ArchiveBox Agent Guide

ArchiveBox is the full self-hosted web archiving app. Keep this repo on the dev branch.

Shared Standards

Use uv and uv run for Python commands. Do not use system python, direct .venv/bin/python, or pip commands.
Prefer existing repo patterns, helper APIs, fixtures, scripts, and command surfaces.
Keep edits focused and minimal. Do not add wrappers, shims, aliases, or extra abstraction layers unless the current code path requires them.
Do not weaken assertions, skip tests, xfail tests, or accept flaky behavior.
No mocks, monkeypatches, fakes, simulated handlers, fake binaries, fake hooks, fake buses, or direct shortcuts around user-facing flows.
Tests and verification should use real CLI commands, REST/API calls, browser UI flows, real hooks, real installs, real subprocesses, real DB rows, real files, and existing fixtures.
Assertions must verify real correctness: exit codes, returned values, DB state, filesystem contents, field values, rendered output, and side effects.
Start behavior fixes with a red failing test when a test is requested or practical.
Trace root causes from observed behavior. Do not paper over failures with retries, wider timeouts, broad fallbacks, or looser assertions.
Read README.md for the full setup, CLI, Docker, API, and release surface.

Concurrency Contract

A collection has one orchestrator at a time. Local PID/process checks may warn about obvious same-machine duplicates, but must not claim to enforce ownership across machines or shared filesystems.
SQLite remains supported with concurrent short writes from CLI and server processes. Keep the SQLite database on a local filesystem, never NFS/SMB, and use short autocommit/CAS updates instead of long transactions or database locks.
Network calls, hook execution, filesystem migrations, and other long work belong in the orchestrator. Never hold a database transaction or lock across that work.
PostgreSQL and shared data directories are the path to future multi-machine scheduling. Coordinate that work through database-backed per-crawl/per-snapshot claims; do not introduce file leases or timer-based orchestrator election.

Development Setup

uv sync --dev --all-extras
mkdir -p data
cd data
uv run --project .. archivebox init --install

Run collection commands from inside an initialized data directory:

cd data
uv run --project .. archivebox status
uv run --project .. archivebox add --plugins=parse_txt_urls 'https://example.com/'
uv run --project .. archivebox run

User-Facing Setup

Recommended CLI install:

uv tool install --python 3.13 --prerelease explicit --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
mkdir -p ~/archivebox/data
cd ~/archivebox/data
archivebox init --install
archivebox add --plugins=parse_txt_urls 'https://example.com/'

Alternative install methods:

Docker Compose / Docker
Homebrew
Debian package
pip

Basic Usage

cd ~/archivebox/data
archivebox version
archivebox help
archivebox status
archivebox install
archivebox add --plugins=parse_txt_urls 'https://example.com/docs-basic-usage'
archivebox list --json --with-headers
archivebox search 'example'
archivebox update --filter-type=domain example.com
archivebox remove --yes --delete --filter-type=exact 'https://example.com/docs-basic-usage'
archivebox run

Verification

Use targeted tests for focused work:

uv run pytest archivebox/tests/test_cli_add.py::test_add_help_shows_depth_and_tag_options -q
uv run prek run --all-files

Releases are published only by .github/workflows/release.yml after the complete dev CI workflow succeeds. Local development and deployment commands must not publish packages, images, tags, or GitHub releases.

3.7 KiB Raw Blame History