mirror of
https://github.com/ArchiveBox/ArchiveBox.git
synced 2026-09-12 19:50:57 +05:00
88 lines
3.7 KiB
Markdown
88 lines
3.7 KiB
Markdown
# ArchiveBox Agent Guide
|
|
|
|
ArchiveBox is the full self-hosted web archiving app. Keep this repo on the `dev` branch.
|
|
|
|
## Shared Standards
|
|
|
|
- Use `uv` and `uv run` for Python commands. Do not use system `python`, direct `.venv/bin/python`, or `pip` commands.
|
|
- Prefer existing repo patterns, helper APIs, fixtures, scripts, and command surfaces.
|
|
- Keep edits focused and minimal. Do not add wrappers, shims, aliases, or extra abstraction layers unless the current code path requires them.
|
|
- Do not weaken assertions, skip tests, xfail tests, or accept flaky behavior.
|
|
- No mocks, monkeypatches, fakes, simulated handlers, fake binaries, fake hooks, fake buses, or direct shortcuts around user-facing flows.
|
|
- Tests and verification should use real CLI commands, REST/API calls, browser UI flows, real hooks, real installs, real subprocesses, real DB rows, real files, and existing fixtures.
|
|
- Assertions must verify real correctness: exit codes, returned values, DB state, filesystem contents, field values, rendered output, and side effects.
|
|
- Start behavior fixes with a red failing test when a test is requested or practical.
|
|
- Trace root causes from observed behavior. Do not paper over failures with retries, wider timeouts, broad fallbacks, or looser assertions.
|
|
- Read `README.md` for the full setup, CLI, Docker, API, and release surface.
|
|
|
|
## Concurrency Contract
|
|
|
|
- A collection has one orchestrator at a time. Local PID/process checks may warn about obvious same-machine duplicates, but must not claim to enforce ownership across machines or shared filesystems.
|
|
- SQLite remains supported with concurrent short writes from CLI and server processes. Keep the SQLite database on a local filesystem, never NFS/SMB, and use short autocommit/CAS updates instead of long transactions or database locks.
|
|
- Network calls, hook execution, filesystem migrations, and other long work belong in the orchestrator. Never hold a database transaction or lock across that work.
|
|
- PostgreSQL and shared data directories are the path to future multi-machine scheduling. Coordinate that work through database-backed per-crawl/per-snapshot claims; do not introduce file leases or timer-based orchestrator election.
|
|
|
|
## Development Setup
|
|
|
|
```bash
|
|
uv sync --dev --all-extras
|
|
mkdir -p data
|
|
cd data
|
|
uv run --project .. archivebox init --install
|
|
```
|
|
|
|
Run collection commands from inside an initialized data directory:
|
|
|
|
```bash
|
|
cd data
|
|
uv run --project .. archivebox status
|
|
uv run --project .. archivebox add --plugins=parse_txt_urls 'https://example.com/'
|
|
uv run --project .. archivebox run
|
|
```
|
|
|
|
## User-Facing Setup
|
|
|
|
Recommended CLI install:
|
|
|
|
```bash
|
|
uv tool install --python 3.13 --prerelease explicit --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
|
|
mkdir -p ~/archivebox/data
|
|
cd ~/archivebox/data
|
|
archivebox init --install
|
|
archivebox add --plugins=parse_txt_urls 'https://example.com/'
|
|
```
|
|
|
|
Alternative install methods:
|
|
|
|
- Docker Compose / Docker
|
|
- Homebrew
|
|
- Debian package
|
|
- pip
|
|
|
|
## Basic Usage
|
|
|
|
```bash
|
|
cd ~/archivebox/data
|
|
archivebox version
|
|
archivebox help
|
|
archivebox status
|
|
archivebox install
|
|
archivebox add --plugins=parse_txt_urls 'https://example.com/docs-basic-usage'
|
|
archivebox list --json --with-headers
|
|
archivebox search 'example'
|
|
archivebox update --filter-type=domain example.com
|
|
archivebox remove --yes --delete --filter-type=exact 'https://example.com/docs-basic-usage'
|
|
archivebox run
|
|
```
|
|
|
|
## Verification
|
|
|
|
Use targeted tests for focused work:
|
|
|
|
```bash
|
|
uv run pytest archivebox/tests/test_cli_add.py::test_add_help_shows_depth_and_tag_options -q
|
|
uv run prek run --all-files
|
|
```
|
|
|
|
Releases are published only by `.github/workflows/release.yml` after the complete `dev` CI workflow succeeds. Local development and deployment commands must not publish packages, images, tags, or GitHub releases.
|