The_Great_Archive/ArchiveBox
1
0
Fork 0
mirror of https://github.com/ArchiveBox/ArchiveBox.git synced 2026-09-12 19:50:57 +05:00
Code Issues Actions 4 Packages Projects Releases Wiki Activity

chore: publish local release fixes

Browse Source
This commit is contained in:
Nick Sweeting 2026-06-14 17:26:49 -07:00
parent 322b4cbe3f
commit a729790b8e
No known key found for this signature in database
6 changed files with 191 additions and 248 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

99
AGENTS.md
View File

@ -1,14 +1,87 @@
I found 31 unique `no mocking` prompts in recent Codex history, across 21 session transcripts. Only one transcript literally started with `no mocking`; most had it later as testing guidance. Consolidated advice:
# ArchiveBox Agent Guide
- Tests must hit real user-facing code paths: CLI commands, REST/API calls, browser UI, real hooks, real ArchiveBox data dirs, real pytest fixtures, and real subprocess/binary behavior.
- No mocking, faking, simulating, monkey patching, handwritten fake objects, fake buses, fake hooks, fake binaries, fake handlers, or direct-post shortcuts when the user path is through UI/extension/CLI.
- No skipped, xfailed, flaky, or “works around platform” tests. Flakiness is treated as a bug, especially on macOS/browser flows.
- Prefer live integration tests over narrow unit tests when behavior depends on browsers, binaries, ArchiveBox crawls, plugins, LLMs, or server state.
- Assertions must validate real correctness: returned values, exit codes, DB rows, filesystem contents, field values, uploaded files, rendered output, and side effects. “No error occurred” or “attribute exists” is not enough.
- Start fixes with failing red tests that reproduce the missing behavior or regression, then implement the minimal fix and confirm the test passes.
- Use realistic setup patterns “like a user would”: events + bus + handlers, real browser pages/CDP sessions, real URLs or `pytest-httpserver`, real rows, real snapshots, real installs, real local browser/server state.
- For ArchiveBox/API tests, use existing `conftest.py` fixtures and test harnesses, real test DB rows/data dirs, and user-facing commands/APIs rather than bespoke helpers.
- For browser/extension tests, trigger behavior through the real extension UI or actual browser session, not direct posting or mocked browser/session objects.
- For binary/provider tests, use real binaries and real installs; verify constraints and final installed package metadata, not just install success.
- For coverage quality, keep tests strict, deterministic, grouped consistently, and use a few larger realistic tests when that gives better surface coverage than many tiny fake unit tests.
- Avoid weakening test coverage, adding compatibility/shim/fallback layers, or guessing from code shape. Trace root causes, verify assumptions with tests/scripts, and let real type/parse errors surface normally.
ArchiveBox is the full self-hosted web archiving app. Keep this repo on the `dev` branch.
## Shared Standards
- Use `uv` and `uv run` for Python commands. Do not use system `python`, direct `.venv/bin/python`, or `pip` commands.
- Prefer existing repo patterns, helper APIs, fixtures, scripts, and command surfaces.
- Keep edits focused and minimal. Do not add wrappers, shims, aliases, or extra abstraction layers unless the current code path requires them.
- Do not weaken assertions, skip tests, xfail tests, or accept flaky behavior.
- No mocks, monkeypatches, fakes, simulated handlers, fake binaries, fake hooks, fake buses, or direct shortcuts around user-facing flows.
- Tests and verification should use real CLI commands, REST/API calls, browser UI flows, real hooks, real installs, real subprocesses, real DB rows, real files, and existing fixtures.
- Assertions must verify real correctness: exit codes, returned values, DB state, filesystem contents, field values, rendered output, and side effects.
- Start behavior fixes with a red failing test when a test is requested or practical.
- Trace root causes from observed behavior. Do not paper over failures with retries, wider timeouts, broad fallbacks, or looser assertions.
- Read `README.md` for the full setup, CLI, Docker, API, and release surface.
## Development Setup
```bash
uv sync --dev --all-extras
mkdir -p data
cd data
uv run --project .. archivebox init --install
```
Run collection commands from inside an initialized data directory:
```bash
cd data
uv run --project .. archivebox status
uv run --project .. archivebox add 'https://example.com'
uv run --project .. archivebox run
```
## User-Facing Setup
Recommended CLI install:
```bash
uv tool install archivebox
mkdir -p ~/archivebox/data
cd ~/archivebox/data
archivebox init --install
archivebox add 'https://example.com'
archivebox server 0.0.0.0:8000
```
Alternative install methods:
- Docker Compose / Docker
- Homebrew
- Debian package
- pip
## Basic Usage
```bash
archivebox version
archivebox help
archivebox status
archivebox install
archivebox add 'https://example.com'
archivebox add --extract=title,screenshot,pdf 'https://example.com'
archivebox list --json --with-headers
archivebox search 'example'
archivebox update --filter-type=domain example.com
archivebox remove --filter-type=exact 'https://example.com'
archivebox run
```
## Verification
Use targeted tests for focused work:
<!--pytest.mark.skip(reason="pytest invocation")-->
```bash
uv run pytest archivebox/tests/test_cli_add.py -q
uv run prek run --all-files
```
Use the full release/deploy loop only when requested:
<!--pytest.mark.skip(reason="release/deploy script")-->
```bash
./bin/release_dev_stack.sh
```

1
Dockerfile
View File

@ -51,7 +51,6 @@ ENV CODE_DIR=/app \
ABXPKG_LIB_DIR=/opt/archivebox/lib \
PLAYWRIGHT_BROWSERS_PATH=/opt/archivebox/lib/playwright/cache \
PERSONAS_DIR=/data/personas \
CHROME_USER_DATA_DIR=/data/personas/Default/chrome_profile \
CHROME_HEADLESS=true \
CHROME_SANDBOX=false \
CHROME_ISOLATION=crawl \

26
archivebox/services/archive_result_service.py
View File

@ -237,12 +237,12 @@ def _should_update_snapshot_title(current_title: str, next_title: str, *, snapsh
return len(next_title) > len(current)
def _has_content_files(output_files: Any) -> bool:
return any(Path(path).suffix not in {".log", ".pid", ".sh"} for path in _normalize_output_files(output_files))
def _is_signal_interrupted_exit(exit_code: int) -> bool:
return exit_code < 0 or (exit_code >= 128 and exit_code != PROCESS_EXIT_SKIPPED)
def _status_for_process_without_archive_result(event: ProcessCompletedEvent) -> str:
if event.exit_code == PROCESS_EXIT_SKIPPED:
return "skipped"
if event.exit_code != 0:
return "failed"
return "noresults"
def _iter_archiveresult_records(stdout: str) -> list[dict]:
@ -444,24 +444,14 @@ class ArchiveResultService(BaseService):
).now()
return
# TODO: consider moving this fallback derivation into abx-dl itself.
# First try both patterns: if the whole abx-dl process crashes, restarting
# the snapshot may be enough, but don't guess before validating it.
process_interrupted = _is_signal_interrupted_exit(event.exit_code)
process_failed = event.exit_code not in (0, PROCESS_EXIT_SKIPPED) and not process_interrupted
process_failed = _status_for_process_without_archive_result(event) == "failed"
with _perf_span("archivebox.ArchiveResultService.on_ProcessCompletedEvent.emit_archive_result_fallback"):
await event.emit(
ArchiveResultEvent(
snapshot_id=snapshot_event.snapshot_id,
plugin=event.plugin_name,
hook_name=event.hook_name,
status=(
"queued"
if process_interrupted
else "failed"
if process_failed
else ("succeeded" if _has_content_files(event.output_files) else "noresult")
),
status=_status_for_process_without_archive_result(event),
output_str=event.stderr if process_failed else "",
output_files=event.output_files,
start_ts=event.start_ts,

51
archivebox/tests/test_archive_result_service.py
View File

@ -3,7 +3,7 @@ import pytest
from abxpkg.binary_service import BinaryRequestEvent
from abx_dl.events import ArchiveResultEvent, ProcessEvent, ProcessStartedEvent
from abx_dl.events import ArchiveResultEvent, ProcessCompletedEvent, ProcessEvent, ProcessStartedEvent, SnapshotEvent
from abx_dl.orchestrator import create_bus
from abx_dl.output_files import OutputFile
@ -331,6 +331,55 @@ def test_process_completed_projects_noresults_archiveresult():
assert result.output_str == "No title found"
def test_process_completed_without_archive_result_does_not_infer_success_from_output_files(snapshot):
from archivebox.core.models import ArchiveResult
from archivebox.services.archive_result_service import ArchiveResultService
import asyncio
plugin_dir = Path(snapshot.output_dir) / "wget"
plugin_dir.mkdir(parents=True, exist_ok=True)
(plugin_dir / "index.html").write_text("<html>downloaded but not reported</html>")
bus = create_bus(name="test_process_completed_without_archive_result_output_files")
ArchiveResultService(bus)
snapshot_event = SnapshotEvent(
url=snapshot.url,
snapshot_id=str(snapshot.id),
output_dir=str(snapshot.output_dir),
)
completed_event = ProcessCompletedEvent(
plugin_name="wget",
hook_name="on_Snapshot__06_wget.finite.bg",
hook_path="/usr/bin/env",
hook_args=[],
env={},
timeout=60,
stdout="",
stderr="",
exit_code=0,
status="succeeded",
output_dir=str(plugin_dir),
output_files=[OutputFile(path="index.html", extension="html", mimetype="text/html", size=36)],
start_ts="2026-03-22T12:00:00+00:00",
end_ts="2026-03-22T12:00:01+00:00",
event_parent_id=snapshot_event.event_id,
)
async def emit_events() -> None:
await bus.emit(snapshot_event).now()
await bus.emit(completed_event).now()
await bus.wait_until_idle()
asyncio.run(emit_events())
result = ArchiveResult.objects.get(snapshot=snapshot, plugin="wget", hook_name="on_Snapshot__06_wget.finite.bg")
assert result.status == ArchiveResult.StatusChoices.NORESULTS
assert result.output_str == ""
assert result.output_files == {"index.html": {"extension": "html", "mimetype": "text/html", "size": 36}}
_cleanup_machine_process_rows()
def test_retry_failed_archiveresults_requeues_snapshot_in_queued_state():
from archivebox.core.models import ArchiveResult, Snapshot

3
pyproject.toml
View File

@ -242,7 +242,8 @@ ignore-words-list = "abx,archivebox,adminsnapshots,bu,wit,dont,cant,wont,havent,
skip = "*.json,*.min.js,*.min.css,uv.lock,old/*,publicsite/*,docs/apidocs/*"
[tool.pytest.ini_options]
testpaths = [ "archivebox/tests" ]
testpaths = [ "archivebox/tests", "README.md", "AGENTS.md", "skills", "docs" ]
addopts = ["--codeblocks"]
norecursedirs = ["archivebox/tests/data"]
timeout = 600
timeout_method = "thread"

259
skills/archivebox/SKILL.md
View File

@ -1,250 +1,81 @@
---
name: archivebox
description: Use this when an agent needs to install, run, automate, inspect, or troubleshoot ArchiveBox collections. Covers setup, data-folder workflow, CLI usage, Docker usage, Admin UI, REST API, SQLite index.sqlite3 access, filesystem layout, and `archivebox shell -c '...'` Django ORM snippets.
description: Use this when working on the ArchiveBox app, CLI, server, Docker image, Admin UI, REST API, data dirs, crawls, snapshots, and release/deploy scripts.
---
# ArchiveBox
Use this skill when an agent needs to operate a full ArchiveBox collection: add URLs, run the server, inspect archived snapshots, automate via API, query `index.sqlite3`, or use the Django shell/ORM.
## Purpose
ArchiveBox is the full self-hosted archiving app. For one-off extraction without a collection/server, prefer the `abx-dl` skill.
ArchiveBox is the full self-hosted web archiving app. Use this skill for collection operations, app code, Docker, Admin UI, API, and release work.
## Core Model
## Shared Rules
- Always `cd` into the ArchiveBox data directory before running `archivebox ...`.
- A data directory contains `ArchiveBox.conf`, `index.sqlite3`, and `archive/`.
- Run `archivebox init` before other collection commands; it is safe to rerun and applies migrations.
- Do not rely on `DATA_DIR=/path/to/data archivebox ...` for agent workflows; commands must be run from inside the initialized `data/` directory.
- Avoid running local/dev ArchiveBox commands as root. The official Docker image handles its own runtime environment.
- Keep this repo on branch `dev`.
- Use `uv` and `uv run` for Python commands.
- Do not use system `python`, direct `.venv/bin/python`, or `pip` commands.
- Use existing ArchiveBox CLI/API/UI/runner paths for setup and side effects.
- Do not mock, monkeypatch, fake, simulate, skip, xfail, or weaken tests.
- Verify behavior with real commands, real data dirs, real DB rows, real hooks, real browsers, and real filesystem outputs.
- Read `README.md` for the full setup, CLI, Docker, API, and release surface.
Current default layout:
```text
data/
ArchiveBox.conf
index.sqlite3
archive/
users/
<username>/
crawls/
YYYYMMDD/
<domain>/
<crawl-uuid>/
...crawl-level state
snapshots/
YYYYMMDD/
<domain>/
<snapshot-uuid>/
index.jsonl
<plugin>/
...plugin output files
```
## Setup
Preferred Docker Compose workflow:
## Development Setup
```bash
mkdir -p ~/archivebox/data
cd ~/archivebox
curl -fsSL 'https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/dev/docker-compose.yml' -o docker-compose.yml
docker compose run archivebox init
docker compose up
```
Local checkout workflow:
```bash
cd /path/to/ArchiveBox/archivebox
uv sync --dev --all-extras
mkdir -p ./data
cd ./data
mkdir -p data
cd data
uv run --project .. archivebox init --install
```
Published CLI workflow:
## User-Facing Setup
Recommended CLI install:
```bash
uv tool install archivebox
mkdir -p ~/archivebox/data
cd ~/archivebox/data
archivebox init --install
```
## Basic CLI Usage
Run commands from inside the data directory:
```bash
cd ~/archivebox/data
archivebox version
archivebox help
archivebox status
archivebox add 'https://example.com'
archivebox add --depth=1 'https://news.ycombinator.com'
echo 'https://example.com' | archivebox add
archivebox list --json --with-headers > index.json
archivebox list --html --with-headers > index.html
archivebox search 'example'
archivebox update --filter-type=domain example.com
archivebox remove --filter-type=exact 'https://example.com'
```
Docker equivalents:
```bash
docker compose run archivebox init --install
docker compose run archivebox add --depth=1 'https://example.com'
echo 'https://example.com' | docker compose run -T archivebox add
docker compose run -T archivebox list --json --with-headers > index.json
```
Useful subcommands:
- `archivebox init`, `install`, `config`, `status`, `version`, `help`
- `archivebox add`, `update`, `list`, `search`, `remove`, `schedule`
- `archivebox server`, `manage`, `shell`
- Model-oriented commands: `crawl`, `snapshot`, `archiveresult`, `tag`, `binary`, `process`, `machine`, `persona`
## Configuration
ArchiveBox config can come from environment variables, `ArchiveBox.conf`, or `archivebox config --set`.
```bash
archivebox config
archivebox config --get CHROME_BINARY
archivebox config --set TIMEOUT=240
archivebox config --set CHECK_SSL_VALIDITY=False
archivebox config --set PUBLIC_INDEX=False PUBLIC_SNAPSHOTS=False PUBLIC_ADD_VIEW=False
CHROME_BINARY=chromium archivebox add 'https://example.com'
```
Common knobs:
- `TIMEOUT`, `USER_AGENT`, `CHECK_SSL_VALIDITY`
- `PUBLIC_INDEX`, `PUBLIC_SNAPSHOTS`, `PUBLIC_ADD_VIEW`
- `CHROME_BINARY`, `SAVE_WGET`, `SAVE_DOM`, `SAVE_PDF`, `SAVE_SCREENSHOT`
- `ABXPKG_LIB_DIR`, `TMP_DIR`
## Admin UI
Create an admin user and start the server:
```bash
archivebox manage createsuperuser
archivebox server 0.0.0.0:8000
```
Docker Compose:
Alternative install methods:
- Docker Compose / Docker
- Homebrew
- Debian package
- pip
## Basic Usage
Run from inside an initialized data dir:
```bash
docker compose run archivebox manage createsuperuser
docker compose up
archivebox version
archivebox status
archivebox install
archivebox add 'https://example.com'
archivebox add --extract=title,screenshot,pdf 'https://example.com'
archivebox list --json --with-headers
archivebox search 'example'
archivebox update --filter-type=domain example.com
archivebox remove --filter-type=exact 'https://example.com'
archivebox run
```
Open:
- Public UI: `http://web.archivebox.localhost:8000`
- Admin UI: `http://admin.archivebox.localhost:8000`
- API docs: `http://web.archivebox.localhost:8000/api/v1/docs`
If custom hostnames are not resolving, try `/admin` and `/api/v1/docs` on the server base URL.
## REST API
The REST API is alpha and served by Django Ninja. Always check the live Swagger docs for the exact schema:
```text
GET /api/v1/docs
GET /api/v1/openapi.json
```
Get an API token:
## Verification
<!--pytest.mark.skip(reason="pytest invocation")-->
```bash
curl -sS -X POST 'http://web.archivebox.localhost:8000/api/v1/auth/get_api_token' \
-H 'Content-Type: application/json' \
-d '{"username":"admin","password":"password"}'
uv run pytest archivebox/tests/test_cli_add.py -q
uv run prek run --all-files
```
Authenticate with any of:
Use the full release/deploy loop only when requested:
<!--pytest.mark.skip(reason="release/deploy script")-->
```bash
Authorization: Bearer <token>
X-ArchiveBox-API-Key: <token>
?api_key=<token>
./bin/release_dev_stack.sh
```
Useful endpoints:
- `POST /api/v1/cli/add` mirrors `archivebox add` and queues background work.
- `GET /api/v1/core/snapshots`
- `GET /api/v1/core/snapshot/{snapshot_id}`
- `POST /api/v1/core/snapshots`
- `GET /api/v1/crawls/crawls`
- `POST /api/v1/crawls/crawls`
- `GET /api/v1/machine/binaries`
Example add via API:
```bash
TOKEN='...'
curl -sS -X POST 'http://web.archivebox.localhost:8000/api/v1/cli/add' \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"urls":["https://example.com"],"depth":0,"tag":"agent-run"}'
```
Keep `archivebox server` running to process queued UI/API jobs.
## SQLite Index
The main metadata database is `index.sqlite3` in the data directory. Prefer the ORM for writes. Direct SQLite is useful for read-only inspection, exports, and debugging.
```bash
sqlite3 ./index.sqlite3 '.tables'
sqlite3 ./index.sqlite3 'SELECT COUNT(*) FROM core_snapshot;'
sqlite3 ./index.sqlite3 "SELECT timestamp, url, title, status FROM core_snapshot ORDER BY bookmarked_at DESC LIMIT 20;"
sqlite3 ./index.sqlite3 "SELECT plugin, status, COUNT(*) FROM core_archiveresult GROUP BY plugin, status ORDER BY plugin, status;"
sqlite3 ./index.sqlite3 "SELECT id, status, substr(urls, 1, 120) FROM crawls_crawl ORDER BY created_at DESC LIMIT 20;"
```
Useful tables usually include:
- `core_snapshot`: URL, title, timestamp, status, crawl linkage, output size.
- `core_archiveresult`: extractor/plugin run status and output metadata.
- `core_tag`, `core_snapshottag`: tags and snapshot-tag links.
- `crawls_crawl`: crawl batches, queued URLs, config, status.
- Django tables such as `auth_user` and `django_migrations`.
For large or active archives, avoid long write transactions and keep `index.sqlite3` on local disk/SSD when possible.
## Django Shell
Use `archivebox shell` for interactive ORM access and `archivebox shell -c '...'` for agent-friendly one-liners. Run from the data directory.
Read-only examples:
```bash
archivebox shell -c 'from archivebox.core.models import Snapshot; print(Snapshot.objects.count())'
archivebox shell -c 'from archivebox.core.models import Snapshot; print(list(Snapshot.objects.values("timestamp", "url", "title", "status").order_by("-bookmarked_at")[:10]))'
archivebox shell -c 'from archivebox.core.models import ArchiveResult; print(list(ArchiveResult.objects.values("plugin", "status").order_by("plugin", "status").distinct()))'
archivebox shell -c 'from archivebox.crawls.models import Crawl; print(list(Crawl.objects.values("id", "status", "urls").order_by("-created_at")[:5]))'
```
Safe write examples:
```bash
archivebox shell -c 'from archivebox.core.models import Snapshot; s=Snapshot.objects.get(timestamp="1700000000"); s.notes="reviewed by agent"; s.save(update_fields=["notes", "modified_at"])'
archivebox shell -c 'from archivebox.core.models import Snapshot, Tag; s=Snapshot.objects.get(timestamp="1700000000"); t,_=Tag.objects.get_or_create(name="important"); s.tags.add(t)'
```
Prefer `archivebox add`, the REST API, or model methods for creating crawl/snapshot work. Use direct ORM writes only when you understand the model lifecycle.
## Recommended Agent Workflow
1. Identify the data directory and run `archivebox status`.
2. Use `archivebox config` to confirm privacy, timeout, and browser settings before ingesting sensitive URLs.
3. Add URLs with `archivebox add` or `POST /api/v1/cli/add`.
4. Inspect progress with the Admin UI, `archivebox list`, `archivebox shell -c`, or read-only SQLite queries.
5. Use filesystem outputs under `archive/` for artifacts, but use `index.sqlite3`/ORM/API for authoritative metadata.