From a729790b8edffa792faa4628991757700f4209c4 Mon Sep 17 00:00:00 2001 From: Nick Sweeting Date: Sun, 14 Jun 2026 17:26:49 -0700 Subject: [PATCH] chore: publish local release fixes --- AGENTS.md | 99 ++++++- Dockerfile | 1 - archivebox/services/archive_result_service.py | 26 +- .../tests/test_archive_result_service.py | 51 +++- pyproject.toml | 3 +- skills/archivebox/SKILL.md | 259 +++--------------- 6 files changed, 191 insertions(+), 248 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6077e39c..ae6089e5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,14 +1,87 @@ -I found 31 unique `no mocking` prompts in recent Codex history, across 21 session transcripts. Only one transcript literally started with `no mocking`; most had it later as testing guidance. Consolidated advice: +# ArchiveBox Agent Guide -- Tests must hit real user-facing code paths: CLI commands, REST/API calls, browser UI, real hooks, real ArchiveBox data dirs, real pytest fixtures, and real subprocess/binary behavior. -- No mocking, faking, simulating, monkey patching, handwritten fake objects, fake buses, fake hooks, fake binaries, fake handlers, or direct-post shortcuts when the user path is through UI/extension/CLI. -- No skipped, xfailed, flaky, or “works around platform” tests. Flakiness is treated as a bug, especially on macOS/browser flows. -- Prefer live integration tests over narrow unit tests when behavior depends on browsers, binaries, ArchiveBox crawls, plugins, LLMs, or server state. -- Assertions must validate real correctness: returned values, exit codes, DB rows, filesystem contents, field values, uploaded files, rendered output, and side effects. “No error occurred” or “attribute exists” is not enough. -- Start fixes with failing red tests that reproduce the missing behavior or regression, then implement the minimal fix and confirm the test passes. -- Use realistic setup patterns “like a user would”: events + bus + handlers, real browser pages/CDP sessions, real URLs or `pytest-httpserver`, real rows, real snapshots, real installs, real local browser/server state. -- For ArchiveBox/API tests, use existing `conftest.py` fixtures and test harnesses, real test DB rows/data dirs, and user-facing commands/APIs rather than bespoke helpers. -- For browser/extension tests, trigger behavior through the real extension UI or actual browser session, not direct posting or mocked browser/session objects. -- For binary/provider tests, use real binaries and real installs; verify constraints and final installed package metadata, not just install success. -- For coverage quality, keep tests strict, deterministic, grouped consistently, and use a few larger realistic tests when that gives better surface coverage than many tiny fake unit tests. -- Avoid weakening test coverage, adding compatibility/shim/fallback layers, or guessing from code shape. Trace root causes, verify assumptions with tests/scripts, and let real type/parse errors surface normally. +ArchiveBox is the full self-hosted web archiving app. Keep this repo on the `dev` branch. + +## Shared Standards + +- Use `uv` and `uv run` for Python commands. Do not use system `python`, direct `.venv/bin/python`, or `pip` commands. +- Prefer existing repo patterns, helper APIs, fixtures, scripts, and command surfaces. +- Keep edits focused and minimal. Do not add wrappers, shims, aliases, or extra abstraction layers unless the current code path requires them. +- Do not weaken assertions, skip tests, xfail tests, or accept flaky behavior. +- No mocks, monkeypatches, fakes, simulated handlers, fake binaries, fake hooks, fake buses, or direct shortcuts around user-facing flows. +- Tests and verification should use real CLI commands, REST/API calls, browser UI flows, real hooks, real installs, real subprocesses, real DB rows, real files, and existing fixtures. +- Assertions must verify real correctness: exit codes, returned values, DB state, filesystem contents, field values, rendered output, and side effects. +- Start behavior fixes with a red failing test when a test is requested or practical. +- Trace root causes from observed behavior. Do not paper over failures with retries, wider timeouts, broad fallbacks, or looser assertions. +- Read `README.md` for the full setup, CLI, Docker, API, and release surface. + +## Development Setup + +```bash +uv sync --dev --all-extras +mkdir -p data +cd data +uv run --project .. archivebox init --install +``` + +Run collection commands from inside an initialized data directory: + +```bash +cd data +uv run --project .. archivebox status +uv run --project .. archivebox add 'https://example.com' +uv run --project .. archivebox run +``` + +## User-Facing Setup + +Recommended CLI install: + +```bash +uv tool install archivebox +mkdir -p ~/archivebox/data +cd ~/archivebox/data +archivebox init --install +archivebox add 'https://example.com' +archivebox server 0.0.0.0:8000 +``` + +Alternative install methods: + +- Docker Compose / Docker +- Homebrew +- Debian package +- pip + +## Basic Usage + +```bash +archivebox version +archivebox help +archivebox status +archivebox install +archivebox add 'https://example.com' +archivebox add --extract=title,screenshot,pdf 'https://example.com' +archivebox list --json --with-headers +archivebox search 'example' +archivebox update --filter-type=domain example.com +archivebox remove --filter-type=exact 'https://example.com' +archivebox run +``` + +## Verification + +Use targeted tests for focused work: + + +```bash +uv run pytest archivebox/tests/test_cli_add.py -q +uv run prek run --all-files +``` + +Use the full release/deploy loop only when requested: + + +```bash +./bin/release_dev_stack.sh +``` diff --git a/Dockerfile b/Dockerfile index de8e235f..5ede94c7 100644 --- a/Dockerfile +++ b/Dockerfile @@ -51,7 +51,6 @@ ENV CODE_DIR=/app \ ABXPKG_LIB_DIR=/opt/archivebox/lib \ PLAYWRIGHT_BROWSERS_PATH=/opt/archivebox/lib/playwright/cache \ PERSONAS_DIR=/data/personas \ - CHROME_USER_DATA_DIR=/data/personas/Default/chrome_profile \ CHROME_HEADLESS=true \ CHROME_SANDBOX=false \ CHROME_ISOLATION=crawl \ diff --git a/archivebox/services/archive_result_service.py b/archivebox/services/archive_result_service.py index b5a7abb7..15151514 100644 --- a/archivebox/services/archive_result_service.py +++ b/archivebox/services/archive_result_service.py @@ -237,12 +237,12 @@ def _should_update_snapshot_title(current_title: str, next_title: str, *, snapsh return len(next_title) > len(current) -def _has_content_files(output_files: Any) -> bool: - return any(Path(path).suffix not in {".log", ".pid", ".sh"} for path in _normalize_output_files(output_files)) - - -def _is_signal_interrupted_exit(exit_code: int) -> bool: - return exit_code < 0 or (exit_code >= 128 and exit_code != PROCESS_EXIT_SKIPPED) +def _status_for_process_without_archive_result(event: ProcessCompletedEvent) -> str: + if event.exit_code == PROCESS_EXIT_SKIPPED: + return "skipped" + if event.exit_code != 0: + return "failed" + return "noresults" def _iter_archiveresult_records(stdout: str) -> list[dict]: @@ -444,24 +444,14 @@ class ArchiveResultService(BaseService): ).now() return - # TODO: consider moving this fallback derivation into abx-dl itself. - # First try both patterns: if the whole abx-dl process crashes, restarting - # the snapshot may be enough, but don't guess before validating it. - process_interrupted = _is_signal_interrupted_exit(event.exit_code) - process_failed = event.exit_code not in (0, PROCESS_EXIT_SKIPPED) and not process_interrupted + process_failed = _status_for_process_without_archive_result(event) == "failed" with _perf_span("archivebox.ArchiveResultService.on_ProcessCompletedEvent.emit_archive_result_fallback"): await event.emit( ArchiveResultEvent( snapshot_id=snapshot_event.snapshot_id, plugin=event.plugin_name, hook_name=event.hook_name, - status=( - "queued" - if process_interrupted - else "failed" - if process_failed - else ("succeeded" if _has_content_files(event.output_files) else "noresult") - ), + status=_status_for_process_without_archive_result(event), output_str=event.stderr if process_failed else "", output_files=event.output_files, start_ts=event.start_ts, diff --git a/archivebox/tests/test_archive_result_service.py b/archivebox/tests/test_archive_result_service.py index c310d732..6e7583a1 100644 --- a/archivebox/tests/test_archive_result_service.py +++ b/archivebox/tests/test_archive_result_service.py @@ -3,7 +3,7 @@ import pytest from abxpkg.binary_service import BinaryRequestEvent -from abx_dl.events import ArchiveResultEvent, ProcessEvent, ProcessStartedEvent +from abx_dl.events import ArchiveResultEvent, ProcessCompletedEvent, ProcessEvent, ProcessStartedEvent, SnapshotEvent from abx_dl.orchestrator import create_bus from abx_dl.output_files import OutputFile @@ -331,6 +331,55 @@ def test_process_completed_projects_noresults_archiveresult(): assert result.output_str == "No title found" +def test_process_completed_without_archive_result_does_not_infer_success_from_output_files(snapshot): + from archivebox.core.models import ArchiveResult + from archivebox.services.archive_result_service import ArchiveResultService + import asyncio + + plugin_dir = Path(snapshot.output_dir) / "wget" + plugin_dir.mkdir(parents=True, exist_ok=True) + (plugin_dir / "index.html").write_text("downloaded but not reported") + + bus = create_bus(name="test_process_completed_without_archive_result_output_files") + ArchiveResultService(bus) + + snapshot_event = SnapshotEvent( + url=snapshot.url, + snapshot_id=str(snapshot.id), + output_dir=str(snapshot.output_dir), + ) + completed_event = ProcessCompletedEvent( + plugin_name="wget", + hook_name="on_Snapshot__06_wget.finite.bg", + hook_path="/usr/bin/env", + hook_args=[], + env={}, + timeout=60, + stdout="", + stderr="", + exit_code=0, + status="succeeded", + output_dir=str(plugin_dir), + output_files=[OutputFile(path="index.html", extension="html", mimetype="text/html", size=36)], + start_ts="2026-03-22T12:00:00+00:00", + end_ts="2026-03-22T12:00:01+00:00", + event_parent_id=snapshot_event.event_id, + ) + + async def emit_events() -> None: + await bus.emit(snapshot_event).now() + await bus.emit(completed_event).now() + await bus.wait_until_idle() + + asyncio.run(emit_events()) + + result = ArchiveResult.objects.get(snapshot=snapshot, plugin="wget", hook_name="on_Snapshot__06_wget.finite.bg") + assert result.status == ArchiveResult.StatusChoices.NORESULTS + assert result.output_str == "" + assert result.output_files == {"index.html": {"extension": "html", "mimetype": "text/html", "size": 36}} + _cleanup_machine_process_rows() + + def test_retry_failed_archiveresults_requeues_snapshot_in_queued_state(): from archivebox.core.models import ArchiveResult, Snapshot diff --git a/pyproject.toml b/pyproject.toml index 9fe4bf53..3d4c0225 100755 --- a/pyproject.toml +++ b/pyproject.toml @@ -242,7 +242,8 @@ ignore-words-list = "abx,archivebox,adminsnapshots,bu,wit,dont,cant,wont,havent, skip = "*.json,*.min.js,*.min.css,uv.lock,old/*,publicsite/*,docs/apidocs/*" [tool.pytest.ini_options] -testpaths = [ "archivebox/tests" ] +testpaths = [ "archivebox/tests", "README.md", "AGENTS.md", "skills", "docs" ] +addopts = ["--codeblocks"] norecursedirs = ["archivebox/tests/data"] timeout = 600 timeout_method = "thread" diff --git a/skills/archivebox/SKILL.md b/skills/archivebox/SKILL.md index af40e160..255e424f 100644 --- a/skills/archivebox/SKILL.md +++ b/skills/archivebox/SKILL.md @@ -1,250 +1,81 @@ --- name: archivebox -description: Use this when an agent needs to install, run, automate, inspect, or troubleshoot ArchiveBox collections. Covers setup, data-folder workflow, CLI usage, Docker usage, Admin UI, REST API, SQLite index.sqlite3 access, filesystem layout, and `archivebox shell -c '...'` Django ORM snippets. +description: Use this when working on the ArchiveBox app, CLI, server, Docker image, Admin UI, REST API, data dirs, crawls, snapshots, and release/deploy scripts. --- # ArchiveBox -Use this skill when an agent needs to operate a full ArchiveBox collection: add URLs, run the server, inspect archived snapshots, automate via API, query `index.sqlite3`, or use the Django shell/ORM. +## Purpose -ArchiveBox is the full self-hosted archiving app. For one-off extraction without a collection/server, prefer the `abx-dl` skill. +ArchiveBox is the full self-hosted web archiving app. Use this skill for collection operations, app code, Docker, Admin UI, API, and release work. -## Core Model +## Shared Rules -- Always `cd` into the ArchiveBox data directory before running `archivebox ...`. -- A data directory contains `ArchiveBox.conf`, `index.sqlite3`, and `archive/`. -- Run `archivebox init` before other collection commands; it is safe to rerun and applies migrations. -- Do not rely on `DATA_DIR=/path/to/data archivebox ...` for agent workflows; commands must be run from inside the initialized `data/` directory. -- Avoid running local/dev ArchiveBox commands as root. The official Docker image handles its own runtime environment. +- Keep this repo on branch `dev`. +- Use `uv` and `uv run` for Python commands. +- Do not use system `python`, direct `.venv/bin/python`, or `pip` commands. +- Use existing ArchiveBox CLI/API/UI/runner paths for setup and side effects. +- Do not mock, monkeypatch, fake, simulate, skip, xfail, or weaken tests. +- Verify behavior with real commands, real data dirs, real DB rows, real hooks, real browsers, and real filesystem outputs. +- Read `README.md` for the full setup, CLI, Docker, API, and release surface. -Current default layout: - -```text -data/ - ArchiveBox.conf - index.sqlite3 - archive/ - users/ - / - crawls/ - YYYYMMDD/ - / - / - ...crawl-level state - snapshots/ - YYYYMMDD/ - / - / - index.jsonl - / - ...plugin output files -``` - - -## Setup - -Preferred Docker Compose workflow: +## Development Setup ```bash -mkdir -p ~/archivebox/data -cd ~/archivebox -curl -fsSL 'https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/dev/docker-compose.yml' -o docker-compose.yml -docker compose run archivebox init -docker compose up -``` - -Local checkout workflow: - -```bash -cd /path/to/ArchiveBox/archivebox uv sync --dev --all-extras -mkdir -p ./data -cd ./data +mkdir -p data +cd data uv run --project .. archivebox init --install ``` -Published CLI workflow: +## User-Facing Setup + +Recommended CLI install: ```bash +uv tool install archivebox mkdir -p ~/archivebox/data cd ~/archivebox/data archivebox init --install -``` - -## Basic CLI Usage - -Run commands from inside the data directory: - -```bash -cd ~/archivebox/data -archivebox version -archivebox help -archivebox status archivebox add 'https://example.com' -archivebox add --depth=1 'https://news.ycombinator.com' -echo 'https://example.com' | archivebox add -archivebox list --json --with-headers > index.json -archivebox list --html --with-headers > index.html -archivebox search 'example' -archivebox update --filter-type=domain example.com -archivebox remove --filter-type=exact 'https://example.com' -``` - -Docker equivalents: - -```bash -docker compose run archivebox init --install -docker compose run archivebox add --depth=1 'https://example.com' -echo 'https://example.com' | docker compose run -T archivebox add -docker compose run -T archivebox list --json --with-headers > index.json -``` - -Useful subcommands: - -- `archivebox init`, `install`, `config`, `status`, `version`, `help` -- `archivebox add`, `update`, `list`, `search`, `remove`, `schedule` -- `archivebox server`, `manage`, `shell` -- Model-oriented commands: `crawl`, `snapshot`, `archiveresult`, `tag`, `binary`, `process`, `machine`, `persona` - -## Configuration - -ArchiveBox config can come from environment variables, `ArchiveBox.conf`, or `archivebox config --set`. - -```bash -archivebox config -archivebox config --get CHROME_BINARY -archivebox config --set TIMEOUT=240 -archivebox config --set CHECK_SSL_VALIDITY=False -archivebox config --set PUBLIC_INDEX=False PUBLIC_SNAPSHOTS=False PUBLIC_ADD_VIEW=False -CHROME_BINARY=chromium archivebox add 'https://example.com' -``` - -Common knobs: - -- `TIMEOUT`, `USER_AGENT`, `CHECK_SSL_VALIDITY` -- `PUBLIC_INDEX`, `PUBLIC_SNAPSHOTS`, `PUBLIC_ADD_VIEW` -- `CHROME_BINARY`, `SAVE_WGET`, `SAVE_DOM`, `SAVE_PDF`, `SAVE_SCREENSHOT` -- `ABXPKG_LIB_DIR`, `TMP_DIR` - -## Admin UI - -Create an admin user and start the server: - -```bash -archivebox manage createsuperuser archivebox server 0.0.0.0:8000 ``` -Docker Compose: +Alternative install methods: + +- Docker Compose / Docker +- Homebrew +- Debian package +- pip + +## Basic Usage + +Run from inside an initialized data dir: ```bash -docker compose run archivebox manage createsuperuser -docker compose up +archivebox version +archivebox status +archivebox install +archivebox add 'https://example.com' +archivebox add --extract=title,screenshot,pdf 'https://example.com' +archivebox list --json --with-headers +archivebox search 'example' +archivebox update --filter-type=domain example.com +archivebox remove --filter-type=exact 'https://example.com' +archivebox run ``` -Open: - -- Public UI: `http://web.archivebox.localhost:8000` -- Admin UI: `http://admin.archivebox.localhost:8000` -- API docs: `http://web.archivebox.localhost:8000/api/v1/docs` - -If custom hostnames are not resolving, try `/admin` and `/api/v1/docs` on the server base URL. - -## REST API - -The REST API is alpha and served by Django Ninja. Always check the live Swagger docs for the exact schema: - -```text -GET /api/v1/docs -GET /api/v1/openapi.json -``` - -Get an API token: +## Verification + ```bash -curl -sS -X POST 'http://web.archivebox.localhost:8000/api/v1/auth/get_api_token' \ - -H 'Content-Type: application/json' \ - -d '{"username":"admin","password":"password"}' +uv run pytest archivebox/tests/test_cli_add.py -q +uv run prek run --all-files ``` -Authenticate with any of: +Use the full release/deploy loop only when requested: + ```bash -Authorization: Bearer -X-ArchiveBox-API-Key: -?api_key= +./bin/release_dev_stack.sh ``` - -Useful endpoints: - -- `POST /api/v1/cli/add` mirrors `archivebox add` and queues background work. -- `GET /api/v1/core/snapshots` -- `GET /api/v1/core/snapshot/{snapshot_id}` -- `POST /api/v1/core/snapshots` -- `GET /api/v1/crawls/crawls` -- `POST /api/v1/crawls/crawls` -- `GET /api/v1/machine/binaries` - -Example add via API: - -```bash -TOKEN='...' -curl -sS -X POST 'http://web.archivebox.localhost:8000/api/v1/cli/add' \ - -H "Authorization: Bearer $TOKEN" \ - -H 'Content-Type: application/json' \ - -d '{"urls":["https://example.com"],"depth":0,"tag":"agent-run"}' -``` - -Keep `archivebox server` running to process queued UI/API jobs. - -## SQLite Index - -The main metadata database is `index.sqlite3` in the data directory. Prefer the ORM for writes. Direct SQLite is useful for read-only inspection, exports, and debugging. - -```bash -sqlite3 ./index.sqlite3 '.tables' -sqlite3 ./index.sqlite3 'SELECT COUNT(*) FROM core_snapshot;' -sqlite3 ./index.sqlite3 "SELECT timestamp, url, title, status FROM core_snapshot ORDER BY bookmarked_at DESC LIMIT 20;" -sqlite3 ./index.sqlite3 "SELECT plugin, status, COUNT(*) FROM core_archiveresult GROUP BY plugin, status ORDER BY plugin, status;" -sqlite3 ./index.sqlite3 "SELECT id, status, substr(urls, 1, 120) FROM crawls_crawl ORDER BY created_at DESC LIMIT 20;" -``` - -Useful tables usually include: - -- `core_snapshot`: URL, title, timestamp, status, crawl linkage, output size. -- `core_archiveresult`: extractor/plugin run status and output metadata. -- `core_tag`, `core_snapshottag`: tags and snapshot-tag links. -- `crawls_crawl`: crawl batches, queued URLs, config, status. -- Django tables such as `auth_user` and `django_migrations`. - -For large or active archives, avoid long write transactions and keep `index.sqlite3` on local disk/SSD when possible. - -## Django Shell - -Use `archivebox shell` for interactive ORM access and `archivebox shell -c '...'` for agent-friendly one-liners. Run from the data directory. - -Read-only examples: - -```bash -archivebox shell -c 'from archivebox.core.models import Snapshot; print(Snapshot.objects.count())' -archivebox shell -c 'from archivebox.core.models import Snapshot; print(list(Snapshot.objects.values("timestamp", "url", "title", "status").order_by("-bookmarked_at")[:10]))' -archivebox shell -c 'from archivebox.core.models import ArchiveResult; print(list(ArchiveResult.objects.values("plugin", "status").order_by("plugin", "status").distinct()))' -archivebox shell -c 'from archivebox.crawls.models import Crawl; print(list(Crawl.objects.values("id", "status", "urls").order_by("-created_at")[:5]))' -``` - -Safe write examples: - -```bash -archivebox shell -c 'from archivebox.core.models import Snapshot; s=Snapshot.objects.get(timestamp="1700000000"); s.notes="reviewed by agent"; s.save(update_fields=["notes", "modified_at"])' -archivebox shell -c 'from archivebox.core.models import Snapshot, Tag; s=Snapshot.objects.get(timestamp="1700000000"); t,_=Tag.objects.get_or_create(name="important"); s.tags.add(t)' -``` - -Prefer `archivebox add`, the REST API, or model methods for creating crawl/snapshot work. Use direct ORM writes only when you understand the model lifecycle. - -## Recommended Agent Workflow - -1. Identify the data directory and run `archivebox status`. -2. Use `archivebox config` to confirm privacy, timeout, and browser settings before ingesting sensitive URLs. -3. Add URLs with `archivebox add` or `POST /api/v1/cli/add`. -4. Inspect progress with the Admin UI, `archivebox list`, `archivebox shell -c`, or read-only SQLite queries. -5. Use filesystem outputs under `archive/` for artifacts, but use `index.sqlite3`/ORM/API for authoritative metadata.