Clean rebuild of docs/apidocs/archivebox/ via autodoc2:
- Removes 19 stale module pages whose source files no longer exist
(cli_utils, host_utils, schedule_utils — renamed to *_util; actors,
ideas, debugging, folders, legacy, progress_layout, tests_piping,
config_tags, personas.runtime/views, orchestrator*, worker, tasks).
- Adds 29 new module pages for code that was added since the previous
generation but not yet documented.
- Updates 100 existing pages to reflect API surface changes (e.g.
ENABLED_PLUGINS Field removed, SNAPSHOTS_PER_PAGE clamp removed,
default bumped to 50, etc.).
156 source modules <-> 156 apidoc files (zero drift). Build clean
under sphinx-build -W --keep-going.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
admin_snapshots.py:571 had min(max(50, SNAPSHOTS_PER_PAGE), 500), and
admin_archiveresults.py:501 had min(max(5, SNAPSHOTS_PER_PAGE), 5000).
Both clamps silently overrode the configured value — a documented
default of 40 was inaccessible in the Snapshot admin, and the
ArchiveResult admin also reused the same setting without being mentioned
in the docs.
- Drop both clamps; admin changelists now use SNAPSHOTS_PER_PAGE as-is.
- Bump the default in common.py from 40 to 50 (matches what users were
actually seeing in the admin under the old floor).
- Add ge=1 validation so non-positive values are rejected at config
parse time instead of producing broken pagination.
- Update Configuration.md: new default 50, clarify the option drives
both Snapshot and ArchiveResult admin changelists plus the public
index, and that it must be >= 1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Several inaccuracies + over-documentation cleaned up in one pass:
- ONLY_NEW: completely rewrite. The old prose ("ArchiveBox will never
re-download sites that have already succeeded previously") was carried
over from 0.7.x and is wrong in 0.9.x — setting ONLY_NEW=False (or
--no-only-new) explicitly creates a new Snapshot and re-runs every
extractor. Now describes the actual behavior: skip URL entirely vs.
create a new Snapshot for it.
- CRAWL_MAX_CONCURRENT_SNAPSHOTS: fix the "each concurrent Snapshot
launches its own Chrome instance" claim. Chrome is crawl-scoped by
default (CHROME_ISOLATION="crawl") — concurrent Snapshots share the
crawl's Chrome via tabs, not separate browser processes.
- BASE_URL: drop the "admin.admin.admin.<host> compounding bug"
reference. Config docs shouldn't explain legacy bugs.
- Remove derived/runtime-only options that are NOT user-settable:
ACTIVE_PERSONA (set by persona resolver), CRAWL_DIR/SNAP_DIR (injected
by orchestrator per-call), DATA_DIR (derived from cwd), ARCHIVE_DIR
(derived from DATA_DIR/archive), USERS_DIR (derived from ARCHIVE_DIR),
PERSONAS_DIR (derived from DATA_DIR), LIB_BIN_DIR (tracks LIB_DIR),
DATABASE_NAME (derived from DATA_DIR/index.sqlite3). Backward-compat
<a id="..."></a> anchors preserved for all of them above the nearest
surviving heading so external links still resolve.
- LIB_DIR: fix default path. The doc claimed "<DATA_DIR>/lib/<arch>-<os>"
but constants.py:117 uses platformdirs.user_config_path("abx") / "lib"
— the XDG user-config dir, not inside the data folder. Updated to the
actual default.
- ENABLED_PLUGINS section dropped (option removed in a separate commit);
anchor redirected to PLUGINS.
- Drop the "Pydantic config" implementation-detail mention in PUID/PGID.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ENABLED_PLUGINS and PLUGINS were two near-identical config keys: PLUGINS
was the CLI/per-run whitelist (--plugins flag, runner config), while
ENABLED_PLUGINS was the UI/API "persisted enabled set" — but both ended
up steering the same plugin resolution. Consolidating on PLUGINS as the
single source of truth.
- archivebox/config/common.py: drop the ENABLED_PLUGINS Field entirely
(no alias, no compat shim — the migration is one-shot).
- archivebox/hooks.py:get_enabled_plugins(): read PLUGINS instead of
ENABLED_PLUGINS. Function name kept (describes the return value).
- archivebox/templates/core/add.html: admin "Add" form JS now writes to
PLUGINS; help text updated to reference PLUGINS.
views.py:1302 and runner.py:585 already read/wrote PLUGINS; they're now
consistent with the resolver.
abx-dl is unaffected — it receives selected_plugins as a Python list
argument and never reads either config key.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This rewrite (now reapplied on top of the wiki subtree) covers the full
session's work on Configuration.md:
- Add crawl/snapshot limits (CRAWL_MAX_URLS/SIZE/TIMEOUT,
CRAWL_MAX_CONCURRENT_SNAPSHOTS, SNAPSHOT_MAX_SIZE), DELETE_AFTER,
PERMISSIONS, PLUGINS/ENABLED_PLUGINS/ACTIVE_PERSONA.
- Add new Database Settings section (SQLITE_* tuning + DATABASE_NAME).
- Add SERVER_SECURITY_MODE deep-dive (4 modes, host-layout table).
- Add Storage path overrides (DATA_DIR, ARCHIVE_DIR, USERS_DIR,
PERSONAS_DIR, CRAWL_DIR, SNAP_DIR, ALLOW_NO_UNIX_SOCKETS).
- Remove ALLOWED_HOSTS + CSRF_TRUSTED_ORIGINS as user-settable; both
auto-derived from BASE_URL + SERVER_SECURITY_MODE. Backward-compat
anchors preserved on BASE_URL with the 0.7.3 -> 0.9 legacy upgrade note.
- Remove the entire Plugin Settings tree (~200 options, 41 subsections);
replace with prominent redirect to https://archivebox.github.io/abx-plugins/
and a "shared core options that plugins fall back to" table.
- Add 231 backward-compat <a id="..."></a> anchors so old URLs to plugin
sections / removed options / multi-option headers all still resolve
(e.g. #wget_args -> Plugin Configuration section, #public_snapshots ->
PERMISSIONS, #ssl_enabled -> Plugin Configuration, #admin_username ->
ADMIN_USERNAME/PASSWORD heading, #dir_output_permissions ->
OUTPUT_PERMISSIONS, #url_blacklist -> URL_DENYLIST).
- Fix wrong default: PUBLIC_ADD_VIEW is False, not True.
- Drop the 7 TRAFILATURA_OUTPUT_* per-format flags (replaced by single
TRAFILATURA_OUTPUT_FORMATS in plugin); SSL_ENABLED/SSL_TIMEOUT (wrong
plugin namespace) — anchors redirected to Plugin Configuration.
- Reframe COOKIES_FILE as low-level escape hatch; personas are the
preferred auth path.
- Link every named plugin to its specific anchor on the abx-plugins page
(e.g. WGET_TIMEOUT -> #wget, SONIC_HOST -> #search_backend_sonic).
- Strip implementation-detail mentions (Pydantic, etc.).
- Slim Shell Options to only user-settable (DEBUG, USE_COLOR,
SHOW_PROGRESS); drop IS_TTY/IN_DOCKER/IN_QEMU.
- Restructure: General -> Server (+LDAP) -> Storage -> Database (new) ->
Search -> Shell -> Plugin Configuration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Remove ALLOWED_HOSTS + CSRF_TRUSTED_ORIGINS as documented user options;
both are now auto-derived from BASE_URL + SERVER_SECURITY_MODE. Backward-compat
anchors preserved above BASE_URL with a brief 0.7.3->0.9 legacy upgrade note.
- Drop the lone pydantic reference in PUID/PGID; users don't care about
implementation details, only names/defaults/behavior/why.
- Reframe COOKIES_FILE as the low-level escape hatch and surface personas as
the preferred auth path (TIP admonition + persona-first example).
- Link every named plugin override to its specific anchor on the abx-plugins
page (e.g. WGET_TIMEOUT -> #wget, SONIC_HOST -> #search_backend_sonic);
retain the generic top-of-page link only where no single plugin is named.
- Drop spurious CURL_* override examples (no curl plugin exists in abx-plugins).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-0.9.0 crawl tables had ``max_urls``, ``crawl_max_size``, and
``snapshot_max_size`` as NOT NULL ``IntegerField`` columns. The 0011
migration RemoveField'd them, but on long-lived DBs where a historical
0011 ran with different semantics the columns stayed on disk while
being absent from the model. Every Crawl.objects.create() through the
ORM (e.g. the /add/ form view) skips them, and SQLite refuses the
insert with "NOT NULL constraint failed: crawls_crawl.max_urls" —
an HTTP 500 on /add/. Cabbage hit this on the rc51 UI test.
The new 0017 migration introspects the live table, copies any
pre-existing values into ``config`` under the canonical
CRAWL_MAX_URLS / CRAWL_MAX_SIZE / SNAPSHOT_MAX_SIZE keys so nothing
gets silently dropped, then ALTER TABLE DROP COLUMNs each legacy
field. Fresh installs already have these columns removed and the
migration is a no-op there. Requires SQLite 3.35+ for DROP COLUMN.
Plus: bin/take_screenshot.js bumps Puppeteer's CDP protocolTimeout
from the 30s default to 5 minutes — admin pages with many DB-backed
partials (snapshot grid w/ 23K rows, progress monitor, etc.) blow
past 30s on the screenshot capture under load.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Django 6.0 wraps every ASGI request in `async with ThreadSensitiveContext():`
(django/core/handlers/asgi.py:169). On exit, asgiref calls
``executor.shutdown()`` with the default ``wait=True`` (asgiref/sync.py:148),
which is a synchronous ``Thread.join()`` inside an async function — so it
blocks the daphne event loop until the executor's worker thread exits.
That's normally fine: the request handler has already awaited every
``sync_to_async`` it submitted, the worker is idle, and the shutdown
sentinel makes it exit immediately. The blocking becomes catastrophic
when a client disconnects mid-request:
* ``SyncToAsync.__call__`` shields the executor work via
``await asyncio.shield(exec_coro)`` (asgiref/sync.py:506) so the
sync DB call doesn't get torn down halfway.
* On cancellation it calls ``exec_coro.cancel()`` (line 522), which
only flips the asyncio Future to cancelled — the underlying thread
keeps running the SQL query.
* Control unwinds back to ``__aexit__`` while the orphaned thread is
still mid-query. ``shutdown(wait=True)`` blocks the event loop
until that orphan finishes.
Under heavy SQLite write contention (the cabbage load-test scenario:
23K snapshots, 100K+ archive_results, hundreds of inflight extractor
writes) orphan queries routinely take 30s+ waiting for locks. Daphne is
single-threaded, so every orphan stalls every concurrent request. After
a few rounds of Cloudflare-side disconnects, the loop is hung,
healthchecks time out, the container goes ``unhealthy``, and the demo
goes down. py-spy on the stuck process showed every worker idle and the
main thread parked in ``concurrent/futures/thread.py:join`` from
``ThreadSensitiveContext.__aexit__``.
Switching to ``shutdown(wait=False)`` queues the sentinel and returns
immediately; the worker still exits cleanly once its current task
finishes, and asgiref's ``WeakKeyDictionary`` releases the executor as
soon as the request context is GC'd. There was no caller relying on the
per-request "thread is dead before next request starts" guarantee — the
ThreadPoolExecutor only ever has max_workers=1 and is per-context.
Patched in archivebox/core/asgi.py at module import (before
``get_asgi_application()``) so daphne picks it up on first boot.
Idempotent and safe to re-import.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Beta-tester / cabbage-style DBs upgraded incrementally through the
0.8.x → 0.9.x rc chain have crawls/0013_crawl_permissions,
personas/0003_persona_permissions, and core/0041_snapshot_permissions
all marked applied in django_migrations — but the historical migrations
with those names predate the current GeneratedField design. The columns
they were supposed to add never actually landed on the tables. When the
downstream hydration migrations (crawls/0016_hydrate_crawl_permissions,
personas/0004_hydrate_persona_permissions) run and try to filter on
.permissions, the query fails with no such column: permissions and
bricks startup.
Add a defensive _ensure_permissions_column pass at the top of each
hydration migration that ALTER TABLEs the column in if absent. For the
snapshot side (no hydration migration to attach to) introduce a new
core/0046_repair_snapshot_permissions migration that runs the same
guard. Fresh installs already have the column from the initial 0013 /
0003 / 0041 migrations, so the guard no-ops there.
SQLite ALTER TABLE ADD COLUMN only accepts VIRTUAL generated columns
("cannot add a STORED column"), so the repaired columns evaluate the
JSON extract on read rather than write — runtime queries behave
identically.
Verified end-to-end against actual cabbage data (23517 snapshots,
229 crawls, 103122 archive_results): container now boots healthy on
rc48 with the legacy schema in place.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0045_archiveresult_unique_hook adds a UniqueConstraint on
(snapshot, plugin, hook_name) but real long-lived collections (cabbage's
demo, beta-tester DBs) accumulated multiple rows per hook across the dev
rc chain. Without a cleanup pass first, the constraint fails with
``UNIQUE constraint failed`` mid-migration and bricks startup — cabbage
hit this on the rc48 deploy with 1382 duplicate (snapshot, plugin,
hook_name) groups already present.
Add a RunPython dedup pass that keeps the highest-id (most recent) row
per tuple, then applies the constraint. Fresh installs no-op the cleanup
since there are no duplicates to find.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Renames (no functional change, just consistency with the rest of the codebase):
- cli/cli_utils.py → cli/cli_util.py
- core/host_utils.py → core/host_util.py
- core/tag_utils.py → core/tag_util.py
- crawls/schedule_utils.py → crawls/schedule_util.py
- machine/env_utils.py → machine/env_util.py
Functional fixes:
- archivebox add --index-only now materializes Snapshot rows synchronously
via crawl.create_snapshots_from_urls() instead of just queueing the Crawl
and leaving the index empty. The previous behavior broke every test that
expected --index-only to populate the index, since the runner is never
started in index-only mode.
- config/collection.py: add _coerce_from_str_dict as the inverse of
_coerce_to_str_dict so JSON-encoded INI values are decoded back to native
dict/list types when mirrored into Machine.config (a JSONField). Without
this, downstream consumers like MachineEvent / abx-dl get raw JSON
strings where they expect dicts.
Plus matching admin / middleware / model touch-ups, the registration
password_change_form template, and assorted small cleanups the user
worked through while validating the deploy path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The newer-DB guard added in 65dc2521 refused to start on any collection
whose django_migrations table referenced migrations that have since been
squashed away from source (core/0023→0075, api/0002→0009, machine/0002→
0004, plus every transient dev-branch rename). That's the state of every
beta-tester DB that was upgraded incrementally through the 0.8.x → 0.9.x
rc chain — cabbage's demo deployment was the first concrete casualty.
Django's standard `replaces=` pattern doesn't fit here: its all-or-none
semantics split the migration graph whenever only a subset of the listed
replaces is applied, which is exactly what happens when different beta
testers stopped at different intermediate dev branches. Instead, keep an
explicit HISTORICAL_GHOST_MIGRATIONS allowlist in misc/db.py enumerating
every squashed-away name, and subtract it from missing_from_code in
migration_state(). The hard SystemExit(3) for truly unknown future names
stays — only known-historical ghosts are absorbed.
Adds test_init_recovers_from_pre_squash_dev_history covering the gap the
pre-existing migration tests missed (they only seed published 0.4/0.7/
0.8 schemas, never a post-squash dev DB).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Snapshot detail page: embed scoped live-progress monitor (same-origin
/progress.json on whichever host the page is served from); hide admin
action buttons when scoped; per-snapshot perms via can_view_snapshot.
- crawl_file API: respect crawl-level permissions; PUBLIC/UNLISTED served
to guests, PRIVATE returns 404 for non-admin/non-owner.
- CrawlRunner: replace allow_paused_snapshot_maintenance with
allow_maintenance_on_inactive_crawl so SEALED crawls don't short-circuit
the cancellation guard for legitimate maintenance hooks (search backend
backfill, fs migration, etc.). Fixes infinite STARTED loop on snapshots
with queued search_backend results.
- Universal `--init` flag: works on any subcommand (server, update, add,
shell, install, ...). Detected at module load, stripped from argv, and
consumed in the dispatcher so subprocesses inherit a clean env.
- supervisord_util.run_runner_worker: route Ctrl+C through
supervisor.signalProcess(name, "SIGINT") instead of raw os.kill on a
cached pid, gated on statename=RUNNING. Prevents killing unrelated
processes when the worker's pid has been reused by the OS.
- Login page: remove non-functional password-reset links; add
has_real_admin_users template tag to gate the bootstrap hint.
- Add page: hide underline on the "Get the extension" link.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
