Helper reduction:
- Move the Django DATABASES/SQLITE_CONNECTION_OPTIONS assembly into
core/settings.py, dropping get_database_settings() and
get_sqlite_connection_options() from misc/db.py.
- Inline the single-use migration_table_columns() into its one migration.
Review fixes:
- search: match only scalar JSON *values* on postgres (jsonb_path_query
over '$.**' scalar leaves), mirroring SQLite json_tree.atom so config
keys no longer match.
- CharField clamp now also runs in SnapshotQuerySet.bulk_create (bulk paths
bypass the pre_save signal); truncate_overlong_charfields is dual-use.
- Restore reverse-migration parity on postgres: crawls/machine/api initial
migrations drop their rebuilt tables on reverse via drop_models_on_postgres.
- docs: give DATABASE_NAME its own section so the anchor resolves correctly.
- CI: only install postgres binaries on the shard that runs the postgres test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YHSjZM6TstSAMN2PhgfUg
Add DATABASE_ENGINE=postgres (plus DATABASE_HOST/PORT/USER/PASSWORD/NAME)
config and centralize all sqlite-vs-postgres branching in
archivebox.misc.db:
- get_database_settings() builds DATABASES for either backend; the sqlite
path is unchanged (custom lock-retry backend, same PRAGMAs).
- database_exists()/ensure_database_ready() replace index.sqlite3 file
checks; init auto-creates the postgres database when missing.
- approximate_row_counts() serves admin index counts from sqlite_stat1 or
pg_class.reltuples; missing-table detection covers both vendors.
- rebuild_models_from_migration_state() lets historical sqlite-only raw
SQL migrations resync postgres schema from Django migration state at
every divergence point (postgres can never hold legacy data, so
affected tables are empty when these run). All raw-DDL and PRAGMA
migrations are now vendor-gated with sqlite behavior byte-for-byte
unchanged.
- A pre_save clamp truncates CharField values to max_length: sqlite
never enforced varchar(n) but postgres does (e.g. long crawl labels).
- Collation-sensitive URL range scans branch to escaped LIKE on postgres
(with a text_pattern_ops index) since linguistic collations break
bytewise range tricks; the crawl-config JSON search wave gets a
jsonb-text implementation.
Verified on real PostgreSQL 16: fresh init applies the entire migration
graph, schema matches models exactly (column-level parity check +
makemigrations --check), and add/run/list/search/status/remove all work
end-to-end. New test_postgres_backend.py suite boots a real throwaway
postgres cluster (initdb + pg_ctl); CI workflows install postgres server
binaries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YHSjZM6TstSAMN2PhgfUg
Direct URL inputs from CLI/UI/API now seed Crawl.urls as explicit
{type:CrawlSeed,url,depth} JSONL rows; raw stdin/UI/API import text
stays verbatim. The runner's create_initial_snapshots() is now the
single place that either expands seed rows or creates the synthetic
archivebox://internal root + staticfile/stdin.txt, so add paths no
longer perform DB/FS side effects and the parser hooks run through
the same Snapshot lifecycle as every other extractor.
Pre-0.9.0 crawl tables had ``max_urls``, ``crawl_max_size``, and
``snapshot_max_size`` as NOT NULL ``IntegerField`` columns. The 0011
migration RemoveField'd them, but on long-lived DBs where a historical
0011 ran with different semantics the columns stayed on disk while
being absent from the model. Every Crawl.objects.create() through the
ORM (e.g. the /add/ form view) skips them, and SQLite refuses the
insert with "NOT NULL constraint failed: crawls_crawl.max_urls" —
an HTTP 500 on /add/. Cabbage hit this on the rc51 UI test.
The new 0017 migration introspects the live table, copies any
pre-existing values into ``config`` under the canonical
CRAWL_MAX_URLS / CRAWL_MAX_SIZE / SNAPSHOT_MAX_SIZE keys so nothing
gets silently dropped, then ALTER TABLE DROP COLUMNs each legacy
field. Fresh installs already have these columns removed and the
migration is a no-op there. Requires SQLite 3.35+ for DROP COLUMN.
Plus: bin/take_screenshot.js bumps Puppeteer's CDP protocolTimeout
from the 30s default to 5 minutes — admin pages with many DB-backed
partials (snapshot grid w/ 23K rows, progress monitor, etc.) blow
past 30s on the screenshot capture under load.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Beta-tester / cabbage-style DBs upgraded incrementally through the
0.8.x → 0.9.x rc chain have crawls/0013_crawl_permissions,
personas/0003_persona_permissions, and core/0041_snapshot_permissions
all marked applied in django_migrations — but the historical migrations
with those names predate the current GeneratedField design. The columns
they were supposed to add never actually landed on the tables. When the
downstream hydration migrations (crawls/0016_hydrate_crawl_permissions,
personas/0004_hydrate_persona_permissions) run and try to filter on
.permissions, the query fails with no such column: permissions and
bricks startup.
Add a defensive _ensure_permissions_column pass at the top of each
hydration migration that ALTER TABLEs the column in if absent. For the
snapshot side (no hydration migration to attach to) introduce a new
core/0046_repair_snapshot_permissions migration that runs the same
guard. Fresh installs already have the column from the initial 0013 /
0003 / 0041 migrations, so the guard no-ops there.
SQLite ALTER TABLE ADD COLUMN only accepts VIRTUAL generated columns
("cannot add a STORED column"), so the repaired columns evaluate the
JSON extract on read rather than write — runtime queries behave
identically.
Verified end-to-end against actual cabbage data (23517 snapshots,
229 crawls, 103122 archive_results): container now boots healthy on
rc48 with the legacy schema in place.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Renames (no functional change, just consistency with the rest of the codebase):
- cli/cli_utils.py → cli/cli_util.py
- core/host_utils.py → core/host_util.py
- core/tag_utils.py → core/tag_util.py
- crawls/schedule_utils.py → crawls/schedule_util.py
- machine/env_utils.py → machine/env_util.py
Functional fixes:
- archivebox add --index-only now materializes Snapshot rows synchronously
via crawl.create_snapshots_from_urls() instead of just queueing the Crawl
and leaving the index empty. The previous behavior broke every test that
expected --index-only to populate the index, since the runner is never
started in index-only mode.
- config/collection.py: add _coerce_from_str_dict as the inverse of
_coerce_to_str_dict so JSON-encoded INI values are decoded back to native
dict/list types when mirrored into Machine.config (a JSONField). Without
this, downstream consumers like MachineEvent / abx-dl get raw JSON
strings where they expect dicts.
Plus matching admin / middleware / model touch-ups, the registration
password_change_form template, and assorted small cleanups the user
worked through while validating the deploy path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Snapshot detail page: embed scoped live-progress monitor (same-origin
/progress.json on whichever host the page is served from); hide admin
action buttons when scoped; per-snapshot perms via can_view_snapshot.
- crawl_file API: respect crawl-level permissions; PUBLIC/UNLISTED served
to guests, PRIVATE returns 404 for non-admin/non-owner.
- CrawlRunner: replace allow_paused_snapshot_maintenance with
allow_maintenance_on_inactive_crawl so SEALED crawls don't short-circuit
the cancellation guard for legitimate maintenance hooks (search backend
backfill, fs migration, etc.). Fixes infinite STARTED loop on snapshots
with queued search_backend results.
- Universal `--init` flag: works on any subcommand (server, update, add,
shell, install, ...). Detected at module load, stripped from argv, and
consumed in the dispatcher so subprocesses inherit a clean env.
- supervisord_util.run_runner_worker: route Ctrl+C through
supervisor.signalProcess(name, "SIGINT") instead of raw os.kill on a
cached pid, gated on statename=RUNNING. Prevents killing unrelated
processes when the worker's pid has been reused by the OS.
- Login page: remove non-functional password-reset links; add
has_real_admin_users template tag to gate the bootstrap hint.
- Add page: hide underline on the "Get the extension" link.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
