Helper reduction:
- Move the Django DATABASES/SQLITE_CONNECTION_OPTIONS assembly into
core/settings.py, dropping get_database_settings() and
get_sqlite_connection_options() from misc/db.py.
- Inline the single-use migration_table_columns() into its one migration.
Review fixes:
- search: match only scalar JSON *values* on postgres (jsonb_path_query
over '$.**' scalar leaves), mirroring SQLite json_tree.atom so config
keys no longer match.
- CharField clamp now also runs in SnapshotQuerySet.bulk_create (bulk paths
bypass the pre_save signal); truncate_overlong_charfields is dual-use.
- Restore reverse-migration parity on postgres: crawls/machine/api initial
migrations drop their rebuilt tables on reverse via drop_models_on_postgres.
- docs: give DATABASE_NAME its own section so the anchor resolves correctly.
- CI: only install postgres binaries on the shard that runs the postgres test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YHSjZM6TstSAMN2PhgfUg
Re-lock uv.lock with the CI-pinned uv version so the diff is limited to the
psycopg addition (a newer local uv had rewritten platform markers and
exclude-newer, breaking `uv sync --locked` in CI).
Consolidate the misc/db.py adapter surface: fold database_backend() into
is_postgres(), drop the redundant vendor-name constants, remove the unused
migration_table_exists() helper, and inline the single-use missing-table check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YHSjZM6TstSAMN2PhgfUg
Add DATABASE_ENGINE=postgres (plus DATABASE_HOST/PORT/USER/PASSWORD/NAME)
config and centralize all sqlite-vs-postgres branching in
archivebox.misc.db:
- get_database_settings() builds DATABASES for either backend; the sqlite
path is unchanged (custom lock-retry backend, same PRAGMAs).
- database_exists()/ensure_database_ready() replace index.sqlite3 file
checks; init auto-creates the postgres database when missing.
- approximate_row_counts() serves admin index counts from sqlite_stat1 or
pg_class.reltuples; missing-table detection covers both vendors.
- rebuild_models_from_migration_state() lets historical sqlite-only raw
SQL migrations resync postgres schema from Django migration state at
every divergence point (postgres can never hold legacy data, so
affected tables are empty when these run). All raw-DDL and PRAGMA
migrations are now vendor-gated with sqlite behavior byte-for-byte
unchanged.
- A pre_save clamp truncates CharField values to max_length: sqlite
never enforced varchar(n) but postgres does (e.g. long crawl labels).
- Collation-sensitive URL range scans branch to escaped LIKE on postgres
(with a text_pattern_ops index) since linguistic collations break
bytewise range tricks; the crawl-config JSON search wave gets a
jsonb-text implementation.
Verified on real PostgreSQL 16: fresh init applies the entire migration
graph, schema matches models exactly (column-level parity check +
makemigrations --check), and add/run/list/search/status/remove all work
end-to-end. New test_postgres_backend.py suite boots a real throwaway
postgres cluster (initdb + pg_ctl); CI workflows install postgres server
binaries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YHSjZM6TstSAMN2PhgfUg
The column change to a variable-length, still-indexed TextField is what
matters for supporting long URLs efficiently — no reason to lower the
supported limit. Adjust the long-URL tests to the real 65535 boundary.
Snapshot.url was a CharField(max_length=65535) which reserves a fixed-width
column and is too long to index or constrain on real DB backends. Store it as a
variable-length TextField instead, so short URLs don't waste space and very long
URLs (up to MAX_URL_LENGTH=8000) are supported, while keeping a normal index on
the field so exact, prefix, and substring (icontains) URL lookups all stay fast.
- misc/util.py: MAX_URL_LENGTH 65535 -> 8000 (the practical web-server limit)
- core/models.py: Snapshot.url CharField -> TextField(db_index=True)
- migration 0049_alter_snapshot_url
- tests covering 8000-char persistence, exact/prefix/substring lookups,
over-length rejection, and per-crawl uniqueness for long URLs
https://claude.ai/code/session_01BLnGTL5GSoouD4ihaYp55n
Renames (no functional change, just consistency with the rest of the codebase):
- cli/cli_utils.py → cli/cli_util.py
- core/host_utils.py → core/host_util.py
- core/tag_utils.py → core/tag_util.py
- crawls/schedule_utils.py → crawls/schedule_util.py
- machine/env_utils.py → machine/env_util.py
Functional fixes:
- archivebox add --index-only now materializes Snapshot rows synchronously
via crawl.create_snapshots_from_urls() instead of just queueing the Crawl
and leaving the index empty. The previous behavior broke every test that
expected --index-only to populate the index, since the runner is never
started in index-only mode.
- config/collection.py: add _coerce_from_str_dict as the inverse of
_coerce_to_str_dict so JSON-encoded INI values are decoded back to native
dict/list types when mirrored into Machine.config (a JSONField). Without
this, downstream consumers like MachineEvent / abx-dl get raw JSON
strings where they expect dicts.
Plus matching admin / middleware / model touch-ups, the registration
password_change_form template, and assorted small cleanups the user
worked through while validating the deploy path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The newer-DB guard added in 65dc2521 refused to start on any collection
whose django_migrations table referenced migrations that have since been
squashed away from source (core/0023→0075, api/0002→0009, machine/0002→
0004, plus every transient dev-branch rename). That's the state of every
beta-tester DB that was upgraded incrementally through the 0.8.x → 0.9.x
rc chain — cabbage's demo deployment was the first concrete casualty.
Django's standard `replaces=` pattern doesn't fit here: its all-or-none
semantics split the migration graph whenever only a subset of the listed
replaces is applied, which is exactly what happens when different beta
testers stopped at different intermediate dev branches. Instead, keep an
explicit HISTORICAL_GHOST_MIGRATIONS allowlist in misc/db.py enumerating
every squashed-away name, and subtract it from missing_from_code in
migration_state(). The hard SystemExit(3) for truly unknown future names
stays — only known-historical ghosts are absorbed.
Adds test_init_recovers_from_pre_squash_dev_history covering the gap the
pre-existing migration tests missed (they only seed published 0.4/0.7/
0.8 schemas, never a post-squash dev DB).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Snapshot detail page: embed scoped live-progress monitor (same-origin
/progress.json on whichever host the page is served from); hide admin
action buttons when scoped; per-snapshot perms via can_view_snapshot.
- crawl_file API: respect crawl-level permissions; PUBLIC/UNLISTED served
to guests, PRIVATE returns 404 for non-admin/non-owner.
- CrawlRunner: replace allow_paused_snapshot_maintenance with
allow_maintenance_on_inactive_crawl so SEALED crawls don't short-circuit
the cancellation guard for legitimate maintenance hooks (search backend
backfill, fs migration, etc.). Fixes infinite STARTED loop on snapshots
with queued search_backend results.
- Universal `--init` flag: works on any subcommand (server, update, add,
shell, install, ...). Detected at module load, stripped from argv, and
consumed in the dispatcher so subprocesses inherit a clean env.
- supervisord_util.run_runner_worker: route Ctrl+C through
supervisor.signalProcess(name, "SIGINT") instead of raw os.kill on a
cached pid, gated on statename=RUNNING. Prevents killing unrelated
processes when the worker's pid has been reused by the OS.
- Login page: remove non-functional password-reset links; add
has_real_admin_users template tag to gate the bootstrap hint.
- Add page: hide underline on the "Get the extension" link.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
