Add DATABASE_ENGINE=postgres (plus DATABASE_HOST/PORT/USER/PASSWORD/NAME)
config and centralize all sqlite-vs-postgres branching in
archivebox.misc.db:
- get_database_settings() builds DATABASES for either backend; the sqlite
path is unchanged (custom lock-retry backend, same PRAGMAs).
- database_exists()/ensure_database_ready() replace index.sqlite3 file
checks; init auto-creates the postgres database when missing.
- approximate_row_counts() serves admin index counts from sqlite_stat1 or
pg_class.reltuples; missing-table detection covers both vendors.
- rebuild_models_from_migration_state() lets historical sqlite-only raw
SQL migrations resync postgres schema from Django migration state at
every divergence point (postgres can never hold legacy data, so
affected tables are empty when these run). All raw-DDL and PRAGMA
migrations are now vendor-gated with sqlite behavior byte-for-byte
unchanged.
- A pre_save clamp truncates CharField values to max_length: sqlite
never enforced varchar(n) but postgres does (e.g. long crawl labels).
- Collation-sensitive URL range scans branch to escaped LIKE on postgres
(with a text_pattern_ops index) since linguistic collations break
bytewise range tricks; the crawl-config JSON search wave gets a
jsonb-text implementation.
Verified on real PostgreSQL 16: fresh init applies the entire migration
graph, schema matches models exactly (column-level parity check +
makemigrations --check), and add/run/list/search/status/remove all work
end-to-end. New test_postgres_backend.py suite boots a real throwaway
postgres cluster (initdb + pg_ctl); CI workflows install postgres server
binaries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YHSjZM6TstSAMN2PhgfUg
Remove imports of deleted folder utility functions and rewrite
status command to query Snapshot model directly. This aligns with
the fs_version refactor where the DB is the single source of truth.
- Use Snapshot.objects queries for indexed/archived/unarchived counts
- Scan filesystem directly for present/orphaned directory counts
- Simplify output to focus on essential status information
