## Summary
This PR adds full PostgreSQL database backend support to ArchiveBox,
allowing users to choose between SQLite (default) and PostgreSQL via the
new `DATABASE_ENGINE` configuration option. All
database-backend-specific logic is centralized in a new
`archivebox/misc/db.py` module with helper functions for connection
management, schema initialization, and backend detection.
## Related issues
Enables PostgreSQL as an alternative to SQLite for deployments requiring
better concurrency, larger datasets, or managed database services.
## Changes these areas
- [x] Feature behavior
- [x] Configuration options
- [x] Internal architecture
## Key Changes
### New Database Backend Abstraction (`archivebox/misc/db.py`)
- Centralized all SQLite vs PostgreSQL branching logic in a single
module
- Added helper functions:
- `database_backend()` / `is_postgres()` - detect configured backend
- `get_database_settings()` - generate Django DATABASES config per
backend
- `postgres_db_params()` - extract PostgreSQL connection parameters
- `database_exists()` - check if database is initialized (works before
Django setup)
- `ensure_database_ready()` - verify PostgreSQL server is reachable and
create database if needed
- `approximate_row_counts()` - get optimizer stats from either backend
- `truncate_tables()` / `rebuild_models_from_migration_state()` - schema
management helpers
### Configuration
- Added `DATABASE_ENGINE` config option (default: "sqlite") in
`archivebox/config/common.py`
- Added PostgreSQL-specific config options: `DATABASE_NAME`,
`DATABASE_USER`, `DATABASE_PASSWORD`, `DATABASE_HOST`, `DATABASE_PORT`
- Updated documentation in `docs/Configuration.md`
### Django Settings Integration
- Modified `archivebox/core/settings.py` to use
`get_database_settings()` for dynamic backend selection
- Removed hardcoded SQLite-only connection options
### Migration Strategy
- Refactored existing migrations to support both backends:
- SQLite migrations execute raw DDL via `RunSQL` (byte-for-byte
identical to original)
- PostgreSQL migrations use `rebuild_models_from_migration_state()` to
sync schema from Django models
- Added backend-specific migration functions (`_run_sqlite_only_sql`,
`_pg_sync_schema`)
- Affected migrations: `crawls/0001_initial.py`, `api/0001_initial.py`,
`machine/0001_initial.py`, `core/0024_assign_default_crawl.py`, and
others
### Query Compatibility
- Updated `archivebox/search/query.py` to handle both SQLite's
`json_tree()` and PostgreSQL's `jsonb` operators
- Updated `archivebox/search/views.py` URL prefix search to use
backend-specific range queries
- Added PostgreSQL pattern-ops index migration
(`core/0051_postgres_url_pattern_ops_index.py`) for efficient LIKE
queries
### Testing
- Added comprehensive PostgreSQL backend test suite
(`archivebox/tests/test_postgres_backend.py`)
- Spins up real throwaway PostgreSQL cluster for end-to-end testing
- Tests init, status, add, list, remove operations
- Validates schema parity between models and database
- Requires PostgreSQL server binaries (initdb/pg_ctl)
- Added database benchmarking tool (`bin/benchmark_db_backends.py`) for
performance comparison
### Admin UI
- Updated `archivebox/core/admin_site.py` to use centralized
`approximate_row_counts()` helper (works on both backends)
### CLI Integration
- Updated `archivebox_init.py` to call `ensure_database_ready()` before
migrations (PostgreSQL-specific setup)
- Updated `archivebox_status.py` to use `database_display_location()`
for user-friendly output
## Test Plan
- CI runs new PostgreSQL backend tests on supported platforms (macOS,
Python 3.14)
- Existing SQLite tests continue to pass (no behavioral changes to
default backend)
- Schema parity test validates all Django models match database schema
on PostgreSQL
- Benchmark tool available for performance
https://claude.ai/code/session_019YHSjZM6TstSAMN2PhgfUg
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds first‑class PostgreSQL support alongside SQLite with backend‑aware
migrations and queries. SQLite stays the default; nothing changes unless
you set `DATABASE_ENGINE=postgres`.
- **New Features**
- Choose Postgres with `DATABASE_ENGINE` and
`DATABASE_HOST/PORT/USER/PASSWORD/NAME` (SQLite remains default).
- Centralized helpers: `is_postgres()`, `postgres_db_params()`,
`database_exists()`, `ensure_database_ready()` (auto‑create DB),
`approximate_row_counts()`, and schema rebuild/drop on non‑SQLite.
- Migrations keep raw SQLite DDL byte‑for‑byte; on Postgres, SQLite‑only
steps are skipped and tables are rebuilt from migration state (reverse
drops tables on Postgres). Adds a Postgres `text_pattern_ops` index for
URL queries.
- Query parity: crawl‑config search matches scalar JSON values on
Postgres via `jsonb_path_query`; URL prefix and fragmentless URL checks
use escaped `LIKE` on Postgres (uses the pattern‑ops index) and bytewise
range scans on SQLite.
- Consistent field limits: clamp overlong `CharField` values on save and
in `bulk_create` so writes behave the same on both backends.
- CLI/Admin: `init` verifies Postgres connectivity and creates the DB if
missing; status shows DB DSN or file; admin counts come from backend
optimizer stats.
- Tests run against a real Postgres cluster; CI only installs Postgres
binaries on the shard running `test_postgres_backend.py`. Added
benchmarking tool for hot‑path queries. New dependency:
`psycopg[binary]`.
- **Migration**
- SQLite users: no action needed; behavior unchanged.
- To use Postgres: set `DATABASE_ENGINE=postgres` and `DATABASE_*` vars,
then run `archivebox init` (creates the DB and applies migrations).
Choose your backend on first init; there’s no built‑in tool to move an
existing index between SQLite and Postgres.
<sup>Written for commit 2df28d9a72.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1839?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
Re-apply the postgres-server install step in dev's new per-file sharded
test-parallel structure, conditioned on the shard whose matrix.test.path is
the postgres backend test so other shards skip the apt install.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YHSjZM6TstSAMN2PhgfUg
The postgres ArchiveBox.conf example added to Configuration.md is an
illustration snippet; add it to docs/codeblocks.toml so the docs manifest
inventory check passes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YHSjZM6TstSAMN2PhgfUg
Helper reduction:
- Move the Django DATABASES/SQLITE_CONNECTION_OPTIONS assembly into
core/settings.py, dropping get_database_settings() and
get_sqlite_connection_options() from misc/db.py.
- Inline the single-use migration_table_columns() into its one migration.
Review fixes:
- search: match only scalar JSON *values* on postgres (jsonb_path_query
over '$.**' scalar leaves), mirroring SQLite json_tree.atom so config
keys no longer match.
- CharField clamp now also runs in SnapshotQuerySet.bulk_create (bulk paths
bypass the pre_save signal); truncate_overlong_charfields is dual-use.
- Restore reverse-migration parity on postgres: crawls/machine/api initial
migrations drop their rebuilt tables on reverse via drop_models_on_postgres.
- docs: give DATABASE_NAME its own section so the anchor resolves correctly.
- CI: only install postgres binaries on the shard that runs the postgres test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YHSjZM6TstSAMN2PhgfUg
Re-lock uv.lock with the CI-pinned uv version so the diff is limited to the
psycopg addition (a newer local uv had rewritten platform markers and
exclude-newer, breaking `uv sync --locked` in CI).
Consolidate the misc/db.py adapter surface: fold database_backend() into
is_postgres(), drop the redundant vendor-name constants, remove the unused
migration_table_exists() helper, and inline the single-use missing-table check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YHSjZM6TstSAMN2PhgfUg
Add DATABASE_ENGINE=postgres (plus DATABASE_HOST/PORT/USER/PASSWORD/NAME)
config and centralize all sqlite-vs-postgres branching in
archivebox.misc.db:
- get_database_settings() builds DATABASES for either backend; the sqlite
path is unchanged (custom lock-retry backend, same PRAGMAs).
- database_exists()/ensure_database_ready() replace index.sqlite3 file
checks; init auto-creates the postgres database when missing.
- approximate_row_counts() serves admin index counts from sqlite_stat1 or
pg_class.reltuples; missing-table detection covers both vendors.
- rebuild_models_from_migration_state() lets historical sqlite-only raw
SQL migrations resync postgres schema from Django migration state at
every divergence point (postgres can never hold legacy data, so
affected tables are empty when these run). All raw-DDL and PRAGMA
migrations are now vendor-gated with sqlite behavior byte-for-byte
unchanged.
- A pre_save clamp truncates CharField values to max_length: sqlite
never enforced varchar(n) but postgres does (e.g. long crawl labels).
- Collation-sensitive URL range scans branch to escaped LIKE on postgres
(with a text_pattern_ops index) since linguistic collations break
bytewise range tricks; the crawl-config JSON search wave gets a
jsonb-text implementation.
Verified on real PostgreSQL 16: fresh init applies the entire migration
graph, schema matches models exactly (column-level parity check +
makemigrations --check), and add/run/list/search/status/remove all work
end-to-end. New test_postgres_backend.py suite boots a real throwaway
postgres cluster (initdb + pg_ctl); CI workflows install postgres server
binaries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YHSjZM6TstSAMN2PhgfUg
