Add DATABASE_ENGINE=postgres (plus DATABASE_HOST/PORT/USER/PASSWORD/NAME)
config and centralize all sqlite-vs-postgres branching in
archivebox.misc.db:
- get_database_settings() builds DATABASES for either backend; the sqlite
path is unchanged (custom lock-retry backend, same PRAGMAs).
- database_exists()/ensure_database_ready() replace index.sqlite3 file
checks; init auto-creates the postgres database when missing.
- approximate_row_counts() serves admin index counts from sqlite_stat1 or
pg_class.reltuples; missing-table detection covers both vendors.
- rebuild_models_from_migration_state() lets historical sqlite-only raw
SQL migrations resync postgres schema from Django migration state at
every divergence point (postgres can never hold legacy data, so
affected tables are empty when these run). All raw-DDL and PRAGMA
migrations are now vendor-gated with sqlite behavior byte-for-byte
unchanged.
- A pre_save clamp truncates CharField values to max_length: sqlite
never enforced varchar(n) but postgres does (e.g. long crawl labels).
- Collation-sensitive URL range scans branch to escaped LIKE on postgres
(with a text_pattern_ops index) since linguistic collations break
bytewise range tricks; the crawl-config JSON search wave gets a
jsonb-text implementation.
Verified on real PostgreSQL 16: fresh init applies the entire migration
graph, schema matches models exactly (column-level parity check +
makemigrations --check), and add/run/list/search/status/remove all work
end-to-end. New test_postgres_backend.py suite boots a real throwaway
postgres cluster (initdb + pg_ctl); CI workflows install postgres server
binaries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YHSjZM6TstSAMN2PhgfUg
Snapshot.url was a CharField(max_length=65535) which reserves a fixed-width
column and is too long to index or constrain on real DB backends. Store it as a
variable-length TextField instead, so short URLs don't waste space and very long
URLs (up to MAX_URL_LENGTH=8000) are supported, while keeping a normal index on
the field so exact, prefix, and substring (icontains) URL lookups all stay fast.
- misc/util.py: MAX_URL_LENGTH 65535 -> 8000 (the practical web-server limit)
- core/models.py: Snapshot.url CharField -> TextField(db_index=True)
- migration 0049_alter_snapshot_url
- tests covering 8000-char persistence, exact/prefix/substring lookups,
over-length rejection, and per-crawl uniqueness for long URLs
https://claude.ai/code/session_01BLnGTL5GSoouD4ihaYp55n
Beta-tester / cabbage-style DBs upgraded incrementally through the
0.8.x → 0.9.x rc chain have crawls/0013_crawl_permissions,
personas/0003_persona_permissions, and core/0041_snapshot_permissions
all marked applied in django_migrations — but the historical migrations
with those names predate the current GeneratedField design. The columns
they were supposed to add never actually landed on the tables. When the
downstream hydration migrations (crawls/0016_hydrate_crawl_permissions,
personas/0004_hydrate_persona_permissions) run and try to filter on
.permissions, the query fails with no such column: permissions and
bricks startup.
Add a defensive _ensure_permissions_column pass at the top of each
hydration migration that ALTER TABLEs the column in if absent. For the
snapshot side (no hydration migration to attach to) introduce a new
core/0046_repair_snapshot_permissions migration that runs the same
guard. Fresh installs already have the column from the initial 0013 /
0003 / 0041 migrations, so the guard no-ops there.
SQLite ALTER TABLE ADD COLUMN only accepts VIRTUAL generated columns
("cannot add a STORED column"), so the repaired columns evaluate the
JSON extract on read rather than write — runtime queries behave
identically.
Verified end-to-end against actual cabbage data (23517 snapshots,
229 crawls, 103122 archive_results): container now boots healthy on
rc48 with the legacy schema in place.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0045_archiveresult_unique_hook adds a UniqueConstraint on
(snapshot, plugin, hook_name) but real long-lived collections (cabbage's
demo, beta-tester DBs) accumulated multiple rows per hook across the dev
rc chain. Without a cleanup pass first, the constraint fails with
``UNIQUE constraint failed`` mid-migration and bricks startup — cabbage
hit this on the rc48 deploy with 1382 duplicate (snapshot, plugin,
hook_name) groups already present.
Add a RunPython dedup pass that keeps the highest-id (most recent) row
per tuple, then applies the constraint. Fresh installs no-op the cleanup
since there are no duplicates to find.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tags now support full unicode with no restrictions. URL-encode the tag
name wherever it previously used the slug (export filenames, lookups).
- Remove `slug` field, `_generate_unique_slug`, and slug handling in save()
- Add migration 0034 to drop the slug column
- `get_tag_by_ref` now resolves by URL-decoded exact name match
- Tag search/autocomplete/export filenames use the name directly
- Drop slug from admin search_fields/readonly_fields/fieldsets
- Remove slug display from similar-tag cards and client download filename
