Nick Sweeting
d4ff0d033f
Merge remote-tracking branch 'origin/dev' into codex/all-tests-ci
2026-07-25 11:51:21 -07:00
Nick Sweeting
7658df9208
Fix ArchiveBox CI runtime paths and docs validation
2026-07-25 11:51:15 -07:00
Claude
f7c182d76c
Merge remote-tracking branch 'origin/dev' into claude/archivebox-postgresql-support-xrlgim
...
# Conflicts:
# .github/workflows/test-parallel.yml
# .github/workflows/test.yml
# uv.lock
2026-07-24 10:30:55 +00:00
Nick Sweeting
eacfae5d6c
Run all ArchiveBox tests through unified CI
2026-07-24 00:50:40 -07:00
Claude
7167c5dd8a
Add native PostgreSQL support alongside SQLite
...
Add DATABASE_ENGINE=postgres (plus DATABASE_HOST/PORT/USER/PASSWORD/NAME)
config and centralize all sqlite-vs-postgres branching in
archivebox.misc.db:
- get_database_settings() builds DATABASES for either backend; the sqlite
path is unchanged (custom lock-retry backend, same PRAGMAs).
- database_exists()/ensure_database_ready() replace index.sqlite3 file
checks; init auto-creates the postgres database when missing.
- approximate_row_counts() serves admin index counts from sqlite_stat1 or
pg_class.reltuples; missing-table detection covers both vendors.
- rebuild_models_from_migration_state() lets historical sqlite-only raw
SQL migrations resync postgres schema from Django migration state at
every divergence point (postgres can never hold legacy data, so
affected tables are empty when these run). All raw-DDL and PRAGMA
migrations are now vendor-gated with sqlite behavior byte-for-byte
unchanged.
- A pre_save clamp truncates CharField values to max_length: sqlite
never enforced varchar(n) but postgres does (e.g. long crawl labels).
- Collation-sensitive URL range scans branch to escaped LIKE on postgres
(with a text_pattern_ops index) since linguistic collations break
bytewise range tricks; the crawl-config JSON search wave gets a
jsonb-text implementation.
Verified on real PostgreSQL 16: fresh init applies the entire migration
graph, schema matches models exactly (column-level parity check +
makemigrations --check), and add/run/list/search/status/remove all work
end-to-end. New test_postgres_backend.py suite boots a real throwaway
postgres cluster (initdb + pg_ctl); CI workflows install postgres server
binaries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019YHSjZM6TstSAMN2PhgfUg
2026-07-23 17:08:47 +00:00
Nick Sweeting
a3c5e13ff8
Release ArchiveBox through complete CI coverage
CI / Linters (push) Waiting to run
CI / Complete test suite (push) Waiting to run
CI / Sharded and plugin tests (push) Waiting to run
CI / Documentation and root tests (push) Waiting to run
CI / CodeQL (push) Waiting to run
CI / Tested Python artifacts (push) Waiting to run
CI / Tested Docker images (push) Waiting to run
CI / All required CI lanes (push) Blocked by required conditions
Deploy Publicsite to GitHub Pages / deploy (push) Has been cancelled
2026-07-23 07:00:05 -07:00
Nick Sweeting
6592af0d19
Exercise ArchiveBox through unified runtime lifecycles
2026-07-21 19:26:42 -07:00
Nick Sweeting
ee91a9e366
Filter version binaries before resolution
2026-07-20 16:42:04 -07:00
Nick Sweeting
81ea053f97
Avoid empty binary resolution buses
2026-07-20 16:22:53 -07:00
Nick Sweeting
5a30ab0451
Scope install status to requested binaries
2026-07-20 15:54:33 -07:00
Nick Sweeting
01f5666dbc
Route runtime binaries through abxpkg
2026-07-20 12:50:14 -07:00
Nick Sweeting
ee0e3eb43c
Enforce the sealed search maintenance boundary
2026-07-19 08:54:32 -07:00
Nick Sweeting
3fa3794bf3
Skip disabled plugin binary checks in version
2026-06-24 09:51:28 -07:00
Nick Sweeting
0e0759a680
Skip binary probes outside data dirs
2026-06-21 05:42:26 -07:00
Nick Sweeting
59a956bf9f
Show dynamic DB binaries in version output
2026-06-21 05:24:15 -07:00
Nick Sweeting
9dbeece35b
fix version command exit status
2026-06-21 02:41:18 -07:00
Nick Sweeting
d259ac8095
fix machine network interface identity
2026-06-21 02:29:53 -07:00
Nick Sweeting
00e818018a
release: v0.9.35rc46
2026-06-21 01:44:30 -07:00
Nick Sweeting
8b57085827
chore: commit local archivebox changes
2026-06-14 11:44:38 -07:00
Nick Sweeting
6f321797c9
fix: publish dev build with updated package deps
2026-06-14 09:34:32 -07:00
Nick Sweeting
88a92b6548
fix: keep version informational without installed binaries
2026-06-14 09:28:41 -07:00
Nick Sweeting
0ddda66ee3
release: archivebox 0.9.35rc35
2026-06-14 08:55:48 -07:00
Nick Sweeting
1dbde4776f
release: archivebox 0.9.35rc27
2026-06-13 23:12:36 -07:00
Nick Sweeting
e547abbf27
Align direct URL Crawl flow with historical depth=0 convention
...
archivebox add and other entry points now seed Crawl.urls as
CrawlSeed JSONL at depth=0 (the input layer) with max_depth=depth
for direct URLs and depth+1 only for stdin/import text where the
synthetic archivebox://internal root lives at depth=0. The runner
also accepts one plain URL per line for ORM/crawl-create/schedule
callers so every Crawl row goes through the same expansion path
without scattering CrawlSeed knowledge across the codebase.
Tests updated to match restored convention.
2026-06-13 18:04:45 -07:00
Nick Sweeting
c9e63ccffd
Centralize synthetic root snapshot creation in CrawlRunner
...
Direct URL inputs from CLI/UI/API now seed Crawl.urls as explicit
{type:CrawlSeed,url,depth} JSONL rows; raw stdin/UI/API import text
stays verbatim. The runner's create_initial_snapshots() is now the
single place that either expands seed rows or creates the synthetic
archivebox://internal root + staticfile/stdin.txt, so add paths no
longer perform DB/FS side effects and the parser hooks run through
the same Snapshot lifecycle as every other extractor.
2026-06-13 17:10:06 -07:00
Nick Sweeting
6a635d3cd6
Fix UI add direct URL runner flow
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-amd64, docker-amd64, linux/amd64, ubuntu-24.04) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-arm64, docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Waiting to run
Build Docker image / publish multiarch tags (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Release State / release-state (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Parallel Tests / ${{ matrix.plugin.name }} (push) Blocked by required conditions
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
Update Homebrew tap / dispatch (push) Waiting to run
2026-06-11 09:28:28 -07:00
Nick Sweeting
14d43c88f5
Repair stale binaries in version output
2026-06-11 07:52:05 -07:00
Nick Sweeting
b6921d5e03
Restore direct URL add snapshots
2026-06-11 06:21:46 -07:00
Nick Sweeting
f6c98b67d7
Limit stale binary checks to version output
2026-06-11 01:32:30 -07:00
Nick Sweeting
98aecfaf91
release: archivebox 0.9.35rc8
2026-06-09 23:12:22 -07:00
Nick Sweeting
2bfb3ad4eb
release: archivebox 0.9.35rc5
2026-06-09 22:09:28 -07:00
Nick Sweeting
a1449c2822
Use plugin URL patterns for source imports
2026-06-08 23:40:25 -07:00
Nick Sweeting
5adec53b4c
Expand add flow runtime handling
2026-06-08 23:27:08 -07:00
Nick Sweeting
4fa90e484a
release: archivebox 0.9.34rc71
2026-06-07 20:51:28 -07:00
Nick Sweeting
2659f20dc4
fix runner takeover for scoped snapshot workers
2026-06-07 11:41:17 -07:00
Nick Sweeting
1ba5281343
release: archivebox 0.9.34rc68
2026-06-07 04:19:40 -07:00
Nick Sweeting
87b518314a
release: archivebox 0.9.34rc67
2026-06-07 04:06:45 -07:00
Nick Sweeting
1b19736b2f
Recover interrupted hook work by hook identity
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-amd64, docker-amd64, linux/amd64, ubuntu-24.04) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-arm64, docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Waiting to run
Build Docker image / publish multiarch tags (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Release State / release-state (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Parallel Tests / ${{ matrix.plugin.name }} (push) Blocked by required conditions
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
Update Homebrew tap / dispatch (push) Waiting to run
2026-06-05 03:25:38 -07:00
Nick Sweeting
4e4ee8cdb0
fix runner stdin and update maintenance lifecycle
2026-06-04 22:38:22 -07:00
Nick Sweeting
73587a1a4d
use archivebox plugin discovery for extraction queues
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-amd64, docker-amd64, linux/amd64, ubuntu-24.04) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-arm64, docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Waiting to run
Build Docker image / publish multiarch tags (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Release State / release-state (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Parallel Tests / ${{ matrix.plugin.name }} (push) Blocked by required conditions
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
Update Homebrew tap / dispatch (push) Waiting to run
2026-06-04 21:57:33 -07:00
Nick Sweeting
7f8af6357d
fix index and binary runner lifecycle
2026-06-04 21:40:58 -07:00
Nick Sweeting
c0fb8eb532
release: archivebox 0.9.34rc39
2026-06-03 17:19:48 -07:00
Nick Sweeting
83a2099851
fix: scope update search backfill runner
2026-06-02 21:54:25 -07:00
Nick Sweeting
3669133a05
fix: allow install to initialize collections
2026-06-02 21:25:20 -07:00
Nick Sweeting
9f0544857c
test: require success in cli workflows
2026-06-02 21:19:21 -07:00
Nick Sweeting
e4ec848da8
fix: keep background add crawls runnable
2026-06-02 18:53:59 -07:00
Nick Sweeting
39eac65ed0
Stabilize frozen config CLI test flows
...
(cherry picked from commit 2bca869e32
2026-06-02 18:44:57 -07:00
Nick Sweeting
b46d142cc6
test cleanup
2026-06-02 12:13:47 -07:00
Nick Sweeting
96437e1ffd
Publish local ArchiveBox changes
2026-06-02 02:25:52 -07:00
Nick Sweeting
7dd738b5b7
release: archivebox 0.9.34rc37
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Debian package / build (amd64) (push) Waiting to run
Build Debian package / build (arm64) (push) Waiting to run
Build Debian package / test (amd64, ubuntu-24.04) (push) Blocked by required conditions
Build Debian package / test (arm64, ubuntu-24.04-arm) (push) Blocked by required conditions
Build Debian package / release (push) Blocked by required conditions
Build Docker image / build ${{ matrix.platform }} (digest-linux-amd64, docker-amd64, linux/amd64, ubuntu-24.04) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-arm64, docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Waiting to run
Build Docker image / publish multiarch tags (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Release State / release-state (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Parallel Tests / ${{ matrix.plugin.name }} (push) Blocked by required conditions
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
2026-06-01 21:44:23 -07:00
