Commit Graph

6690 Commits

Author SHA1 Message Date
Nick Sweeting
30998a0d16
Stop installing build dependencies in docs jobs 2026-09-02 12:18:30 -07:00
Nick Sweeting
809bd938a3
Preflight plugin dependencies before crawl hooks 2026-09-02 12:18:06 -07:00
ArchiveBox Release Bot
0a8f15e130 Bump release version to 0.9.35rc398
Some checks are pending
CI / Linters (push) Waiting to run
CI / Install and CLI platform compatibility (push) Waiting to run
CI / Discovered test matrix (push) Waiting to run
CI / Documentation and root tests (push) Waiting to run
CI / CodeQL (push) Waiting to run
CI / Tested Python artifacts (push) Waiting to run
CI / Tested Docker images (push) Waiting to run
CI / All required CI lanes (push) Blocked by required conditions
Deploy Publicsite to GitHub Pages / deploy (push) Waiting to run
Release candidate / prepare (push) Waiting to run
Release candidate / python-artifacts (push) Blocked by required conditions
Release candidate / docker-digests (push) Blocked by required conditions
2026-09-02 15:51:53 +00:00
ArchiveBox Release Bot
cb0155f88a Bump for abx-dl 1.12.251 2026-09-02 15:51:31 +00:00
ArchiveBox Release Bot
eeb47938c4 Bump release version to 0.9.35rc397 2026-09-02 15:26:39 +00:00
ArchiveBox Release Bot
1769f8b6e3 Bump for abx-dl 1.12.250 2026-09-02 15:26:21 +00:00
ArchiveBox Release Bot
71c410e791 Bump release version to 0.9.35rc396 2026-09-02 14:22:17 +00:00
ArchiveBox Release Bot
2b6f379453 Bump for abx-dl 1.12.249 2026-09-02 14:21:55 +00:00
ArchiveBox Release Bot
9e9a04bec8 Bump release version to 0.9.35rc395 2026-09-02 14:13:04 +00:00
ArchiveBox Release Bot
f45f7e33a6 Bump for abx-dl 1.12.248 2026-09-02 14:12:40 +00:00
ArchiveBox Release Bot
8c0cb7499f Bump release version to 0.9.35rc394 2026-09-02 13:51:11 +00:00
ArchiveBox Release Bot
bd7775e4f4 Bump for abx-dl 1.12.246 2026-09-02 13:50:49 +00:00
ArchiveBox Release Bot
8dbec73ee6 Bump release version to 0.9.35rc393 2026-09-02 13:38:01 +00:00
ArchiveBox Release Bot
8702aa33cc Bump for abx-dl 1.12.245 2026-09-02 13:37:34 +00:00
ArchiveBox Release Bot
11df983090 Bump release version to 0.9.35rc392 2026-09-02 13:15:51 +00:00
ArchiveBox Release Bot
8ed30dddb0 Bump for abx-dl 1.12.244 2026-09-02 13:15:29 +00:00
ArchiveBox Release Bot
045d7fd886 Bump release version to 0.9.35rc391 2026-09-02 12:59:18 +00:00
ArchiveBox Release Bot
b4421b8f92 Bump for abx-dl 1.12.243 2026-09-02 12:58:39 +00:00
ArchiveBox Release Bot
1594184239 Bump release version to 0.9.35rc390 2026-09-02 12:34:31 +00:00
ArchiveBox Release Bot
79b958e482 Bump for abx-dl 1.12.242 2026-09-02 12:34:16 +00:00
ArchiveBox Release Bot
a512884e9e Bump release version to 0.9.35rc389 2026-09-02 12:24:12 +00:00
Nick Sweeting
663017924b
refactor: consume canonical abx-dl execution plans 2026-09-02 05:23:11 -07:00
ArchiveBox Release Bot
b27f75647c Bump release version to 0.9.35rc388 2026-09-02 12:20:19 +00:00
ArchiveBox Release Bot
6d7d069cc0 Bump for abx-dl 1.12.241 2026-09-02 12:19:42 +00:00
ArchiveBox Release Bot
f4aa4d31c5 Bump release version to 0.9.35rc387 2026-09-02 12:08:57 +00:00
Nick Sweeting
8ed80ded07
Merge remote-tracking branch 'origin/dev' into dev 2026-09-02 05:08:07 -07:00
Nick Sweeting
f3a8c6945d
Clarify targeted plugin maintenance contract 2026-09-02 04:59:40 -07:00
Nick Sweeting
07975eccc7
Merge remote-tracking branch 'origin/dev' into dev 2026-09-02 04:59:27 -07:00
ArchiveBox Release Bot
1af2e8f544 Bump release version to 0.9.35rc386 2026-09-02 11:59:24 +00:00
ArchiveBox Release Bot
d6b1b269ca Bump for abx-dl 1.12.240 2026-09-02 11:59:08 +00:00
Nick Sweeting
e7b3f1c485
Adopt canonical abx-dl execution plan API 2026-09-02 04:44:55 -07:00
Nick Sweeting
23b91d6840
Merge remote-tracking branch 'origin/dev' into dev 2026-09-02 04:34:29 -07:00
Nick Sweeting
95a7e64dd7
Centralize queued plugin execution 2026-09-02 04:34:22 -07:00
ArchiveBox Release Bot
071bc0cce6 Bump release version to 0.9.35rc385 2026-09-02 11:14:34 +00:00
Nick Sweeting
3e58261cd5
Expose yt-dlp media metadata for snapshot cards (#1819)
Supersedes #1818.

This keeps yt-dlp-specific player rendering out of ArchiveBox core. Core
now:

- exposes media type metadata to plugin card templates: `is_video`,
`is_audio`, `is_browser_playable`
- sorts yt-dlp media with browser-playable video/audio first
- prefers browser-playable yt-dlp outputs before thumbnails and
non-browser containers when choosing the primary card output
- avoids pinning resumed snapshots to stale selected hook maps, fixing a
pause/resume runner regression surfaced by CI
- makes `update --index-only` discover the configured search backend
even when `PLUGINS` is restricted for test/runtime isolation
- stabilizes CI-only recursive/takeover/public-UI tests that were
failing due external/heavy plugin coverage, hard-kill recovery races,
and transient full-server `/add/` startup responses

The actual click-to-load player UI lives in the companion plugin PR:
ArchiveBox/abx-plugins#34.

Verification:
```text
pytest archivebox/tests/test_ui_admin_snapshot.py -q
48 passed

pytest archivebox/tests/test_cli_update_reindex_snapshots.py -q
9 passed

pytest -xvs archivebox/tests/test_api_v1_crawls_crawl_crawl_id.py --basetemp=tests/out --ignore=archivebox/pkgs
7 passed

pytest -xvs archivebox/tests/test_takeover_util.py --basetemp=tests/out --ignore=archivebox/pkgs
19 passed

pytest -xvs archivebox/tests/test_takeover_util.py::test_live_add_update_jobs_survive_server_and_cli_owner_exits --basetemp=tests/out --ignore=archivebox/pkgs
1 passed

pytest -xvs archivebox/tests/test_ui_public_snapshot.py --basetemp=tests/out --ignore=archivebox/pkgs
6 passed

pytest -q archivebox/tests/test_recursive_crawl.py::test_recursive_crawl_depth_two_all_plugins_runs_snapshots_in_parallel --basetemp=tests/out --ignore=archivebox/pkgs
1 passed
```
2026-09-02 04:14:15 -07:00
ArchiveBox Release Bot
0f1a402940 Bump release version to 0.9.35rc384 2026-09-02 11:05:26 +00:00
Nick Sweeting
5bead960b3
Merge remote-tracking branch 'origin/dev' into dev 2026-09-02 04:04:59 -07:00
Nick Sweeting
d5ff47b7e6
Refresh documentation snippet inventory 2026-09-02 04:04:45 -07:00
ArchiveBox Release Bot
457e616d1d Bump release version to 0.9.35rc383 2026-09-02 11:03:09 +00:00
Nick Sweeting
7f9d9ca2c1
Remove transient ArchiveResult migration blip 2026-09-02 04:02:47 -07:00
ArchiveBox Release Bot
dbb9f45f53 Bump release version to 0.9.35rc382 2026-09-02 10:50:22 +00:00
Nick Sweeting
ef7d590655
Eliminate request-side SQLite write transactions (#1846)
## Summary

- replace request-side `get_or_create()` / M2M transaction boundaries
with autocommit insert-conflict resolution and guarded CAS updates
- remove the ArchiveResult upload `transaction.atomic()` /
`select_for_update()` path while preserving exact `(snapshot, plugin,
hook_name)` identity and parent output-size accounting
- keep intermediate chunk uploads filesystem-only; persist ArchiveResult
metadata once the file is complete
- reuse the same insert-conflict helpers in pending-result creation,
event projection, recovery, tag projection, and admin tag editing

## Why

Browser screenshot/MHTML uploads were colliding with the runner on
SQLite. Django helper methods opened hidden read-then-write
transactions, and every chunk rewrote ArchiveResult/Snapshot rows. A 50+
MiB upload therefore repeatedly entered the SQLite writer path and could
hit an immediate WAL read-to-write upgrade failure.

The new path uses short autocommit statements. Unique constraints
resolve create races, `modified_at` guards ArchiveResult/Snapshot
updates, and a losing writer re-reads and re-merges before retrying the
CAS. Runner lifecycle locks and event serialization remain unchanged.

## Validation

- `uv run pytest archivebox/tests/test_api_v1_core_*.py -q` (38 passed)
- focused runner/service suite (27 passed)
- `uv run pytest archivebox/tests/test_ui_admin_snapshot.py -q` (57
passed)
- tag/list/admin suites (44 passed)
- `uv run prek run --all-files`

No migrations.

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Eliminates request-side SQLite write transactions so large browser
uploads no longer hit WAL read-to-write upgrade failures while the
runner holds the writer lock.

- Request-side create/update paths now use autocommit statements with
unique-constraint insert-conflict resolution and `modified_at`-guarded
compare-and-swap updates.
- ArchiveResult identity is now `(snapshot, plugin, hook_name)`;
migration `0054` drops the previously published plugin-unique constraint
so a plugin can keep several sibling hooks.
- A losing CAS writer re-reads and re-merges current state before
retrying; after three attempts the request fails with HTTP 409.
- Intermediate chunk uploads write to the filesystem only; the
`ArchiveResult` row is persisted once the upload completes, keeping the
parent snapshot's output-size accounting in sync.
- `create_snapshot` no longer blocks on the crawl lifecycle lock, and
`archivebox run` recreates exact hook requests rather than re-queueing a
plugin.
- Reused the insert-conflict helpers in pending-result creation, event
projection, recovery, tag projection, and admin tag editing; admin tag
edits now attribute new tags to the acting user.

<sup>Written for commit fbff88fb0e.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1846?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
2026-09-02 03:50:03 -07:00
Nick Sweeting
0c6f017f0a
Preserve ArchiveResult hook history through migrations 2026-09-02 03:49:53 -07:00
Nick Sweeting
fbff88fb0e
Merge remote-tracking branch 'origin/dev' into audit-pr1846
# Conflicts:
#	archivebox/cli/archivebox_extract.py
#	archivebox/core/recovery_util.py
#	archivebox/services/archive_result_service.py
#	archivebox/services/runner.py
2026-09-01 15:43:48 -07:00
ArchiveBox Release Bot
07df9c76a8 Bump release version to 0.9.35rc381
Some checks are pending
CI / Linters (push) Waiting to run
CI / Install and CLI platform compatibility (push) Waiting to run
CI / Discovered test matrix (push) Waiting to run
CI / Documentation and root tests (push) Waiting to run
CI / CodeQL (push) Waiting to run
CI / Tested Python artifacts (push) Waiting to run
CI / Tested Docker images (push) Waiting to run
CI / All required CI lanes (push) Blocked by required conditions
Deploy Publicsite to GitHub Pages / deploy (push) Waiting to run
Release candidate / prepare (push) Waiting to run
Release candidate / python-artifacts (push) Blocked by required conditions
Release candidate / docker-digests (push) Blocked by required conditions
2026-09-01 22:08:21 +00:00
Nick Sweeting
97136703f6
docs: state the collection concurrency contract 2026-09-01 15:07:58 -07:00
ArchiveBox Release Bot
2f761615f0 Bump release version to 0.9.35rc380 2026-09-01 22:07:50 +00:00
ArchiveBox Release Bot
8dc91f90be Bump for abx-dl 1.12.233 2026-09-01 22:07:20 +00:00
ArchiveBox Release Bot
2af88da0d4 Bump release version to 0.9.35rc379 2026-09-01 22:07:14 +00:00
Nick Sweeting
032c20bdf5
cleanup: delete obsolete model lifecycle helpers 2026-09-01 15:06:48 -07:00