Supersedes #1818.
This keeps yt-dlp-specific player rendering out of ArchiveBox core. Core
now:
- exposes media type metadata to plugin card templates: `is_video`,
`is_audio`, `is_browser_playable`
- sorts yt-dlp media with browser-playable video/audio first
- prefers browser-playable yt-dlp outputs before thumbnails and
non-browser containers when choosing the primary card output
- avoids pinning resumed snapshots to stale selected hook maps, fixing a
pause/resume runner regression surfaced by CI
- makes `update --index-only` discover the configured search backend
even when `PLUGINS` is restricted for test/runtime isolation
- stabilizes CI-only recursive/takeover/public-UI tests that were
failing due external/heavy plugin coverage, hard-kill recovery races,
and transient full-server `/add/` startup responses
The actual click-to-load player UI lives in the companion plugin PR:
ArchiveBox/abx-plugins#34.
Verification:
```text
pytest archivebox/tests/test_ui_admin_snapshot.py -q
48 passed
pytest archivebox/tests/test_cli_update_reindex_snapshots.py -q
9 passed
pytest -xvs archivebox/tests/test_api_v1_crawls_crawl_crawl_id.py --basetemp=tests/out --ignore=archivebox/pkgs
7 passed
pytest -xvs archivebox/tests/test_takeover_util.py --basetemp=tests/out --ignore=archivebox/pkgs
19 passed
pytest -xvs archivebox/tests/test_takeover_util.py::test_live_add_update_jobs_survive_server_and_cli_owner_exits --basetemp=tests/out --ignore=archivebox/pkgs
1 passed
pytest -xvs archivebox/tests/test_ui_public_snapshot.py --basetemp=tests/out --ignore=archivebox/pkgs
6 passed
pytest -q archivebox/tests/test_recursive_crawl.py::test_recursive_crawl_depth_two_all_plugins_runs_snapshots_in_parallel --basetemp=tests/out --ignore=archivebox/pkgs
1 passed
```
## Summary
- replace request-side `get_or_create()` / M2M transaction boundaries
with autocommit insert-conflict resolution and guarded CAS updates
- remove the ArchiveResult upload `transaction.atomic()` /
`select_for_update()` path while preserving exact `(snapshot, plugin,
hook_name)` identity and parent output-size accounting
- keep intermediate chunk uploads filesystem-only; persist ArchiveResult
metadata once the file is complete
- reuse the same insert-conflict helpers in pending-result creation,
event projection, recovery, tag projection, and admin tag editing
## Why
Browser screenshot/MHTML uploads were colliding with the runner on
SQLite. Django helper methods opened hidden read-then-write
transactions, and every chunk rewrote ArchiveResult/Snapshot rows. A 50+
MiB upload therefore repeatedly entered the SQLite writer path and could
hit an immediate WAL read-to-write upgrade failure.
The new path uses short autocommit statements. Unique constraints
resolve create races, `modified_at` guards ArchiveResult/Snapshot
updates, and a losing writer re-reads and re-merges before retrying the
CAS. Runner lifecycle locks and event serialization remain unchanged.
## Validation
- `uv run pytest archivebox/tests/test_api_v1_core_*.py -q` (38 passed)
- focused runner/service suite (27 passed)
- `uv run pytest archivebox/tests/test_ui_admin_snapshot.py -q` (57
passed)
- tag/list/admin suites (44 passed)
- `uv run prek run --all-files`
No migrations.
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Eliminates request-side SQLite write transactions so large browser
uploads no longer hit WAL read-to-write upgrade failures while the
runner holds the writer lock.
- Request-side create/update paths now use autocommit statements with
unique-constraint insert-conflict resolution and `modified_at`-guarded
compare-and-swap updates.
- ArchiveResult identity is now `(snapshot, plugin, hook_name)`;
migration `0054` drops the previously published plugin-unique constraint
so a plugin can keep several sibling hooks.
- A losing CAS writer re-reads and re-merges current state before
retrying; after three attempts the request fails with HTTP 409.
- Intermediate chunk uploads write to the filesystem only; the
`ArchiveResult` row is persisted once the upload completes, keeping the
parent snapshot's output-size accounting in sync.
- `create_snapshot` no longer blocks on the crawl lifecycle lock, and
`archivebox run` recreates exact hook requests rather than re-queueing a
plugin.
- Reused the insert-conflict helpers in pending-result creation, event
projection, recovery, tag projection, and admin tag editing; admin tag
edits now attribute new tags to the acting user.
<sup>Written for commit fbff88fb0e.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1846?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
## Summary
This is the ArchiveBox side of a coordinated three-repository boundary
cleanup.
- delete the second python-statemachine lifecycle engine and replace it
with explicit model-owned queue transitions
- reduce workers/models.py to durable queue fields, leases, claims,
pause, resume, and guarded updates
- delete the legacy Crawl direct-hook lifecycle and duplicate
hook-record projector
- make abx-dl the sole hook subprocess runtime through PluginCatalog,
ExecutionPlan, execute_hook, and generic event services
- make the shared catalog the single source for packaged and
data-directory plugins, including admin plugin/config presentation
- make PluginConfigResolver the shared schema, alias, precedence, and
enabled-plugin resolver
- make OutputManifest the single output-files, mimetype, and size
implementation across models, APIs, recovery, template helpers, and
services
- remove python-statemachine plus the pydot and pyparsing diagram-only
dependency closure
Net ArchiveBox change: 585 insertions and 2,625 deletions.
## User impact
Existing CLI, REST, Django admin, custom plugin, queued-work,
pause/resume/cancel, and ArchiveResult surfaces are preserved. The
plugin catalog and configuration remain visible in admin; presentation
metadata now comes from plugin manifests rather than ArchiveBox
hardcoded lists.
## Layer boundary
ArchiveBox retains Django/database persistence, scheduling, projectors,
and product policy. abx-dl owns discovery, selection, execution, phase
setup, and output metadata without knowing about Django or ArchiveBox.
abx-plugins remains standalone and knows neither host.
## Coordinated dependencies and promotion order
- ArchiveBox/abx-plugins#59
- ArchiveBox/abx-dl#20
Promotion is intentionally serial because all three products use exact
release pins:
1. Merge and release abx-plugins#59.
2. Advance the abx-plugins pin on abx-dl#20, rerun CI, merge, and
release it.
3. Advance both exact pins and the lockfile on this branch, rerun native
ArchiveBox CI, then rerun native ArchiveBox CI before merging this PR.
This PR is ready for architectural review; merging remains gated on the
release and exact-pin sequence above.
This avoids temporary shims, git dependencies, and an unsatisfiable pair
of exact package requirements. The current native ArchiveBox install
matrix is expected to fail while it installs the older released abx-dl
API; local integration verification used editable sibling worktrees.
## Validation
- prek run --all-files
- django-admin makemigrations --check --dry-run: no changes detected
- 104 lifecycle, runner, output, and binary tests passed in the final
focused run
- 91 catalog, admin, config, runner, and output tests passed
- 31 direct-hook adapter tests passed
- real custom data-directory plugin extraction plus admin plugin/config
rendering: 3 passed
- additional binary, process, config, API, and lifecycle checks: 21
passed
