Commit Graph

6727 Commits

Author SHA1 Message Date
ArchiveBox Release Bot
72f6c70ffa Bump for abx-dl 1.12.260 2026-09-03 02:47:35 +00:00
ArchiveBox Release Bot
08f8baabbd Bump release version to 0.9.35rc412
Some checks are pending
CI / Linters (push) Waiting to run
CI / Install and CLI platform compatibility (push) Waiting to run
CI / Discovered test matrix (push) Waiting to run
CI / Documentation and root tests (push) Waiting to run
CI / CodeQL (push) Waiting to run
CI / Tested Python artifacts (push) Waiting to run
CI / Tested Docker images (push) Waiting to run
CI / All required CI lanes (push) Blocked by required conditions
Deploy Publicsite to GitHub Pages / deploy (push) Waiting to run
Release candidate / prepare (push) Waiting to run
Release candidate / python-artifacts (push) Blocked by required conditions
Release candidate / docker-digests (push) Blocked by required conditions
2026-09-03 01:46:21 +00:00
Nick Sweeting
31bdeb4534
Allow same-origin OpenCode agent framing (#1855)
* Allow same-origin OpenCode agent framing

* Reuse OpenCode proxy prefix for framing policy
2026-09-02 18:45:51 -07:00
ArchiveBox Release Bot
e7ab7b087b Bump release version to 0.9.35rc411 2026-09-03 00:25:50 +00:00
Nick Sweeting
480ad08cf0
Fix snapshot backfill orchestration regressions (#1854)
* fix snapshot backfill orchestration regressions

* address snapshot orchestration review

* fix admin process projection fixture

* fix noresult projection fixture
2026-09-02 17:25:31 -07:00
ArchiveBox Release Bot
b799742f51 Bump release version to 0.9.35rc410 2026-09-02 22:19:25 +00:00
ArchiveBox Release Bot
5c34a68ad8 Bump for abx-dl 1.12.259 2026-09-02 22:19:08 +00:00
ArchiveBox Release Bot
d6106a456f Bump release version to 0.9.35rc409 2026-09-02 22:07:06 +00:00
Nick Sweeting
ebaeb9ea3f
refactor: simplify snapshot maintenance orchestration (#1853)
## Summary

- remove archivebox update and internal pseudo crawl URL paths
- parse submitted documents directly through abx-dl and project
discovered snapshots at depth zero
- keep ArchiveBox ownership at Crawl to Snapshot while abx-dl owns
Snapshot to ArchiveResult execution
- replace queued ArchiveResult search backfills with indexed
missing-result discovery and snapshot-scoped execution that leaves
sealed snapshots sealed
- make update schedules direct maintenance dispatches instead of
synthetic crawls
- keep filesystem migrations explicit, resumable, keyset-batched, and
indexed by fs_version
- preserve live Process and ArchiveResult projection for the progress UI

## Related PRs

- ArchiveBox/abx-dl#21 provides direct input parsing and explicit
event-bus orchestration
- ArchiveBox/abx-plugins#60 removes the obsolete pseudo URL and Sonic
consolidation branches

## Verification

- abx-dl full suite: 167 passed
- ArchiveBox runner and ArchiveResult projection suite: 89 passed
- add, API, and UI input coverage: 78 passed
- schedule suite: 9 passed
- filesystem migration coverage: 9 passed
- live progress coverage: 16 passed
- real https://sweeting.me abx-dl run: 80.74 seconds, 34 succeeded, 11
noresults, 0 failed, no surviving processes
- commit hooks and diff checks passed

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Simplifies snapshot orchestration by having `abx-dl` parse crawl inputs
and run snapshot hooks directly, replacing internal pseudo-URLs and
pre-created `ArchiveResult` queues. Scheduled maintenance now dispatches
directly, while concurrent snapshot completion preserves retry work
owned by the active run.

- Store submitted URLs and imported documents directly on `Crawl`;
remove pseudo-URL and input-root snapshots, and reject multiline URL
items.
- Keep Crawl-to-Snapshot ownership in ArchiveBox while `abx-dl` owns
Snapshot-to-ArchiveResult execution.
- Discover missing indexed results and dispatch snapshot-scoped
maintenance for sealed snapshots without changing their lifecycle.
- Preserve unbounded maintenance pagination and run filesystem
migrations explicitly in resumable keyset batches using indexed
`fs_version`.
- Bound retry scheduling and correlate completion leases so stale or
concurrent completions cannot consume newer retry work.
- Preserve live `Process` and `ArchiveResult` projections for progress
reporting.

**Migration**

- Apply the database migration before rollout; it removes `current_step`
and adds the `fs_version` index.

<sup>Written for commit a29dd2edbc.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1853?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
2026-09-02 15:04:47 -07:00
Nick Sweeting
0a9d8fd525
Merge remote-tracking branch 'origin/dev' into refactor/simplify-snapshot-orchestration
# Conflicts:
#	archivebox/services/runner.py
2026-09-02 15:04:37 -07:00
Nick Sweeting
a29dd2edbc
test: exercise keyset maintenance pagination 2026-09-02 14:51:37 -07:00
Nick Sweeting
ec4b8ef2ef
fix: preserve unbounded maintenance pagination 2026-09-02 14:43:13 -07:00
Nick Sweeting
e878007b6a
fix: bound snapshot scheduling ownership 2026-09-02 14:35:57 -07:00
ArchiveBox Release Bot
ae9399b5fd Bump release version to 0.9.35rc408 2026-09-02 21:26:05 +00:00
Nick Sweeting
2e0c0f14e9
Keep UUID migration check migration-only 2026-09-02 14:25:51 -07:00
Nick Sweeting
35e014ae90
fix: correlate snapshot completion leases 2026-09-02 14:18:00 -07:00
ArchiveBox Release Bot
baa3e37c59 Bump release version to 0.9.35rc407 2026-09-02 21:09:04 +00:00
Nick Sweeting
b7b4f34090
Keep CI dependency roots isolated 2026-09-02 14:06:15 -07:00
ArchiveBox Release Bot
ec1d3381b3 Bump release version to 0.9.35rc406 2026-09-02 21:06:05 +00:00
ArchiveBox Release Bot
410d5db1bf Bump for abx-dl 1.12.257 2026-09-02 21:05:45 +00:00
Nick Sweeting
8bc9f40e9f
fix: harden concurrent snapshot retries 2026-09-02 13:52:42 -07:00
ArchiveBox Release Bot
c791eabfb0 Bump release version to 0.9.35rc405 2026-09-02 20:39:29 +00:00
Nick Sweeting
1a5c6686c2
Provide build headers to LDAP docs example 2026-09-02 13:39:08 -07:00
Nick Sweeting
a85d417d0f
fix: preserve queued snapshot retry work 2026-09-02 13:33:21 -07:00
ArchiveBox Release Bot
68f5caaa29 Bump release version to 0.9.35rc404 2026-09-02 20:27:46 +00:00
ArchiveBox Release Bot
c3b99ceece Bump for abx-dl 1.12.256 2026-09-02 20:27:27 +00:00
ArchiveBox Release Bot
ac7598aae2 Bump release version to 0.9.35rc403 2026-09-02 19:56:48 +00:00
Nick Sweeting
9ee0dcf165
fix: address orchestration review findings 2026-09-02 12:56:23 -07:00
ArchiveBox Release Bot
1c3b4bc5ce Bump for abx-dl 1.12.254 2026-09-02 19:56:13 +00:00
ArchiveBox Release Bot
344dc86d83 Bump release version to 0.9.35rc402 2026-09-02 19:43:41 +00:00
Nick Sweeting
718beb3020
Limit docs build dependencies to setup examples 2026-09-02 12:43:17 -07:00
ArchiveBox Release Bot
27efbe6bb0 Bump release version to 0.9.35rc401 2026-09-02 19:31:11 +00:00
ArchiveBox Release Bot
94a9a4d444 Bump for abx-dl 1.12.252 2026-09-02 19:30:09 +00:00
ArchiveBox Release Bot
970c1adedf Bump release version to 0.9.35rc400 2026-09-02 19:28:37 +00:00
Nick Sweeting
40e6ded0f7
Scope machine service test to liteparse 2026-09-02 12:27:56 -07:00
ArchiveBox Release Bot
3416a1f707 Bump release version to 0.9.35rc399 2026-09-02 19:21:59 +00:00
Nick Sweeting
30998a0d16
Stop installing build dependencies in docs jobs 2026-09-02 12:18:30 -07:00
Nick Sweeting
809bd938a3
Preflight plugin dependencies before crawl hooks 2026-09-02 12:18:06 -07:00
Nick Sweeting
2bba758314
refactor: simplify snapshot maintenance orchestration 2026-09-02 12:05:09 -07:00
ArchiveBox Release Bot
0a8f15e130 Bump release version to 0.9.35rc398
Some checks are pending
CI / Linters (push) Waiting to run
CI / Install and CLI platform compatibility (push) Waiting to run
CI / Discovered test matrix (push) Waiting to run
CI / Documentation and root tests (push) Waiting to run
CI / CodeQL (push) Waiting to run
CI / Tested Python artifacts (push) Waiting to run
CI / Tested Docker images (push) Waiting to run
CI / All required CI lanes (push) Blocked by required conditions
Deploy Publicsite to GitHub Pages / deploy (push) Waiting to run
Release candidate / prepare (push) Waiting to run
Release candidate / python-artifacts (push) Blocked by required conditions
Release candidate / docker-digests (push) Blocked by required conditions
2026-09-02 15:51:53 +00:00
ArchiveBox Release Bot
cb0155f88a Bump for abx-dl 1.12.251 2026-09-02 15:51:31 +00:00
ArchiveBox Release Bot
eeb47938c4 Bump release version to 0.9.35rc397 2026-09-02 15:26:39 +00:00
ArchiveBox Release Bot
1769f8b6e3 Bump for abx-dl 1.12.250 2026-09-02 15:26:21 +00:00
ArchiveBox Release Bot
71c410e791 Bump release version to 0.9.35rc396 2026-09-02 14:22:17 +00:00
ArchiveBox Release Bot
2b6f379453 Bump for abx-dl 1.12.249 2026-09-02 14:21:55 +00:00
ArchiveBox Release Bot
9e9a04bec8 Bump release version to 0.9.35rc395 2026-09-02 14:13:04 +00:00
ArchiveBox Release Bot
f45f7e33a6 Bump for abx-dl 1.12.248 2026-09-02 14:12:40 +00:00
ArchiveBox Release Bot
8c0cb7499f Bump release version to 0.9.35rc394 2026-09-02 13:51:11 +00:00
ArchiveBox Release Bot
bd7775e4f4 Bump for abx-dl 1.12.246 2026-09-02 13:50:49 +00:00
ArchiveBox Release Bot
8dbec73ee6 Bump release version to 0.9.35rc393 2026-09-02 13:38:01 +00:00