Commit Graph

6751 Commits

Author SHA1 Message Date
Nick Sweeting
91aa70cc73
Track OpenCode readiness separately from liveness 2026-09-03 00:14:34 -07:00
Nick Sweeting
8a12e3399c
Honor the OpenCode recovery deadline 2026-09-02 23:57:59 -07:00
Nick Sweeting
944907bfee
Keep OpenCode proxy traffic out of recovery 2026-09-02 23:56:46 -07:00
Nick Sweeting
7f1767a2db
Avoid restarting OpenCode on transient health misses 2026-09-02 23:49:58 -07:00
ArchiveBox Release Bot
3327edba24 Bump release version to 0.9.35rc418 2026-09-03 06:24:45 +00:00
Nick Sweeting
a71ef00b79
Reload stale OpenCode frames after server recovery (#1860)
## Summary

- monitor the proxied OpenCode health endpoint from the stable
ArchiveBox parent page
- reload the iframe once after an unavailable -> healthy transition
- trigger an immediate check again when the browser comes online or the
tab becomes visible

## Why

A real RC416 -> RC417 DigestBox restart left the already-open OpenCode
1.17.14 SPA showing its fatal error page against the newly started
OpenCode 1.17.15 server. A full page reload recovered immediately. The
parent ArchiveBox document survives the container restart, so it can
perform that one iframe reload automatically after the server is healthy
again.

## Verification

- `test_opencode_agent_superuser_gets_admin_wrapper` passes with
assertions for the recovery monitor
- focused pre-commit hooks pass
- real stale-tab restart reproduction will be repeated on DigestBox
after release

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Reloads stale OpenCode iframes after the ArchiveBox server recovers from
a restart, so the previously fatal error page is replaced automatically.
The parent page polls the OpenCode health endpoint every 3 seconds,
reloading the iframe once when the server transitions from unavailable
to healthy or when the server version changes, and also re-checks
immediately when the browser comes online or the tab becomes visible.

- Health monitoring uses a separate endpoint that reports status without
starting the server; recovery restarts a running but unhealthy OpenCode
process, and polling skips while hidden when the server is down.

**Verification**
- `test_opencode_agent_superuser_gets_admin_wrapper` passes with
assertions for the recovery monitor.
- Focused pre-commit hooks pass.

<sup>Written for commit a33b333b6a.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1860?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
2026-09-02 23:23:56 -07:00
Nick Sweeting
a33b333b6a
Keep OpenCode recovery nonblocking 2026-09-02 23:19:30 -07:00
Nick Sweeting
00e867d949
Separate OpenCode health monitoring from recovery 2026-09-02 23:02:59 -07:00
Nick Sweeting
69ca67b37d
Make OpenCode recovery detect server changes 2026-09-02 22:47:15 -07:00
Nick Sweeting
4149b6f578
Reload stale OpenCode frames after server recovery 2026-09-02 22:37:53 -07:00
ArchiveBox Release Bot
2d23264bd6 Bump release version to 0.9.35rc417 2026-09-03 05:27:30 +00:00
Nick Sweeting
e9feb8162a
Keep OpenCode SSE reconnects off blocked ASGI request threads (#1859)
## Summary

- return OpenCode SSE headers before waiting for a stopped server to
restart
- run reconnect-driven OpenCode startup in a non-thread-sensitive worker
inside the async stream
- remove the obsolete global `ThreadSensitiveContext` monkeypatch now
that asgiref 3.12.1 implements non-blocking shutdown upstream
- add a real ASGI regression test that holds the real startup lock and
verifies headers arrive before restart can finish

## Why

After deploying RC416, a stale `/admin/agent` tab successfully recovered
across the container restart, but Daphne logged three
`CurrentThreadExecutor already quit or is broken` exceptions while early
EventSource reconnects were cancelled during synchronous OpenCode
startup. Normal proxy traffic subsequently returned 200 and no data was
lost, but the request cleanup path was not clean.

## Verification

- `uv run pytest archivebox/tests/test_opencode_agent.py -q` — 15 passed
- focused SSE tests after final refactor — 2 passed
- `uv run prek run --all-files` — passed
- no OpenCode server remained after fixture teardown

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Makes OpenCode SSE reconnects return headers before the stopped server
finishes restarting, so early EventSource reconnects no longer cancel
while startup runs on a blocked ASGI request thread. This removes the
`CurrentThreadExecutor already quit or is broken` exceptions Daphne
logged after container restarts.

- Runs reconnect-driven OpenCode startup in a non-thread-sensitive
worker inside the async stream instead of blocking the request thread.
- Surfaces OpenCode startup failures as an error event on the SSE stream
instead of returning an error response.
- Removes the obsolete global `ThreadSensitiveContext` monkeypatch from
`archivebox/core/asgi.py`; asgiref 3.12.1 handles non-blocking shutdown
upstream.
- Adds an ASGI regression test that holds the real startup lock,
verifies headers arrive before restart can finish, avoids starting a
background server, and joins the startup worker before teardown.

<sup>Written for commit 7d7e219962.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1859?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
2026-09-02 22:26:39 -07:00
Nick Sweeting
7d7e219962
Join the SSE startup worker before test teardown 2026-09-02 22:21:52 -07:00
Nick Sweeting
24470ca551
Surface OpenCode startup failures on the event stream 2026-09-02 22:11:40 -07:00
Nick Sweeting
ff2766c007
Keep SSE disconnect test from starting a background server 2026-09-02 22:10:42 -07:00
Nick Sweeting
75cf3f1261
Keep OpenCode SSE startup off ASGI request threads 2026-09-02 22:02:03 -07:00
ArchiveBox Release Bot
b44ab723d2 Bump release version to 0.9.35rc416 2026-09-03 04:33:09 +00:00
Nick Sweeting
ce529842e3
Stop the full OpenCode process group (#1858)
* Stop the full OpenCode process group

* Fall back when process-group signaling fails

* Guarantee fallback test process cleanup
2026-09-02 21:31:03 -07:00
ArchiveBox Release Bot
378133e44c Bump release version to 0.9.35rc415 2026-09-03 03:43:52 +00:00
Nick Sweeting
3954119a0e
Keep embedded OpenCode proxy resilient (#1857)
* Keep embedded OpenCode proxy resilient

* Serialize OpenCode startup readiness

* Re-run review after dependency rebase
2026-09-02 20:43:21 -07:00
ArchiveBox Release Bot
1e53928ed7 Bump release version to 0.9.35rc414 2026-09-03 03:39:41 +00:00
ArchiveBox Release Bot
6241102ea7 Bump for abx-dl 1.12.261 2026-09-03 03:38:39 +00:00
ArchiveBox Release Bot
9a00c2f0b4 Bump release version to 0.9.35rc413 2026-09-03 02:49:30 +00:00
Nick Sweeting
4a31c409ca
Avoid OpenCode proxy startup stalls (#1856)
* Avoid OpenCode proxy startup stalls

* Handle concurrent OpenCode startup and read failures

* Hide OpenCode upstream errors from clients
2026-09-02 19:49:01 -07:00
ArchiveBox Release Bot
72f6c70ffa Bump for abx-dl 1.12.260 2026-09-03 02:47:35 +00:00
ArchiveBox Release Bot
08f8baabbd Bump release version to 0.9.35rc412
Some checks are pending
CI / Linters (push) Waiting to run
CI / Install and CLI platform compatibility (push) Waiting to run
CI / Discovered test matrix (push) Waiting to run
CI / Documentation and root tests (push) Waiting to run
CI / CodeQL (push) Waiting to run
CI / Tested Python artifacts (push) Waiting to run
CI / Tested Docker images (push) Waiting to run
CI / All required CI lanes (push) Blocked by required conditions
Deploy Publicsite to GitHub Pages / deploy (push) Waiting to run
Release candidate / prepare (push) Waiting to run
Release candidate / python-artifacts (push) Blocked by required conditions
Release candidate / docker-digests (push) Blocked by required conditions
2026-09-03 01:46:21 +00:00
Nick Sweeting
31bdeb4534
Allow same-origin OpenCode agent framing (#1855)
* Allow same-origin OpenCode agent framing

* Reuse OpenCode proxy prefix for framing policy
2026-09-02 18:45:51 -07:00
ArchiveBox Release Bot
e7ab7b087b Bump release version to 0.9.35rc411 2026-09-03 00:25:50 +00:00
Nick Sweeting
480ad08cf0
Fix snapshot backfill orchestration regressions (#1854)
* fix snapshot backfill orchestration regressions

* address snapshot orchestration review

* fix admin process projection fixture

* fix noresult projection fixture
2026-09-02 17:25:31 -07:00
ArchiveBox Release Bot
b799742f51 Bump release version to 0.9.35rc410 2026-09-02 22:19:25 +00:00
ArchiveBox Release Bot
5c34a68ad8 Bump for abx-dl 1.12.259 2026-09-02 22:19:08 +00:00
ArchiveBox Release Bot
d6106a456f Bump release version to 0.9.35rc409 2026-09-02 22:07:06 +00:00
Nick Sweeting
ebaeb9ea3f
refactor: simplify snapshot maintenance orchestration (#1853)
## Summary

- remove archivebox update and internal pseudo crawl URL paths
- parse submitted documents directly through abx-dl and project
discovered snapshots at depth zero
- keep ArchiveBox ownership at Crawl to Snapshot while abx-dl owns
Snapshot to ArchiveResult execution
- replace queued ArchiveResult search backfills with indexed
missing-result discovery and snapshot-scoped execution that leaves
sealed snapshots sealed
- make update schedules direct maintenance dispatches instead of
synthetic crawls
- keep filesystem migrations explicit, resumable, keyset-batched, and
indexed by fs_version
- preserve live Process and ArchiveResult projection for the progress UI

## Related PRs

- ArchiveBox/abx-dl#21 provides direct input parsing and explicit
event-bus orchestration
- ArchiveBox/abx-plugins#60 removes the obsolete pseudo URL and Sonic
consolidation branches

## Verification

- abx-dl full suite: 167 passed
- ArchiveBox runner and ArchiveResult projection suite: 89 passed
- add, API, and UI input coverage: 78 passed
- schedule suite: 9 passed
- filesystem migration coverage: 9 passed
- live progress coverage: 16 passed
- real https://sweeting.me abx-dl run: 80.74 seconds, 34 succeeded, 11
noresults, 0 failed, no surviving processes
- commit hooks and diff checks passed

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Simplifies snapshot orchestration by having `abx-dl` parse crawl inputs
and run snapshot hooks directly, replacing internal pseudo-URLs and
pre-created `ArchiveResult` queues. Scheduled maintenance now dispatches
directly, while concurrent snapshot completion preserves retry work
owned by the active run.

- Store submitted URLs and imported documents directly on `Crawl`;
remove pseudo-URL and input-root snapshots, and reject multiline URL
items.
- Keep Crawl-to-Snapshot ownership in ArchiveBox while `abx-dl` owns
Snapshot-to-ArchiveResult execution.
- Discover missing indexed results and dispatch snapshot-scoped
maintenance for sealed snapshots without changing their lifecycle.
- Preserve unbounded maintenance pagination and run filesystem
migrations explicitly in resumable keyset batches using indexed
`fs_version`.
- Bound retry scheduling and correlate completion leases so stale or
concurrent completions cannot consume newer retry work.
- Preserve live `Process` and `ArchiveResult` projections for progress
reporting.

**Migration**

- Apply the database migration before rollout; it removes `current_step`
and adds the `fs_version` index.

<sup>Written for commit a29dd2edbc.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1853?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
2026-09-02 15:04:47 -07:00
Nick Sweeting
0a9d8fd525
Merge remote-tracking branch 'origin/dev' into refactor/simplify-snapshot-orchestration
# Conflicts:
#	archivebox/services/runner.py
2026-09-02 15:04:37 -07:00
Nick Sweeting
a29dd2edbc
test: exercise keyset maintenance pagination 2026-09-02 14:51:37 -07:00
Nick Sweeting
ec4b8ef2ef
fix: preserve unbounded maintenance pagination 2026-09-02 14:43:13 -07:00
Nick Sweeting
e878007b6a
fix: bound snapshot scheduling ownership 2026-09-02 14:35:57 -07:00
ArchiveBox Release Bot
ae9399b5fd Bump release version to 0.9.35rc408 2026-09-02 21:26:05 +00:00
Nick Sweeting
2e0c0f14e9
Keep UUID migration check migration-only 2026-09-02 14:25:51 -07:00
Nick Sweeting
35e014ae90
fix: correlate snapshot completion leases 2026-09-02 14:18:00 -07:00
ArchiveBox Release Bot
baa3e37c59 Bump release version to 0.9.35rc407 2026-09-02 21:09:04 +00:00
Nick Sweeting
b7b4f34090
Keep CI dependency roots isolated 2026-09-02 14:06:15 -07:00
ArchiveBox Release Bot
ec1d3381b3 Bump release version to 0.9.35rc406 2026-09-02 21:06:05 +00:00
ArchiveBox Release Bot
410d5db1bf Bump for abx-dl 1.12.257 2026-09-02 21:05:45 +00:00
Nick Sweeting
8bc9f40e9f
fix: harden concurrent snapshot retries 2026-09-02 13:52:42 -07:00
ArchiveBox Release Bot
c791eabfb0 Bump release version to 0.9.35rc405 2026-09-02 20:39:29 +00:00
Nick Sweeting
1a5c6686c2
Provide build headers to LDAP docs example 2026-09-02 13:39:08 -07:00
Nick Sweeting
a85d417d0f
fix: preserve queued snapshot retry work 2026-09-02 13:33:21 -07:00
ArchiveBox Release Bot
68f5caaa29 Bump release version to 0.9.35rc404 2026-09-02 20:27:46 +00:00
ArchiveBox Release Bot
c3b99ceece Bump for abx-dl 1.12.256 2026-09-02 20:27:27 +00:00