Commit Graph

6759 Commits

Author SHA1 Message Date
Nick Sweeting
747f4a07ac
Remove duplicate issue detection workflow 2026-09-03 09:36:12 -07:00
ArchiveBox Release Bot
ca55f7f8d6 Bump release version to 0.9.35rc420
Some checks are pending
CI / Linters (push) Waiting to run
CI / Install and CLI platform compatibility (push) Waiting to run
CI / Discovered test matrix (push) Waiting to run
CI / Documentation and root tests (push) Waiting to run
CI / CodeQL (push) Waiting to run
CI / Tested Python artifacts (push) Waiting to run
CI / Tested Docker images (push) Waiting to run
CI / All required CI lanes (push) Blocked by required conditions
Deploy Publicsite to GitHub Pages / deploy (push) Waiting to run
Release candidate / prepare (push) Waiting to run
Release candidate / python-artifacts (push) Blocked by required conditions
Release candidate / docker-digests (push) Blocked by required conditions
2026-09-03 08:18:57 +00:00
Nick Sweeting
79fc5c34d2
Trust ready OpenCode processes (#1862)
The rc419 DigestBox test showed that health-based replacement still
killed OpenCode during legitimate initial UI load: health became
temporarily unavailable for more than five seconds while the child was
busy, causing repeated process replacement and 502s.

This removes the health-latency heuristic entirely. ArchiveBox now
distinguishes ready from merely running, trusts a ready live owned PID,
serializes cold starts until readiness, and only replaces children that
exited or never became ready. Stale tabs still recover after an
ArchiveBox/container restart because process state resets and the
detached health wake starts a new child.

This is also a net cleanup: 38 lines removed from the prior lifecycle
implementation.

Verification:
- `uv run pytest archivebox/tests/test_opencode_agent.py -q` (19 passed)
- focused real startup/readiness/proxy/shutdown tests (5 passed)
- `uv run prek run --all-files`
- independent gpt-5.6-medium lifecycle review clean

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Removes the health-latency heuristic that replaced OpenCode processes
during legitimate initial UI loads, causing 502s. Ready owned processes
are now trusted without probing, and startup health checks are bounded
so they can't run past the startup deadline.

- Only replaces children that exited or never became ready.
- Cold starts are serialized until readiness; stale tabs still recover
after a restart.

<sup>Written for commit 57e7820a0b.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1862?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
2026-09-03 01:18:37 -07:00
Nick Sweeting
57e7820a0b
Bound OpenCode startup health probes 2026-09-03 01:13:40 -07:00
Nick Sweeting
57ee874d6e
Trust ready OpenCode processes 2026-09-03 00:59:38 -07:00
ArchiveBox Release Bot
f5234360d7 Bump release version to 0.9.35rc419 2026-09-03 07:24:54 +00:00
Nick Sweeting
e5ac2709f3
Avoid restarting OpenCode on transient health misses (#1861)
The rc418 live DigestBox restart test exposed OpenCode process flapping
under iframe load: a single 2-second health timeout caused
`_ensure_opencode` to kill an otherwise live owned process, after which
the reloading iframe repeated the cycle.

This gives an owned process a bounded 5-second health recovery window
before replacing it. A real SIGSTOP/SIGCONT test proves that a
transiently unavailable process keeps the same PID, while the existing
real unhealthy-process test still proves eventual replacement.

Verification:
- `uv run pytest archivebox/tests/test_opencode_agent.py -q` (18 passed)
- `uv run prek run --all-files`

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Stops OpenCode from restarting on a single 2-second health timeout,
which caused process flapping under iframe load. An owned process now
gets a bounded 5-second health recovery window before being replaced, so
a transiently unavailable process keeps the same PID. Proxy and
event-stream requests no longer block on the recovery check when the
process is already running.

- Adds a SIGSTOP/SIGCONT test proving a transiently stopped process
keeps its PID, while the existing unhealthy-process test still verifies
eventual replacement.
- Adds tests proving proxy requests skip the recovery check while the
process is running and wait for readiness when it is not.

<sup>Written for commit 650697f3c2.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1861?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
2026-09-03 00:23:27 -07:00
Nick Sweeting
650697f3c2
Make OpenCode readiness checks race-safe 2026-09-03 00:17:53 -07:00
Nick Sweeting
91aa70cc73
Track OpenCode readiness separately from liveness 2026-09-03 00:14:34 -07:00
Nick Sweeting
8a12e3399c
Honor the OpenCode recovery deadline 2026-09-02 23:57:59 -07:00
Nick Sweeting
944907bfee
Keep OpenCode proxy traffic out of recovery 2026-09-02 23:56:46 -07:00
Nick Sweeting
7f1767a2db
Avoid restarting OpenCode on transient health misses 2026-09-02 23:49:58 -07:00
ArchiveBox Release Bot
3327edba24 Bump release version to 0.9.35rc418 2026-09-03 06:24:45 +00:00
Nick Sweeting
a71ef00b79
Reload stale OpenCode frames after server recovery (#1860)
## Summary

- monitor the proxied OpenCode health endpoint from the stable
ArchiveBox parent page
- reload the iframe once after an unavailable -> healthy transition
- trigger an immediate check again when the browser comes online or the
tab becomes visible

## Why

A real RC416 -> RC417 DigestBox restart left the already-open OpenCode
1.17.14 SPA showing its fatal error page against the newly started
OpenCode 1.17.15 server. A full page reload recovered immediately. The
parent ArchiveBox document survives the container restart, so it can
perform that one iframe reload automatically after the server is healthy
again.

## Verification

- `test_opencode_agent_superuser_gets_admin_wrapper` passes with
assertions for the recovery monitor
- focused pre-commit hooks pass
- real stale-tab restart reproduction will be repeated on DigestBox
after release

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Reloads stale OpenCode iframes after the ArchiveBox server recovers from
a restart, so the previously fatal error page is replaced automatically.
The parent page polls the OpenCode health endpoint every 3 seconds,
reloading the iframe once when the server transitions from unavailable
to healthy or when the server version changes, and also re-checks
immediately when the browser comes online or the tab becomes visible.

- Health monitoring uses a separate endpoint that reports status without
starting the server; recovery restarts a running but unhealthy OpenCode
process, and polling skips while hidden when the server is down.

**Verification**
- `test_opencode_agent_superuser_gets_admin_wrapper` passes with
assertions for the recovery monitor.
- Focused pre-commit hooks pass.

<sup>Written for commit a33b333b6a.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1860?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
2026-09-02 23:23:56 -07:00
Nick Sweeting
a33b333b6a
Keep OpenCode recovery nonblocking 2026-09-02 23:19:30 -07:00
Nick Sweeting
00e867d949
Separate OpenCode health monitoring from recovery 2026-09-02 23:02:59 -07:00
Nick Sweeting
69ca67b37d
Make OpenCode recovery detect server changes 2026-09-02 22:47:15 -07:00
Nick Sweeting
4149b6f578
Reload stale OpenCode frames after server recovery 2026-09-02 22:37:53 -07:00
ArchiveBox Release Bot
2d23264bd6 Bump release version to 0.9.35rc417 2026-09-03 05:27:30 +00:00
Nick Sweeting
e9feb8162a
Keep OpenCode SSE reconnects off blocked ASGI request threads (#1859)
## Summary

- return OpenCode SSE headers before waiting for a stopped server to
restart
- run reconnect-driven OpenCode startup in a non-thread-sensitive worker
inside the async stream
- remove the obsolete global `ThreadSensitiveContext` monkeypatch now
that asgiref 3.12.1 implements non-blocking shutdown upstream
- add a real ASGI regression test that holds the real startup lock and
verifies headers arrive before restart can finish

## Why

After deploying RC416, a stale `/admin/agent` tab successfully recovered
across the container restart, but Daphne logged three
`CurrentThreadExecutor already quit or is broken` exceptions while early
EventSource reconnects were cancelled during synchronous OpenCode
startup. Normal proxy traffic subsequently returned 200 and no data was
lost, but the request cleanup path was not clean.

## Verification

- `uv run pytest archivebox/tests/test_opencode_agent.py -q` — 15 passed
- focused SSE tests after final refactor — 2 passed
- `uv run prek run --all-files` — passed
- no OpenCode server remained after fixture teardown

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Makes OpenCode SSE reconnects return headers before the stopped server
finishes restarting, so early EventSource reconnects no longer cancel
while startup runs on a blocked ASGI request thread. This removes the
`CurrentThreadExecutor already quit or is broken` exceptions Daphne
logged after container restarts.

- Runs reconnect-driven OpenCode startup in a non-thread-sensitive
worker inside the async stream instead of blocking the request thread.
- Surfaces OpenCode startup failures as an error event on the SSE stream
instead of returning an error response.
- Removes the obsolete global `ThreadSensitiveContext` monkeypatch from
`archivebox/core/asgi.py`; asgiref 3.12.1 handles non-blocking shutdown
upstream.
- Adds an ASGI regression test that holds the real startup lock,
verifies headers arrive before restart can finish, avoids starting a
background server, and joins the startup worker before teardown.

<sup>Written for commit 7d7e219962.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1859?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
2026-09-02 22:26:39 -07:00
Nick Sweeting
7d7e219962
Join the SSE startup worker before test teardown 2026-09-02 22:21:52 -07:00
Nick Sweeting
24470ca551
Surface OpenCode startup failures on the event stream 2026-09-02 22:11:40 -07:00
Nick Sweeting
ff2766c007
Keep SSE disconnect test from starting a background server 2026-09-02 22:10:42 -07:00
Nick Sweeting
75cf3f1261
Keep OpenCode SSE startup off ASGI request threads 2026-09-02 22:02:03 -07:00
ArchiveBox Release Bot
b44ab723d2 Bump release version to 0.9.35rc416 2026-09-03 04:33:09 +00:00
Nick Sweeting
ce529842e3
Stop the full OpenCode process group (#1858)
* Stop the full OpenCode process group

* Fall back when process-group signaling fails

* Guarantee fallback test process cleanup
2026-09-02 21:31:03 -07:00
ArchiveBox Release Bot
378133e44c Bump release version to 0.9.35rc415 2026-09-03 03:43:52 +00:00
Nick Sweeting
3954119a0e
Keep embedded OpenCode proxy resilient (#1857)
* Keep embedded OpenCode proxy resilient

* Serialize OpenCode startup readiness

* Re-run review after dependency rebase
2026-09-02 20:43:21 -07:00
ArchiveBox Release Bot
1e53928ed7 Bump release version to 0.9.35rc414 2026-09-03 03:39:41 +00:00
ArchiveBox Release Bot
6241102ea7 Bump for abx-dl 1.12.261 2026-09-03 03:38:39 +00:00
ArchiveBox Release Bot
9a00c2f0b4 Bump release version to 0.9.35rc413 2026-09-03 02:49:30 +00:00
Nick Sweeting
4a31c409ca
Avoid OpenCode proxy startup stalls (#1856)
* Avoid OpenCode proxy startup stalls

* Handle concurrent OpenCode startup and read failures

* Hide OpenCode upstream errors from clients
2026-09-02 19:49:01 -07:00
ArchiveBox Release Bot
72f6c70ffa Bump for abx-dl 1.12.260 2026-09-03 02:47:35 +00:00
ArchiveBox Release Bot
08f8baabbd Bump release version to 0.9.35rc412
Some checks are pending
CI / Linters (push) Waiting to run
CI / Install and CLI platform compatibility (push) Waiting to run
CI / Discovered test matrix (push) Waiting to run
CI / Documentation and root tests (push) Waiting to run
CI / CodeQL (push) Waiting to run
CI / Tested Python artifacts (push) Waiting to run
CI / Tested Docker images (push) Waiting to run
CI / All required CI lanes (push) Blocked by required conditions
Deploy Publicsite to GitHub Pages / deploy (push) Waiting to run
Release candidate / prepare (push) Waiting to run
Release candidate / python-artifacts (push) Blocked by required conditions
Release candidate / docker-digests (push) Blocked by required conditions
2026-09-03 01:46:21 +00:00
Nick Sweeting
31bdeb4534
Allow same-origin OpenCode agent framing (#1855)
* Allow same-origin OpenCode agent framing

* Reuse OpenCode proxy prefix for framing policy
2026-09-02 18:45:51 -07:00
ArchiveBox Release Bot
e7ab7b087b Bump release version to 0.9.35rc411 2026-09-03 00:25:50 +00:00
Nick Sweeting
480ad08cf0
Fix snapshot backfill orchestration regressions (#1854)
* fix snapshot backfill orchestration regressions

* address snapshot orchestration review

* fix admin process projection fixture

* fix noresult projection fixture
2026-09-02 17:25:31 -07:00
ArchiveBox Release Bot
b799742f51 Bump release version to 0.9.35rc410 2026-09-02 22:19:25 +00:00
ArchiveBox Release Bot
5c34a68ad8 Bump for abx-dl 1.12.259 2026-09-02 22:19:08 +00:00
ArchiveBox Release Bot
d6106a456f Bump release version to 0.9.35rc409 2026-09-02 22:07:06 +00:00
Nick Sweeting
ebaeb9ea3f
refactor: simplify snapshot maintenance orchestration (#1853)
## Summary

- remove archivebox update and internal pseudo crawl URL paths
- parse submitted documents directly through abx-dl and project
discovered snapshots at depth zero
- keep ArchiveBox ownership at Crawl to Snapshot while abx-dl owns
Snapshot to ArchiveResult execution
- replace queued ArchiveResult search backfills with indexed
missing-result discovery and snapshot-scoped execution that leaves
sealed snapshots sealed
- make update schedules direct maintenance dispatches instead of
synthetic crawls
- keep filesystem migrations explicit, resumable, keyset-batched, and
indexed by fs_version
- preserve live Process and ArchiveResult projection for the progress UI

## Related PRs

- ArchiveBox/abx-dl#21 provides direct input parsing and explicit
event-bus orchestration
- ArchiveBox/abx-plugins#60 removes the obsolete pseudo URL and Sonic
consolidation branches

## Verification

- abx-dl full suite: 167 passed
- ArchiveBox runner and ArchiveResult projection suite: 89 passed
- add, API, and UI input coverage: 78 passed
- schedule suite: 9 passed
- filesystem migration coverage: 9 passed
- live progress coverage: 16 passed
- real https://sweeting.me abx-dl run: 80.74 seconds, 34 succeeded, 11
noresults, 0 failed, no surviving processes
- commit hooks and diff checks passed

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Simplifies snapshot orchestration by having `abx-dl` parse crawl inputs
and run snapshot hooks directly, replacing internal pseudo-URLs and
pre-created `ArchiveResult` queues. Scheduled maintenance now dispatches
directly, while concurrent snapshot completion preserves retry work
owned by the active run.

- Store submitted URLs and imported documents directly on `Crawl`;
remove pseudo-URL and input-root snapshots, and reject multiline URL
items.
- Keep Crawl-to-Snapshot ownership in ArchiveBox while `abx-dl` owns
Snapshot-to-ArchiveResult execution.
- Discover missing indexed results and dispatch snapshot-scoped
maintenance for sealed snapshots without changing their lifecycle.
- Preserve unbounded maintenance pagination and run filesystem
migrations explicitly in resumable keyset batches using indexed
`fs_version`.
- Bound retry scheduling and correlate completion leases so stale or
concurrent completions cannot consume newer retry work.
- Preserve live `Process` and `ArchiveResult` projections for progress
reporting.

**Migration**

- Apply the database migration before rollout; it removes `current_step`
and adds the `fs_version` index.

<sup>Written for commit a29dd2edbc.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1853?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
2026-09-02 15:04:47 -07:00
Nick Sweeting
0a9d8fd525
Merge remote-tracking branch 'origin/dev' into refactor/simplify-snapshot-orchestration
# Conflicts:
#	archivebox/services/runner.py
2026-09-02 15:04:37 -07:00
Nick Sweeting
a29dd2edbc
test: exercise keyset maintenance pagination 2026-09-02 14:51:37 -07:00
Nick Sweeting
ec4b8ef2ef
fix: preserve unbounded maintenance pagination 2026-09-02 14:43:13 -07:00
Nick Sweeting
e878007b6a
fix: bound snapshot scheduling ownership 2026-09-02 14:35:57 -07:00
ArchiveBox Release Bot
ae9399b5fd Bump release version to 0.9.35rc408 2026-09-02 21:26:05 +00:00
Nick Sweeting
2e0c0f14e9
Keep UUID migration check migration-only 2026-09-02 14:25:51 -07:00
Nick Sweeting
35e014ae90
fix: correlate snapshot completion leases 2026-09-02 14:18:00 -07:00
ArchiveBox Release Bot
baa3e37c59 Bump release version to 0.9.35rc407 2026-09-02 21:09:04 +00:00
Nick Sweeting
b7b4f34090
Keep CI dependency roots isolated 2026-09-02 14:06:15 -07:00