The column change to a variable-length, still-indexed TextField is what
matters for supporting long URLs efficiently — no reason to lower the
supported limit. Adjust the long-URL tests to the real 65535 boundary.
Snapshot.url was a CharField(max_length=65535) which reserves a fixed-width
column and is too long to index or constrain on real DB backends. Store it as a
variable-length TextField instead, so short URLs don't waste space and very long
URLs (up to MAX_URL_LENGTH=8000) are supported, while keeping a normal index on
the field so exact, prefix, and substring (icontains) URL lookups all stay fast.
- misc/util.py: MAX_URL_LENGTH 65535 -> 8000 (the practical web-server limit)
- core/models.py: Snapshot.url CharField -> TextField(db_index=True)
- migration 0049_alter_snapshot_url
- tests covering 8000-char persistence, exact/prefix/substring lookups,
over-length rejection, and per-crawl uniqueness for long URLs
https://claude.ai/code/session_01BLnGTL5GSoouD4ihaYp55n
Renames (no functional change, just consistency with the rest of the codebase):
- cli/cli_utils.py → cli/cli_util.py
- core/host_utils.py → core/host_util.py
- core/tag_utils.py → core/tag_util.py
- crawls/schedule_utils.py → crawls/schedule_util.py
- machine/env_utils.py → machine/env_util.py
Functional fixes:
- archivebox add --index-only now materializes Snapshot rows synchronously
via crawl.create_snapshots_from_urls() instead of just queueing the Crawl
and leaving the index empty. The previous behavior broke every test that
expected --index-only to populate the index, since the runner is never
started in index-only mode.
- config/collection.py: add _coerce_from_str_dict as the inverse of
_coerce_to_str_dict so JSON-encoded INI values are decoded back to native
dict/list types when mirrored into Machine.config (a JSONField). Without
this, downstream consumers like MachineEvent / abx-dl get raw JSON
strings where they expect dicts.
Plus matching admin / middleware / model touch-ups, the registration
password_change_form template, and assorted small cleanups the user
worked through while validating the deploy path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The newer-DB guard added in 65dc2521 refused to start on any collection
whose django_migrations table referenced migrations that have since been
squashed away from source (core/0023→0075, api/0002→0009, machine/0002→
0004, plus every transient dev-branch rename). That's the state of every
beta-tester DB that was upgraded incrementally through the 0.8.x → 0.9.x
rc chain — cabbage's demo deployment was the first concrete casualty.
Django's standard `replaces=` pattern doesn't fit here: its all-or-none
semantics split the migration graph whenever only a subset of the listed
replaces is applied, which is exactly what happens when different beta
testers stopped at different intermediate dev branches. Instead, keep an
explicit HISTORICAL_GHOST_MIGRATIONS allowlist in misc/db.py enumerating
every squashed-away name, and subtract it from missing_from_code in
migration_state(). The hard SystemExit(3) for truly unknown future names
stays — only known-historical ghosts are absorbed.
Adds test_init_recovers_from_pre_squash_dev_history covering the gap the
pre-existing migration tests missed (they only seed published 0.4/0.7/
0.8 schemas, never a post-squash dev DB).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Snapshot detail page: embed scoped live-progress monitor (same-origin
/progress.json on whichever host the page is served from); hide admin
action buttons when scoped; per-snapshot perms via can_view_snapshot.
- crawl_file API: respect crawl-level permissions; PUBLIC/UNLISTED served
to guests, PRIVATE returns 404 for non-admin/non-owner.
- CrawlRunner: replace allow_paused_snapshot_maintenance with
allow_maintenance_on_inactive_crawl so SEALED crawls don't short-circuit
the cancellation guard for legitimate maintenance hooks (search backend
backfill, fs migration, etc.). Fixes infinite STARTED loop on snapshots
with queued search_backend results.
- Universal `--init` flag: works on any subcommand (server, update, add,
shell, install, ...). Detected at module load, stripped from argv, and
consumed in the dispatcher so subprocesses inherit a clean env.
- supervisord_util.run_runner_worker: route Ctrl+C through
supervisor.signalProcess(name, "SIGINT") instead of raw os.kill on a
cached pid, gated on statename=RUNNING. Prevents killing unrelated
processes when the worker's pid has been reused by the OS.
- Login page: remove non-functional password-reset links; add
has_real_admin_users template tag to gate the bootstrap hint.
- Add page: hide underline on the "Get the extension" link.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes https://github.com/ArchiveBox/ArchiveBox/issues/239
## Summary
- add `SERVER_SECURITY_MODE` presets for safe subdomain replay, safe
one-domain no-JS replay, unsafe one-domain no-admin, and dangerous
one-domain full replay
- make host routing, replay URLs, static serving, and control-plane
access mode-aware
- add strict routing/header coverage plus a browser-backed
Chrome/Puppeteer test that verifies real same-origin behavior in all
four modes
## Testing
- `uv run pytest archivebox/tests/test_urls.py -v`
- `uv run pytest archivebox/tests/test_admin_views.py -v`
- `uv run pytest archivebox/tests/test_server_security_browser.py -v`
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/archivebox/archivebox/pull/1773"
target="_blank">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds configurable server security modes to isolate admin/API from
archived content, with a safe subdomain default and single-domain
fallbacks. Routing, replay endpoints, headers, and middleware are
mode-aware, with browser tests validating same-origin behavior.
- New Features
- Introduced SERVER_SECURITY_MODE with presets:
safe-subdomains-fullreplay (default), safe-onedomain-nojsreplay,
unsafe-onedomain-noadmin, danger-onedomain-fullreplay.
- Mode-aware routing and base URLs; one-domain modes use path-based
replay: /snapshot/<id>/... and /original/<domain>/....
- Control plane gate: block admin/API and non-GET methods in
unsafe-onedomain-noadmin; allow full access in
danger-onedomain-fullreplay.
- Safer replay: detect risky HTML/SVG and apply CSP sandbox (no scripts)
in safe-onedomain-nojsreplay; add X-ArchiveBox-Security-Mode and
X-Content-Type-Options: nosniff on replay responses.
- Middleware and serving: added ServerSecurityModeMiddleware, improved
HostRouting, and static server byte-range/CSP handling.
- Tests: added Chrome/Puppeteer browser tests and stricter URL routing
tests covering all modes.
- Migration
- Default requires wildcard subdomains for full isolation (admin., web.,
api., and snapshot-id.<base>).
- To run on one domain, set SERVER_SECURITY_MODE to a one-domain preset;
URLs switch to /snapshot/<id>/ and /original/<domain>/ paths.
- For production, prefer safe-subdomains-fullreplay; lower-security
modes print a startup warning.
<sup>Written for commit ad41b15581.
Summary will update on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
