Commit Graph

622 Commits

Author SHA1 Message Date
Nick Sweeting
cadd3f517d
release: archivebox 0.9.31rc30 2026-05-24 02:05:13 -07:00
Nick Sweeting
035132ded5
release: archivebox 0.9.31rc27 2026-05-24 01:24:50 -07:00
Nick Sweeting
8c761cb424
Pass query data to SavePageNow form 2026-05-23 15:29:10 -07:00
Nick Sweeting
92339e08f0
Prepare dev demo deploy 2026-05-22 20:12:48 -07:00
Nick Sweeting
c1593fe55b
Report supervisord runner in live progress 2026-05-18 02:10:45 -07:00
Nick Sweeting
ea4ce5b641
Prepare ArchiveBox 0.9.30rc81 demo release 2026-05-18 01:51:55 -07:00
Nick Sweeting
0e2ae425ce
Start Sonic worker under supervisord 2026-05-17 21:24:02 -07:00
Nick Sweeting
33b3944c2a
Gate debug urls on enabled debug apps
Some checks are pending
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Debian package / build (amd64) (push) Waiting to run
Build Debian package / build (arm64) (push) Waiting to run
Build Debian package / test (amd64, ubuntu-24.04) (push) Blocked by required conditions
Build Debian package / test (arm64, ubuntu-24.04-arm) (push) Blocked by required conditions
Build Debian package / release (push) Blocked by required conditions
Build Docker image / buildx (push) Waiting to run
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Release State / release-state (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Parallel Tests / ${{ matrix.plugin.name }} (push) Blocked by required conditions
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
2026-05-17 00:17:46 -07:00
Nick Sweeting
0cc4251c8a
Update ArchiveBox for abx dependency releases 2026-05-16 23:50:16 -07:00
Nick Sweeting
5f8098c632
fix recursive crawls and docker runtime 2026-05-13 08:13:33 -07:00
Claude
7c3a3e0dba
Put tag slug back in JS download filename
Address pirate's review: restore the slug in the client-side
download fallback filename. Expose tag.slug as data-slug on the
card element and in the search card schema so the JS can read it
directly without slugifying client-side.
2026-04-21 17:35:57 +00:00
Claude
2ea66d05d1
Move tag slug logic onto Tag.slug @property
Replaces the tag_filename_safe() helper with a Tag.slug property
that returns the slugified form via django.utils.text.slugify.
Call sites now just use tag.slug directly.
2026-04-21 17:32:25 +00:00
Claude
0041a2d407
Sanitize tag export filenames via django.utils.text.slugify
Some checks failed
Build Docker image / buildx (push) Has been cancelled
Run linters / lint (push) Has been cancelled
Build Pip package / build (push) Has been cancelled
Release State / release-state (push) Has been cancelled
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Has been cancelled
Run tests / docker_tests (push) Has been cancelled
Addresses review feedback from cubic and devin: quote()'s percent-
encoding isn't decoded by browsers in Content-Disposition's filename
parameter (Safari saves literal %20). Switch to Django's slugify()
which does NFKD normalization, ASCII transliteration, and replaces
punctuation with hyphens — producing clean names like
"tag-alpha-research-urls.txt".

- Add tag_filename_safe(name) helper wrapping slugify
- Use it in both tag export endpoints
- Drop the now-unneeded JS fallback name (server always sets
  Content-Disposition)
2026-04-21 17:30:50 +00:00
Claude
ec9c7c89f4
Drop Tag slug column and use URL-encoded names
Tags now support full unicode with no restrictions. URL-encode the tag
name wherever it previously used the slug (export filenames, lookups).

- Remove `slug` field, `_generate_unique_slug`, and slug handling in save()
- Add migration 0034 to drop the slug column
- `get_tag_by_ref` now resolves by URL-decoded exact name match
- Tag search/autocomplete/export filenames use the name directly
- Drop slug from admin search_fields/readonly_fields/fieldsets
- Remove slug display from similar-tag cards and client download filename
2026-04-20 17:05:07 +00:00
Nick Sweeting
0b9b3b7e54
split tag editor issue 2026-04-07 20:21:58 -07:00
Nick Sweeting
4d66996569
small fixes
Some checks are pending
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Docker image / buildx (push) Waiting to run
Deploy static content to Pages / deploy (push) Waiting to run
Build GitHub Pages website / build (push) Waiting to run
Build GitHub Pages website / deploy (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Release State / release-state (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Parallel Tests / Plugin tests (push) Waiting to run
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
2026-04-06 23:47:38 -07:00
Nick Sweeting
f3622d8cd3 update working changes
Some checks failed
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Has been cancelled
Build Debian package / build (amd64) (push) Has been cancelled
Build Debian package / build (arm64) (push) Has been cancelled
Build Docker image / buildx (push) Has been cancelled
Deploy static content to Pages / deploy (push) Has been cancelled
Build GitHub Pages website / build (push) Has been cancelled
Run linters / lint (push) Has been cancelled
Build Pip package / build (push) Has been cancelled
Release State / release-state (push) Has been cancelled
Parallel Tests / Discover test files (push) Has been cancelled
Parallel Tests / Plugin tests (push) Has been cancelled
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Has been cancelled
Run tests / docker_tests (push) Has been cancelled
Build Debian package / test (amd64, ubuntu-24.04) (push) Has been cancelled
Build Debian package / test (arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build Debian package / release (push) Has been cancelled
Build GitHub Pages website / deploy (push) Has been cancelled
Parallel Tests / ${{ matrix.test.name }} (push) Has been cancelled
2026-03-25 05:36:07 -07:00
Nick Sweeting
50286d3c38 Reuse cached binaries in archivebox runtime
Some checks are pending
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Docker image / buildx (push) Waiting to run
Deploy static content to Pages / deploy (push) Waiting to run
Build GitHub Pages website / build (push) Waiting to run
Build GitHub Pages website / deploy (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Release State / release-state (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Parallel Tests / Plugin tests (push) Waiting to run
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
2026-03-24 11:03:43 -07:00
Nick Sweeting
39450111dd
Update CI uv handling and runner changes
Some checks are pending
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Debian package / build (amd64) (push) Waiting to run
Build Debian package / build (arm64) (push) Waiting to run
Build Debian package / test (amd64, ubuntu-24.04) (push) Blocked by required conditions
Build Debian package / test (arm64, ubuntu-24.04-arm) (push) Blocked by required conditions
Build Debian package / release (push) Blocked by required conditions
Build Docker image / buildx (push) Waiting to run
Deploy static content to Pages / deploy (push) Waiting to run
Build GitHub Pages website / build (push) Waiting to run
Build GitHub Pages website / deploy (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Release State / release-state (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Parallel Tests / Plugin tests (push) Waiting to run
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
2026-03-23 13:27:23 -07:00
Nick Sweeting
25f935b9d1
split CrawlSetup into Install phase with new Binary + BinaryRequest events 2026-03-23 13:15:41 -07:00
Nick Sweeting
b749b26c5d
wip 2026-03-23 03:58:32 -07:00
Nick Sweeting
f400a2cd67
WIP: checkpoint working tree before rebasing onto dev 2026-03-22 20:25:18 -07:00
Nick Sweeting
a6548df8d0
Add configurable server security modes (#1773)
Some checks are pending
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Docker image / buildx (push) Waiting to run
Deploy static content to Pages / deploy (push) Waiting to run
Build GitHub Pages website / build (push) Waiting to run
Build GitHub Pages website / deploy (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Parallel Tests / Plugin tests (push) Waiting to run
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
Fixes https://github.com/ArchiveBox/ArchiveBox/issues/239

## Summary
- add `SERVER_SECURITY_MODE` presets for safe subdomain replay, safe
one-domain no-JS replay, unsafe one-domain no-admin, and dangerous
one-domain full replay
- make host routing, replay URLs, static serving, and control-plane
access mode-aware
- add strict routing/header coverage plus a browser-backed
Chrome/Puppeteer test that verifies real same-origin behavior in all
four modes

## Testing
- `uv run pytest archivebox/tests/test_urls.py -v`
- `uv run pytest archivebox/tests/test_admin_views.py -v`
- `uv run pytest archivebox/tests/test_server_security_browser.py -v`

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/archivebox/archivebox/pull/1773"
target="_blank">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds configurable server security modes to isolate admin/API from
archived content, with a safe subdomain default and single-domain
fallbacks. Routing, replay endpoints, headers, and middleware are
mode-aware, with browser tests validating same-origin behavior.

- New Features
- Introduced SERVER_SECURITY_MODE with presets:
safe-subdomains-fullreplay (default), safe-onedomain-nojsreplay,
unsafe-onedomain-noadmin, danger-onedomain-fullreplay.
- Mode-aware routing and base URLs; one-domain modes use path-based
replay: /snapshot/<id>/... and /original/<domain>/....
- Control plane gate: block admin/API and non-GET methods in
unsafe-onedomain-noadmin; allow full access in
danger-onedomain-fullreplay.
- Safer replay: detect risky HTML/SVG and apply CSP sandbox (no scripts)
in safe-onedomain-nojsreplay; add X-ArchiveBox-Security-Mode and
X-Content-Type-Options: nosniff on replay responses.
- Middleware and serving: added ServerSecurityModeMiddleware, improved
HostRouting, and static server byte-range/CSP handling.
- Tests: added Chrome/Puppeteer browser tests and stricter URL routing
tests covering all modes.

- Migration
- Default requires wildcard subdomains for full isolation (admin., web.,
api., and snapshot-id.<base>).
- To run on one domain, set SERVER_SECURITY_MODE to a one-domain preset;
URLs switch to /snapshot/<id>/ and /original/<domain>/ paths.
- For production, prefer safe-subdomains-fullreplay; lower-security
modes print a startup warning.

<sup>Written for commit ad41b15581.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-03-22 20:17:21 -07:00
Nick Sweeting
c87079aa0a
Refactor ArchiveBox onto abx-dl bus runner
Some checks failed
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Has been cancelled
Build Debian package / build (amd64) (push) Has been cancelled
Build Debian package / build (arm64) (push) Has been cancelled
Build Docker image / buildx (push) Has been cancelled
Deploy static content to Pages / deploy (push) Has been cancelled
Build GitHub Pages website / build (push) Has been cancelled
Run linters / lint (push) Has been cancelled
Build Pip package / build (push) Has been cancelled
Parallel Tests / Discover test files (push) Has been cancelled
Parallel Tests / Plugin tests (push) Has been cancelled
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Has been cancelled
Run tests / docker_tests (push) Has been cancelled
Build Debian package / test (amd64, ubuntu-24.04) (push) Has been cancelled
Build Debian package / test (arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build Debian package / release (push) Has been cancelled
Build GitHub Pages website / deploy (push) Has been cancelled
Parallel Tests / ${{ matrix.test.name }} (push) Has been cancelled
2026-03-21 11:47:57 -07:00
Nick Sweeting
ad41b15581
Add configurable server security modes 2026-03-15 23:34:40 -07:00
Nick Sweeting
57e11879ec
cleanup archivebox tests 2026-03-15 22:09:56 -07:00
Nick Sweeting
9de084da65
bump package versions 2026-03-15 20:47:28 -07:00
Nick Sweeting
bc21d4bfdb
type and test fixes 2026-03-15 20:12:27 -07:00
Nick Sweeting
3889eb4efa
Tighten config and admin typing 2026-03-15 19:49:52 -07:00
Nick Sweeting
44cabac8d0
fix typing 2026-03-15 19:47:36 -07:00
Nick Sweeting
934e02695b
fix lint 2026-03-15 18:45:29 -07:00
Nick Sweeting
70c9358cf9
Improve scheduling, runtime paths, and API behavior 2026-03-15 18:31:56 -07:00
Nick Sweeting
7d42c6c8b5
bump versions and fix docs 2026-03-15 17:43:07 -07:00
Nick Sweeting
e598614b05
Avoid filesystem lookups in snapshot admin list
Some checks are pending
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Debian package / build (amd64) (push) Waiting to run
Build Debian package / build (arm64) (push) Waiting to run
Build Debian package / test (amd64, ubuntu-24.04) (push) Blocked by required conditions
Build Debian package / test (arm64, ubuntu-24.04-arm) (push) Blocked by required conditions
Build Debian package / release (push) Blocked by required conditions
Build Docker image / buildx (push) Waiting to run
Deploy static content to Pages / deploy (push) Waiting to run
Build GitHub Pages website / build (push) Waiting to run
Build GitHub Pages website / deploy (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Parallel Tests / Plugin tests (push) Waiting to run
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
2026-03-15 17:18:53 -07:00
Nick Sweeting
31e883ec53
Stabilize plugin and crawl integration tests 2026-03-15 08:16:52 -07:00
Nick Sweeting
1f792d7199
Restore CLI compat and plugin dependency handling 2026-03-15 06:06:18 -07:00
Nick Sweeting
c4d30a853f
Restore index-only snapshot output links 2026-03-15 04:58:46 -07:00
Nick Sweeting
4fa701fafe
Update abx dependencies and plugin test harness 2026-03-15 04:37:32 -07:00
Nick Sweeting
ecb1764590
switch to external plugins 2026-03-15 03:46:23 -07:00
Nick Sweeting
07dc880d0b
Harden AddView config overrides to admin-only 2026-03-15 03:45:57 -07:00
Your Name
08b0dfaf12 Fix #1139: Return tags as a JSON list in Snapshot.to_dict() for LLM/RAG integration
Previously, `archivebox search --json` exported tags as a comma-separated
string (e.g. "tag1,tag2"), which required manual parsing by consumers like
LlamaIndex, LangChain, and other RAG frameworks.

Now `to_dict()` returns tags as a proper JSON array (e.g. ["tag1", "tag2"]),
making the export directly usable as structured metadata in LLM/RAG pipelines
without additional preprocessing.

`from_json()` is updated to accept both list and string formats for backward
compatibility with existing JSON imports.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 21:21:38 -08:00
Nick Sweeting
ec4b27056e
wip
Some checks failed
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Has been cancelled
Build Debian package / build (push) Has been cancelled
Deploy static content to Pages / deploy (push) Has been cancelled
Build Homebrew package / build (push) Has been cancelled
Build GitHub Pages website / build (push) Has been cancelled
Run linters / lint (push) Has been cancelled
Build Pip package / build (push) Has been cancelled
Parallel Tests / Discover test files (push) Has been cancelled
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Has been cancelled
Run tests / docker_tests (push) Has been cancelled
Build GitHub Pages website / deploy (push) Has been cancelled
Parallel Tests / ${{ matrix.test.name }} (push) Has been cancelled
2026-01-21 03:19:56 -08:00
Nick Sweeting
f3f55d3395
perfect snapshot detail cards 2026-01-19 14:56:15 -08:00
Nick Sweeting
86e7973334
cleanup tui, startup, card templtes, and more
Some checks failed
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Has been cancelled
Build Debian package / build (push) Has been cancelled
Deploy static content to Pages / deploy (push) Has been cancelled
Build Homebrew package / build (push) Has been cancelled
Build GitHub Pages website / build (push) Has been cancelled
Run linters / lint (push) Has been cancelled
Build Pip package / build (push) Has been cancelled
Parallel Tests / Discover test files (push) Has been cancelled
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Has been cancelled
Run tests / docker_tests (push) Has been cancelled
Build GitHub Pages website / deploy (push) Has been cancelled
Parallel Tests / ${{ matrix.test.name }} (push) Has been cancelled
2026-01-19 14:33:20 -08:00
Nick Sweeting
c7b2217cd6
tons of fixes with codex 2026-01-19 01:00:53 -08:00
claude[bot]
c2bb4b25cb Implement native LDAP authentication support
- Create archivebox/config/ldap.py with LDAPConfig class
- Create archivebox/ldap/ Django app with custom auth backend
- Update core/settings.py to conditionally load LDAP when enabled
- Add LDAP_CREATE_SUPERUSER support to auto-grant superuser privileges
- Add comprehensive tests in test_auth_ldap.py (no mocks, no skips)
- LDAP only activates if django-auth-ldap is installed and LDAP_ENABLED=True
- Helpful error messages when LDAP libraries are missing or config is incomplete

Fixes #1664

Co-authored-by: Nick Sweeting <pirate@users.noreply.github.com>
2026-01-05 21:30:26 +00:00
Nick Sweeting
7ceaeae2d9
rename archive_org to archivedotorg, add BinaryWorker, fix config pass-through 2026-01-04 22:38:15 -08:00
Nick Sweeting
456aaee287
more migration id/uuid and config propagation fixes
Some checks are pending
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Debian package / build (push) Waiting to run
Deploy static content to Pages / deploy (push) Waiting to run
Build Homebrew package / build (push) Waiting to run
Build GitHub Pages website / build (push) Waiting to run
Build GitHub Pages website / deploy (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
2026-01-04 16:16:26 -08:00
Nick Sweeting
839ae744cf
simplify entrypoints for orchestrator and workers
Some checks are pending
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Debian package / build (push) Waiting to run
Deploy static content to Pages / deploy (push) Waiting to run
Build Homebrew package / build (push) Waiting to run
Build GitHub Pages website / build (push) Waiting to run
Build GitHub Pages website / deploy (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
2026-01-04 13:17:07 -08:00
Nick Sweeting
3da523fc74
more consistent crawl, snapshot, hook cleanup and Process tracking 2026-01-02 04:27:38 -08:00