Commit Graph

5455 Commits

Author SHA1 Message Date
archivebox
2fea001d36
refactor(proxy): replace caddy+lego with a single Traefik container
The https profile is now ONE container (was 3): Traefik terminates TLS and, via
its embedded go-acme/lego (~100 DNS providers), fetches + auto-renews a single
*.<domain> wildcard cert over DNS-01; no DNS provider -> Traefik's default
self-signed cert. Catch-all router -> archivebox:8000 via an inline file-provider
config (no docker socket, no extra files). Config validated against the real
traefik v3.7 binary. Comments document the one manual step (wildcard DNS A-record).

Also address review: derive BASE_URL_IS_HTTPS from the RESOLVED base url
(get_base_url(), so CSRF_TRUSTED_ORIGINS-only upgrades keep HTTPS hardening);
clarify CLOUDFLARE_API_KEY is an API Token (Bearer), not the legacy global key.
2026-06-05 04:18:41 +00:00
archivebox
f7117a40cd
refactor(proxy): derive scheme from BASE_URL, provider-agnostic certs, slim tunnel-init
- Drop REVERSE_PROXY_TRUST_FORWARDED_PROTO; derive SECURE_PROXY_SSL_HEADER +
  Secure cookies from BASE_URL's https:// scheme (one knob, no overlapping config).
- Caddy https profile is no longer Cloudflare-specific: wildcard cert via lego
  with ANY of ~150 DNS providers (ARCHIVEBOX_ACME_DNS + provider's native creds
  passed through .env), OR no DNS provider at all -> plain Let's Encrypt on-demand
  per-host (HTTP-01), OR self-signed internal locally. Caddyfiles validated.
- tunnel-init: use python:3-alpine instead of the heavy archivebox image.
- Rename CLOUDFLARE_DNS_API_TOKEN -> CLOUDFLARE_API_KEY for the tunnel's CF API use.
2026-06-05 02:29:49 +00:00
archivebox
f3af045974
fix(proxy): make token chown best-effort so it can't crash tunnel-init
Always chmod 0600 (never world-readable), then best-effort chown to the
cloudflared uid 65532; if the mount disallows chown, warn and keep the private
0600 file rather than crashing tunnel-init or falling back to world-readable.
2026-06-05 00:08:43 +00:00
archivebox
fc7011d988
fix(proxy): drop 0644 token fallback; chown 0600 (tunnel-init runs as root)
The archivebox image starts as root (privilege drop happens in its entrypoint,
which tunnel-init overrides), so chown to the cloudflared uid always succeeds and
the world-readable 0644 fallback was dead code that could leak the token. Always
write the connector token owned by 65532:65532 with 0600.
2026-06-05 00:04:37 +00:00
archivebox
d719fb6916
fix(proxy): make tunnel token readable by the cloudflared connector
The cloudflared image runs as non-root (uid 65532), so a root-owned 0600 token
was unreadable by the connector. chown the token to 65532:65532 and keep 0600;
if tunnel-init isn't root (can't chown), fall back to 0644 so it stays readable.
2026-06-05 00:01:10 +00:00
archivebox
102a3ab4bc
fix(proxy): address review — secure cookies, zone lookup, token perms
- settings.py: gate CSRF_COOKIE_SECURE/SESSION_COOKIE_SECURE on
  REVERSE_PROXY_TRUST_FORWARDED_PROTO so HTTPS deployments never send auth
  cookies over plain HTTP (default-off path unchanged).
- docker-compose.yml tunnel-init: resolve the Cloudflare zone by walking the
  domain labels to the registrable zone, so a subdomain BASE_URL no longer
  crashes provisioning; write the connector token with 0o600 perms.
- Remove .env.example; the env vars are documented inline in docker-compose.yml.
- etc/README.md: drop the .env.example reference.
2026-06-04 23:54:52 +00:00
archivebox
66fea50e81
docs(etc): fix README refs to deleted proxy overlay files
The reverse-proxy/TLS setup now lives entirely in the main docker-compose.yml as
the env-driven https/tunnel profiles; update etc/README.md which still pointed at
the removed docker-compose.proxy.yml, bin/proxy_lego.sh, and .env.proxy.example.
2026-06-04 23:29:58 +00:00
archivebox
f5e47d8197
feat(docker): zero-touch Cloudflare Tunnel provisioning from API token
Make the tunnel ingress fully automatic from env vars — no dashboard clicks,
no cloudflared login, no cert.pem. A one-shot tunnel-init step (reusing the
archivebox image's python, so no extra image) uses CLOUDFLARE_DNS_API_TOKEN to:
create/reuse a tunnel named archivebox-<domain>, set its ingress
(*.<domain> + <domain> -> archivebox:8000, Host-routed), upsert the proxied
apex+wildcard CNAMEs to the tunnel, and write the connector token; cloudflared
then runs it (depends_on service_completed_successfully). Idempotent.

Add .env.example documenting BASE_URL / SERVER_SECURITY_MODE / COMPOSE_PROFILES
/ CLOUDFLARE_DNS_API_TOKEN as the only required config.
2026-06-04 23:06:30 +00:00
archivebox
f50aa0f274
feat(docker): fold ingress into main compose, env-driven https + tunnel profiles
Consolidate the reverse-proxy setup into the single docker-compose.yml (no
overlay file, no separate Dockerfile or shell scripts — all inline) and drive
everything from normal env vars. Two opt-in ingress profiles:

  https  - caddy + go-acme/lego: obtains ONE *.<domain> Let's Encrypt wildcard
           cert via DNS-01 (CLOUDFLARE_DNS_API_TOKEN) covering unlimited snap-*
           subdomains, and GRACEFULLY DEGRADES to a self-signed internal cert if
           no cert is available yet so the site always serves https.
  tunnel - cloudflared: Cloudflare edge terminates TLS and routes *.<domain>
           through a SINGLE tunnel to archivebox:8000, which routes by Host; no
           public IP / wildcard cert needed (TUNNEL_TOKEN).

Base domain is derived from BASE_URL; lego v5 CLI syntax. Removes
docker-compose.proxy.yml, etc/Caddyfile, bin/proxy_lego.sh, .env.proxy.example.
2026-06-04 22:52:49 +00:00
archivebox
ead3921bf9
feat(docker): add opt-in wildcard DNS resolver to proxy profile
Adds a dnsmasq 'resolver' service (--profile proxy-resolver) that answers
the entire base domain with ONE suffix-wildcard rule
(address=/<base>/<target>), never enumerating the millions of dynamically
generated snap-*/role subdomains. Covers the gap where non-browser clients
(wget/python/go) on hosts without systemd-resolved, or LAN/NAS internal
domains, need to resolve snapshot subdomains. Browsers, curl, and
systemd-resolved hosts already resolve *.localhost and need nothing.

Rule validated with 'dnsmasq --test'; env-driven via ARCHIVEBOX_PROXY_BASE
and ARCHIVEBOX_PROXY_DNS_TARGET.
2026-06-04 22:20:55 +00:00
archivebox
a7514a7eb8
feat(docker): add opt-in bundled reverse-proxy profile (Caddy + lego)
Make the default safe-subdomains-fullreplay mode painless to deploy
without baking heavy proxy/cert deps into the archivebox image.

docker-compose.proxy.yml is an opt-in overlay that adds:
  - caddy: wildcard catch-all reverse proxy (etc/Caddyfile) that routes
    every dynamically-generated snapshot/role subdomain to archivebox by
    suffix wildcard (never an enumerated list), preserves Host, and sets
    X-Forwarded-Proto. Local default serves *.archivebox.localhost via
    Caddy's internal CA.
  - lego (proxy-public profile): a go-acme/lego DNS-01 sidecar
    (bin/proxy_lego.sh) that fetches ONE *.<domain> wildcard cert covering
    unlimited snap-* subdomains. Provider coverage (~150: cloudflare,
    route53, gcloud, digitalocean, hetzner, namecheap, godaddy, ...) is
    delegated to lego via standard env vars, with no per-provider code.

When proxied, the archivebox service stops publishing :8000 directly and
sets REVERSE_PROXY_TRUST_FORWARDED_PROTO=True so secure cookies + https
links work end-to-end. Replaces the stale nginx example in
docker-compose.yml. See .env.proxy.example for public setup.
2026-06-04 22:15:35 +00:00
archivebox
a9cc7f887b
feat(server): honor X-Forwarded-Proto behind a trusted reverse proxy
Add REVERSE_PROXY_TRUST_FORWARDED_PROTO (default off). When enabled,
Django's SECURE_PROXY_SSL_HEADER is set so request.is_secure() and
request.scheme reflect the real client-facing scheme behind a
TLS-terminating proxy (the bundled Caddy profile, a user's own
caddy/traefik/nginx, or an ingress like Cloudflare).

Without it, the proxy->archivebox hop is plain HTTP, so the
cross-subdomain login-hint cookie loses its Secure flag and
request-derived schemes fall back to http. Gated behind a config flag
because trusting a forwarded header is only safe when a controlled proxy
always sets it.
2026-06-04 22:04:21 +00:00
Nick Sweeting
bdff741c85
release: archivebox 0.9.34rc59
Some checks are pending
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-amd64, docker-amd64, linux/amd64, ubuntu-24.04) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-arm64, docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Waiting to run
Build Docker image / publish multiarch tags (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Release State / release-state (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Parallel Tests / ${{ matrix.plugin.name }} (push) Blocked by required conditions
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
Update Homebrew tap / dispatch (push) Waiting to run
2026-06-04 10:05:16 -07:00
Nick Sweeting
ba895db80e
release: archivebox 0.9.34rc58 2026-06-04 09:20:18 -07:00
Nick Sweeting
364efdf409
fix release pypi retry loop 2026-06-04 09:17:34 -07:00
Nick Sweeting
9a1919c2f4
fix docker env syntax 2026-06-04 09:15:16 -07:00
Nick Sweeting
91f6106434
release: archivebox 0.9.34rc56 2026-06-04 09:14:36 -07:00
Nick Sweeting
cf4bd959c0
fix private snapshot replay auth 2026-06-04 09:12:56 -07:00
Nick Sweeting
8367578f9b
Update snapshot admin selection behavior 2026-06-04 08:00:25 -07:00
Nick Sweeting
5de536e06d
release: archivebox 0.9.34rc55
Some checks are pending
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-amd64, docker-amd64, linux/amd64, ubuntu-24.04) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-arm64, docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Waiting to run
Build Docker image / publish multiarch tags (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Release State / release-state (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Parallel Tests / ${{ matrix.plugin.name }} (push) Blocked by required conditions
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
Update Homebrew tap / dispatch (push) Waiting to run
2026-06-04 01:26:32 -07:00
Nick Sweeting
caf0acdec8
release: archivebox 0.9.34rc54 2026-06-04 00:48:47 -07:00
Nick Sweeting
70b325f959
release: archivebox 0.9.34rc53 2026-06-04 00:19:42 -07:00
Nick Sweeting
083ee8cb7d
Remove abx-plugins Docker commit pin 2026-06-04 00:01:29 -07:00
Nick Sweeting
79b59945e5
release: v0.9.34rc52 2026-06-03 23:55:32 -07:00
Nick Sweeting
3c6d94d03a
release: v0.9.34rc51 2026-06-03 23:54:12 -07:00
Nick Sweeting
2e9686b901
release: v0.9.34rc51 2026-06-03 23:21:37 -07:00
Nick Sweeting
40d5dc06a4
release: v0.9.34rc50 2026-06-03 22:59:35 -07:00
Nick Sweeting
bb0bf81e02
release: v0.9.34rc49 2026-06-03 22:58:26 -07:00
Nick Sweeting
d26e27d139
release: archivebox 0.9.34rc49 2026-06-03 22:37:51 -07:00
Nick Sweeting
327e378a9e
release: archivebox 0.9.34rc48 2026-06-03 22:07:19 -07:00
Nick Sweeting
77ee827a58
release: archivebox 0.9.34rc47 2026-06-03 21:37:13 -07:00
Nick Sweeting
5b74eb5238
fix: publish pip builds from pypi environment
Some checks failed
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-amd64, docker-amd64, linux/amd64, ubuntu-24.04) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-arm64, docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Waiting to run
Build Docker image / publish multiarch tags (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Release State / release-state (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Parallel Tests / ${{ matrix.plugin.name }} (push) Blocked by required conditions
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
Build Debian package / build (amd64) (push) Has been cancelled
Build Debian package / build (arm64) (push) Has been cancelled
Build Debian package / test (amd64, ubuntu-24.04) (push) Has been cancelled
Build Debian package / test (arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build Debian package / release (push) Has been cancelled
2026-06-03 20:01:42 -07:00
Nick Sweeting
8e1c7dccaa
fix: wait for released deps before docker build 2026-06-03 19:45:43 -07:00
Nick Sweeting
50ff26b524
release: archivebox 0.9.34rc46 2026-06-03 19:42:16 -07:00
Nick Sweeting
9e06c32c18
fix: install archivebox extras explicitly in ci 2026-06-03 19:34:08 -07:00
Nick Sweeting
bcd10ea139
fix: resolve local package deps in archivebox ci 2026-06-03 19:30:42 -07:00
Nick Sweeting
b8b70f3525
release: archivebox 0.9.34rc45 2026-06-03 19:13:11 -07:00
Nick Sweeting
74690eb52b
fix: wait for released deps before pip build 2026-06-03 19:02:35 -07:00
Nick Sweeting
afa958a6d5
release: archivebox 0.9.34rc44 2026-06-03 18:57:23 -07:00
Nick Sweeting
1cf0d18444
release: archivebox 0.9.34rc43 2026-06-03 18:24:54 -07:00
Nick Sweeting
7698155cb9
release: archivebox 0.9.34rc42 2026-06-03 17:54:41 -07:00
Nick Sweeting
1ab99027d8
release: archivebox 0.9.34rc41 2026-06-03 17:41:51 -07:00
Nick Sweeting
e0c3ca67e4
release: archivebox 0.9.34rc40 2026-06-03 17:32:17 -07:00
Nick Sweeting
c0fb8eb532
release: archivebox 0.9.34rc39 2026-06-03 17:19:48 -07:00
Nick Sweeting
6da1d3f5af
ci: give archivebox chrome install a real timeout
Some checks are pending
CodeQL / Analyze (${{ matrix.language }}) (none, python) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-amd64, docker-amd64, linux/amd64, ubuntu-24.04) (push) Waiting to run
Build Docker image / build ${{ matrix.platform }} (digest-linux-arm64, docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Waiting to run
Build Docker image / publish multiarch tags (push) Blocked by required conditions
Run linters / lint (push) Waiting to run
Build Pip package / build (push) Waiting to run
Release State / release-state (push) Waiting to run
Parallel Tests / Discover test files (push) Waiting to run
Parallel Tests / ${{ matrix.test.name }} (push) Blocked by required conditions
Parallel Tests / ${{ matrix.plugin.name }} (push) Blocked by required conditions
Run tests / python_tests (ubuntu-22.04, 3.13) (push) Waiting to run
Run tests / docker_tests (push) Waiting to run
2026-06-03 00:02:26 -07:00
Nick Sweeting
83a2099851
fix: scope update search backfill runner 2026-06-02 21:54:25 -07:00
Nick Sweeting
3669133a05
fix: allow install to initialize collections 2026-06-02 21:25:20 -07:00
Nick Sweeting
4e9c8b1475
ci: install chrome through archivebox 2026-06-02 21:21:39 -07:00
Nick Sweeting
9f0544857c
test: require success in cli workflows 2026-06-02 21:19:21 -07:00
Nick Sweeting
acc830d30f
test: exercise extract cli with real outputs 2026-06-02 21:07:52 -07:00