mirror of
https://github.com/ArchiveBox/ArchiveBox.git
synced 2026-09-12 11:40:25 +05:00
Clarify startup and local runner ownership
This commit is contained in:
Nick Sweeting
parent
fedb091b87
commit
70e477c1c8
47
README.md
47
README.md
@ -80,8 +80,6 @@ docker compose up -d --wait # ini
|
||||
<br/>
|
||||
# Option B: Or use it as a plain Docker container:
|
||||
mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
docker run --rm -it -v "$PWD:/data" archivebox/archivebox:dev init
|
||||
docker run --rm -it -v "$PWD:/data" archivebox/archivebox:dev install
|
||||
docker run -d --name archivebox -v "$PWD:/data" -p 8000:8000 archivebox/archivebox:dev
|
||||
# open http://admin.archivebox.localhost:8000 to finish setup
|
||||
# docker run -it -v $PWD:/data archivebox/archivebox:dev add 'https://example.com'
|
||||
@ -187,17 +185,16 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
|
||||
<br/>
|
||||
<ol>
|
||||
<li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
|
||||
<li>Create a new empty directory and initialize your collection (can be anywhere).
|
||||
<li>Create a new empty directory and start the server, which initializes the collection automatically (can be anywhere).
|
||||
<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
docker run --rm -v $PWD:/data -it archivebox/archivebox:dev init
|
||||
docker run --rm -v $PWD:/data -it archivebox/archivebox:dev install
|
||||
docker run -d --name archivebox -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
|
||||
</code></pre>
|
||||
</li>
|
||||
<li>Optional: Start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
|
||||
<pre lang="bash"><code style="white-space: pre-line">docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
|
||||
<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
|
||||
<pre lang="bash"><code style="white-space: pre-line">
|
||||
# completely optional, CLI can always be used without running a server
|
||||
# docker run -v $PWD:/data -it archivebox/archivebox:dev [subcommand] [--help]
|
||||
docker run -v $PWD:/data -it archivebox/archivebox:dev help
|
||||
# docker exec archivebox archivebox [subcommand] [--help]
|
||||
docker exec archivebox archivebox help
|
||||
</code></pre>
|
||||
<i>For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-a-docker--docker-compose-setup-%EF%B8%8F">Install: Docker Compose</a> in the Wiki. ➡️</i>
|
||||
</li>
|
||||
@ -497,12 +494,10 @@ archivebox help # get list of archivebox subcommands that can be ru
|
||||
<pre lang="bash"><code style="white-space: pre-line">
|
||||
# make sure you have `docker-compose.yml` from the Quickstart instructions first
|
||||
<br/>
|
||||
# docker compose run --rm archivebox [subcommand] [--help]
|
||||
docker compose run --rm archivebox init
|
||||
docker compose run --rm archivebox install
|
||||
docker compose run --rm archivebox version
|
||||
docker compose run --rm archivebox help
|
||||
docker compose run --rm archivebox add 'https://example.com'
|
||||
# docker compose exec archivebox archivebox [subcommand] [--help]
|
||||
docker compose exec archivebox archivebox version
|
||||
docker compose exec archivebox archivebox help
|
||||
docker compose exec archivebox archivebox add 'https://example.com'
|
||||
# to start webserver: docker compose up
|
||||
</code></pre>
|
||||
<i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage">Usage: Docker Compose CLI</a> wiki. ➡️</i>
|
||||
@ -514,15 +509,12 @@ docker compose run --rm archivebox add 'https://example.com'
|
||||
<summary><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="22px" align="top"/> <b>CLI Usage Examples: Docker</b></summary>
|
||||
<br/>
|
||||
<pre lang="bash"><code style="white-space: pre-line">
|
||||
# make sure you create and cd into in a new empty directory first
|
||||
# make sure the `archivebox` server container from the Quickstart is running first
|
||||
<br/>
|
||||
# docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]
|
||||
docker run -v $PWD:/data -it archivebox/archivebox:dev init
|
||||
docker run -v $PWD:/data -it archivebox/archivebox:dev install
|
||||
docker run -v $PWD:/data -it archivebox/archivebox:dev version
|
||||
docker run -v $PWD:/data -it archivebox/archivebox:dev help
|
||||
docker run -v $PWD:/data -it archivebox/archivebox:dev add 'https://example.com'
|
||||
# to start webserver: docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
|
||||
# docker exec archivebox archivebox [subcommand] [--help]
|
||||
docker exec archivebox archivebox version
|
||||
docker exec archivebox archivebox help
|
||||
docker exec archivebox archivebox add 'https://example.com'
|
||||
</code></pre>
|
||||
<i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage-1">Usage: Docker CLI</a> wiki. ➡️</i>
|
||||
</details>
|
||||
@ -1345,14 +1337,11 @@ archivebox server 0.0.0.0:8000
|
||||
# inside the container will reload and pick up your changes
|
||||
./bin/build_docker.sh dev
|
||||
|
||||
docker run -it -v $PWD/data:/data archivebox/archivebox:dev init
|
||||
docker run -it -v $PWD/data:/data archivebox/archivebox:dev install
|
||||
|
||||
# Run the development server w/ autoreloading (but no bg workers)
|
||||
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --debug --reload 0.0.0.0:8000
|
||||
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --init --debug --reload 0.0.0.0:8000
|
||||
|
||||
# Run the production server (with bg workers but no autoreloading)
|
||||
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server
|
||||
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --init
|
||||
|
||||
# (remove the --reload flag and add the --nothreading flag when profiling with the django debug toolbar)
|
||||
# When using --reload, make sure any files you create can be read by the user in the Docker container, eg with 'chmod a+rX'.
|
||||
@ -1413,7 +1402,7 @@ services:
|
||||
|
||||
# or with plain Docker:
|
||||
docker build -t archivebox:dev https://github.com/ArchiveBox/ArchiveBox.git#dev
|
||||
docker run -it -v $PWD:/data archivebox:dev init
|
||||
docker run -it -v $PWD:/data -p 8000:8000 archivebox:dev
|
||||
|
||||
# or with uv:
|
||||
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
|
||||
|
||||
@ -1,3 +1,17 @@
|
||||
"""Coordinate local foreground processes without pretending to provide a distributed lock.
|
||||
|
||||
ArchiveBox wants one orchestrator per collection, but Process rows can only prove
|
||||
liveness for PIDs visible on the current machine/PID namespace. We therefore
|
||||
enforce one active runner per ``(Machine, DATA_DIR)`` locally, retire stale rows
|
||||
from sequential containers on that machine, and only warn about rows owned by a
|
||||
different machine. Foreign-machine rows must never block progress or be killed:
|
||||
multi-machine coordination belongs in the Crawl/Snapshot CAS claim layer, not in
|
||||
process takeover.
|
||||
|
||||
These helpers only hand local supervisord/runner ownership between CLI parents.
|
||||
They must not hold database transactions or filesystem locks while work runs.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
@ -15,25 +29,6 @@ RUNNER_WAITING_WORKER_TYPE = "runner_waiting"
|
||||
RUNNER_GATE_WORKER_TYPES = (RUNNER_ACTIVE_WORKER_TYPE, RUNNER_WAITING_WORKER_TYPE, "")
|
||||
|
||||
|
||||
def runtime_stack_owner_types():
|
||||
from archivebox.machine.models import Process
|
||||
|
||||
return (
|
||||
Process.TypeChoices.SERVER,
|
||||
Process.TypeChoices.ORCHESTRATOR,
|
||||
)
|
||||
|
||||
|
||||
def foreground_runner_owner_types():
|
||||
from archivebox.machine.models import Process
|
||||
|
||||
return (
|
||||
Process.TypeChoices.SERVER,
|
||||
Process.TypeChoices.ADD,
|
||||
Process.TypeChoices.UPDATE,
|
||||
)
|
||||
|
||||
|
||||
def current_command(process_type: str, *, data_dir: str | Path, url: str | None = None):
|
||||
from archivebox.machine.models import Process
|
||||
|
||||
@ -42,31 +37,8 @@ def current_command(process_type: str, *, data_dir: str | Path, url: str | None
|
||||
return proc
|
||||
|
||||
|
||||
def live_processes(*, process_type: str, data_dir: str | Path, url: str | None = None):
|
||||
from archivebox.machine.models import Machine, Process
|
||||
|
||||
qs = Process.objects.filter(
|
||||
machine=Machine.current(),
|
||||
process_type=process_type,
|
||||
status=Process.StatusChoices.RUNNING,
|
||||
pwd=str(data_dir),
|
||||
)
|
||||
if url is not None:
|
||||
qs = qs.filter(url=url)
|
||||
return [proc for proc in qs.order_by("-created_at", "-modified_at").iterator(chunk_size=50) if proc.is_running]
|
||||
|
||||
|
||||
def newest_live_process(*, process_type: str, data_dir: str | Path, url: str | None = None):
|
||||
processes = live_processes(process_type=process_type, data_dir=data_dir, url=url)
|
||||
return processes[0] if processes else None
|
||||
|
||||
|
||||
def command_is_newest(command, *, process_type: str, data_dir: str | Path, url: str | None = None) -> bool:
|
||||
leader = newest_live_process(process_type=process_type, data_dir=data_dir, url=url)
|
||||
return bool(leader and leader.id == command.id)
|
||||
|
||||
|
||||
def runtime_stack_owner(*, data_dir: str | Path, exclude_id=None):
|
||||
"""Return the live local parent allowed to own the server runtime stack."""
|
||||
from archivebox.machine.models import Machine, Process
|
||||
|
||||
machine = Machine.current()
|
||||
@ -74,7 +46,7 @@ def runtime_stack_owner(*, data_dir: str | Path, exclude_id=None):
|
||||
machine=machine,
|
||||
status=Process.StatusChoices.RUNNING,
|
||||
pwd=str(data_dir),
|
||||
process_type__in=runtime_stack_owner_types(),
|
||||
process_type__in=(Process.TypeChoices.SERVER, Process.TypeChoices.ORCHESTRATOR),
|
||||
)
|
||||
if exclude_id is not None:
|
||||
base_qs = base_qs.exclude(id=exclude_id)
|
||||
@ -103,6 +75,7 @@ def command_owns_runtime_stack(command, *, data_dir: str | Path) -> bool:
|
||||
|
||||
|
||||
def foreground_runner_owner(*, data_dir: str | Path, exclude_id=None):
|
||||
"""Return the newest live local parent allowed to borrow runner/sonic."""
|
||||
from archivebox.machine.models import Machine, Process
|
||||
|
||||
machine = Machine.current()
|
||||
@ -110,7 +83,7 @@ def foreground_runner_owner(*, data_dir: str | Path, exclude_id=None):
|
||||
machine=machine,
|
||||
status=Process.StatusChoices.RUNNING,
|
||||
pwd=str(data_dir),
|
||||
process_type__in=foreground_runner_owner_types(),
|
||||
process_type__in=(Process.TypeChoices.SERVER, Process.TypeChoices.ADD, Process.TypeChoices.UPDATE),
|
||||
)
|
||||
if exclude_id is not None:
|
||||
qs = qs.exclude(id=exclude_id)
|
||||
@ -126,26 +99,6 @@ def command_owns_foreground_runner(command, *, data_dir: str | Path) -> bool:
|
||||
return bool(owner and owner.id == command.id)
|
||||
|
||||
|
||||
def runtime_stack_component_label(*, owner=None, data_dir: str | Path) -> str:
|
||||
try:
|
||||
from archivebox.workers.supervisord_util import active_supervisord_runtime_components
|
||||
|
||||
components = active_supervisord_runtime_components()
|
||||
except Exception:
|
||||
components = []
|
||||
|
||||
names = list(components)
|
||||
if not names and owner is not None:
|
||||
from archivebox.machine.models import Process
|
||||
|
||||
if owner.process_type == Process.TypeChoices.SERVER:
|
||||
names = ["orchestrator", "server"]
|
||||
elif owner.process_type == Process.TypeChoices.ORCHESTRATOR:
|
||||
names = ["orchestrator"]
|
||||
|
||||
return ", ".join(dict.fromkeys(names)) or "runtime stack"
|
||||
|
||||
|
||||
def ensure_daemon_stack(*, reason: str = ""):
|
||||
from archivebox.config.common import get_config
|
||||
from archivebox.workers.supervisord_util import (
|
||||
@ -181,31 +134,15 @@ def ensure_daemon_stack(*, reason: str = ""):
|
||||
return start_worker(supervisor, sonic_worker)
|
||||
|
||||
|
||||
def healthy_orchestrator(*, data_dir: str | Path):
|
||||
from archivebox.machine.models import Machine, Process
|
||||
from archivebox.workers.supervisord_util import get_existing_supervisord_process, get_worker
|
||||
def live_runner_processes(*, data_dir: str | Path):
|
||||
"""Return locally verifiable runners and warn about unsupported overlap.
|
||||
|
||||
supervisor = get_existing_supervisord_process()
|
||||
worker = get_worker(supervisor, "worker_runner") if supervisor else None
|
||||
if isinstance(worker, dict) and worker.get("statename") in ("STARTING", "RUNNING"):
|
||||
return worker
|
||||
|
||||
for proc in Process.objects.filter(
|
||||
machine=Machine.current(),
|
||||
process_type=Process.TypeChoices.ORCHESTRATOR,
|
||||
status=Process.StatusChoices.RUNNING,
|
||||
pwd=str(data_dir),
|
||||
).order_by("-created_at"):
|
||||
if proc.is_running:
|
||||
return proc
|
||||
return None
|
||||
|
||||
|
||||
def _runner_sort_key(process):
|
||||
return (process.started_at or process.created_at, process.created_at, str(process.id))
|
||||
|
||||
|
||||
def live_runner_processes(*, data_dir: str | Path, exclude_id=None):
|
||||
A Process row from another machine is observability only: its PID cannot be
|
||||
checked or signalled here, so it neither joins the local election nor gets
|
||||
mutated. A row for this same Machine from another PID namespace represents
|
||||
a previous sequential container under the supported model; warn, retire the
|
||||
unreachable row, and let the new container continue.
|
||||
"""
|
||||
from archivebox.machine.models import Machine, Process
|
||||
|
||||
machine = Machine.current()
|
||||
@ -218,9 +155,6 @@ def live_runner_processes(*, data_dir: str | Path, exclude_id=None):
|
||||
)
|
||||
foreign_machine_exists = qs.exclude(machine=machine).exists()
|
||||
qs = qs.filter(machine=machine)
|
||||
if exclude_id is not None:
|
||||
qs = qs.exclude(id=exclude_id)
|
||||
|
||||
live = []
|
||||
foreign_namespace_ids = []
|
||||
for process in qs.order_by("started_at", "created_at").iterator(chunk_size=20):
|
||||
@ -250,13 +184,15 @@ def live_runner_processes(*, data_dir: str | Path, exclude_id=None):
|
||||
|
||||
def enter_single_runner_gate(command, *, data_dir: str | Path, graceful_timeout: float = 5.0) -> bool:
|
||||
"""
|
||||
Admit exactly one active runner for this DATA_DIR using Process rows.
|
||||
Admit one active runner for this Machine and DATA_DIR using Process rows.
|
||||
|
||||
The current process is a real OS process while it waits, so we keep its
|
||||
Process row RUNNING but mark worker_type=runner_waiting. Only the process
|
||||
that wins takeover is promoted to worker_type=worker_runner, which is
|
||||
protected by a partial unique DB constraint. Older runners are terminated
|
||||
and fully waited out before promotion, so the runner work loop never overlaps.
|
||||
protected by a partial unique DB constraint scoped to (Machine, DATA_DIR).
|
||||
Older locally verifiable runners are terminated and fully waited out before
|
||||
promotion, so runner work never overlaps on one machine. Foreign machines
|
||||
are intentionally outside this gate and only produce a warning above.
|
||||
"""
|
||||
from archivebox.machine.models import Process
|
||||
|
||||
@ -278,7 +214,7 @@ def enter_single_runner_gate(command, *, data_dir: str | Path, graceful_timeout:
|
||||
)
|
||||
runners = live_runner_processes(data_dir=data_dir)
|
||||
|
||||
newest = max(runners, key=_runner_sort_key)
|
||||
newest = max(runners, key=lambda process: (process.started_at or process.created_at, process.created_at, str(process.id)))
|
||||
if newest.id != command.id:
|
||||
rprint(
|
||||
f"[yellow][*] Newer ArchiveBox runner pid={newest.pid} is taking over; exiting this runner.[/yellow]",
|
||||
@ -310,24 +246,9 @@ def enter_single_runner_gate(command, *, data_dir: str | Path, graceful_timeout:
|
||||
time.sleep(0.1)
|
||||
|
||||
|
||||
def standby_until_leader_needed(command, *, process_type: str, data_dir: str | Path, url: str | None = None, interval: float = 2.0) -> None:
|
||||
from archivebox.workers.supervisord_util import reap_foreground_supervisord_process
|
||||
|
||||
announced = False
|
||||
while not command_is_newest(command, process_type=process_type, data_dir=data_dir, url=url):
|
||||
reap_foreground_supervisord_process()
|
||||
if not announced:
|
||||
leader = newest_live_process(process_type=process_type, data_dir=data_dir, url=url)
|
||||
leader_pid = leader.pid if leader else "unknown"
|
||||
rprint(f"[yellow][*] Standing by; newer ArchiveBox process pid={leader_pid} is running the orchestrator and server.[/yellow]")
|
||||
announced = True
|
||||
time.sleep(interval)
|
||||
command.modified_at = timezone.now()
|
||||
command.save(update_fields=["modified_at"])
|
||||
|
||||
|
||||
def standby_until_runtime_stack_needed(command, *, data_dir: str | Path, interval: float = 2.0) -> dict[str, object]:
|
||||
from archivebox.workers.supervisord_util import reap_foreground_supervisord_process
|
||||
from archivebox.machine.models import Process
|
||||
from archivebox.workers.supervisord_util import active_supervisord_runtime_components, reap_foreground_supervisord_process
|
||||
|
||||
announced = False
|
||||
previous_owner_pid = None
|
||||
@ -336,7 +257,16 @@ def standby_until_runtime_stack_needed(command, *, data_dir: str | Path, interva
|
||||
if not announced:
|
||||
owner = runtime_stack_owner(data_dir=data_dir)
|
||||
owner_pid = owner.pid if owner else "unknown"
|
||||
components = runtime_stack_component_label(owner=owner, data_dir=data_dir)
|
||||
try:
|
||||
component_names = list(active_supervisord_runtime_components())
|
||||
except Exception:
|
||||
component_names = []
|
||||
if not component_names and owner is not None:
|
||||
if owner.process_type == Process.TypeChoices.SERVER:
|
||||
component_names = ["orchestrator", "server"]
|
||||
elif owner.process_type == Process.TypeChoices.ORCHESTRATOR:
|
||||
component_names = ["orchestrator"]
|
||||
components = ", ".join(dict.fromkeys(component_names)) or "runtime stack"
|
||||
previous_owner_pid = owner_pid
|
||||
rprint(
|
||||
f"[yellow][*] A newer archivebox process took over the {components} "
|
||||
|
||||
@ -1156,6 +1156,11 @@ class Process(ModelWithDeleteAfter, models.Model):
|
||||
models.Index(fields=["machine", "status", "process_type"], name="mach_proc_running_idx"),
|
||||
]
|
||||
constraints = [
|
||||
# This is deliberately machine-scoped. It prevents two locally
|
||||
# verifiable runners from working the same collection, while not
|
||||
# claiming to coordinate independent hosts that share a database.
|
||||
# Cross-machine work ownership belongs to short Crawl/Snapshot CAS
|
||||
# claims so PostgreSQL deployments can support that model later.
|
||||
models.UniqueConstraint(
|
||||
fields=["machine", "pwd"],
|
||||
condition=Q(status="running", process_type="orchestrator", worker_type="worker_runner"),
|
||||
@ -1362,7 +1367,7 @@ class Process(ModelWithDeleteAfter, models.Model):
|
||||
self.save(update_fields=updates)
|
||||
|
||||
def heartbeat(self) -> None:
|
||||
"""Touch modified_at so standby/leader selection can see this parent is alive."""
|
||||
"""Keep a long-lived watcher visible in recent-process monitoring."""
|
||||
self.save(update_fields=["modified_at"])
|
||||
|
||||
def mark_exited(self, *, exit_code: int = 0) -> None:
|
||||
|
||||
@ -483,6 +483,11 @@ class CrawlRunner:
|
||||
await self.enqueue_pending_snapshots_from_projection()
|
||||
|
||||
async def heartbeat_active_leases(self) -> None:
|
||||
# These are resumable work-item leases, not orchestrator-election
|
||||
# heartbeats. Each update is a short autocommit statement; network and
|
||||
# filesystem work continues outside a database transaction. A future
|
||||
# PostgreSQL multi-machine runner uses these Crawl/Snapshot claims as
|
||||
# its coordination boundary while SQLite keeps one local orchestrator.
|
||||
if self._run_task is None:
|
||||
return
|
||||
now_monotonic = time.monotonic()
|
||||
|
||||
@ -192,13 +192,12 @@ Never enable on-demand TLS or request individual certificates for `snap-*` hostn
|
||||
|
||||
### Setup
|
||||
|
||||
Fetch and run the ArchiveBox Docker image to create your initial archive.
|
||||
Fetch and run the ArchiveBox Docker image. Starting the server creates the initial archive automatically.
|
||||
|
||||
```bash
|
||||
docker pull archivebox/archivebox:dev
|
||||
|
||||
mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
docker run --rm -it -v "$PWD:/data" archivebox/archivebox:dev init
|
||||
docker run -d --name archivebox -v "$PWD:/data" -p 8000:8000 archivebox/archivebox:dev
|
||||
```
|
||||
|
||||
|
||||
@ -17,7 +17,7 @@
|
||||
- right now, the paths of the extractor output are scattered all over the codebase, e.g. `output.pdf` (should be moved to constants at the top of the plugin config file)
|
||||
- make out_dir, link_dir, extractor_dir, naming consistent across codebase
|
||||
- remove `timestamps` as primary keys in favor of hashes, UUIDs, or some other slug https://github.com/ArchiveBox/ArchiveBox/issues/74
|
||||
- create a migration system for folder layout independent of the index (`mv` is atomic at the FS level, so we just need a `transaction.atomic(): move(oldpath, newpath); snap.data_dir = newpath; snap.save()`)
|
||||
- create a migration system for folder layout independent of the index
|
||||
- make `Tag` a real model `ManyToMany` with Snapshots
|
||||
- allow multiple Snapshots of the same site over time + CLI / UI to manage those, + migration from old style `#2020-01-01` hack to proper versioned snapshots
|
||||
- upgrade from Django 3 to Django 5 https://github.com/ArchiveBox/ArchiveBox/issues/988
|
||||
|
||||
@ -18,12 +18,11 @@ archivebox install
|
||||
archivebox update --migrate-only
|
||||
archivebox status
|
||||
|
||||
# Docker Compose install
|
||||
# Docker Compose upgrade
|
||||
cd ~/archivebox
|
||||
docker compose down
|
||||
docker compose pull
|
||||
docker compose run --rm archivebox init
|
||||
docker compose run --rm archivebox install
|
||||
docker compose run --rm archivebox update --migrate-only
|
||||
docker compose up -d
|
||||
```
|
||||
@ -35,7 +34,7 @@ docker compose up -d
|
||||
2. **Read the release notes carefully** for any instructions or extra steps around upgrading for each release you're skipping or installing
|
||||
3. **Stop any running ArchiveBox server, scheduler, and worker processes**, then back up the entire collection data directory before upgrading. `archivebox config --get ...` and a database-only backup do not include archived outputs.
|
||||
`cd ~/archivebox && tar -czf "archivebox-data-$(date +%s).tar.gz" data/`
|
||||
4. Follow the steps below for your installation method, then run `archivebox init`, `archivebox install`, and `archivebox update --migrate-only` inside the collection
|
||||
4. Follow the steps below for your installation method. Bare-metal installs run `archivebox init`, `archivebox install`, and `archivebox update --migrate-only` inside the collection; Docker images already include runtime dependencies.
|
||||
5. Confirm the upgrade succeeded and check for any orphan/corrupted snapshots with `archivebox status`
|
||||
|
||||
💬 [Open an issue](https://github.com/ArchiveBox/ArchiveBox/issues/new/choose) in our bug tracker if you experience any problems with upgrading/merging/modifying collections.
|
||||
@ -49,7 +48,7 @@ docker compose up -d
|
||||
|
||||
**ℹ️ How it works internally:**
|
||||
|
||||
The same command is used for initializing a new archive and upgrading an existing database. `archivebox init` is idempotent and can safely be run multiple times; it applies database migrations and prepares collection-level state. `archivebox install` resolves runtime dependencies for the new version. `archivebox update --migrate-only` performs filesystem migrations and reconciles Snapshot metadata with the current layout without scheduling normal archive maintenance jobs. `archivebox status` checks collection health afterward.
|
||||
The same command is used for initializing a new archive and upgrading an existing database. `archivebox init` is idempotent and can safely be run multiple times; it applies database migrations and prepares collection-level state. For bare-metal installs, `archivebox install` resolves runtime dependencies for the new version; Docker images include those dependencies at build time. `archivebox update --migrate-only` performs filesystem migrations and reconciles Snapshot metadata with the current layout without scheduling normal archive maintenance jobs. `archivebox status` checks collection health afterward.
|
||||
|
||||
There are three main areas on disk that ArchiveBox modifies during upgrades:
|
||||
- `index.sqlite3` contains the SQLite3 DB index that gets upgraded automatically by Django based on the changes in [`archivebox/core/models.py`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/models.py).
|
||||
@ -76,7 +75,6 @@ cd ~/archivebox # or wherever your folder containing docker-compose.yml i
|
||||
docker compose down # stop the currently running ArchiveBox containers
|
||||
docker compose pull # pull the latest image version from Docker Hub
|
||||
docker compose run --rm archivebox init
|
||||
docker compose run --rm archivebox install
|
||||
docker compose run --rm archivebox update --migrate-only
|
||||
docker compose up -d
|
||||
```
|
||||
@ -97,7 +95,6 @@ docker stop CONTAINER_ID
|
||||
cd ~/archivebox/data # or wherever your existing collection is stored
|
||||
docker pull archivebox/archivebox:dev
|
||||
docker run --rm -v $PWD:/data -it archivebox/archivebox:dev init
|
||||
docker run --rm -v $PWD:/data -it archivebox/archivebox:dev install
|
||||
docker run --rm -v $PWD:/data -it archivebox/archivebox:dev update --migrate-only
|
||||
|
||||
# restart the archivebox server container if needed
|
||||
|
||||
@ -15,30 +15,10 @@
|
||||
:class: autosummary longtable
|
||||
:align: left
|
||||
|
||||
* - {py:obj}`runtime_stack_owner_types <archivebox.core.takeover_util.runtime_stack_owner_types>`
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.runtime_stack_owner_types
|
||||
:summary:
|
||||
```
|
||||
* - {py:obj}`foreground_runner_owner_types <archivebox.core.takeover_util.foreground_runner_owner_types>`
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.foreground_runner_owner_types
|
||||
:summary:
|
||||
```
|
||||
* - {py:obj}`current_command <archivebox.core.takeover_util.current_command>`
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.current_command
|
||||
:summary:
|
||||
```
|
||||
* - {py:obj}`live_processes <archivebox.core.takeover_util.live_processes>`
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.live_processes
|
||||
:summary:
|
||||
```
|
||||
* - {py:obj}`newest_live_process <archivebox.core.takeover_util.newest_live_process>`
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.newest_live_process
|
||||
:summary:
|
||||
```
|
||||
* - {py:obj}`command_is_newest <archivebox.core.takeover_util.command_is_newest>`
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.command_is_newest
|
||||
:summary:
|
||||
```
|
||||
* - {py:obj}`runtime_stack_owner <archivebox.core.takeover_util.runtime_stack_owner>`
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.runtime_stack_owner
|
||||
:summary:
|
||||
@ -55,22 +35,10 @@
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.command_owns_foreground_runner
|
||||
:summary:
|
||||
```
|
||||
* - {py:obj}`runtime_stack_component_label <archivebox.core.takeover_util.runtime_stack_component_label>`
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.runtime_stack_component_label
|
||||
:summary:
|
||||
```
|
||||
* - {py:obj}`ensure_daemon_stack <archivebox.core.takeover_util.ensure_daemon_stack>`
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.ensure_daemon_stack
|
||||
:summary:
|
||||
```
|
||||
* - {py:obj}`healthy_orchestrator <archivebox.core.takeover_util.healthy_orchestrator>`
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.healthy_orchestrator
|
||||
:summary:
|
||||
```
|
||||
* - {py:obj}`_runner_sort_key <archivebox.core.takeover_util._runner_sort_key>`
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util._runner_sort_key
|
||||
:summary:
|
||||
```
|
||||
* - {py:obj}`live_runner_processes <archivebox.core.takeover_util.live_runner_processes>`
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.live_runner_processes
|
||||
:summary:
|
||||
@ -79,10 +47,6 @@
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.enter_single_runner_gate
|
||||
:summary:
|
||||
```
|
||||
* - {py:obj}`standby_until_leader_needed <archivebox.core.takeover_util.standby_until_leader_needed>`
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.standby_until_leader_needed
|
||||
:summary:
|
||||
```
|
||||
* - {py:obj}`standby_until_runtime_stack_needed <archivebox.core.takeover_util.standby_until_runtime_stack_needed>`
|
||||
- ```{autodoc2-docstring} archivebox.core.takeover_util.standby_until_runtime_stack_needed
|
||||
:summary:
|
||||
@ -145,20 +109,6 @@
|
||||
|
||||
````
|
||||
|
||||
````{py:function} runtime_stack_owner_types()
|
||||
:canonical: archivebox.core.takeover_util.runtime_stack_owner_types
|
||||
|
||||
```{autodoc2-docstring} archivebox.core.takeover_util.runtime_stack_owner_types
|
||||
```
|
||||
````
|
||||
|
||||
````{py:function} foreground_runner_owner_types()
|
||||
:canonical: archivebox.core.takeover_util.foreground_runner_owner_types
|
||||
|
||||
```{autodoc2-docstring} archivebox.core.takeover_util.foreground_runner_owner_types
|
||||
```
|
||||
````
|
||||
|
||||
````{py:function} current_command(process_type: str, *, data_dir: str | pathlib.Path, url: str | None = None)
|
||||
:canonical: archivebox.core.takeover_util.current_command
|
||||
|
||||
@ -166,27 +116,6 @@
|
||||
```
|
||||
````
|
||||
|
||||
````{py:function} live_processes(*, process_type: str, data_dir: str | pathlib.Path, url: str | None = None)
|
||||
:canonical: archivebox.core.takeover_util.live_processes
|
||||
|
||||
```{autodoc2-docstring} archivebox.core.takeover_util.live_processes
|
||||
```
|
||||
````
|
||||
|
||||
````{py:function} newest_live_process(*, process_type: str, data_dir: str | pathlib.Path, url: str | None = None)
|
||||
:canonical: archivebox.core.takeover_util.newest_live_process
|
||||
|
||||
```{autodoc2-docstring} archivebox.core.takeover_util.newest_live_process
|
||||
```
|
||||
````
|
||||
|
||||
````{py:function} command_is_newest(command, *, process_type: str, data_dir: str | pathlib.Path, url: str | None = None) -> bool
|
||||
:canonical: archivebox.core.takeover_util.command_is_newest
|
||||
|
||||
```{autodoc2-docstring} archivebox.core.takeover_util.command_is_newest
|
||||
```
|
||||
````
|
||||
|
||||
````{py:function} runtime_stack_owner(*, data_dir: str | pathlib.Path, exclude_id=None)
|
||||
:canonical: archivebox.core.takeover_util.runtime_stack_owner
|
||||
|
||||
@ -215,13 +144,6 @@
|
||||
```
|
||||
````
|
||||
|
||||
````{py:function} runtime_stack_component_label(*, owner=None, data_dir: str | pathlib.Path) -> str
|
||||
:canonical: archivebox.core.takeover_util.runtime_stack_component_label
|
||||
|
||||
```{autodoc2-docstring} archivebox.core.takeover_util.runtime_stack_component_label
|
||||
```
|
||||
````
|
||||
|
||||
````{py:function} ensure_daemon_stack(*, reason: str = '')
|
||||
:canonical: archivebox.core.takeover_util.ensure_daemon_stack
|
||||
|
||||
@ -229,21 +151,7 @@
|
||||
```
|
||||
````
|
||||
|
||||
````{py:function} healthy_orchestrator(*, data_dir: str | pathlib.Path)
|
||||
:canonical: archivebox.core.takeover_util.healthy_orchestrator
|
||||
|
||||
```{autodoc2-docstring} archivebox.core.takeover_util.healthy_orchestrator
|
||||
```
|
||||
````
|
||||
|
||||
````{py:function} _runner_sort_key(process)
|
||||
:canonical: archivebox.core.takeover_util._runner_sort_key
|
||||
|
||||
```{autodoc2-docstring} archivebox.core.takeover_util._runner_sort_key
|
||||
```
|
||||
````
|
||||
|
||||
````{py:function} live_runner_processes(*, data_dir: str | pathlib.Path, exclude_id=None)
|
||||
````{py:function} live_runner_processes(*, data_dir: str | pathlib.Path)
|
||||
:canonical: archivebox.core.takeover_util.live_runner_processes
|
||||
|
||||
```{autodoc2-docstring} archivebox.core.takeover_util.live_runner_processes
|
||||
@ -257,13 +165,6 @@
|
||||
```
|
||||
````
|
||||
|
||||
````{py:function} standby_until_leader_needed(command, *, process_type: str, data_dir: str | pathlib.Path, url: str | None = None, interval: float = 2.0) -> None
|
||||
:canonical: archivebox.core.takeover_util.standby_until_leader_needed
|
||||
|
||||
```{autodoc2-docstring} archivebox.core.takeover_util.standby_until_leader_needed
|
||||
```
|
||||
````
|
||||
|
||||
````{py:function} standby_until_runtime_stack_needed(command, *, data_dir: str | pathlib.Path, interval: float = 2.0) -> dict[str, object]
|
||||
:canonical: archivebox.core.takeover_util.standby_until_runtime_stack_needed
|
||||
|
||||
|
||||
@ -99,8 +99,9 @@
|
||||
<pre><span class="ab-dim"># Docker Compose is the recommended setup</span>
|
||||
<span class="ab-prompt">$</span> <span class="ab-cmd">mkdir -p ~/archivebox/data && cd ~/archivebox</span>
|
||||
<span class="ab-prompt">$</span> <span class="ab-cmd">curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml</span>
|
||||
<span class="ab-prompt">$</span> <span class="ab-cmd">docker compose run archivebox init</span>
|
||||
<span class="ab-out">-> created ./data/index.sqlite3</span>
|
||||
<span class="ab-prompt">$</span> <span class="ab-cmd">docker compose up -d --wait</span>
|
||||
<span class="ab-out">-> initialized ./data/index.sqlite3</span>
|
||||
<span class="ab-ok">ok listening on http://127.0.0.1:8000</span>
|
||||
<span class="ab-prompt">$</span> <span class="ab-cursor" aria-hidden="true"> </span></pre>
|
||||
</div>
|
||||
</div>
|
||||
@ -207,16 +208,15 @@
|
||||
<div class="step">
|
||||
<span>3</span>
|
||||
<div>
|
||||
<h3>Initialize and start</h3>
|
||||
<div class="ab-terminal ab-terminal--inline"><div class="ab-terminal__body"><pre><span class="ab-prompt">$</span> <span class="ab-cmd">docker compose run archivebox init</span>
|
||||
<span class="ab-prompt">$</span> <span class="ab-cmd">docker compose up</span></pre></div></div>
|
||||
<h3>Start ArchiveBox (initializes automatically)</h3>
|
||||
<div class="ab-terminal ab-terminal--inline"><div class="ab-terminal__body"><pre><span class="ab-prompt">$</span> <span class="ab-cmd">docker compose up -d --wait</span></pre></div></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="step">
|
||||
<span>4</span>
|
||||
<div>
|
||||
<h3>Add your first URL</h3>
|
||||
<div class="ab-terminal ab-terminal--inline"><div class="ab-terminal__body"><pre><span class="ab-prompt">$</span> <span class="ab-cmd">docker compose run archivebox add 'https://example.com'</span></pre></div></div>
|
||||
<div class="ab-terminal ab-terminal--inline"><div class="ab-terminal__body"><pre><span class="ab-prompt">$</span> <span class="ab-cmd">docker compose exec archivebox archivebox add 'https://example.com'</span></pre></div></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Loading…
Reference in New Issue
Block a user