Reverts 6a635d3c's piecewise depth-shift in test_recursive_crawl.py
plus matching test_cli_add tagged-URL, test_api_v1_cli_workflow,
test_config_MAX_limits assertions. Direct URLs land at snapshot.depth=0
and max_depth==depth as the single user-facing contract; only stdin/
import text adds the +1 hop for the synthetic archivebox://internal
root.
archivebox add and other entry points now seed Crawl.urls as
CrawlSeed JSONL at depth=0 (the input layer) with max_depth=depth
for direct URLs and depth+1 only for stdin/import text where the
synthetic archivebox://internal root lives at depth=0. The runner
also accepts one plain URL per line for ORM/crawl-create/schedule
callers so every Crawl row goes through the same expansion path
without scattering CrawlSeed knowledge across the codebase.
Tests updated to match restored convention.
Direct URL inputs from CLI/UI/API now seed Crawl.urls as explicit
{type:CrawlSeed,url,depth} JSONL rows; raw stdin/UI/API import text
stays verbatim. The runner's create_initial_snapshots() is now the
single place that either expands seed rows or creates the synthetic
archivebox://internal root + staticfile/stdin.txt, so add paths no
longer perform DB/FS side effects and the parser hooks run through
the same Snapshot lifecycle as every other extractor.
## Summary
Changes the `Snapshot.url` field from a fixed-length
`CharField(max_length=65535)` to a variable-length `TextField` while
enforcing a practical 8000-character limit. This allows short URLs to
use minimal storage space while still supporting very long URLs (e.g.,
data: URLs, deeply nested query strings) that are common in real-world
scenarios.
The TextField with a normal database index maintains fast exact, prefix,
and substring lookups while reducing storage overhead for typical URLs.
## Related issues
Improves URL handling to match practical web server/proxy limits (8000
chars is the de facto standard enforced by most infrastructure).
## Changes these areas
- [x] Internal architecture
- [x] Snapshot data layout on disk
## Details
### Changes Made
1. **`archivebox/misc/util.py`**: Reduced `MAX_URL_LENGTH` from 65535 to
8000 with detailed comments explaining the rationale
2. **`archivebox/core/models.py`**: Changed `Snapshot.url` field from
`CharField(max_length=MAX_URL_LENGTH)` to `TextField(db_index=True)`
with validation in the model's `save()` method
3. **`archivebox/core/migrations/0049_alter_snapshot_url.py`**: Django
migration to alter the field type
4. **`archivebox/tests/test_snapshot_url_length.py`**: Comprehensive
test suite covering:
- URL length validation at the limit and over the limit
- Full round-trip persistence through the database
- Query operations (exact match, prefix search, substring search)
- Uniqueness constraints per crawl
- Cross-crawl URL reuse
### Why This Change
- **Storage efficiency**: Variable-length TextField doesn't reserve 8000
bytes for every URL
- **Practical limit**: 8000 chars matches the upper bound enforced by
most web servers and proxies
- **Query performance**: Normal database index on TextField maintains
fast lookups for all query types
- **Real-world support**: Handles legitimate long URLs (data: URIs,
complex query strings) while rejecting pathological cases
## Test Plan
Added comprehensive test suite (`test_snapshot_url_length.py`) that
validates:
- URLs at exactly the 8000-char limit are accepted and persisted
correctly
- URLs exceeding the limit are rejected with validation errors
- Database queries (exact, prefix, substring) work correctly on long
URLs
- Uniqueness constraints per crawl are still enforced
- Same URL can exist in different crawls
All tests pass with the new implementation.
https://claude.ai/code/session_01BLnGTL5GSoouD4ihaYp55n
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Store `Snapshot.url` as an indexed `TextField` to support very long URLs
efficiently while keeping exact, prefix, and substring searches fast.
The supported URL length remains 65535 characters.
- **Refactors**
- Changed `Snapshot.url` from `CharField` to `TextField(db_index=True)`.
- Kept `MAX_URL_LENGTH` at 65535 and validate in model/utilities; tests
updated to the 65535 boundary, lookups, and per-crawl uniqueness.
- **Migration**
- Apply `0049_alter_snapshot_url`.
- No data rewrite; existing rows remain. New/updated URLs must be ≤65535
chars.
<sup>Written for commit d6d479f50b.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1817?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
