fix(snapshots): honor SNAPSHOTS_PER_PAGE without silent clamping; bump default to 50
admin_snapshots.py:571 had min(max(50, SNAPSHOTS_PER_PAGE), 500), and
admin_archiveresults.py:501 had min(max(5, SNAPSHOTS_PER_PAGE), 5000).
Both clamps silently overrode the configured value — a documented
default of 40 was inaccessible in the Snapshot admin, and the
ArchiveResult admin also reused the same setting without being mentioned
in the docs.
- Drop both clamps; admin changelists now use SNAPSHOTS_PER_PAGE as-is.
- Bump the default in common.py from 40 to 50 (matches what users were
actually seeing in the admin under the old floor).
- Add ge=1 validation so non-positive values are rejected at config
parse time instead of producing broken pagination.
- Update Configuration.md: new default 50, clarify the option drives
both Snapshot and ArchiveResult admin changelists plus the public
index, and that it must be >= 1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Several inaccuracies + over-documentation cleaned up in one pass:
- ONLY_NEW: completely rewrite. The old prose ("ArchiveBox will never
re-download sites that have already succeeded previously") was carried
over from 0.7.x and is wrong in 0.9.x — setting ONLY_NEW=False (or
--no-only-new) explicitly creates a new Snapshot and re-runs every
extractor. Now describes the actual behavior: skip URL entirely vs.
create a new Snapshot for it.
- CRAWL_MAX_CONCURRENT_SNAPSHOTS: fix the "each concurrent Snapshot
launches its own Chrome instance" claim. Chrome is crawl-scoped by
default (CHROME_ISOLATION="crawl") — concurrent Snapshots share the
crawl's Chrome via tabs, not separate browser processes.
- BASE_URL: drop the "admin.admin.admin.<host> compounding bug"
reference. Config docs shouldn't explain legacy bugs.
- Remove derived/runtime-only options that are NOT user-settable:
ACTIVE_PERSONA (set by persona resolver), CRAWL_DIR/SNAP_DIR (injected
by orchestrator per-call), DATA_DIR (derived from cwd), ARCHIVE_DIR
(derived from DATA_DIR/archive), USERS_DIR (derived from ARCHIVE_DIR),
PERSONAS_DIR (derived from DATA_DIR), LIB_BIN_DIR (tracks LIB_DIR),
DATABASE_NAME (derived from DATA_DIR/index.sqlite3). Backward-compat
<a id="..."></a> anchors preserved for all of them above the nearest
surviving heading so external links still resolve.
- LIB_DIR: fix default path. The doc claimed "<DATA_DIR>/lib/<arch>-<os>"
but constants.py:117 uses platformdirs.user_config_path("abx") / "lib"
— the XDG user-config dir, not inside the data folder. Updated to the
actual default.
- ENABLED_PLUGINS section dropped (option removed in a separate commit);
anchor redirected to PLUGINS.
- Drop the "Pydantic config" implementation-detail mention in PUID/PGID.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This rewrite (now reapplied on top of the wiki subtree) covers the full
session's work on Configuration.md:
- Add crawl/snapshot limits (CRAWL_MAX_URLS/SIZE/TIMEOUT,
CRAWL_MAX_CONCURRENT_SNAPSHOTS, SNAPSHOT_MAX_SIZE), DELETE_AFTER,
PERMISSIONS, PLUGINS/ENABLED_PLUGINS/ACTIVE_PERSONA.
- Add new Database Settings section (SQLITE_* tuning + DATABASE_NAME).
- Add SERVER_SECURITY_MODE deep-dive (4 modes, host-layout table).
- Add Storage path overrides (DATA_DIR, ARCHIVE_DIR, USERS_DIR,
PERSONAS_DIR, CRAWL_DIR, SNAP_DIR, ALLOW_NO_UNIX_SOCKETS).
- Remove ALLOWED_HOSTS + CSRF_TRUSTED_ORIGINS as user-settable; both
auto-derived from BASE_URL + SERVER_SECURITY_MODE. Backward-compat
anchors preserved on BASE_URL with the 0.7.3 -> 0.9 legacy upgrade note.
- Remove the entire Plugin Settings tree (~200 options, 41 subsections);
replace with prominent redirect to https://archivebox.github.io/abx-plugins/
and a "shared core options that plugins fall back to" table.
- Add 231 backward-compat <a id="..."></a> anchors so old URLs to plugin
sections / removed options / multi-option headers all still resolve
(e.g. #wget_args -> Plugin Configuration section, #public_snapshots ->
PERMISSIONS, #ssl_enabled -> Plugin Configuration, #admin_username ->
ADMIN_USERNAME/PASSWORD heading, #dir_output_permissions ->
OUTPUT_PERMISSIONS, #url_blacklist -> URL_DENYLIST).
- Fix wrong default: PUBLIC_ADD_VIEW is False, not True.
- Drop the 7 TRAFILATURA_OUTPUT_* per-format flags (replaced by single
TRAFILATURA_OUTPUT_FORMATS in plugin); SSL_ENABLED/SSL_TIMEOUT (wrong
plugin namespace) — anchors redirected to Plugin Configuration.
- Reframe COOKIES_FILE as low-level escape hatch; personas are the
preferred auth path.
- Link every named plugin to its specific anchor on the abx-plugins page
(e.g. WGET_TIMEOUT -> #wget, SONIC_HOST -> #search_backend_sonic).
- Strip implementation-detail mentions (Pydantic, etc.).
- Slim Shell Options to only user-settable (DEBUG, USE_COLOR,
SHOW_PROGRESS); drop IS_TTY/IN_DOCKER/IN_QEMU.
- Restructure: General -> Server (+LDAP) -> Storage -> Database (new) ->
Search -> Shell -> Plugin Configuration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace all config tables with flat #### headings for hotlinking
Every config option now has its own #### `KEY` heading so GitHub wiki
anchors work for direct linking (e.g. Configuration#SEO_ENABLED).
Converted 9 tables: Chrome sub-extractors, Chrome extensions, URL
parsers, Sonic search, SQLite FTS, Infinite scroll, 2captcha, Modal
closer, and LDAP settings.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document all 189 plugin config options with hotlinkable headings
Every config option now has its own #### `KEY` heading so it can be
directly linked as Configuration#KEY from the wiki or other docs.
Covers all extractor plugins, Chrome sub-extractors, Chrome extensions,
URL parsers, AI plugins, and search backends with their full set of
_ENABLED, _TIMEOUT, _BINARY, _ARGS, _ARGS_EXTRA, _USER_AGENT,
_CHECK_SSL_VALIDITY, _COOKIES_FILE, and unique options.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
