fix(config): derive BACKEND_CONSUMED from .env.example; scan src/ for undocumented env reads (0.87.25)

0.87.12 added a regression test for backend env wiring, but its BACKEND_CONSUMED list was hand-maintained and drifted: ~22 vars read by backend/src/ never made it into compose's backend.environment. An operator setting ANALYSIS_TEMPERATURE=0.3 in .env got a silent no-op — exactly the bug class the test was supposed to catch. Make .env.example the single source of truth: * docker_compose_wires_every_documented_backend_env_var now reads .env.example at test time, filters NOT_BACKEND_CONSUMED_IN_ENV_EXAMPLE (compose-side composed vars, frontend-only vars, sidecar vars), and requires every remaining key to be wired into compose with a matching ${KEY...} interpolation OR allowlisted via COMPOSE_RHS_EXEMPT. * every_backend_src_env_read_is_documented_or_allowlisted scans backend/src/ for env::var / env_bool / env_i64 / ... reads and requires each key to be in .env.example OR in INTERNAL_NOT_CONFIGURABLE with a per-entry rationale. Catches the silent-no-op bug from the other side. Wire the previously-missing env vars into docker-compose.yml's backend.environment (CRAWLER_DAEMON, CRAWLER_DAILY_AT, CRAWLER_TZ, CRAWLER_IDLE_TIMEOUT_S, CRAWLER_CHAPTER_WORKERS, CRAWLER_JOB_RETENTION_DAYS, CRAWL_METRICS_RETENTION_DAYS, CRAWLER_RATE_MS, CRAWLER_CDN_RATE_MS, CRAWLER_USER_AGENT, CRAWLER_PHPSESSID, CRAWLER_COOKIE_DOMAIN, CRAWLER_TOR_CONTROL_COOKIE_PATH, all ANALYSIS_* tuning knobs). Document each new var in .env.example with the same default value shown in compose's `${KEY:-default}` form. Allowlist internal/system vars (HOME = chromium cache dir fallback, CRAWLER_BROWSER_*, CRAWLER_SKIP_*, the multi-paragraph ANALYSIS_*_PROMPT seeds) with per-entry rationale. Mutation-tested both sides: adding `env::var("MANGALORD_FAKE_NEW_KNOB")` to main.rs fails the scanner; removing the new compose wiring fails the documented-var test with all 23 missing keys named. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-24 19:28:00 +02:00
parent 579e6ade0f
commit ae708a72d0
6 changed files with 381 additions and 74 deletions
--- a/.env.example
+++ b/.env.example
@@ -122,6 +122,39 @@ CRAWLER_LIMIT=0
 # A job that exceeds the budget is acked failed (with exponential
 # backoff) instead of wedging a worker. Default 600s.
 CRAWLER_JOB_TIMEOUT_SECS=600
+# Idle seconds the daemon waits for new jobs before parking (between
+# scheduled passes). Lower for snappier dev loops; higher for fewer
+# wakeups in steady state. Default 600.
+CRAWLER_IDLE_TIMEOUT_S=600
+# Concurrent chapter-content workers. Each pulls one chapter at a time
+# from the queue. Higher = faster on multi-core hosts BUT also more
+# pressure on the source (raise CRAWLER_RATE_MS in tandem). Default 1.
+CRAWLER_CHAPTER_WORKERS=1
+# Days of completed `crawler_jobs` rows kept before vacuum (per-row
+# audit trail; failed rows survive to make retries traceable). Default 7.
+CRAWLER_JOB_RETENTION_DAYS=7
+# Days of `crawl_metrics` rows kept (per-run aggregate dashboard
+# data). Longer history → bigger dashboard charts; shorter → less DB.
+# Default 90.
+CRAWL_METRICS_RETENTION_DAYS=90
+# Politeness rate limit between requests to the source (milliseconds).
+# Pause inserted after each fetch. Raising slows the crawl but is
+# friendlier to small sources. Default 1000ms.
+CRAWLER_RATE_MS=1000
+# Politeness rate for the CDN host (CRAWLER_CDN_HOST). Defaults to
+# CRAWLER_RATE_MS when unset — set lower if the CDN tolerates faster
+# pulls, or higher if it rate-limits aggressively. Default 1000ms.
+CRAWLER_CDN_RATE_MS=1000
+# User-Agent string for crawler HTTP requests. Leave unset for a
+# generic default. Some sources fingerprint UA — set a realistic
+# browser UA if the source 403s the default.
+CRAWLER_USER_AGENT=
+# Source session cookie value (PHPSESSID). Required when the source
+# gates listings behind a logged-in session. Treat as a secret.
+CRAWLER_PHPSESSID=
+# Cookie domain the crawler scopes its session to. Usually the source
+# host (e.g. `mangadex.org`). Leave unset to let reqwest infer it.
+CRAWLER_COOKIE_DOMAIN=
 # Consecutive metadata-pass `fetch_manga` failures that abort the pass
 # (circuit breaker for a source outage). The pass does NOT mark a clean
 # exit, so the next tick does a recovery sweep. Default 10.
@@ -162,6 +195,11 @@ CRAWLER_PROXY=socks5h://tor:9050
 CRAWLER_TOR_CONTROL_URL=tcp://tor:9051
 # Max NEWNYM-and-retry cycles per recircuit-eligible failure. Default 3.
 CRAWLER_TOR_RECIRCUIT_MAX_ATTEMPTS=3
+# Path to a tor cookie-auth file. Alternative to TOR_CONTROL_PASSWORD —
+# the TorController prefers cookie when both are present. Useful when
+# running against an externally managed tor daemon that uses
+# `CookieAuthentication 1` instead of HashedControlPassword. Default unset.
+CRAWLER_TOR_CONTROL_COOKIE_PATH=

 # ----- TOR control-port password -----
 # Shared between the bundled dockurr/tor service (which hashes it into
@@ -213,6 +251,14 @@ BACKEND_PROXY_TIMEOUT_MS=300000
 #                       seed the download allowlist.
 CRAWLER_START_URL=
 CRAWLER_CDN_HOST=
+# CRAWLER_DAEMON / CRAWLER_DAILY_AT / CRAWLER_TZ — the daemon sweep
+# schedule. CRAWLER_DAEMON=false keeps the in-process daemon off; clicks
+# in the dashboard still work. DAILY_AT is `HH:MM` in the chosen TZ;
+# TZ is an IANA name (e.g. `Europe/Berlin`). Default: daemon on, runs
+# at 00:00 UTC.
+CRAWLER_DAEMON=true
+CRAWLER_DAILY_AT=00:00
+CRAWLER_TZ=UTC
 #
 # New analysis knobs (all optional, all seed defaults):
 #   ANALYSIS_TEMPERATURE       Sampling temperature. Default 0 (deterministic).
@@ -245,6 +291,37 @@ ANALYSIS_JOB_TIMEOUT_SECS=600
 # ANALYSIS_API_KEY  env-ONLY. Bearer-attached to every vision call; never
 #                   persisted, never logged.
 ANALYSIS_API_KEY=
+# Analysis tuning — boot seeds (live-editable from the dashboard once
+# persisted). Defaults are tuned for a 4 GiB-class local vision model;
+# raise/lower as the model and budget allow.
+#   ANALYSIS_MAX_TOKENS         Output-token cap per call. Default 4096.
+#   ANALYSIS_MAX_PIXELS         Per-image pixel budget; we resize to fit.
+#                               Default 1_000_000 (~1 MP).
+#   ANALYSIS_MIN_SLICE_HEIGHT   Slice grid height for tall pages, px.
+#                               Default 640.
+#   ANALYSIS_SLICE_OVERLAP      Fractional overlap between adjacent slices
+#                               so text on a cut survives. Default 0.12.
+#   ANALYSIS_TALL_ASPECT        Slice only when `height/width` exceeds this;
+#                               normal pages take a single call. Default 1.6.
+#   ANALYSIS_MAX_SLICES         Hard cap on slices per page (coarsens
+#                               beyond cap). Default 16.
+#   ANALYSIS_MAX_IMAGE_BYTES    Per-image byte cap before downscale.
+#                               Default 8388608 (8 MiB).
+#   ANALYSIS_RESPONSE_FORMAT    `json_schema` | `json_object` | `none`.
+#                               Default `json_schema`.
+#   ANALYSIS_FREQUENCY_PENALTY  Discourages repetition loops. Default 0.3.
+#   ANALYSIS_TEMPERATURE        Sampling temperature; 0 = deterministic.
+#                               Default 0.
+ANALYSIS_MAX_TOKENS=4096
+ANALYSIS_MAX_PIXELS=1000000
+ANALYSIS_MIN_SLICE_HEIGHT=640
+ANALYSIS_SLICE_OVERLAP=0.12
+ANALYSIS_TALL_ASPECT=1.6
+ANALYSIS_MAX_SLICES=16
+ANALYSIS_MAX_IMAGE_BYTES=8388608
+ANALYSIS_RESPONSE_FORMAT=json_schema
+ANALYSIS_FREQUENCY_PENALTY=0.3
+ANALYSIS_TEMPERATURE=0.0

 # ----- Vision autoscaling (the `ai` compose profile) -----
 # The mangalord-vision (llama.cpp) container pins ~4.4 GiB and has no