fix(config): derive BACKEND_CONSUMED from .env.example; scan src/ for undocumented env reads (0.87.25)

0.87.12 added a regression test for backend env wiring, but its
BACKEND_CONSUMED list was hand-maintained and drifted: ~22 vars read
by backend/src/ never made it into compose's backend.environment.
An operator setting ANALYSIS_TEMPERATURE=0.3 in .env got a silent
no-op — exactly the bug class the test was supposed to catch.

Make .env.example the single source of truth:

  * docker_compose_wires_every_documented_backend_env_var now reads
    .env.example at test time, filters NOT_BACKEND_CONSUMED_IN_ENV_EXAMPLE
    (compose-side composed vars, frontend-only vars, sidecar vars),
    and requires every remaining key to be wired into compose with a
    matching ${KEY...} interpolation OR allowlisted via COMPOSE_RHS_EXEMPT.
  * every_backend_src_env_read_is_documented_or_allowlisted scans
    backend/src/ for env::var / env_bool / env_i64 / ... reads and
    requires each key to be in .env.example OR in INTERNAL_NOT_CONFIGURABLE
    with a per-entry rationale. Catches the silent-no-op bug from the
    other side.

Wire the previously-missing env vars into docker-compose.yml's
backend.environment (CRAWLER_DAEMON, CRAWLER_DAILY_AT, CRAWLER_TZ,
CRAWLER_IDLE_TIMEOUT_S, CRAWLER_CHAPTER_WORKERS, CRAWLER_JOB_RETENTION_DAYS,
CRAWL_METRICS_RETENTION_DAYS, CRAWLER_RATE_MS, CRAWLER_CDN_RATE_MS,
CRAWLER_USER_AGENT, CRAWLER_PHPSESSID, CRAWLER_COOKIE_DOMAIN,
CRAWLER_TOR_CONTROL_COOKIE_PATH, all ANALYSIS_* tuning knobs).
Document each new var in .env.example with the same default value
shown in compose's `${KEY:-default}` form. Allowlist internal/system
vars (HOME = chromium cache dir fallback, CRAWLER_BROWSER_*,
CRAWLER_SKIP_*, the multi-paragraph ANALYSIS_*_PROMPT seeds) with
per-entry rationale.

Mutation-tested both sides: adding `env::var("MANGALORD_FAKE_NEW_KNOB")`
to main.rs fails the scanner; removing the new compose wiring fails
the documented-var test with all 23 missing keys named.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

docker-compose.yml

@@ -104,6 +104,22 @@ services:
       CRAWLER_JOB_TIMEOUT_SECS: ${CRAWLER_JOB_TIMEOUT_SECS:-600}
       CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES: ${CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES:-10}
       CRAWLER_BROWSER_RESTART_THRESHOLD: ${CRAWLER_BROWSER_RESTART_THRESHOLD:-3}
+      # Crawler daemon schedule + retention. CRAWLER_DAEMON=false keeps
+      # the in-process scheduler off; the dashboard force-resync still works.
+      CRAWLER_DAEMON: ${CRAWLER_DAEMON:-true}
+      CRAWLER_DAILY_AT: ${CRAWLER_DAILY_AT:-00:00}
+      CRAWLER_TZ: ${CRAWLER_TZ:-UTC}
+      CRAWLER_IDLE_TIMEOUT_S: ${CRAWLER_IDLE_TIMEOUT_S:-600}
+      CRAWLER_CHAPTER_WORKERS: ${CRAWLER_CHAPTER_WORKERS:-1}
+      CRAWLER_JOB_RETENTION_DAYS: ${CRAWLER_JOB_RETENTION_DAYS:-7}
+      CRAWL_METRICS_RETENTION_DAYS: ${CRAWL_METRICS_RETENTION_DAYS:-90}
+      # Crawler politeness — pause between source / CDN fetches (ms).
+      CRAWLER_RATE_MS: ${CRAWLER_RATE_MS:-1000}
+      CRAWLER_CDN_RATE_MS: ${CRAWLER_CDN_RATE_MS:-1000}
+      # Crawler session/identity to the source (cookies, UA).
+      CRAWLER_USER_AGENT: ${CRAWLER_USER_AGENT:-}
+      CRAWLER_PHPSESSID: ${CRAWLER_PHPSESSID:-}
+      CRAWLER_COOKIE_DOMAIN: ${CRAWLER_COOKIE_DOMAIN:-}
       # Crawler download safety — env-only, never persisted.
       CRAWLER_DOWNLOAD_ALLOWLIST: ${CRAWLER_DOWNLOAD_ALLOWLIST:-}
       CRAWLER_ALLOW_ANY_HOST: ${CRAWLER_ALLOW_ANY_HOST:-false}
@@ -123,6 +139,10 @@ services:
       CRAWLER_TOR_CONTROL_URL: ${CRAWLER_TOR_CONTROL_URL-tcp://tor:9051}
       CRAWLER_TOR_CONTROL_PASSWORD: ${TOR_CONTROL_PASSWORD:?TOR_CONTROL_PASSWORD must be set in .env}
       CRAWLER_TOR_RECIRCUIT_MAX_ATTEMPTS: ${CRAWLER_TOR_RECIRCUIT_MAX_ATTEMPTS:-3}
+      # Tor cookie-file auth alternative — the controller prefers cookie
+      # when both are present. Leave unset for the bundled
+      # HashedControlPassword path above.
+      CRAWLER_TOR_CONTROL_COOKIE_PATH: ${CRAWLER_TOR_CONTROL_COOKIE_PATH:-}
       # Analysis worker (vision) — boot seeds + env-only API key.
       ANALYSIS_ENABLED: ${ANALYSIS_ENABLED:-false}
       ANALYSIS_VISION_URL: ${ANALYSIS_VISION_URL:-}
@@ -132,6 +152,17 @@ services:
       # API key never gets persisted to app_settings; bearer-attached to
       # every vision call.
       ANALYSIS_API_KEY: ${ANALYSIS_API_KEY:-}
+      # Analysis tuning knobs — boot seeds, then live-editable in dashboard.
+      ANALYSIS_MAX_TOKENS: ${ANALYSIS_MAX_TOKENS:-4096}
+      ANALYSIS_MAX_PIXELS: ${ANALYSIS_MAX_PIXELS:-1000000}
+      ANALYSIS_MIN_SLICE_HEIGHT: ${ANALYSIS_MIN_SLICE_HEIGHT:-640}
+      ANALYSIS_SLICE_OVERLAP: ${ANALYSIS_SLICE_OVERLAP:-0.12}
+      ANALYSIS_TALL_ASPECT: ${ANALYSIS_TALL_ASPECT:-1.6}
+      ANALYSIS_MAX_SLICES: ${ANALYSIS_MAX_SLICES:-16}
+      ANALYSIS_MAX_IMAGE_BYTES: ${ANALYSIS_MAX_IMAGE_BYTES:-8388608}
+      ANALYSIS_RESPONSE_FORMAT: ${ANALYSIS_RESPONSE_FORMAT:-json_schema}
+      ANALYSIS_FREQUENCY_PENALTY: ${ANALYSIS_FREQUENCY_PENALTY:-0.3}
+      ANALYSIS_TEMPERATURE: ${ANALYSIS_TEMPERATURE:-0.0}
       # Vision readiness gate (env-ONLY, not a dashboard setting). When set,
       # the analysis worker refuses to lease a page until this answers 2xx, so
       # the vision-manager autoscaler can idle-stop the vision container