Files
Mangalord/.env.example
MechaCat02 0a9e727530 feat(vision-manager): memory-pressure yield gate
Add a fourth gate to the vision autoscaler poll loop: when host memory
gets tight, proactively SIGTERM-stop the mangalord-vision container so a
transient spike elsewhere (a crawl, CI, a backend burst) can finish
instead of the kernel OOM-killer shooting something stateful.

- read_mem_used_pct: host-wide used% from /proc/meminfo MemAvailable
  (not MemFree), ERR sentinel when MemTotal or MemAvailable is missing.
- mem_check_and_stop: active stop over MEM_HIGH_WATERMARK_PCT + arm a
  restart cooldown; only inspects docker once already over the mark.
- mem_yield_active: start inhibit while cooldown active or used% >=
  MEM_LOW_WATERMARK_PCT (two watermarks = hysteresis).
- Inhibit reuses the existing idle path via pending=0; flip-only logging;
  stop_vision gains a reason arg so idle/MAX_UPTIME/memory-yield stops are
  distinctly logged.
- Faster MEM_POLL_INTERVAL sub-poll between backlog ticks to catch spikes.

Backend verifications V1-V4 (see VISION-MEMORY-YIELD.md) all pass against
the current queue, so no backend changes are needed. RESPECT_CRAWL_MUTEX
is kept for now. New VISION_MEM_* knobs wired through docker-compose.yml
and documented in .env.example.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 14:33:34 +02:00

233 lines
12 KiB
Plaintext

# Copy to .env for `docker compose up --build`. Local-dev runs (cargo run
# / npm run dev) read backend/.env if present, or pick up the variables
# from your shell.
#
# Production note: COOKIE_SECURE=true (the default below) makes browsers
# refuse to send the session cookie over plain HTTP. Run with a TLS-
# terminating reverse proxy (Caddy, Traefik, nginx) in front — the
# compose file here doesn't ship one. Local/dev runs without HTTPS
# should set COOKIE_SECURE=false.
# ----- Postgres -----
# These are read by the Postgres container *and* by DATABASE_URL below;
# changing them after the first boot won't migrate existing data, so set
# them up front for any new deployment.
#
# POSTGRES_PASSWORD is REQUIRED — docker-compose.yml fails fast if it
# isn't set in this file, to prevent a deploy without an .env booting
# Postgres with a publicly-known credential.
POSTGRES_USER=mangalord
POSTGRES_PASSWORD=change-me-to-a-strong-random-string
POSTGRES_DB=mangalord
# ----- Backend -----
DATABASE_URL=postgres://mangalord:mangalord@postgres:5432/mangalord
BIND_ADDRESS=0.0.0.0:8080
STORAGE_DIR=/var/lib/mangalord/storage
RUST_LOG=info,mangalord=debug,chromiumoxide::conn=off,chromiumoxide::handler=off
# ----- Auth / cookies -----
# COOKIE_SECURE controls whether the `Secure` flag is set on the session
# cookie. Keep `true` in production (HTTPS); set to `false` if you're
# serving over plain HTTP locally (e.g., behind a dev reverse proxy).
COOKIE_SECURE=true
# COOKIE_DOMAIN scopes the session cookie. Leave empty to default to the
# requesting host. Set when serving the API and frontend on subdomains of
# a shared parent (e.g., `.example.com`) so the cookie is shared.
COOKIE_DOMAIN=
# Session lifetime in days. Expired sessions are no longer accepted and
# get reaped lazily.
SESSION_TTL_DAYS=30
# ----- Auth brute-force rate limits -----
# Token-bucket budget shared across /auth/login, /auth/register, and
# /auth/me/password. Set per_sec=0 to disable (e.g. behind a
# rate-limiting reverse proxy that already enforces a budget).
AUTH_RATE_PER_SEC=5
AUTH_RATE_BURST=10
# ----- CORS -----
# Comma-separated origins allowed to call the API with credentials.
# Default is empty: same-origin only. Set when frontend and backend live
# on different hosts. Example: https://app.example.com,https://app.example.de
CORS_ALLOWED_ORIGINS=
# ----- Admin CSRF allowlist -----
# Browser origins (scheme + host[:port]) permitted to POST to
# /api/v1/admin/* mutating endpoints. Defends the session-cookie-
# authenticated admin surface against SameSite=Lax form-POST CSRF.
# Same shape as CORS_ALLOWED_ORIGINS (comma-separated). Compare against
# the request's Origin header (falling back to Referer when absent);
# safe methods (GET/HEAD/OPTIONS) are not checked, and requests with
# neither Origin nor Referer (curl, server-to-server callers) are
# always allowed.
#
# Default is empty: CSRF check disabled (operator opt-out). For a
# browser-exposed deployment this should be set to the SvelteKit
# origin, e.g. https://app.example.com. For a same-origin
# docker-compose deploy where only one origin exists, set the same
# value the browser uses.
ADMIN_ALLOWED_ORIGINS=
# ----- Upload limits -----
# Per-request body cap. axum rejects oversized requests with 413 before
# our handlers run. Default 200 MiB.
MAX_REQUEST_BYTES=209715200
# Per-image-part cap. Enforced after reading each part, so a single
# oversized image is rejected even when the total request fits.
# Default 20 MiB.
MAX_FILE_BYTES=20971520
# ----- Crawler download safety -----
# Hosts the crawler is allowed to fetch images/covers from, in addition
# to CRAWLER_START_URL's host and CRAWLER_CDN_HOST. Comma-separated.
# Defends against SSRF via scraped <img src="http://10.0.0.1/...">.
CRAWLER_DOWNLOAD_ALLOWLIST=
# Bypass the host allowlist entirely. Intended for sources that shard
# images across numbered CDN subdomains (cdn1/cdn2/…) where enumerating
# every host upfront is impractical. The private-IP / localhost / non-
# http(s) scheme defenses STAY ON — a scraped <img src="http://10.0.0.1/">
# is still refused with this flag set.
CRAWLER_ALLOW_ANY_HOST=false
# Hard cap on a single image body. Default 32 MiB.
CRAWLER_MAX_IMAGE_BYTES=33554432
# Max manga detail fetches per metadata pass (both the in-process daemon
# and the `bin/crawler` CLI). 0 means no cap — let the source walker run
# to completion. Useful for capped test runs against a new source.
CRAWLER_LIMIT=0
# ----- Crawler reliability knobs -----
# Hard upper bound on a single chapter-content job dispatch (seconds).
# A job that exceeds the budget is acked failed (with exponential
# backoff) instead of wedging a worker. Default 600s.
CRAWLER_JOB_TIMEOUT_SECS=600
# Consecutive metadata-pass `fetch_manga` failures that abort the pass
# (circuit breaker for a source outage). The pass does NOT mark a clean
# exit, so the next tick does a recovery sweep. Default 10.
CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES=10
# Consecutive transient chapter failures (after TOR recircuit is
# exhausted) that trigger an automatic coordinated browser restart.
# Default 3.
CRAWLER_BROWSER_RESTART_THRESHOLD=3
# Path to a system Chromium binary. When set, the crawler skips the
# bundled-fetcher download. Required on platforms without a usable
# upstream Chromium build (notably Linux_arm64 / Raspberry Pi). On
# Debian: /usr/bin/chromium-headless-shell or /usr/bin/chromium. On
# Ubuntu the package is chromium-browser (different path). Pair with
# `docker compose build --build-arg INSTALL_CHROMIUM=true backend` so
# the image actually contains the binary.
CRAWLER_CHROMIUM_BINARY=
# ----- Crawler TOR proxy + recircuit -----
# The compose stack ships a `tor` service (dockurr/tor) and defaults
# CRAWLER_PROXY to it, so by default all crawler traffic exits via the
# TOR network. To opt out, set CRAWLER_PROXY= (empty) AND
# CRAWLER_TOR_CONTROL_URL= (empty) below — the tor service can stay
# running, it just won't be used.
#
# Going through TOR adds latency to every fetch; image downloads in
# particular slow noticeably. The win is on sites that rate-limit or
# fingerprint by exit IP — NEWNYM recirculation makes a fresh exit
# cheap to reach for.
#
# CRAWLER_PROXY: SOCKS5(h) URL. Use `socks5h://` (not `socks5://`) so
# DNS resolution also goes through TOR, avoiding leaks via the host's
# resolver. Leave unset to talk to the upstream directly.
CRAWLER_PROXY=socks5h://tor:9050
# Control-port URL for SIGNAL NEWNYM ("get a fresh circuit"). Triggered
# automatically on bad pages (broken-page body, missing #logo) and on
# the Unauthenticated session probe outcome. Leave unset to disable
# the recircuit feature (the SOCKS proxy still works).
CRAWLER_TOR_CONTROL_URL=tcp://tor:9051
# Max NEWNYM-and-retry cycles per recircuit-eligible failure. Default 3.
CRAWLER_TOR_RECIRCUIT_MAX_ATTEMPTS=3
# ----- TOR control-port password -----
# Shared between the bundled dockurr/tor service (which hashes it into
# its HashedControlPassword) and the backend's
# CRAWLER_TOR_CONTROL_PASSWORD. REQUIRED — docker-compose.yml fails
# fast if absent. Generate a strong random string; rotate by setting
# a new value and restarting both `tor` and `backend`.
#
# Operators running their own non-dockurr tor daemon with cookie-file
# auth can ignore this var and instead set
# CRAWLER_TOR_CONTROL_COOKIE_PATH on the backend — the TorController
# prefers cookie when both are present.
TOR_CONTROL_PASSWORD=change-me-to-a-strong-random-string
# ----- Frontend -----
# The frontend container runs SvelteKit's Node adapter on :3000 and
# proxies /api/* to BACKEND_URL via src/hooks.server.ts. In compose the
# default `http://backend:8080` reaches the backend service over the
# internal docker network. Override only if you're running the
# frontend container against a backend somewhere else.
BACKEND_URL=http://backend:8080
# Per-request wall-clock cap for the /api/* reverse proxy (milliseconds).
# Default 300000 (5 min) covers a typical 200 MiB chapter upload over
# 25 Mbps; raise for users on slower upstream links or lower if a
# tighter front proxy already bounds the request lifetime.
BACKEND_PROXY_TIMEOUT_MS=300000
# ----- Runtime settings (crawler + analysis) -----
# The crawler and analysis subsystems are configured at runtime from the
# `app_settings` table and edited live in the admin dashboard
# (Admin → Settings) — no restart needed. The CRAWLER_* / ANALYSIS_* env
# vars below act as the BOOT SEED: on first boot (when the row is absent)
# the env values populate the DB; thereafter the DB is the source of
# truth and env changes are ignored for those fields.
#
# Host/infra and secret fields stay env-ONLY (never persisted, shown
# read-only in the dashboard): the chromium binary/dir, browser mode/args,
# CRAWLER_PROXY, all CRAWLER_TOR_CONTROL_*, the cookie domain, and the
# analysis ANALYSIS_API_KEY. The important site levers (PRIVATE_MODE,
# ALLOW_SELF_REGISTER) also remain env-only by design.
#
# New analysis knobs (all optional, all seed defaults):
# ANALYSIS_TEMPERATURE Sampling temperature. Default 0 (deterministic).
# ANALYSIS_SYSTEM_PROMPT Override the single-call vision system prompt.
# ANALYSIS_OCR_PROMPT Override the tall-page OCR (pass A) prompt.
# ANALYSIS_GROUNDING_PROMPT Override the tags/scene/safety (pass B) prompt.
# Leave the prompt vars unset to use the built-in defaults (also editable,
# with a per-prompt "reset to default", in the dashboard).
# ----- Vision autoscaling (the `ai` compose profile) -----
# The mangalord-vision (llama.cpp) container pins ~4.4 GiB and has no
# idle-unload, so the `vision-manager` sidecar starts it on demand and
# idle-stops it. Bring the two helper containers up with:
# docker compose --profile ai up -d
# The vision container itself is NOT defined in compose — it lives elsewhere
# on the host and the manager drives it by name. See VISION-AUTOSCALE.md.
#
# ANALYSIS_VISION_HEALTH_URL — env-ONLY readiness gate for the analysis
# worker. When set, the worker refuses to lease a page until this answers
# 2xx, so the manager can stop vision mid-idle without jobs burning their
# retries / landing `failed` rows. MUST point at the same vision instance the
# worker analyses against. Leave empty to disable the gate (always-on setups).
ANALYSIS_VISION_HEALTH_URL=http://mangalord-vision:8000/health
#
# vision-manager knobs:
# VISION_MANAGER_DATABASE_URL — REQUIRED for the `ai` profile. A read-only
# role, NOT the backend creds: apply vision-manager/readonly-role.sql once,
# then set e.g.
# VISION_MANAGER_DATABASE_URL=postgres://vision_manager:<pw>@postgres:5432/mangalord
VISION_MANAGER_DATABASE_URL=
# VISION_CONTAINER Container name the manager starts/stops. Default mangalord-vision.
# VISION_HEALTH_URL /health URL the manager polls after start. Default http://mangalord-vision:8000/health.
# VISION_POLL_INTERVAL Backlog poll cadence, seconds. Default 20.
# VISION_STOP_DEBOUNCE Idle seconds before stopping vision. Default 600 (debounces bursty enqueues).
# VISION_START_HEALTH_TIMEOUT Max seconds to wait for /health 200 after start. Default 300 (cold model load).
# VISION_RESPECT_CRAWL_MUTEX 1 = don't start vision while a crawl runs (RAM mutex on the 8 GiB box). Default 1.
# VISION_MAX_UPTIME >0 = force-stop vision after N seconds running (backstop). Default 0 (disabled).
#
# Memory-pressure yield — the dynamic generalization of the crawl mutex. The
# manager reads host-wide used% (from /proc/meminfo MemAvailable) and stops a
# RUNNING vision when memory gets tight, so a spike elsewhere (a crawl, CI, a
# backend burst) can finish instead of the kernel OOM-killer shooting something
# stateful. Two watermarks give hysteresis; the cooldown prevents stop/restart
# thrash when vision itself is the hog. See VISION-MEMORY-YIELD.md.
# VISION_MEM_YIELD_ENABLED 1 = enable the gate, 0 = disable entirely. Default 1.
# VISION_MEM_HIGH_WATERMARK_PCT used% >= this → actively stop a running vision. Default 92.
# VISION_MEM_LOW_WATERMARK_PCT used% >= this → inhibit starts (keep HIGH - LOW >= ~10 so the band beats jitter). Default 80.
# VISION_MEM_YIELD_COOLDOWN Seconds after a pressure-stop during which restart is refused regardless of backlog. Default 300.
# VISION_MEM_POLL_INTERVAL Mem sub-poll cadence, seconds (<= VISION_POLL_INTERVAL); catches spikes between backlog ticks. Default 5.