d6ac648ac9
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.
Backend:
- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
into status / control / dead_jobs / backlog modules. SSE stream
composes in-memory status with DB-derived queue counts, memoizes
the counts for 1s and debounces watch pokes for 250ms (~10x QPS
reduction per subscriber). One-shot GET /admin/crawler shares the
same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
(409 Conflict on overlapping click); browser restart goes through
the coordinated_restart gate (drain + relaunch + auto-clear of the
sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
validation, never logged, audit row carries SHA-256 fingerprint).
Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
scope=all requires confirm:true; DISTINCT ON dedup keeps the
partial unique index from rejecting requeues for chapters with
multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
panics; last-writer-wins on cover so concurrent dispatches don't
clobber each other's slot. Pure functions (should_stop /
should_mark_clean_exit / should_abort_pass) with named regression
tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
per-job timeout, circuit breaker on consecutive failures, BrowserManager
coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
Pages stream through with peak memory ~one HTTP chunk + 64-byte
sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
and crawler_jobs_dead_idx, both ordered by updated_at DESC to
match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
admin_no_store_guard (Cache-Control: no-store on admin
responses), audit rows carry per-scope target_id.
Frontend:
- /admin/crawler page decomposed into lib/components/crawler/
(11 components: ProgressBar, SearchBar, CrawlerHero,
CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
MissingCoversTable, DeadJobsTable, RestartConfirmModal,
RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
pageshow (BFCache); after 5 consecutive errors probes the status
endpoint so a 401 routes through the global on401Hook instead of
infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
AbortControllers; refresh after a control action only runs when
the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
sync_state locally (no full chapter-list refetch); proper
aria-label. Requeue-all gets its own confirm modal; both confirm
modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
for Accept: text/event-stream; pure shouldBypassProxyTimeout
helper covered by unit tests.
Config / docs:
- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
CRAWLER_BROWSER_RESTART_THRESHOLD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
170 lines
8.0 KiB
Plaintext
170 lines
8.0 KiB
Plaintext
# Copy to .env for `docker compose up --build`. Local-dev runs (cargo run
|
|
# / npm run dev) read backend/.env if present, or pick up the variables
|
|
# from your shell.
|
|
#
|
|
# Production note: COOKIE_SECURE=true (the default below) makes browsers
|
|
# refuse to send the session cookie over plain HTTP. Run with a TLS-
|
|
# terminating reverse proxy (Caddy, Traefik, nginx) in front — the
|
|
# compose file here doesn't ship one. Local/dev runs without HTTPS
|
|
# should set COOKIE_SECURE=false.
|
|
|
|
# ----- Postgres -----
|
|
# These are read by the Postgres container *and* by DATABASE_URL below;
|
|
# changing them after the first boot won't migrate existing data, so set
|
|
# them up front for any new deployment.
|
|
#
|
|
# POSTGRES_PASSWORD is REQUIRED — docker-compose.yml fails fast if it
|
|
# isn't set in this file, to prevent a deploy without an .env booting
|
|
# Postgres with a publicly-known credential.
|
|
POSTGRES_USER=mangalord
|
|
POSTGRES_PASSWORD=change-me-to-a-strong-random-string
|
|
POSTGRES_DB=mangalord
|
|
|
|
# ----- Backend -----
|
|
DATABASE_URL=postgres://mangalord:mangalord@postgres:5432/mangalord
|
|
BIND_ADDRESS=0.0.0.0:8080
|
|
STORAGE_DIR=/var/lib/mangalord/storage
|
|
RUST_LOG=info,mangalord=debug,chromiumoxide::conn=off,chromiumoxide::handler=off
|
|
|
|
# ----- Auth / cookies -----
|
|
# COOKIE_SECURE controls whether the `Secure` flag is set on the session
|
|
# cookie. Keep `true` in production (HTTPS); set to `false` if you're
|
|
# serving over plain HTTP locally (e.g., behind a dev reverse proxy).
|
|
COOKIE_SECURE=true
|
|
# COOKIE_DOMAIN scopes the session cookie. Leave empty to default to the
|
|
# requesting host. Set when serving the API and frontend on subdomains of
|
|
# a shared parent (e.g., `.example.com`) so the cookie is shared.
|
|
COOKIE_DOMAIN=
|
|
# Session lifetime in days. Expired sessions are no longer accepted and
|
|
# get reaped lazily.
|
|
SESSION_TTL_DAYS=30
|
|
|
|
# ----- Auth brute-force rate limits -----
|
|
# Token-bucket budget shared across /auth/login, /auth/register, and
|
|
# /auth/me/password. Set per_sec=0 to disable (e.g. behind a
|
|
# rate-limiting reverse proxy that already enforces a budget).
|
|
AUTH_RATE_PER_SEC=5
|
|
AUTH_RATE_BURST=10
|
|
|
|
# ----- CORS -----
|
|
# Comma-separated origins allowed to call the API with credentials.
|
|
# Default is empty: same-origin only. Set when frontend and backend live
|
|
# on different hosts. Example: https://app.example.com,https://app.example.de
|
|
CORS_ALLOWED_ORIGINS=
|
|
|
|
# ----- Admin CSRF allowlist -----
|
|
# Browser origins (scheme + host[:port]) permitted to POST to
|
|
# /api/v1/admin/* mutating endpoints. Defends the session-cookie-
|
|
# authenticated admin surface against SameSite=Lax form-POST CSRF.
|
|
# Same shape as CORS_ALLOWED_ORIGINS (comma-separated). Compare against
|
|
# the request's Origin header (falling back to Referer when absent);
|
|
# safe methods (GET/HEAD/OPTIONS) are not checked, and requests with
|
|
# neither Origin nor Referer (curl, server-to-server callers) are
|
|
# always allowed.
|
|
#
|
|
# Default is empty: CSRF check disabled (operator opt-out). For a
|
|
# browser-exposed deployment this should be set to the SvelteKit
|
|
# origin, e.g. https://app.example.com. For a same-origin
|
|
# docker-compose deploy where only one origin exists, set the same
|
|
# value the browser uses.
|
|
ADMIN_ALLOWED_ORIGINS=
|
|
|
|
# ----- Upload limits -----
|
|
# Per-request body cap. axum rejects oversized requests with 413 before
|
|
# our handlers run. Default 200 MiB.
|
|
MAX_REQUEST_BYTES=209715200
|
|
# Per-image-part cap. Enforced after reading each part, so a single
|
|
# oversized image is rejected even when the total request fits.
|
|
# Default 20 MiB.
|
|
MAX_FILE_BYTES=20971520
|
|
|
|
# ----- Crawler download safety -----
|
|
# Hosts the crawler is allowed to fetch images/covers from, in addition
|
|
# to CRAWLER_START_URL's host and CRAWLER_CDN_HOST. Comma-separated.
|
|
# Defends against SSRF via scraped <img src="http://10.0.0.1/...">.
|
|
CRAWLER_DOWNLOAD_ALLOWLIST=
|
|
# Bypass the host allowlist entirely. Intended for sources that shard
|
|
# images across numbered CDN subdomains (cdn1/cdn2/…) where enumerating
|
|
# every host upfront is impractical. The private-IP / localhost / non-
|
|
# http(s) scheme defenses STAY ON — a scraped <img src="http://10.0.0.1/">
|
|
# is still refused with this flag set.
|
|
CRAWLER_ALLOW_ANY_HOST=false
|
|
# Hard cap on a single image body. Default 32 MiB.
|
|
CRAWLER_MAX_IMAGE_BYTES=33554432
|
|
# Max manga detail fetches per metadata pass (both the in-process daemon
|
|
# and the `bin/crawler` CLI). 0 means no cap — let the source walker run
|
|
# to completion. Useful for capped test runs against a new source.
|
|
CRAWLER_LIMIT=0
|
|
|
|
# ----- Crawler reliability knobs -----
|
|
# Hard upper bound on a single chapter-content job dispatch (seconds).
|
|
# A job that exceeds the budget is acked failed (with exponential
|
|
# backoff) instead of wedging a worker. Default 600s.
|
|
CRAWLER_JOB_TIMEOUT_SECS=600
|
|
# Consecutive metadata-pass `fetch_manga` failures that abort the pass
|
|
# (circuit breaker for a source outage). The pass does NOT mark a clean
|
|
# exit, so the next tick does a recovery sweep. Default 10.
|
|
CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES=10
|
|
# Consecutive transient chapter failures (after TOR recircuit is
|
|
# exhausted) that trigger an automatic coordinated browser restart.
|
|
# Default 3.
|
|
CRAWLER_BROWSER_RESTART_THRESHOLD=3
|
|
# Path to a system Chromium binary. When set, the crawler skips the
|
|
# bundled-fetcher download. Required on platforms without a usable
|
|
# upstream Chromium build (notably Linux_arm64 / Raspberry Pi). On
|
|
# Debian: /usr/bin/chromium-headless-shell or /usr/bin/chromium. On
|
|
# Ubuntu the package is chromium-browser (different path). Pair with
|
|
# `docker compose build --build-arg INSTALL_CHROMIUM=true backend` so
|
|
# the image actually contains the binary.
|
|
CRAWLER_CHROMIUM_BINARY=
|
|
|
|
# ----- Crawler TOR proxy + recircuit -----
|
|
# The compose stack ships a `tor` service (dockurr/tor) and defaults
|
|
# CRAWLER_PROXY to it, so by default all crawler traffic exits via the
|
|
# TOR network. To opt out, set CRAWLER_PROXY= (empty) AND
|
|
# CRAWLER_TOR_CONTROL_URL= (empty) below — the tor service can stay
|
|
# running, it just won't be used.
|
|
#
|
|
# Going through TOR adds latency to every fetch; image downloads in
|
|
# particular slow noticeably. The win is on sites that rate-limit or
|
|
# fingerprint by exit IP — NEWNYM recirculation makes a fresh exit
|
|
# cheap to reach for.
|
|
#
|
|
# CRAWLER_PROXY: SOCKS5(h) URL. Use `socks5h://` (not `socks5://`) so
|
|
# DNS resolution also goes through TOR, avoiding leaks via the host's
|
|
# resolver. Leave unset to talk to the upstream directly.
|
|
CRAWLER_PROXY=socks5h://tor:9050
|
|
# Control-port URL for SIGNAL NEWNYM ("get a fresh circuit"). Triggered
|
|
# automatically on bad pages (broken-page body, missing #logo) and on
|
|
# the Unauthenticated session probe outcome. Leave unset to disable
|
|
# the recircuit feature (the SOCKS proxy still works).
|
|
CRAWLER_TOR_CONTROL_URL=tcp://tor:9051
|
|
# Max NEWNYM-and-retry cycles per recircuit-eligible failure. Default 3.
|
|
CRAWLER_TOR_RECIRCUIT_MAX_ATTEMPTS=3
|
|
|
|
# ----- TOR control-port password -----
|
|
# Shared between the bundled dockurr/tor service (which hashes it into
|
|
# its HashedControlPassword) and the backend's
|
|
# CRAWLER_TOR_CONTROL_PASSWORD. REQUIRED — docker-compose.yml fails
|
|
# fast if absent. Generate a strong random string; rotate by setting
|
|
# a new value and restarting both `tor` and `backend`.
|
|
#
|
|
# Operators running their own non-dockurr tor daemon with cookie-file
|
|
# auth can ignore this var and instead set
|
|
# CRAWLER_TOR_CONTROL_COOKIE_PATH on the backend — the TorController
|
|
# prefers cookie when both are present.
|
|
TOR_CONTROL_PASSWORD=change-me-to-a-strong-random-string
|
|
|
|
# ----- Frontend -----
|
|
# The frontend container runs SvelteKit's Node adapter on :3000 and
|
|
# proxies /api/* to BACKEND_URL via src/hooks.server.ts. In compose the
|
|
# default `http://backend:8080` reaches the backend service over the
|
|
# internal docker network. Override only if you're running the
|
|
# frontend container against a backend somewhere else.
|
|
BACKEND_URL=http://backend:8080
|
|
# Per-request wall-clock cap for the /api/* reverse proxy (milliseconds).
|
|
# Default 300000 (5 min) covers a typical 200 MiB chapter upload over
|
|
# 25 Mbps; raise for users on slower upstream links or lower if a
|
|
# tighter front proxy already bounds the request lifetime.
|
|
BACKEND_PROXY_TIMEOUT_MS=300000
|