Files
Mangalord/frontend/src/hooks.server.ts
MechaCat02 d6ac648ac9 feat(admin): crawler observability dashboard + reliability hardening (0.55.0)
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.

Backend:

- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
  into status / control / dead_jobs / backlog modules. SSE stream
  composes in-memory status with DB-derived queue counts, memoizes
  the counts for 1s and debounces watch pokes for 250ms (~10x QPS
  reduction per subscriber). One-shot GET /admin/crawler shares the
  same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
  (409 Conflict on overlapping click); browser restart goes through
  the coordinated_restart gate (drain + relaunch + auto-clear of the
  sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
  validation, never logged, audit row carries SHA-256 fingerprint).
  Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
  scope=all requires confirm:true; DISTINCT ON dedup keeps the
  partial unique index from rejecting requeues for chapters with
  multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
  panics; last-writer-wins on cover so concurrent dispatches don't
  clobber each other's slot. Pure functions (should_stop /
  should_mark_clean_exit / should_abort_pass) with named regression
  tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
  per-job timeout, circuit breaker on consecutive failures, BrowserManager
  coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
  LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
  Pages stream through with peak memory ~one HTTP chunk + 64-byte
  sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
  and crawler_jobs_dead_idx, both ordered by updated_at DESC to
  match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
  on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
  admin_no_store_guard (Cache-Control: no-store on admin
  responses), audit rows carry per-scope target_id.

Frontend:

- /admin/crawler page decomposed into lib/components/crawler/
  (11 components: ProgressBar, SearchBar, CrawlerHero,
  CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
  MissingCoversTable, DeadJobsTable, RestartConfirmModal,
  RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
  orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
  pageshow (BFCache); after 5 consecutive errors probes the status
  endpoint so a 401 routes through the global on401Hook instead of
  infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
  AbortControllers; refresh after a control action only runs when
  the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
  sync_state locally (no full chapter-list refetch); proper
  aria-label. Requeue-all gets its own confirm modal; both confirm
  modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
  for Accept: text/event-stream; pure shouldBypassProxyTimeout
  helper covered by unit tests.

Config / docs:

- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
  CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
  CRAWLER_BROWSER_RESTART_THRESHOLD.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-06 18:49:56 +02:00

140 lines
5.6 KiB
TypeScript

import type { Handle } from '@sveltejs/kit';
// Reverse-proxy `/api/*` requests through to the backend container.
//
// Mangalord's compose runs SvelteKit (this process) on :3000 and axum on
// :8080. The browser only ever talks to :3000, so cookies stay
// same-origin and `CORS_ALLOWED_ORIGINS` can stay empty in the default
// deploy. The backend hostname comes from `BACKEND_URL` (compose wires
// `http://backend:8080`); for `npm run dev` we fall back to the same
// localhost target the vite proxy uses, which keeps the dev story
// consistent even if someone bypasses the vite proxy.
const BACKEND_URL = process.env.BACKEND_URL ?? 'http://localhost:8080';
/**
* Hop-by-hop headers per RFC 7230 §6.1. These are scoped to a single
* transport-level connection and must not be forwarded by a proxy.
* Plus `host` and `content-length`: `host` would mislead the backend
* about its origin, and `content-length` is recomputed by the upstream
* fetch from the body stream.
*/
const HOP_BY_HOP_HEADERS = [
'host',
'content-length',
'connection',
'keep-alive',
'proxy-authenticate',
'proxy-authorization',
'te',
'trailer',
'transfer-encoding',
'upgrade'
];
/**
* Cap each proxied request at 5 minutes. The bound exists to surface
* a wedged backend (stuck on a slow DB query, deadlocked, etc.) as a
* 502 rather than letting the browser request hang indefinitely.
*
* The default leans toward the slow-upload end of the spectrum: at a
* 1 Mbps upstream, a 200 MiB chapter upload (the default
* `MAX_REQUEST_BYTES` cap) needs ~27 minutes; 300 s covers the more
* realistic 25 Mbps urban-broadband case (~64 s for the same upload)
* with comfortable headroom. Operators serving very slow clients
* should raise `BACKEND_PROXY_TIMEOUT_MS`; operators behind a
* tighter upstream proxy may want to lower it. A future improvement
* is an idle-based timeout (reset per chunk) instead of this
* wall-clock budget — that's a fair bit more code, deferred.
*
* SSE streams (`Accept: text/event-stream`) are exempt — see
* `shouldBypassProxyTimeout`. A wall-clock abort on a long-lived
* stream would tear the connection down every 5 min and force the
* browser to reconnect, which flickers the admin dashboard.
*/
const PROXY_TIMEOUT_MS = (() => {
const raw = process.env.BACKEND_PROXY_TIMEOUT_MS;
const n = raw ? Number(raw) : 300_000;
return Number.isFinite(n) && n > 0 ? n : 300_000;
})();
/**
* Whether the proxy should skip its wall-clock timeout for this
* request. SSE clients open a single connection and stay subscribed
* for the lifetime of the page; aborting on a wall-clock budget would
* tear down the live admin dashboard every 5 min. Exported for unit
* test coverage.
*/
export function shouldBypassProxyTimeout(headers: Headers): boolean {
const accept = headers.get('accept') ?? '';
return accept.toLowerCase().includes('text/event-stream');
}
export const handle: Handle = async ({ event, resolve }) => {
if (event.url.pathname.startsWith('/api/')) {
const target = `${BACKEND_URL}${event.url.pathname}${event.url.search}`;
const headers = new Headers(event.request.headers);
for (const h of HOP_BY_HOP_HEADERS) headers.delete(h);
// AbortController times the upstream fetch out so a backend
// wedged on a slow DB query doesn't keep the browser request
// hanging forever. The `signal` is also wired into the
// RequestInit so the body stream is cancelled cleanly. For
// SSE streams the timer is suppressed so a long-lived stream
// isn't torn down on the 5-minute mark — see T1 in the audit.
const bypassTimeout = shouldBypassProxyTimeout(event.request.headers);
const ctrl = new AbortController();
const timeoutHandle = bypassTimeout
? null
: setTimeout(() => ctrl.abort(), PROXY_TIMEOUT_MS);
const init: RequestInit & { duplex?: 'half' } = {
method: event.request.method,
headers,
redirect: 'manual',
signal: ctrl.signal
};
if (event.request.method !== 'GET' && event.request.method !== 'HEAD') {
init.body = event.request.body;
// Node's fetch requires `duplex: 'half'` when streaming a
// request body; otherwise the stream is rejected.
init.duplex = 'half';
}
let upstream: Response;
try {
upstream = await fetch(target, init);
} catch (e) {
// Network-layer failure (DNS / connection refused / TLS
// handshake / abort by timeout) — most commonly "backend
// container restarting". SvelteKit's default 500 would be
// an HTML page that client.ts can't .json(), which masks
// the real cause. Emit the standard envelope with a
// dedicated code instead.
console.error('Proxy to backend failed:', e);
if (timeoutHandle) clearTimeout(timeoutHandle);
return new Response(
JSON.stringify({
error: {
code: 'upstream_unavailable',
message: 'backend unreachable'
}
}),
{
status: 502,
headers: { 'content-type': 'application/json' }
}
);
}
if (timeoutHandle) clearTimeout(timeoutHandle);
return new Response(upstream.body, {
status: upstream.status,
statusText: upstream.statusText,
headers: upstream.headers
});
}
return resolve(event);
};