feat(admin): crawler observability dashboard + reliability hardening (0.55.0)
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.
Backend:
- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
into status / control / dead_jobs / backlog modules. SSE stream
composes in-memory status with DB-derived queue counts, memoizes
the counts for 1s and debounces watch pokes for 250ms (~10x QPS
reduction per subscriber). One-shot GET /admin/crawler shares the
same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
(409 Conflict on overlapping click); browser restart goes through
the coordinated_restart gate (drain + relaunch + auto-clear of the
sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
validation, never logged, audit row carries SHA-256 fingerprint).
Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
scope=all requires confirm:true; DISTINCT ON dedup keeps the
partial unique index from rejecting requeues for chapters with
multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
panics; last-writer-wins on cover so concurrent dispatches don't
clobber each other's slot. Pure functions (should_stop /
should_mark_clean_exit / should_abort_pass) with named regression
tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
per-job timeout, circuit breaker on consecutive failures, BrowserManager
coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
Pages stream through with peak memory ~one HTTP chunk + 64-byte
sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
and crawler_jobs_dead_idx, both ordered by updated_at DESC to
match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
admin_no_store_guard (Cache-Control: no-store on admin
responses), audit rows carry per-scope target_id.
Frontend:
- /admin/crawler page decomposed into lib/components/crawler/
(11 components: ProgressBar, SearchBar, CrawlerHero,
CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
MissingCoversTable, DeadJobsTable, RestartConfirmModal,
RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
pageshow (BFCache); after 5 consecutive errors probes the status
endpoint so a 401 routes through the global on401Hook instead of
infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
AbortControllers; refresh after a control action only runs when
the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
sync_state locally (no full chapter-list refetch); proper
aria-label. Requeue-all gets its own confirm modal; both confirm
modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
for Accept: text/event-stream; pure shouldBypassProxyTimeout
helper covered by unit tests.
Config / docs:
- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
CRAWLER_BROWSER_RESTART_THRESHOLD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -46,6 +46,11 @@ const HOP_BY_HOP_HEADERS = [
|
||||
* tighter upstream proxy may want to lower it. A future improvement
|
||||
* is an idle-based timeout (reset per chunk) instead of this
|
||||
* wall-clock budget — that's a fair bit more code, deferred.
|
||||
*
|
||||
* SSE streams (`Accept: text/event-stream`) are exempt — see
|
||||
* `shouldBypassProxyTimeout`. A wall-clock abort on a long-lived
|
||||
* stream would tear the connection down every 5 min and force the
|
||||
* browser to reconnect, which flickers the admin dashboard.
|
||||
*/
|
||||
const PROXY_TIMEOUT_MS = (() => {
|
||||
const raw = process.env.BACKEND_PROXY_TIMEOUT_MS;
|
||||
@@ -53,6 +58,18 @@ const PROXY_TIMEOUT_MS = (() => {
|
||||
return Number.isFinite(n) && n > 0 ? n : 300_000;
|
||||
})();
|
||||
|
||||
/**
|
||||
* Whether the proxy should skip its wall-clock timeout for this
|
||||
* request. SSE clients open a single connection and stay subscribed
|
||||
* for the lifetime of the page; aborting on a wall-clock budget would
|
||||
* tear down the live admin dashboard every 5 min. Exported for unit
|
||||
* test coverage.
|
||||
*/
|
||||
export function shouldBypassProxyTimeout(headers: Headers): boolean {
|
||||
const accept = headers.get('accept') ?? '';
|
||||
return accept.toLowerCase().includes('text/event-stream');
|
||||
}
|
||||
|
||||
export const handle: Handle = async ({ event, resolve }) => {
|
||||
if (event.url.pathname.startsWith('/api/')) {
|
||||
const target = `${BACKEND_URL}${event.url.pathname}${event.url.search}`;
|
||||
@@ -63,9 +80,14 @@ export const handle: Handle = async ({ event, resolve }) => {
|
||||
// AbortController times the upstream fetch out so a backend
|
||||
// wedged on a slow DB query doesn't keep the browser request
|
||||
// hanging forever. The `signal` is also wired into the
|
||||
// RequestInit so the body stream is cancelled cleanly.
|
||||
// RequestInit so the body stream is cancelled cleanly. For
|
||||
// SSE streams the timer is suppressed so a long-lived stream
|
||||
// isn't torn down on the 5-minute mark — see T1 in the audit.
|
||||
const bypassTimeout = shouldBypassProxyTimeout(event.request.headers);
|
||||
const ctrl = new AbortController();
|
||||
const timeoutHandle = setTimeout(() => ctrl.abort(), PROXY_TIMEOUT_MS);
|
||||
const timeoutHandle = bypassTimeout
|
||||
? null
|
||||
: setTimeout(() => ctrl.abort(), PROXY_TIMEOUT_MS);
|
||||
|
||||
const init: RequestInit & { duplex?: 'half' } = {
|
||||
method: event.request.method,
|
||||
@@ -91,7 +113,7 @@ export const handle: Handle = async ({ event, resolve }) => {
|
||||
// the real cause. Emit the standard envelope with a
|
||||
// dedicated code instead.
|
||||
console.error('Proxy to backend failed:', e);
|
||||
clearTimeout(timeoutHandle);
|
||||
if (timeoutHandle) clearTimeout(timeoutHandle);
|
||||
return new Response(
|
||||
JSON.stringify({
|
||||
error: {
|
||||
@@ -106,7 +128,7 @@ export const handle: Handle = async ({ event, resolve }) => {
|
||||
);
|
||||
}
|
||||
|
||||
clearTimeout(timeoutHandle);
|
||||
if (timeoutHandle) clearTimeout(timeoutHandle);
|
||||
return new Response(upstream.body, {
|
||||
status: upstream.status,
|
||||
statusText: upstream.statusText,
|
||||
|
||||
Reference in New Issue
Block a user