feat(admin): crawler observability dashboard + reliability hardening (0.55.0)

Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.

Backend:

- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
  into status / control / dead_jobs / backlog modules. SSE stream
  composes in-memory status with DB-derived queue counts, memoizes
  the counts for 1s and debounces watch pokes for 250ms (~10x QPS
  reduction per subscriber). One-shot GET /admin/crawler shares the
  same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
  (409 Conflict on overlapping click); browser restart goes through
  the coordinated_restart gate (drain + relaunch + auto-clear of the
  sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
  validation, never logged, audit row carries SHA-256 fingerprint).
  Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
  scope=all requires confirm:true; DISTINCT ON dedup keeps the
  partial unique index from rejecting requeues for chapters with
  multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
  panics; last-writer-wins on cover so concurrent dispatches don't
  clobber each other's slot. Pure functions (should_stop /
  should_mark_clean_exit / should_abort_pass) with named regression
  tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
  per-job timeout, circuit breaker on consecutive failures, BrowserManager
  coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
  LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
  Pages stream through with peak memory ~one HTTP chunk + 64-byte
  sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
  and crawler_jobs_dead_idx, both ordered by updated_at DESC to
  match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
  on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
  admin_no_store_guard (Cache-Control: no-store on admin
  responses), audit rows carry per-scope target_id.

Frontend:

- /admin/crawler page decomposed into lib/components/crawler/
  (11 components: ProgressBar, SearchBar, CrawlerHero,
  CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
  MissingCoversTable, DeadJobsTable, RestartConfirmModal,
  RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
  orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
  pageshow (BFCache); after 5 consecutive errors probes the status
  endpoint so a 401 routes through the global on401Hook instead of
  infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
  AbortControllers; refresh after a control action only runs when
  the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
  sync_state locally (no full chapter-list refetch); proper
  aria-label. Requeue-all gets its own confirm modal; both confirm
  modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
  for Accept: text/event-stream; pure shouldBypassProxyTimeout
  helper covered by unit tests.

Config / docs:

- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
  CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
  CRAWLER_BROWSER_RESTART_THRESHOLD.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-06 18:49:56 +02:00
parent 679abae736
commit d6ac648ac9
54 changed files with 6076 additions and 137 deletions

View File

@@ -0,0 +1,112 @@
<script lang="ts">
import type { CrawlerStatus } from '$lib/api/admin';
import ProgressBar from './ProgressBar.svelte';
let { status }: { status: CrawlerStatus } = $props();
function chapterPercent(c: { pages_done: number; pages_total: number | null }): number | null {
return c.pages_total && c.pages_total > 0
? Math.min(100, (c.pages_done / c.pages_total) * 100)
: null;
}
</script>
<section class="grid2">
<article>
<h2>Queue</h2>
<dl>
<dt>Pending</dt>
<dd>{status.queue.pending}</dd>
<dt>Running</dt>
<dd>{status.queue.running}</dd>
<dt>Dead</dt>
<dd>{status.queue.dead}</dd>
<dt>Covers queued</dt>
<dd>{status.covers_queued}</dd>
</dl>
</article>
<article>
<h2>Active chapters ({status.active_chapters.length}/{status.worker_count})</h2>
{#if status.active_chapters.length === 0}
<p class="muted">idle — no chapters downloading</p>
{:else}
<table class="active">
<tbody>
{#each status.active_chapters as c (c.chapter_id)}
<tr>
<td>{c.manga_title} · ch.{c.chapter_number}</td>
<td class="pagecount" data-testid="active-pages">
{c.pages_done}/{c.pages_total ?? '?'}
</td>
<td class="pagebar">
{#if chapterPercent(c) !== null}
<ProgressBar percent={chapterPercent(c) ?? 0} />
{/if}
</td>
</tr>
{/each}
</tbody>
</table>
{/if}
</article>
</section>
<style>
h2 {
margin: 0 0 var(--space-3) 0;
font-size: var(--font-sm);
color: var(--text-muted);
text-transform: uppercase;
letter-spacing: 0.04em;
}
.grid2 {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(16rem, 1fr));
gap: var(--space-3);
margin-bottom: var(--space-4);
}
article {
padding: var(--space-3);
border: 1px solid var(--border);
border-radius: var(--radius-md);
background: var(--surface);
}
dl {
display: grid;
grid-template-columns: max-content 1fr;
gap: var(--space-1) var(--space-3);
margin: 0;
font-size: var(--font-sm);
}
dt {
color: var(--text-muted);
}
dd {
margin: 0;
font-family: var(--font-mono, monospace);
}
table {
width: 100%;
border-collapse: collapse;
}
td {
padding: var(--space-2);
text-align: left;
border-bottom: 1px solid var(--border);
font-size: var(--font-sm);
}
.muted {
color: var(--text-muted);
}
.pagecount {
font-family: var(--font-mono, monospace);
font-size: var(--font-xs);
white-space: nowrap;
}
.pagebar {
width: 8rem;
}
table.active td {
vertical-align: middle;
}
</style>

View File

@@ -0,0 +1,101 @@
<script lang="ts">
import Pager from '$lib/components/Pager.svelte';
import type { ActiveJob } from '$lib/api/admin';
import SearchBar from './SearchBar.svelte';
let {
jobs,
total,
page,
totalPages,
search = $bindable(''),
onSearch,
onPageChange
}: {
jobs: ActiveJob[];
total: number;
page: number;
totalPages: number;
search?: string;
onSearch: () => void;
onPageChange: (p: number) => void;
} = $props();
</script>
<section class="backlog">
<div class="deadhead">
<h2>Queued chapters ({total})</h2>
<div class="deadtools">
<SearchBar bind:value={search} {onSearch} />
</div>
</div>
{#if jobs.length === 0}
<p class="muted">No chapters queued.</p>
{:else}
<table class="dead">
<thead>
<tr>
<th>Manga / Chapter</th>
<th>State</th>
<th>Att.</th>
</tr>
</thead>
<tbody>
{#each jobs as j (j.id)}
<tr>
<td>
{j.manga_title ?? '(unknown)'}
{#if j.chapter_number != null}· ch.{j.chapter_number}{/if}
</td>
<td>
<span
class={`badge ${j.state === 'running' ? 'badge-downloading' : 'badge-not_downloaded'}`}
>{j.state}</span
>
</td>
<td>{j.attempts}/{j.max_attempts}</td>
</tr>
{/each}
</tbody>
</table>
<Pager {page} {totalPages} onChange={onPageChange} />
{/if}
</section>
<style>
h2 {
margin: 0 0 var(--space-3) 0;
font-size: var(--font-sm);
color: var(--text-muted);
text-transform: uppercase;
letter-spacing: 0.04em;
}
table {
width: 100%;
border-collapse: collapse;
}
th,
td {
padding: var(--space-2);
text-align: left;
border-bottom: 1px solid var(--border);
font-size: var(--font-sm);
}
.deadhead {
display: flex;
justify-content: space-between;
align-items: center;
gap: var(--space-3);
flex-wrap: wrap;
}
.deadtools {
display: flex;
gap: var(--space-2);
}
.muted {
color: var(--text-muted);
}
.backlog {
margin-top: var(--space-4);
}
</style>

View File

@@ -0,0 +1,43 @@
<script lang="ts">
import type { CrawlerStatus } from '$lib/api/admin';
let {
status,
busy,
onRunPass,
onOpenRestart,
onOpenSession,
onClearExpired
}: {
status: CrawlerStatus;
busy: boolean;
onRunPass: () => void;
onOpenRestart: () => void;
onOpenSession: () => void;
onClearExpired: () => void;
} = $props();
</script>
<section class="controls">
<button onclick={onRunPass} disabled={busy || status.daemon !== 'running'}
>Run metadata pass now</button
>
<button onclick={onOpenRestart} disabled={busy || status.daemon !== 'running'}
>Restart browser</button
>
<button onclick={onOpenSession} disabled={busy || status.daemon !== 'running'}
>Manage session…</button
>
{#if status.session.expired}
<button onclick={onClearExpired} disabled={busy}>Clear expired flag</button>
{/if}
</section>
<style>
.controls {
display: flex;
gap: var(--space-2);
flex-wrap: wrap;
margin-bottom: var(--space-4);
}
</style>

View File

@@ -0,0 +1,148 @@
<script lang="ts">
import type { CrawlerStatus, CrawlerPhase } from '$lib/api/admin';
import ProgressBar from './ProgressBar.svelte';
let { status }: { status: CrawlerStatus } = $props();
function phaseLabel(p: CrawlerPhase | null): string {
if (!p) return 'Daemon disabled';
switch (p.state) {
case 'idle':
return p.next_fire
? `Idle — next pass ${new Date(p.next_fire).toLocaleString()}`
: 'Idle';
case 'walking_list':
return 'Walking source list';
case 'fetching_metadata':
return `Fetching metadata · ${p.index}/${p.total ?? '?'} · ${p.title}`;
case 'cover_backfill':
return `Backfilling covers · ${p.index + 1}/${p.total}`;
default: {
// Exhaustive default: if a new phase state ships
// without a label, TypeScript flags the missing case
// at compile time. Keeps the UI from showing
// `undefined` if the server gets ahead of the client.
const _exhaustive: never = p;
void _exhaustive;
return 'Unknown phase';
}
}
}
function phasePercent(p: CrawlerPhase | null): number | null {
if (p && p.state === 'fetching_metadata' && p.total && p.total > 0) {
return Math.min(100, (p.index / p.total) * 100);
}
return null;
}
function sessionPill(s: CrawlerStatus): { cls: string; text: string } {
if (s.daemon === 'disabled') return { cls: 'badge-not_downloaded', text: 'n/a' };
if (s.session.expired) return { cls: 'badge-in_progress', text: 'Expired' };
if (!s.session.configured) return { cls: 'badge-not_downloaded', text: 'Not set' };
return { cls: 'badge-synced', text: 'OK' };
}
function browserPill(s: CrawlerStatus): { cls: string; text: string } {
switch (s.browser) {
case 'healthy':
return { cls: 'badge-synced', text: 'Up' };
case 'draining':
case 'restarting':
return { cls: 'badge-in_progress', text: s.browser };
default:
return { cls: 'badge-not_downloaded', text: 'Down' };
}
}
</script>
<section class="hero" data-testid="crawler-hero">
<div class="pills">
<span class="pill"
>Daemon
<span
class={`badge ${status.daemon === 'running' ? 'badge-synced' : 'badge-not_downloaded'}`}
>{status.daemon}</span
></span
>
<span class="pill"
>Session
<span class={`badge ${sessionPill(status).cls}`}>{sessionPill(status).text}</span></span
>
<span class="pill"
>Browser
<span class={`badge ${browserPill(status).cls}`}>{browserPill(status).text}</span></span
>
</div>
<p class="phase" data-testid="crawler-phase">{phaseLabel(status.phase)}</p>
{#if phasePercent(status.phase) !== null}
<ProgressBar percent={phasePercent(status.phase) ?? 0} />
{/if}
{#if status.session.expired}
<p class="warn">
⚠ Chapter downloads paused — session expired. Metadata + list crawl continue.
</p>
{/if}
{#if status.current_cover}
<p class="cover" data-testid="current-cover">
🖼 Fetching cover: <strong>{status.current_cover.manga_title}</strong>
</p>
{/if}
<p class="lastpass">
Last pass:
{#if status.last_pass.at}
{new Date(status.last_pass.at).toLocaleString()} ·
{status.last_pass.discovered} seen · {status.last_pass.upserted} upserted ·
{status.last_pass.mangas_failed} failed
{:else}
— none yet this session
{/if}
</p>
</section>
<style>
.hero {
padding: var(--space-4);
border: 1px solid var(--border);
border-radius: var(--radius-md);
background: var(--surface);
margin-bottom: var(--space-4);
}
.pills {
display: flex;
gap: var(--space-4);
flex-wrap: wrap;
margin-bottom: var(--space-3);
}
.pill {
font-size: var(--font-sm);
color: var(--text-muted);
display: inline-flex;
align-items: center;
gap: var(--space-2);
}
.phase {
font-size: var(--font-lg);
font-weight: var(--weight-semibold);
margin: var(--space-2) 0;
}
.lastpass {
color: var(--text-muted);
font-size: var(--font-sm);
}
.warn {
color: #92400e;
background: #fef3c7;
border: 1px solid #fcd34d;
padding: var(--space-2) var(--space-3);
border-radius: var(--radius-md);
font-size: var(--font-sm);
}
.cover {
font-size: var(--font-sm);
}
</style>

View File

@@ -0,0 +1,132 @@
<script lang="ts">
import Pager from '$lib/components/Pager.svelte';
import type { DeadJob, RequeueScope } from '$lib/api/admin';
import SearchBar from './SearchBar.svelte';
let {
jobs,
total,
page,
totalPages,
search = $bindable(''),
busy,
onSearch,
onPageChange,
onRequeue,
onRequeueAll
}: {
jobs: DeadJob[];
total: number;
page: number;
totalPages: number;
search?: string;
busy: boolean;
onSearch: () => void;
onPageChange: (p: number) => void;
onRequeue: (scope: RequeueScope) => void;
onRequeueAll: () => void;
} = $props();
</script>
<section class="deadjobs">
<div class="deadhead">
<h2>Dead jobs ({total})</h2>
<div class="deadtools">
<SearchBar bind:value={search} {onSearch} />
<button onclick={onRequeueAll} disabled={busy || total === 0}
>Requeue all ({total})</button
>
</div>
</div>
{#if jobs.length === 0}
<p class="muted">No dead jobs 🎉</p>
{:else}
<table class="dead">
<thead>
<tr>
<th>Manga / Chapter</th>
<th>Att.</th>
<th>Failed</th>
<th>Last error</th>
<th class="actions">Action</th>
</tr>
</thead>
<tbody>
{#each jobs as j (j.id)}
<tr>
<td>
{j.manga_title ?? '(unknown)'}
{#if j.chapter_number != null}· ch.{j.chapter_number}{/if}
</td>
<td>{j.attempts}/{j.max_attempts}</td>
<td>{new Date(j.updated_at).toLocaleDateString()}</td>
<td class="err" title={j.last_error ?? ''}>{j.last_error ?? '—'}</td>
<td class="actions">
<button
onclick={() => onRequeue({ scope: 'job', job_id: j.id })}
disabled={busy}>Requeue</button
>
{#if j.manga_id}
<button
class="secondary"
onclick={() => onRequeue({ scope: 'manga', manga_id: j.manga_id! })}
disabled={busy}>Manga</button
>
{/if}
</td>
</tr>
{/each}
</tbody>
</table>
<Pager {page} {totalPages} onChange={onPageChange} />
{/if}
</section>
<style>
h2 {
margin: 0 0 var(--space-3) 0;
font-size: var(--font-sm);
color: var(--text-muted);
text-transform: uppercase;
letter-spacing: 0.04em;
}
table {
width: 100%;
border-collapse: collapse;
}
th,
td {
padding: var(--space-2);
text-align: left;
border-bottom: 1px solid var(--border);
font-size: var(--font-sm);
}
.actions {
text-align: right;
}
.err {
max-width: 22rem;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
color: var(--text-muted);
}
.deadhead {
display: flex;
justify-content: space-between;
align-items: center;
gap: var(--space-3);
flex-wrap: wrap;
}
.deadtools {
display: flex;
gap: var(--space-2);
}
button.secondary {
background: var(--surface-elevated);
}
.muted {
color: var(--text-muted);
}
</style>

View File

@@ -0,0 +1,85 @@
<script lang="ts">
import Pager from '$lib/components/Pager.svelte';
import type { MissingCover } from '$lib/api/admin';
import SearchBar from './SearchBar.svelte';
let {
covers,
total,
page,
totalPages,
search = $bindable(''),
onSearch,
onPageChange
}: {
covers: MissingCover[];
total: number;
page: number;
totalPages: number;
search?: string;
onSearch: () => void;
onPageChange: (p: number) => void;
} = $props();
</script>
<section class="backlog">
<div class="deadhead">
<h2>Queued covers ({total})</h2>
<div class="deadtools">
<SearchBar bind:value={search} {onSearch} />
</div>
</div>
{#if covers.length === 0}
<p class="muted">No covers queued 🎉</p>
{:else}
<table class="dead">
<thead>
<tr><th>Manga</th></tr>
</thead>
<tbody>
{#each covers as c (c.manga_id)}
<tr><td>{c.manga_title}</td></tr>
{/each}
</tbody>
</table>
<Pager {page} {totalPages} onChange={onPageChange} />
{/if}
</section>
<style>
h2 {
margin: 0 0 var(--space-3) 0;
font-size: var(--font-sm);
color: var(--text-muted);
text-transform: uppercase;
letter-spacing: 0.04em;
}
table {
width: 100%;
border-collapse: collapse;
}
th,
td {
padding: var(--space-2);
text-align: left;
border-bottom: 1px solid var(--border);
font-size: var(--font-sm);
}
.deadhead {
display: flex;
justify-content: space-between;
align-items: center;
gap: var(--space-3);
flex-wrap: wrap;
}
.deadtools {
display: flex;
gap: var(--space-2);
}
.muted {
color: var(--text-muted);
}
.backlog {
margin-top: var(--space-4);
}
</style>

View File

@@ -0,0 +1,35 @@
<script lang="ts">
// Tiny percent bar reused by the phase progress + per-chapter page
// progress in the admin crawler dashboard. Was an inline `{#snippet}`
// before; extracted so the per-component tables can use it too.
let { percent }: { percent: number } = $props();
</script>
<div class="bar" role="progressbar" aria-valuenow={percent} aria-valuemin="0" aria-valuemax="100">
<div class="fill" style:width="{Math.min(100, Math.max(0, percent))}%"></div>
<span class="label">{percent.toFixed(0)}%</span>
</div>
<style>
.bar {
position: relative;
background: var(--surface-elevated);
border-radius: var(--radius-sm, 4px);
height: 1.5rem;
margin: var(--space-2) 0;
overflow: hidden;
}
.fill {
height: 100%;
background: #22c55e;
transition: width 0.3s ease;
}
.label {
position: absolute;
top: 50%;
left: 50%;
transform: translate(-50%, -50%);
font-size: var(--font-xs);
font-weight: var(--weight-semibold);
}
</style>

View File

@@ -0,0 +1,32 @@
<script lang="ts">
import Modal from '$lib/components/Modal.svelte';
let {
open,
total,
busy,
onCancel,
onConfirm
}: {
open: boolean;
total: number;
busy: boolean;
onCancel: () => void;
onConfirm: () => void;
} = $props();
</script>
<Modal {open} title="Requeue all dead jobs" onClose={onCancel} size="sm">
{#snippet children()}
<p>
This will flip every dead job ({total}) back to <code>pending</code>.
Chapters with a live job are skipped automatically; the rest will be
picked up by the next chapter worker.
</p>
{/snippet}
{#snippet footer()}
<!-- svelte-ignore a11y_autofocus -->
<button autofocus onclick={onCancel}>Cancel</button>
<button class="primary" onclick={onConfirm} disabled={busy}>Requeue {total}</button>
{/snippet}
</Modal>

View File

@@ -0,0 +1,42 @@
<script lang="ts">
import Modal from '$lib/components/Modal.svelte';
let {
open,
busy,
onCancel,
onConfirm
}: {
open: boolean;
busy: boolean;
onCancel: () => void;
onConfirm: () => void;
} = $props();
</script>
<Modal {open} title="Restart browser" onClose={onCancel} size="sm">
{#snippet children()}
<p>This relaunches Chromium and re-injects the session cookie.</p>
<ul class="coord">
<li>In-flight jobs are allowed to finish (bounded), then forced.</li>
<li>New jobs pause until the relaunch completes.</li>
<li>The metadata pass yields at its next checkpoint.</li>
</ul>
{/snippet}
{#snippet footer()}
<!-- Cancel autofocused so an accidental Enter on the modal
dismisses rather than triggering the destructive action. -->
<!-- svelte-ignore a11y_autofocus -->
<button autofocus onclick={onCancel}>Cancel</button>
<button class="primary" onclick={onConfirm} disabled={busy}>Restart</button>
{/snippet}
</Modal>
<style>
.coord {
margin: var(--space-2) 0;
padding-left: var(--space-4);
font-size: var(--font-sm);
color: var(--text-muted);
}
</style>

View File

@@ -0,0 +1,22 @@
<script lang="ts">
// Search input + button used by all three backlog tables on the
// crawler dashboard. The three sites had identical markup before;
// the audit (F2) flagged the duplication. `bindable` lets the
// parent keep `value` in $state and read it after Enter / click.
let {
value = $bindable(''),
placeholder = 'Search manga…',
onSearch
}: {
value?: string;
placeholder?: string;
onSearch: () => void;
} = $props();
</script>
<input
{placeholder}
bind:value
onkeydown={(e) => e.key === 'Enter' && onSearch()}
/>
<button onclick={onSearch}>Search</button>

View File

@@ -0,0 +1,49 @@
<script lang="ts">
import Modal from '$lib/components/Modal.svelte';
let {
open,
busy,
phpsessid = $bindable(''),
result,
onCancel,
onSave
}: {
open: boolean;
busy: boolean;
phpsessid?: string;
result: string | null;
onCancel: () => void;
onSave: () => void;
} = $props();
</script>
<Modal {open} title="Manage crawler session" onClose={onCancel} size="md">
{#snippet children()}
<label for="phpsessid">PHPSESSID</label>
<input id="phpsessid" type="password" bind:value={phpsessid} autocomplete="off" />
<p class="hint">
Saving rewrites the cookie everywhere, persists it, restarts the browser, and re-probes.
</p>
{#if result}
<p class="sessionresult">{result}</p>
{/if}
{/snippet}
{#snippet footer()}
<button onclick={onCancel}>Cancel</button>
<button class="primary" onclick={onSave} disabled={busy || phpsessid.trim() === ''}
>Save &amp; validate</button
>
{/snippet}
</Modal>
<style>
.hint {
color: var(--text-muted);
font-size: var(--font-sm);
}
.sessionresult {
margin-top: var(--space-2);
font-size: var(--font-sm);
}
</style>