fabi/Mangalord
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(admin): crawler observability dashboard + reliability hardening (0.55.0)

Browse Source 
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.

Backend:

- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
  into status / control / dead_jobs / backlog modules. SSE stream
  composes in-memory status with DB-derived queue counts, memoizes
  the counts for 1s and debounces watch pokes for 250ms (~10x QPS
  reduction per subscriber). One-shot GET /admin/crawler shares the
  same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
  (409 Conflict on overlapping click); browser restart goes through
  the coordinated_restart gate (drain + relaunch + auto-clear of the
  sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
  validation, never logged, audit row carries SHA-256 fingerprint).
  Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
  scope=all requires confirm:true; DISTINCT ON dedup keeps the
  partial unique index from rejecting requeues for chapters with
  multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
  panics; last-writer-wins on cover so concurrent dispatches don't
  clobber each other's slot. Pure functions (should_stop /
  should_mark_clean_exit / should_abort_pass) with named regression
  tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
  per-job timeout, circuit breaker on consecutive failures, BrowserManager
  coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
  LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
  Pages stream through with peak memory ~one HTTP chunk + 64-byte
  sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
  and crawler_jobs_dead_idx, both ordered by updated_at DESC to
  match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
  on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
  admin_no_store_guard (Cache-Control: no-store on admin
  responses), audit rows carry per-scope target_id.

Frontend:

- /admin/crawler page decomposed into lib/components/crawler/
  (11 components: ProgressBar, SearchBar, CrawlerHero,
  CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
  MissingCoversTable, DeadJobsTable, RestartConfirmModal,
  RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
  orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
  pageshow (BFCache); after 5 consecutive errors probes the status
  endpoint so a 401 routes through the global on401Hook instead of
  infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
  AbortControllers; refresh after a control action only runs when
  the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
  sync_state locally (no full chapter-list refetch); proper
  aria-label. Requeue-all gets its own confirm modal; both confirm
  modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
  for Accept: text/event-stream; pure shouldBypassProxyTimeout
  helper covered by unit tests.

Config / docs:

- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
  CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
  CRAWLER_BROWSER_RESTART_THRESHOLD.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-06 18:49:56 +02:00
parent 679abae736
commit d6ac648ac9
54 changed files with 6076 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files

148
frontend/src/lib/components/crawler/CrawlerHero.svelte Normal file
View File

@@ -0,0 +1,148 @@
<script lang="ts">
import type { CrawlerStatus, CrawlerPhase } from '$lib/api/admin';
import ProgressBar from './ProgressBar.svelte';
let { status }: { status: CrawlerStatus } = $props();
function phaseLabel(p: CrawlerPhase | null): string {
if (!p) return 'Daemon disabled';
switch (p.state) {
case 'idle':
return p.next_fire
? `Idle — next pass ${new Date(p.next_fire).toLocaleString()}`
: 'Idle';
case 'walking_list':
return 'Walking source list';
case 'fetching_metadata':
return `Fetching metadata · ${p.index}/${p.total ?? '?'} · ${p.title}`;
case 'cover_backfill':
return `Backfilling covers · ${p.index + 1}/${p.total}`;
default: {
// Exhaustive default: if a new phase state ships
// without a label, TypeScript flags the missing case
// at compile time. Keeps the UI from showing
// `undefined` if the server gets ahead of the client.
const _exhaustive: never = p;
void _exhaustive;
return 'Unknown phase';
}
}
}
function phasePercent(p: CrawlerPhase | null): number | null {
if (p && p.state === 'fetching_metadata' && p.total && p.total > 0) {
return Math.min(100, (p.index / p.total) * 100);
}
return null;
}
function sessionPill(s: CrawlerStatus): { cls: string; text: string } {
if (s.daemon === 'disabled') return { cls: 'badge-not_downloaded', text: 'n/a' };
if (s.session.expired) return { cls: 'badge-in_progress', text: 'Expired' };
if (!s.session.configured) return { cls: 'badge-not_downloaded', text: 'Not set' };
return { cls: 'badge-synced', text: 'OK' };
}
function browserPill(s: CrawlerStatus): { cls: string; text: string } {
switch (s.browser) {
case 'healthy':
return { cls: 'badge-synced', text: 'Up' };
case 'draining':
case 'restarting':
return { cls: 'badge-in_progress', text: s.browser };
default:
return { cls: 'badge-not_downloaded', text: 'Down' };
}
}
</script>
<section class="hero" data-testid="crawler-hero">
<div class="pills">
<span class="pill"
>Daemon
<span
class={`badge ${status.daemon === 'running' ? 'badge-synced' : 'badge-not_downloaded'}`}
>{status.daemon}</span
></span
>
<span class="pill"
>Session
<span class={`badge ${sessionPill(status).cls}`}>{sessionPill(status).text}</span></span
>
<span class="pill"
>Browser
<span class={`badge ${browserPill(status).cls}`}>{browserPill(status).text}</span></span
>
</div>
<p class="phase" data-testid="crawler-phase">{phaseLabel(status.phase)}</p>
{#if phasePercent(status.phase) !== null}
<ProgressBar percent={phasePercent(status.phase) ?? 0} />
{/if}
{#if status.session.expired}
<p class="warn">
⚠ Chapter downloads paused — session expired. Metadata + list crawl continue.
</p>
{/if}
{#if status.current_cover}
<p class="cover" data-testid="current-cover">
🖼 Fetching cover: <strong>{status.current_cover.manga_title}</strong>
</p>
{/if}
<p class="lastpass">
Last pass:
{#if status.last_pass.at}
{new Date(status.last_pass.at).toLocaleString()} ·
{status.last_pass.discovered} seen · {status.last_pass.upserted} upserted ·
{status.last_pass.mangas_failed} failed
{:else}
— none yet this session
{/if}
</p>
</section>
<style>
.hero {
padding: var(--space-4);
border: 1px solid var(--border);
border-radius: var(--radius-md);
background: var(--surface);
margin-bottom: var(--space-4);
}
.pills {
display: flex;
gap: var(--space-4);
flex-wrap: wrap;
margin-bottom: var(--space-3);
}
.pill {
font-size: var(--font-sm);
color: var(--text-muted);
display: inline-flex;
align-items: center;
gap: var(--space-2);
}
.phase {
font-size: var(--font-lg);
font-weight: var(--weight-semibold);
margin: var(--space-2) 0;
}
.lastpass {
color: var(--text-muted);
font-size: var(--font-sm);
}
.warn {
color: #92400e;
background: #fef3c7;
border: 1px solid #fcd34d;
padding: var(--space-2) var(--space-3);
border-radius: var(--radius-md);
font-size: var(--font-sm);
}
.cover {
font-size: var(--font-sm);
}
</style>