feat(crawler): reconcile pass to enqueue mangas missing from the DB (0.85.0)

The interleaved metadata pass misses mangas to list drift (a title slips a
pagination slot during the slow detail walk). Add a reconcile pass: a cheap,
full, list-only walk (refs only, no detail visit, no early stop) that
set-diffs the walked keys against manga_sources and enqueues the strictly-
missing ones as SyncManga jobs. A set-diff is immune to drift — a manga only
has to appear somewhere in the list.

- Build the previously-dead SyncManga worker by extending RealChapterDispatcher
  to run the shared pipeline::process_manga_ref (fetch → upsert → cover →
  chapters), refactored out of run_metadata_pass so both paths stay in lockstep.
  The crawl worker now leases both sync_chapter_content and sync_manga
  (jobs::lease_kinds); both serialize on the single exclusive browser.
- SyncManga payload carries url + title so the worker can rebuild the ref and
  the dead-jobs/history UI can label a missing manga that has no manga row yet.
- "Missing" = strict NOT EXISTS (dropped rows count as present). Re-enqueue is
  skipped when a pending/running/dead SyncManga job already exists, so a gone
  manga (detail 404 → retries → dead) is left dead and not retried.
- New POST /v1/admin/crawler/reconcile (fire-and-forget, shares
  manual_pass_lock) + "Reconcile missing" admin button + Reconciling status
  phase streamed over SSE.
- dead-jobs/history queries surface payload title/url/key; tables fall back to
  them for sync_manga rows; both searches match the payload title.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-16 21:50:54 +02:00
parent 35664bccc7
commit dd300a150c
26 changed files with 1387 additions and 218 deletions

View File

@@ -71,7 +71,8 @@
await load();
}
/** A human "Manga · Ch N · pP" target, or the source key / em-dash. */
/** A human "Manga · Ch N · pP" target. Falls back to the payload title
* (sync_manga jobs with no manga row yet), then the source key. */
function target(r: CrawlerHistoryRow): string {
if (r.manga_title) {
let s = r.manga_title;
@@ -79,7 +80,7 @@
if (r.page_number != null) s += ` · p${r.page_number}`;
return s;
}
return r.source_key ?? '—';
return r.payload_title ?? r.source_key ?? '—';
}
function fmtAgo(iso: string): string {
@@ -172,7 +173,15 @@
<span class="badge state-{r.state}">{r.state}</span>
</td>
<td class="kind">{r.kind ?? '—'}</td>
<td>{target(r)}</td>
<td>
{#if !r.manga_title && r.payload_title && r.source_url}
<a href={r.source_url} target="_blank" rel="noreferrer noopener"
>{target(r)}</a
>
{:else}
{target(r)}
{/if}
</td>
<td class="num">{fmtDuration(r.duration_ms)}</td>
<td>{r.attempts}/{r.max_attempts}</td>
<td title={new Date(r.updated_at).toLocaleString()}