feat(crawler): reconcile pass to enqueue mangas missing from the DB (0.85.0)
The interleaved metadata pass misses mangas to list drift (a title slips a pagination slot during the slow detail walk). Add a reconcile pass: a cheap, full, list-only walk (refs only, no detail visit, no early stop) that set-diffs the walked keys against manga_sources and enqueues the strictly- missing ones as SyncManga jobs. A set-diff is immune to drift — a manga only has to appear somewhere in the list. - Build the previously-dead SyncManga worker by extending RealChapterDispatcher to run the shared pipeline::process_manga_ref (fetch → upsert → cover → chapters), refactored out of run_metadata_pass so both paths stay in lockstep. The crawl worker now leases both sync_chapter_content and sync_manga (jobs::lease_kinds); both serialize on the single exclusive browser. - SyncManga payload carries url + title so the worker can rebuild the ref and the dead-jobs/history UI can label a missing manga that has no manga row yet. - "Missing" = strict NOT EXISTS (dropped rows count as present). Re-enqueue is skipped when a pending/running/dead SyncManga job already exists, so a gone manga (detail 404 → retries → dead) is left dead and not retried. - New POST /v1/admin/crawler/reconcile (fire-and-forget, shares manual_pass_lock) + "Reconcile missing" admin button + Reconciling status phase streamed over SSE. - dead-jobs/history queries surface payload title/url/key; tables fall back to them for sync_manga rows; both searches match the payload title. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -71,7 +71,8 @@
|
||||
await load();
|
||||
}
|
||||
|
||||
/** A human "Manga · Ch N · pP" target, or the source key / em-dash. */
|
||||
/** A human "Manga · Ch N · pP" target. Falls back to the payload title
|
||||
* (sync_manga jobs with no manga row yet), then the source key. */
|
||||
function target(r: CrawlerHistoryRow): string {
|
||||
if (r.manga_title) {
|
||||
let s = r.manga_title;
|
||||
@@ -79,7 +80,7 @@
|
||||
if (r.page_number != null) s += ` · p${r.page_number}`;
|
||||
return s;
|
||||
}
|
||||
return r.source_key ?? '—';
|
||||
return r.payload_title ?? r.source_key ?? '—';
|
||||
}
|
||||
|
||||
function fmtAgo(iso: string): string {
|
||||
@@ -172,7 +173,15 @@
|
||||
<span class="badge state-{r.state}">{r.state}</span>
|
||||
</td>
|
||||
<td class="kind">{r.kind ?? '—'}</td>
|
||||
<td>{target(r)}</td>
|
||||
<td>
|
||||
{#if !r.manga_title && r.payload_title && r.source_url}
|
||||
<a href={r.source_url} target="_blank" rel="noreferrer noopener"
|
||||
>{target(r)}</a
|
||||
>
|
||||
{:else}
|
||||
{target(r)}
|
||||
{/if}
|
||||
</td>
|
||||
<td class="num">{fmtDuration(r.duration_ms)}</td>
|
||||
<td>{r.attempts}/{r.max_attempts}</td>
|
||||
<td title={new Date(r.updated_at).toLocaleString()}
|
||||
|
||||
Reference in New Issue
Block a user