feat(crawler): reconcile pass to enqueue mangas missing from the DB (0.85.0)
The interleaved metadata pass misses mangas to list drift (a title slips a pagination slot during the slow detail walk). Add a reconcile pass: a cheap, full, list-only walk (refs only, no detail visit, no early stop) that set-diffs the walked keys against manga_sources and enqueues the strictly- missing ones as SyncManga jobs. A set-diff is immune to drift — a manga only has to appear somewhere in the list. - Build the previously-dead SyncManga worker by extending RealChapterDispatcher to run the shared pipeline::process_manga_ref (fetch → upsert → cover → chapters), refactored out of run_metadata_pass so both paths stay in lockstep. The crawl worker now leases both sync_chapter_content and sync_manga (jobs::lease_kinds); both serialize on the single exclusive browser. - SyncManga payload carries url + title so the worker can rebuild the ref and the dead-jobs/history UI can label a missing manga that has no manga row yet. - "Missing" = strict NOT EXISTS (dropped rows count as present). Re-enqueue is skipped when a pending/running/dead SyncManga job already exists, so a gone manga (detail 404 → retries → dead) is left dead and not retried. - New POST /v1/admin/crawler/reconcile (fire-and-forget, shares manual_pass_lock) + "Reconcile missing" admin button + Reconciling status phase streamed over SSE. - dead-jobs/history queries surface payload title/url/key; tables fall back to them for sync_manga rows; both searches match the payload title. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -16,6 +16,7 @@
|
||||
getCrawlerStatus,
|
||||
crawlerStatusStreamUrl,
|
||||
runCrawlerPass,
|
||||
reconcileCrawler,
|
||||
restartCrawlerBrowser,
|
||||
updateCrawlerSession,
|
||||
clearCrawlerSessionExpired,
|
||||
@@ -308,6 +309,13 @@
|
||||
});
|
||||
}
|
||||
|
||||
async function onReconcile() {
|
||||
await withBusy('reconcile', async () => {
|
||||
await reconcileCrawler();
|
||||
notice = 'Reconcile started — walking the full list for missing mangas.';
|
||||
});
|
||||
}
|
||||
|
||||
async function onConfirmRestart() {
|
||||
restartModalOpen = false;
|
||||
await withBusy('restart browser', async () => {
|
||||
@@ -427,6 +435,7 @@
|
||||
{status}
|
||||
{busy}
|
||||
{onRunPass}
|
||||
{onReconcile}
|
||||
onOpenRestart={() => (restartModalOpen = true)}
|
||||
onOpenSession={() => {
|
||||
sessionModalOpen = true;
|
||||
|
||||
Reference in New Issue
Block a user