fabi/Mangalord
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(crawler): reconcile pass to enqueue mangas missing from the DB (0.85.0)

Browse Source 
The interleaved metadata pass misses mangas to list drift (a title slips a
pagination slot during the slow detail walk). Add a reconcile pass: a cheap,
full, list-only walk (refs only, no detail visit, no early stop) that
set-diffs the walked keys against manga_sources and enqueues the strictly-
missing ones as SyncManga jobs. A set-diff is immune to drift — a manga only
has to appear somewhere in the list.

- Build the previously-dead SyncManga worker by extending RealChapterDispatcher
  to run the shared pipeline::process_manga_ref (fetch → upsert → cover →
  chapters), refactored out of run_metadata_pass so both paths stay in lockstep.
  The crawl worker now leases both sync_chapter_content and sync_manga
  (jobs::lease_kinds); both serialize on the single exclusive browser.
- SyncManga payload carries url + title so the worker can rebuild the ref and
  the dead-jobs/history UI can label a missing manga that has no manga row yet.
- "Missing" = strict NOT EXISTS (dropped rows count as present). Re-enqueue is
  skipped when a pending/running/dead SyncManga job already exists, so a gone
  manga (detail 404 → retries → dead) is left dead and not retried.
- New POST /v1/admin/crawler/reconcile (fire-and-forget, shares
  manual_pass_lock) + "Reconcile missing" admin button + Reconciling status
  phase streamed over SSE.
- dead-jobs/history queries surface payload title/url/key; tables fall back to
  them for sync_manga rows; both searches match the payload title.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-16 21:50:54 +02:00
parent 35664bccc7
commit dd300a150c
26 changed files with 1387 additions and 218 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
frontend/src/lib/api/admin.ts
View File

@@ -332,7 +332,8 @@ export type CrawlerPhase =
| { state: 'idle'; next_fire: string | null }
| { state: 'walking_list' }
| { state: 'fetching_metadata'; index: number; total: number | null; title: string }
| { state: 'cover_backfill'; index: number; total: number };
| { state: 'cover_backfill'; index: number; total: number }
| { state: 'reconciling'; walked: number; enqueued: number };
/** A chapter being crawled right now, with a live page count. */
export type ActiveChapter = {
@@ -382,6 +383,12 @@ export async function runCrawlerPass(): Promise<{ started: boolean }> {
return request('/v1/admin/crawler/run', { method: 'POST' });
}
/** POST /v1/admin/crawler/reconcile — full list-only walk that enqueues
* mangas missing from the DB as sync_manga jobs. */
export async function reconcileCrawler(): Promise<{ started: boolean }> {
return request('/v1/admin/crawler/reconcile', { method: 'POST' });
}
/** POST /v1/admin/crawler/browser/restart — coordinated Chromium restart. */
export async function restartCrawlerBrowser(): Promise<{ ok: boolean; error: string | null }> {
return request('/v1/admin/crawler/browser/restart', { method: 'POST' });
@@ -410,6 +417,12 @@ export type DeadJob = {
manga_id: string | null;
manga_title: string | null;
chapter_number: number | null;
/** Payload title (fallback label for sync_manga jobs with no manga row). */
payload_title: string | null;
/** Source detail URL from the payload (sync_manga). */
source_url: string | null;
/** Source-native key from the payload (sync_manga). */
source_key: string | null;
attempts: number;
max_attempts: number;
last_error: string | null;
@@ -513,6 +526,10 @@ export type CrawlerHistoryRow = {
chapter_number: number | null;
page_number: number | null;
source_key: string | null;
/** Payload title (fallback label for sync_manga jobs with no manga row). */
payload_title: string | null;
/** Source detail URL from the payload (sync_manga). */
source_url: string | null;
attempts: number;
max_attempts: number;
last_error: string | null;