feat(crawler): reconcile pass to enqueue mangas missing from the DB (0.85.0)

The interleaved metadata pass misses mangas to list drift (a title slips a
pagination slot during the slow detail walk). Add a reconcile pass: a cheap,
full, list-only walk (refs only, no detail visit, no early stop) that
set-diffs the walked keys against manga_sources and enqueues the strictly-
missing ones as SyncManga jobs. A set-diff is immune to drift — a manga only
has to appear somewhere in the list.

- Build the previously-dead SyncManga worker by extending RealChapterDispatcher
  to run the shared pipeline::process_manga_ref (fetch → upsert → cover →
  chapters), refactored out of run_metadata_pass so both paths stay in lockstep.
  The crawl worker now leases both sync_chapter_content and sync_manga
  (jobs::lease_kinds); both serialize on the single exclusive browser.
- SyncManga payload carries url + title so the worker can rebuild the ref and
  the dead-jobs/history UI can label a missing manga that has no manga row yet.
- "Missing" = strict NOT EXISTS (dropped rows count as present). Re-enqueue is
  skipped when a pending/running/dead SyncManga job already exists, so a gone
  manga (detail 404 → retries → dead) is left dead and not retried.
- New POST /v1/admin/crawler/reconcile (fire-and-forget, shares
  manual_pass_lock) + "Reconcile missing" admin button + Reconciling status
  phase streamed over SSE.
- dead-jobs/history queries surface payload title/url/key; tables fall back to
  them for sync_manga rows; both searches match the payload title.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-16 21:50:54 +02:00
parent 35664bccc7
commit dd300a150c
26 changed files with 1387 additions and 218 deletions

View File

@@ -24,6 +24,7 @@ use super::require_crawler;
pub(super) fn routes() -> Router<AppState> {
Router::new()
.route("/admin/crawler/run", post(run_now))
.route("/admin/crawler/reconcile", post(reconcile_now))
.route("/admin/crawler/browser/restart", post(restart_browser))
.route("/admin/crawler/session", post(update_session))
.route(
@@ -71,6 +72,45 @@ async fn run_now(
Ok(Json(RunResponse { started: true }))
}
async fn reconcile_now(
State(state): State<AppState>,
admin: RequireAdmin,
) -> AppResult<Json<RunResponse>> {
let c = require_crawler(&state)?;
let rp = c.reconcile_pass.as_ref().ok_or_else(|| {
AppError::ServiceUnavailable("no source configured (CRAWLER_START_URL unset)".into())
})?;
// Share `manual_pass_lock` with the metadata pass: reconcile's list walk
// and a metadata pass both contend for the single browser lease, so they
// must not run concurrently. A click while either is in flight gets 409.
let pass_guard = c
.manual_pass_lock
.clone()
.try_lock_owned()
.map_err(|_| AppError::Conflict("a metadata or reconcile pass is already running".into()))?;
let rp = std::sync::Arc::clone(rp);
// Fire-and-forget: the full list walk runs for minutes; progress streams
// over SSE (the Reconciling phase). The guard moves into the task so the
// lock releases only when the walk + enqueue finish.
tokio::spawn(async move {
let _pass_guard = pass_guard;
match rp.run().await {
Ok(stats) => tracing::info!(?stats, "manual reconcile pass complete"),
Err(e) => tracing::warn!(error = ?e, "manual reconcile pass failed"),
}
});
repo::admin_audit::insert(
&state.db,
admin.0.id,
"crawler_reconcile",
"crawler",
None,
json!({}),
)
.await?;
Ok(Json(RunResponse { started: true }))
}
#[derive(Debug, Serialize)]
struct RestartResponse {
ok: bool,