feat(crawler): reconcile pass to enqueue mangas missing from the DB (0.85.0)
The interleaved metadata pass misses mangas to list drift (a title slips a pagination slot during the slow detail walk). Add a reconcile pass: a cheap, full, list-only walk (refs only, no detail visit, no early stop) that set-diffs the walked keys against manga_sources and enqueues the strictly- missing ones as SyncManga jobs. A set-diff is immune to drift — a manga only has to appear somewhere in the list. - Build the previously-dead SyncManga worker by extending RealChapterDispatcher to run the shared pipeline::process_manga_ref (fetch → upsert → cover → chapters), refactored out of run_metadata_pass so both paths stay in lockstep. The crawl worker now leases both sync_chapter_content and sync_manga (jobs::lease_kinds); both serialize on the single exclusive browser. - SyncManga payload carries url + title so the worker can rebuild the ref and the dead-jobs/history UI can label a missing manga that has no manga row yet. - "Missing" = strict NOT EXISTS (dropped rows count as present). Re-enqueue is skipped when a pending/running/dead SyncManga job already exists, so a gone manga (detail 404 → retries → dead) is left dead and not retried. - New POST /v1/admin/crawler/reconcile (fire-and-forget, shares manual_pass_lock) + "Reconcile missing" admin button + Reconciling status phase streamed over SSE. - dead-jobs/history queries surface payload title/url/key; tables fall back to them for sync_manga rows; both searches match the payload title. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -24,6 +24,7 @@ use super::require_crawler;
|
||||
pub(super) fn routes() -> Router<AppState> {
|
||||
Router::new()
|
||||
.route("/admin/crawler/run", post(run_now))
|
||||
.route("/admin/crawler/reconcile", post(reconcile_now))
|
||||
.route("/admin/crawler/browser/restart", post(restart_browser))
|
||||
.route("/admin/crawler/session", post(update_session))
|
||||
.route(
|
||||
@@ -71,6 +72,45 @@ async fn run_now(
|
||||
Ok(Json(RunResponse { started: true }))
|
||||
}
|
||||
|
||||
async fn reconcile_now(
|
||||
State(state): State<AppState>,
|
||||
admin: RequireAdmin,
|
||||
) -> AppResult<Json<RunResponse>> {
|
||||
let c = require_crawler(&state)?;
|
||||
let rp = c.reconcile_pass.as_ref().ok_or_else(|| {
|
||||
AppError::ServiceUnavailable("no source configured (CRAWLER_START_URL unset)".into())
|
||||
})?;
|
||||
// Share `manual_pass_lock` with the metadata pass: reconcile's list walk
|
||||
// and a metadata pass both contend for the single browser lease, so they
|
||||
// must not run concurrently. A click while either is in flight gets 409.
|
||||
let pass_guard = c
|
||||
.manual_pass_lock
|
||||
.clone()
|
||||
.try_lock_owned()
|
||||
.map_err(|_| AppError::Conflict("a metadata or reconcile pass is already running".into()))?;
|
||||
let rp = std::sync::Arc::clone(rp);
|
||||
// Fire-and-forget: the full list walk runs for minutes; progress streams
|
||||
// over SSE (the Reconciling phase). The guard moves into the task so the
|
||||
// lock releases only when the walk + enqueue finish.
|
||||
tokio::spawn(async move {
|
||||
let _pass_guard = pass_guard;
|
||||
match rp.run().await {
|
||||
Ok(stats) => tracing::info!(?stats, "manual reconcile pass complete"),
|
||||
Err(e) => tracing::warn!(error = ?e, "manual reconcile pass failed"),
|
||||
}
|
||||
});
|
||||
repo::admin_audit::insert(
|
||||
&state.db,
|
||||
admin.0.id,
|
||||
"crawler_reconcile",
|
||||
"crawler",
|
||||
None,
|
||||
json!({}),
|
||||
)
|
||||
.await?;
|
||||
Ok(Json(RunResponse { started: true }))
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
struct RestartResponse {
|
||||
ok: bool,
|
||||
|
||||
Reference in New Issue
Block a user