fix(crawler): harden crash & shutdown job recovery
Three failure-mode fixes surfaced by a crawler recovery audit: - Graceful shutdown mid-dispatch now releases the in-flight job back to pending without burning a retry attempt. process_lease wraps the dispatch in a biased tokio::select! cancel arm that aborts the heartbeat and calls jobs::release; previously a mid-job SIGTERM left the row 'running' until lease expiry and cost one of max_attempts. - Boot-time jobs::reclaim_orphaned resets 'running' jobs with an expired lease back to pending (attempt refunded), run once at daemon startup before workers start. Crash recovery is now immediate instead of waiting a full lease window for the lazy lease-expiry path. Safe under multi-replica: only already-expired leases are touched, which a healthy heartbeating peer never has. - Manga-list parsing now warns when listing anchors are dropped for a missing/empty href or title (split into parse_manga_list_anchors so the drop count is testable), turning silent source markup drift into an observable signal. Returned refs are byte-identical to before. Tests added: shutdown_mid_dispatch_releases_lease_without_burning_attempt, reclaim_orphaned_resets_only_expired_running_jobs, and a drop-count unit test. Patch bump 0.81.0 -> 0.81.1 (both manifests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -645,6 +645,17 @@ async fn spawn_crawler_daemon(
|
||||
})
|
||||
};
|
||||
|
||||
// Reclaim jobs orphaned by a previous crash/kill (running with an
|
||||
// expired lease) before workers start, so recovery is immediate instead
|
||||
// of waiting a full lease window for `lease`'s expiry clause. Safe under
|
||||
// multi-replica: only already-expired leases are touched. Best-effort —
|
||||
// a failure here just defers recovery to the lazy lease path.
|
||||
match crate::crawler::jobs::reclaim_orphaned(&db).await {
|
||||
Ok(0) => {}
|
||||
Ok(n) => tracing::info!(reclaimed = n, "crawler: reclaimed orphaned in-flight jobs at startup"),
|
||||
Err(e) => tracing::warn!(?e, "crawler: reclaim_orphaned at startup failed"),
|
||||
}
|
||||
|
||||
let daemon_handle = daemon::spawn(
|
||||
db,
|
||||
cancel,
|
||||
|
||||
Reference in New Issue
Block a user