fix(crawler): harden crash & shutdown job recovery
Three failure-mode fixes surfaced by a crawler recovery audit: - Graceful shutdown mid-dispatch now releases the in-flight job back to pending without burning a retry attempt. process_lease wraps the dispatch in a biased tokio::select! cancel arm that aborts the heartbeat and calls jobs::release; previously a mid-job SIGTERM left the row 'running' until lease expiry and cost one of max_attempts. - Boot-time jobs::reclaim_orphaned resets 'running' jobs with an expired lease back to pending (attempt refunded), run once at daemon startup before workers start. Crash recovery is now immediate instead of waiting a full lease window for the lazy lease-expiry path. Safe under multi-replica: only already-expired leases are touched, which a healthy heartbeating peer never has. - Manga-list parsing now warns when listing anchors are dropped for a missing/empty href or title (split into parse_manga_list_anchors so the drop count is testable), turning silent source markup drift into an observable signal. Returned refs are byte-identical to before. Tests added: shutdown_mid_dispatch_releases_lease_without_burning_attempt, reclaim_orphaned_resets_only_expired_running_jobs, and a drop-count unit test. Patch bump 0.81.0 -> 0.81.1 (both manifests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -405,7 +405,30 @@ impl WorkerContext {
|
||||
// failed (exponential backoff) rather than wedging the worker.
|
||||
let dispatch = AssertUnwindSafe(self.dispatcher.dispatch(lease.payload.clone()))
|
||||
.catch_unwind();
|
||||
let outcome = tokio::time::timeout(self.job_timeout, dispatch).await;
|
||||
let outcome = tokio::select! {
|
||||
// `biased` so a shutdown that arrives while the dispatch is also
|
||||
// ready still takes the cancel arm and releases the lease.
|
||||
biased;
|
||||
_ = self.cancel.cancelled() => {
|
||||
// Graceful shutdown mid-dispatch: drop the in-flight dispatch
|
||||
// future (cancelling its browser work) and return the job to
|
||||
// `pending` WITHOUT burning a retry attempt, so a clean
|
||||
// restart doesn't march healthy jobs toward `max_attempts`.
|
||||
// `release` refunds the lease's `attempts` increment, mirroring
|
||||
// the session-expired path. Previously the worker had no cancel
|
||||
// branch here, so a mid-dispatch SIGTERM left the row `running`
|
||||
// until lease expiry and cost one attempt.
|
||||
heartbeat.abort();
|
||||
let _ = jobs::release(&self.pool, lease.id).await;
|
||||
tracing::info!(
|
||||
worker = self.id,
|
||||
lease_id = %lease.id,
|
||||
"worker: shutdown mid-dispatch — released lease without burning an attempt"
|
||||
);
|
||||
return;
|
||||
}
|
||||
o = tokio::time::timeout(self.job_timeout, dispatch) => o,
|
||||
};
|
||||
heartbeat.abort();
|
||||
|
||||
let outcome = match outcome {
|
||||
|
||||
Reference in New Issue
Block a user