fix(crawler): harden crash & shutdown job recovery
All checks were successful
deploy / test-backend (push) Successful in 19m5s
deploy / test-frontend (push) Successful in 9m54s
deploy / build-and-push (push) Successful in 10m11s
deploy / deploy (push) Successful in 12s

Three failure-mode fixes surfaced by a crawler recovery audit:

- Graceful shutdown mid-dispatch now releases the in-flight job back to
  pending without burning a retry attempt. process_lease wraps the
  dispatch in a biased tokio::select! cancel arm that aborts the
  heartbeat and calls jobs::release; previously a mid-job SIGTERM left
  the row 'running' until lease expiry and cost one of max_attempts.

- Boot-time jobs::reclaim_orphaned resets 'running' jobs with an expired
  lease back to pending (attempt refunded), run once at daemon startup
  before workers start. Crash recovery is now immediate instead of
  waiting a full lease window for the lazy lease-expiry path. Safe under
  multi-replica: only already-expired leases are touched, which a
  healthy heartbeating peer never has.

- Manga-list parsing now warns when listing anchors are dropped for a
  missing/empty href or title (split into parse_manga_list_anchors so
  the drop count is testable), turning silent source markup drift into
  an observable signal. Returned refs are byte-identical to before.

Tests added: shutdown_mid_dispatch_releases_lease_without_burning_attempt,
reclaim_orphaned_resets_only_expired_running_jobs, and a drop-count unit
test. Patch bump 0.81.0 -> 0.81.1 (both manifests).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-14 16:59:46 +02:00
committed by fabi
parent 54530d67ef
commit ce6d96c5e1
9 changed files with 275 additions and 22 deletions

View File

@@ -405,7 +405,30 @@ impl WorkerContext {
// failed (exponential backoff) rather than wedging the worker.
let dispatch = AssertUnwindSafe(self.dispatcher.dispatch(lease.payload.clone()))
.catch_unwind();
let outcome = tokio::time::timeout(self.job_timeout, dispatch).await;
let outcome = tokio::select! {
// `biased` so a shutdown that arrives while the dispatch is also
// ready still takes the cancel arm and releases the lease.
biased;
_ = self.cancel.cancelled() => {
// Graceful shutdown mid-dispatch: drop the in-flight dispatch
// future (cancelling its browser work) and return the job to
// `pending` WITHOUT burning a retry attempt, so a clean
// restart doesn't march healthy jobs toward `max_attempts`.
// `release` refunds the lease's `attempts` increment, mirroring
// the session-expired path. Previously the worker had no cancel
// branch here, so a mid-dispatch SIGTERM left the row `running`
// until lease expiry and cost one attempt.
heartbeat.abort();
let _ = jobs::release(&self.pool, lease.id).await;
tracing::info!(
worker = self.id,
lease_id = %lease.id,
"worker: shutdown mid-dispatch — released lease without burning an attempt"
);
return;
}
o = tokio::time::timeout(self.job_timeout, dispatch) => o,
};
heartbeat.abort();
let outcome = match outcome {