fix(crawler): harden crash & shutdown job recovery
Three failure-mode fixes surfaced by a crawler recovery audit: - Graceful shutdown mid-dispatch now releases the in-flight job back to pending without burning a retry attempt. process_lease wraps the dispatch in a biased tokio::select! cancel arm that aborts the heartbeat and calls jobs::release; previously a mid-job SIGTERM left the row 'running' until lease expiry and cost one of max_attempts. - Boot-time jobs::reclaim_orphaned resets 'running' jobs with an expired lease back to pending (attempt refunded), run once at daemon startup before workers start. Crash recovery is now immediate instead of waiting a full lease window for the lazy lease-expiry path. Safe under multi-replica: only already-expired leases are touched, which a healthy heartbeating peer never has. - Manga-list parsing now warns when listing anchors are dropped for a missing/empty href or title (split into parse_manga_list_anchors so the drop count is testable), turning silent source markup drift into an observable signal. Returned refs are byte-identical to before. Tests added: shutdown_mid_dispatch_releases_lease_without_burning_attempt, reclaim_orphaned_resets_only_expired_running_jobs, and a drop-count unit test. Patch bump 0.81.0 -> 0.81.1 (both manifests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -314,6 +314,40 @@ pub async fn release(pool: &PgPool, lease_id: Uuid) -> sqlx::Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Reclaim jobs orphaned by a crashed/killed worker: those still `running`
|
||||
/// whose `leased_until` has already lapsed. Each is returned to `pending`
|
||||
/// with its lease cleared and the lease's `attempts` increment refunded
|
||||
/// (`GREATEST(0, attempts - 1)`, mirroring [`release`]) — a crash that
|
||||
/// interrupted an attempt mid-flight shouldn't count against `max_attempts`.
|
||||
/// Returns the number reclaimed.
|
||||
///
|
||||
/// Intended to run once at daemon startup so crash recovery is immediate
|
||||
/// rather than waiting up to a full lease window for `lease`'s expiry clause
|
||||
/// to re-pick the row. It is safe under multi-replica deployment precisely
|
||||
/// because it only touches **already-expired** leases: a healthy peer
|
||||
/// heartbeats (`renew`) every ~20s, keeping its in-flight jobs' `leased_until`
|
||||
/// in the future, so this never steals live work — it only does eagerly what
|
||||
/// `lease` would do lazily, minus the attempt burn.
|
||||
///
|
||||
/// Trade-off: a job whose dispatch reliably hard-kills the process (e.g. OOM
|
||||
/// on a pathological payload) is refunded each boot and could loop without
|
||||
/// dead-lettering. That window is bounded in practice by the per-image size
|
||||
/// cap and the worker's `job_timeout` (a hang is acked-failed normally, which
|
||||
/// *does* burn an attempt); only a true process-killer evades it, which is an
|
||||
/// infrastructure signal worth surfacing rather than silently dead-lettering
|
||||
/// everyone's chapters during a crash loop.
|
||||
pub async fn reclaim_orphaned(pool: &PgPool) -> sqlx::Result<u64> {
|
||||
let result = sqlx::query(
|
||||
"UPDATE crawler_jobs \
|
||||
SET state = 'pending', leased_until = NULL, \
|
||||
attempts = GREATEST(0, attempts - 1), updated_at = now() \
|
||||
WHERE state = 'running' AND leased_until < now()",
|
||||
)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
Ok(result.rows_affected())
|
||||
}
|
||||
|
||||
/// Delete `done` jobs whose `updated_at` is older than `retention_days`
|
||||
/// days. `0` disables the reaper without touching the table. Returns the
|
||||
/// number of rows removed.
|
||||
|
||||
Reference in New Issue
Block a user