fabi/Mangalord
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(jobs): per-lease generation token closes the ack-from-dead-lease race (0.87.26)

Browse Source 
0.87.20 narrowed but did not close the force-analyze race:
ack_done / ack_failed / renew matched only on (id, state='running'),
so a worker A whose lease was released mid-flight (force-analyze,
reclaim_orphaned) could still come back later — once a successor B
re-leased the same id and put it back to state='running' — and
clobber B's lease. A's stale result became the row's `done` payload;
B's in-flight work was silently lost. The pre-existing test even
documented this hole: "We can't make ack_done's lease_id distinguish
A from B today".

Add `lease_generation BIGINT NOT NULL DEFAULT 0` to crawler_jobs
(migration 0032). It is bumped by every lease, release / release_unowned,
and reclaim_orphaned, so lease identity is `(id, generation)` rather
than just `id`. Each Lease struct carries its generation. ack_done /
ack_failed / renew / release match on `(id, generation, state='running')`
so a stale ack from a prior generation finds zero rows and falls
back to the existing warn-and-skip branch.

Split the release surface in two:
  * release(lease_id, lease_generation) — owner-aware path for
    workers (graceful shutdown, session-expired), matches the
    caller's specific lease.
  * release_unowned(lease_id) — id-only path for force-analyze and
    ops tools that drop a running lease without holding a Lease
    struct; unconditionally bumps generation so any pending ack
    from the in-flight original becomes a no-op.

Force-analyze in repo::page_analysis now uses release_unowned; the
test in api_admin_analysis still passes unchanged (it only asserts
the steady-state outcome). The pre-existing `ack_done_no_ops_when_
lease_was_stolen` test is strengthened: it now mints both A and B
leases, asserts their generations differ, and verifies A's stale
ack does NOT clobber B's running row — the assertion the old test
explicitly couldn't make.

Five new tests in tests/crawler_jobs.rs pin every facet:
  * lease_assigns_strictly_increasing_generation_per_release
  * ack_done_from_dead_lease_after_release_unowned_is_a_no_op
  * ack_failed_from_dead_lease_after_release_unowned_is_a_no_op
  * renew_from_dead_lease_is_a_no_op
  * release_unowned_bumps_generation_even_when_lease_is_held

Mutation-confirmed: relaxing the ack_done guard to
`lease_generation >= $2` (bind 0) makes the race test fail with
state=done — the exact prior-bug shape.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-24 20:05:55 +02:00
parent ae708a72d0
commit 93b7e451bf
10 changed files with 419 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
backend/src/analysis/daemon.rs
View File

@@ -200,7 +200,7 @@ impl WorkerContext {
// Shouldn't happen — we lease only analyze_page — but ack done
// so a misrouted job doesn't loop forever.
tracing::warn!(worker = self.id, "analysis worker: non-analyze payload leased");
let _ = jobs::ack_done(&self.pool, lease.id).await;
let _ = jobs::ack_done(&self.pool, lease.id, lease.lease_generation).await;
return;
};
@@ -210,7 +210,7 @@ impl WorkerContext {
if !force {
if let Ok(Some(row)) = repo::page_analysis::load(&self.pool, page_id).await {
if row.status == crate::domain::page_analysis::AnalysisStatus::Done {
let _ = jobs::ack_done(&self.pool, lease.id).await;
let _ = jobs::ack_done(&self.pool, lease.id, lease.lease_generation).await;
return;
}
}
@@ -241,10 +241,11 @@ impl WorkerContext {
let heartbeat = {
let hb_pool = self.pool.clone();
let hb_id = lease.id;
let hb_gen = lease.lease_generation;
tokio::spawn(async move {
loop {
tokio::time::sleep(LEASE_HEARTBEAT).await;
match jobs::renew(&hb_pool, hb_id, LEASE_DURATION).await {
match jobs::renew(&hb_pool, hb_id, hb_gen, LEASE_DURATION).await {
Ok(true) => {}
Ok(false) => break,
Err(e) => tracing::warn!(lease_id = %hb_id, ?e, "heartbeat renew failed"),
@@ -274,7 +275,7 @@ impl WorkerContext {
lease_id = %lease.id,
"analysis worker: cancelled mid-dispatch — releasing lease"
);
let _ = jobs::release(&self.pool, lease.id).await;
let _ = jobs::release(&self.pool, lease.id, lease.lease_generation).await;
// Don't publish Failed (the page didn't actually fail) and
// don't write a duration row — neither outcome is true. DO
// publish Cancelled so the dashboard's "now analyzing" banner
@@ -342,7 +343,7 @@ impl WorkerContext {
match result {
Ok(()) => {
let _ = jobs::ack_done(&self.pool, lease.id).await;
let _ = jobs::ack_done(&self.pool, lease.id, lease.lease_generation).await;
}
Err(msg) => {
tracing::warn!(
@@ -357,6 +358,7 @@ impl WorkerContext {
&msg,
lease.attempts,
lease.max_attempts,
lease.lease_generation,
)
.await;
// Terminal failure (retries exhausted) → record a `failed`