93b7e451bf
0.87.20 narrowed but did not close the force-analyze race:
ack_done / ack_failed / renew matched only on (id, state='running'),
so a worker A whose lease was released mid-flight (force-analyze,
reclaim_orphaned) could still come back later — once a successor B
re-leased the same id and put it back to state='running' — and
clobber B's lease. A's stale result became the row's `done` payload;
B's in-flight work was silently lost. The pre-existing test even
documented this hole: "We can't make ack_done's lease_id distinguish
A from B today".
Add `lease_generation BIGINT NOT NULL DEFAULT 0` to crawler_jobs
(migration 0032). It is bumped by every lease, release / release_unowned,
and reclaim_orphaned, so lease identity is `(id, generation)` rather
than just `id`. Each Lease struct carries its generation. ack_done /
ack_failed / renew / release match on `(id, generation, state='running')`
so a stale ack from a prior generation finds zero rows and falls
back to the existing warn-and-skip branch.
Split the release surface in two:
* release(lease_id, lease_generation) — owner-aware path for
workers (graceful shutdown, session-expired), matches the
caller's specific lease.
* release_unowned(lease_id) — id-only path for force-analyze and
ops tools that drop a running lease without holding a Lease
struct; unconditionally bumps generation so any pending ack
from the in-flight original becomes a no-op.
Force-analyze in repo::page_analysis now uses release_unowned; the
test in api_admin_analysis still passes unchanged (it only asserts
the steady-state outcome). The pre-existing `ack_done_no_ops_when_
lease_was_stolen` test is strengthened: it now mints both A and B
leases, asserts their generations differ, and verifies A's stale
ack does NOT clobber B's running row — the assertion the old test
explicitly couldn't make.
Five new tests in tests/crawler_jobs.rs pin every facet:
* lease_assigns_strictly_increasing_generation_per_release
* ack_done_from_dead_lease_after_release_unowned_is_a_no_op
* ack_failed_from_dead_lease_after_release_unowned_is_a_no_op
* renew_from_dead_lease_is_a_no_op
* release_unowned_bumps_generation_even_when_lease_is_held
Mutation-confirmed: relaxing the ack_done guard to
`lease_generation >= $2` (bind 0) makes the race test fail with
state=done — the exact prior-bug shape.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
24 lines
1.1 KiB
SQL
24 lines
1.1 KiB
SQL
-- Per-lease generation token to close the lease-identity race that
|
|
-- 0.87.20 only partially addressed.
|
|
--
|
|
-- Before this, `ack_done` / `ack_failed` / `renew` matched on
|
|
-- `(id, state='running')`. A worker whose lease was released mid-
|
|
-- flight (force-analyze, reclaim) could still ack-done the row once a
|
|
-- successor leased it back to `state='running'` — clobbering the
|
|
-- successor's lease with the original's stale result. The bug class is
|
|
-- ack-from-a-dead-lease.
|
|
--
|
|
-- Adding `lease_generation` makes lease identity = `(id, generation)`:
|
|
-- * `lease`/`lease_kinds` bump generation by 1 at lease time
|
|
-- * `release` (force-analyze, graceful shutdown) bumps generation too
|
|
-- * `reclaim_orphaned` bumps generation for every reclaimed row
|
|
-- * `ack_done` / `ack_failed` / `renew` match on `(id, generation,
|
|
-- state='running')`, so a stale ack from a prior generation finds
|
|
-- no row and falls back to the existing warn-and-skip branch.
|
|
--
|
|
-- BIGINT so the counter can't realistically overflow even for a row
|
|
-- that's been crash-leased thousands of times.
|
|
|
|
ALTER TABLE crawler_jobs
|
|
ADD COLUMN lease_generation BIGINT NOT NULL DEFAULT 0;
|