fix(outbox): reclaim stale claims — a crash mid-dispatch stranded events forever
Chasing the e2e flakiness turned up a production durability bug, not a test bug. The dispatcher claims an outbox row, executes it, then either deletes it (success) or reschedules it (failure) — both of which clear the claim. If the PROCESS DIES in between, neither runs. And `claim_due` only ever selects `claimed_at IS NULL`. Nothing else in the codebase clears `outbox.claimed_at` — grep it: there are exactly three writers, and those are two of them. So a crash or restart mid-dispatch stranded every in-flight row PERMANENTLY. Its trigger never fired and no retry could notice. The outbox is the universal trigger path, so the loss covered kv / docs / files / cron / pubsub / email / invoke_async / dead-letter alike. This is the same durability class as the audit's #6 (lost trigger event), which was just fixed at the WRITE end — this is the same hole at the READ end. Every other claim-based store already had the safety net: `queue_messages` and `group_queue_messages` have `reclaim_visibility_timeouts`, `workflow_steps` has its own reclaim. The outbox was the one that didn't. `OutboxRepo::reclaim_stale_claims(timeout)` returns rows whose claim is older than `PICLOUD_OUTBOX_CLAIM_TIMEOUT_SEC` (default 600s), run from the dispatcher's existing reclaim ticker. The default is deliberately generous — a script may run for up to 300s (the `scripts.timeout_seconds` CHECK), so a claim held past twice that is abandoned rather than slow; reclaiming a row a LIVE dispatcher is still working on would double-execute it (survivable — dispatch is at-least-once — but not worth courting). A reclaim does NOT bump `attempt_count`: the handler never ran, so it must not consume the row's retry budget, or repeated restarts would dead-letter an event that executed zero times. (Same reasoning as the transient queue `release` fixed earlier in this branch.) This is also the root cause of the flaky e2e suites: each test drops its dispatcher, and `claim_due` is not scoped per app or per dispatcher, so a test's dispatcher could claim another test's row and strand it on teardown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -231,6 +231,8 @@ impl Dispatcher {
|
||||
// contend with the per-100ms dispatcher tick.
|
||||
let reclaim_queue = self.queue.clone();
|
||||
let reclaim_group_queue = self.group_queue.clone();
|
||||
let reclaim_outbox = self.outbox.clone();
|
||||
let outbox_claim_timeout = self.config.outbox_claim_timeout_secs;
|
||||
let reclaim_interval =
|
||||
Duration::from_millis(u64::from(self.config.queue_reclaim_interval_ms));
|
||||
tokio::spawn(async move {
|
||||
@@ -238,6 +240,21 @@ impl Dispatcher {
|
||||
ticker.tick().await;
|
||||
loop {
|
||||
ticker.tick().await;
|
||||
// A dispatcher that died mid-dispatch left its claimed rows
|
||||
// stranded — `claim_due` only takes unclaimed rows, so without
|
||||
// this they would never fire again.
|
||||
match reclaim_outbox
|
||||
.reclaim_stale_claims(outbox_claim_timeout)
|
||||
.await
|
||||
{
|
||||
Ok(0) => {}
|
||||
Ok(n) => tracing::warn!(
|
||||
reclaimed = n,
|
||||
timeout_secs = outbox_claim_timeout,
|
||||
"reclaimed stale outbox claims — a dispatcher died mid-dispatch"
|
||||
),
|
||||
Err(e) => tracing::warn!(?e, "outbox reclaim task errored"),
|
||||
}
|
||||
match reclaim_queue.reclaim_visibility_timeouts().await {
|
||||
Ok(0) => {}
|
||||
Ok(n) => tracing::info!(reclaimed = n, "queue visibility-timeout reclaim"),
|
||||
@@ -2344,6 +2361,12 @@ mod tests {
|
||||
|
||||
#[async_trait]
|
||||
impl OutboxRepo for UnusedOutbox {
|
||||
async fn reclaim_stale_claims(
|
||||
&self,
|
||||
_timeout_secs: u32,
|
||||
) -> Result<u64, crate::outbox_repo::OutboxRepoError> {
|
||||
Ok(0)
|
||||
}
|
||||
async fn insert(
|
||||
&self,
|
||||
_row: NewOutboxRow,
|
||||
@@ -2649,6 +2672,12 @@ mod tests {
|
||||
|
||||
#[async_trait]
|
||||
impl OutboxRepo for RecordingOutbox {
|
||||
async fn reclaim_stale_claims(
|
||||
&self,
|
||||
_timeout_secs: u32,
|
||||
) -> Result<u64, crate::outbox_repo::OutboxRepoError> {
|
||||
Ok(0)
|
||||
}
|
||||
async fn insert(
|
||||
&self,
|
||||
_row: NewOutboxRow,
|
||||
|
||||
Reference in New Issue
Block a user