The compression job is handed to the worker by a bare `tokio::spawn` that runs
AFTER `tx.commit()`, and the commit is a suspension point. If the guest walks
out of range inside it, Postgres applies the COMMIT and axum drops the future
before the spawn is reached: the row lands durably at
`compression_status = 'pending'` with no job behind it.
Nothing looked at it again. `startup_recovery` rescues only `'processing'`, and
`backfill_stale_derivatives` — whose predicate WOULD match — runs once at boot
and never on a timer. The photo keeps its feed entry, never gets a derivative,
forces every viewer to pull the full original instead, and is skipped by the
diashow for the rest of the event. Silent, and permanent.
This is the same hazard `TempFileGuard` already covers for the file, reached one
line later, so it gets the same answer: an `EnqueueGuard` armed before the commit
and stood down only once the job is with the worker. `Drop` runs on cancellation,
which closes the window — and it covers the indeterminate-commit error path too,
which returns Err on a row that may well be live. Queueing a job for a row that
did not commit is harmless: the status update matches zero rows and the task
retires.
A guard cannot survive a SIGKILL, so `requeue_stuck_pending` sweeps for the same
state at boot and every ten minutes. The grace window is what makes it safe to
re-enter the live path: a task flips the row to `'processing'` as its FIRST act
after taking a permit, so a row still `'pending'` ten minutes on is lost rather
than merely queued. The event simulation peaked at 120 queued with a p99 of 97s
while being fed thirty times a real event's arrival rate, so the margin is wide.
Hit 1 upload in 932 during that run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cacf616c2d