fix(upload): a client vanishing mid-commit lost the photo's preview forever

The compression job is handed to the worker by a bare `tokio::spawn` that runs
AFTER `tx.commit()`, and the commit is a suspension point. If the guest walks
out of range inside it, Postgres applies the COMMIT and axum drops the future
before the spawn is reached: the row lands durably at
`compression_status = 'pending'` with no job behind it.

Nothing looked at it again. `startup_recovery` rescues only `'processing'`, and
`backfill_stale_derivatives` — whose predicate WOULD match — runs once at boot
and never on a timer. The photo keeps its feed entry, never gets a derivative,
forces every viewer to pull the full original instead, and is skipped by the
diashow for the rest of the event. Silent, and permanent.

This is the same hazard `TempFileGuard` already covers for the file, reached one
line later, so it gets the same answer: an `EnqueueGuard` armed before the commit
and stood down only once the job is with the worker. `Drop` runs on cancellation,
which closes the window — and it covers the indeterminate-commit error path too,
which returns Err on a row that may well be live. Queueing a job for a row that
did not commit is harmless: the status update matches zero rows and the task
retires.

A guard cannot survive a SIGKILL, so `requeue_stuck_pending` sweeps for the same
state at boot and every ten minutes. The grace window is what makes it safe to
re-enter the live path: a task flips the row to `'processing'` as its FIRST act
after taking a permit, so a row still `'pending'` ten minutes on is lost rather
than merely queued. The event simulation peaked at 120 queued with a p99 of 97s
while being fed thirty times a real event's arrival rate, so the margin is wide.

Hit 1 upload in 932 during that run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

backend/src/services/compression.rs

@@ -411,6 +411,66 @@ impl CompressionWorker {
     /// `derivative_attempts` stops a fatal row being replayed on every boot, `BACKFILL_BATCH`
     /// stops one start queueing unbounded work, and the whole thing runs as ONE task walking
     /// the rows sequentially rather than N tasks racing for the same semaphore.
+    /// How long an upload may sit at `'pending'` before the sweeper assumes its job was lost.
+    ///
+    /// Must comfortably exceed the deepest real queue wait, or the sweeper re-enqueues photos
+    /// that are merely waiting their turn. The live path makes that easy to bound: a task's
+    /// FIRST action after taking a permit is to flip the row to `'processing'`, so a row still
+    /// `'pending'` after this long is either behind that much work or genuinely lost. The
+    /// event simulation peaked at 120 queued with a p99 of 97s while being fed thirty times a
+    /// real event's arrival rate, so ten minutes is a wide margin rather than a tight one.
+    const PENDING_GRACE: chrono::Duration = chrono::Duration::minutes(10);
+
+    /// Re-queue uploads whose compression job was lost between the commit and the spawn.
+    ///
+    /// `EnqueueGuard` closes that window for a cancelled request, but nothing in-process
+    /// survives a SIGKILL or an OOM kill, and `startup_recovery` only rescues `'processing'`
+    /// — so a row orphaned at `'pending'` had no path back at all. Without this, the failure is
+    /// silent and permanent: the photo keeps its feed entry, never gets a derivative, and is
+    /// dropped from the diashow for the rest of the event.
+    ///
+    /// Re-entering the live `process` path (rather than regenerating derivatives inline, as
+    /// `backfill_stale_derivatives` does) is deliberate: it reuses the status transitions, the
+    /// retry ladder and the SSE broadcast, so a rescued photo behaves exactly like one that was
+    /// never lost.
+    pub async fn requeue_stuck_pending(&self) {
+        let cutoff = chrono::Utc::now() - Self::PENDING_GRACE;
+        let rows = sqlx::query_as::<_, (Uuid, String, String)>(
+            "SELECT id, original_path, mime_type FROM upload
+             WHERE deleted_at IS NULL
+               AND compression_status = 'pending'
+               AND original_path <> ''
+               AND created_at < $1
+               AND derivative_attempts < $2
+             ORDER BY created_at
+             LIMIT $3",
+        )
+        .bind(cutoff)
+        .bind(Self::MAX_DERIVATIVE_ATTEMPTS)
+        .bind(Self::BACKFILL_BATCH)
+        .fetch_all(&self.pool)
+        .await;
+        let rows = match rows {
+            Ok(r) => r,
+            Err(e) => {
+                tracing::warn!(error = ?e, "stuck-pending sweep query failed");
+                return;
+            }
+        };
+        if rows.is_empty() {
+            return;
+        }
+        // WARN, not INFO: reaching this means the guard did not run, which is worth noticing
+        // even though the photo is being rescued.
+        tracing::warn!(
+            count = rows.len(),
+            "found upload(s) stuck at 'pending' with no compression job; re-queueing"
+        );
+        for (id, original_path, mime_type) in rows {
+            self.process(id, original_path, mime_type);
+        }
+    }
+
     pub async fn backfill_stale_derivatives(&self) {
         // `original_path IS NOT NULL` was dead — the column is NOT NULL. What actually needs
         // excluding is the blanked path `cleanup_deleted_media` leaves behind.

backend/src/services/maintenance.rs

@@ -144,7 +144,36 @@ pub fn spawn_periodic_tasks(
     rate_limiter: RateLimiter,
     sse_tickets: SseTicketStore,
     media_path: PathBuf,
+    compression: crate::services::compression::CompressionWorker,
 ) {
+    // Rescuing a lost compression job gets its OWN loop rather than a line in the hourly one.
+    // Cadence is the whole point: an upload stuck at 'pending' is a photo missing from the
+    // diashow and showing without a preview, so an hour of that during a five-hour party is a
+    // guest-visible hole. Ten minutes bounds it while still sitting far above the deepest
+    // observed queue wait (see `PENDING_GRACE`). Supervised for the same reason as the loop
+    // below: silently dying is how this class of safety net stops existing.
+    tokio::spawn(async move {
+        loop {
+            let worker = compression.clone();
+            let inner = tokio::spawn(async move {
+                // `interval` fires its first tick immediately, and that is deliberate here:
+                // a restart is exactly when stranded rows exist (the drop guard cannot
+                // survive a SIGKILL), and `startup_recovery` rescues only 'processing'. The
+                // grace window makes the boot pass safe — nothing uploaded in the last ten
+                // minutes can match, so this can never race a job that is merely queued.
+                let mut tick = tokio::time::interval(Duration::from_secs(600));
+                loop {
+                    tick.tick().await;
+                    worker.requeue_stuck_pending().await;
+                }
+            });
+            match inner.await {
+                Ok(()) => tracing::error!("stuck-pending sweeper returned; restarting it"),
+                Err(e) => tracing::error!(error = ?e, "stuck-pending sweeper died; restarting it"),
+            }
+            tokio::time::sleep(Duration::from_secs(60)).await;
+        }
+    });
     // Supervised, because this one task carries EVERY piece of recurring hygiene in the app:
     // session pruning, media reclamation, the orphan-temp sweep, and the rate-limiter and
     // SSE-ticket maps. As a bare `tokio::spawn` with no retained handle, a single panic anywhere