fix: close the nine ways an unattended event loses photos or dies

Every one of these was found in the pre-event audit, verified against source, and
survives to production on the current main. Grouped by what actually goes wrong.

PHOTOS DISAPPEAR

* compression.rs no longer soft-deletes on a failed derivative. The guest got a
  201, watched the card appear, then watched it vanish — the row left v_feed,
  find_visible_media and BOTH keepsakes, while its bytes sat on disk for 14 days
  waiting for a cleanup nothing announced. No screen anywhere lists compression
  failures, so recovery meant hand-written SQL that also had to re-add the
  refunded quota. Now it does exactly what the ENOSPC arm beside it already did
  and documented as correct: keep the row, serve the original, retry on the next
  boot (bounded by derivative_attempts). `upload-deleted` is no longer emitted;
  `upload-processed` is, so the card re-renders instead of sitting on a
  placeholder.

* A 413 is now a reversible lock, so the blob survives. The quota moves — free
  disk falls, uploader count rises — so a guest goes over it having done nothing,
  and treating that as permanent meant a 400 MB video was pushed across cellular
  in full and THEN deleted from IndexedDB. Gone on both sides, and unrecoverable
  for an in-app camera capture that exists nowhere else.

* quota_limit_bytes gained a floor and a stable divisor. The ceiling used to
  decrease monotonically all evening; it now settles at max(uploaders,
  estimated_guest_count) — a config key that was seeded, validated in the admin
  whitelist, and read by no code at all. The floor is clamped to what the disk
  can actually back, so a full volume still yields zero rather than handing out
  an allowance it cannot honour.

* Because that floor gives up the aggregate guarantee the formula used to imply,
  uploads now check a hard 10 GB reserve first, independent of every quota
  toggle. postgres_data, media_data and exports_data share one filesystem: the
  end state was not a degraded feature, it was Postgres unable to write WAL.

THE ARCHIVE DISAPPEARS

* prune_superseded_archives runs only after the new generation lands. It ran
  before the preflight, reasoning the old archive was already unreachable — true
  of reachability, false of recoverability. An epoch is a value that can be
  rolled back; deleted bytes cannot. Any failed rebuild left the event with NO
  keepsake at all.

* The export preflight reserves the same 10 GB. `free < needed` authorised an
  export sized at exactly free, which ran for half an hour and landed the box at
  zero with the keepsake still unfinished.

THE APP DIES

* The feed reconcile re-reads the id set after its awaits instead of reusing one
  captured up to three round-trips earlier. The new-upload SSE handler prepends
  during exactly that window, so the row was both already present and absent from
  the stale set — prepended twice, and a duplicate key in a keyed {#each} throws
  in production, not just dev. The SSE handler and loadMore now dedupe too.

* Added routes/+error.svelte. Without it any uncaught error fell through to
  SvelteKit's unstyled English 500 with no reload control — inside a chromeless
  standalone PWA with no URL bar, for the rest of the evening.

THE OPERATOR IS LOCKED OUT

* admin_login verifies the password BEFORE charging the rate bucket, and a
  correct password is never throttled. The old order made this a denial of
  service against its own operator: every guest shares one NAT IP, the check ran
  first, so five requests a minute from any phone in the room kept the bucket
  full — and the escape hatch needed the admin session being blocked. A generous
  separate ceiling still bounds bcrypt CPU.

THE PROJECTOR DIES

* The preload budget is now strictly inside the dwell. At the 3s option the 4s
  budget could never land a commit on a slow uplink, so the wall froze on one
  photo while the queue drained silently behind it.

* The wake lock retries every 30s while visible, and the page says so on screen
  when the browser has no wake lock API. visibilitychange was the only retry
  trigger and a kiosk never changes visibility, so one refusal — iOS in Low Power
  Mode, say — was permanent.

* Caddy: /api/v1/upload/*/display joins the cacheable carve-out. The backend set
  max-age=300 on it and the blanket no-store silently replaced it, so a projector
  re-fetched a full-size JPEG per slide, ~2-4 GB over an evening on the uplink
  the guests are uploading over.

Also removes Upload::soft_delete, now unreferenced and an unscoped footgun next
to soft_delete_in_event.

Verified: 146/146 backend tests against a live Postgres, clippy clean, 51/51
vitest, svelte-check 0 errors, eslint clean, vite build, caddy validate, compose
YAML parse.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-08-08 21:32:52 +02:00
parent 61119be817
commit 1d9fb11c7b
12 changed files with 448 additions and 106 deletions

View File

@@ -165,35 +165,43 @@ impl CompressionWorker {
tracing::error!(
"compression failed for upload {upload_id} after {attempt} attempt(s): {e:#}"
);
// Refund + soft-delete (one tx, so v_feed excludes it) so a failed
// transcode doesn't leave a permanently broken feed card or silently
// charge the uploader's quota. Then tell the uploader (upload-error
// toast) and evict the card everywhere (upload-deleted).
// KEEP THE ROW. This used to soft-delete, which made a derivative failure
// indistinguishable — to the guest — from their photo being deleted: they
// got a `201 Created`, watched the card appear, and then watched it vanish.
// The row left `v_feed`, `find_visible_media` and BOTH keepsake archives,
// so the photo was gone from the product's core promise while its bytes sat
// on disk for 14 days waiting for a `cleanup_deleted_media` that nothing
// told anyone about. There is no host or admin screen listing compression
// failures, so recovery meant hand-written SQL that also had to re-add the
// refunded quota bytes. Against "0 lost uploads", that was silent per-photo
// loss on any error the ENOSPC arm above doesn't catch — a HEIC that slipped
// the allowlist, a truncated frame, an ffmpeg hiccup, a pool blip.
//
// The ORIGINAL IS DELIBERATELY KEPT. This path used to `remove_file` it
// unconditionally, which meant any transient error — a disk-full blip
// while saving a derivative, a pool hiccup, a panic in the image codec —
// irreversibly destroyed the guest's only copy of a photo they can never
// retake. The row is only soft-deleted, so keeping the bytes makes the
// upload fully recoverable; the file is orphaned rather than lost, and
// the path is logged so it can be found. `backfill_stale_derivatives`
// already refuses to destroy data on error for exactly this reason.
// This is exactly what the ENOSPC arm already does and documents as correct:
// every client falls back to the original when `preview_url` and
// `thumbnail_url` are NULL, so the photo stays visible and downloadable —
// just uncompressed — and `backfill_stale_derivatives` retries it on the
// next boot, now bounded by `derivative_attempts` so a poisoned row cannot
// loop. The quota stays charged, which is correct: the bytes are still on
// disk and still the guest's.
let _ = Upload::set_compression_status(&worker.pool, upload_id, "failed").await;
if let Err(del) = Upload::soft_delete(&worker.pool, upload_id).await {
tracing::warn!(error = ?del, %upload_id, "failed to soft-delete after compression failure");
}
tracing::warn!(
%upload_id,
path = %worker.media_path.join(&original_path).display(),
"original retained for recovery after compression failure"
"derivatives failed; the upload is kept and served from its original"
);
// `upload-error` still fires so the uploader learns the photo will look
// uncompressed. `upload-deleted` deliberately does NOT — nothing was
// deleted, and evicting the card was the visible half of the data loss.
let _ = worker.sse_tx.send(SseEvent {
event_type: "upload-error".to_string(),
data: serde_json::json!({ "upload_id": upload_id, "error": e.to_string() })
.to_string(),
});
// Tell every client to refetch, so the card re-renders from the original
// instead of sitting on a stale "processing" placeholder forever.
let _ = worker.sse_tx.send(SseEvent {
event_type: "upload-deleted".to_string(),
event_type: "upload-processed".to_string(),
data: serde_json::json!({ "upload_id": upload_id }).to_string(),
});
}