fix: close the nine ways an unattended event loses photos or dies
Every one of these was found in the pre-event audit, verified against source, and
survives to production on the current main. Grouped by what actually goes wrong.
PHOTOS DISAPPEAR
* compression.rs no longer soft-deletes on a failed derivative. The guest got a
201, watched the card appear, then watched it vanish — the row left v_feed,
find_visible_media and BOTH keepsakes, while its bytes sat on disk for 14 days
waiting for a cleanup nothing announced. No screen anywhere lists compression
failures, so recovery meant hand-written SQL that also had to re-add the
refunded quota. Now it does exactly what the ENOSPC arm beside it already did
and documented as correct: keep the row, serve the original, retry on the next
boot (bounded by derivative_attempts). `upload-deleted` is no longer emitted;
`upload-processed` is, so the card re-renders instead of sitting on a
placeholder.
* A 413 is now a reversible lock, so the blob survives. The quota moves — free
disk falls, uploader count rises — so a guest goes over it having done nothing,
and treating that as permanent meant a 400 MB video was pushed across cellular
in full and THEN deleted from IndexedDB. Gone on both sides, and unrecoverable
for an in-app camera capture that exists nowhere else.
* quota_limit_bytes gained a floor and a stable divisor. The ceiling used to
decrease monotonically all evening; it now settles at max(uploaders,
estimated_guest_count) — a config key that was seeded, validated in the admin
whitelist, and read by no code at all. The floor is clamped to what the disk
can actually back, so a full volume still yields zero rather than handing out
an allowance it cannot honour.
* Because that floor gives up the aggregate guarantee the formula used to imply,
uploads now check a hard 10 GB reserve first, independent of every quota
toggle. postgres_data, media_data and exports_data share one filesystem: the
end state was not a degraded feature, it was Postgres unable to write WAL.
THE ARCHIVE DISAPPEARS
* prune_superseded_archives runs only after the new generation lands. It ran
before the preflight, reasoning the old archive was already unreachable — true
of reachability, false of recoverability. An epoch is a value that can be
rolled back; deleted bytes cannot. Any failed rebuild left the event with NO
keepsake at all.
* The export preflight reserves the same 10 GB. `free < needed` authorised an
export sized at exactly free, which ran for half an hour and landed the box at
zero with the keepsake still unfinished.
THE APP DIES
* The feed reconcile re-reads the id set after its awaits instead of reusing one
captured up to three round-trips earlier. The new-upload SSE handler prepends
during exactly that window, so the row was both already present and absent from
the stale set — prepended twice, and a duplicate key in a keyed {#each} throws
in production, not just dev. The SSE handler and loadMore now dedupe too.
* Added routes/+error.svelte. Without it any uncaught error fell through to
SvelteKit's unstyled English 500 with no reload control — inside a chromeless
standalone PWA with no URL bar, for the rest of the evening.
THE OPERATOR IS LOCKED OUT
* admin_login verifies the password BEFORE charging the rate bucket, and a
correct password is never throttled. The old order made this a denial of
service against its own operator: every guest shares one NAT IP, the check ran
first, so five requests a minute from any phone in the room kept the bucket
full — and the escape hatch needed the admin session being blocked. A generous
separate ceiling still bounds bcrypt CPU.
THE PROJECTOR DIES
* The preload budget is now strictly inside the dwell. At the 3s option the 4s
budget could never land a commit on a slow uplink, so the wall froze on one
photo while the queue drained silently behind it.
* The wake lock retries every 30s while visible, and the page says so on screen
when the browser has no wake lock API. visibilitychange was the only retry
trigger and a kiosk never changes visibility, so one refusal — iOS in Low Power
Mode, say — was permanent.
* Caddy: /api/v1/upload/*/display joins the cacheable carve-out. The backend set
max-age=300 on it and the blanket no-store silently replaced it, so a projector
re-fetched a full-size JPEG per slide, ~2-4 GB over an evening on the uplink
the guests are uploading over.
Also removes Upload::soft_delete, now unreferenced and an unscoped footgun next
to soft_delete_in_event.
Verified: 146/146 backend tests against a live Postgres, clippy clean, 51/51
vitest, svelte-check 0 errors, eslint clean, vite build, caddy validate, compose
YAML parse.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -499,9 +499,6 @@ async fn run_zip_export(
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Reclaim BEFORE measuring: the superseded archive is already unreachable, and the space it
|
||||
// holds is very often exactly the space this rebuild needs.
|
||||
prune_superseded_archives(pool, export_path, "Gallery", event_id, epoch).await;
|
||||
// AFTER the claim, not before. A preflight that bailed before claiming would leave the row
|
||||
// `pending` with no worker and no error — the spinner-forever state `mark_failed`'s status
|
||||
// guard was widened to prevent. Failing here goes through the caller's `mark_failed`, so the
|
||||
@@ -517,6 +514,25 @@ async fn run_zip_export(
|
||||
tokio::fs::remove_file(export_path.join(gen_name(event_id, "Gallery", epoch, ".tmp")))
|
||||
.await;
|
||||
}
|
||||
|
||||
// Reclaim the PREVIOUS generation only once this one has actually landed.
|
||||
//
|
||||
// This used to run before the preflight, reasoning that the superseded archive is already
|
||||
// unreachable and its space is usually exactly what the rebuild needs. That is true about
|
||||
// REACHABILITY and false about RECOVERABILITY: an epoch is a database value that can be
|
||||
// rolled back, deleted bytes cannot. Any rebuild that then failed — ENOSPC mid-write, an
|
||||
// OOM, a hung ffmpeg, a host tapping "Neu erzeugen" on a bad day — left the event with NO
|
||||
// archive at all, which is the one outcome the whole product exists to prevent, at the one
|
||||
// moment nobody is watching.
|
||||
//
|
||||
// The cost of deferring is that a rebuild now needs room for both generations at once, and
|
||||
// `ensure_export_space` above no longer gets to count the old archive's bytes as available.
|
||||
// That is the correct trade: it converts "silently destroyed the only copy" into "refused
|
||||
// to start, and said why".
|
||||
if res.is_ok() {
|
||||
prune_superseded_archives(pool, export_path, "Gallery", event_id, epoch).await;
|
||||
}
|
||||
|
||||
abandon_if_superseded("ZIP", event_id, epoch, res)
|
||||
}
|
||||
|
||||
@@ -690,9 +706,8 @@ async fn run_html_export(
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// See run_zip_export: reclaim the superseded generation first, then refuse at the door rather
|
||||
// than ENOSPC mid-write.
|
||||
prune_superseded_archives(pool, export_path, "Memories", event_id, epoch).await;
|
||||
// See run_zip_export: refuse at the door rather than ENOSPC mid-write, and reclaim the
|
||||
// superseded generation only AFTER this one lands.
|
||||
ensure_export_space(pool, event_id, export_path).await?;
|
||||
|
||||
let res = run_html_export_inner(
|
||||
@@ -716,6 +731,13 @@ async fn run_html_export(
|
||||
tokio::fs::remove_dir_all(export_path.join(format!("viewer_tmp_{event_id}_{epoch}")))
|
||||
.await;
|
||||
}
|
||||
|
||||
// Only once the new keepsake exists — see the reasoning in run_zip_export. A failed rebuild
|
||||
// must never be the reason the previous one is gone.
|
||||
if res.is_ok() {
|
||||
prune_superseded_archives(pool, export_path, "Memories", event_id, epoch).await;
|
||||
}
|
||||
|
||||
abandon_if_superseded("HTML", event_id, epoch, res)
|
||||
}
|
||||
|
||||
@@ -1383,18 +1405,32 @@ async fn ensure_export_space(pool: &PgPool, event_id: Uuid, export_path: &Path)
|
||||
return Ok(());
|
||||
};
|
||||
|
||||
if free < needed {
|
||||
// The archive may not consume the last byte of the volume. `needed` alone authorised an
|
||||
// export sized at exactly `free`: it would pass the check, run for half an hour, and land
|
||||
// the box at zero — at which point Postgres cannot write WAL and the event is over, with
|
||||
// the keepsake still unfinished. `postgres_data`, `media_data` and `exports_data` share one
|
||||
// filesystem, so "enough room for the archive" was never the same question as "enough room
|
||||
// for the archive AND a working database".
|
||||
//
|
||||
// Same reserve the upload path and the host dashboard's low-disk banner use, so all three
|
||||
// agree on what "full" means.
|
||||
let required = needed.saturating_add(crate::handlers::upload::DISK_RESERVE_BYTES as u64);
|
||||
|
||||
if free < required {
|
||||
let gb = |b: u64| b as f64 / 1_000_000_000.0;
|
||||
tracing::error!(
|
||||
needed,
|
||||
required,
|
||||
free,
|
||||
armed,
|
||||
"export preflight: not enough free space to build the keepsake for event {event_id}"
|
||||
);
|
||||
anyhow::bail!(
|
||||
"Nicht genug Speicherplatz für das Keepsake: benötigt ca. {:.1} GB, frei sind {:.1} GB. \
|
||||
Bitte Speicher freigeben und das Keepsake anschließend neu erstellen.",
|
||||
"Nicht genug Speicherplatz für das Keepsake: benötigt ca. {:.1} GB (plus {:.0} GB \
|
||||
Reserve), frei sind {:.1} GB. Bitte Speicher freigeben und das Keepsake \
|
||||
anschließend neu erstellen.",
|
||||
gb(needed),
|
||||
gb(crate::handlers::upload::DISK_RESERVE_BYTES as u64),
|
||||
gb(free)
|
||||
);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user