fix: close the nine ways an unattended event loses photos or dies
Every one of these was found in the pre-event audit, verified against source, and
survives to production on the current main. Grouped by what actually goes wrong.
PHOTOS DISAPPEAR
* compression.rs no longer soft-deletes on a failed derivative. The guest got a
201, watched the card appear, then watched it vanish — the row left v_feed,
find_visible_media and BOTH keepsakes, while its bytes sat on disk for 14 days
waiting for a cleanup nothing announced. No screen anywhere lists compression
failures, so recovery meant hand-written SQL that also had to re-add the
refunded quota. Now it does exactly what the ENOSPC arm beside it already did
and documented as correct: keep the row, serve the original, retry on the next
boot (bounded by derivative_attempts). `upload-deleted` is no longer emitted;
`upload-processed` is, so the card re-renders instead of sitting on a
placeholder.
* A 413 is now a reversible lock, so the blob survives. The quota moves — free
disk falls, uploader count rises — so a guest goes over it having done nothing,
and treating that as permanent meant a 400 MB video was pushed across cellular
in full and THEN deleted from IndexedDB. Gone on both sides, and unrecoverable
for an in-app camera capture that exists nowhere else.
* quota_limit_bytes gained a floor and a stable divisor. The ceiling used to
decrease monotonically all evening; it now settles at max(uploaders,
estimated_guest_count) — a config key that was seeded, validated in the admin
whitelist, and read by no code at all. The floor is clamped to what the disk
can actually back, so a full volume still yields zero rather than handing out
an allowance it cannot honour.
* Because that floor gives up the aggregate guarantee the formula used to imply,
uploads now check a hard 10 GB reserve first, independent of every quota
toggle. postgres_data, media_data and exports_data share one filesystem: the
end state was not a degraded feature, it was Postgres unable to write WAL.
THE ARCHIVE DISAPPEARS
* prune_superseded_archives runs only after the new generation lands. It ran
before the preflight, reasoning the old archive was already unreachable — true
of reachability, false of recoverability. An epoch is a value that can be
rolled back; deleted bytes cannot. Any failed rebuild left the event with NO
keepsake at all.
* The export preflight reserves the same 10 GB. `free < needed` authorised an
export sized at exactly free, which ran for half an hour and landed the box at
zero with the keepsake still unfinished.
THE APP DIES
* The feed reconcile re-reads the id set after its awaits instead of reusing one
captured up to three round-trips earlier. The new-upload SSE handler prepends
during exactly that window, so the row was both already present and absent from
the stale set — prepended twice, and a duplicate key in a keyed {#each} throws
in production, not just dev. The SSE handler and loadMore now dedupe too.
* Added routes/+error.svelte. Without it any uncaught error fell through to
SvelteKit's unstyled English 500 with no reload control — inside a chromeless
standalone PWA with no URL bar, for the rest of the evening.
THE OPERATOR IS LOCKED OUT
* admin_login verifies the password BEFORE charging the rate bucket, and a
correct password is never throttled. The old order made this a denial of
service against its own operator: every guest shares one NAT IP, the check ran
first, so five requests a minute from any phone in the room kept the bucket
full — and the escape hatch needed the admin session being blocked. A generous
separate ceiling still bounds bcrypt CPU.
THE PROJECTOR DIES
* The preload budget is now strictly inside the dwell. At the 3s option the 4s
budget could never land a commit on a slow uplink, so the wall froze on one
photo while the queue drained silently behind it.
* The wake lock retries every 30s while visible, and the page says so on screen
when the browser has no wake lock API. visibilitychange was the only retry
trigger and a kiosk never changes visibility, so one refusal — iOS in Low Power
Mode, say — was permanent.
* Caddy: /api/v1/upload/*/display joins the cacheable carve-out. The backend set
max-age=300 on it and the blanket no-store silently replaced it, so a projector
re-fetched a full-size JPEG per slide, ~2-4 GB over an evening on the uplink
the guests are uploading over.
Also removes Upload::soft_delete, now unreferenced and an unscoped footgun next
to soft_delete_in_event.
Verified: 146/146 backend tests against a live Postgres, clippy clean, 51/51
vitest, svelte-check 0 errors, eslint clean, vite build, caddy validate, compose
YAML parse.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -443,6 +443,38 @@ pub async fn upload(
|
||||
// concurrent uploads from the same user (e.g. phone + laptop) both pass this stale
|
||||
// pre-check and both increment, blowing past the quota. The pre-check stays as a
|
||||
// fast path that avoids the disk write when the user is already clearly over.
|
||||
// GLOBAL RESERVE, checked before the per-user ceiling and independent of every quota
|
||||
// toggle. The per-user quota is a fairness mechanism, not a disk guarantee — and since it
|
||||
// now carries a floor (MIN_QUOTA_LIMIT_BYTES) so a guest's allowance stops shrinking as
|
||||
// the party fills up, the aggregate ceiling it used to imply is gone entirely. Something
|
||||
// has to own "do not fill the volume", because `postgres_data`, `media_data` and
|
||||
// `exports_data` share one filesystem: the end state is not a degraded feature, it is
|
||||
// Postgres unable to write WAL and the whole event down with nobody watching.
|
||||
//
|
||||
// Deliberately NOT gated behind `quota_enabled`. That switch exists so an operator can
|
||||
// stop rationing space between guests; it was never meant to authorise running the disk
|
||||
// to zero, and an operator flipping it at 23:00 to unblock a guest should not silently
|
||||
// disarm the last thing standing between the party and a dead database.
|
||||
if let Some(free) = crate::services::disk::free_bytes(&state.config.media_path) {
|
||||
let remaining = (free as i64).saturating_sub(size);
|
||||
if remaining < DISK_RESERVE_BYTES {
|
||||
tracing::error!(
|
||||
free_bytes = free,
|
||||
upload_size = size,
|
||||
reserve = DISK_RESERVE_BYTES,
|
||||
"refusing upload: it would take the media volume below the reserve"
|
||||
);
|
||||
return Err(AppError::QuotaExceeded(
|
||||
"Der Speicher des Events ist voll. Bitte sag einem Host Bescheid — neue \
|
||||
Uploads sind vorübergehend nicht möglich."
|
||||
.into(),
|
||||
));
|
||||
}
|
||||
}
|
||||
// Failing OPEN when the disk can't be read is deliberate and matches the per-user quota
|
||||
// above: refusing every upload because a `statfs` failed would be a worse outage than the
|
||||
// one being guarded against.
|
||||
|
||||
let mut quota_limit: Option<i64> = None;
|
||||
if quota_on && storage_quota_on {
|
||||
let estimate = compute_storage_quota(&state).await;
|
||||
@@ -956,11 +988,52 @@ pub struct QuotaEstimate {
|
||||
pub tolerance: f64,
|
||||
}
|
||||
|
||||
/// Pure per-user quota formula: `floor((free_disk * tolerance) / max(active, 1))`.
|
||||
/// The smallest per-user ceiling this formula is ever allowed to produce.
|
||||
///
|
||||
/// Without a floor the quota is not a limit, it is a moving target: the numerator (free disk)
|
||||
/// only falls and the denominator (uploaders who have posted) only rises, so the ceiling
|
||||
/// decreases monotonically across the event. A guest comfortably under it at 20:00 is over it
|
||||
/// at 22:00 having done nothing, and because a delete refunds the quota but does not free the
|
||||
/// bytes for 24h, the remedy the error message names ("delete older posts") cannot move it
|
||||
/// back either.
|
||||
///
|
||||
/// 500 MB is chosen to clear `max_video_size_mb` (500, seeded in 005) — below that the ceiling
|
||||
/// could refuse a single legal video outright, which is the worst version of this: the guest
|
||||
/// pushes 500 MB across cellular and is rejected on arrival, every time, with no way to comply.
|
||||
///
|
||||
/// This deliberately trades the quota's disk guarantee for a usability floor. The disk is now
|
||||
/// bounded by the low-disk warning and the export preflight rather than by this formula alone —
|
||||
/// see the reserve check in `ensure_export_space`.
|
||||
const MIN_QUOTA_LIMIT_BYTES: i64 = 500 * 1024 * 1024;
|
||||
|
||||
/// Free space on the media volume that uploads may never consume, whatever any quota says.
|
||||
///
|
||||
/// Matches the host dashboard's low-disk threshold (`handlers::host`), so the banner the host
|
||||
/// sees and the wall guests hit are the same number rather than two unrelated opinions. 10 GB
|
||||
/// is chosen to leave Postgres, its WAL and a rotation of container logs comfortable room on
|
||||
/// the shared filesystem long after new uploads have been refused.
|
||||
pub const DISK_RESERVE_BYTES: i64 = 10_000_000_000;
|
||||
|
||||
/// Pure per-user quota formula: `max(floor((free_disk * tolerance) / divisor), MIN)`.
|
||||
///
|
||||
/// `divisor` is the LARGER of the observed uploader count and the operator's
|
||||
/// `estimated_guest_count`, so the ceiling settles at its final value early instead of sliding
|
||||
/// down all evening as guests arrive. (Before this, `estimated_guest_count` was seeded and
|
||||
/// validated in the admin whitelist but read by no code at all — an operator who set it
|
||||
/// expecting a stable divisor changed nothing.) It also blunts the abuse case, where the
|
||||
/// divisor was attacker-controlled: ~1000 throwaway accounts drove every real guest's ceiling
|
||||
/// to ~52 MB.
|
||||
///
|
||||
/// Extracted from `compute_storage_quota` so it's unit-testable without a DB or disk.
|
||||
fn quota_limit_bytes(free_disk: i64, tolerance: f64, active_uploaders: i64) -> i64 {
|
||||
let active = active_uploaders.max(1);
|
||||
((free_disk as f64 * tolerance) / active as f64).floor() as i64
|
||||
fn quota_limit_bytes(free_disk: i64, tolerance: f64, active_uploaders: i64, expected: i64) -> i64 {
|
||||
let divisor = active_uploaders.max(expected).max(1);
|
||||
let budget = (free_disk as f64 * tolerance).max(0.0);
|
||||
let computed = (budget / divisor as f64).floor() as i64;
|
||||
// The floor may never exceed what the disk can actually back. Raising a ceiling the volume
|
||||
// cannot honour would hand out an allowance on a full disk — turning the quota from a
|
||||
// usability floor into a way to finish filling the filesystem that Postgres writes WAL to.
|
||||
let backed_floor = MIN_QUOTA_LIMIT_BYTES.min(budget as i64);
|
||||
computed.max(backed_floor)
|
||||
}
|
||||
|
||||
/// Computes the per-user storage quota using
|
||||
@@ -979,6 +1052,9 @@ pub async fn compute_storage_quota(state: &AppState) -> QuotaEstimate {
|
||||
.await
|
||||
.unwrap_or((0,));
|
||||
let active = active_count.max(1);
|
||||
// The operator's expected headcount, used as a FLOOR on the divisor so the ceiling doesn't
|
||||
// slide down as guests arrive — see `quota_limit_bytes`. Admin-editable at runtime.
|
||||
let expected_guests = config::get_i64(&state.config_cache, "estimated_guest_count", 100).await;
|
||||
|
||||
// Cached disk reading. `None` means we couldn't resolve the media filesystem.
|
||||
let disk = state.disk_cache.snapshot(&state.config.media_path);
|
||||
@@ -986,7 +1062,12 @@ pub async fn compute_storage_quota(state: &AppState) -> QuotaEstimate {
|
||||
|
||||
let limit_bytes = if quota_on && storage_quota_on {
|
||||
match disk {
|
||||
Some(d) => Some(quota_limit_bytes(d.free as i64, tolerance, active)),
|
||||
Some(d) => Some(quota_limit_bytes(
|
||||
d.free as i64,
|
||||
tolerance,
|
||||
active,
|
||||
expected_guests,
|
||||
)),
|
||||
// Fail OPEN, not closed: if the disk can't be read we don't know the real
|
||||
// free space, and enforcing a 0-byte limit would reject every upload with a
|
||||
// spurious "quota reached". Skip enforcement this round and warn instead.
|
||||
@@ -1289,7 +1370,7 @@ pub async fn get_thumbnail(
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{RangeSpec, parse_range, quota_limit_bytes};
|
||||
use super::{MIN_QUOTA_LIMIT_BYTES, RangeSpec, parse_range, quota_limit_bytes};
|
||||
|
||||
// `Range` handling exists because iOS Safari probes every `<video>` with
|
||||
// `Range: bytes=0-1` and abandons the load without a 206. These pin the forms a
|
||||
@@ -1392,33 +1473,70 @@ mod tests {
|
||||
assert_eq!(parse_range(None, 0), RangeSpec::Full);
|
||||
}
|
||||
|
||||
const GB: i64 = 1024 * 1024 * 1024;
|
||||
|
||||
#[test]
|
||||
fn divides_free_space_by_uploaders_with_tolerance() {
|
||||
// 1000 * 0.75 / 3 = 250
|
||||
assert_eq!(quota_limit_bytes(1000, 0.75, 3), 250);
|
||||
// 100 GB * 0.75 / 3 uploaders, well above the floor so the formula shows through.
|
||||
assert_eq!(quota_limit_bytes(100 * GB, 0.75, 3, 1), 26_843_545_600);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn floors_fractional_results() {
|
||||
// 1000 * 0.75 / 7 = 107.14… → 107
|
||||
assert_eq!(quota_limit_bytes(1000, 0.75, 7), 107);
|
||||
// 100 GB * 0.75 / 7 = 11_504_376_685.71… → truncated, not rounded.
|
||||
assert_eq!(quota_limit_bytes(100 * GB, 0.75, 7, 1), 11_504_376_685);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn active_uploaders_below_one_is_clamped_to_one() {
|
||||
fn divisor_below_one_is_clamped_to_one() {
|
||||
// Guards against divide-by-zero when no one has uploaded yet.
|
||||
assert_eq!(quota_limit_bytes(1000, 1.0, 0), 1000);
|
||||
assert_eq!(quota_limit_bytes(1000, 1.0, -5), 1000);
|
||||
assert_eq!(quota_limit_bytes(10 * GB, 1.0, 0, 0), 10 * GB);
|
||||
assert_eq!(quota_limit_bytes(10 * GB, 1.0, -5, 0), 10 * GB);
|
||||
}
|
||||
|
||||
/// The property the floor exists for: a guest's ceiling must not keep shrinking as more
|
||||
/// guests arrive. Same disk, 10 uploaders vs 1000 — the second must not be starved.
|
||||
#[test]
|
||||
fn zero_free_disk_yields_zero() {
|
||||
assert_eq!(quota_limit_bytes(0, 0.75, 3), 0);
|
||||
fn the_ceiling_stops_falling_once_it_reaches_the_floor() {
|
||||
let ten = quota_limit_bytes(70 * GB, 0.75, 10, 1);
|
||||
let thousand = quota_limit_bytes(70 * GB, 0.75, 1000, 1);
|
||||
assert!(ten > MIN_QUOTA_LIMIT_BYTES, "10 uploaders should be roomy");
|
||||
assert_eq!(
|
||||
thousand, MIN_QUOTA_LIMIT_BYTES,
|
||||
"1000 uploaders (or 1000 fake accounts) must not drive the ceiling below the floor"
|
||||
);
|
||||
assert!(
|
||||
thousand >= 500 * 1024 * 1024,
|
||||
"the floor must still clear a single max-size video"
|
||||
);
|
||||
}
|
||||
|
||||
/// `estimated_guest_count` is a FLOOR on the divisor, so the ceiling settles early instead
|
||||
/// of sliding down all evening as guests arrive.
|
||||
#[test]
|
||||
fn expected_headcount_holds_the_divisor_steady_while_guests_arrive() {
|
||||
let early = quota_limit_bytes(70 * GB, 0.75, 5, 100);
|
||||
let late = quota_limit_bytes(70 * GB, 0.75, 100, 100);
|
||||
assert_eq!(
|
||||
early, late,
|
||||
"the 5th guest and the 100th must see the same ceiling"
|
||||
);
|
||||
}
|
||||
|
||||
/// The floor must never write a cheque the volume cannot cash — otherwise a full disk
|
||||
/// still hands out a 500 MB allowance and the filesystem Postgres needs fills up.
|
||||
#[test]
|
||||
fn the_floor_never_exceeds_what_the_disk_can_back() {
|
||||
assert_eq!(quota_limit_bytes(0, 0.75, 3, 1), 0, "no disk, no allowance");
|
||||
// 400 MB free * 0.75 = 300 MB — below the floor, so the disk wins.
|
||||
let tight = quota_limit_bytes(400 * 1024 * 1024, 0.75, 3, 1);
|
||||
assert_eq!(tight, 300 * 1024 * 1024);
|
||||
assert!(tight < MIN_QUOTA_LIMIT_BYTES);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn full_tolerance_is_identity_for_a_single_uploader() {
|
||||
assert_eq!(quota_limit_bytes(500, 1.0, 1), 500);
|
||||
assert_eq!(quota_limit_bytes(50 * GB, 1.0, 1, 1), 50 * GB);
|
||||
}
|
||||
|
||||
/// The guard is the only reclaim mechanism that survives a client disconnect, so its three
|
||||
|
||||
Reference in New Issue
Block a user