fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: gate uploads on keepsake headroom, and close five unattended-event gaps

Browse Source 
The box is 2 vCPU / 4 GB / 40 GB, not the 4 vCPU / 8 GB / 80 GB that the audit,
the committed comments and README's sizing section all assumed. That correction
is what the first change is about; the rest are the remaining pre-event items.

THE ARCHIVE COULD BECOME UNBUILDABLE WHILE UPLOADS KEPT SUCCEEDING

`required_free_bytes` is `media × 1.1 × 2` — the ZIP and the HTML viewer are each
gallery-sized — and the export preflight also wants DISK_RESERVE_BYTES on top. The
upload gate, though, only refused below a FLAT 10 GB reserve. On 40 GB that let
uploads run to ~25 GB of media while a release needed `2.2 × 25 + 10` = 65 GB free.
Every upload in that band succeeded and the keepsake could then never be built: the
product's entire promise, failing silently at the end of the night with nobody there.

The gate now enforces the invariant that actually matters — never accept an upload
that would make the keepsake unbuildable — sharing `required_free_bytes` with the
preflight so the two cannot drift into disagreeing about the same question. Uploads
stop at ~8 GB of media on this disk, with a German message naming the cause.
Refusing the 1001st photo beats losing all 1000.

`media_total.rs` backs it: SUM(user.total_upload_bytes) over ~100 rows, cached 5s,
rather than `estimate_export_bytes`'s join across every upload. It counts hidden and
banned users' bytes, which the export excludes — skew in the SAFE direction, so the
gate closes marginally early rather than late. Fails open on a query error.

A test pins the gate against the preflight across the whole gallery-size range, and
a second asserts the per-user floor alone would over-commit the volume — i.e. that
the global gate is what must bind.

THE WATCHDOG ABORTED HEALTHY UPLOADS EVERY TIME A PHONE WAS POCKETED

`Date.now()` advances while a backgrounded phone is frozen but `setInterval` does
not, so the first tick after a screen lock read the whole sleep as silence and
aborted — re-sending a video from byte zero and burning one of five PERMANENT
auto-attempts. The interval is now its own suspension detector: a tick that arrives
125s late for a 5s schedule credits that window back, because a period the watchdog
could not observe is not evidence of silence.

Chosen over a `visibilitychange` listener, which only covers causes that fire that
event — a throttled-but-visible tab, a closed lid and an occluded window all freeze
timers without one — and which would have needed module state, an SSR guard and a
teardown for strictly less coverage. `performance.now()` was rejected because Safari
pauses it across system sleep on some paths and Chrome does not.

The credit buys one fresh window, not immunity: a socket iOS reaped while
backgrounded still aborts ~90s after resume rather than hanging for `xhr.timeout`
(5-60 min) with the queue's `processing` latch held.

Two latent leaks found while in there: `xhr.abort()` on a request already in
readyState DONE emits no `abort` event, so `settle()` never ran and the interval
re-aborted every 5s forever while `activeUploads` kept a stale entry (the ✕ button
silently stopped working); and a synchronous throw from `xhr.send` — a blob whose
backing store the OS purged — leaked the same way. Both closed.

OKLCH MADE THE DELETE BUTTON INVISIBLE ON SAMSUNG'S DEFAULT BROWSER

red/amber/green were never in the @theme block and fell through to Tailwind v4's
`oklch()` defaults, which Safari <15.4, Chrome <111 and Samsung Internet <22 cannot
parse: `var(--color-red-600)` is then invalid at computed-value time, `background-color`
falls back to transparent, and `.btn-danger` renders white text on nothing. Pinned to
Tailwind's own defaults gamut-mapped to sRGB by Lightning CSS — the converter already
in this pipeline — so modern browsers render exactly what they render today. Verified
against seven hex fallbacks it had already emitted for the /alpha forms. rose and teal
(avatar chips) had the same leak. The app CSS goes from 40 oklch declarations to 0.

Also fixes `--color-purple-950`, which was simply missing: `dark:bg-purple-950/50` on
the host dashboard was rendering default violet on EVERY browser, off-brand.

The keepsake viewer only picks this up on a rebuild, so its committed artefact is
rebuilt here too — still single-file, still zero external references.

A BRICKED BOOT LOOKED LIKE A SPINNER FOREVER

With `ssr = false` the page is empty until the bundle mounts, so a chunk 404 after a
redeploy or a dead uplink left the guest on the boot spinner with no message, no
reload control, and in a standalone PWA no URL bar. A 15s timeout in the existing
nonce'd IIFE (no CSP change) swaps in German copy and a reload button. Deliberately a
timeout rather than feature detection: a SyntaxError in the bundle is invisible to any
capability check. Plus a <noscript>, since there was nothing at all to see without JS.

EVERY 4xx WAS INVISIBLE AT ANY LOG LEVEL

tower_http counts 4xx as a success, so it logs at DEBUG while production runs at info.
If guests spend the evening hitting 429s or 413s, the post-event logs said nothing.
Now one WARN per client error; 5xx excluded because Internal already logs its source
chain and the pool-exhaustion 503 logs at construction.

A DEAD FRONTEND SERVED A BLANK 502

`handle_errors 5xx` with an inline German page (the caddy service mounts only the
Caddyfile, so there is no volume to ship a static file through). Verified empirically
against this config, not from documentation: an upstream 404 through `reverse_proxy`
still arrives as untouched `application/json`, and only a dial failure renders the
page. That mattered — the keepsake download navigates a hidden iframe and DEPENDS on a
real 404/429 arriving, and swallowing those would have been worse than the blank 502.

CONFIG CORRECTIONS FOR THE REAL HARDWARE

DATABASE_MAX_CONNECTIONS 30 → 15: sized to 2 vCPU rather than to the guest count.
Since migration 024 a feed page costs well under a millisecond, so connections are no
longer spent waiting, and 30 backends crowd the db container's 1 GB on a 4 GB host.
COMPRESSION_WORKER_CONCURRENCY stays at 2 — the merged heavy-image permit already
serialises anything over 150 MiB, so the "two 48 MP photos" worst case that number was
sized against is unreachable; dropping to 1 would halve light-path throughput and push
more feed tiles onto full-size originals. README's sizing section rewritten for the
actual disk.

Verified: 149/149 backend tests against a live Postgres, clippy clean, 57/57 vitest,
svelte-check 0 errors, eslint clean, vite build, export-viewer rebuild, caddy validate,
compose YAML parse.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-08-08 22:07:56 +02:00
parent 1d9fb11c7b
commit eb0e405562
16 changed files with 624 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files

126
backend/src/handlers/upload.rs
View File

@@ -438,43 +438,62 @@ pub async fn upload(
let quota_on = config::get_bool(&state.config_cache, "quota_enabled", true).await;
let storage_quota_on =
config::get_bool(&state.config_cache, "storage_quota_enabled", true).await;
// When quota is enforced, this holds the byte ceiling so the increment UPDATE below can
// enforce it atomically (`WHERE total + size <= limit`). Without that guard, two
// concurrent uploads from the same user (e.g. phone + laptop) both pass this stale
// pre-check and both increment, blowing past the quota. The pre-check stays as a
// fast path that avoids the disk write when the user is already clearly over.
// GLOBAL RESERVE, checked before the per-user ceiling and independent of every quota
// GLOBAL DISK GATE, checked before the per-user ceiling and independent of every quota
// toggle. The per-user quota is a fairness mechanism, not a disk guarantee — and since it
// now carries a floor (MIN_QUOTA_LIMIT_BYTES) so a guest's allowance stops shrinking as
// the party fills up, the aggregate ceiling it used to imply is gone entirely. Something
// has to own "do not fill the volume", because `postgres_data`, `media_data` and
// `exports_data` share one filesystem: the end state is not a degraded feature, it is
// Postgres unable to write WAL and the whole event down with nobody watching.
// carries a floor (MIN_QUOTA_LIMIT_BYTES) so a guest's allowance stops shrinking as the
// party fills up, the aggregate ceiling it used to imply is gone entirely. Something has to
// own "do not fill the volume", because `postgres_data`, `media_data` and `exports_data`
// share one filesystem: the end state is not a degraded feature, it is Postgres unable to
// write WAL and the whole event down with nobody watching.
//
// Deliberately NOT gated behind `quota_enabled`. That switch exists so an operator can
// stop rationing space between guests; it was never meant to authorise running the disk
// to zero, and an operator flipping it at 23:00 to unblock a guest should not silently
// disarm the last thing standing between the party and a dead database.
// WHAT IS RESERVED IS NOT A CONSTANT. A flat reserve answers "can Postgres still write",
// which is necessary and not sufficient: the keepsake needs room for BOTH halves at once —
// `required_free_bytes` is `media × 1.1 × 2`, since the ZIP and the HTML viewer are each
// gallery-sized. On the 40 GB box this runs on, a flat 10 GB reserve let uploads continue to
// roughly 25 GB of media while the release needed `2.2 × 25 + 10` = 65 GB free. Every upload
// in that band succeeded and then the archive could never be built — the product's entire
// promise, failing silently at the end of the night with nobody there to notice.
//
// So the gate enforces the invariant that actually matters: never accept an upload that
// would make the keepsake unbuildable. It shares `required_free_bytes` with the export
// preflight so the two cannot drift into disagreeing about the same question.
//
// Deliberately NOT gated behind `quota_enabled`. That switch exists so an operator can stop
// rationing space between guests; it was never meant to authorise running the disk to zero,
// and an operator flipping it at 23:00 to unblock a guest should not silently disarm the
// last thing standing between the party and a dead database.
if let Some(free) = crate::services::disk::free_bytes(&state.config.media_path) {
let remaining = (free as i64).saturating_sub(size);
if remaining < DISK_RESERVE_BYTES {
let media_after = state.media_total.get(&state.pool).await.saturating_add(size);
let keepsake_needs =
crate::services::export::required_free_bytes(media_after.max(0) as u64, 2) as i64;
let free_after = (free as i64).saturating_sub(size);
let required = keepsake_needs.saturating_add(DISK_RESERVE_BYTES);
if free_after < required {
tracing::error!(
free_bytes = free,
upload_size = size,
media_after,
keepsake_needs,
reserve = DISK_RESERVE_BYTES,
"refusing upload: it would take the media volume below the reserve"
"refusing upload: it would leave too little room to build the keepsake"
);
return Err(AppError::QuotaExceeded(
"Der Speicher des Events ist voll. Bitte sag einem Host Bescheid — neue \
Uploads sind vorübergehend nicht möglich."
"Der Speicher des Events ist fast voll — damit die Galerie am Ende noch als \
Download erstellt werden kann, sind neue Uploads jetzt gesperrt. Bitte sag \
einem Host Bescheid."
.into(),
));
}
}
// Failing OPEN when the disk can't be read is deliberate and matches the per-user quota
// above: refusing every upload because a `statfs` failed would be a worse outage than the
// below: refusing every upload because a `statfs` failed would be a worse outage than the
// one being guarded against.
// When quota is enforced, this holds the byte ceiling so the increment UPDATE below can
// enforce it atomically (`WHERE total + size <= limit`). Without that guard, two
// concurrent uploads from the same user (e.g. phone + laptop) both pass this stale
// pre-check and both increment, blowing past the quota. The pre-check stays as a
// fast path that avoids the disk write when the user is already clearly over.
let mut quota_limit: Option<i64> = None;
if quota_on && storage_quota_on {
let estimate = compute_storage_quota(&state).await;
@@ -1370,7 +1389,9 @@ pub async fn get_thumbnail(
#[cfg(test)]
mod tests {
use super::{MIN_QUOTA_LIMIT_BYTES, RangeSpec, parse_range, quota_limit_bytes};
use super::{
DISK_RESERVE_BYTES, MIN_QUOTA_LIMIT_BYTES, RangeSpec, parse_range, quota_limit_bytes,
};
// `Range` handling exists because iOS Safari probes every `<video>` with
// `Range: bytes=0-1` and abandons the load without a 206. These pin the forms a
@@ -1523,6 +1544,67 @@ mod tests {
);
}
/// THE INVARIANT THE UPLOAD GATE EXISTS FOR: if an upload is accepted, the keepsake must
/// still be buildable afterwards.
///
/// The gate and `ensure_export_space` answer the same question at different times, from the
/// same `required_free_bytes`. If they ever drift, the failure is silent and terminal — every
/// upload succeeds and the archive can never be built, discovered only when the host taps
/// release and there is nobody left to fix it. This pins the two together.
///
/// Models the real box: 40 GB volume, ~5 GB consumed by OS, images and Postgres.
#[test]
fn an_accepted_upload_always_leaves_room_to_build_the_keepsake() {
const USABLE: i64 = 35 * GB;
let reserve = DISK_RESERVE_BYTES;
// Walk the gallery upward in 250 MB steps and assert the two agree at every point.
let mut media: i64 = 0;
let step: i64 = 250 * 1024 * 1024;
let mut last_accepted = 0i64;
while media < USABLE {
let free = USABLE - media;
let media_after = media + step;
let free_after = free - step;
let required =
crate::services::export::required_free_bytes(media_after as u64, 2) as i64 + reserve;
let gate_accepts = free_after >= required;
if gate_accepts {
// The export preflight must agree, using the SAME arithmetic it will run later.
let preflight_needs =
crate::services::export::required_free_bytes(media_after as u64, 2) as i64
+ reserve;
assert!(
free_after >= preflight_needs,
"gate accepted at media={media_after} but the preflight would refuse"
);
last_accepted = media_after;
}
media = media_after;
}
// Sanity-check the ceiling is where the arithmetic says: 35 = 2.2·M + 10 ⇒ M ≈ 7.8 GB.
// Pinned loosely (6–9 GB) so a deliberate change to the overhead multiplier or the
// reserve fails this test loudly rather than silently moving the cliff.
assert!(
(6 * GB..=9 * GB).contains(&last_accepted),
"expected the gallery ceiling near 7.8 GB on a 35 GB volume, got {last_accepted} bytes"
);
}
/// The gate must be the binding constraint, not the per-user floor. With 100 guests each
/// allowed 500 MB, the per-user quota alone would authorise ~50 GB on a 40 GB disk.
#[test]
fn the_global_gate_binds_before_the_per_user_floor_can_overfill_the_disk() {
let per_user_total = MIN_QUOTA_LIMIT_BYTES * 100;
assert!(
per_user_total > 35 * GB,
"premise: the per-user floor alone over-commits the volume, so the global gate \
is what must stop it"
);
}
/// The floor must never write a cheque the volume cannot cash — otherwise a full disk
/// still hands out a 500 MB allowance and the filesystem Postgres needs fills up.
#[test]