fix: gate uploads on keepsake headroom, and close five unattended-event gaps

The box is 2 vCPU / 4 GB / 40 GB, not the 4 vCPU / 8 GB / 80 GB that the audit,
the committed comments and README's sizing section all assumed. That correction
is what the first change is about; the rest are the remaining pre-event items.

THE ARCHIVE COULD BECOME UNBUILDABLE WHILE UPLOADS KEPT SUCCEEDING

`required_free_bytes` is `media × 1.1 × 2` — the ZIP and the HTML viewer are each
gallery-sized — and the export preflight also wants DISK_RESERVE_BYTES on top. The
upload gate, though, only refused below a FLAT 10 GB reserve. On 40 GB that let
uploads run to ~25 GB of media while a release needed `2.2 × 25 + 10` = 65 GB free.
Every upload in that band succeeded and the keepsake could then never be built: the
product's entire promise, failing silently at the end of the night with nobody there.

The gate now enforces the invariant that actually matters — never accept an upload
that would make the keepsake unbuildable — sharing `required_free_bytes` with the
preflight so the two cannot drift into disagreeing about the same question. Uploads
stop at ~8 GB of media on this disk, with a German message naming the cause.
Refusing the 1001st photo beats losing all 1000.

`media_total.rs` backs it: SUM(user.total_upload_bytes) over ~100 rows, cached 5s,
rather than `estimate_export_bytes`'s join across every upload. It counts hidden and
banned users' bytes, which the export excludes — skew in the SAFE direction, so the
gate closes marginally early rather than late. Fails open on a query error.

A test pins the gate against the preflight across the whole gallery-size range, and
a second asserts the per-user floor alone would over-commit the volume — i.e. that
the global gate is what must bind.

THE WATCHDOG ABORTED HEALTHY UPLOADS EVERY TIME A PHONE WAS POCKETED

`Date.now()` advances while a backgrounded phone is frozen but `setInterval` does
not, so the first tick after a screen lock read the whole sleep as silence and
aborted — re-sending a video from byte zero and burning one of five PERMANENT
auto-attempts. The interval is now its own suspension detector: a tick that arrives
125s late for a 5s schedule credits that window back, because a period the watchdog
could not observe is not evidence of silence.

Chosen over a `visibilitychange` listener, which only covers causes that fire that
event — a throttled-but-visible tab, a closed lid and an occluded window all freeze
timers without one — and which would have needed module state, an SSR guard and a
teardown for strictly less coverage. `performance.now()` was rejected because Safari
pauses it across system sleep on some paths and Chrome does not.

The credit buys one fresh window, not immunity: a socket iOS reaped while
backgrounded still aborts ~90s after resume rather than hanging for `xhr.timeout`
(5-60 min) with the queue's `processing` latch held.

Two latent leaks found while in there: `xhr.abort()` on a request already in
readyState DONE emits no `abort` event, so `settle()` never ran and the interval
re-aborted every 5s forever while `activeUploads` kept a stale entry (the ✕ button
silently stopped working); and a synchronous throw from `xhr.send` — a blob whose
backing store the OS purged — leaked the same way. Both closed.

OKLCH MADE THE DELETE BUTTON INVISIBLE ON SAMSUNG'S DEFAULT BROWSER

red/amber/green were never in the @theme block and fell through to Tailwind v4's
`oklch()` defaults, which Safari <15.4, Chrome <111 and Samsung Internet <22 cannot
parse: `var(--color-red-600)` is then invalid at computed-value time, `background-color`
falls back to transparent, and `.btn-danger` renders white text on nothing. Pinned to
Tailwind's own defaults gamut-mapped to sRGB by Lightning CSS — the converter already
in this pipeline — so modern browsers render exactly what they render today. Verified
against seven hex fallbacks it had already emitted for the /alpha forms. rose and teal
(avatar chips) had the same leak. The app CSS goes from 40 oklch declarations to 0.

Also fixes `--color-purple-950`, which was simply missing: `dark:bg-purple-950/50` on
the host dashboard was rendering default violet on EVERY browser, off-brand.

The keepsake viewer only picks this up on a rebuild, so its committed artefact is
rebuilt here too — still single-file, still zero external references.

A BRICKED BOOT LOOKED LIKE A SPINNER FOREVER

With `ssr = false` the page is empty until the bundle mounts, so a chunk 404 after a
redeploy or a dead uplink left the guest on the boot spinner with no message, no
reload control, and in a standalone PWA no URL bar. A 15s timeout in the existing
nonce'd IIFE (no CSP change) swaps in German copy and a reload button. Deliberately a
timeout rather than feature detection: a SyntaxError in the bundle is invisible to any
capability check. Plus a <noscript>, since there was nothing at all to see without JS.

EVERY 4xx WAS INVISIBLE AT ANY LOG LEVEL

tower_http counts 4xx as a success, so it logs at DEBUG while production runs at info.
If guests spend the evening hitting 429s or 413s, the post-event logs said nothing.
Now one WARN per client error; 5xx excluded because Internal already logs its source
chain and the pool-exhaustion 503 logs at construction.

A DEAD FRONTEND SERVED A BLANK 502

`handle_errors 5xx` with an inline German page (the caddy service mounts only the
Caddyfile, so there is no volume to ship a static file through). Verified empirically
against this config, not from documentation: an upstream 404 through `reverse_proxy`
still arrives as untouched `application/json`, and only a dial failure renders the
page. That mattered — the keepsake download navigates a hidden iframe and DEPENDS on a
real 404/429 arriving, and swallowing those would have been worse than the blank 502.

CONFIG CORRECTIONS FOR THE REAL HARDWARE

DATABASE_MAX_CONNECTIONS 30 → 15: sized to 2 vCPU rather than to the guest count.
Since migration 024 a feed page costs well under a millisecond, so connections are no
longer spent waiting, and 30 backends crowd the db container's 1 GB on a 4 GB host.
COMPRESSION_WORKER_CONCURRENCY stays at 2 — the merged heavy-image permit already
serialises anything over 150 MiB, so the "two 48 MP photos" worst case that number was
sized against is unreachable; dropping to 1 would halve light-path throughput and push
more feed tiles onto full-size originals. README's sizing section rewritten for the
actual disk.

Verified: 149/149 backend tests against a live Postgres, clippy clean, 57/57 vitest,
svelte-check 0 errors, eslint clean, vite build, export-viewer rebuild, caddy validate,
compose YAML parse.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-08-08 22:07:56 +02:00
parent 1d9fb11c7b
commit eb0e405562
16 changed files with 624 additions and 64 deletions

View File

@@ -36,13 +36,20 @@ DATABASE_URL=postgres://eventsnap:CHANGE_ME_use_a_strong_password@db:5432/events
POSTGRES_USER=eventsnap
POSTGRES_PASSWORD=CHANGE_ME_use_a_strong_password
POSTGRES_DB=eventsnap
# Connection pool size. Default 10. For a busy event (~100 guests polling the feed
# + SSE + uploads at once) raise to ~30 so requests don't queue on a pool permit.
# PAIRED WITH THE DB CONTAINER'S MEMORY LIMIT: 30 backends plus Postgres 16's default
# shared_buffers is already snug in the 1G that docker-compose.yml allots the `db`
# service. If you raise this, raise `db.deploy.resources.limits.memory` with it — an
# OOM in Postgres doesn't degrade one feature, it takes the whole event down.
DATABASE_MAX_CONNECTIONS=30
# Connection pool size. The code default is 10 (backend/src/db.rs) — set it explicitly,
# because a `.env` written by hand from this file's secrets is otherwise silently on 10.
#
# SIZE IT TO THE CORES, NOT TO THE GUESTS. The earlier advice here was ~30, reasoned from
# "~100 guests polling the feed at once" back when a feed page cost ~449 ms and connections
# were spent waiting. Migration 024 replaced the feed view's GROUP BY with scalar subqueries
# and a page now costs well under a millisecond, so concurrency is no longer where the time
# goes. On a 2 vCPU box 30 simultaneous queries cannot run — they queue on the CPU instead of
# on the pool, which is the same wait wearing a different hat, and 30 Postgres backends plus
# shared_buffers is snug in the 1G that docker-compose.yml allots `db`.
#
# 15 on 2 vCPU / 4 GB. Raise toward 30 only alongside more cores AND a bigger `db` memory
# limit — an OOM in Postgres doesn't degrade one feature, it takes the whole event down.
DATABASE_MAX_CONNECTIONS=15
# Log level. `info` is the right production default: at `debug` the tower-http trace
# layer writes a line per request AND per response, which on a busy event is a large
@@ -124,12 +131,23 @@ EXPORT_PATH=/exports
# display resize: ~145 MB at 12 MP, ~223 MB at 24 MP, ~354 MB at 48 MP.
#
# So on a 2 vCPU / 4 GB box (e.g. Hetzner CX22) KEEP THIS AT 2:
# * concurrency 2, two 48 MP photos ≈ 800 MB against the 1G app limit — ~25% margin.
# * concurrency 4, the same pair ≈ 1.5 GB — OOM.
# * concurrency 4 would put two giants at ~1.5 GB against the 1G app limit — OOM.
# * and app=2G + db=1G + frontend/caddy 256M each + ~370 MB of OS/Docker exceeds the
# ~3910 MiB a "4 GB" VM actually reports. Raising the limit oversubscribes the host.
# 4 is only reasonable on the 4 vCPU / 8 GB box README.md documents.
#
# The "two 48 MP photos at once" worst case this number used to be sized against is no
# longer reachable: compression.rs takes an EXCLUSIVE `heavy` permit for any job whose
# estimated peak exceeds HEAVY_IMAGE_BYTES (150 MiB), so two giants serialise no matter what
# this is set to. What concurrency 2 now buys is two ORDINARY phone photos in parallel
# (~145 MB peak each), which is both memory-safe and short enough not to starve the two
# tokio worker threads a 2 vCPU box gets.
#
# Do NOT drop this to 1 hoping to protect the CPU. It halves throughput on the common light
# path for a heavy path that is already serialised, and a longer compression backlog means
# more feed tiles served from full-size originals (VirtualFeed falls back to /original while
# derivatives are pending) — trading a little CPU for a lot of venue-wifi bandwidth.
#
# Throughput at 2 is not the bottleneck anyone thinks it is: ~2.5s per 12 MP photo, so
# 100 photos is ~250 CPU-seconds spread over an entire evening.
COMPRESSION_WORKER_CONCURRENCY=2

View File

@@ -75,4 +75,59 @@
# Everything else goes to SvelteKit frontend
reverse_proxy frontend:3001
# Last-resort page for when Caddy itself cannot reach an upstream — the app or frontend
# container down, restarting, or still warming up after a host reboot. Without it a guest
# gets Caddy's bodiless 502: a completely blank page, which reads as "the whole thing is
# gone" rather than "try again in a moment".
#
# THIS DOES NOT TOUCH APPLICATION ERRORS. `handle_errors` fires only on errors CADDY
# generates; a status the app returns through `reverse_proxy` is written back verbatim and
# never reaches here. That distinction is load-bearing rather than incidental: the keepsake
# download navigates a HIDDEN IFRAME and depends on a real 404/429 arriving from the app
# (frontend/src/routes/export/+page.svelte), and every API route answers 403/404/429 as
# ordinary JSON that the client parses. Swallowing those into an HTML page would be a far
# worse regression than the blank 502 this fixes. Verified against this exact config: an
# upstream 404 through `reverse_proxy` still arrives as `Content-Type: application/json`
# with its body intact, while only a dial failure renders the page below.
#
# Scoped to 5xx so a hypothetical future Caddy-generated 4xx (there is none today) still
# returns plainly instead of claiming the server is restarting.
#
# The body is inline because the caddy service mounts ONLY ./Caddyfile and caddy_data —
# there is no volume to ship an HTML file through and the image has no build step, so a
# static file would mean changing the deployed stack's compose definition. No external
# font, stylesheet or image is referenced: the app may be exactly what is down.
#
# `handle_errors` has NO position in the directive order — Caddy hoists it into a separate
# `errors` route list — so it cannot disturb the "first `header` directive wins" hazard
# documented at the top of this file. The site-wide security headers still apply to it.
handle_errors 5xx {
header Content-Type "text/html; charset=utf-8"
header Cache-Control "no-store"
# {err.status_code} preserves the real status. Hardcoding 503 would mislabel a genuine
# 502 for anything watching from outside.
respond `<!doctype html>
<html lang="de">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Gleich zurück</title>
<style>
html{background:#faf9f7;color:#1a1918;font-family:system-ui,-apple-system,"Segoe UI",Roboto,sans-serif}
body{margin:0;min-height:100vh;display:flex;align-items:center;justify-content:center;padding:2rem;text-align:center}
h1{font-family:Georgia,"Times New Roman",serif;font-weight:600;font-size:1.5rem;margin:0 0 .75rem}
p{margin:0;color:#545350;line-height:1.5}
@media (prefers-color-scheme:dark){html{background:#100f0f;color:#f5f4f2}p{color:#a6a4a1}}
</style>
</head>
<body>
<main>
<h1>Wir sind gleich zurück</h1>
<p>Die Seite wird gerade neu gestartet.<br>Bitte lade in einem Moment neu deine Fotos bleiben gespeichert.</p>
</main>
</body>
</html>
` {err.status_code}
}
}

View File

@@ -280,35 +280,49 @@ so a host takedown or a ban actually revokes access to the bytes.
degrade one subsystem — Postgres stops being able to write and the whole event goes
down.
Uploads are self-limiting. `per_user_limit = free_disk × quota_tolerance ÷
active_uploaders` is recomputed against live free space on every upload, so guests
converge on a fixed point at `tolerance / (1 + tolerance)` of the free space you
started with — **43%** at the default 0.75. On an 80 GB box with ~70 GB free after
the OS and images, media settles at ~30 GB and stops.
**`Gallery.zip` and `Memories.zip` are each roughly a second copy of every original.**
Both write their media `Compression::Stored`, and `Memories.zip` streams the untouched
original for every video and for every image at or under 5 MB. So a release wants room
for **two more copies of the gallery** on top of the gallery itself — which is what
`required_free_bytes` encodes as `media × 1.1 × 2`.
**The keepsake is what the 80 GB baseline does not cover.** `Gallery.zip` and
`Memories.zip` are built concurrently and each is roughly a second copy of every
original: both write their media `Compression::Stored`, and `Memories.zip` streams the
untouched original for every video and for every image at or under 5 MB. So a release
wants room for **two more copies of the gallery** on top of the gallery itself.
The per-user quota does **not** bound this. It is a fairness mechanism that divides
free space between guests, and since it carries a floor (`MIN_QUOTA_LIMIT_BYTES`, so a
guest's allowance stops shrinking as the party fills up) the aggregate ceiling it used
to imply is gone. What bounds the disk is the **global gate in the upload handler**,
which refuses any upload that would leave too little room to build the keepsake:
| Stage | Used | Free (80 GB box) |
|---|---|---|
| Fresh box (OS + images) | ~10 GB | ~70 GB |
| Guests reach the quota fixed point | ~40 GB | ~40 GB |
| Host releases → both archives | ~100 GB | **ENOSPC** |
```
free_after_upload < media_after × 1.1 × 2 + DISK_RESERVE_BYTES → refused
```
Two ways to size for it:
Solving that for the gallery size gives the real ceiling. On the **40 GB box this runs
on**, with ~5 GB for the OS, Docker images (the runbook pre-pulls the rollback tag too)
and Postgres:
| Volume | Usable after baseline | Media ceiling | Free at release |
|---|---|---|---|
| 40 GB | ~35 GB | **~8 GB** | ~27 GB → both archives fit |
| 80 GB | ~70 GB | ~19 GB | ~51 GB → both archives fit |
**Uploads therefore stop at roughly 8 GB of media on a 40 GB box, not when the disk is
full.** That is deliberate. 1000 photos at ~3.5 MB is ~3.5 GB and fits comfortably;
video is what consumes the budget, so lower `max_video_size_mb` (seeded at 500) if you
expect a lot of it. Refusing the 1001st upload is a far better outcome than accepting it
and discovering at 01:00 that the archive can never be built.
Two ways to buy headroom:
- **Provision ~3× your expected media** on one volume (media + two archives), or
- **give `exports_data` its own volume** so a full export cannot reach Postgres, and
size that one at ~2× expected media.
This is no longer silent. The export refuses up front with the two numbers rather than
hitting ENOSPC halfway through a multi-GB write, a rebuild reclaims the superseded
generation before it starts (so peak is one generation, not two), and the host
dashboard warns as soon as the keepsake would not fit — which is the only point at
which anyone can still do something about it.
None of this is silent. The upload gate refuses with a German message naming the cause,
the export preflight refuses up front with both numbers rather than hitting ENOSPC
halfway through a multi-GB write, a rebuild only reclaims the superseded generation
**after** the new one lands (so a failed rebuild can never leave you with no archive at
all), and the host dashboard warns as soon as the keepsake would not fit — which is the
only point at which anyone can still do something about it.
---

View File

@@ -78,6 +78,26 @@ impl IntoResponse for AppError {
};
let message = self.message();
// Log every 4xx. Until now they were invisible at ANY log level: tower_http's
// `ServerErrorsAsFailures` classifier counts a 4xx as a *success*, so it goes to
// `DefaultOnResponse` at DEBUG, and production runs at `info`. The consequence is that a
// misconfigured limit leaves no trace at all — if guests spend the evening hitting 429s
// on `upload_rate_per_hour`, or 413s on the storage quota, `docker compose logs` after
// the event contains nothing about it and the cause is unknowable.
//
// WARN rather than INFO because every variant here is a request that did not do what
// the guest asked. 5xx is excluded: `Internal` already logs with its full source chain
// in `message()` above, and the pool-exhaustion 503 logs at construction — logging again
// here would double every server-side failure.
//
// No request context is available: `into_response` receives only the error, so there is
// no path, method or user id to attach. Status + code + message is what can honestly be
// reported from here, and it is enough to see the SHAPE of a bad evening. Raising
// `tower_http` to DEBUG instead was considered and rejected — see the note in main.rs.
if status.is_client_error() {
tracing::warn!(status = status.as_u16(), code, %message, "request rejected");
}
let mut body = serde_json::json!({
"error": code,
"message": message,
@@ -161,6 +181,43 @@ mod tests {
}
}
/// 4xx must be logged and 5xx must not be logged HERE — `Internal` logs its source chain in
/// `message()` and the pool-exhaustion 503 logs at construction, so a second line in
/// `into_response` would double every server-side failure in the post-event logs.
///
/// The guard is `status.is_client_error()`, so this pins the classification rather than the
/// logging itself (which needs a subscriber to observe).
#[test]
fn only_client_errors_are_in_the_logged_band() {
for err in [
AppError::BadRequest("x".into()),
AppError::Unauthorized("x".into()),
AppError::Forbidden("x".into()),
AppError::UploadsLocked("x".into()),
AppError::NotFound("x".into()),
AppError::Conflict("x".into()),
AppError::TooManyRequests("x".into(), Some(1)),
AppError::QuotaExceeded("x".into()),
] {
let (status, _) = err.status_and_code();
assert!(
status.is_client_error(),
"{status} should be in the 4xx band this logs"
);
}
for err in [
AppError::ServiceUnavailable("x".into(), Some(3)),
AppError::Internal(anyhow::anyhow!("boom")),
] {
let (status, _) = err.status_and_code();
assert!(
!status.is_client_error(),
"{status} logs elsewhere; logging it here would double it"
);
}
}
/// Pool saturation is load, not a bug. A 500 makes the frontend's retry classifier pile
/// straight back into the saturated pool with no backoff to pace it.
#[test]

View File

@@ -111,6 +111,12 @@ pub async fn truncate_all(
// steers the per-user limit off `free_disk_bytes`), i.e. two holes were masking each other.
state.disk_cache.invalidate();
// `media_total` caches SUM(user.total_upload_bytes) for the upload gate's keepsake-headroom
// check. TRUNCATE has just zeroed every one of those rows, so a surviving reading would make
// the next test's first upload measure its headroom against the previous test's gallery —
// and that gate REFUSES uploads, so the failure would look like a spurious quota rejection.
state.media_total.invalidate();
// `sse_tickets` maps a ticket to a session token hash. TRUNCATE deletes the sessions, so every
// surviving ticket is a dangling reference to a user that no longer exists.
state.sse_tickets.clear();

View File

@@ -438,43 +438,62 @@ pub async fn upload(
let quota_on = config::get_bool(&state.config_cache, "quota_enabled", true).await;
let storage_quota_on =
config::get_bool(&state.config_cache, "storage_quota_enabled", true).await;
// When quota is enforced, this holds the byte ceiling so the increment UPDATE below can
// enforce it atomically (`WHERE total + size <= limit`). Without that guard, two
// concurrent uploads from the same user (e.g. phone + laptop) both pass this stale
// pre-check and both increment, blowing past the quota. The pre-check stays as a
// fast path that avoids the disk write when the user is already clearly over.
// GLOBAL RESERVE, checked before the per-user ceiling and independent of every quota
// GLOBAL DISK GATE, checked before the per-user ceiling and independent of every quota
// toggle. The per-user quota is a fairness mechanism, not a disk guarantee — and since it
// now carries a floor (MIN_QUOTA_LIMIT_BYTES) so a guest's allowance stops shrinking as
// the party fills up, the aggregate ceiling it used to imply is gone entirely. Something
// has to own "do not fill the volume", because `postgres_data`, `media_data` and
// `exports_data` share one filesystem: the end state is not a degraded feature, it is
// Postgres unable to write WAL and the whole event down with nobody watching.
// carries a floor (MIN_QUOTA_LIMIT_BYTES) so a guest's allowance stops shrinking as the
// party fills up, the aggregate ceiling it used to imply is gone entirely. Something has to
// own "do not fill the volume", because `postgres_data`, `media_data` and `exports_data`
// share one filesystem: the end state is not a degraded feature, it is Postgres unable to
// write WAL and the whole event down with nobody watching.
//
// Deliberately NOT gated behind `quota_enabled`. That switch exists so an operator can
// stop rationing space between guests; it was never meant to authorise running the disk
// to zero, and an operator flipping it at 23:00 to unblock a guest should not silently
// disarm the last thing standing between the party and a dead database.
// WHAT IS RESERVED IS NOT A CONSTANT. A flat reserve answers "can Postgres still write",
// which is necessary and not sufficient: the keepsake needs room for BOTH halves at once —
// `required_free_bytes` is `media × 1.1 × 2`, since the ZIP and the HTML viewer are each
// gallery-sized. On the 40 GB box this runs on, a flat 10 GB reserve let uploads continue to
// roughly 25 GB of media while the release needed `2.2 × 25 + 10` = 65 GB free. Every upload
// in that band succeeded and then the archive could never be built — the product's entire
// promise, failing silently at the end of the night with nobody there to notice.
//
// So the gate enforces the invariant that actually matters: never accept an upload that
// would make the keepsake unbuildable. It shares `required_free_bytes` with the export
// preflight so the two cannot drift into disagreeing about the same question.
//
// Deliberately NOT gated behind `quota_enabled`. That switch exists so an operator can stop
// rationing space between guests; it was never meant to authorise running the disk to zero,
// and an operator flipping it at 23:00 to unblock a guest should not silently disarm the
// last thing standing between the party and a dead database.
if let Some(free) = crate::services::disk::free_bytes(&state.config.media_path) {
let remaining = (free as i64).saturating_sub(size);
if remaining < DISK_RESERVE_BYTES {
let media_after = state.media_total.get(&state.pool).await.saturating_add(size);
let keepsake_needs =
crate::services::export::required_free_bytes(media_after.max(0) as u64, 2) as i64;
let free_after = (free as i64).saturating_sub(size);
let required = keepsake_needs.saturating_add(DISK_RESERVE_BYTES);
if free_after < required {
tracing::error!(
free_bytes = free,
upload_size = size,
media_after,
keepsake_needs,
reserve = DISK_RESERVE_BYTES,
"refusing upload: it would take the media volume below the reserve"
"refusing upload: it would leave too little room to build the keepsake"
);
return Err(AppError::QuotaExceeded(
"Der Speicher des Events ist voll. Bitte sag einem Host Bescheid — neue \
Uploads sind vorübergehend nicht möglich."
"Der Speicher des Events ist fast voll — damit die Galerie am Ende noch als \
Download erstellt werden kann, sind neue Uploads jetzt gesperrt. Bitte sag \
einem Host Bescheid."
.into(),
));
}
}
// Failing OPEN when the disk can't be read is deliberate and matches the per-user quota
// above: refusing every upload because a `statfs` failed would be a worse outage than the
// below: refusing every upload because a `statfs` failed would be a worse outage than the
// one being guarded against.
// When quota is enforced, this holds the byte ceiling so the increment UPDATE below can
// enforce it atomically (`WHERE total + size <= limit`). Without that guard, two
// concurrent uploads from the same user (e.g. phone + laptop) both pass this stale
// pre-check and both increment, blowing past the quota. The pre-check stays as a
// fast path that avoids the disk write when the user is already clearly over.
let mut quota_limit: Option<i64> = None;
if quota_on && storage_quota_on {
let estimate = compute_storage_quota(&state).await;
@@ -1370,7 +1389,9 @@ pub async fn get_thumbnail(
#[cfg(test)]
mod tests {
use super::{MIN_QUOTA_LIMIT_BYTES, RangeSpec, parse_range, quota_limit_bytes};
use super::{
DISK_RESERVE_BYTES, MIN_QUOTA_LIMIT_BYTES, RangeSpec, parse_range, quota_limit_bytes,
};
// `Range` handling exists because iOS Safari probes every `<video>` with
// `Range: bytes=0-1` and abandons the load without a 206. These pin the forms a
@@ -1523,6 +1544,67 @@ mod tests {
);
}
/// THE INVARIANT THE UPLOAD GATE EXISTS FOR: if an upload is accepted, the keepsake must
/// still be buildable afterwards.
///
/// The gate and `ensure_export_space` answer the same question at different times, from the
/// same `required_free_bytes`. If they ever drift, the failure is silent and terminal — every
/// upload succeeds and the archive can never be built, discovered only when the host taps
/// release and there is nobody left to fix it. This pins the two together.
///
/// Models the real box: 40 GB volume, ~5 GB consumed by OS, images and Postgres.
#[test]
fn an_accepted_upload_always_leaves_room_to_build_the_keepsake() {
const USABLE: i64 = 35 * GB;
let reserve = DISK_RESERVE_BYTES;
// Walk the gallery upward in 250 MB steps and assert the two agree at every point.
let mut media: i64 = 0;
let step: i64 = 250 * 1024 * 1024;
let mut last_accepted = 0i64;
while media < USABLE {
let free = USABLE - media;
let media_after = media + step;
let free_after = free - step;
let required =
crate::services::export::required_free_bytes(media_after as u64, 2) as i64 + reserve;
let gate_accepts = free_after >= required;
if gate_accepts {
// The export preflight must agree, using the SAME arithmetic it will run later.
let preflight_needs =
crate::services::export::required_free_bytes(media_after as u64, 2) as i64
+ reserve;
assert!(
free_after >= preflight_needs,
"gate accepted at media={media_after} but the preflight would refuse"
);
last_accepted = media_after;
}
media = media_after;
}
// Sanity-check the ceiling is where the arithmetic says: 35 = 2.2·M + 10 ⇒ M ≈ 7.8 GB.
// Pinned loosely (69 GB) so a deliberate change to the overhead multiplier or the
// reserve fails this test loudly rather than silently moving the cliff.
assert!(
(6 * GB..=9 * GB).contains(&last_accepted),
"expected the gallery ceiling near 7.8 GB on a 35 GB volume, got {last_accepted} bytes"
);
}
/// The gate must be the binding constraint, not the per-user floor. With 100 guests each
/// allowed 500 MB, the per-user quota alone would authorise ~50 GB on a 40 GB disk.
#[test]
fn the_global_gate_binds_before_the_per_user_floor_can_overfill_the_disk() {
let per_user_total = MIN_QUOTA_LIMIT_BYTES * 100;
assert!(
per_user_total > 35 * GB,
"premise: the per-user floor alone over-commits the volume, so the global gate \
is what must stop it"
);
}
/// The floor must never write a cheque the volume cannot cash — otherwise a full disk
/// still hands out a 500 MB allowance and the filesystem Postgres needs fills up.
#[test]

View File

@@ -1354,7 +1354,7 @@ const EXPORT_SIZE_OVERHEAD_PCT: u64 = 110;
/// Computed in `u128` and clamped, NOT with `saturating_mul`: saturating first and then dividing by
/// 100 quietly turns an overflow into a number ~100x too small, which is the one direction that
/// matters here — an under-estimate authorises the very write the preflight exists to refuse.
fn required_free_bytes(media_bytes: u64, armed: i64) -> u64 {
pub(crate) fn required_free_bytes(media_bytes: u64, armed: i64) -> u64 {
let needed = media_bytes as u128 * EXPORT_SIZE_OVERHEAD_PCT as u128 / 100
* armed.max(1).min(i64::from(u32::MAX)) as u128;
needed.min(u64::MAX as u128) as u64

View File

@@ -0,0 +1,82 @@
//! Cached sum of all media bytes the event is holding.
//!
//! The upload gate needs to know "how big would the keepsake be if we accept this file", because
//! the archive needs room for BOTH halves at once (`export::required_free_bytes` is
//! `media × 1.1 × 2` — the ZIP and the HTML viewer are each gallery-sized). Asking that question
//! per upload has to be cheap, and it has to be cheap on the busiest write path in the app.
//!
//! `export::estimate_export_bytes` answers the same question exactly, but it aggregates
//! `original_size_bytes` across every upload row joined to `user` — fine once per release,
//! wasteful per upload and growing all evening. This sums `user.total_upload_bytes` instead:
//! one row per guest (~100), already maintained transactionally by the quota path, already
//! refunded on delete.
//!
//! The two differ slightly — this one counts uploads belonging to banned or hidden users, which
//! the export filters out. That skew is in the SAFE direction: it over-estimates the archive, so
//! the gate closes marginally early rather than marginally late. Never swap it for a cheaper
//! query that could under-estimate; an under-estimate authorises the very upload that makes the
//! keepsake unbuildable, which is the failure this exists to prevent.
use std::sync::{Arc, RwLock};
use std::time::{Duration, Instant};
use sqlx::PgPool;
/// How long a reading is trusted. Shorter than [`crate::services::disk`]'s TTL because this
/// number only ever grows and does so in the same request path that reads it — a stale value
/// under-counts the newest uploads, and under-counting is the direction that matters.
const TTL: Duration = Duration::from_secs(5);
/// Cheap-to-clone cache of the event's total media bytes. Lives in `AppState`.
#[derive(Clone)]
pub struct MediaTotalCache {
inner: Arc<RwLock<Option<(i64, Instant)>>>,
}
impl MediaTotalCache {
pub fn new() -> Self {
Self {
inner: Arc::new(RwLock::new(None)),
}
}
/// Drop the cached reading so the next `get()` re-queries.
///
/// Used by the e2e TRUNCATE endpoint for the same reason `DiskCache::invalidate` exists:
/// truncation removes every upload, and a surviving reading would make the next test's
/// gate compute against the previous test's data.
pub fn invalidate(&self) {
*self.inner.write().unwrap() = None;
}
/// Total bytes of media the event is holding, cached for [`TTL`].
///
/// Returns 0 when the query fails. That is a deliberate FAIL-OPEN, consistent with the
/// quota path and the export preflight: a database blip must not turn into "every upload
/// refused". The disk-space half of the gate still applies, so a failure here degrades the
/// check to the old flat-reserve behaviour rather than disabling it.
pub async fn get(&self, pool: &PgPool) -> i64 {
if let Some((bytes, at)) = *self.inner.read().unwrap()
&& at.elapsed() < TTL
{
return bytes;
}
let bytes = sqlx::query_scalar::<_, Option<i64>>(
"SELECT SUM(total_upload_bytes)::bigint FROM \"user\"",
)
.fetch_one(pool)
.await
.ok()
.flatten()
.unwrap_or(0)
.max(0);
*self.inner.write().unwrap() = Some((bytes, Instant::now()));
bytes
}
}
impl Default for MediaTotalCache {
fn default() -> Self {
Self::new()
}
}

View File

@@ -4,6 +4,7 @@ pub mod disk;
pub mod export;
pub mod imaging;
pub mod maintenance;
pub mod media_total;
pub mod rate_limiter;
pub mod sse_tickets;
pub mod video;

View File

@@ -5,6 +5,7 @@ use crate::config::AppConfig;
use crate::services::compression::CompressionWorker;
use crate::services::config::ConfigCache;
use crate::services::disk::DiskCache;
use crate::services::media_total::MediaTotalCache;
use crate::services::rate_limiter::RateLimiter;
use crate::services::sse_tickets::SseTicketStore;
@@ -38,6 +39,8 @@ pub struct AppState {
pub config_cache: ConfigCache,
/// Cached total/free bytes for the media filesystem (quota + admin stats).
pub disk_cache: DiskCache,
/// Cached sum of all media bytes, for the upload gate's keepsake-headroom check.
pub media_total: MediaTotalCache,
}
impl AppState {
@@ -63,6 +66,7 @@ impl AppState {
sse_tickets: SseTicketStore::new(),
config_cache,
disk_cache: DiskCache::new(),
media_total: MediaTotalCache::new(),
}
}
}

File diff suppressed because one or more lines are too long

View File

@@ -31,14 +31,16 @@ services:
deploy:
resources:
limits:
# 1G, not 512M. DATABASE_MAX_CONNECTIONS defaults to 30 for a ~100-guest event
# (feed polling + SSE + uploads at once), and 30 backends plus Postgres 16's
# default shared_buffers leaves very little headroom at 512M. An OOM here does
# not degrade one feature — it takes the event down, because every request
# path touches the database. Memory is the cheaper knob than shrinking the
# pool back and reintroducing the queueing it was raised to fix.
# 1G, not 512M. Postgres 16's default shared_buffers plus a pool of backends
# leaves very little headroom at 512M, and an OOM here does not degrade one
# feature — it takes the event down, because every request path touches the
# database.
#
# Raising DATABASE_MAX_CONNECTIONS further means raising this too.
# `.env.example` now sets DATABASE_MAX_CONNECTIONS=15, sized to the 2 vCPU this
# box has rather than to the guest count: since migration 024 replaced the feed
# view's GROUP BY with scalar subqueries, a feed page costs well under a
# millisecond, so connections are no longer spent waiting. Raising it back
# toward 30 means raising this limit with it.
memory: 1G
app:

View File

@@ -46,6 +46,33 @@
document.head.appendChild(s);
}
} catch (_) {}
// Boot backstop. With `ssr = false` the page is EMPTY until the bundle mounts, so
// anything that stops it mounting leaves the guest on the spinner below forever:
// a chunk 404 after a redeploy, a dead uplink, or untranspiled syntax on an old
// phone. There is no message, no reload control, and in a standalone PWA no URL
// bar to escape from — the app is simply bricked for that guest, all evening.
//
// A TIMEOUT, deliberately, not feature detection: a SyntaxError in the bundle is
// invisible to any capability check, whereas "still not painted" catches every
// cause at once. The root layout removes #app-boot on mount, so its continued
// presence IS the failure signal and nothing needs to cancel this.
//
// 15s is well beyond a slow-but-healthy load on venue wifi (the bundle is ~130 kB
// gzipped); erring long matters more than erring short, because a false positive
// here would tell a guest something is broken while it is quietly working.
setTimeout(function () {
var boot = document.getElementById('app-boot');
if (!boot) return; // app mounted — nothing to do
boot.innerHTML =
'<div style="max-width:20rem;text-align:center">' +
'<p style="font-family:Georgia,serif;font-size:1.125rem;font-weight:600;margin:0 0 .5rem">Die App konnte nicht geladen werden</p>' +
'<p style="margin:0 0 1.25rem;color:#545350;line-height:1.5">Bitte prüf deine Verbindung und lade die Seite neu.</p>' +
'<button id="app-boot-reload" style="font:inherit;font-weight:600;cursor:pointer;border:0;border-radius:.5rem;padding:.625rem 1.25rem;background:#8a6a2b;color:#fff">Neu laden</button>' +
'</div>';
var btn = document.getElementById('app-boot-reload');
if (btn) btn.addEventListener('click', function () { location.reload(); });
}, 15000);
})();
</script>
%sveltekit.head%
@@ -65,6 +92,14 @@
<span class="app-boot__spinner"></span>
<span class="app-boot__label">EventSnap</span>
</div>
<!-- The app is client-rendered end to end, so with JS off there is nothing at all to
show. Say so, rather than leaving a permanent spinner over a blank page. -->
<noscript>
<div class="app-boot__noscript">
<p><strong>EventSnap braucht JavaScript</strong></p>
<p>Bitte aktiviere JavaScript im Browser und lade die Seite neu.</p>
</div>
</noscript>
<style>
#app-boot {
position: fixed;
@@ -103,6 +138,26 @@
transform: rotate(360deg);
}
}
/* Sits above #app-boot, which is `position: fixed` and would otherwise cover it. */
.app-boot__noscript {
position: fixed;
inset: 0;
z-index: 1;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
gap: 0.25rem;
padding: 2rem;
text-align: center;
background: #faf9f7;
font-family: system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif;
color: #545350;
}
html.dark .app-boot__noscript {
background: #100f0f;
color: #a6a4a1;
}
</style>
</body>
</html>

View File

@@ -3,7 +3,8 @@ import {
classifyUploadStatus,
isReversibleLock,
entryToQueueItem,
shouldAbortForStall
shouldAbortForStall,
suspendedSinceLastTick
} from './upload-queue';
/**
@@ -140,4 +141,55 @@ describe('shouldAbortForStall', () => {
expect(shouldAbortForStall(now - 91_000, now, true)).toBe(false);
expect(shouldAbortForStall(now - 121_000, now, true)).toBe(true);
});
});
/**
* The watchdog measures SILENCE via `Date.now()`, but a backgrounded phone freezes the
* interval while the clock keeps running. Without crediting the un-run window back, the first
* tick after a screen lock reads the whole sleep as a stall and aborts a healthy upload —
* re-sending from byte zero and spending one of five permanent auto-attempts. That is what
* every phone does between shots at a party.
*
* Detecting the freeze from the tick gap (rather than from `visibilitychange`) also covers the
* causes that fire no visibility event at all: a throttled-but-visible tab, a closed lid, an
* occluded window.
*/
describe('suspendedSinceLastTick', () => {
const now = 1_000_000;
it('credits nothing for a tick that arrived on schedule', () => {
expect(suspendedSinceLastTick(now - 5_000, now, 5_000)).toBe(0);
});
it('credits nothing for ordinary timer jitter or throttling', () => {
expect(suspendedSinceLastTick(now - 6_900, now, 5_000)).toBe(0);
});
it('credits the whole frozen window when the interval did not run', () => {
// Screen locked ~2 minutes: a 5s interval arriving 130s late.
expect(suspendedSinceLastTick(now - 130_000, now, 5_000)).toBe(125_000);
});
it('credits nothing when the clock jumps backwards (NTP correction)', () => {
expect(suspendedSinceLastTick(now + 60_000, now, 5_000)).toBe(0);
});
it('a suspension longer than the stall ceiling does not abort a healthy upload', () => {
// The bug, end to end: 3 minutes suspended, interval resumes, no bytes since.
const lastProgressAt = now - 180_000;
const credited = Math.min(
now,
lastProgressAt + suspendedSinceLastTick(now - 185_000, now, 5_000)
);
expect(shouldAbortForStall(credited, now, false)).toBe(false);
});
it('but a socket still silent 91s AFTER resume is aborted, never left to xhr.timeout', () => {
// iOS reaps backgrounded sockets without firing `error`. The credit buys one fresh
// window, not immunity — otherwise a dead upload would hang for 5-60 minutes holding
// the queue's `processing` latch.
const resumedAt = now - 91_000;
expect(shouldAbortForStall(resumedAt, now, false)).toBe(true);
});
});

View File

@@ -94,6 +94,45 @@ export function shouldAbortForStall(
return now - lastActivityAt > ceiling;
}
/**
* Normal timer jitter/throttling budget. A tick later than `interval + this` did not run
* because the page was suspended, not because it was merely late.
*/
const SUSPEND_TOLERANCE_MS = 2_000;
/**
* Wall-clock the watchdog interval FAILED to cover because the page was suspended.
*
* `Date.now()` keeps advancing while a backgrounded phone is frozen, but `setInterval` does
* not run. So the first tick after a screen lock saw the entire sleep as "no bytes moved" and
* aborted a connection that was very possibly healthy — restarting a 200 MB video from byte
* zero, burning one of five PERMANENT auto-attempts (`chargeAttempt`), and breaking the drain
* loop for every other queued photo. A phone in a pocket between shots is the common case at a
* party, not an edge case.
*
* The interval is its own suspension detector: a tick scheduled 5s out that arrives 130s late
* means the page was frozen for ~125s, and that is exactly the window the watchdog had no
* right to measure. Deliberately chosen over a `visibilitychange` listener, which only covers
* the causes that happen to fire that event — a throttled-but-visible tab, a closed laptop lid
* and an occluded window all freeze timers without one. It also needs no listener, no
* module-level state, no SSR guard and no teardown.
*
* `performance.now()` was rejected as the clock source: Safari pauses it across system sleep
* on some paths while Chrome does not, which is precisely the non-uniformity that makes it
* unusable as the sole signal.
*
* Returns 0 for a normal tick, and 0 if the clock jumps BACKWARDS (an NTP correction) — that
* fails open, and the wall-clock `xhr.timeout` still bounds the request.
*/
export function suspendedSinceLastTick(
lastTickAt: number,
now: number,
intervalMs: number = STALL_CHECK_INTERVAL_MS
): number {
const overshoot = now - lastTickAt - intervalMs;
return overshoot > SUSPEND_TOLERANCE_MS ? overshoot : 0;
}
/**
* Wall-clock cap for one attempt, scaled by file size assuming a floor of ~8 kB/s — a
* deliberately pessimistic rate, because killing a slow-but-progressing upload would lose
@@ -905,11 +944,25 @@ async function uploadItem(id: string): Promise<void> {
// connection that never errors and never completes. Only "no bytes moved" catches
// that without also punishing a healthy slow link.
let lastProgressAt = Date.now();
let lastTickAt = Date.now();
let bodySent = false;
let stalled = false;
const stallTimer = setInterval(() => {
if (!shouldAbortForStall(lastProgressAt, Date.now(), bodySent)) return;
// Never fire twice. `xhr.abort()` on a request already in `readyState === DONE`
// emits NO `abort` event, so `settle()` would never run: the interval would keep
// running forever, re-aborting every 5s, and `activeUploads` would keep a stale
// entry so the guest's ✕ button silently did nothing.
if (stalled) return;
const now = Date.now();
// Credit back the window the page was frozen. The watchdog measures SILENCE, and
// a period in which it could not observe anything is not evidence of silence.
// Clamped to `now` so a progress event delivered right at resume cannot push the
// timestamp into the future.
lastProgressAt = Math.min(now, lastProgressAt + suspendedSinceLastTick(lastTickAt, now));
lastTickAt = now;
if (!shouldAbortForStall(lastProgressAt, now, bodySent)) return;
stalled = true;
clearInterval(stallTimer);
xhr.abort();
}, STALL_CHECK_INTERVAL_MS);
const settle = (fn: () => void) => {
@@ -1018,7 +1071,16 @@ async function uploadItem(id: string): Promise<void> {
else reject(new NetworkError('Abgebrochen'));
})
);
// `send` can throw SYNCHRONOUSLY — most plausibly on a phone whose OS purged the
// backing store for the blob, leaving a neutered File. The executor would turn that
// into a rejection and `uploadItem` would recover, but `settle()` never runs: the
// stall interval leaks and `activeUploads` keeps a stale entry, so the ✕ button on
// that item stops working for the rest of the session.
try {
xhr.send(formData);
} catch {
settle(() => reject(new NetworkError('Netzwerkfehler')));
}
});
// Success — remove blob from IndexedDB, mark done

View File

@@ -95,11 +95,81 @@
--color-purple-700: #6f5523;
--color-purple-800: #59441e;
--color-purple-900: #493819;
/* The ramp stopped at 900 while blue/primary both run to 950, so
* `dark:bg-purple-950/50` (routes/host/+page.svelte) fell through to Tailwind's default
* violet — off-brand on every browser, modern ones included. Mirrors `--color-blue-950`. */
--color-purple-950: #29200d;
--color-violet-500: #ab8433;
--color-violet-600: #8a6a2b;
--color-accent-500: #ab8433;
--color-accent-600: #8a6a2b;
/* ── Semantic: red / amber / green ───────────────────────────────────────────
* PINNED TO HEX rather than inherited. Tailwind v4 ships these families as
* `oklch()`, which Safari <15.4, Chrome <111 and Samsung Internet <22 (the default
* browser on Samsung phones) cannot parse. The declaration is accepted but
* `var(--color-red-600)` is then invalid at computed-value time, so `background-color`
* falls back to `initial` — transparent — and `.btn-danger` in the component layer
* renders white text on nothing. An invisible delete-confirm button.
*
* Note this is NOT the `@apply` hard-fail described for `primary` above: these families
* have Tailwind defaults, so an unpinned stop does not break the build, it silently
* reverts to oklch. Full 50-950 ramps are therefore about closing that silent-leak class
* permanently, not about compiling.
*
* Values are Tailwind 4.2.2's own defaults gamut-mapped to sRGB by Lightning CSS — the
* same converter already in this build pipeline, which is why they match the hex
* fallbacks it emits for the `/alpha` opacity forms (verified against `#bf000f`,
* `#460809`, `#461901`, `#032e15`, `#82181a`, `#ffa3a3`, `#0d542b` in the shipped CSS).
* Naive channel clipping gives different, wrong values for the out-of-gamut stops.
* Modern browsers therefore render exactly what they render today. */
--color-red-50: #fef2f2;
--color-red-100: #ffe2e2;
--color-red-200: #ffcaca;
--color-red-300: #ffa3a3;
--color-red-400: #ff6568;
--color-red-500: #fb2c36;
--color-red-600: #e40014;
--color-red-700: #bf000f;
--color-red-800: #9f0712;
--color-red-900: #82181a;
--color-red-950: #460809;
--color-amber-50: #fffbeb;
--color-amber-100: #fef3c6;
--color-amber-200: #fee685;
--color-amber-300: #ffd236;
--color-amber-400: #fcbb00;
--color-amber-500: #f99c00;
--color-amber-600: #dd7400;
--color-amber-700: #b75000;
--color-amber-800: #953d00;
--color-amber-900: #7b3306;
--color-amber-950: #461901;
--color-green-50: #f0fdf4;
--color-green-100: #dcfce7;
--color-green-200: #b9f8cf;
--color-green-300: #7bf1a8;
--color-green-400: #05df72;
--color-green-500: #00c758;
--color-green-600: #00a544;
--color-green-700: #008138;
--color-green-800: #016630;
--color-green-900: #0d542b;
--color-green-950: #032e15;
/* Avatar chips (lib/avatar.ts) — same oklch problem, same fix. Only the stops those
* chips actually use; nothing else in the app references rose or teal. */
--color-rose-100: #ffe4e6;
--color-rose-200: #ffccd3;
--color-rose-700: #c20039;
--color-rose-900: #8b0836;
--color-teal-100: #cbfbf1;
--color-teal-200: #96f7e4;
--color-teal-700: #00776e;
--color-teal-900: #0b4f4a;
/* ── Neutrals: pearl → silver → graphite (remaps `gray-*`). Whisper-warm
* pearl at the light end (research: "warm pearl, not stark white"), turning
* neutral-cool through the mids/darks so structure reads as silver, not sand. */