fix: gate uploads on keepsake headroom, and close five unattended-event gaps
The box is 2 vCPU / 4 GB / 40 GB, not the 4 vCPU / 8 GB / 80 GB that the audit, the committed comments and README's sizing section all assumed. That correction is what the first change is about; the rest are the remaining pre-event items. THE ARCHIVE COULD BECOME UNBUILDABLE WHILE UPLOADS KEPT SUCCEEDING `required_free_bytes` is `media × 1.1 × 2` — the ZIP and the HTML viewer are each gallery-sized — and the export preflight also wants DISK_RESERVE_BYTES on top. The upload gate, though, only refused below a FLAT 10 GB reserve. On 40 GB that let uploads run to ~25 GB of media while a release needed `2.2 × 25 + 10` = 65 GB free. Every upload in that band succeeded and the keepsake could then never be built: the product's entire promise, failing silently at the end of the night with nobody there. The gate now enforces the invariant that actually matters — never accept an upload that would make the keepsake unbuildable — sharing `required_free_bytes` with the preflight so the two cannot drift into disagreeing about the same question. Uploads stop at ~8 GB of media on this disk, with a German message naming the cause. Refusing the 1001st photo beats losing all 1000. `media_total.rs` backs it: SUM(user.total_upload_bytes) over ~100 rows, cached 5s, rather than `estimate_export_bytes`'s join across every upload. It counts hidden and banned users' bytes, which the export excludes — skew in the SAFE direction, so the gate closes marginally early rather than late. Fails open on a query error. A test pins the gate against the preflight across the whole gallery-size range, and a second asserts the per-user floor alone would over-commit the volume — i.e. that the global gate is what must bind. THE WATCHDOG ABORTED HEALTHY UPLOADS EVERY TIME A PHONE WAS POCKETED `Date.now()` advances while a backgrounded phone is frozen but `setInterval` does not, so the first tick after a screen lock read the whole sleep as silence and aborted — re-sending a video from byte zero and burning one of five PERMANENT auto-attempts. The interval is now its own suspension detector: a tick that arrives 125s late for a 5s schedule credits that window back, because a period the watchdog could not observe is not evidence of silence. Chosen over a `visibilitychange` listener, which only covers causes that fire that event — a throttled-but-visible tab, a closed lid and an occluded window all freeze timers without one — and which would have needed module state, an SSR guard and a teardown for strictly less coverage. `performance.now()` was rejected because Safari pauses it across system sleep on some paths and Chrome does not. The credit buys one fresh window, not immunity: a socket iOS reaped while backgrounded still aborts ~90s after resume rather than hanging for `xhr.timeout` (5-60 min) with the queue's `processing` latch held. Two latent leaks found while in there: `xhr.abort()` on a request already in readyState DONE emits no `abort` event, so `settle()` never ran and the interval re-aborted every 5s forever while `activeUploads` kept a stale entry (the ✕ button silently stopped working); and a synchronous throw from `xhr.send` — a blob whose backing store the OS purged — leaked the same way. Both closed. OKLCH MADE THE DELETE BUTTON INVISIBLE ON SAMSUNG'S DEFAULT BROWSER red/amber/green were never in the @theme block and fell through to Tailwind v4's `oklch()` defaults, which Safari <15.4, Chrome <111 and Samsung Internet <22 cannot parse: `var(--color-red-600)` is then invalid at computed-value time, `background-color` falls back to transparent, and `.btn-danger` renders white text on nothing. Pinned to Tailwind's own defaults gamut-mapped to sRGB by Lightning CSS — the converter already in this pipeline — so modern browsers render exactly what they render today. Verified against seven hex fallbacks it had already emitted for the /alpha forms. rose and teal (avatar chips) had the same leak. The app CSS goes from 40 oklch declarations to 0. Also fixes `--color-purple-950`, which was simply missing: `dark:bg-purple-950/50` on the host dashboard was rendering default violet on EVERY browser, off-brand. The keepsake viewer only picks this up on a rebuild, so its committed artefact is rebuilt here too — still single-file, still zero external references. A BRICKED BOOT LOOKED LIKE A SPINNER FOREVER With `ssr = false` the page is empty until the bundle mounts, so a chunk 404 after a redeploy or a dead uplink left the guest on the boot spinner with no message, no reload control, and in a standalone PWA no URL bar. A 15s timeout in the existing nonce'd IIFE (no CSP change) swaps in German copy and a reload button. Deliberately a timeout rather than feature detection: a SyntaxError in the bundle is invisible to any capability check. Plus a <noscript>, since there was nothing at all to see without JS. EVERY 4xx WAS INVISIBLE AT ANY LOG LEVEL tower_http counts 4xx as a success, so it logs at DEBUG while production runs at info. If guests spend the evening hitting 429s or 413s, the post-event logs said nothing. Now one WARN per client error; 5xx excluded because Internal already logs its source chain and the pool-exhaustion 503 logs at construction. A DEAD FRONTEND SERVED A BLANK 502 `handle_errors 5xx` with an inline German page (the caddy service mounts only the Caddyfile, so there is no volume to ship a static file through). Verified empirically against this config, not from documentation: an upstream 404 through `reverse_proxy` still arrives as untouched `application/json`, and only a dial failure renders the page. That mattered — the keepsake download navigates a hidden iframe and DEPENDS on a real 404/429 arriving, and swallowing those would have been worse than the blank 502. CONFIG CORRECTIONS FOR THE REAL HARDWARE DATABASE_MAX_CONNECTIONS 30 → 15: sized to 2 vCPU rather than to the guest count. Since migration 024 a feed page costs well under a millisecond, so connections are no longer spent waiting, and 30 backends crowd the db container's 1 GB on a 4 GB host. COMPRESSION_WORKER_CONCURRENCY stays at 2 — the merged heavy-image permit already serialises anything over 150 MiB, so the "two 48 MP photos" worst case that number was sized against is unreachable; dropping to 1 would halve light-path throughput and push more feed tiles onto full-size originals. README's sizing section rewritten for the actual disk. Verified: 149/149 backend tests against a live Postgres, clippy clean, 57/57 vitest, svelte-check 0 errors, eslint clean, vite build, export-viewer rebuild, caddy validate, compose YAML parse. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
36
.env.example
36
.env.example
@@ -36,13 +36,20 @@ DATABASE_URL=postgres://eventsnap:CHANGE_ME_use_a_strong_password@db:5432/events
|
|||||||
POSTGRES_USER=eventsnap
|
POSTGRES_USER=eventsnap
|
||||||
POSTGRES_PASSWORD=CHANGE_ME_use_a_strong_password
|
POSTGRES_PASSWORD=CHANGE_ME_use_a_strong_password
|
||||||
POSTGRES_DB=eventsnap
|
POSTGRES_DB=eventsnap
|
||||||
# Connection pool size. Default 10. For a busy event (~100 guests polling the feed
|
# Connection pool size. The code default is 10 (backend/src/db.rs) — set it explicitly,
|
||||||
# + SSE + uploads at once) raise to ~30 so requests don't queue on a pool permit.
|
# because a `.env` written by hand from this file's secrets is otherwise silently on 10.
|
||||||
# PAIRED WITH THE DB CONTAINER'S MEMORY LIMIT: 30 backends plus Postgres 16's default
|
#
|
||||||
# shared_buffers is already snug in the 1G that docker-compose.yml allots the `db`
|
# SIZE IT TO THE CORES, NOT TO THE GUESTS. The earlier advice here was ~30, reasoned from
|
||||||
# service. If you raise this, raise `db.deploy.resources.limits.memory` with it — an
|
# "~100 guests polling the feed at once" back when a feed page cost ~449 ms and connections
|
||||||
# OOM in Postgres doesn't degrade one feature, it takes the whole event down.
|
# were spent waiting. Migration 024 replaced the feed view's GROUP BY with scalar subqueries
|
||||||
DATABASE_MAX_CONNECTIONS=30
|
# and a page now costs well under a millisecond, so concurrency is no longer where the time
|
||||||
|
# goes. On a 2 vCPU box 30 simultaneous queries cannot run — they queue on the CPU instead of
|
||||||
|
# on the pool, which is the same wait wearing a different hat, and 30 Postgres backends plus
|
||||||
|
# shared_buffers is snug in the 1G that docker-compose.yml allots `db`.
|
||||||
|
#
|
||||||
|
# 15 on 2 vCPU / 4 GB. Raise toward 30 only alongside more cores AND a bigger `db` memory
|
||||||
|
# limit — an OOM in Postgres doesn't degrade one feature, it takes the whole event down.
|
||||||
|
DATABASE_MAX_CONNECTIONS=15
|
||||||
|
|
||||||
# Log level. `info` is the right production default: at `debug` the tower-http trace
|
# Log level. `info` is the right production default: at `debug` the tower-http trace
|
||||||
# layer writes a line per request AND per response, which on a busy event is a large
|
# layer writes a line per request AND per response, which on a busy event is a large
|
||||||
@@ -124,12 +131,23 @@ EXPORT_PATH=/exports
|
|||||||
# display resize: ~145 MB at 12 MP, ~223 MB at 24 MP, ~354 MB at 48 MP.
|
# display resize: ~145 MB at 12 MP, ~223 MB at 24 MP, ~354 MB at 48 MP.
|
||||||
#
|
#
|
||||||
# So on a 2 vCPU / 4 GB box (e.g. Hetzner CX22) KEEP THIS AT 2:
|
# So on a 2 vCPU / 4 GB box (e.g. Hetzner CX22) KEEP THIS AT 2:
|
||||||
# * concurrency 2, two 48 MP photos ≈ 800 MB against the 1G app limit — ~25% margin.
|
# * concurrency 4 would put two giants at ~1.5 GB against the 1G app limit — OOM.
|
||||||
# * concurrency 4, the same pair ≈ 1.5 GB — OOM.
|
|
||||||
# * and app=2G + db=1G + frontend/caddy 256M each + ~370 MB of OS/Docker exceeds the
|
# * and app=2G + db=1G + frontend/caddy 256M each + ~370 MB of OS/Docker exceeds the
|
||||||
# ~3910 MiB a "4 GB" VM actually reports. Raising the limit oversubscribes the host.
|
# ~3910 MiB a "4 GB" VM actually reports. Raising the limit oversubscribes the host.
|
||||||
# 4 is only reasonable on the 4 vCPU / 8 GB box README.md documents.
|
# 4 is only reasonable on the 4 vCPU / 8 GB box README.md documents.
|
||||||
#
|
#
|
||||||
|
# The "two 48 MP photos at once" worst case this number used to be sized against is no
|
||||||
|
# longer reachable: compression.rs takes an EXCLUSIVE `heavy` permit for any job whose
|
||||||
|
# estimated peak exceeds HEAVY_IMAGE_BYTES (150 MiB), so two giants serialise no matter what
|
||||||
|
# this is set to. What concurrency 2 now buys is two ORDINARY phone photos in parallel
|
||||||
|
# (~145 MB peak each), which is both memory-safe and short enough not to starve the two
|
||||||
|
# tokio worker threads a 2 vCPU box gets.
|
||||||
|
#
|
||||||
|
# Do NOT drop this to 1 hoping to protect the CPU. It halves throughput on the common light
|
||||||
|
# path for a heavy path that is already serialised, and a longer compression backlog means
|
||||||
|
# more feed tiles served from full-size originals (VirtualFeed falls back to /original while
|
||||||
|
# derivatives are pending) — trading a little CPU for a lot of venue-wifi bandwidth.
|
||||||
|
#
|
||||||
# Throughput at 2 is not the bottleneck anyone thinks it is: ~2.5s per 12 MP photo, so
|
# Throughput at 2 is not the bottleneck anyone thinks it is: ~2.5s per 12 MP photo, so
|
||||||
# 100 photos is ~250 CPU-seconds spread over an entire evening.
|
# 100 photos is ~250 CPU-seconds spread over an entire evening.
|
||||||
COMPRESSION_WORKER_CONCURRENCY=2
|
COMPRESSION_WORKER_CONCURRENCY=2
|
||||||
|
|||||||
55
Caddyfile
55
Caddyfile
@@ -75,4 +75,59 @@
|
|||||||
|
|
||||||
# Everything else goes to SvelteKit frontend
|
# Everything else goes to SvelteKit frontend
|
||||||
reverse_proxy frontend:3001
|
reverse_proxy frontend:3001
|
||||||
|
|
||||||
|
# Last-resort page for when Caddy itself cannot reach an upstream — the app or frontend
|
||||||
|
# container down, restarting, or still warming up after a host reboot. Without it a guest
|
||||||
|
# gets Caddy's bodiless 502: a completely blank page, which reads as "the whole thing is
|
||||||
|
# gone" rather than "try again in a moment".
|
||||||
|
#
|
||||||
|
# THIS DOES NOT TOUCH APPLICATION ERRORS. `handle_errors` fires only on errors CADDY
|
||||||
|
# generates; a status the app returns through `reverse_proxy` is written back verbatim and
|
||||||
|
# never reaches here. That distinction is load-bearing rather than incidental: the keepsake
|
||||||
|
# download navigates a HIDDEN IFRAME and depends on a real 404/429 arriving from the app
|
||||||
|
# (frontend/src/routes/export/+page.svelte), and every API route answers 403/404/429 as
|
||||||
|
# ordinary JSON that the client parses. Swallowing those into an HTML page would be a far
|
||||||
|
# worse regression than the blank 502 this fixes. Verified against this exact config: an
|
||||||
|
# upstream 404 through `reverse_proxy` still arrives as `Content-Type: application/json`
|
||||||
|
# with its body intact, while only a dial failure renders the page below.
|
||||||
|
#
|
||||||
|
# Scoped to 5xx so a hypothetical future Caddy-generated 4xx (there is none today) still
|
||||||
|
# returns plainly instead of claiming the server is restarting.
|
||||||
|
#
|
||||||
|
# The body is inline because the caddy service mounts ONLY ./Caddyfile and caddy_data —
|
||||||
|
# there is no volume to ship an HTML file through and the image has no build step, so a
|
||||||
|
# static file would mean changing the deployed stack's compose definition. No external
|
||||||
|
# font, stylesheet or image is referenced: the app may be exactly what is down.
|
||||||
|
#
|
||||||
|
# `handle_errors` has NO position in the directive order — Caddy hoists it into a separate
|
||||||
|
# `errors` route list — so it cannot disturb the "first `header` directive wins" hazard
|
||||||
|
# documented at the top of this file. The site-wide security headers still apply to it.
|
||||||
|
handle_errors 5xx {
|
||||||
|
header Content-Type "text/html; charset=utf-8"
|
||||||
|
header Cache-Control "no-store"
|
||||||
|
# {err.status_code} preserves the real status. Hardcoding 503 would mislabel a genuine
|
||||||
|
# 502 for anything watching from outside.
|
||||||
|
respond `<!doctype html>
|
||||||
|
<html lang="de">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<title>Gleich zurück</title>
|
||||||
|
<style>
|
||||||
|
html{background:#faf9f7;color:#1a1918;font-family:system-ui,-apple-system,"Segoe UI",Roboto,sans-serif}
|
||||||
|
body{margin:0;min-height:100vh;display:flex;align-items:center;justify-content:center;padding:2rem;text-align:center}
|
||||||
|
h1{font-family:Georgia,"Times New Roman",serif;font-weight:600;font-size:1.5rem;margin:0 0 .75rem}
|
||||||
|
p{margin:0;color:#545350;line-height:1.5}
|
||||||
|
@media (prefers-color-scheme:dark){html{background:#100f0f;color:#f5f4f2}p{color:#a6a4a1}}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<main>
|
||||||
|
<h1>Wir sind gleich zurück</h1>
|
||||||
|
<p>Die Seite wird gerade neu gestartet.<br>Bitte lade in einem Moment neu — deine Fotos bleiben gespeichert.</p>
|
||||||
|
</main>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
` {err.status_code}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
56
README.md
56
README.md
@@ -280,35 +280,49 @@ so a host takedown or a ban actually revokes access to the bytes.
|
|||||||
degrade one subsystem — Postgres stops being able to write and the whole event goes
|
degrade one subsystem — Postgres stops being able to write and the whole event goes
|
||||||
down.
|
down.
|
||||||
|
|
||||||
Uploads are self-limiting. `per_user_limit = free_disk × quota_tolerance ÷
|
**`Gallery.zip` and `Memories.zip` are each roughly a second copy of every original.**
|
||||||
active_uploaders` is recomputed against live free space on every upload, so guests
|
Both write their media `Compression::Stored`, and `Memories.zip` streams the untouched
|
||||||
converge on a fixed point at `tolerance / (1 + tolerance)` of the free space you
|
original for every video and for every image at or under 5 MB. So a release wants room
|
||||||
started with — **43%** at the default 0.75. On an 80 GB box with ~70 GB free after
|
for **two more copies of the gallery** on top of the gallery itself — which is what
|
||||||
the OS and images, media settles at ~30 GB and stops.
|
`required_free_bytes` encodes as `media × 1.1 × 2`.
|
||||||
|
|
||||||
**The keepsake is what the 80 GB baseline does not cover.** `Gallery.zip` and
|
The per-user quota does **not** bound this. It is a fairness mechanism that divides
|
||||||
`Memories.zip` are built concurrently and each is roughly a second copy of every
|
free space between guests, and since it carries a floor (`MIN_QUOTA_LIMIT_BYTES`, so a
|
||||||
original: both write their media `Compression::Stored`, and `Memories.zip` streams the
|
guest's allowance stops shrinking as the party fills up) the aggregate ceiling it used
|
||||||
untouched original for every video and for every image at or under 5 MB. So a release
|
to imply is gone. What bounds the disk is the **global gate in the upload handler**,
|
||||||
wants room for **two more copies of the gallery** on top of the gallery itself.
|
which refuses any upload that would leave too little room to build the keepsake:
|
||||||
|
|
||||||
| Stage | Used | Free (80 GB box) |
|
```
|
||||||
|---|---|---|
|
free_after_upload < media_after × 1.1 × 2 + DISK_RESERVE_BYTES → refused
|
||||||
| Fresh box (OS + images) | ~10 GB | ~70 GB |
|
```
|
||||||
| Guests reach the quota fixed point | ~40 GB | ~40 GB |
|
|
||||||
| Host releases → both archives | ~100 GB | **ENOSPC** |
|
|
||||||
|
|
||||||
Two ways to size for it:
|
Solving that for the gallery size gives the real ceiling. On the **40 GB box this runs
|
||||||
|
on**, with ~5 GB for the OS, Docker images (the runbook pre-pulls the rollback tag too)
|
||||||
|
and Postgres:
|
||||||
|
|
||||||
|
| Volume | Usable after baseline | Media ceiling | Free at release |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 40 GB | ~35 GB | **~8 GB** | ~27 GB → both archives fit |
|
||||||
|
| 80 GB | ~70 GB | ~19 GB | ~51 GB → both archives fit |
|
||||||
|
|
||||||
|
**Uploads therefore stop at roughly 8 GB of media on a 40 GB box, not when the disk is
|
||||||
|
full.** That is deliberate. 1000 photos at ~3.5 MB is ~3.5 GB and fits comfortably;
|
||||||
|
video is what consumes the budget, so lower `max_video_size_mb` (seeded at 500) if you
|
||||||
|
expect a lot of it. Refusing the 1001st upload is a far better outcome than accepting it
|
||||||
|
and discovering at 01:00 that the archive can never be built.
|
||||||
|
|
||||||
|
Two ways to buy headroom:
|
||||||
|
|
||||||
- **Provision ~3× your expected media** on one volume (media + two archives), or
|
- **Provision ~3× your expected media** on one volume (media + two archives), or
|
||||||
- **give `exports_data` its own volume** so a full export cannot reach Postgres, and
|
- **give `exports_data` its own volume** so a full export cannot reach Postgres, and
|
||||||
size that one at ~2× expected media.
|
size that one at ~2× expected media.
|
||||||
|
|
||||||
This is no longer silent. The export refuses up front with the two numbers rather than
|
None of this is silent. The upload gate refuses with a German message naming the cause,
|
||||||
hitting ENOSPC halfway through a multi-GB write, a rebuild reclaims the superseded
|
the export preflight refuses up front with both numbers rather than hitting ENOSPC
|
||||||
generation before it starts (so peak is one generation, not two), and the host
|
halfway through a multi-GB write, a rebuild only reclaims the superseded generation
|
||||||
dashboard warns as soon as the keepsake would not fit — which is the only point at
|
**after** the new one lands (so a failed rebuild can never leave you with no archive at
|
||||||
which anyone can still do something about it.
|
all), and the host dashboard warns as soon as the keepsake would not fit — which is the
|
||||||
|
only point at which anyone can still do something about it.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -78,6 +78,26 @@ impl IntoResponse for AppError {
|
|||||||
};
|
};
|
||||||
let message = self.message();
|
let message = self.message();
|
||||||
|
|
||||||
|
// Log every 4xx. Until now they were invisible at ANY log level: tower_http's
|
||||||
|
// `ServerErrorsAsFailures` classifier counts a 4xx as a *success*, so it goes to
|
||||||
|
// `DefaultOnResponse` at DEBUG, and production runs at `info`. The consequence is that a
|
||||||
|
// misconfigured limit leaves no trace at all — if guests spend the evening hitting 429s
|
||||||
|
// on `upload_rate_per_hour`, or 413s on the storage quota, `docker compose logs` after
|
||||||
|
// the event contains nothing about it and the cause is unknowable.
|
||||||
|
//
|
||||||
|
// WARN rather than INFO because every variant here is a request that did not do what
|
||||||
|
// the guest asked. 5xx is excluded: `Internal` already logs with its full source chain
|
||||||
|
// in `message()` above, and the pool-exhaustion 503 logs at construction — logging again
|
||||||
|
// here would double every server-side failure.
|
||||||
|
//
|
||||||
|
// No request context is available: `into_response` receives only the error, so there is
|
||||||
|
// no path, method or user id to attach. Status + code + message is what can honestly be
|
||||||
|
// reported from here, and it is enough to see the SHAPE of a bad evening. Raising
|
||||||
|
// `tower_http` to DEBUG instead was considered and rejected — see the note in main.rs.
|
||||||
|
if status.is_client_error() {
|
||||||
|
tracing::warn!(status = status.as_u16(), code, %message, "request rejected");
|
||||||
|
}
|
||||||
|
|
||||||
let mut body = serde_json::json!({
|
let mut body = serde_json::json!({
|
||||||
"error": code,
|
"error": code,
|
||||||
"message": message,
|
"message": message,
|
||||||
@@ -161,6 +181,43 @@ mod tests {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// 4xx must be logged and 5xx must not be logged HERE — `Internal` logs its source chain in
|
||||||
|
/// `message()` and the pool-exhaustion 503 logs at construction, so a second line in
|
||||||
|
/// `into_response` would double every server-side failure in the post-event logs.
|
||||||
|
///
|
||||||
|
/// The guard is `status.is_client_error()`, so this pins the classification rather than the
|
||||||
|
/// logging itself (which needs a subscriber to observe).
|
||||||
|
#[test]
|
||||||
|
fn only_client_errors_are_in_the_logged_band() {
|
||||||
|
for err in [
|
||||||
|
AppError::BadRequest("x".into()),
|
||||||
|
AppError::Unauthorized("x".into()),
|
||||||
|
AppError::Forbidden("x".into()),
|
||||||
|
AppError::UploadsLocked("x".into()),
|
||||||
|
AppError::NotFound("x".into()),
|
||||||
|
AppError::Conflict("x".into()),
|
||||||
|
AppError::TooManyRequests("x".into(), Some(1)),
|
||||||
|
AppError::QuotaExceeded("x".into()),
|
||||||
|
] {
|
||||||
|
let (status, _) = err.status_and_code();
|
||||||
|
assert!(
|
||||||
|
status.is_client_error(),
|
||||||
|
"{status} should be in the 4xx band this logs"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
for err in [
|
||||||
|
AppError::ServiceUnavailable("x".into(), Some(3)),
|
||||||
|
AppError::Internal(anyhow::anyhow!("boom")),
|
||||||
|
] {
|
||||||
|
let (status, _) = err.status_and_code();
|
||||||
|
assert!(
|
||||||
|
!status.is_client_error(),
|
||||||
|
"{status} logs elsewhere; logging it here would double it"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Pool saturation is load, not a bug. A 500 makes the frontend's retry classifier pile
|
/// Pool saturation is load, not a bug. A 500 makes the frontend's retry classifier pile
|
||||||
/// straight back into the saturated pool with no backoff to pace it.
|
/// straight back into the saturated pool with no backoff to pace it.
|
||||||
#[test]
|
#[test]
|
||||||
|
|||||||
@@ -111,6 +111,12 @@ pub async fn truncate_all(
|
|||||||
// steers the per-user limit off `free_disk_bytes`), i.e. two holes were masking each other.
|
// steers the per-user limit off `free_disk_bytes`), i.e. two holes were masking each other.
|
||||||
state.disk_cache.invalidate();
|
state.disk_cache.invalidate();
|
||||||
|
|
||||||
|
// `media_total` caches SUM(user.total_upload_bytes) for the upload gate's keepsake-headroom
|
||||||
|
// check. TRUNCATE has just zeroed every one of those rows, so a surviving reading would make
|
||||||
|
// the next test's first upload measure its headroom against the previous test's gallery —
|
||||||
|
// and that gate REFUSES uploads, so the failure would look like a spurious quota rejection.
|
||||||
|
state.media_total.invalidate();
|
||||||
|
|
||||||
// `sse_tickets` maps a ticket to a session token hash. TRUNCATE deletes the sessions, so every
|
// `sse_tickets` maps a ticket to a session token hash. TRUNCATE deletes the sessions, so every
|
||||||
// surviving ticket is a dangling reference to a user that no longer exists.
|
// surviving ticket is a dangling reference to a user that no longer exists.
|
||||||
state.sse_tickets.clear();
|
state.sse_tickets.clear();
|
||||||
|
|||||||
@@ -438,43 +438,62 @@ pub async fn upload(
|
|||||||
let quota_on = config::get_bool(&state.config_cache, "quota_enabled", true).await;
|
let quota_on = config::get_bool(&state.config_cache, "quota_enabled", true).await;
|
||||||
let storage_quota_on =
|
let storage_quota_on =
|
||||||
config::get_bool(&state.config_cache, "storage_quota_enabled", true).await;
|
config::get_bool(&state.config_cache, "storage_quota_enabled", true).await;
|
||||||
// When quota is enforced, this holds the byte ceiling so the increment UPDATE below can
|
// GLOBAL DISK GATE, checked before the per-user ceiling and independent of every quota
|
||||||
// enforce it atomically (`WHERE total + size <= limit`). Without that guard, two
|
|
||||||
// concurrent uploads from the same user (e.g. phone + laptop) both pass this stale
|
|
||||||
// pre-check and both increment, blowing past the quota. The pre-check stays as a
|
|
||||||
// fast path that avoids the disk write when the user is already clearly over.
|
|
||||||
// GLOBAL RESERVE, checked before the per-user ceiling and independent of every quota
|
|
||||||
// toggle. The per-user quota is a fairness mechanism, not a disk guarantee — and since it
|
// toggle. The per-user quota is a fairness mechanism, not a disk guarantee — and since it
|
||||||
// now carries a floor (MIN_QUOTA_LIMIT_BYTES) so a guest's allowance stops shrinking as
|
// carries a floor (MIN_QUOTA_LIMIT_BYTES) so a guest's allowance stops shrinking as the
|
||||||
// the party fills up, the aggregate ceiling it used to imply is gone entirely. Something
|
// party fills up, the aggregate ceiling it used to imply is gone entirely. Something has to
|
||||||
// has to own "do not fill the volume", because `postgres_data`, `media_data` and
|
// own "do not fill the volume", because `postgres_data`, `media_data` and `exports_data`
|
||||||
// `exports_data` share one filesystem: the end state is not a degraded feature, it is
|
// share one filesystem: the end state is not a degraded feature, it is Postgres unable to
|
||||||
// Postgres unable to write WAL and the whole event down with nobody watching.
|
// write WAL and the whole event down with nobody watching.
|
||||||
//
|
//
|
||||||
// Deliberately NOT gated behind `quota_enabled`. That switch exists so an operator can
|
// WHAT IS RESERVED IS NOT A CONSTANT. A flat reserve answers "can Postgres still write",
|
||||||
// stop rationing space between guests; it was never meant to authorise running the disk
|
// which is necessary and not sufficient: the keepsake needs room for BOTH halves at once —
|
||||||
// to zero, and an operator flipping it at 23:00 to unblock a guest should not silently
|
// `required_free_bytes` is `media × 1.1 × 2`, since the ZIP and the HTML viewer are each
|
||||||
// disarm the last thing standing between the party and a dead database.
|
// gallery-sized. On the 40 GB box this runs on, a flat 10 GB reserve let uploads continue to
|
||||||
|
// roughly 25 GB of media while the release needed `2.2 × 25 + 10` = 65 GB free. Every upload
|
||||||
|
// in that band succeeded and then the archive could never be built — the product's entire
|
||||||
|
// promise, failing silently at the end of the night with nobody there to notice.
|
||||||
|
//
|
||||||
|
// So the gate enforces the invariant that actually matters: never accept an upload that
|
||||||
|
// would make the keepsake unbuildable. It shares `required_free_bytes` with the export
|
||||||
|
// preflight so the two cannot drift into disagreeing about the same question.
|
||||||
|
//
|
||||||
|
// Deliberately NOT gated behind `quota_enabled`. That switch exists so an operator can stop
|
||||||
|
// rationing space between guests; it was never meant to authorise running the disk to zero,
|
||||||
|
// and an operator flipping it at 23:00 to unblock a guest should not silently disarm the
|
||||||
|
// last thing standing between the party and a dead database.
|
||||||
if let Some(free) = crate::services::disk::free_bytes(&state.config.media_path) {
|
if let Some(free) = crate::services::disk::free_bytes(&state.config.media_path) {
|
||||||
let remaining = (free as i64).saturating_sub(size);
|
let media_after = state.media_total.get(&state.pool).await.saturating_add(size);
|
||||||
if remaining < DISK_RESERVE_BYTES {
|
let keepsake_needs =
|
||||||
|
crate::services::export::required_free_bytes(media_after.max(0) as u64, 2) as i64;
|
||||||
|
let free_after = (free as i64).saturating_sub(size);
|
||||||
|
let required = keepsake_needs.saturating_add(DISK_RESERVE_BYTES);
|
||||||
|
if free_after < required {
|
||||||
tracing::error!(
|
tracing::error!(
|
||||||
free_bytes = free,
|
free_bytes = free,
|
||||||
upload_size = size,
|
upload_size = size,
|
||||||
|
media_after,
|
||||||
|
keepsake_needs,
|
||||||
reserve = DISK_RESERVE_BYTES,
|
reserve = DISK_RESERVE_BYTES,
|
||||||
"refusing upload: it would take the media volume below the reserve"
|
"refusing upload: it would leave too little room to build the keepsake"
|
||||||
);
|
);
|
||||||
return Err(AppError::QuotaExceeded(
|
return Err(AppError::QuotaExceeded(
|
||||||
"Der Speicher des Events ist voll. Bitte sag einem Host Bescheid — neue \
|
"Der Speicher des Events ist fast voll — damit die Galerie am Ende noch als \
|
||||||
Uploads sind vorübergehend nicht möglich."
|
Download erstellt werden kann, sind neue Uploads jetzt gesperrt. Bitte sag \
|
||||||
|
einem Host Bescheid."
|
||||||
.into(),
|
.into(),
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Failing OPEN when the disk can't be read is deliberate and matches the per-user quota
|
// Failing OPEN when the disk can't be read is deliberate and matches the per-user quota
|
||||||
// above: refusing every upload because a `statfs` failed would be a worse outage than the
|
// below: refusing every upload because a `statfs` failed would be a worse outage than the
|
||||||
// one being guarded against.
|
// one being guarded against.
|
||||||
|
|
||||||
|
// When quota is enforced, this holds the byte ceiling so the increment UPDATE below can
|
||||||
|
// enforce it atomically (`WHERE total + size <= limit`). Without that guard, two
|
||||||
|
// concurrent uploads from the same user (e.g. phone + laptop) both pass this stale
|
||||||
|
// pre-check and both increment, blowing past the quota. The pre-check stays as a
|
||||||
|
// fast path that avoids the disk write when the user is already clearly over.
|
||||||
let mut quota_limit: Option<i64> = None;
|
let mut quota_limit: Option<i64> = None;
|
||||||
if quota_on && storage_quota_on {
|
if quota_on && storage_quota_on {
|
||||||
let estimate = compute_storage_quota(&state).await;
|
let estimate = compute_storage_quota(&state).await;
|
||||||
@@ -1370,7 +1389,9 @@ pub async fn get_thumbnail(
|
|||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::{MIN_QUOTA_LIMIT_BYTES, RangeSpec, parse_range, quota_limit_bytes};
|
use super::{
|
||||||
|
DISK_RESERVE_BYTES, MIN_QUOTA_LIMIT_BYTES, RangeSpec, parse_range, quota_limit_bytes,
|
||||||
|
};
|
||||||
|
|
||||||
// `Range` handling exists because iOS Safari probes every `<video>` with
|
// `Range` handling exists because iOS Safari probes every `<video>` with
|
||||||
// `Range: bytes=0-1` and abandons the load without a 206. These pin the forms a
|
// `Range: bytes=0-1` and abandons the load without a 206. These pin the forms a
|
||||||
@@ -1523,6 +1544,67 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// THE INVARIANT THE UPLOAD GATE EXISTS FOR: if an upload is accepted, the keepsake must
|
||||||
|
/// still be buildable afterwards.
|
||||||
|
///
|
||||||
|
/// The gate and `ensure_export_space` answer the same question at different times, from the
|
||||||
|
/// same `required_free_bytes`. If they ever drift, the failure is silent and terminal — every
|
||||||
|
/// upload succeeds and the archive can never be built, discovered only when the host taps
|
||||||
|
/// release and there is nobody left to fix it. This pins the two together.
|
||||||
|
///
|
||||||
|
/// Models the real box: 40 GB volume, ~5 GB consumed by OS, images and Postgres.
|
||||||
|
#[test]
|
||||||
|
fn an_accepted_upload_always_leaves_room_to_build_the_keepsake() {
|
||||||
|
const USABLE: i64 = 35 * GB;
|
||||||
|
let reserve = DISK_RESERVE_BYTES;
|
||||||
|
|
||||||
|
// Walk the gallery upward in 250 MB steps and assert the two agree at every point.
|
||||||
|
let mut media: i64 = 0;
|
||||||
|
let step: i64 = 250 * 1024 * 1024;
|
||||||
|
let mut last_accepted = 0i64;
|
||||||
|
while media < USABLE {
|
||||||
|
let free = USABLE - media;
|
||||||
|
let media_after = media + step;
|
||||||
|
let free_after = free - step;
|
||||||
|
let required =
|
||||||
|
crate::services::export::required_free_bytes(media_after as u64, 2) as i64 + reserve;
|
||||||
|
let gate_accepts = free_after >= required;
|
||||||
|
|
||||||
|
if gate_accepts {
|
||||||
|
// The export preflight must agree, using the SAME arithmetic it will run later.
|
||||||
|
let preflight_needs =
|
||||||
|
crate::services::export::required_free_bytes(media_after as u64, 2) as i64
|
||||||
|
+ reserve;
|
||||||
|
assert!(
|
||||||
|
free_after >= preflight_needs,
|
||||||
|
"gate accepted at media={media_after} but the preflight would refuse"
|
||||||
|
);
|
||||||
|
last_accepted = media_after;
|
||||||
|
}
|
||||||
|
media = media_after;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sanity-check the ceiling is where the arithmetic says: 35 = 2.2·M + 10 ⇒ M ≈ 7.8 GB.
|
||||||
|
// Pinned loosely (6–9 GB) so a deliberate change to the overhead multiplier or the
|
||||||
|
// reserve fails this test loudly rather than silently moving the cliff.
|
||||||
|
assert!(
|
||||||
|
(6 * GB..=9 * GB).contains(&last_accepted),
|
||||||
|
"expected the gallery ceiling near 7.8 GB on a 35 GB volume, got {last_accepted} bytes"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The gate must be the binding constraint, not the per-user floor. With 100 guests each
|
||||||
|
/// allowed 500 MB, the per-user quota alone would authorise ~50 GB on a 40 GB disk.
|
||||||
|
#[test]
|
||||||
|
fn the_global_gate_binds_before_the_per_user_floor_can_overfill_the_disk() {
|
||||||
|
let per_user_total = MIN_QUOTA_LIMIT_BYTES * 100;
|
||||||
|
assert!(
|
||||||
|
per_user_total > 35 * GB,
|
||||||
|
"premise: the per-user floor alone over-commits the volume, so the global gate \
|
||||||
|
is what must stop it"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
/// The floor must never write a cheque the volume cannot cash — otherwise a full disk
|
/// The floor must never write a cheque the volume cannot cash — otherwise a full disk
|
||||||
/// still hands out a 500 MB allowance and the filesystem Postgres needs fills up.
|
/// still hands out a 500 MB allowance and the filesystem Postgres needs fills up.
|
||||||
#[test]
|
#[test]
|
||||||
|
|||||||
@@ -1354,7 +1354,7 @@ const EXPORT_SIZE_OVERHEAD_PCT: u64 = 110;
|
|||||||
/// Computed in `u128` and clamped, NOT with `saturating_mul`: saturating first and then dividing by
|
/// Computed in `u128` and clamped, NOT with `saturating_mul`: saturating first and then dividing by
|
||||||
/// 100 quietly turns an overflow into a number ~100x too small, which is the one direction that
|
/// 100 quietly turns an overflow into a number ~100x too small, which is the one direction that
|
||||||
/// matters here — an under-estimate authorises the very write the preflight exists to refuse.
|
/// matters here — an under-estimate authorises the very write the preflight exists to refuse.
|
||||||
fn required_free_bytes(media_bytes: u64, armed: i64) -> u64 {
|
pub(crate) fn required_free_bytes(media_bytes: u64, armed: i64) -> u64 {
|
||||||
let needed = media_bytes as u128 * EXPORT_SIZE_OVERHEAD_PCT as u128 / 100
|
let needed = media_bytes as u128 * EXPORT_SIZE_OVERHEAD_PCT as u128 / 100
|
||||||
* armed.max(1).min(i64::from(u32::MAX)) as u128;
|
* armed.max(1).min(i64::from(u32::MAX)) as u128;
|
||||||
needed.min(u64::MAX as u128) as u64
|
needed.min(u64::MAX as u128) as u64
|
||||||
|
|||||||
82
backend/src/services/media_total.rs
Normal file
82
backend/src/services/media_total.rs
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
//! Cached sum of all media bytes the event is holding.
|
||||||
|
//!
|
||||||
|
//! The upload gate needs to know "how big would the keepsake be if we accept this file", because
|
||||||
|
//! the archive needs room for BOTH halves at once (`export::required_free_bytes` is
|
||||||
|
//! `media × 1.1 × 2` — the ZIP and the HTML viewer are each gallery-sized). Asking that question
|
||||||
|
//! per upload has to be cheap, and it has to be cheap on the busiest write path in the app.
|
||||||
|
//!
|
||||||
|
//! `export::estimate_export_bytes` answers the same question exactly, but it aggregates
|
||||||
|
//! `original_size_bytes` across every upload row joined to `user` — fine once per release,
|
||||||
|
//! wasteful per upload and growing all evening. This sums `user.total_upload_bytes` instead:
|
||||||
|
//! one row per guest (~100), already maintained transactionally by the quota path, already
|
||||||
|
//! refunded on delete.
|
||||||
|
//!
|
||||||
|
//! The two differ slightly — this one counts uploads belonging to banned or hidden users, which
|
||||||
|
//! the export filters out. That skew is in the SAFE direction: it over-estimates the archive, so
|
||||||
|
//! the gate closes marginally early rather than marginally late. Never swap it for a cheaper
|
||||||
|
//! query that could under-estimate; an under-estimate authorises the very upload that makes the
|
||||||
|
//! keepsake unbuildable, which is the failure this exists to prevent.
|
||||||
|
|
||||||
|
use std::sync::{Arc, RwLock};
|
||||||
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
|
use sqlx::PgPool;
|
||||||
|
|
||||||
|
/// How long a reading is trusted. Shorter than [`crate::services::disk`]'s TTL because this
|
||||||
|
/// number only ever grows and does so in the same request path that reads it — a stale value
|
||||||
|
/// under-counts the newest uploads, and under-counting is the direction that matters.
|
||||||
|
const TTL: Duration = Duration::from_secs(5);
|
||||||
|
|
||||||
|
/// Cheap-to-clone cache of the event's total media bytes. Lives in `AppState`.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct MediaTotalCache {
|
||||||
|
inner: Arc<RwLock<Option<(i64, Instant)>>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MediaTotalCache {
|
||||||
|
pub fn new() -> Self {
|
||||||
|
Self {
|
||||||
|
inner: Arc::new(RwLock::new(None)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Drop the cached reading so the next `get()` re-queries.
|
||||||
|
///
|
||||||
|
/// Used by the e2e TRUNCATE endpoint for the same reason `DiskCache::invalidate` exists:
|
||||||
|
/// truncation removes every upload, and a surviving reading would make the next test's
|
||||||
|
/// gate compute against the previous test's data.
|
||||||
|
pub fn invalidate(&self) {
|
||||||
|
*self.inner.write().unwrap() = None;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Total bytes of media the event is holding, cached for [`TTL`].
|
||||||
|
///
|
||||||
|
/// Returns 0 when the query fails. That is a deliberate FAIL-OPEN, consistent with the
|
||||||
|
/// quota path and the export preflight: a database blip must not turn into "every upload
|
||||||
|
/// refused". The disk-space half of the gate still applies, so a failure here degrades the
|
||||||
|
/// check to the old flat-reserve behaviour rather than disabling it.
|
||||||
|
pub async fn get(&self, pool: &PgPool) -> i64 {
|
||||||
|
if let Some((bytes, at)) = *self.inner.read().unwrap()
|
||||||
|
&& at.elapsed() < TTL
|
||||||
|
{
|
||||||
|
return bytes;
|
||||||
|
}
|
||||||
|
let bytes = sqlx::query_scalar::<_, Option<i64>>(
|
||||||
|
"SELECT SUM(total_upload_bytes)::bigint FROM \"user\"",
|
||||||
|
)
|
||||||
|
.fetch_one(pool)
|
||||||
|
.await
|
||||||
|
.ok()
|
||||||
|
.flatten()
|
||||||
|
.unwrap_or(0)
|
||||||
|
.max(0);
|
||||||
|
*self.inner.write().unwrap() = Some((bytes, Instant::now()));
|
||||||
|
bytes
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for MediaTotalCache {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self::new()
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -4,6 +4,7 @@ pub mod disk;
|
|||||||
pub mod export;
|
pub mod export;
|
||||||
pub mod imaging;
|
pub mod imaging;
|
||||||
pub mod maintenance;
|
pub mod maintenance;
|
||||||
|
pub mod media_total;
|
||||||
pub mod rate_limiter;
|
pub mod rate_limiter;
|
||||||
pub mod sse_tickets;
|
pub mod sse_tickets;
|
||||||
pub mod video;
|
pub mod video;
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ use crate::config::AppConfig;
|
|||||||
use crate::services::compression::CompressionWorker;
|
use crate::services::compression::CompressionWorker;
|
||||||
use crate::services::config::ConfigCache;
|
use crate::services::config::ConfigCache;
|
||||||
use crate::services::disk::DiskCache;
|
use crate::services::disk::DiskCache;
|
||||||
|
use crate::services::media_total::MediaTotalCache;
|
||||||
use crate::services::rate_limiter::RateLimiter;
|
use crate::services::rate_limiter::RateLimiter;
|
||||||
use crate::services::sse_tickets::SseTicketStore;
|
use crate::services::sse_tickets::SseTicketStore;
|
||||||
|
|
||||||
@@ -38,6 +39,8 @@ pub struct AppState {
|
|||||||
pub config_cache: ConfigCache,
|
pub config_cache: ConfigCache,
|
||||||
/// Cached total/free bytes for the media filesystem (quota + admin stats).
|
/// Cached total/free bytes for the media filesystem (quota + admin stats).
|
||||||
pub disk_cache: DiskCache,
|
pub disk_cache: DiskCache,
|
||||||
|
/// Cached sum of all media bytes, for the upload gate's keepsake-headroom check.
|
||||||
|
pub media_total: MediaTotalCache,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl AppState {
|
impl AppState {
|
||||||
@@ -63,6 +66,7 @@ impl AppState {
|
|||||||
sse_tickets: SseTicketStore::new(),
|
sse_tickets: SseTicketStore::new(),
|
||||||
config_cache,
|
config_cache,
|
||||||
disk_cache: DiskCache::new(),
|
disk_cache: DiskCache::new(),
|
||||||
|
media_total: MediaTotalCache::new(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
@@ -31,14 +31,16 @@ services:
|
|||||||
deploy:
|
deploy:
|
||||||
resources:
|
resources:
|
||||||
limits:
|
limits:
|
||||||
# 1G, not 512M. DATABASE_MAX_CONNECTIONS defaults to 30 for a ~100-guest event
|
# 1G, not 512M. Postgres 16's default shared_buffers plus a pool of backends
|
||||||
# (feed polling + SSE + uploads at once), and 30 backends plus Postgres 16's
|
# leaves very little headroom at 512M, and an OOM here does not degrade one
|
||||||
# default shared_buffers leaves very little headroom at 512M. An OOM here does
|
# feature — it takes the event down, because every request path touches the
|
||||||
# not degrade one feature — it takes the event down, because every request
|
# database.
|
||||||
# path touches the database. Memory is the cheaper knob than shrinking the
|
|
||||||
# pool back and reintroducing the queueing it was raised to fix.
|
|
||||||
#
|
#
|
||||||
# Raising DATABASE_MAX_CONNECTIONS further means raising this too.
|
# `.env.example` now sets DATABASE_MAX_CONNECTIONS=15, sized to the 2 vCPU this
|
||||||
|
# box has rather than to the guest count: since migration 024 replaced the feed
|
||||||
|
# view's GROUP BY with scalar subqueries, a feed page costs well under a
|
||||||
|
# millisecond, so connections are no longer spent waiting. Raising it back
|
||||||
|
# toward 30 means raising this limit with it.
|
||||||
memory: 1G
|
memory: 1G
|
||||||
|
|
||||||
app:
|
app:
|
||||||
|
|||||||
@@ -46,6 +46,33 @@
|
|||||||
document.head.appendChild(s);
|
document.head.appendChild(s);
|
||||||
}
|
}
|
||||||
} catch (_) {}
|
} catch (_) {}
|
||||||
|
|
||||||
|
// Boot backstop. With `ssr = false` the page is EMPTY until the bundle mounts, so
|
||||||
|
// anything that stops it mounting leaves the guest on the spinner below forever:
|
||||||
|
// a chunk 404 after a redeploy, a dead uplink, or untranspiled syntax on an old
|
||||||
|
// phone. There is no message, no reload control, and in a standalone PWA no URL
|
||||||
|
// bar to escape from — the app is simply bricked for that guest, all evening.
|
||||||
|
//
|
||||||
|
// A TIMEOUT, deliberately, not feature detection: a SyntaxError in the bundle is
|
||||||
|
// invisible to any capability check, whereas "still not painted" catches every
|
||||||
|
// cause at once. The root layout removes #app-boot on mount, so its continued
|
||||||
|
// presence IS the failure signal and nothing needs to cancel this.
|
||||||
|
//
|
||||||
|
// 15s is well beyond a slow-but-healthy load on venue wifi (the bundle is ~130 kB
|
||||||
|
// gzipped); erring long matters more than erring short, because a false positive
|
||||||
|
// here would tell a guest something is broken while it is quietly working.
|
||||||
|
setTimeout(function () {
|
||||||
|
var boot = document.getElementById('app-boot');
|
||||||
|
if (!boot) return; // app mounted — nothing to do
|
||||||
|
boot.innerHTML =
|
||||||
|
'<div style="max-width:20rem;text-align:center">' +
|
||||||
|
'<p style="font-family:Georgia,serif;font-size:1.125rem;font-weight:600;margin:0 0 .5rem">Die App konnte nicht geladen werden</p>' +
|
||||||
|
'<p style="margin:0 0 1.25rem;color:#545350;line-height:1.5">Bitte prüf deine Verbindung und lade die Seite neu.</p>' +
|
||||||
|
'<button id="app-boot-reload" style="font:inherit;font-weight:600;cursor:pointer;border:0;border-radius:.5rem;padding:.625rem 1.25rem;background:#8a6a2b;color:#fff">Neu laden</button>' +
|
||||||
|
'</div>';
|
||||||
|
var btn = document.getElementById('app-boot-reload');
|
||||||
|
if (btn) btn.addEventListener('click', function () { location.reload(); });
|
||||||
|
}, 15000);
|
||||||
})();
|
})();
|
||||||
</script>
|
</script>
|
||||||
%sveltekit.head%
|
%sveltekit.head%
|
||||||
@@ -65,6 +92,14 @@
|
|||||||
<span class="app-boot__spinner"></span>
|
<span class="app-boot__spinner"></span>
|
||||||
<span class="app-boot__label">EventSnap</span>
|
<span class="app-boot__label">EventSnap</span>
|
||||||
</div>
|
</div>
|
||||||
|
<!-- The app is client-rendered end to end, so with JS off there is nothing at all to
|
||||||
|
show. Say so, rather than leaving a permanent spinner over a blank page. -->
|
||||||
|
<noscript>
|
||||||
|
<div class="app-boot__noscript">
|
||||||
|
<p><strong>EventSnap braucht JavaScript</strong></p>
|
||||||
|
<p>Bitte aktiviere JavaScript im Browser und lade die Seite neu.</p>
|
||||||
|
</div>
|
||||||
|
</noscript>
|
||||||
<style>
|
<style>
|
||||||
#app-boot {
|
#app-boot {
|
||||||
position: fixed;
|
position: fixed;
|
||||||
@@ -103,6 +138,26 @@
|
|||||||
transform: rotate(360deg);
|
transform: rotate(360deg);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
/* Sits above #app-boot, which is `position: fixed` and would otherwise cover it. */
|
||||||
|
.app-boot__noscript {
|
||||||
|
position: fixed;
|
||||||
|
inset: 0;
|
||||||
|
z-index: 1;
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
gap: 0.25rem;
|
||||||
|
padding: 2rem;
|
||||||
|
text-align: center;
|
||||||
|
background: #faf9f7;
|
||||||
|
font-family: system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif;
|
||||||
|
color: #545350;
|
||||||
|
}
|
||||||
|
html.dark .app-boot__noscript {
|
||||||
|
background: #100f0f;
|
||||||
|
color: #a6a4a1;
|
||||||
|
}
|
||||||
</style>
|
</style>
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
|
|||||||
@@ -3,7 +3,8 @@ import {
|
|||||||
classifyUploadStatus,
|
classifyUploadStatus,
|
||||||
isReversibleLock,
|
isReversibleLock,
|
||||||
entryToQueueItem,
|
entryToQueueItem,
|
||||||
shouldAbortForStall
|
shouldAbortForStall,
|
||||||
|
suspendedSinceLastTick
|
||||||
} from './upload-queue';
|
} from './upload-queue';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -140,4 +141,55 @@ describe('shouldAbortForStall', () => {
|
|||||||
expect(shouldAbortForStall(now - 91_000, now, true)).toBe(false);
|
expect(shouldAbortForStall(now - 91_000, now, true)).toBe(false);
|
||||||
expect(shouldAbortForStall(now - 121_000, now, true)).toBe(true);
|
expect(shouldAbortForStall(now - 121_000, now, true)).toBe(true);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The watchdog measures SILENCE via `Date.now()`, but a backgrounded phone freezes the
|
||||||
|
* interval while the clock keeps running. Without crediting the un-run window back, the first
|
||||||
|
* tick after a screen lock reads the whole sleep as a stall and aborts a healthy upload —
|
||||||
|
* re-sending from byte zero and spending one of five permanent auto-attempts. That is what
|
||||||
|
* every phone does between shots at a party.
|
||||||
|
*
|
||||||
|
* Detecting the freeze from the tick gap (rather than from `visibilitychange`) also covers the
|
||||||
|
* causes that fire no visibility event at all: a throttled-but-visible tab, a closed lid, an
|
||||||
|
* occluded window.
|
||||||
|
*/
|
||||||
|
describe('suspendedSinceLastTick', () => {
|
||||||
|
const now = 1_000_000;
|
||||||
|
|
||||||
|
it('credits nothing for a tick that arrived on schedule', () => {
|
||||||
|
expect(suspendedSinceLastTick(now - 5_000, now, 5_000)).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('credits nothing for ordinary timer jitter or throttling', () => {
|
||||||
|
expect(suspendedSinceLastTick(now - 6_900, now, 5_000)).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('credits the whole frozen window when the interval did not run', () => {
|
||||||
|
// Screen locked ~2 minutes: a 5s interval arriving 130s late.
|
||||||
|
expect(suspendedSinceLastTick(now - 130_000, now, 5_000)).toBe(125_000);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('credits nothing when the clock jumps backwards (NTP correction)', () => {
|
||||||
|
expect(suspendedSinceLastTick(now + 60_000, now, 5_000)).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('a suspension longer than the stall ceiling does not abort a healthy upload', () => {
|
||||||
|
// The bug, end to end: 3 minutes suspended, interval resumes, no bytes since.
|
||||||
|
const lastProgressAt = now - 180_000;
|
||||||
|
const credited = Math.min(
|
||||||
|
now,
|
||||||
|
lastProgressAt + suspendedSinceLastTick(now - 185_000, now, 5_000)
|
||||||
|
);
|
||||||
|
expect(shouldAbortForStall(credited, now, false)).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('but a socket still silent 91s AFTER resume is aborted, never left to xhr.timeout', () => {
|
||||||
|
// iOS reaps backgrounded sockets without firing `error`. The credit buys one fresh
|
||||||
|
// window, not immunity — otherwise a dead upload would hang for 5-60 minutes holding
|
||||||
|
// the queue's `processing` latch.
|
||||||
|
const resumedAt = now - 91_000;
|
||||||
|
expect(shouldAbortForStall(resumedAt, now, false)).toBe(true);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -94,6 +94,45 @@ export function shouldAbortForStall(
|
|||||||
return now - lastActivityAt > ceiling;
|
return now - lastActivityAt > ceiling;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Normal timer jitter/throttling budget. A tick later than `interval + this` did not run
|
||||||
|
* because the page was suspended, not because it was merely late.
|
||||||
|
*/
|
||||||
|
const SUSPEND_TOLERANCE_MS = 2_000;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Wall-clock the watchdog interval FAILED to cover because the page was suspended.
|
||||||
|
*
|
||||||
|
* `Date.now()` keeps advancing while a backgrounded phone is frozen, but `setInterval` does
|
||||||
|
* not run. So the first tick after a screen lock saw the entire sleep as "no bytes moved" and
|
||||||
|
* aborted a connection that was very possibly healthy — restarting a 200 MB video from byte
|
||||||
|
* zero, burning one of five PERMANENT auto-attempts (`chargeAttempt`), and breaking the drain
|
||||||
|
* loop for every other queued photo. A phone in a pocket between shots is the common case at a
|
||||||
|
* party, not an edge case.
|
||||||
|
*
|
||||||
|
* The interval is its own suspension detector: a tick scheduled 5s out that arrives 130s late
|
||||||
|
* means the page was frozen for ~125s, and that is exactly the window the watchdog had no
|
||||||
|
* right to measure. Deliberately chosen over a `visibilitychange` listener, which only covers
|
||||||
|
* the causes that happen to fire that event — a throttled-but-visible tab, a closed laptop lid
|
||||||
|
* and an occluded window all freeze timers without one. It also needs no listener, no
|
||||||
|
* module-level state, no SSR guard and no teardown.
|
||||||
|
*
|
||||||
|
* `performance.now()` was rejected as the clock source: Safari pauses it across system sleep
|
||||||
|
* on some paths while Chrome does not, which is precisely the non-uniformity that makes it
|
||||||
|
* unusable as the sole signal.
|
||||||
|
*
|
||||||
|
* Returns 0 for a normal tick, and 0 if the clock jumps BACKWARDS (an NTP correction) — that
|
||||||
|
* fails open, and the wall-clock `xhr.timeout` still bounds the request.
|
||||||
|
*/
|
||||||
|
export function suspendedSinceLastTick(
|
||||||
|
lastTickAt: number,
|
||||||
|
now: number,
|
||||||
|
intervalMs: number = STALL_CHECK_INTERVAL_MS
|
||||||
|
): number {
|
||||||
|
const overshoot = now - lastTickAt - intervalMs;
|
||||||
|
return overshoot > SUSPEND_TOLERANCE_MS ? overshoot : 0;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Wall-clock cap for one attempt, scaled by file size assuming a floor of ~8 kB/s — a
|
* Wall-clock cap for one attempt, scaled by file size assuming a floor of ~8 kB/s — a
|
||||||
* deliberately pessimistic rate, because killing a slow-but-progressing upload would lose
|
* deliberately pessimistic rate, because killing a slow-but-progressing upload would lose
|
||||||
@@ -905,11 +944,25 @@ async function uploadItem(id: string): Promise<void> {
|
|||||||
// connection that never errors and never completes. Only "no bytes moved" catches
|
// connection that never errors and never completes. Only "no bytes moved" catches
|
||||||
// that without also punishing a healthy slow link.
|
// that without also punishing a healthy slow link.
|
||||||
let lastProgressAt = Date.now();
|
let lastProgressAt = Date.now();
|
||||||
|
let lastTickAt = Date.now();
|
||||||
let bodySent = false;
|
let bodySent = false;
|
||||||
let stalled = false;
|
let stalled = false;
|
||||||
const stallTimer = setInterval(() => {
|
const stallTimer = setInterval(() => {
|
||||||
if (!shouldAbortForStall(lastProgressAt, Date.now(), bodySent)) return;
|
// Never fire twice. `xhr.abort()` on a request already in `readyState === DONE`
|
||||||
|
// emits NO `abort` event, so `settle()` would never run: the interval would keep
|
||||||
|
// running forever, re-aborting every 5s, and `activeUploads` would keep a stale
|
||||||
|
// entry so the guest's ✕ button silently did nothing.
|
||||||
|
if (stalled) return;
|
||||||
|
const now = Date.now();
|
||||||
|
// Credit back the window the page was frozen. The watchdog measures SILENCE, and
|
||||||
|
// a period in which it could not observe anything is not evidence of silence.
|
||||||
|
// Clamped to `now` so a progress event delivered right at resume cannot push the
|
||||||
|
// timestamp into the future.
|
||||||
|
lastProgressAt = Math.min(now, lastProgressAt + suspendedSinceLastTick(lastTickAt, now));
|
||||||
|
lastTickAt = now;
|
||||||
|
if (!shouldAbortForStall(lastProgressAt, now, bodySent)) return;
|
||||||
stalled = true;
|
stalled = true;
|
||||||
|
clearInterval(stallTimer);
|
||||||
xhr.abort();
|
xhr.abort();
|
||||||
}, STALL_CHECK_INTERVAL_MS);
|
}, STALL_CHECK_INTERVAL_MS);
|
||||||
const settle = (fn: () => void) => {
|
const settle = (fn: () => void) => {
|
||||||
@@ -1018,7 +1071,16 @@ async function uploadItem(id: string): Promise<void> {
|
|||||||
else reject(new NetworkError('Abgebrochen'));
|
else reject(new NetworkError('Abgebrochen'));
|
||||||
})
|
})
|
||||||
);
|
);
|
||||||
|
// `send` can throw SYNCHRONOUSLY — most plausibly on a phone whose OS purged the
|
||||||
|
// backing store for the blob, leaving a neutered File. The executor would turn that
|
||||||
|
// into a rejection and `uploadItem` would recover, but `settle()` never runs: the
|
||||||
|
// stall interval leaks and `activeUploads` keeps a stale entry, so the ✕ button on
|
||||||
|
// that item stops working for the rest of the session.
|
||||||
|
try {
|
||||||
xhr.send(formData);
|
xhr.send(formData);
|
||||||
|
} catch {
|
||||||
|
settle(() => reject(new NetworkError('Netzwerkfehler')));
|
||||||
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
// Success — remove blob from IndexedDB, mark done
|
// Success — remove blob from IndexedDB, mark done
|
||||||
|
|||||||
@@ -95,11 +95,81 @@
|
|||||||
--color-purple-700: #6f5523;
|
--color-purple-700: #6f5523;
|
||||||
--color-purple-800: #59441e;
|
--color-purple-800: #59441e;
|
||||||
--color-purple-900: #493819;
|
--color-purple-900: #493819;
|
||||||
|
/* The ramp stopped at 900 while blue/primary both run to 950, so
|
||||||
|
* `dark:bg-purple-950/50` (routes/host/+page.svelte) fell through to Tailwind's default
|
||||||
|
* violet — off-brand on every browser, modern ones included. Mirrors `--color-blue-950`. */
|
||||||
|
--color-purple-950: #29200d;
|
||||||
--color-violet-500: #ab8433;
|
--color-violet-500: #ab8433;
|
||||||
--color-violet-600: #8a6a2b;
|
--color-violet-600: #8a6a2b;
|
||||||
--color-accent-500: #ab8433;
|
--color-accent-500: #ab8433;
|
||||||
--color-accent-600: #8a6a2b;
|
--color-accent-600: #8a6a2b;
|
||||||
|
|
||||||
|
/* ── Semantic: red / amber / green ───────────────────────────────────────────
|
||||||
|
* PINNED TO HEX rather than inherited. Tailwind v4 ships these families as
|
||||||
|
* `oklch()`, which Safari <15.4, Chrome <111 and Samsung Internet <22 (the default
|
||||||
|
* browser on Samsung phones) cannot parse. The declaration is accepted but
|
||||||
|
* `var(--color-red-600)` is then invalid at computed-value time, so `background-color`
|
||||||
|
* falls back to `initial` — transparent — and `.btn-danger` in the component layer
|
||||||
|
* renders white text on nothing. An invisible delete-confirm button.
|
||||||
|
*
|
||||||
|
* Note this is NOT the `@apply` hard-fail described for `primary` above: these families
|
||||||
|
* have Tailwind defaults, so an unpinned stop does not break the build, it silently
|
||||||
|
* reverts to oklch. Full 50-950 ramps are therefore about closing that silent-leak class
|
||||||
|
* permanently, not about compiling.
|
||||||
|
*
|
||||||
|
* Values are Tailwind 4.2.2's own defaults gamut-mapped to sRGB by Lightning CSS — the
|
||||||
|
* same converter already in this build pipeline, which is why they match the hex
|
||||||
|
* fallbacks it emits for the `/alpha` opacity forms (verified against `#bf000f`,
|
||||||
|
* `#460809`, `#461901`, `#032e15`, `#82181a`, `#ffa3a3`, `#0d542b` in the shipped CSS).
|
||||||
|
* Naive channel clipping gives different, wrong values for the out-of-gamut stops.
|
||||||
|
* Modern browsers therefore render exactly what they render today. */
|
||||||
|
--color-red-50: #fef2f2;
|
||||||
|
--color-red-100: #ffe2e2;
|
||||||
|
--color-red-200: #ffcaca;
|
||||||
|
--color-red-300: #ffa3a3;
|
||||||
|
--color-red-400: #ff6568;
|
||||||
|
--color-red-500: #fb2c36;
|
||||||
|
--color-red-600: #e40014;
|
||||||
|
--color-red-700: #bf000f;
|
||||||
|
--color-red-800: #9f0712;
|
||||||
|
--color-red-900: #82181a;
|
||||||
|
--color-red-950: #460809;
|
||||||
|
|
||||||
|
--color-amber-50: #fffbeb;
|
||||||
|
--color-amber-100: #fef3c6;
|
||||||
|
--color-amber-200: #fee685;
|
||||||
|
--color-amber-300: #ffd236;
|
||||||
|
--color-amber-400: #fcbb00;
|
||||||
|
--color-amber-500: #f99c00;
|
||||||
|
--color-amber-600: #dd7400;
|
||||||
|
--color-amber-700: #b75000;
|
||||||
|
--color-amber-800: #953d00;
|
||||||
|
--color-amber-900: #7b3306;
|
||||||
|
--color-amber-950: #461901;
|
||||||
|
|
||||||
|
--color-green-50: #f0fdf4;
|
||||||
|
--color-green-100: #dcfce7;
|
||||||
|
--color-green-200: #b9f8cf;
|
||||||
|
--color-green-300: #7bf1a8;
|
||||||
|
--color-green-400: #05df72;
|
||||||
|
--color-green-500: #00c758;
|
||||||
|
--color-green-600: #00a544;
|
||||||
|
--color-green-700: #008138;
|
||||||
|
--color-green-800: #016630;
|
||||||
|
--color-green-900: #0d542b;
|
||||||
|
--color-green-950: #032e15;
|
||||||
|
|
||||||
|
/* Avatar chips (lib/avatar.ts) — same oklch problem, same fix. Only the stops those
|
||||||
|
* chips actually use; nothing else in the app references rose or teal. */
|
||||||
|
--color-rose-100: #ffe4e6;
|
||||||
|
--color-rose-200: #ffccd3;
|
||||||
|
--color-rose-700: #c20039;
|
||||||
|
--color-rose-900: #8b0836;
|
||||||
|
--color-teal-100: #cbfbf1;
|
||||||
|
--color-teal-200: #96f7e4;
|
||||||
|
--color-teal-700: #00776e;
|
||||||
|
--color-teal-900: #0b4f4a;
|
||||||
|
|
||||||
/* ── Neutrals: pearl → silver → graphite (remaps `gray-*`). Whisper-warm
|
/* ── Neutrals: pearl → silver → graphite (remaps `gray-*`). Whisper-warm
|
||||||
* pearl at the light end (research: "warm pearl, not stark white"), turning
|
* pearl at the light end (research: "warm pearl, not stark white"), turning
|
||||||
* neutral-cool through the mids/darks so structure reads as silver, not sand. */
|
* neutral-cool through the mids/darks so structure reads as silver, not sand. */
|
||||||
|
|||||||
Reference in New Issue
Block a user