fix: gate uploads on keepsake headroom, and close five unattended-event gaps
The box is 2 vCPU / 4 GB / 40 GB, not the 4 vCPU / 8 GB / 80 GB that the audit, the committed comments and README's sizing section all assumed. That correction is what the first change is about; the rest are the remaining pre-event items. THE ARCHIVE COULD BECOME UNBUILDABLE WHILE UPLOADS KEPT SUCCEEDING `required_free_bytes` is `media × 1.1 × 2` — the ZIP and the HTML viewer are each gallery-sized — and the export preflight also wants DISK_RESERVE_BYTES on top. The upload gate, though, only refused below a FLAT 10 GB reserve. On 40 GB that let uploads run to ~25 GB of media while a release needed `2.2 × 25 + 10` = 65 GB free. Every upload in that band succeeded and the keepsake could then never be built: the product's entire promise, failing silently at the end of the night with nobody there. The gate now enforces the invariant that actually matters — never accept an upload that would make the keepsake unbuildable — sharing `required_free_bytes` with the preflight so the two cannot drift into disagreeing about the same question. Uploads stop at ~8 GB of media on this disk, with a German message naming the cause. Refusing the 1001st photo beats losing all 1000. `media_total.rs` backs it: SUM(user.total_upload_bytes) over ~100 rows, cached 5s, rather than `estimate_export_bytes`'s join across every upload. It counts hidden and banned users' bytes, which the export excludes — skew in the SAFE direction, so the gate closes marginally early rather than late. Fails open on a query error. A test pins the gate against the preflight across the whole gallery-size range, and a second asserts the per-user floor alone would over-commit the volume — i.e. that the global gate is what must bind. THE WATCHDOG ABORTED HEALTHY UPLOADS EVERY TIME A PHONE WAS POCKETED `Date.now()` advances while a backgrounded phone is frozen but `setInterval` does not, so the first tick after a screen lock read the whole sleep as silence and aborted — re-sending a video from byte zero and burning one of five PERMANENT auto-attempts. The interval is now its own suspension detector: a tick that arrives 125s late for a 5s schedule credits that window back, because a period the watchdog could not observe is not evidence of silence. Chosen over a `visibilitychange` listener, which only covers causes that fire that event — a throttled-but-visible tab, a closed lid and an occluded window all freeze timers without one — and which would have needed module state, an SSR guard and a teardown for strictly less coverage. `performance.now()` was rejected because Safari pauses it across system sleep on some paths and Chrome does not. The credit buys one fresh window, not immunity: a socket iOS reaped while backgrounded still aborts ~90s after resume rather than hanging for `xhr.timeout` (5-60 min) with the queue's `processing` latch held. Two latent leaks found while in there: `xhr.abort()` on a request already in readyState DONE emits no `abort` event, so `settle()` never ran and the interval re-aborted every 5s forever while `activeUploads` kept a stale entry (the ✕ button silently stopped working); and a synchronous throw from `xhr.send` — a blob whose backing store the OS purged — leaked the same way. Both closed. OKLCH MADE THE DELETE BUTTON INVISIBLE ON SAMSUNG'S DEFAULT BROWSER red/amber/green were never in the @theme block and fell through to Tailwind v4's `oklch()` defaults, which Safari <15.4, Chrome <111 and Samsung Internet <22 cannot parse: `var(--color-red-600)` is then invalid at computed-value time, `background-color` falls back to transparent, and `.btn-danger` renders white text on nothing. Pinned to Tailwind's own defaults gamut-mapped to sRGB by Lightning CSS — the converter already in this pipeline — so modern browsers render exactly what they render today. Verified against seven hex fallbacks it had already emitted for the /alpha forms. rose and teal (avatar chips) had the same leak. The app CSS goes from 40 oklch declarations to 0. Also fixes `--color-purple-950`, which was simply missing: `dark:bg-purple-950/50` on the host dashboard was rendering default violet on EVERY browser, off-brand. The keepsake viewer only picks this up on a rebuild, so its committed artefact is rebuilt here too — still single-file, still zero external references. A BRICKED BOOT LOOKED LIKE A SPINNER FOREVER With `ssr = false` the page is empty until the bundle mounts, so a chunk 404 after a redeploy or a dead uplink left the guest on the boot spinner with no message, no reload control, and in a standalone PWA no URL bar. A 15s timeout in the existing nonce'd IIFE (no CSP change) swaps in German copy and a reload button. Deliberately a timeout rather than feature detection: a SyntaxError in the bundle is invisible to any capability check. Plus a <noscript>, since there was nothing at all to see without JS. EVERY 4xx WAS INVISIBLE AT ANY LOG LEVEL tower_http counts 4xx as a success, so it logs at DEBUG while production runs at info. If guests spend the evening hitting 429s or 413s, the post-event logs said nothing. Now one WARN per client error; 5xx excluded because Internal already logs its source chain and the pool-exhaustion 503 logs at construction. A DEAD FRONTEND SERVED A BLANK 502 `handle_errors 5xx` with an inline German page (the caddy service mounts only the Caddyfile, so there is no volume to ship a static file through). Verified empirically against this config, not from documentation: an upstream 404 through `reverse_proxy` still arrives as untouched `application/json`, and only a dial failure renders the page. That mattered — the keepsake download navigates a hidden iframe and DEPENDS on a real 404/429 arriving, and swallowing those would have been worse than the blank 502. CONFIG CORRECTIONS FOR THE REAL HARDWARE DATABASE_MAX_CONNECTIONS 30 → 15: sized to 2 vCPU rather than to the guest count. Since migration 024 a feed page costs well under a millisecond, so connections are no longer spent waiting, and 30 backends crowd the db container's 1 GB on a 4 GB host. COMPRESSION_WORKER_CONCURRENCY stays at 2 — the merged heavy-image permit already serialises anything over 150 MiB, so the "two 48 MP photos" worst case that number was sized against is unreachable; dropping to 1 would halve light-path throughput and push more feed tiles onto full-size originals. README's sizing section rewritten for the actual disk. Verified: 149/149 backend tests against a live Postgres, clippy clean, 57/57 vitest, svelte-check 0 errors, eslint clean, vite build, export-viewer rebuild, caddy validate, compose YAML parse. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
56
README.md
56
README.md
@@ -280,35 +280,49 @@ so a host takedown or a ban actually revokes access to the bytes.
|
||||
degrade one subsystem — Postgres stops being able to write and the whole event goes
|
||||
down.
|
||||
|
||||
Uploads are self-limiting. `per_user_limit = free_disk × quota_tolerance ÷
|
||||
active_uploaders` is recomputed against live free space on every upload, so guests
|
||||
converge on a fixed point at `tolerance / (1 + tolerance)` of the free space you
|
||||
started with — **43%** at the default 0.75. On an 80 GB box with ~70 GB free after
|
||||
the OS and images, media settles at ~30 GB and stops.
|
||||
**`Gallery.zip` and `Memories.zip` are each roughly a second copy of every original.**
|
||||
Both write their media `Compression::Stored`, and `Memories.zip` streams the untouched
|
||||
original for every video and for every image at or under 5 MB. So a release wants room
|
||||
for **two more copies of the gallery** on top of the gallery itself — which is what
|
||||
`required_free_bytes` encodes as `media × 1.1 × 2`.
|
||||
|
||||
**The keepsake is what the 80 GB baseline does not cover.** `Gallery.zip` and
|
||||
`Memories.zip` are built concurrently and each is roughly a second copy of every
|
||||
original: both write their media `Compression::Stored`, and `Memories.zip` streams the
|
||||
untouched original for every video and for every image at or under 5 MB. So a release
|
||||
wants room for **two more copies of the gallery** on top of the gallery itself.
|
||||
The per-user quota does **not** bound this. It is a fairness mechanism that divides
|
||||
free space between guests, and since it carries a floor (`MIN_QUOTA_LIMIT_BYTES`, so a
|
||||
guest's allowance stops shrinking as the party fills up) the aggregate ceiling it used
|
||||
to imply is gone. What bounds the disk is the **global gate in the upload handler**,
|
||||
which refuses any upload that would leave too little room to build the keepsake:
|
||||
|
||||
| Stage | Used | Free (80 GB box) |
|
||||
|---|---|---|
|
||||
| Fresh box (OS + images) | ~10 GB | ~70 GB |
|
||||
| Guests reach the quota fixed point | ~40 GB | ~40 GB |
|
||||
| Host releases → both archives | ~100 GB | **ENOSPC** |
|
||||
```
|
||||
free_after_upload < media_after × 1.1 × 2 + DISK_RESERVE_BYTES → refused
|
||||
```
|
||||
|
||||
Two ways to size for it:
|
||||
Solving that for the gallery size gives the real ceiling. On the **40 GB box this runs
|
||||
on**, with ~5 GB for the OS, Docker images (the runbook pre-pulls the rollback tag too)
|
||||
and Postgres:
|
||||
|
||||
| Volume | Usable after baseline | Media ceiling | Free at release |
|
||||
|---|---|---|---|
|
||||
| 40 GB | ~35 GB | **~8 GB** | ~27 GB → both archives fit |
|
||||
| 80 GB | ~70 GB | ~19 GB | ~51 GB → both archives fit |
|
||||
|
||||
**Uploads therefore stop at roughly 8 GB of media on a 40 GB box, not when the disk is
|
||||
full.** That is deliberate. 1000 photos at ~3.5 MB is ~3.5 GB and fits comfortably;
|
||||
video is what consumes the budget, so lower `max_video_size_mb` (seeded at 500) if you
|
||||
expect a lot of it. Refusing the 1001st upload is a far better outcome than accepting it
|
||||
and discovering at 01:00 that the archive can never be built.
|
||||
|
||||
Two ways to buy headroom:
|
||||
|
||||
- **Provision ~3× your expected media** on one volume (media + two archives), or
|
||||
- **give `exports_data` its own volume** so a full export cannot reach Postgres, and
|
||||
size that one at ~2× expected media.
|
||||
|
||||
This is no longer silent. The export refuses up front with the two numbers rather than
|
||||
hitting ENOSPC halfway through a multi-GB write, a rebuild reclaims the superseded
|
||||
generation before it starts (so peak is one generation, not two), and the host
|
||||
dashboard warns as soon as the keepsake would not fit — which is the only point at
|
||||
which anyone can still do something about it.
|
||||
None of this is silent. The upload gate refuses with a German message naming the cause,
|
||||
the export preflight refuses up front with both numbers rather than hitting ENOSPC
|
||||
halfway through a multi-GB write, a rebuild only reclaims the superseded generation
|
||||
**after** the new one lands (so a failed rebuild can never leave you with no archive at
|
||||
all), and the host dashboard warns as soon as the keepsake would not fit — which is the
|
||||
only point at which anyone can still do something about it.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user