fix(backend): three ways the end of the night could go wrong

**1. Releasing the gallery could arm the keepsake with no worker.**
`release_gallery` ran `tx.commit()` -> SSE `event-closed` -> `audit::record().await`
-> `spawn_export_jobs`. The audit write is two pool round-trips, each able to wait
the full 5s acquire timeout, and it runs in the same instant `event-closed` fans
out to ~100 phones whose upload queues all hit the API at once. Axum drops the
handler future when the client disconnects — the host taps "Freigeben" and
pockets the phone. The release has COMMITTED: event closed, uploads locked, epoch
bumped, both `export_job` rows pending, and no worker. `/export/*` 404s, the page
sits on "Wird vorbereitet…", `recover_exports` only runs at boot, and a second
release is refused. Every other regen call site spawns first; `me.rs` says so in
a comment. This was the sole violator, and the only path that arms the FIRST
build of the keepsake. Spawn moved immediately after the commit.

**2. The event could be left with no operator.**
`remaining_operators` was an unlocked pool COUNT followed by a separate UPDATE,
so `ban_user` and `set_role` raced each other and `DELETE /me`: an admin demotes
host B while host A deletes themselves, each check sees the other still present,
both commit, and nobody can moderate, release the gallery, or appoint anyone —
appointing requires being an operator. The count now runs inside the writing
transaction behind the same advisory lock `delete_account` uses, via one shared
helper so the key cannot drift between copies.

The lock is taken FIRST in all three, and the order is load-bearing:
`delete_account` previously took it last, after row locks on `upload` and
`event`, while the two new call sites take it before locking those same rows —
an ABBA that Postgres would resolve by killing one transaction with a 500. The
ordering rule is documented on the helper.

**3. The keepsake could become unbuildable the moment uploads stopped.**
The upload gate and the export preflight computed the IDENTICAL threshold
(`required_free_bytes(media, 2) + DISK_RESERVE_BYTES`), leaving zero margin
between them. Once the gate refused its first upload the preflight was already at
its own limit, so anything written afterwards decided the keepsake's fate: WAL up
to `max_wal_size`, 30 MB x 4 of container logs, and the compression backlog
draining at exactly that hour. The release commits before the workers bail, so
the failure lands at 01:00 with no second release possible. The gate now demands
`UPLOAD_GATE_HEADROOM_BYTES` more than the preflight, costing ~0.5 GB of media
ceiling — the trade README already argues for. The dashboard banner mirrors the
new threshold so its lead is unchanged, and a new test pins gate-before-preflight
at six gallery sizes.

Also: the global disk gate fails OPEN when the mount cannot be read, which is
deliberate, but did it SILENTLY — no log line at all, while the export preflight
warns on the identical condition. Inside a container `/` is an overlay rather
than a `/dev` device, so this is reachable, and when it happens the only global
disk bound is gone and the box fills until Postgres cannot write WAL.

README's sizing table was also arithmetically self-contradictory (it showed
~27 GB free against a ~27.6 GB requirement); recomputed for the new gate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
fabi
2026-08-13 19:45:24 +02:00
parent 06e0bea0e9
commit 20c15c3500
4 changed files with 265 additions and 94 deletions

View File

@@ -311,19 +311,29 @@ to imply is gone. What bounds the disk is the **global gate in the upload handle
which refuses any upload that would leave too little room to build the keepsake:
```
free_after_upload < media_after × 1.1 × 2 + DISK_RESERVE_BYTES → refused
free_after_upload < media_after × 1.1 × 2 + DISK_RESERVE_BYTES
+ UPLOAD_GATE_HEADROOM_BYTES → refused
```
Solving that for the gallery size gives the real ceiling. On the **40 GB box this runs
on**, with ~5 GB for the OS, Docker images (the runbook pre-pulls the rollback tag too)
and Postgres:
That last term is what separates this gate from the export preflight, which bails at
`media × 1.1 × 2 + DISK_RESERVE_BYTES` — the same expression **minus** the headroom. The
two used to be identical, which meant the preflight was already sitting on its limit at
the exact moment uploads stopped: every byte written between the last refused upload and
the host tapping *Galerie freigeben* (Postgres WAL, container logs, the compression
backlog draining at precisely that hour) pushed it under, and the release commits before
the workers fail. The headroom buys 1.5 GB of slack so that cannot happen.
| Volume | Usable after baseline | Media ceiling | Free at release |
|---|---|---|---|
| 40 GB | ~35 GB | **~8 GB** | ~27 GB → both archives fit |
| 80 GB | ~70 GB | ~19 GB | ~51 GB → both archives fit |
Solving the gate for the gallery size gives the real ceiling — the gate's equilibrium is
`3.2 × media`, so each GB of reserve or headroom costs ~0.31 GB of gallery. On the
**40 GB box this runs on**, with ~5 GB for the OS, Docker images (the runbook pre-pulls
the rollback tag too) and Postgres:
**Uploads therefore stop at roughly 8 GB of media on a 40 GB box, not when the disk is
| Volume | Usable after baseline | Media ceiling | Free at release | Preflight needs |
|---|---|---|---|---|
| 40 GB | ~35 GB | **~7.3 GB** | ~27.7 GB | ~26.2 GB → fits, 1.5 GB spare |
| 80 GB | ~70 GB | ~18.3 GB | ~51.7 GB | ~50.2 GB → fits, 1.5 GB spare |
**Uploads therefore stop at roughly 7 GB of media on a 40 GB box, not when the disk is
full.** That is deliberate. 1000 photos at ~3.5 MB is ~3.5 GB and fits comfortably;
video is what consumes the budget, so lower `max_video_size_mb` (seeded at 500) if you
expect a lot of it. Refusing the 1001st upload is a far better outcome than accepting it