fix(backend): three ways the end of the night could go wrong
**1. Releasing the gallery could arm the keepsake with no worker.** `release_gallery` ran `tx.commit()` -> SSE `event-closed` -> `audit::record().await` -> `spawn_export_jobs`. The audit write is two pool round-trips, each able to wait the full 5s acquire timeout, and it runs in the same instant `event-closed` fans out to ~100 phones whose upload queues all hit the API at once. Axum drops the handler future when the client disconnects — the host taps "Freigeben" and pockets the phone. The release has COMMITTED: event closed, uploads locked, epoch bumped, both `export_job` rows pending, and no worker. `/export/*` 404s, the page sits on "Wird vorbereitet…", `recover_exports` only runs at boot, and a second release is refused. Every other regen call site spawns first; `me.rs` says so in a comment. This was the sole violator, and the only path that arms the FIRST build of the keepsake. Spawn moved immediately after the commit. **2. The event could be left with no operator.** `remaining_operators` was an unlocked pool COUNT followed by a separate UPDATE, so `ban_user` and `set_role` raced each other and `DELETE /me`: an admin demotes host B while host A deletes themselves, each check sees the other still present, both commit, and nobody can moderate, release the gallery, or appoint anyone — appointing requires being an operator. The count now runs inside the writing transaction behind the same advisory lock `delete_account` uses, via one shared helper so the key cannot drift between copies. The lock is taken FIRST in all three, and the order is load-bearing: `delete_account` previously took it last, after row locks on `upload` and `event`, while the two new call sites take it before locking those same rows — an ABBA that Postgres would resolve by killing one transaction with a 500. The ordering rule is documented on the helper. **3. The keepsake could become unbuildable the moment uploads stopped.** The upload gate and the export preflight computed the IDENTICAL threshold (`required_free_bytes(media, 2) + DISK_RESERVE_BYTES`), leaving zero margin between them. Once the gate refused its first upload the preflight was already at its own limit, so anything written afterwards decided the keepsake's fate: WAL up to `max_wal_size`, 30 MB x 4 of container logs, and the compression backlog draining at exactly that hour. The release commits before the workers bail, so the failure lands at 01:00 with no second release possible. The gate now demands `UPLOAD_GATE_HEADROOM_BYTES` more than the preflight, costing ~0.5 GB of media ceiling — the trade README already argues for. The dashboard banner mirrors the new threshold so its lead is unchanged, and a new test pins gate-before-preflight at six gallery sizes. Also: the global disk gate fails OPEN when the mount cannot be read, which is deliberate, but did it SILENTLY — no log line at all, while the export preflight warns on the identical condition. Inside a container `/` is an overlay rather than a `/dev` device, so this is reachable, and when it happens the only global disk bound is gone and the box fills until Postgres cannot write WAL. README's sizing table was also arithmetically self-contradictory (it showed ~27 GB free against a ~27.6 GB requirement); recomputed for the new gate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
28
README.md
28
README.md
@@ -311,19 +311,29 @@ to imply is gone. What bounds the disk is the **global gate in the upload handle
|
||||
which refuses any upload that would leave too little room to build the keepsake:
|
||||
|
||||
```
|
||||
free_after_upload < media_after × 1.1 × 2 + DISK_RESERVE_BYTES → refused
|
||||
free_after_upload < media_after × 1.1 × 2 + DISK_RESERVE_BYTES
|
||||
+ UPLOAD_GATE_HEADROOM_BYTES → refused
|
||||
```
|
||||
|
||||
Solving that for the gallery size gives the real ceiling. On the **40 GB box this runs
|
||||
on**, with ~5 GB for the OS, Docker images (the runbook pre-pulls the rollback tag too)
|
||||
and Postgres:
|
||||
That last term is what separates this gate from the export preflight, which bails at
|
||||
`media × 1.1 × 2 + DISK_RESERVE_BYTES` — the same expression **minus** the headroom. The
|
||||
two used to be identical, which meant the preflight was already sitting on its limit at
|
||||
the exact moment uploads stopped: every byte written between the last refused upload and
|
||||
the host tapping *Galerie freigeben* (Postgres WAL, container logs, the compression
|
||||
backlog draining at precisely that hour) pushed it under, and the release commits before
|
||||
the workers fail. The headroom buys 1.5 GB of slack so that cannot happen.
|
||||
|
||||
| Volume | Usable after baseline | Media ceiling | Free at release |
|
||||
|---|---|---|---|
|
||||
| 40 GB | ~35 GB | **~8 GB** | ~27 GB → both archives fit |
|
||||
| 80 GB | ~70 GB | ~19 GB | ~51 GB → both archives fit |
|
||||
Solving the gate for the gallery size gives the real ceiling — the gate's equilibrium is
|
||||
`3.2 × media`, so each GB of reserve or headroom costs ~0.31 GB of gallery. On the
|
||||
**40 GB box this runs on**, with ~5 GB for the OS, Docker images (the runbook pre-pulls
|
||||
the rollback tag too) and Postgres:
|
||||
|
||||
**Uploads therefore stop at roughly 8 GB of media on a 40 GB box, not when the disk is
|
||||
| Volume | Usable after baseline | Media ceiling | Free at release | Preflight needs |
|
||||
|---|---|---|---|---|
|
||||
| 40 GB | ~35 GB | **~7.3 GB** | ~27.7 GB | ~26.2 GB → fits, 1.5 GB spare |
|
||||
| 80 GB | ~70 GB | ~18.3 GB | ~51.7 GB | ~50.2 GB → fits, 1.5 GB spare |
|
||||
|
||||
**Uploads therefore stop at roughly 7 GB of media on a 40 GB box, not when the disk is
|
||||
full.** That is deliberate. 1000 photos at ~3.5 MB is ~3.5 GB and fits comfortably;
|
||||
video is what consumes the budget, so lower `max_video_size_mb` (seeded at 500) if you
|
||||
expect a lot of it. Refusing the 1001st upload is a far better outcome than accepting it
|
||||
|
||||
Reference in New Issue
Block a user