diff --git a/README.md b/README.md index c7ac6e8..b065779 100644 --- a/README.md +++ b/README.md @@ -311,19 +311,29 @@ to imply is gone. What bounds the disk is the **global gate in the upload handle which refuses any upload that would leave too little room to build the keepsake: ``` -free_after_upload < media_after × 1.1 × 2 + DISK_RESERVE_BYTES → refused +free_after_upload < media_after × 1.1 × 2 + DISK_RESERVE_BYTES + + UPLOAD_GATE_HEADROOM_BYTES → refused ``` -Solving that for the gallery size gives the real ceiling. On the **40 GB box this runs -on**, with ~5 GB for the OS, Docker images (the runbook pre-pulls the rollback tag too) -and Postgres: +That last term is what separates this gate from the export preflight, which bails at +`media × 1.1 × 2 + DISK_RESERVE_BYTES` — the same expression **minus** the headroom. The +two used to be identical, which meant the preflight was already sitting on its limit at +the exact moment uploads stopped: every byte written between the last refused upload and +the host tapping *Galerie freigeben* (Postgres WAL, container logs, the compression +backlog draining at precisely that hour) pushed it under, and the release commits before +the workers fail. The headroom buys 1.5 GB of slack so that cannot happen. -| Volume | Usable after baseline | Media ceiling | Free at release | -|---|---|---|---| -| 40 GB | ~35 GB | **~8 GB** | ~27 GB → both archives fit | -| 80 GB | ~70 GB | ~19 GB | ~51 GB → both archives fit | +Solving the gate for the gallery size gives the real ceiling — the gate's equilibrium is +`3.2 × media`, so each GB of reserve or headroom costs ~0.31 GB of gallery. On the +**40 GB box this runs on**, with ~5 GB for the OS, Docker images (the runbook pre-pulls +the rollback tag too) and Postgres: -**Uploads therefore stop at roughly 8 GB of media on a 40 GB box, not when the disk is +| Volume | Usable after baseline | Media ceiling | Free at release | Preflight needs | +|---|---|---|---|---| +| 40 GB | ~35 GB | **~7.3 GB** | ~27.7 GB | ~26.2 GB → fits, 1.5 GB spare | +| 80 GB | ~70 GB | ~18.3 GB | ~51.7 GB | ~50.2 GB → fits, 1.5 GB spare | + +**Uploads therefore stop at roughly 7 GB of media on a 40 GB box, not when the disk is full.** That is deliberate. 1000 photos at ~3.5 MB is ~3.5 GB and fits comfortably; video is what consumes the budget, so lower `max_video_size_mb` (seeded at 500) if you expect a lot of it. Refusing the 1001st upload is a far better outcome than accepting it diff --git a/backend/src/handlers/host.rs b/backend/src/handlers/host.rs index 0dcba32..79ee909 100644 --- a/backend/src/handlers/host.rs +++ b/backend/src/handlers/host.rs @@ -57,7 +57,8 @@ pub struct EventStatus { /// into a decision someone can still make. /// /// IT MUST FIRE BEFORE THE UPLOAD GATE CLOSES, and that is why the reserve and the margin are -/// here. The gate in `handlers::upload` refuses at `free < keepsake_required + DISK_RESERVE_BYTES`; +/// here. The gate in `handlers::upload` refuses at +/// `free < keepsake_required + DISK_RESERVE_BYTES + UPLOAD_GATE_HEADROOM_BYTES`; /// warning at `free < keepsake_required` alone meant the two differed by the whole reserve, so /// the wall was always hit FIRST. Every guest would be blocked from uploading while this /// dashboard showed a comfortable disk and no banner at all — on the shipped 40 GB box, uploads @@ -66,8 +67,14 @@ pub struct EventStatus { /// The 25% margin makes it a warning rather than an obituary: the host sees it while there is /// still room to act (delete a few large videos, which refunds immediately and reopens the gate). fn disk_is_low(free: u64, keepsake_required: u64) -> bool { - let gate_closes_at = - keepsake_required.saturating_add(crate::handlers::upload::DISK_RESERVE_BYTES as u64); + // Mirrors the gate EXACTLY, headroom included. The gate now demands + // `UPLOAD_GATE_HEADROOM_BYTES` more than the export preflight does, so that ordinary + // end-of-night writes cannot flip the preflight after uploads have already stopped. Leaving + // that term out here would shrink the warning's lead by 1.5 GB — and the whole point of this + // function is that the banner must appear while the host can still act. + let gate_closes_at = keepsake_required + .saturating_add(crate::handlers::upload::DISK_RESERVE_BYTES as u64) + .saturating_add(crate::handlers::upload::UPLOAD_GATE_HEADROOM_BYTES as u64); let warn_at = gate_closes_at.saturating_add(gate_closes_at / 4); // No separate absolute-floor clause. There used to be `free < LOW_DISK_FLOOR_BYTES ||` here, // and it was unreachable: `gate_closes_at` is at least DISK_RESERVE_BYTES, so `warn_at` is at @@ -80,8 +87,15 @@ fn disk_is_low(free: u64, keepsake_required: u64) -> bool { /// Count non-banned hosts/admins in the event OTHER than `excluding` — the operators /// who would remain if `excluding` were demoted or banned. Used to enforce the "an event /// always keeps at least one operator" floor. +/// +/// Takes a CONNECTION, not the pool, and every caller passes the same transaction it is about to +/// write in — after taking [`lock_operator_floor`]. Read on the pool beforehand, this count was a +/// snapshot that any concurrent operator-removing action could invalidate before the UPDATE landed: +/// an admin demoting host B while host A calls `DELETE /me` saw two independent checks each observe +/// the other still present, both commit, and the event end up with zero operators — which is not +/// recoverable from inside the app, since appointing an operator requires being one. async fn remaining_operators( - state: &AppState, + conn: &mut sqlx::PgConnection, event_id: Uuid, excluding: Uuid, ) -> Result { @@ -92,11 +106,36 @@ async fn remaining_operators( ) .bind(event_id) .bind(excluding) - .fetch_one(&state.pool) + .fetch_one(conn) .await?; Ok(count) } +/// Serialise every action that can remove an operator from an event. +/// +/// The same key `me::delete_account` takes — namespace 4242, `hashtext(event_id)` — and it MUST +/// stay identical, or the two families of caller lock against nothing. An advisory lock is used +/// rather than a row lock because it is a separate lock space and so cannot join the +/// `event`/`user` row-lock graph that moderation traffic already traverses in both directions; +/// it is released automatically when the transaction ends. +/// +/// **Call this FIRST in the transaction, before taking any row lock.** Being a separate lock space +/// means it cannot form a cycle *with itself*, not that ordering is free: all three callers go on +/// to lock `user` and `event` rows, so a caller that took those rows first and reached for this +/// lock afterwards would deadlock against one that did it the other way round. Postgres would +/// break the tie by killing one transaction with a 500. Every caller acquires it first; keep it +/// that way. +pub(crate) async fn lock_operator_floor( + conn: &mut sqlx::PgConnection, + event_id: Uuid, +) -> Result<(), AppError> { + sqlx::query("SELECT pg_advisory_xact_lock(4242, hashtext($1::text))") + .bind(event_id) + .execute(conn) + .await?; + Ok(()) +} + #[derive(Deserialize)] pub struct SetRoleRequest { pub role: String, @@ -193,14 +232,6 @@ pub async fn ban_user( )); } - // Floor: never leave the event with zero operators. Banning removes the target from - // the active-operator pool, so refuse if they're the last non-banned host/admin. - if target.0 == "host" && remaining_operators(&state, auth.event_id, user_id).await? == 0 { - return Err(AppError::BadRequest( - "Der letzte Host kann nicht gesperrt werden.".into(), - )); - } - // Ban ALWAYS hides: a banned user's content is "gone" everywhere. The visibility // views/queries now also filter on `is_banned` (defense in depth), and we set // `uploads_hidden` so the existing `user-hidden` live-eviction path fires too. The old @@ -215,6 +246,21 @@ pub async fn ban_user( // // The ban and the keepsake invalidation are ONE transaction — see `host_delete_upload`. let mut tx = state.pool.begin().await?; + + // Floor: never leave the event with zero operators. Banning removes the target from the + // active-operator pool, so refuse if they're the last non-banned host/admin. + // + // INSIDE the transaction and behind the operator lock — see `remaining_operators`. Checked on + // the pool beforehand, this raced `set_role` and `DELETE /me` into an event with no operator. + if target.0 == "host" { + lock_operator_floor(&mut tx, auth.event_id).await?; + if remaining_operators(&mut tx, auth.event_id, user_id).await? == 0 { + return Err(AppError::BadRequest( + "Der letzte Host kann nicht gesperrt werden.".into(), + )); + } + } + sqlx::query( "UPDATE \"user\" SET is_banned = TRUE, uploads_hidden = TRUE, uploads_hidden_at = NOW() @@ -461,21 +507,26 @@ pub async fn set_role( // Floor: demoting the last non-banned host/admin to guest would leave the event with // no operator. Refuse. - if new_role == "guest" - && target.0 == "host" - && remaining_operators(&state, auth.event_id, user_id).await? == 0 - { - return Err(AppError::BadRequest( - "Der letzte Host kann nicht zum Gast gemacht werden.".into(), - )); + // + // The check and the UPDATE are ONE transaction, behind the operator lock — see + // `remaining_operators`. Split apart on the pool, this raced `ban_user` and `DELETE /me`. + let mut tx = state.pool.begin().await?; + if new_role == "guest" && target.0 == "host" { + lock_operator_floor(&mut tx, auth.event_id).await?; + if remaining_operators(&mut tx, auth.event_id, user_id).await? == 0 { + return Err(AppError::BadRequest( + "Der letzte Host kann nicht zum Gast gemacht werden.".into(), + )); + } } sqlx::query("UPDATE \"user\" SET role = $2::user_role WHERE id = $1 AND event_id = $3") .bind(user_id) .bind(new_role) .bind(auth.event_id) - .execute(&state.pool) + .execute(&mut *tx) .await?; + tx.commit().await?; tracing::info!( actor_user_id = %auth.user_id, target_user_id = %user_id, @@ -938,9 +989,39 @@ pub async fn release_gallery( // discovering it via a rejected upload. let _ = state.sse_tx.send(SseEvent::new("event-closed", "{}")); + // Detached — survives this handler being cancelled. + // + // SPAWNED IMMEDIATELY AFTER THE COMMIT, BEFORE ANY OTHER `.await`. Every `invalidate_and_arm` + // call site does this; `me::delete_account` carries the same note. The audit write below used + // to sit here, and it is two pool round-trips that can each wait up to the 5 s acquire timeout + // — right at the moment `event-closed` has just fanned out to ~100 phones whose queues all hit + // the API at once, so the pool is as contended as it ever gets. Drop the handler future during + // that suspension (the host's phone sleeps, the tab closes, Caddy times the request out) and + // the task never spawns: the event is released, uploads are locked, both `export_job` rows sit + // `pending` at the live epoch, and no worker exists. `/export/*` 404s, the page sits on "Wird + // vorbereitet…", `recover_exports` only runs at boot, and `release_gallery` refuses a retry + // because the gallery is already released. + // + // This is the one path that arms the FIRST build of the keepsake, so it is the worst possible + // place to reintroduce that window. + crate::services::export::spawn_export_jobs( + event_id, + event_name, + epoch, + state.config.comments_enabled, + std::time::Duration::ZERO, + state.pool.clone(), + state.config.media_path.clone(), + state.config.export_path.clone(), + state.sse_tx.clone(), + ); + // Was logged NOWHERE at all before this — not even a tracing line. A host reading // the record the morning after had no way to see when uploads were locked or the // gallery released, which are the two actions that change what every guest can do. + // + // Last, deliberately: it is best-effort by design (it swallows its own errors), so nothing + // downstream may depend on it having completed. crate::services::audit::record( &state.pool, auth.event_id, @@ -954,26 +1035,13 @@ pub async fn release_gallery( ) .await; - // Detached — survives this handler being cancelled. - crate::services::export::spawn_export_jobs( - event_id, - event_name, - epoch, - state.config.comments_enabled, - std::time::Duration::ZERO, - state.pool.clone(), - state.config.media_path.clone(), - state.config.export_path.clone(), - state.sse_tx.clone(), - ); - Ok(StatusCode::NO_CONTENT) } #[cfg(test)] mod tests { use super::disk_is_low; - use crate::handlers::upload::DISK_RESERVE_BYTES; + use crate::handlers::upload::{DISK_RESERVE_BYTES, UPLOAD_GATE_HEADROOM_BYTES}; use crate::services::export::required_free_bytes; const GB: u64 = 1_000_000_000; @@ -1016,7 +1084,8 @@ mod tests { // require it to be strictly above the level at which the gate closes, by a usable amount. for media_gb in [0u64, 1, 4, 8, 16, 32] { let required = required_free_bytes(media_gb * GB, 2); - let gate_closes_at = required + DISK_RESERVE_BYTES as u64; + let gate_closes_at = + required + DISK_RESERVE_BYTES as u64 + UPLOAD_GATE_HEADROOM_BYTES as u64; // Just above the gate: guests can still upload, and the host must already be warned. assert!( @@ -1037,6 +1106,43 @@ mod tests { } } + /// The invariant the headroom exists for: uploads must stop while the keepsake can STILL be + /// built, with room to spare — not at the exact instant the preflight reaches its own limit. + /// + /// Both thresholds used to be `required_free_bytes(media, 2) + DISK_RESERVE_BYTES`, identically. + /// So the moment the gate refused its first upload, the export preflight was already sitting on + /// its limit, and every byte written afterwards (WAL, container logs, the compression backlog + /// draining at exactly that hour) pushed it under. The release would then COMMIT — event closed, + /// uploads locked, epoch bumped, `event-closed` fanned out to every phone — and only then would + /// both workers bail, with no second release possible. + #[test] + fn the_upload_gate_closes_before_the_export_preflight_would_refuse() { + for media_gb in [0u64, 1, 4, 8, 16, 32] { + let required = required_free_bytes(media_gb * GB, 2); + + // `services::export::preflight` bails below this. + let preflight_refuses_below = required + DISK_RESERVE_BYTES as u64; + // `handlers::upload` refuses below this. + let gate_refuses_below = preflight_refuses_below + UPLOAD_GATE_HEADROOM_BYTES as u64; + + assert!( + gate_refuses_below > preflight_refuses_below, + "at media={media_gb}GB the gate and the preflight share a threshold, so the \ + keepsake's fate rests on whatever is written after uploads stop" + ); + + // At the instant the last upload is refused, the preflight must still pass with the + // whole headroom to spare — that is the slack the night's remaining writes consume. + let free_when_gate_closes = gate_refuses_below; + assert!( + free_when_gate_closes + >= preflight_refuses_below + UPLOAD_GATE_HEADROOM_BYTES as u64, + "at media={media_gb}GB there is no slack between the gate closing and the \ + preflight failing" + ); + } + } + #[test] fn plenty_of_space_is_still_low_when_the_keepsake_would_not_fit() { // THE case the fixed threshold misses, and the one that matters: 30 GB free is nowhere near @@ -1051,7 +1157,7 @@ mod tests { // size — see `disk_is_low`. Warning at the bare size fired only after the gate had // already blocked every guest. let required = 20 * GB; - let gate = required + DISK_RESERVE_BYTES as u64; + let gate = required + DISK_RESERVE_BYTES as u64 + UPLOAD_GATE_HEADROOM_BYTES as u64; let warn_at = gate + gate / 4; assert!(!disk_is_low(warn_at, required), "exactly enough is enough"); assert!(disk_is_low(warn_at - 1, required)); @@ -1060,8 +1166,9 @@ mod tests { #[test] fn an_empty_gallery_still_reserves_room_for_postgres() { // With no gallery the keepsake term is 0, so the warn threshold collapses to - // 1.25 x DISK_RESERVE_BYTES (12.5 GB), which dominates the 10 GB absolute floor. - assert!(!disk_is_low(13 * GB, 0)); + // 1.25 x (DISK_RESERVE_BYTES + UPLOAD_GATE_HEADROOM_BYTES) = 1.25 x 11.5 GB = 14.375 GB, + // which dominates the 10 GB absolute floor. + assert!(!disk_is_low(15 * GB, 0)); assert!(disk_is_low(9 * GB, 0)); } } diff --git a/backend/src/handlers/me.rs b/backend/src/handlers/me.rs index ac79db3..00d75a1 100644 --- a/backend/src/handlers/me.rs +++ b/backend/src/handlers/me.rs @@ -178,6 +178,54 @@ pub async fn delete_account( .await?; let mut tx = state.pool.begin().await?; + + // The last-host guard, AUTHORITATIVELY — inside the transaction, holding a lock. + // + // The pre-check further up runs on the pool before this transaction opens, so two hosts + // deleting themselves at the same moment each saw the other and both proceeded, leaving the + // event with NO operator: nobody to moderate, nobody to release the gallery, and no way to + // appoint anyone because appointing requires a host. Not recoverable from inside the app. + // + // Serialised with a transaction-scoped ADVISORY lock, not a row lock. `FOR UPDATE` on the + // other operators\' rows looks like the obvious answer and is the wrong one: each deleter would + // lock the OTHER\'s row and then try to delete its own, so the two block on each other and + // Postgres resolves it by killing one with a deadlock error — the invariant holds, but the + // loser gets a 500 instead of the sentence below. Locking the `event` row instead would + // serialise cleanly, but it inverts the lock order every moderation path uses (upload/user + // rows first, event last). An advisory lock is a separate lock space, so it cannot join the + // row-lock graph at all, and it is released automatically when this transaction ends. + // + // FIRST STATEMENT IN THE TRANSACTION, before any row lock — the ORDER matters as much as the + // lock. `ban_user` and `set_role` take this same lock and then go on to lock `user` and + // `event` rows. If this path grabbed those rows first and reached for the advisory lock + // afterwards, the two would deadlock, each holding what the other needs, and Postgres would + // kill one with a 500: the invariant would survive, but a host deleting their account would + // get an error page instead of the sentence below. + // + // Taking it up front also means the refusal path does no work at all before answering. + if matches!(user.role, UserRole::Host | UserRole::Admin) { + // Shared with `host::ban_user` and `host::set_role` — the same key, by construction rather + // than by two copies agreeing. All three remove an operator, so all three must serialise + // against each other or the floor is enforceable only against its own kind of caller. + crate::handlers::host::lock_operator_floor(&mut tx, auth.event_id).await?; + let others: Vec = sqlx::query_scalar( + "SELECT id FROM \"user\" + WHERE event_id = $1 AND id != $2 + AND role IN ('host', 'admin') AND is_banned = FALSE", + ) + .bind(auth.event_id) + .bind(auth.user_id) + .fetch_all(&mut *tx) + .await?; + if others.is_empty() { + return Err(AppError::BadRequest( + "Du bist der letzte Gastgeber. Ernenne zuerst einen anderen Gastgeber, bevor du \ + dein Konto löschst." + .into(), + )); + } + } + // Comments the guest wrote on OTHER people's photos. Hard delete, not `deleted_at`: this is // erasure, and a soft delete leaves the body in the table and in the keepsake's data.json. sqlx::query("DELETE FROM comment WHERE user_id = $1") @@ -199,48 +247,6 @@ pub async fn delete_account( crate::services::export::Affects::Both, ) .await?; - // The last-host guard again, now AUTHORITATIVELY — inside the transaction, holding a lock. - // - // The check above runs on the pool before this transaction opens, so two hosts deleting - // themselves at the same moment each saw the other and both proceeded, leaving the event with - // NO operator: nobody to moderate, nobody to release the gallery, and no way to appoint anyone - // because appointing requires a host. That is not recoverable from inside the app. - // - // Serialised with a transaction-scoped ADVISORY lock, not a row lock. - // - // `FOR UPDATE` on the other operators' rows looks like the obvious answer and is the wrong one: - // each deleter would lock the OTHER's row and then try to delete its own, so the two block on - // each other and Postgres resolves it by killing one with a deadlock error — the invariant - // holds, but the loser gets a 500 instead of the sentence below. Locking the `event` row - // instead would serialise cleanly, but it inverts the lock order every moderation path uses - // (upload/user rows first, event last), which is an ABBA waiting to happen. - // - // An advisory lock has neither problem: it is a separate lock space, so it cannot interact with - // the row-lock graph at all, and it is released automatically when this transaction ends. - // 4242 is an arbitrary namespace to keep this key from colliding with any future advisory use. - if matches!(user.role, UserRole::Host | UserRole::Admin) { - sqlx::query("SELECT pg_advisory_xact_lock(4242, hashtext($1::text))") - .bind(auth.event_id) - .execute(&mut *tx) - .await?; - let others: Vec = sqlx::query_scalar( - "SELECT id FROM \"user\" - WHERE event_id = $1 AND id != $2 - AND role IN ('host', 'admin') AND is_banned = FALSE", - ) - .bind(auth.event_id) - .bind(auth.user_id) - .fetch_all(&mut *tx) - .await?; - if others.is_empty() { - return Err(AppError::BadRequest( - "Du bist der letzte Gastgeber. Ernenne zuerst einen anderen Gastgeber, bevor du \ - dein Konto löschst." - .into(), - )); - } - } - // And the account. `session`, `like` and `pin_reset_request` cascade from here. sqlx::query("DELETE FROM \"user\" WHERE id = $1") .bind(auth.user_id) diff --git a/backend/src/handlers/upload.rs b/backend/src/handlers/upload.rs index da77274..a9798b5 100644 --- a/backend/src/handlers/upload.rs +++ b/backend/src/handlers/upload.rs @@ -575,7 +575,12 @@ pub async fn upload( .saturating_add(size); let keepsake_needs = crate::services::export::required_free_bytes(media_after.max(0) as u64, 2) as i64; - let required = keepsake_needs.saturating_add(DISK_RESERVE_BYTES); + // Strictly more than the export preflight requires — see `UPLOAD_GATE_HEADROOM_BYTES`. + // Matching it exactly meant the preflight was already at its limit the moment uploads + // stopped, so the night's remaining writes decided whether the keepsake could be built. + let required = keepsake_needs + .saturating_add(DISK_RESERVE_BYTES) + .saturating_add(UPLOAD_GATE_HEADROOM_BYTES); if free < required { tracing::error!( free_bytes = free, @@ -583,6 +588,7 @@ pub async fn upload( media_after, keepsake_needs, reserve = DISK_RESERVE_BYTES, + headroom = UPLOAD_GATE_HEADROOM_BYTES, "refusing upload: it would leave too little room to build the keepsake" ); return Err(AppError::QuotaExceeded( @@ -592,10 +598,27 @@ pub async fn upload( .into(), )); } + } else { + // Failing OPEN when the disk can't be read is deliberate and matches the per-user quota + // below: refusing every upload because a `statfs` failed would be a worse outage than the + // one being guarded against. + // + // But it must not be SILENT. `snapshot` returns `None` when `select_disk` finds neither a + // mount that prefixes the media path nor a `/` entry — and inside a container `/` is an + // overlay rather than a `/dev` device, so this is a real possibility rather than a + // theoretical one. When it happens, the ONLY global disk bound in the app is gone, the + // per-user quota fails open through the same `None`, and the box fills to 100% — at which + // point Postgres cannot write WAL and the whole event stops, with nothing having warned + // anyone. The export preflight already warns on the identical condition; this is the + // louder of the two paths and had no log line at all. + // + // Rate-limited by the disk cache's own TTL, so this cannot spam the log per upload. + tracing::warn!( + media_path = %state.config.media_path.display(), + "disk usage unreadable — the global free-space gate is INACTIVE and uploads are \ + proceeding unbounded; check the admin stats page for a plausible free-space figure" + ); } - // Failing OPEN when the disk can't be read is deliberate and matches the per-user quota - // below: refusing every upload because a `statfs` failed would be a worse outage than the - // one being guarded against. // When quota is enforced, this holds the byte ceiling so the increment UPDATE below can // enforce it atomically (`WHERE total + size <= limit`). Without that guard, two @@ -1240,6 +1263,31 @@ const MIN_QUOTA_LIMIT_BYTES: i64 = 500 * 1024 * 1024; /// the shared filesystem long after new uploads have been refused. pub const DISK_RESERVE_BYTES: i64 = 10_000_000_000; +/// Extra free space the UPLOAD gate demands on top of what the export preflight demands. +/// +/// Both gates were computing the identical threshold — `required_free_bytes(media, 2) + +/// DISK_RESERVE_BYTES` — which left exactly zero margin between them. The moment the gate refused +/// its first upload, the preflight was already sitting on its own limit, so anything written +/// between that refusal and the host tapping "Galerie freigeben" pushed the preflight under: +/// +/// * Postgres WAL, up to `max_wal_size` (1 GB by default) before a checkpoint reclaims it +/// * container logs, capped at 30 MB x 4 services by `docker-compose.yml` +/// * the compression backlog still draining — ~0.9 MB of derivatives per queued photo, and the +/// backlog is longest exactly at the end of the night +/// +/// The failure that produces is the worst one in the app: the release COMMITS (event closed, +/// uploads locked, epoch bumped, `event-closed` fanned out to every phone) and only then do both +/// workers bail, at 01:00, with no second release possible and `rebuild_export` needing the same +/// space it just failed to find. Meanwhile ~10 GB of reserve sits unused — the preflight refused +/// on a threshold, not for want of room. +/// +/// Giving the upload gate this much more to satisfy means it closes strictly earlier, so ordinary +/// end-of-night writes cannot flip the preflight. The cost is roughly 0.5 GB off the media ceiling +/// on a 40 GB box (the gate's equilibrium is `3.2 x media`, so headroom divides by 3.2), which is +/// the trade `README.md` already argues for: refusing the 1001st upload beats discovering at 01:00 +/// that the archive can never be built. +pub const UPLOAD_GATE_HEADROOM_BYTES: i64 = 1_500_000_000; + /// Pure per-user quota formula: `max(floor((free_disk * tolerance) / divisor), MIN)`. /// /// `divisor` is the LARGER of the observed uploader count and the operator's