fix: close what nine adversarial reviews found, most of it mine
Some checks failed
Checks / Backend — cargo test + clippy + fmt (push) Failing after 1m5s
Checks / Frontend — vitest + svelte-check (push) Failing after 5m55s
Checks / E2E — typecheck + lint (push) Failing after 49s
E2E / Playwright E2E (chromium + webkit) (push) Failing after 10m42s
E2E / Cross-UA smoke matrix (push) Failing after 7m57s
Audit / cargo audit (backend) (push) Failing after 11m12s
Audit / npm audit (frontend) (push) Successful in 44s

Nine focused reviews (export state machine, upload path, auth/abuse, client
queue, guest UI, database, deploy/ops, regression hunt, test honesty). Every
finding below was re-verified against the code before being acted on; several
plausible-sounding ones were checked and rejected.

## Data loss and denial of service

**One request could OOM-kill the app container.** `client_upload_id` was read
with `Field::text()` — axum builds its multipart reader with no SizeLimit, so
the only bound was the route's 576 MiB body limit, then decoded into a second
full String. `caption` and `hashtags` go through `read_text_field_bounded` for
exactly this reason; this field arrived later and missed it. Any guest, one
request, and every SSE stream drops and every in-flight temp file is stranded.

**Nothing bounded concurrent upload bodies.** The headroom gate can only refuse
to COMMIT — the body is already streamed to a temp file by the time it runs, and
neither axum, the tower stack nor Caddy limits how many stream at once. ~100
guests tapping "upload all" after the ceremony puts 10-20 GB of .tmp on a 40 GB
volume, invisible to the gate, eating the reserve that keeps Postgres able to
write WAL. New `UploadAdmission` budgets bytes (not requests, so one video and
two hundred photos coexist) via a permit that releases on drop, so every exit
path returns it.

**The export decode bypassed the memory permit the compression path takes.**
Same class of work — decode + resize every image in the gallery — in a bare
spawn_blocking. A release fired while the last photos were still compressing put
both in the same 1 GiB cgroup; the OOM kill marks the export failed and
`recover_exports` re-spawns it into the same conditions on the next boot. The
permit is now process-wide in `imaging`, because the constraint it expresses is
the container's memory, not one worker's.

**`MediaTotalCache` cached its own failure as 0.** For the whole TTL the gate
then saw an empty event and collapsed to the flat reserve — the behaviour the
two-halves design replaced — with no log line. And the trigger correlates with
the danger: with max_connections 10 the query fails exactly during a burst. Now
falls back to the last good reading and says so.

**V8's heap ceiling sat above the frontend container's entire budget** (measured:
259 MB inside a 256M limit), so GC could never intervene and the only
backpressure was SIGKILL under an arrival burst.

## Guest-visible

**The feed stopped being newest-first after the first reconcile.** It fetches
whole 100-item server pages while `uploads` grows in 20s, so everything in the
gap was absent from `present`, classified as new, and prepended — ~80 photos
from earlier in the evening above the newest ones. It also stalled infinite
scroll, since the cursor still pointed at item 20 and the observer only re-fires
on a change. The union is now sorted on the server's own (created_at, id) key,
which additionally places an SSE arrival correctly.

**A stale `loadMoreError` outlived every refresh and filter change**, leaving a
false error above a button that returns immediately on `!nextCursor`.

**A failed derivative toasted "Ein Upload konnte nicht verarbeitet werden."** for
a photo sitting right there on screen — the handler still assumed 1d9fb11's
pre-fix behaviour (row deleted, quota refunded, card evicted), none of which is
true any more. It was the last surviving route for the "your photo is gone"
signal that fix set out to remove.

## Enforcement that existed only in comments

`recover_name_rate_per_15min` is clamped at the point of use: the ordering
`3 x ceiling <= PIN_LOCK_THRESHOLD` is the whole control against one source
locking any guest whose name is on the feed, it was asserted in a comment, and
`patch_config` accepted 1..100_000. The test pinned the default constant rather
than the enforced bound; it now pins the bound.

## Tests that could not fail

- The gate test asserted only its own premise (`500MB x 100 > 35GB`) and never
  touched the gate. It now checks both controls against the same state and
  requires them to disagree in the right direction.
- `the_banner_always_fires_before_the_upload_gate_closes` reduced to
  `G < G + G/4` — true for any margin, including zero, so it could not detect
  the banner moving to exactly the gate. It now pins the gap.
- `disk_is_low`'s `free < LOW_DISK_FLOOR_BYTES` clause was unreachable (warn_at
  is always >= 12.5 GB against a 10 GB floor). Two tests were named after it and
  neither could fail if it were deleted. Clause and constant removed.
- The suspension test I added last commit hard-coded the credit cap instead of
  importing it, so changing STALL_TIMEOUT_MS would leave it passing against a
  system that no longer exists. Now imports MAX_SUSPEND_CREDIT_MS.

## Stale comments corrected

The prune doc still argued at length for the pre-build ordering that 1d9fb11
reversed — a reader trusting it would reopen the blocker 0506369 fixed.
DISK_RESERVE_BYTES claimed to equal the banner threshold that 0506369
deliberately offset by 25%. And host.rs kept its own duplicate 10 GB literal
instead of importing the constant.

154/154 backend, 59/59 vitest, clippy clean, svelte-check 0 errors, eslint
clean, both builds, compose + caddy validate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-08-09 14:51:58 +02:00
parent 214f9e3062
commit ef6d3a077a
14 changed files with 453 additions and 89 deletions

View File

@@ -201,6 +201,16 @@ pub async fn join(
/// Mirrors migration 023; kept here so the invariant below can be asserted in a test.
const RECOVER_NAME_CEILING_DEFAULT: usize = 4;
/// Hard ceiling on the CONFIGURED per-(IP, name) limit, whatever an operator sets.
///
/// The ordering `3 x ceiling <= PIN_LOCK_THRESHOLD` is the entire control that stops one source
/// from locking a victim out: display names are public on the feed, so if a single IP can spend
/// the whole lock threshold it can lock any guest it likes, repeatedly. That ordering was asserted
/// in a comment and in a test — but the test pinned the DEFAULT constant, while the handler reads
/// the config value, and `patch_config` accepted anything from 1 to 100_000. So an operator
/// raising this key restored the exact DoS the tier ordering exists to prevent, silently.
pub const RECOVER_NAME_CEILING_MAX: usize = (PIN_LOCK_THRESHOLD as usize) / 3;
/// Wrong PINs, from ALL sources, before the ACCOUNT itself is locked for 15 minutes.
///
/// This was 3, which sat BELOW the per-(IP, name) ceiling of 5 — and that ordering, not the
@@ -337,7 +347,11 @@ pub async fn recover(
"recover_name_rate_per_15min",
RECOVER_NAME_CEILING_DEFAULT,
)
.await;
.await
// CLAMPED, not merely defaulted — see RECOVER_NAME_CEILING_MAX. The config value is
// operator-settable and the invariant it has to respect is not expressible in
// `patch_config`'s numeric range, so it is enforced at the point of use.
.min(RECOVER_NAME_CEILING_MAX);
let name_key = display_name.to_lowercase();
if let Err(retry_after_secs) = state.rate_limiter.check_with_retry(
format!("recover:{ip}:{name_key}"),
@@ -756,9 +770,10 @@ mod tests {
#[test]
fn one_ip_cannot_reach_the_account_lock() {
assert!(
PIN_LOCK_THRESHOLD as usize >= RECOVER_NAME_CEILING_DEFAULT * 3,
PIN_LOCK_THRESHOLD as usize >= RECOVER_NAME_CEILING_MAX * 3,
"locking a victim must require at least three distinct sources; \
threshold {PIN_LOCK_THRESHOLD} vs per-IP ceiling {RECOVER_NAME_CEILING_DEFAULT}"
threshold {PIN_LOCK_THRESHOLD} vs the ENFORCED per-IP ceiling \
{RECOVER_NAME_CEILING_MAX} (the default is {RECOVER_NAME_CEILING_DEFAULT})"
);
}

View File

@@ -44,9 +44,6 @@ pub struct EventStatus {
pub disk_low: bool,
}
/// Absolute floor below which free space is worth surfacing regardless of gallery size — the
/// threshold the README has carried on the roadmap since v1.
const LOW_DISK_FLOOR_BYTES: u64 = 10_000_000_000;
/// Is free space low enough that the host needs to know?
///
@@ -72,7 +69,12 @@ const LOW_DISK_FLOOR_BYTES: u64 = 10_000_000_000;
fn disk_is_low(free: u64, keepsake_required: u64) -> bool {
let gate_closes_at = keepsake_required.saturating_add(crate::handlers::upload::DISK_RESERVE_BYTES as u64);
let warn_at = gate_closes_at.saturating_add(gate_closes_at / 4);
free < LOW_DISK_FLOOR_BYTES || free < warn_at
// No separate absolute-floor clause. There used to be `free < LOW_DISK_FLOOR_BYTES ||` here,
// and it was unreachable: `gate_closes_at` is at least DISK_RESERVE_BYTES, so `warn_at` is at
// least 1.25x it (12.5 GB) — always above the 10 GB floor. Two tests were named after that
// clause and neither could fail if it were deleted. Keeping dead code that tests claim to
// cover is worse than not having it.
free < warn_at
}
/// Count non-banned hosts/admins in the event OTHER than `excluding` — the operators
@@ -824,7 +826,7 @@ pub async fn release_gallery(
#[cfg(test)]
mod tests {
use super::{LOW_DISK_FLOOR_BYTES, disk_is_low};
use super::disk_is_low;
use crate::handlers::upload::DISK_RESERVE_BYTES;
use crate::services::export::required_free_bytes;
@@ -836,18 +838,18 @@ mod tests {
assert!(!disk_is_low(60 * GB, 25 * GB));
}
/// Renamed from `the_absolute_floor_fires_...`: there is no separate floor clause any more
/// (see `disk_is_low`). What still has to hold is the behaviour the floor was there FOR — a
/// nearly-empty disk is low even when the gallery is small enough that the keepsake term
/// alone would clear it, because all three volumes share one filesystem and Postgres needs
/// room to write.
#[test]
fn the_absolute_floor_fires_even_when_the_gallery_is_tiny() {
fn a_nearly_empty_disk_is_low_even_when_the_gallery_is_tiny() {
// All three volumes share one filesystem, so running out doesn't degrade one subsystem —
// Postgres stops being able to write and the event goes down. A 1 GB gallery would clear
// the keepsake test comfortably; the floor is what catches this.
assert!(disk_is_low(5 * GB, GB));
assert!(disk_is_low(LOW_DISK_FLOOR_BYTES - 1, 0));
// NOT `!disk_is_low(LOW_DISK_FLOOR_BYTES, 0)` any more. The floor is a lower bound, not
// the boundary: the gate-aware trigger now dominates it (at an empty gallery it warns
// below 1.25 x the reserve, i.e. 12.5 GB). Pinning equality here asserted the very
// behaviour that let the wall arrive before the warning.
assert!(disk_is_low(LOW_DISK_FLOOR_BYTES, 0));
assert!(disk_is_low(9 * GB, 0));
assert!(!disk_is_low(20 * GB, 0), "a roomy empty disk is not low");
}
@@ -858,12 +860,33 @@ mod tests {
/// to upload while the dashboard shows a comfortable disk — with nobody on site to ask.
#[test]
fn the_banner_always_fires_before_the_upload_gate_closes() {
// Asserting `disk_is_low(gate_closes_at, required)` is what this used to do, and it was a
// tautology: `disk_is_low` recomputes the same `gate_closes_at` internally and compares
// against `gate + gate/4`, so the assertion reduced to `G < G + G/4` — true for every G,
// for any margin, even a margin of zero. It could not detect the banner being moved to
// exactly the gate, which is the regression it is named for.
//
// So pin the GAP instead: find the free-space level at which the banner starts, and
// require it to be strictly above the level at which the gate closes, by a usable amount.
for media_gb in [0u64, 1, 4, 8, 16, 32] {
let required = required_free_bytes(media_gb * GB, 2);
let gate_closes_at = required + DISK_RESERVE_BYTES as u64;
// Just above the gate: guests can still upload, and the host must already be warned.
assert!(
disk_is_low(gate_closes_at, required),
"at media={media_gb}GB the gate is about to close but no banner is shown"
disk_is_low(gate_closes_at + 1, required),
"at media={media_gb}GB the banner is not yet showing while the gate still allows uploads"
);
// The warning must lead by a margin the host can act inside, not by one byte.
let mut warn_starts_at = gate_closes_at;
while disk_is_low(warn_starts_at, required) {
warn_starts_at += GB / 10;
}
assert!(
warn_starts_at >= gate_closes_at + gate_closes_at / 5,
"at media={media_gb}GB the banner leads the gate by only {} bytes",
warn_starts_at - gate_closes_at
);
}
}
@@ -889,7 +912,7 @@ mod tests {
}
#[test]
fn an_empty_gallery_needs_nothing_and_only_the_floor_applies() {
fn an_empty_gallery_still_reserves_room_for_postgres() {
// With no gallery the keepsake term is 0, so the warn threshold collapses to
// 1.25 x DISK_RESERVE_BYTES (12.5 GB), which dominates the 10 GB absolute floor.
assert!(!disk_is_low(13 * GB, 0));

View File

@@ -248,6 +248,11 @@ pub async fn upload(
// The client's idempotency key. Optional: an older client, or any other caller, simply
// doesn't send one and gets the previous behaviour.
let mut client_upload_id: Option<Uuid> = None;
// Admission reservation for this body's temp bytes. Declared out here so it lives until the
// handler returns — the temp file exists for that whole span, and releasing early would let
// the next upload reserve space this one is still occupying. Dropping it is the release, so
// every exit path (success, error, client disconnect) returns the budget automatically.
let mut _admission: Option<tokio::sync::OwnedSemaphorePermit> = None;
// The multipart read is wrapped so the field loop can use `?` freely; reclaiming the temp
// file on failure is `file_guard`'s job, not this block's.
@@ -274,6 +279,28 @@ pub async fn upload(
} else {
(max_image_mb.max(max_video_mb) * 1024 * 1024) as usize
};
// ADMISSION BEFORE THE FIRST BYTE TOUCHES DISK. The headroom gate below can
// only refuse to COMMIT an upload — by the time it runs, the body has already
// been streamed to its temp file. Nothing else bounds how many bodies stream
// at once (axum imposes no limit, the tower stack is just TraceLayer, Caddy
// passes through), so ~100 guests tapping "upload all" after the ceremony put
// 10-20 GB of `.tmp` on a 40 GB volume that the gate cannot see, eating the
// reserve that keeps Postgres able to write WAL. The permit is held until the
// handler returns, which is exactly as long as the temp file can exist.
_admission = Some(
state
.upload_admission
.reserve(cap_bytes)
.await
.ok_or_else(|| {
AppError::ServiceUnavailable(
"Gerade laden sehr viele Gäste hoch. Dein Foto bleibt in der \
Warteschlange und wird gleich automatisch gesendet."
.into(),
Some(30),
)
})?,
);
tokio::fs::create_dir_all(&originals_dir)
.await
.map_err(|e| AppError::Internal(e.into()))?;
@@ -286,10 +313,19 @@ pub async fn upload(
hashtags_csv = Some(read_text_field_bounded(field, MAX_HASHTAGS_BYTES).await?);
}
"client_upload_id" => {
let raw = field
.text()
.await
.map_err(|e| AppError::BadRequest(e.to_string()))?;
// BOUNDED, like every other text field here. This used `Field::text()`,
// which buffers without any ceiling of its own: axum builds its multipart
// reader with no `SizeLimit`, so the only bound was this route's 576 MiB
// body limit — and `text()` then decodes that into a second full String.
// One request from any authenticated guest, declaring `client_upload_id`
// and sending 576 MiB of padding, peaks well past the app container's 1 GB
// and gets it OOM-killed: every SSE stream dropped, every in-flight upload's
// temp file stranded, `restart: unless-stopped` cycling it. `caption` and
// `hashtags` were bounded by the helper for exactly this reason; this field
// arrived later (migration 022) and missed it.
//
// 64 bytes fits a hyphenated UUID (36) with room to spare.
let raw = read_text_field_bounded(field, 64).await?;
// A malformed key is not worth rejecting an upload over — the photo is the
// thing the guest cares about. Drop the key and lose only the retry
// protection, which is exactly where we were before it existed.
@@ -1064,8 +1100,10 @@ const MIN_QUOTA_LIMIT_BYTES: i64 = 500 * 1024 * 1024;
/// Free space on the media volume that uploads may never consume, whatever any quota says.
///
/// Matches the host dashboard's low-disk threshold (`handlers::host`), so the banner the host
/// sees and the wall guests hit are the same number rather than two unrelated opinions. 10 GB
/// The host dashboard's low-disk banner is DERIVED from this (`handlers::host::disk_is_low`)
/// rather than equal to it: the banner fires at 1.25x the gate's closing point, deliberately, so
/// the host is warned while there is still room to act instead of at the same instant guests hit
/// the wall. Shared expression, offset threshold — do not "restore" them to one number. 10 GB
/// is chosen to leave Postgres, its WAL and a rotation of container logs comfortable room on
/// the shared filesystem long after new uploads have been refused.
pub const DISK_RESERVE_BYTES: i64 = 10_000_000_000;
@@ -1655,18 +1693,44 @@ mod tests {
/// allowed 500 MB, the per-user quota alone would authorise ~50 GB on a 40 GB disk.
#[test]
fn the_global_gate_binds_before_the_per_user_floor_can_overfill_the_disk() {
let per_user_total = MIN_QUOTA_LIMIT_BYTES * 100;
const USABLE: i64 = 35 * GB;
const GUESTS: i64 = 100;
// Premise (constant, so `const`-asserted rather than pretending to be a test):
// the per-user floor alone over-commits the volume, so something else must bind.
const _: () = assert!(MIN_QUOTA_LIMIT_BYTES * 100 > 35 * GB);
// The property. The previous version of this test asserted ONLY the premise above and
// never touched the gate, so no change to `quota_limit_bytes`, `required_free_bytes` or
// DISK_RESERVE_BYTES could have failed it.
//
// Take a moment where the volume holds 10 GB of media. Check both controls against the
// SAME state and assert they disagree in the required direction: the per-user quota still
// says yes, and the global gate already says no.
let media: i64 = 10 * GB;
let free = USABLE - media;
// Per-user: a guest who has uploaded nothing is still granted a full floor-sized
// allowance, because the formula divides free space and then applies the floor.
let per_user = quota_limit_bytes(free, 0.75, GUESTS, GUESTS);
assert!(
per_user_total > 35 * GB,
"premise: the per-user floor alone over-commits the volume, so the global gate \
is what must stop it"
per_user >= MIN_QUOTA_LIMIT_BYTES,
"the per-user quota is expected to still be permissive here, got {per_user}"
);
// Global: the keepsake needs both halves plus the reserve, and they no longer fit.
let required =
crate::services::export::required_free_bytes(media as u64, 2) as i64 + DISK_RESERVE_BYTES;
assert!(
free < required,
"the global gate must already be closed at {media} bytes of media: free {free} \
vs required {required}"
);
}
/// The floor must never write a cheque the volume cannot cash — otherwise a full disk
/// still hands out a 500 MB allowance and the filesystem Postgres needs fills up.
/// The floor must never write a cheque the volume cannot cash, and — the subtler half —
/// must never hand EVERY uploader the whole remaining budget.
/// The floor must never write a cheque the volume cannot cash — otherwise a full disk still
/// hands out a 500 MB allowance and the filesystem Postgres needs fills up — and, the subtler
/// half, it must never hand EVERY uploader the whole remaining budget.
#[test]
fn the_floor_never_exceeds_what_the_disk_can_back() {
assert_eq!(quota_limit_bytes(0, 0.75, 3, 1), 0, "no disk, no allowance");

View File

@@ -13,9 +13,6 @@ use crate::state::SseEvent;
#[derive(Clone)]
pub struct CompressionWorker {
semaphore: Arc<Semaphore>,
/// Serialises the memory-heavy image jobs — see `HEAVY_IMAGE_BYTES`. Separate from
/// `semaphore` so ordinary photos keep full concurrency.
heavy: Arc<Semaphore>,
pool: PgPool,
media_path: PathBuf,
sse_tx: broadcast::Sender<SseEvent>,
@@ -34,7 +31,6 @@ impl CompressionWorker {
) -> Self {
Self {
semaphore: Arc::new(Semaphore::new(concurrency)),
heavy: Arc::new(Semaphore::new(1)),
pool,
media_path,
sse_tx,
@@ -308,19 +304,6 @@ impl CompressionWorker {
/// saving rather than risk the OOM kill.
const OXIPNG_MAX_PIXELS: u64 = 8_000_000;
/// Estimated peak heap above which an image job takes the exclusive `heavy` permit.
///
/// `compression_concurrency` (default 2) bounds how many jobs run at once, but says
/// nothing about how much memory each one costs, and the container gets 1 GiB total. A
/// single 8000x8000 original measures ~516 MiB peak even with the decode correctly scoped
/// — two of those overlapping is 1032 MiB and another OOM kill, from nothing more exotic
/// than two guests uploading big photos at the same moment.
///
/// 150 MiB sits far above a normal phone photo (a 12 MP JPEG costs ~50 MiB all-in) so the
/// common path never serialises, and far below the point where two jobs stop fitting.
/// Throughput is unaffected for everything except the rare giant, which is exactly the
/// case that must not run in parallel with another giant.
const HEAVY_IMAGE_BYTES: u64 = 150 * 1024 * 1024;
/// Wall-clock ceiling for one oxipng run.
///
@@ -356,13 +339,13 @@ impl CompressionWorker {
let estimate =
crate::services::imaging::estimated_processing_peak_bytes(&original, Self::DISPLAY_MAX_EDGE);
let _heavy_permit = match estimate {
Some(bytes) if bytes > Self::HEAVY_IMAGE_BYTES => {
Some(bytes) if bytes > crate::services::imaging::HEAVY_IMAGE_BYTES => {
tracing::debug!(
%upload_id,
estimated_mib = bytes / (1024 * 1024),
"waiting for the heavy-image permit"
);
Some(self.heavy.acquire().await)
Some(crate::services::imaging::HEAVY_IMAGE_PERMITS.acquire().await)
}
_ => None,
};
@@ -768,7 +751,7 @@ mod tests {
)
.expect("header readable");
assert!(
ordinary_peak <= CompressionWorker::HEAVY_IMAGE_BYTES,
ordinary_peak <= crate::services::imaging::HEAVY_IMAGE_BYTES,
"a 12 MP photo estimated at {} MiB would serialise the common path",
ordinary_peak / 1048576
);
@@ -782,7 +765,7 @@ mod tests {
)
.expect("header readable");
assert!(
giant_peak > CompressionWorker::HEAVY_IMAGE_BYTES,
giant_peak > crate::services::imaging::HEAVY_IMAGE_BYTES,
"an 8000x8000 RGBA original estimated at only {} MiB would be allowed to run \
concurrently with another one — 2x its real ~516 MiB peak does not fit in 1 GiB",
giant_peak / 1048576

View File

@@ -522,6 +522,21 @@ async fn ensure_export_space_reclaiming(
ensure_export_space(pool, event_id, export_path).await
}
/// Take the process-wide heavy-image permit if this file is big enough to need it.
///
/// Mirrors the compression worker's gate exactly (a header probe, no pixels decoded), so the two
/// producers of heavy image work agree on what "heavy" means and serialise against each other
/// rather than each against itself.
async fn heavy_permit_for(path: &Path) -> Option<tokio::sync::SemaphorePermit<'static>> {
let estimate = crate::services::imaging::estimated_processing_peak_bytes(path, 2048);
match estimate {
Some(bytes) if bytes > crate::services::imaging::HEAVY_IMAGE_BYTES => {
crate::services::imaging::HEAVY_IMAGE_PERMITS.acquire().await.ok()
}
_ => None,
}
}
// ── ZIP export ───────────────────────────────────────────────────────────────
async fn run_zip_export(
@@ -888,6 +903,12 @@ async fn run_html_export_inner(
let thumb_path = media_tmp.join(&thumb);
let thumb_path_clone = thumb_path.clone();
// Same process-wide memory permit the compression worker takes. Without it, a
// release fired while the last phone photos were still compressing put an export
// decode and a heavy compression job in the same 1 GiB cgroup — and the OOM kill
// marks the export failed, which `recover_exports` then re-spawns into the same
// conditions on the next boot.
let _heavy = heavy_permit_for(&src).await;
let thumb_result = tokio::task::spawn_blocking(move || -> Result<()> {
// `decode_oriented`, not `image::open`: the latter ignores the EXIF
// orientation tag AND applies no decode limits. Using it here is why every
@@ -922,6 +943,9 @@ async fn run_html_export_inner(
let full_path = media_tmp.join(&full);
let full_path_clone = full_path.clone();
// See the thumbnail above. This branch is the more expensive of the two: it
// only runs for originals over 5 MB, i.e. exactly the giants.
let _heavy = heavy_permit_for(&src).await;
let compress_result = tokio::task::spawn_blocking(move || -> Result<()> {
// Same reason as the thumbnail above. This branch only runs for originals
// over 5 MB, which is why the viewer's full image looked correct for small
@@ -1315,16 +1339,19 @@ async fn protected_files(pool: &PgPool, event_id: Uuid) -> Vec<String> {
.unwrap_or_default()
}
/// Reclaim superseded FINAL archives BEFORE this generation starts writing its own.
/// Reclaim superseded FINAL archives.
///
/// Peak disk usage used to be two full generations, because the only prune ran after the new archive
/// was written, renamed and finalised. That ordering reads as durability ("don't delete the good
/// keepsake before the replacement is safe") but it buys nothing: readiness is derived from
/// `job.epoch = event.export_epoch AND status = 'done'`, so the moment `invalidate_and_arm` bumps
/// the epoch the old archive is ALREADY unreachable — `GET /export/zip` 404s whether the file is on
/// disk or not. Keeping it only reserves gigabytes for a download nobody can perform, and for
/// `Affects::Both` (a takedown) it is content someone has explicitly asked to have removed. So a
/// rebuild reclaims first and peaks at one generation.
/// CALLED AFTER A SUCCESSFUL BUILD, not before one. This doc used to argue the opposite at
/// length — that since readiness is derived from `job.epoch = event.export_epoch`, a superseded
/// archive is already unreachable and keeping it "buys nothing". That reasoning is right about
/// REACHABILITY and wrong about RECOVERABILITY: an epoch is a database value that can be rolled
/// back, deleted bytes cannot. Pruning first meant any rebuild that then failed — ENOSPC, an OOM,
/// a hung ffmpeg — left the event with NO archive at all, which is the one outcome the product
/// exists to prevent, at the one moment nobody is watching.
///
/// The single exception is phase 2 of `ensure_export_space_reclaiming`, where the previous
/// generation's bytes are the only way the rebuild can fit at all. Both call sites carry the full
/// reasoning; do not "restore" a pre-build prune on the strength of this function's convenience.
///
/// Narrower than [`prune_stale_export_files`] on purpose: FINAL archives only. Those are inert — a
/// superseded worker either already renamed its file (and will delete it itself when its guarded

View File

@@ -205,6 +205,32 @@ pub fn decode_oriented(path: &Path) -> Result<DynamicImage> {
Ok(img)
}
/// Process-wide serialisation for memory-heavy image work.
///
/// The `app` container gets 1 GiB. A single 8000x8000 original measures ~516 MiB peak even with
/// the decode correctly scoped, so two overlapping giants is an OOM kill — and the kernel kills
/// the whole process, dropping every SSE stream and stranding every in-flight upload.
///
/// GLOBAL rather than a field on `CompressionWorker`, because the constraint is the container's
/// memory and there is more than one producer of this work. The export's own image path
/// (`services::export`) decodes and resizes every photo in the gallery — a thumbnail for each,
/// plus a 2000px re-encode for every original over 5 MB — and it ran in a bare `spawn_blocking`
/// with no permit at all. So "host taps Freigeben while the last phone photos are still
/// compressing" put an export decode and a heavy compression job in the same cgroup at the same
/// time, which is the scenario the permit exists to make impossible. Worse, it is self-repeating:
/// the OOM kill marks the export failed, and `recover_exports` re-spawns it on boot into the same
/// conditions.
///
/// Held across the blocking section and released on drop, including on error.
pub static HEAVY_IMAGE_PERMITS: std::sync::LazyLock<tokio::sync::Semaphore> =
std::sync::LazyLock::new(|| tokio::sync::Semaphore::new(1));
/// Estimated peak heap above which a job must take [`HEAVY_IMAGE_PERMITS`].
///
/// 150 MiB sits far above a normal phone photo (a 12 MP JPEG costs ~50 MiB all-in) so the common
/// path never serialises, and far below the point where two jobs stop fitting in the container.
pub const HEAVY_IMAGE_BYTES: u64 = 150 * 1024 * 1024;
#[cfg(test)]
mod tests {
use super::*;

View File

@@ -61,15 +61,37 @@ impl MediaTotalCache {
{
return bytes;
}
let bytes = sqlx::query_scalar::<_, Option<i64>>(
let queried = sqlx::query_scalar::<_, Option<i64>>(
"SELECT SUM(total_upload_bytes)::bigint FROM \"user\"",
)
.fetch_one(pool)
.await
.ok()
.flatten()
.unwrap_or(0)
.max(0);
.await;
let bytes = match queried {
Ok(v) => v.unwrap_or(0).max(0),
Err(e) => {
// FAIL OPEN, but do NOT cache the failure, and do NOT let it pass silently.
//
// Storing 0 here pinned the gate's view of the event at "empty" for the whole
// TTL. During that window `media_after` is just this upload, `keepsake_needs`
// collapses to ~2.2x one file, and the gate degrades to the flat 10 GB reserve —
// precisely the behaviour the two-halves design replaced, reappearing with no
// trace in the log. And the trigger correlates with the danger: with
// `max_connections = 10` and a 5s acquire timeout, this query fails exactly when
// a burst is in progress.
//
// Falling back to the LAST GOOD reading (however stale) is strictly better than
// 0: the total only ever grows, so a stale value under-counts slightly, while 0
// under-counts by everything.
let previous = self.inner.read().unwrap().map(|(b, _)| b);
tracing::warn!(
error = %e,
fallback_bytes = previous.unwrap_or(0),
"media total query failed; upload gate is running on a stale reading"
);
return previous.unwrap_or(0);
}
};
*self.inner.write().unwrap() = Some((bytes, Instant::now()));
bytes
}

View File

@@ -7,4 +7,5 @@ pub mod maintenance;
pub mod media_total;
pub mod rate_limiter;
pub mod sse_tickets;
pub mod upload_admission;
pub mod video;

View File

@@ -0,0 +1,155 @@
//! Admission control for upload bodies, budgeted in BYTES rather than requests.
//!
//! ## Why this has to exist
//!
//! The keepsake headroom gate in `handlers::upload` cannot bound a burst, and the reason is
//! structural rather than a bug in the gate: the request body is streamed to a temp file during
//! multipart parsing, so the bytes are already on disk by the time any check runs. The gate can
//! only refuse to COMMIT them. Nothing upstream limited how many bodies stream at once — axum has
//! no such limit, the tower stack is just `TraceLayer`, and Caddy passes requests straight
//! through.
//!
//! So the failure mode is the ordinary one, not an attack: the ceremony ends, ~100 guests tap
//! "upload all", and ~100 bodies stream concurrently. At phone-video sizes that is 10-20 GB of
//! `.tmp` files on a 40 GB volume, none of it visible to the gate, and `DISK_RESERVE_BYTES` — the
//! 10 GB standing between the party and Postgres losing the volume it writes WAL to — is consumed
//! by transient files. The `.tmp` sweeper only reclaims files idle for an hour, correctly, which
//! means nothing reclaims a burst on this timescale.
//!
//! ## Why bytes and not a request count
//!
//! A flat "N concurrent uploads" limit has to be sized for the worst case (a 500 MB video), which
//! makes it absurdly restrictive for the common case (a 3 MB photo). Budgeting bytes lets one
//! 500 MB video and two hundred photos coexist under the same ceiling, and it means the ceiling is
//! stated in the unit the disk actually cares about.
//!
//! The reservation is the streaming CAP, not the real size — the real size is unknowable until the
//! body has been read, which is far too late. Reserving the cap is deliberately pessimistic; that
//! pessimism is the safety margin.
//!
//! ## Why a permit and not a counter
//!
//! `OwnedSemaphorePermit` releases on drop. Every path out of the upload handler — success, error,
//! a client vanishing mid-body, a panic — therefore returns the reservation without any explicit
//! bookkeeping. A hand-rolled `AtomicI64` would need a decrement on each of those paths, and the
//! one that gets missed is the one that leaks the budget until restart.
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::{OwnedSemaphorePermit, Semaphore};
/// Total transient upload bytes allowed on disk at once, in MiB.
///
/// Sized against `DISK_RESERVE_BYTES` (10 GB): the reserve must survive a full burst with room to
/// spare, since Postgres is writing WAL to the same filesystem throughout. 4 GiB leaves ~6 GB of
/// the reserve untouched at the worst moment.
///
/// It is NOT a throughput limit. On 2 vCPU the box cannot usefully ingest more than this at once
/// anyway — compression, ffmpeg, Postgres and TLS all contend for the same two cores — so the
/// budget mostly converts "everything is slow and the disk fills" into "a few uploads wait".
const BUDGET_MIB: u32 = 4096;
/// How long an upload waits for room before being told to come back.
///
/// Long enough to absorb the burst (a photo holds its reservation for well under a second), short
/// enough that a guest is not left staring at a spinner. On timeout the handler answers 503 with
/// `Retry-After`, which the client queue already treats as transient and retries with backoff.
const WAIT: Duration = Duration::from_secs(20);
#[derive(Clone)]
pub struct UploadAdmission {
permits: Arc<Semaphore>,
}
impl UploadAdmission {
pub fn new() -> Self {
Self {
permits: Arc::new(Semaphore::new(BUDGET_MIB as usize)),
}
}
/// Reserve room for a body capped at `cap_bytes`. The returned permit must be held for as long
/// as the temp file exists.
///
/// `None` means the wait timed out and the caller should shed the request.
///
/// A cap larger than the whole budget is clamped rather than refused. Otherwise an operator
/// raising `max_video_size_mb` above the budget would make `acquire_many` unsatisfiable and
/// every video upload would hang until timeout — a config change silently disabling video for
/// the event. Clamped, such an upload simply gets the whole budget to itself, which is the
/// honest interpretation of "one file may fill the machine".
pub async fn reserve(&self, cap_bytes: usize) -> Option<OwnedSemaphorePermit> {
let mib = cap_bytes.div_ceil(1024 * 1024).max(1);
let want = u32::try_from(mib).unwrap_or(BUDGET_MIB).min(BUDGET_MIB);
match tokio::time::timeout(
WAIT,
self.permits.clone().acquire_many_owned(want),
)
.await
{
Ok(Ok(permit)) => Some(permit),
// The semaphore is never closed, so `Err` here is unreachable in practice; treat it
// the same as a timeout rather than panicking on the upload path.
Ok(Err(_)) => None,
Err(_) => {
tracing::warn!(
requested_mib = want,
"upload admission timed out; shedding to keep transient temp files bounded"
);
None
}
}
}
}
impl Default for UploadAdmission {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
/// The budget must actually block once exhausted — otherwise this whole module is decoration.
#[tokio::test]
async fn a_full_budget_sheds_instead_of_admitting() {
let admission = UploadAdmission::new();
let whole = admission
.reserve(BUDGET_MIB as usize * 1024 * 1024)
.await
.expect("first reservation takes the whole budget");
// Nothing left: a second reservation must not be granted. Raced against a short timeout so
// the test does not sit for the full WAIT.
let blocked = tokio::time::timeout(
Duration::from_millis(150),
admission.reserve(1024 * 1024),
)
.await;
assert!(blocked.is_err(), "budget exhausted, yet a reservation was granted");
// ...and releasing the permit makes room again, so the budget is not a one-way latch.
drop(whole);
assert!(
admission.reserve(1024 * 1024).await.is_some(),
"budget did not recover after the permit was dropped"
);
}
/// A cap above the whole budget must be clamped, not left unsatisfiable. Unclamped,
/// `acquire_many` for more permits than exist never completes, so raising
/// `max_video_size_mb` past the budget would silently hang every video upload for 20s and
/// then shed it.
#[tokio::test]
async fn a_cap_larger_than_the_budget_is_clamped_rather_than_unsatisfiable() {
let admission = UploadAdmission::new();
let oversized = (BUDGET_MIB as usize + 4096) * 1024 * 1024;
assert!(
admission.reserve(oversized).await.is_some(),
"an over-budget cap must still be admittable on an idle server"
);
}
}

View File

@@ -8,6 +8,7 @@ use crate::services::disk::DiskCache;
use crate::services::media_total::MediaTotalCache;
use crate::services::rate_limiter::RateLimiter;
use crate::services::sse_tickets::SseTicketStore;
use crate::services::upload_admission::UploadAdmission;
#[derive(Clone, Debug)]
pub struct SseEvent {
@@ -41,6 +42,10 @@ pub struct AppState {
pub disk_cache: DiskCache,
/// Cached sum of all media bytes, for the upload gate's keepsake-headroom check.
pub media_total: MediaTotalCache,
/// Byte budget for upload bodies currently streaming to temp files. The headroom gate can
/// only refuse to COMMIT bytes that are already on disk; this is what bounds how many get
/// there at once.
pub upload_admission: UploadAdmission,
}
impl AppState {
@@ -67,6 +72,7 @@ impl AppState {
config_cache,
disk_cache: DiskCache::new(),
media_total: MediaTotalCache::new(),
upload_admission: UploadAdmission::new(),
}
}
}