fix: close what nine adversarial reviews found, most of it mine
Some checks failed
Checks / Backend — cargo test + clippy + fmt (push) Failing after 1m5s
Checks / Frontend — vitest + svelte-check (push) Failing after 5m55s
Checks / E2E — typecheck + lint (push) Failing after 49s
E2E / Playwright E2E (chromium + webkit) (push) Failing after 10m42s
E2E / Cross-UA smoke matrix (push) Failing after 7m57s
Audit / cargo audit (backend) (push) Failing after 11m12s
Audit / npm audit (frontend) (push) Successful in 44s
Some checks failed
Checks / Backend — cargo test + clippy + fmt (push) Failing after 1m5s
Checks / Frontend — vitest + svelte-check (push) Failing after 5m55s
Checks / E2E — typecheck + lint (push) Failing after 49s
E2E / Playwright E2E (chromium + webkit) (push) Failing after 10m42s
E2E / Cross-UA smoke matrix (push) Failing after 7m57s
Audit / cargo audit (backend) (push) Failing after 11m12s
Audit / npm audit (frontend) (push) Successful in 44s
Nine focused reviews (export state machine, upload path, auth/abuse, client queue, guest UI, database, deploy/ops, regression hunt, test honesty). Every finding below was re-verified against the code before being acted on; several plausible-sounding ones were checked and rejected. ## Data loss and denial of service **One request could OOM-kill the app container.** `client_upload_id` was read with `Field::text()` — axum builds its multipart reader with no SizeLimit, so the only bound was the route's 576 MiB body limit, then decoded into a second full String. `caption` and `hashtags` go through `read_text_field_bounded` for exactly this reason; this field arrived later and missed it. Any guest, one request, and every SSE stream drops and every in-flight temp file is stranded. **Nothing bounded concurrent upload bodies.** The headroom gate can only refuse to COMMIT — the body is already streamed to a temp file by the time it runs, and neither axum, the tower stack nor Caddy limits how many stream at once. ~100 guests tapping "upload all" after the ceremony puts 10-20 GB of .tmp on a 40 GB volume, invisible to the gate, eating the reserve that keeps Postgres able to write WAL. New `UploadAdmission` budgets bytes (not requests, so one video and two hundred photos coexist) via a permit that releases on drop, so every exit path returns it. **The export decode bypassed the memory permit the compression path takes.** Same class of work — decode + resize every image in the gallery — in a bare spawn_blocking. A release fired while the last photos were still compressing put both in the same 1 GiB cgroup; the OOM kill marks the export failed and `recover_exports` re-spawns it into the same conditions on the next boot. The permit is now process-wide in `imaging`, because the constraint it expresses is the container's memory, not one worker's. **`MediaTotalCache` cached its own failure as 0.** For the whole TTL the gate then saw an empty event and collapsed to the flat reserve — the behaviour the two-halves design replaced — with no log line. And the trigger correlates with the danger: with max_connections 10 the query fails exactly during a burst. Now falls back to the last good reading and says so. **V8's heap ceiling sat above the frontend container's entire budget** (measured: 259 MB inside a 256M limit), so GC could never intervene and the only backpressure was SIGKILL under an arrival burst. ## Guest-visible **The feed stopped being newest-first after the first reconcile.** It fetches whole 100-item server pages while `uploads` grows in 20s, so everything in the gap was absent from `present`, classified as new, and prepended — ~80 photos from earlier in the evening above the newest ones. It also stalled infinite scroll, since the cursor still pointed at item 20 and the observer only re-fires on a change. The union is now sorted on the server's own (created_at, id) key, which additionally places an SSE arrival correctly. **A stale `loadMoreError` outlived every refresh and filter change**, leaving a false error above a button that returns immediately on `!nextCursor`. **A failed derivative toasted "Ein Upload konnte nicht verarbeitet werden."** for a photo sitting right there on screen — the handler still assumed 1d9fb11's pre-fix behaviour (row deleted, quota refunded, card evicted), none of which is true any more. It was the last surviving route for the "your photo is gone" signal that fix set out to remove. ## Enforcement that existed only in comments `recover_name_rate_per_15min` is clamped at the point of use: the ordering `3 x ceiling <= PIN_LOCK_THRESHOLD` is the whole control against one source locking any guest whose name is on the feed, it was asserted in a comment, and `patch_config` accepted 1..100_000. The test pinned the default constant rather than the enforced bound; it now pins the bound. ## Tests that could not fail - The gate test asserted only its own premise (`500MB x 100 > 35GB`) and never touched the gate. It now checks both controls against the same state and requires them to disagree in the right direction. - `the_banner_always_fires_before_the_upload_gate_closes` reduced to `G < G + G/4` — true for any margin, including zero, so it could not detect the banner moving to exactly the gate. It now pins the gap. - `disk_is_low`'s `free < LOW_DISK_FLOOR_BYTES` clause was unreachable (warn_at is always >= 12.5 GB against a 10 GB floor). Two tests were named after it and neither could fail if it were deleted. Clause and constant removed. - The suspension test I added last commit hard-coded the credit cap instead of importing it, so changing STALL_TIMEOUT_MS would leave it passing against a system that no longer exists. Now imports MAX_SUSPEND_CREDIT_MS. ## Stale comments corrected The prune doc still argued at length for the pre-build ordering that1d9fb11reversed — a reader trusting it would reopen the blocker0506369fixed. DISK_RESERVE_BYTES claimed to equal the banner threshold that0506369deliberately offset by 25%. And host.rs kept its own duplicate 10 GB literal instead of importing the constant. 154/154 backend, 59/59 vitest, clippy clean, svelte-check 0 errors, eslint clean, both builds, compose + caddy validate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -248,6 +248,11 @@ pub async fn upload(
|
||||
// The client's idempotency key. Optional: an older client, or any other caller, simply
|
||||
// doesn't send one and gets the previous behaviour.
|
||||
let mut client_upload_id: Option<Uuid> = None;
|
||||
// Admission reservation for this body's temp bytes. Declared out here so it lives until the
|
||||
// handler returns — the temp file exists for that whole span, and releasing early would let
|
||||
// the next upload reserve space this one is still occupying. Dropping it is the release, so
|
||||
// every exit path (success, error, client disconnect) returns the budget automatically.
|
||||
let mut _admission: Option<tokio::sync::OwnedSemaphorePermit> = None;
|
||||
|
||||
// The multipart read is wrapped so the field loop can use `?` freely; reclaiming the temp
|
||||
// file on failure is `file_guard`'s job, not this block's.
|
||||
@@ -274,6 +279,28 @@ pub async fn upload(
|
||||
} else {
|
||||
(max_image_mb.max(max_video_mb) * 1024 * 1024) as usize
|
||||
};
|
||||
// ADMISSION BEFORE THE FIRST BYTE TOUCHES DISK. The headroom gate below can
|
||||
// only refuse to COMMIT an upload — by the time it runs, the body has already
|
||||
// been streamed to its temp file. Nothing else bounds how many bodies stream
|
||||
// at once (axum imposes no limit, the tower stack is just TraceLayer, Caddy
|
||||
// passes through), so ~100 guests tapping "upload all" after the ceremony put
|
||||
// 10-20 GB of `.tmp` on a 40 GB volume that the gate cannot see, eating the
|
||||
// reserve that keeps Postgres able to write WAL. The permit is held until the
|
||||
// handler returns, which is exactly as long as the temp file can exist.
|
||||
_admission = Some(
|
||||
state
|
||||
.upload_admission
|
||||
.reserve(cap_bytes)
|
||||
.await
|
||||
.ok_or_else(|| {
|
||||
AppError::ServiceUnavailable(
|
||||
"Gerade laden sehr viele Gäste hoch. Dein Foto bleibt in der \
|
||||
Warteschlange und wird gleich automatisch gesendet."
|
||||
.into(),
|
||||
Some(30),
|
||||
)
|
||||
})?,
|
||||
);
|
||||
tokio::fs::create_dir_all(&originals_dir)
|
||||
.await
|
||||
.map_err(|e| AppError::Internal(e.into()))?;
|
||||
@@ -286,10 +313,19 @@ pub async fn upload(
|
||||
hashtags_csv = Some(read_text_field_bounded(field, MAX_HASHTAGS_BYTES).await?);
|
||||
}
|
||||
"client_upload_id" => {
|
||||
let raw = field
|
||||
.text()
|
||||
.await
|
||||
.map_err(|e| AppError::BadRequest(e.to_string()))?;
|
||||
// BOUNDED, like every other text field here. This used `Field::text()`,
|
||||
// which buffers without any ceiling of its own: axum builds its multipart
|
||||
// reader with no `SizeLimit`, so the only bound was this route's 576 MiB
|
||||
// body limit — and `text()` then decodes that into a second full String.
|
||||
// One request from any authenticated guest, declaring `client_upload_id`
|
||||
// and sending 576 MiB of padding, peaks well past the app container's 1 GB
|
||||
// and gets it OOM-killed: every SSE stream dropped, every in-flight upload's
|
||||
// temp file stranded, `restart: unless-stopped` cycling it. `caption` and
|
||||
// `hashtags` were bounded by the helper for exactly this reason; this field
|
||||
// arrived later (migration 022) and missed it.
|
||||
//
|
||||
// 64 bytes fits a hyphenated UUID (36) with room to spare.
|
||||
let raw = read_text_field_bounded(field, 64).await?;
|
||||
// A malformed key is not worth rejecting an upload over — the photo is the
|
||||
// thing the guest cares about. Drop the key and lose only the retry
|
||||
// protection, which is exactly where we were before it existed.
|
||||
@@ -1064,8 +1100,10 @@ const MIN_QUOTA_LIMIT_BYTES: i64 = 500 * 1024 * 1024;
|
||||
|
||||
/// Free space on the media volume that uploads may never consume, whatever any quota says.
|
||||
///
|
||||
/// Matches the host dashboard's low-disk threshold (`handlers::host`), so the banner the host
|
||||
/// sees and the wall guests hit are the same number rather than two unrelated opinions. 10 GB
|
||||
/// The host dashboard's low-disk banner is DERIVED from this (`handlers::host::disk_is_low`)
|
||||
/// rather than equal to it: the banner fires at 1.25x the gate's closing point, deliberately, so
|
||||
/// the host is warned while there is still room to act instead of at the same instant guests hit
|
||||
/// the wall. Shared expression, offset threshold — do not "restore" them to one number. 10 GB
|
||||
/// is chosen to leave Postgres, its WAL and a rotation of container logs comfortable room on
|
||||
/// the shared filesystem long after new uploads have been refused.
|
||||
pub const DISK_RESERVE_BYTES: i64 = 10_000_000_000;
|
||||
@@ -1655,18 +1693,44 @@ mod tests {
|
||||
/// allowed 500 MB, the per-user quota alone would authorise ~50 GB on a 40 GB disk.
|
||||
#[test]
|
||||
fn the_global_gate_binds_before_the_per_user_floor_can_overfill_the_disk() {
|
||||
let per_user_total = MIN_QUOTA_LIMIT_BYTES * 100;
|
||||
const USABLE: i64 = 35 * GB;
|
||||
const GUESTS: i64 = 100;
|
||||
|
||||
// Premise (constant, so `const`-asserted rather than pretending to be a test):
|
||||
// the per-user floor alone over-commits the volume, so something else must bind.
|
||||
const _: () = assert!(MIN_QUOTA_LIMIT_BYTES * 100 > 35 * GB);
|
||||
|
||||
// The property. The previous version of this test asserted ONLY the premise above and
|
||||
// never touched the gate, so no change to `quota_limit_bytes`, `required_free_bytes` or
|
||||
// DISK_RESERVE_BYTES could have failed it.
|
||||
//
|
||||
// Take a moment where the volume holds 10 GB of media. Check both controls against the
|
||||
// SAME state and assert they disagree in the required direction: the per-user quota still
|
||||
// says yes, and the global gate already says no.
|
||||
let media: i64 = 10 * GB;
|
||||
let free = USABLE - media;
|
||||
|
||||
// Per-user: a guest who has uploaded nothing is still granted a full floor-sized
|
||||
// allowance, because the formula divides free space and then applies the floor.
|
||||
let per_user = quota_limit_bytes(free, 0.75, GUESTS, GUESTS);
|
||||
assert!(
|
||||
per_user_total > 35 * GB,
|
||||
"premise: the per-user floor alone over-commits the volume, so the global gate \
|
||||
is what must stop it"
|
||||
per_user >= MIN_QUOTA_LIMIT_BYTES,
|
||||
"the per-user quota is expected to still be permissive here, got {per_user}"
|
||||
);
|
||||
|
||||
// Global: the keepsake needs both halves plus the reserve, and they no longer fit.
|
||||
let required =
|
||||
crate::services::export::required_free_bytes(media as u64, 2) as i64 + DISK_RESERVE_BYTES;
|
||||
assert!(
|
||||
free < required,
|
||||
"the global gate must already be closed at {media} bytes of media: free {free} \
|
||||
vs required {required}"
|
||||
);
|
||||
}
|
||||
|
||||
/// The floor must never write a cheque the volume cannot cash — otherwise a full disk
|
||||
/// still hands out a 500 MB allowance and the filesystem Postgres needs fills up.
|
||||
/// The floor must never write a cheque the volume cannot cash, and — the subtler half —
|
||||
/// must never hand EVERY uploader the whole remaining budget.
|
||||
/// The floor must never write a cheque the volume cannot cash — otherwise a full disk still
|
||||
/// hands out a 500 MB allowance and the filesystem Postgres needs fills up — and, the subtler
|
||||
/// half, it must never hand EVERY uploader the whole remaining budget.
|
||||
#[test]
|
||||
fn the_floor_never_exceeds_what_the_disk_can_back() {
|
||||
assert_eq!(quota_limit_bytes(0, 0.75, 3, 1), 0, "no disk, no allowance");
|
||||
|
||||
Reference in New Issue
Block a user