fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'fix/deploy-unattended-blockers' into main

Browse Source 
Two independent lines of production hardening diverged at 7d0334b and attacked
overlapping problems. Neither was a superset, so this is a merge of substance
rather than a fast-forward: every conflict was resolved on the merits, and the
losing side's intent was re-checked against the winner rather than assumed.

MIGRATIONS. The branch's 021/022/023 collided with main's already-DEPLOYED
021_hashtag_counts_respect_bans and 022_client_upload_idempotency. Renumbered to
023/024/025 in a prior commit — main's versions are applied in production, so
their version numbers are immutable and the branch's had to move. Verified by
running the full sqlx::test suite, which applies the whole chain from scratch.

RESOLVED IN MAIN'S FAVOUR (the branch would have regressed these):
  * upload-queue.ts wholesale — the branch's copy has ZERO client_upload_id
    references, so taking it would have silently destroyed end-to-end upload
    idempotency, the one thing standing between a lost response and a duplicate
    photo charged twice against the guest's quota.
  * maintenance.rs supervisor — the branch replaced it with a bare tokio::spawn,
    where one panic silently stops session pruning, media reclaim, the temp
    sweep and both HashMap prunes, permanently and with no log line.
  * The decode-budget probe on spawn_blocking, not inline on the async runtime.
  * feed/+page.svelte's 8s debounce + jitter + max-wait + hidden-tab deferral,
    against the branch's naive 800ms — at 100 guests the branch's version walks
    straight into the per-user feed rate limit.
  * db.rs pool tuning, /uploaders, and the docker-compose deployment story.
  * ONE /health, still DB-backed. The branch's split (dependency-free liveness +
    DB-backed readiness) is defensible, but a constant-"ok" /health is the exact
    defect faea555 fixed and verified live, its motive (Caddy's boot gate) is
    already covered by app depends_on db: service_healthy, and the two handlers
    were the same SELECT 1 under two names.

TAKEN FROM THE BRANCH:
  * The large-PNG OOM guard and its bounded-retry counter (023). Together these
    turn a single upload that can OOM-kill a 1G container into a bounded failure
    instead of an infinite restart loop under `restart: unless-stopped`.
  * 024_feed_scalar_counts — the feed no longer aggregates the whole event per
    page. Pure SQL; column names, order and types are unchanged by design.
  * The admin-lockout fix: look the admin up BY ROLE, never by name. 025 also
    frees any guest already squatting on a reserved name.
  * PIN lockout tier ordering, bounded caption/hashtag reads, SSE ticket caps,
    PoolTimedOut -> 503 + Retry-After, and the ffmpeg stderr drain.
  * backfill_video_posters, which main lacked entirely.
  * TempFileGuard, plus sweep_orphan_originals wired into main's SUPERVISED loop
    (not the branch's bare one) — it reclaims final-named originals whose commit
    never happened, a class main's .tmp-only sweep structurally cannot see.
  * shouldAbortForStall, hand-ported into main's upload-queue.ts since that file
    was resolved to main. Widens the watchdog at loadend instead of disarming it,
    bounding a half-open socket at 2 minutes rather than handing the window to
    xhr.timeout (5-60 min) with the whole queue's `processing` latch held.

ALSO: RUST_LOG and EXPORT_PATH pinned in compose. The code fallback was
`debug` (a line per request, all night) and EXPORT_PATH was the one path with a
mount-shaped default that nothing validated.

Verified: cargo check --all-targets, cargo clippy (clean), 144/144 backend tests
against a live Postgres including upload_idempotency and upload_concurrency,
51/51 vitest, svelte-check 0 errors, eslint clean, vite build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-08-08 21:16:31 +02:00
parent d5b4bf0ac1 e1c689d1a7
commit 61119be817
30 changed files with 2192 additions and 439 deletions
Download Patch File Download Diff File Expand all files Collapse all files

368
backend/src/handlers/upload.rs
View File

@@ -17,6 +17,135 @@ use crate::state::AppState;
const MAX_CAPTION_LENGTH: usize = 2000;
/// Byte ceiling for the caption field, enforced WHILE reading it.
///
/// `Field::text()` buffers the entire field before returning, and this is the one route whose
/// `DefaultBodyLimit` is raised to 576 MiB (main.rs) — so `caption=<576 MiB of text>` allocated
/// 576 MiB of heap per concurrent request inside a 1 GiB container, and the
/// `MAX_CAPTION_LENGTH` check only ran afterwards, on a string that had already been built.
/// 4 bytes per code point is the worst case for UTF-8, so this can never reject a caption the
/// character limit would have accepted.
const MAX_CAPTION_BYTES: usize = MAX_CAPTION_LENGTH * 4;
/// Byte ceiling for the raw hashtag CSV. Generous next to what the tag caps below allow.
const MAX_HASHTAGS_BYTES: usize = 4 * 1024;
/// Hashtags stored per upload. The CSV was never length-checked at all and was split into an
/// unbounded `Vec`, then upserted TAG BY TAG inside the commit transaction — which holds a
/// `FOR SHARE` lock on the event row, so one request could stall every other upload behind
/// tens of thousands of round trips.
const MAX_HASHTAGS_PER_UPLOAD: usize = 30;
/// Characters per stored tag. `extract_hashtags` already self-bounds at 40; this covers the CSV
/// path, which had no bound of its own.
const MAX_HASHTAG_LENGTH: usize = 50;
/// Read a multipart text field, refusing it the moment it exceeds `max_bytes`.
///
/// The point is to fail DURING the read rather than after it — `Field::text()` cannot, because
/// it has already allocated the whole thing by the time it returns.
async fn read_text_field_bounded(
mut field: axum::extract::multipart::Field<'_>,
max_bytes: usize,
) -> Result<String, AppError> {
let mut buf: Vec<u8> = Vec::new();
while let Some(chunk) = field
.chunk()
.await
.map_err(|e| AppError::BadRequest(e.to_string()))?
{
if buf.len() + chunk.len() > max_bytes {
return Err(AppError::BadRequest(
"Eingabe ist zu lang.".to_string(),
));
}
buf.extend_from_slice(&chunk);
}
String::from_utf8(buf).map_err(|_| AppError::BadRequest("Ungültige Zeichenkodierung.".into()))
}
/// Normalise, dedupe and CAP the tags for one upload.
///
/// Extracted as a pure function so the caps are testable without standing up multipart, and
/// shared by the upload and edit paths — which previously disagreed: upload lowercased and
/// stripped `#`, while edit upserted raw strings, so `#Party` via edit and `party` via upload
/// became two different hashtag rows.
///
/// Truncates rather than rejecting. `extract_hashtags` legitimately derives tags from a
/// 2000-character caption, and 400-ing a guest for writing an enthusiastic caption would be a
/// worse outcome than silently keeping the first 30.
fn normalize_tags(caption_tags: Vec<String>, csv: Option<&str>) -> Vec<String> {
let mut tags = caption_tags;
if let Some(csv) = csv {
for tag in csv.split(',') {
let t = tag.trim().trim_start_matches('#').to_lowercase();
if !t.is_empty() {
tags.push(t);
}
}
}
tags.sort();
tags.dedup();
tags.retain(|t| t.chars().count() <= MAX_HASHTAG_LENGTH);
tags.truncate(MAX_HASHTAGS_PER_UPLOAD);
tags
}
/// Owns the bytes an in-flight upload has written to disk, and deletes them unless the
/// request reaches the point where a database row takes ownership.
///
/// Reclaim used to be a dozen explicit `remove_file` calls on the handler's return paths.
/// That covers every way the handler can FINISH, and none of the ways it can simply STOP:
/// when a client disconnects mid-body — a phone leaving wifi, iOS killing a backgrounded
/// PWA, the user hitting back — axum drops the handler future at a `.await` inside
/// `field.chunk()`, and no return path runs at all. The partial file then survives forever:
/// it has no upload row, so `cleanup_deleted_media` (which is row-driven) can never see it,
/// and no sweeper existed for the originals directory. Those bytes are also invisible to the
/// quota while still consuming the free disk that `quota_limit_bytes` divides among guests.
///
/// A drop guard is the only construct that survives cancellation, because dropping the future
/// is exactly what runs it.
struct TempFileGuard {
/// `None` once disarmed — a row now owns these bytes.
path: Option<std::path::PathBuf>,
}
impl TempFileGuard {
fn new(path: std::path::PathBuf) -> Self {
Self { path: Some(path) }
}
/// Follow the bytes to their new location after a rename.
///
/// NOT `disarm`. Between the rename and the commit the file exists under its FINAL name
/// with still no row pointing at it, so that window needs guarding just as much as the
/// `.tmp` did — arguably more, since a leftover final-named original looks legitimate.
fn retarget(&mut self, path: std::path::PathBuf) {
self.path = Some(path);
}
/// Hand ownership to the committed row. Only correct after `tx.commit()` succeeds.
fn disarm(&mut self) {
self.path = None;
}
}
impl Drop for TempFileGuard {
fn drop(&mut self) {
let Some(path) = self.path.take() else {
return;
};
// std::fs, not tokio::fs: `Drop` cannot await, and a runtime-dependent unlink is not
// guaranteed a live runtime here (shutdown drops in-flight tasks).
match std::fs::remove_file(&path) {
Ok(()) => tracing::debug!(path = %path.display(), "reclaimed an abandoned upload"),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
Err(e) => {
tracing::warn!(error = ?e, path = %path.display(), "failed to reclaim an abandoned upload")
}
}
}
}
/// Allowlist of accepted media types, keyed by the MIME that `infer` derives from
/// the file's magic bytes. The detected MIME (not the client-declared one) is what
/// we trust, store, and hand to the compression pipeline — so a text-based payload
@@ -107,6 +236,11 @@ pub async fn upload(
.media_path
.join(format!("originals/{event_slug}"));
let temp_abs = originals_dir.join(format!("{upload_id}.tmp"));
// Armed before anything can create the file, so there is no window in which bytes exist
// unowned. From here on, EVERY exit — return, `?`, panic, or the future being dropped
// mid-body by a client disconnect — reclaims them, and the explicit `remove_file` calls
// that used to be sprinkled over the return paths are gone. One owner, one rule.
let mut file_guard = TempFileGuard::new(temp_abs.clone());
let mut streamed: Option<(i64, Vec<u8>)> = None; // (size, head bytes for sniffing)
let mut caption: Option<String> = None;
@@ -115,8 +249,8 @@ pub async fn upload(
// doesn't send one and gets the previous behaviour.
let mut client_upload_id: Option<Uuid> = None;
// Wrap the multipart read so any error after the temp file is created still cleans
// it up (a mid-stream parse failure must not leave a stray `.tmp` on disk).
// The multipart read is wrapped so the field loop can use `?` freely; reclaiming the temp
// file on failure is `file_guard`'s job, not this block's.
let parse_result: Result<(), AppError> = async {
while let Some(field) = multipart
.next_field()
@@ -146,20 +280,10 @@ pub async fn upload(
streamed = Some(stream_field_to_file(field, &temp_abs, cap_bytes).await?);
}
"caption" => {
caption = Some(
field
.text()
.await
.map_err(|e| AppError::BadRequest(e.to_string()))?,
);
caption = Some(read_text_field_bounded(field, MAX_CAPTION_BYTES).await?);
}
"hashtags" => {
hashtags_csv = Some(
field
.text()
.await
.map_err(|e| AppError::BadRequest(e.to_string()))?,
);
hashtags_csv = Some(read_text_field_bounded(field, MAX_HASHTAGS_BYTES).await?);
}
"client_upload_id" => {
let raw = field
@@ -178,10 +302,7 @@ pub async fn upload(
}
.await;
if let Err(e) = parse_result {
let _ = tokio::fs::remove_file(&temp_abs).await;
return Err(e);
}
parse_result?;
// Idempotency, fast path: this key already has a live upload, so the previous attempt DID
// succeed and only its response was lost. Replay that response instead of storing the photo
@@ -192,12 +313,14 @@ pub async fn upload(
// cost and it has already been paid by the time we get here; what has to be prevented is a
// second ROW, and that is what this does. The concurrent case (two retries in flight at once)
// is caught by the unique index inside the transaction below.
//
// The temp file needs no explicit cleanup here: `file_guard` is armed and reclaims it when
// this early return drops it.
if let Some(cid) = client_upload_id
&& let Some(existing) = Upload::find_by_client_upload_id(&state.pool, auth.user_id, cid)
.await
.map_err(AppError::from)?
{
let _ = tokio::fs::remove_file(&temp_abs).await;
tracing::info!(
client_upload_id = %cid, upload_id = %existing.id,
"duplicate upload suppressed; replaying the original response"
@@ -206,8 +329,8 @@ pub async fn upload(
return Ok((StatusCode::OK, Json(dto)));
}
// From here on the temp file may exist; every validation failure removes it before
// returning so a rejected upload never leaves bytes behind.
// From here on the temp file may exist. Every exit reclaims it via `file_guard` — see
// TempFileGuard for why the explicit per-branch cleanup this replaced was not enough.
let (size, head) = match streamed {
Some(s) => s,
None => return Err(AppError::BadRequest("Keine Datei hochgeladen.".into())),
@@ -219,7 +342,6 @@ pub async fn upload(
if let Some(ref cap) = caption
&& cap.chars().count() > MAX_CAPTION_LENGTH
{
let _ = tokio::fs::remove_file(&temp_abs).await;
return Err(AppError::BadRequest(format!(
"Beschreibung ist zu lang. Maximum: {} Zeichen.",
MAX_CAPTION_LENGTH
@@ -234,7 +356,6 @@ pub async fn upload(
let kind = match infer::get(&head) {
Some(k) => k,
None => {
let _ = tokio::fs::remove_file(&temp_abs).await;
return Err(AppError::BadRequest(
"Dateityp nicht erkannt oder nicht unterstützt.".into(),
));
@@ -247,7 +368,6 @@ pub async fn upload(
{
Some(v) => v,
None => {
let _ = tokio::fs::remove_file(&temp_abs).await;
return Err(AppError::BadRequest(format!(
"Dateityp wird nicht unterstützt: {}.",
kind.mime_type()
@@ -262,7 +382,6 @@ pub async fn upload(
max_image_mb * 1024 * 1024
};
if size > max_bytes {
let _ = tokio::fs::remove_file(&temp_abs).await;
return Err(AppError::BadRequest(format!(
"Datei ist zu groß. Maximum: {} MB.",
max_bytes / (1024 * 1024)
@@ -331,7 +450,6 @@ pub async fn upload(
quota_limit = Some(limit);
let prospective_total = user.total_upload_bytes.saturating_add(size);
if prospective_total > limit {
let _ = tokio::fs::remove_file(&temp_abs).await;
return Err(AppError::QuotaExceeded(
// Name the remedy, because the guest cannot see the number. Every quota
// display is staff-gated by design, so a guest hitting this had no idea what
@@ -352,22 +470,20 @@ pub async fn upload(
tokio::fs::rename(&temp_abs, &absolute_path)
.await
.map_err(|e| AppError::Internal(e.into()))?;
// THERE MUST BE NO `.await` BETWEEN THE RENAME AND THIS LINE. Both statements resolve on
// the same poll, so the future cannot be dropped between them and the guard is never
// pointing at a path that no longer holds the bytes. If the rename fails the guard still
// owns `temp_abs`, which is why retargeting comes after it rather than before.
file_guard.retarget(absolute_path.clone());
// Process hashtags from caption and explicit CSV
let mut tags: Vec<String> = Vec::new();
if let Some(ref cap) = caption {
tags.extend(hashtag::extract_hashtags(cap));
}
if let Some(ref csv) = hashtags_csv {
for tag in csv.split(',') {
let t = tag.trim().trim_start_matches('#').to_lowercase();
if !t.is_empty() {
tags.push(t);
}
}
}
tags.sort();
tags.dedup();
// Process hashtags from caption and explicit CSV, capped — see `normalize_tags`.
let tags = normalize_tags(
caption
.as_deref()
.map(hashtag::extract_hashtags)
.unwrap_or_default(),
hashtags_csv.as_deref(),
);
// Quota accounting, the upload row, and its hashtag links must be atomic: a
// crash between the bytes increment and the insert would permanently charge
@@ -466,14 +582,17 @@ pub async fn upload(
}
.await;
// The file is already on disk at `absolute_path`. If the transaction failed, no DB
// row will ever reference it, so remove it now rather than orphan bytes on disk.
// The file is already on disk at `absolute_path`, and `file_guard` was retargeted to it
// above — so every path out of here that is NOT a successful commit leaves the guard armed
// and reclaims the bytes on the way out. That covers the concurrent-duplicate loser below
// as well as the plain error case, and unlike the explicit `remove_file` calls it replaces,
// it also covers axum dropping this future instead of returning.
let upload = match tx_result {
Ok(u) => u,
// The concurrent duplicate resolved inside the transaction. The winner's row is committed;
// answer with it so both retries of the same photo get the same successful reply.
// answer with it so both retries of the same photo get the same successful reply. The
// loser's bytes are reclaimed by the guard when this return drops it.
Err(AppError::Conflict(ref marker)) if marker == DUPLICATE_UPLOAD_MARKER => {
let _ = tokio::fs::remove_file(&absolute_path).await;
let existing = match client_upload_id {
Some(cid) => Upload::find_by_client_upload_id(&state.pool, auth.user_id, cid)
.await
@@ -492,11 +611,10 @@ pub async fn upload(
let dto = replay_upload_dto(&state, &existing, &user.display_name).await;
return Ok((StatusCode::OK, Json(dto)));
}
Err(e) => {
let _ = tokio::fs::remove_file(&absolute_path).await;
return Err(e);
}
Err(e) => return Err(e),
};
// The committed row now references these bytes — hand ownership over.
file_guard.disarm();
// Spawn compression task
state
@@ -551,6 +669,57 @@ pub async fn edit_upload(
return Err(AppError::Forbidden("Nur eigene Uploads bearbeiten.".into()));
}
// This endpoint had no rate limit of any kind, while every other mutating route has one.
let rate_limits_on = config::get_bool(&state.config_cache, "rate_limits_enabled", true).await;
let edit_rate_on = config::get_bool(&state.config_cache, "upload_edit_rate_enabled", true).await;
if rate_limits_on && edit_rate_on {
let edit_rate =
config::get_i64(&state.config_cache, "upload_edit_rate_per_min", 30).await as usize;
if let Err(retry_after_secs) = state.rate_limiter.check_with_retry(
format!("upload_edit:{}", auth.user_id),
edit_rate,
Duration::from_secs(60),
) {
return Err(AppError::TooManyRequests(
"Zu viele Änderungen. Bitte warte kurz.".into(),
Some(retry_after_secs),
));
}
}
// Validate to the same limits as the upload path. This route had none at all, so a caption
// rejected at upload could be set here instead, and the tags went in raw — meaning `#Party`
// via edit and `party` via upload became two different hashtag rows.
if let Some(ref caption) = body.caption
&& caption.chars().count() > MAX_CAPTION_LENGTH
{
return Err(AppError::BadRequest(format!(
"Beschreibung ist zu lang. Maximum: {MAX_CAPTION_LENGTH} Zeichen."
)));
}
let normalized_tags = body
.hashtags
.as_ref()
.map(|tags| normalize_tags(tags.clone(), None));
// A PATCH that changes nothing must not retire the keepsake generation.
//
// `invalidate_and_arm` below ran unconditionally, outside both `if let Some(...)` guards, so
// `PATCH {}` — which any authenticated guest can send in a loop against their own upload —
// bumped export_epoch and armed a fresh pair of full-gallery export workers every time.
// REGEN_DEBOUNCE bounds the rate of that, not the total work, so the keepsake could be kept
// permanently un-downloadable.
//
// Residual, deliberately not fixed: re-sending an IDENTICAL hashtag list still counts as a
// change. Comparing would need another query, and unlike `PATCH {}` it is not a free loop.
let caption_changed = match (&body.caption, &upload.caption) {
(Some(new), existing) => Some(new.as_str()) != existing.as_deref(),
(None, _) => false,
};
if !caption_changed && normalized_tags.is_none() {
return Ok(StatusCode::OK);
}
// Caption update + hashtag wipe-then-relink in one transaction, so a crash
// mid-relink can't leave the upload with its hashtags stripped.
//
@@ -566,7 +735,7 @@ pub async fn edit_upload(
if let Some(ref caption) = body.caption {
Upload::update_caption(&mut *tx, upload_id, Some(caption)).await?;
}
if let Some(ref hashtags) = body.hashtags {
if let Some(ref hashtags) = normalized_tags {
Hashtag::unlink_all_from_upload(&mut *tx, upload_id).await?;
// Sort + dedup before upserting, exactly as the upload path does. `Hashtag::upsert`
// takes row locks, so two transactions touching the same two tags in OPPOSITE order
@@ -1251,4 +1420,103 @@ mod tests {
fn full_tolerance_is_identity_for_a_single_uploader() {
assert_eq!(quota_limit_bytes(500, 1.0, 1), 500);
}
/// The guard is the only reclaim mechanism that survives a client disconnect, so its three
/// states have to be exactly right — a wrong `disarm` leaks bytes forever, a wrong `Drop`
/// deletes a committed guest's photo.
mod temp_file_guard {
use super::super::TempFileGuard;
fn scratch(name: &str) -> std::path::PathBuf {
let dir = std::env::temp_dir().join(format!("es-guard-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let p = dir.join(name);
std::fs::write(&p, b"bytes").unwrap();
p
}
#[test]
fn dropping_an_armed_guard_reclaims_the_file() {
let p = scratch("armed.tmp");
drop(TempFileGuard::new(p.clone()));
assert!(!p.exists(), "an abandoned upload must not survive the request");
}
#[test]
fn a_disarmed_guard_leaves_the_file_alone() {
let p = scratch("committed.jpg");
let mut g = TempFileGuard::new(p.clone());
g.disarm();
drop(g);
assert!(p.exists(), "a committed upload's bytes must never be deleted");
}
#[test]
fn retarget_follows_the_rename_and_forgets_the_old_path() {
let old = scratch("old.tmp");
let new = scratch("new.jpg");
let mut g = TempFileGuard::new(old.clone());
// The rename already moved the bytes; only the new path is at risk now.
std::fs::remove_file(&old).unwrap();
g.retarget(new.clone());
drop(g);
assert!(!new.exists(), "the final-named original is orphaned too until the row commits");
}
#[test]
fn a_guard_whose_file_is_already_gone_is_harmless() {
let p = scratch("vanished.tmp");
std::fs::remove_file(&p).unwrap();
drop(TempFileGuard::new(p)); // must not panic
}
}
/// The CSV hashtag path had no length check at all and was upserted tag-by-tag INSIDE the
/// commit transaction — which holds a FOR SHARE lock on the event row, so one request could
/// stall every other upload behind tens of thousands of round trips.
mod hashtag_caps {
use super::super::{MAX_HASHTAGS_PER_UPLOAD, MAX_HASHTAG_LENGTH, normalize_tags};
#[test]
fn a_huge_csv_is_capped_not_upserted_in_full() {
let csv = (0..10_000)
.map(|i| format!("tag{i}"))
.collect::<Vec<_>>()
.join(",");
let tags = normalize_tags(vec![], Some(&csv));
assert_eq!(tags.len(), MAX_HASHTAGS_PER_UPLOAD);
}
#[test]
fn an_overlong_tag_is_dropped_rather_than_stored() {
let long = "a".repeat(MAX_HASHTAG_LENGTH + 1);
let tags = normalize_tags(vec![], Some(&format!("ok,{long}")));
assert_eq!(tags, vec!["ok"]);
}
#[test]
fn tags_are_normalised_and_deduped_across_both_sources() {
// The upload path lowercased and stripped `#` while the edit path did not, so
// `#Party` and `party` became two different hashtag rows. One helper, one rule.
let tags = normalize_tags(vec!["party".into()], Some("#Party, PARTY ,tanz"));
assert_eq!(tags, vec!["party", "tanz"]);
}
#[test]
fn truncation_keeps_a_stable_prefix_not_an_arbitrary_one() {
// Sorted before truncation, so the same input always yields the same tags —
// otherwise an edit could silently shuffle which 30 survived.
let csv = "zulu,alpha,mike,bravo";
assert_eq!(
normalize_tags(vec![], Some(csv)),
normalize_tags(vec![], Some("bravo,mike,alpha,zulu"))
);
}
#[test]
fn an_empty_or_absent_csv_yields_nothing() {
assert!(normalize_tags(vec![], None).is_empty());
assert!(normalize_tags(vec![], Some(",, ,")).is_empty());
}
}
}