fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(export): single epoch authority; fix upload-path TOCTOU losing photos

Browse Source 
A deep investigative review of the export concurrency (four parallel adversarial
reviewers) found that three rounds of race fixes had been chasing the wrong bug, and
that the model itself was the root cause. This replaces the model and fixes the real
data loss.

THE ACTUAL BUG (upload path, not the state machine)
  `upload` checked `uploads_locked_at` BEFORE streaming the body — minutes, for a large
  video — then committed the row without re-checking. A release landing mid-upload let the
  row commit AFTER the export snapshot: the photo appeared in the live feed and was
  PERMANENTLY missing from the keepsake, with nothing to regenerate it. release_gallery's
  comment claims to prevent exactly this; it never did. Every prior round fixed the DB
  state machine, and the leak was upstream of it.
  Fix: re-assert the lock under `SELECT ... FOR SHARE` INSIDE the commit tx. That row lock
  conflicts with release_gallery's `UPDATE event`, so either the upload commits first (and
  the release — hence the snapshot — is ordered after it) or the release wins and the
  upload is rejected as `uploads_locked` (reversible; the client keeps the blob).
  Regression test verified NON-VACUOUS: with the fix reverted it fails, reporting accepted
  uploads missing from the keepsake.

THE MODEL (migration 014)
  "Which generation is current?" had no single answer: it was assembled from three
  separately-written pieces across two tables (`export_released_at`, a per-job
  `release_seq`, and cached `export_{zip,html}_ready` flags). Keeping them in agreement
  took ~10 hand-written guards; each review round found one more path that slipped through.
  Now: ONE monotonic `event.export_epoch`, bumped in the same UPDATE as any change to
  `export_released_at`. Each job carries a copy. Downloadable IFF
  `released AND job.epoch = event.export_epoch AND status = 'done'` — readiness DERIVED,
  never stored. A retired-epoch worker is INERT BY CONSTRUCTION: losing a race can no
  longer corrupt state, only waste work.
  Deleted: both ready-flag columns, both ready-flip blocks, `if ready { continue }`, the
  reopen seq-bump, open_event's transaction, and the cross-table claim guard.

  That last one was also UNSOUND: under READ COMMITTED, a blocked UPDATE re-evaluates its
  WHERE against the updated target row but answers subqueries on OTHER tables from the
  original snapshot — so the previous `EXISTS (SELECT ... FROM event ...)` claim guard
  could admit a claim against an already-reopened event while RETURNING the post-bump seq.
  Every guard is now same-row (`epoch = $n`), which EPQ re-evaluates correctly.

OTHER REAL DEFECTS FIXED
  - release_gallery committed the release and THEN awaited the enqueue inside the handler
    future. Axum drops that future on client disconnect → event released, uploads locked,
    ZERO export_job rows, host unable to retry ("bereits freigegeben"), restart-only
    recovery. Now one tx; workers spawn (detached) after commit.
  - No flush/fsync before the tmp→final rename. async_zip's close() never flushes the inner
    file and tokio's File drop DISCARDS write errors — so a full disk silently produced a
    truncated archive that was then marked done and published, after which the last good
    generation was pruned. Now flush + sync_all, which also surfaces ENOSPC.
  - Download read the ready flag and the file path in TWO queries; a reopen between them
    served a keepsake that should 404. Now one read through the `export_current` view.
  - `if !src.exists() { continue }` then `open()?` — a TOCTOU that turned a tolerated gap
    into a hard failure of the ENTIRE keepsake (unretryable while released). Now open-first,
    warn, and skip.
  - Recovery was flag-driven and never checked the disk: a `done` row whose archive was gone
    was a permanent dead end needing manual DB surgery. Now verifies the file and re-arms.
  - Temp files/dirs leaked on every error path (the leak is what fills the disk that
    corrupts the next archive). Now cleaned up.
  - Export archives were not event-scoped (`viewer_tmp_` already was), so a second event
    would collide on paths and one event's prune would delete another's live keepsake.
  - Host deletes after release never regenerated the keepsake — a photo removed on request
    lived on in the archive forever. Now bumps the epoch and rebuilds without it.
  - claim_job swallowed DB errors as "someone else won", stranding a job pending at 0%.
  - export_status reported retired-epoch rows (a progress bar that never moves).
  - The startup sweep's single-instance assumption is now documented, not hidden.

Verified: backend 40 tests, svelte-check 0 errors, e2e typecheck clean,
e2e 160 passed / 1 skipped on chromium-desktop (incl. 2 new regressions; the
upload-acceptance one confirmed to fail without the fix).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
fabi
2026-07-14 19:48:38 +02:00
parent 1148c2e906
commit 8e906d7866
12 changed files with 802 additions and 430 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
backend/src/handlers/admin.rs
View File

@@ -288,32 +288,28 @@ pub async fn download_zip(
authenticate_download_ticket(&state, &q.ticket).await?;
enforce_export_rate(&state, &headers).await?;
let event = crate::models::event::Event::find_by_slug(&state.pool, &state.config.event_slug)
.await?
.ok_or_else(|| AppError::NotFound("Event nicht gefunden.".into()))?;
if !event.export_zip_ready {
return Err(AppError::NotFound(
"Der ZIP-Export ist noch nicht verfügbar.".into(),
));
}
let path = resolve_export_file(&state, "zip").await?;
let path = resolve_export_file(&state, "zip", "Der ZIP-Export ist noch nicht verfügbar.").await?;
serve_file(path, "Gallery.zip", "application/zip").await
}
/// Resolve the on-disk path of the CURRENT export generation from `export_job.file_path`
/// rather than a fixed filename. Exports are written to per-generation paths
/// (`Gallery.<seq>.zip`) so a superseded worker can't stomp a fresh keepsake (H1); the
/// download must therefore follow whatever the current 'done' row points at, not guess.
/// Resolve the on-disk path of the CURRENT export generation — readiness check and path lookup in
/// ONE read, through the `export_current` view (migration 014).
///
/// This used to be two statements: the caller checked `event.export_zip_ready`, then this fetched
/// `export_job.file_path`. A reopen landing between them served a keepsake that should have 404'd —
/// the same stale-keepsake class, leaking through the read path, and it existed only because
/// readiness was a stored copy rather than a derivation. `export_current` gives both facts from one
/// consistent snapshot: it yields a row only when the event is released AND the job is `done` at the
/// event's current epoch, so "is it ready" and "which file" can no longer disagree.
async fn resolve_export_file(
state: &AppState,
export_type: &str,
not_ready_msg: &str,
) -> Result<std::path::PathBuf, AppError> {
let file_path: Option<(Option<String>,)> = sqlx::query_as(
"SELECT file_path FROM export_job ej
JOIN event e ON e.id = ej.event_id
WHERE e.slug = $1 AND ej.type = $2::export_type AND ej.status = 'done'",
"SELECT c.file_path FROM export_current c
JOIN event e ON e.id = c.event_id
WHERE e.slug = $1 AND c.type = $2::export_type AND c.status = 'done'",
)
.bind(&state.config.event_slug)
.bind(export_type)
@@ -321,11 +317,12 @@ async fn resolve_export_file(
.await
.map_err(|e| AppError::Internal(e.into()))?;
let Some((Some(rel),)) = file_path else {
return Err(AppError::NotFound(not_ready_msg.into()));
};
// `file_path` is stored as `exports/<name>`; the base dir is already `export_path`, so
// join only the file name (defends against any absolute/`..` content too).
let rel = file_path
.and_then(|(p,)| p)
.ok_or_else(|| AppError::NotFound("Exportdatei nicht gefunden.".into()))?;
let name = std::path::Path::new(&rel)
.file_name()
.ok_or_else(|| AppError::NotFound("Exportdatei nicht gefunden.".into()))?;
@@ -344,17 +341,8 @@ pub async fn download_html(
authenticate_download_ticket(&state, &q.ticket).await?;
enforce_export_rate(&state, &headers).await?;
let event = crate::models::event::Event::find_by_slug(&state.pool, &state.config.event_slug)
.await?
.ok_or_else(|| AppError::NotFound("Event nicht gefunden.".into()))?;
if !event.export_html_ready {
return Err(AppError::NotFound(
"Der HTML-Export ist noch nicht verfügbar.".into(),
));
}
let path = resolve_export_file(&state, "html").await?;
let path =
resolve_export_file(&state, "html", "Der HTML-Export ist noch nicht verfügbar.").await?;
serve_file(path, "Memories.zip", "application/zip").await
}
@@ -400,10 +388,16 @@ pub async fn export_status(
let released = event.export_released_at.is_some();
// Only report jobs at the CURRENT epoch. A row left behind by a retired generation (a worker
// that was superseded mid-run) is meaningless — surfacing its frozen `running`/77% would show
// the host a progress bar that never moves for a keepsake nobody is building. A retired row
// reads as "locked" (no current job), which is exactly what it is.
let jobs: Vec<(String, String, i16)> = sqlx::query_as(
"SELECT type::text, status::text, progress_pct FROM export_job WHERE event_id = $1",
"SELECT type::text, status::text, progress_pct FROM export_job
WHERE event_id = $1 AND epoch = $2",
)
.bind(event.id)
.bind(event.export_epoch)
.fetch_all(&state.pool)
.await?;