fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(export): single epoch authority; fix upload-path TOCTOU losing photos

Browse Source 
A deep investigative review of the export concurrency (four parallel adversarial
reviewers) found that three rounds of race fixes had been chasing the wrong bug, and
that the model itself was the root cause. This replaces the model and fixes the real
data loss.

THE ACTUAL BUG (upload path, not the state machine)
  `upload` checked `uploads_locked_at` BEFORE streaming the body — minutes, for a large
  video — then committed the row without re-checking. A release landing mid-upload let the
  row commit AFTER the export snapshot: the photo appeared in the live feed and was
  PERMANENTLY missing from the keepsake, with nothing to regenerate it. release_gallery's
  comment claims to prevent exactly this; it never did. Every prior round fixed the DB
  state machine, and the leak was upstream of it.
  Fix: re-assert the lock under `SELECT ... FOR SHARE` INSIDE the commit tx. That row lock
  conflicts with release_gallery's `UPDATE event`, so either the upload commits first (and
  the release — hence the snapshot — is ordered after it) or the release wins and the
  upload is rejected as `uploads_locked` (reversible; the client keeps the blob).
  Regression test verified NON-VACUOUS: with the fix reverted it fails, reporting accepted
  uploads missing from the keepsake.

THE MODEL (migration 014)
  "Which generation is current?" had no single answer: it was assembled from three
  separately-written pieces across two tables (`export_released_at`, a per-job
  `release_seq`, and cached `export_{zip,html}_ready` flags). Keeping them in agreement
  took ~10 hand-written guards; each review round found one more path that slipped through.
  Now: ONE monotonic `event.export_epoch`, bumped in the same UPDATE as any change to
  `export_released_at`. Each job carries a copy. Downloadable IFF
  `released AND job.epoch = event.export_epoch AND status = 'done'` — readiness DERIVED,
  never stored. A retired-epoch worker is INERT BY CONSTRUCTION: losing a race can no
  longer corrupt state, only waste work.
  Deleted: both ready-flag columns, both ready-flip blocks, `if ready { continue }`, the
  reopen seq-bump, open_event's transaction, and the cross-table claim guard.

  That last one was also UNSOUND: under READ COMMITTED, a blocked UPDATE re-evaluates its
  WHERE against the updated target row but answers subqueries on OTHER tables from the
  original snapshot — so the previous `EXISTS (SELECT ... FROM event ...)` claim guard
  could admit a claim against an already-reopened event while RETURNING the post-bump seq.
  Every guard is now same-row (`epoch = $n`), which EPQ re-evaluates correctly.

OTHER REAL DEFECTS FIXED
  - release_gallery committed the release and THEN awaited the enqueue inside the handler
    future. Axum drops that future on client disconnect → event released, uploads locked,
    ZERO export_job rows, host unable to retry ("bereits freigegeben"), restart-only
    recovery. Now one tx; workers spawn (detached) after commit.
  - No flush/fsync before the tmp→final rename. async_zip's close() never flushes the inner
    file and tokio's File drop DISCARDS write errors — so a full disk silently produced a
    truncated archive that was then marked done and published, after which the last good
    generation was pruned. Now flush + sync_all, which also surfaces ENOSPC.
  - Download read the ready flag and the file path in TWO queries; a reopen between them
    served a keepsake that should 404. Now one read through the `export_current` view.
  - `if !src.exists() { continue }` then `open()?` — a TOCTOU that turned a tolerated gap
    into a hard failure of the ENTIRE keepsake (unretryable while released). Now open-first,
    warn, and skip.
  - Recovery was flag-driven and never checked the disk: a `done` row whose archive was gone
    was a permanent dead end needing manual DB surgery. Now verifies the file and re-arms.
  - Temp files/dirs leaked on every error path (the leak is what fills the disk that
    corrupts the next archive). Now cleaned up.
  - Export archives were not event-scoped (`viewer_tmp_` already was), so a second event
    would collide on paths and one event's prune would delete another's live keepsake.
  - Host deletes after release never regenerated the keepsake — a photo removed on request
    lived on in the archive forever. Now bumps the epoch and rebuilds without it.
  - claim_job swallowed DB errors as "someone else won", stranding a job pending at 0%.
  - export_status reported retired-epoch rows (a progress bar that never moves).
  - The startup sweep's single-instance assumption is now documented, not hidden.

Verified: backend 40 tests, svelte-check 0 errors, e2e typecheck clean,
e2e 160 passed / 1 skipped on chromium-desktop (incl. 2 new regressions; the
upload-acceptance one confirmed to fail without the fix).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
fabi
2026-07-14 19:48:38 +02:00
parent 1148c2e906
commit 8e906d7866
12 changed files with 802 additions and 430 deletions
Download Patch File Download Diff File Expand all files Collapse all files

169
backend/src/handlers/host.rs
View File

@@ -435,6 +435,48 @@ pub async fn dismiss_pin_reset_request(
Ok(StatusCode::NO_CONTENT)
}
/// Content changed AFTER the gallery was released — regenerate the keepsake.
///
/// Deleting a photo used to remove it from the live feed but leave it in the already-generated
/// archive FOREVER: the old `if ready { continue }` skip guaranteed the export was never rebuilt.
/// For a takedown ("please remove my photo") that is the one place it most needs to disappear.
///
/// Bumping the epoch (while staying released) retires the current generation, so the stale archive
/// stops being downloadable the instant the delete commits, and a fresh worker rebuilds it without
/// the removed content. The download 404s in the meantime, which is the correct answer — serving
/// the old archive would serve the deleted photo.
async fn regenerate_keepsake_if_released(state: &AppState) -> Result<(), AppError> {
let mut tx = state.pool.begin().await?;
let bumped: Option<(Uuid, String, i64)> = sqlx::query_as(
"UPDATE event SET export_epoch = export_epoch + 1
WHERE slug = $1 AND export_released_at IS NOT NULL
RETURNING id, name, export_epoch",
)
.bind(&state.config.event_slug)
.fetch_optional(&mut *tx)
.await?;
// Not released → nothing to regenerate; the export will be built fresh at release time.
let Some((event_id, event_name, epoch)) = bumped else {
return Ok(());
};
crate::services::export::enqueue_jobs_at_epoch(&mut tx, event_id, epoch).await?;
tx.commit().await?;
crate::services::export::spawn_export_jobs(
event_id,
event_name,
epoch,
state.pool.clone(),
state.config.media_path.clone(),
state.config.export_path.clone(),
state.sse_tx.clone(),
);
Ok(())
}
pub async fn host_delete_upload(
State(state): State<AppState>,
RequireHost(auth): RequireHost,
@@ -454,6 +496,9 @@ pub async fn host_delete_upload(
serde_json::json!({ "upload_id": upload.id }).to_string(),
));
// A released keepsake must not keep serving a photo the host just took down.
regenerate_keepsake_if_released(&state).await?;
tracing::info!(
actor_user_id = %auth.user_id,
event_id = %auth.event_id,
@@ -478,6 +523,8 @@ pub async fn host_delete_comment(
"comment-deleted",
serde_json::json!({ "comment_id": comment_id }).to_string(),
));
// The HTML viewer embeds comments — rebuild it so a deleted comment doesn't live on there.
regenerate_keepsake_if_released(&state).await?;
tracing::info!(
actor_user_id = %auth.user_id,
event_id = %auth.event_id,
@@ -511,57 +558,27 @@ pub async fn open_event(
State(state): State<AppState>,
RequireHost(_auth): RequireHost,
) -> Result<StatusCode, AppError> {
// Reopening also invalidates any prior release: the keepsake was snapshotted at
// release time, so allowing new uploads afterwards would silently diverge the live
// feed from the frozen export. Clearing `export_released_at` (and the readiness
// flags) lets the host re-release later to regenerate a correct, complete keepsake.
// Reopening invalidates any prior release: the keepsake was snapshotted at release time, so
// allowing new uploads afterwards would silently diverge the live feed from the frozen export.
//
// BOTH statements run in ONE transaction. They must be atomic: the first makes
// `release_gallery` possible again (it gates on `export_released_at IS NULL`), so if a
// concurrent re-release could slip between them it would enqueue + spawn a FRESH worker at
// seq N+1, and our second statement would then bump that brand-new generation to N+2 —
// orphaning the live worker (its seq-guarded finalize/fail/flip all match nothing), stranding
// the row at `running` with no owner, and leaving the host unable to retry ("bereits
// freigegeben"). Inside a transaction, our row lock on `event` serializes `release_gallery`'s
// claim behind our commit, so it only ever sees the fully-reopened, fully-bumped state.
let mut tx = state.pool.begin().await?;
// ONE statement retires the entire export generation. Bumping `export_epoch` in the same write
// that clears `export_released_at` instantly invalidates every in-flight worker, every `done`
// row and all readiness — because readiness is DERIVED from this epoch (migration 014), not
// stored. There is no export_job row to touch and nothing to keep in sync, so this needs no
// transaction. Any worker still streaming holds the old epoch and is now inert: its
// epoch-guarded finalize matches nothing, and it discards its own output.
let result = sqlx::query(
"UPDATE event
SET uploads_locked_at = NULL,
SET uploads_locked_at = NULL,
export_released_at = NULL,
export_zip_ready = FALSE,
export_html_ready = FALSE
export_epoch = export_epoch + 1
WHERE slug = $1 AND (uploads_locked_at IS NOT NULL OR export_released_at IS NOT NULL)",
)
.bind(&state.config.event_slug)
.execute(&mut *tx)
.execute(&state.pool)
.await?;
let reopened = result.rows_affected() > 0;
if reopened {
// A reopen invalidates any released keepsake, so make it a *supersession point* for the
// export pipeline: bump every export_job generation for this event. An in-flight worker
// — or one that has already `finalize_job`'d but not yet flipped its ready flag — captured
// the pre-reopen `release_seq`, so its guarded finalize/ready-flip now match nothing and
// become no-ops. Without this, the `export_released_at IS NOT NULL` flip guard alone leaves
// a sub-window open: a re-release re-arms `export_released_at` *before* it bumps the seq, and
// a stale worker's flip landing in that gap satisfies both guards and resurrects a
// pre-reopen keepsake (silent data loss). Bumping here retires the old seq for good.
sqlx::query(
"UPDATE export_job SET release_seq = release_seq + 1
FROM event e
WHERE e.id = export_job.event_id AND e.slug = $1",
)
.bind(&state.config.event_slug)
.execute(&mut *tx)
.await?;
}
tx.commit().await?;
if reopened {
if result.rows_affected() > 0 {
let _ = state.sse_tx.send(SseEvent::new("event-opened", "{}"));
}
@@ -572,24 +589,37 @@ pub async fn release_gallery(
State(state): State<AppState>,
RequireHost(_auth): RequireHost,
) -> Result<StatusCode, AppError> {
// Atomic claim: the conditional UPDATE is the sole gate, so two concurrent
// release calls can't both pass a check-then-set and double-enqueue exports.
// rows_affected == 0 means someone already released.
// The release claim, the epoch bump, the upload lock and BOTH job rows are written in ONE
// transaction. Two reasons, both of which were live bugs:
//
// Releasing also locks uploads in the same statement (release ⇒ lock). Otherwise a
// guest whose offline upload reconnects *after* the export snapshot would land in the
// live feed but not in the downloaded keepsake — a silent, non-regenerable data loss.
// `COALESCE` preserves an earlier explicit lock time rather than overwriting it.
let result = sqlx::query(
// 1. Cancellation. This handler used to commit `export_released_at` and only THEN await the
// enqueue. Axum drops the handler future when the client disconnects (closed tab, proxy
// timeout), which left the event released and uploads locked with ZERO export_job rows and
// no workers: downloads 404 forever and the host cannot retry, because release_gallery
// rejects an already-released event ("bereits freigegeben"). Only a restart escaped it.
// Now nothing is committed until every row is written, and the workers are spawned AFTER
// the commit (a detached `tokio::spawn` survives cancellation).
// 2. Atomicity vs. a concurrent reopen. Bumping the epoch in the same statement that sets
// `export_released_at` means no worker and no reader can ever observe "released again but
// the generation hasn't moved on yet" — the window every previous fix kept leaving open.
//
// Release also locks uploads in the same statement (release ⇒ lock), so the export snapshot is
// taken against a frozen upload set. `COALESCE` preserves an earlier explicit lock time.
let mut tx = state.pool.begin().await?;
let claimed: Option<(Uuid, String, i64)> = sqlx::query_as(
"UPDATE event
SET export_released_at = NOW(),
uploads_locked_at = COALESCE(uploads_locked_at, NOW())
WHERE slug = $1 AND export_released_at IS NULL",
uploads_locked_at = COALESCE(uploads_locked_at, NOW()),
export_epoch = export_epoch + 1
WHERE slug = $1 AND export_released_at IS NULL
RETURNING id, name, export_epoch",
)
.bind(&state.config.event_slug)
.execute(&state.pool)
.fetch_optional(&mut *tx)
.await?;
if result.rows_affected() == 0 {
let Some((event_id, event_name, epoch)) = claimed else {
// Distinguish "no such event" from "already released" for a clean error.
let exists = Event::find_by_slug(&state.pool, &state.config.event_slug)
.await?
@@ -599,28 +629,29 @@ pub async fn release_gallery(
} else {
AppError::NotFound("Event nicht gefunden.".into())
});
}
};
// Release locked uploads too — tell any open composer to flip to the locked UI live
// rather than discovering it via a rejected upload.
// Arm both types at THIS epoch. No "skip the type that's already ready" check any more: the
// epoch bump above retired every prior generation, so there is nothing to preserve and nothing
// to be fooled by. (That skip was how a stale ready flag used to suppress regeneration.)
crate::services::export::enqueue_jobs_at_epoch(&mut tx, event_id, epoch).await?;
tx.commit().await?;
// Release locks uploads too — tell any open composer to flip to the locked UI live rather than
// discovering it via a rejected upload.
let _ = state.sse_tx.send(SseEvent::new("event-closed", "{}"));
// We won the claim — load the event for its id/name to enqueue export jobs.
let event = Event::find_by_slug(&state.pool, &state.config.event_slug)
.await?
.ok_or_else(|| AppError::NotFound("Event nicht gefunden.".into()))?;
// Enqueue + spawn via the shared path so a re-release (after reopen) regenerates
// cleanly and startup recovery uses identical logic.
crate::services::export::enqueue_and_spawn_exports(
event.id,
event.name,
// Detached — survives this handler being cancelled.
crate::services::export::spawn_export_jobs(
event_id,
event_name,
epoch,
state.pool.clone(),
state.config.media_path.clone(),
state.config.export_path.clone(),
state.sse_tx.clone(),
)
.await?;
);
Ok(StatusCode::NO_CONTENT)
}