fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(export): refuse an export that cannot fit, and stop peaking at two generations

Browse Source 
Nothing in export.rs ever asked whether the keepsake would fit. Both archives write
their media `Compression::Stored`, so each is essentially a byte-for-byte second copy
of the originals -- Gallery.zip always, and Memories.zip for every video and every
image at or under 5 MB. On the documented CX33 (80 GB, all three volumes on one
filesystem) the upload quota's fixed point leaves ~40 GB free, and a release spawns
BOTH halves concurrently against it.

The failure is not "the export failed", it is "the deliverable is stuck":

  1. ENOSPC lands partway through a multi-GB write.
  2. The epoch has already moved, so the job row is `failed` at the CURRENT
     generation and readiness (epoch = event.export_epoch AND status = 'done') is
     false -- GET /export/zip 404s.
  3. The last good archive sits on disk, unreferenced and unreachable.
  4. POST /host/export/rebuild, the only escape, re-arms the same doomed write.

Three changes.

Reclaim before building. `prune_stale_export_files` ran only after the new archive
was written, renamed and finalised. That reads as durability but buys nothing: the
moment `invalidate_and_arm` bumps the epoch the old archive is ALREADY unreachable,
so keeping it reserves gigabytes for a download nobody can perform -- and for a
takedown it is content someone explicitly asked to have removed. Peak usage is now
one generation. Narrower than the post-finalize prune on purpose: final archives
only, never a `.tmp` or a `viewer_tmp_` dir, since a superseded worker can still be
streaming into those and at build START is far more likely to be alive.

Preflight the space. SUM(original_size_bytes) over exactly `query_uploads`'
visibility filter, +10% for ZIP overhead, multiplied by the number of armed jobs --
without that multiplier each of the two concurrent halves independently sees "it
fits" and together they don't. Runs AFTER claim_job, not before as reported: bailing
before the claim leaves the row `pending` with no worker and no error, the
spinner-forever state `mark_failed`'s status guard exists to prevent. Fails open when
the mount can't be read, exactly as the upload quota does.

Show the host the reason. /export/status returned {status, progress_pct} and nothing
else, so the host dashboard could only render "fehlgeschlagen" next to the retry
button. The message was written to the row and surfaced solely in the ADMIN job list
-- a different screen, possibly a different person. It now travels with the status,
and only on a failure, so a message left on a since-succeeded row can't appear beside
a green "ist bereit".

Tests: 10 unit (the u128 clamp caught a real bug in the first draft -- saturating_mul
then /100 turns an overflow into a number ~100x too small, the one direction that
authorises the write being guarded against; the carried-forward archive must survive
its own older epoch in the filename), 4 DB-backed (the estimate is asserted against
the row set the archive actually contains, not against a restatement of the WHERE
clause, so the two queries cannot drift), 3 e2e over the four-hop plumbing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
fabi
2026-07-29 19:38:26 +02:00
parent 8c93cbb045
commit 281eb3bec7
9 changed files with 705 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
backend/src/handlers/admin.rs
View File

@@ -456,8 +456,13 @@ pub async fn export_status(
// worker superseded mid-run) is meaningless — surfacing its frozen `running`/77% would show a
// progress bar that never moves for a keepsake nobody is building. It reads as "locked" (no
// current job), which is exactly what it is.
let jobs: Vec<(String, String, i16)> = sqlx::query_as(
"SELECT j.type::text, j.status::text, j.progress_pct
// `error_message` is carried here, not just on the admin dashboard's job list. The host is the
// one who releases the keepsake and the one who owns the "Erneut versuchen" button, but this
// endpoint used to hand them a bare `failed` — so a fully actionable reason (notably the disk
// preflight's "needs X GB, Y GB free") was written to the row and then shown to nobody who
// could act on it. An admin-only diagnostic is not a diagnostic for the person on the spot.
let jobs: Vec<(String, String, i16, Option<String>)> = sqlx::query_as(
"SELECT j.type::text, j.status::text, j.progress_pct, j.error_message
FROM export_job j
JOIN event e ON e.id = j.event_id
WHERE e.id = $1 AND j.epoch = e.export_epoch",
@@ -468,9 +473,21 @@ pub async fn export_status(
let job_status = |type_name: &str| {
jobs.iter()
.find(|(t, _, _)| t == type_name)
.map(|(_, status, pct)| serde_json::json!({ "status": status, "progress_pct": pct }))
.unwrap_or_else(|| serde_json::json!({ "status": "locked", "progress_pct": 0 }))
.find(|(t, _, _, _)| t == type_name)
.map(|(_, status, pct, err)| {
serde_json::json!({
"status": status,
"progress_pct": pct,
// Only on a failure. A stale message left on a row that has since been re-armed
// would otherwise show an error next to a running progress bar.
"error_message": if status == "failed" { err.clone() } else { None },
})
})
.unwrap_or_else(|| {
serde_json::json!({
"status": "locked", "progress_pct": 0, "error_message": null,
})
})
};
Ok(Json(serde_json::json!({

11
backend/src/services/disk.rs
View File

@@ -76,6 +76,17 @@ impl Default for DiskCache {
}
}
/// UNCACHED free-space reading for the filesystem backing `path`.
///
/// Deliberately bypasses [`DiskCache`]. The cache exists for the quota poll, where a 15s-stale
/// number is fine because it is only ever advisory. The export preflight is the opposite case: it
/// decides whether to start writing a multi-GB archive, and the sibling export worker running
/// concurrently can move free space by tens of gigabytes well inside the TTL. A stale reading there
/// would authorise exactly the write that fills the disk.
pub fn free_bytes(path: &Path) -> Option<u64> {
read_disk_for_path(path).map(|d| d.free)
}
/// Resolve the filesystem backing `media_path` and read its total/free bytes.
///
/// Snapshots the mount table via `sysinfo`, then delegates the selection to the pure

333
backend/src/services/export.rs
View File

@@ -476,6 +476,15 @@ async fn run_zip_export(
return Ok(());
}
// Reclaim BEFORE measuring: the superseded archive is already unreachable, and the space it
// holds is very often exactly the space this rebuild needs.
prune_superseded_archives(pool, export_path, "Gallery", event_id, epoch).await;
// AFTER the claim, not before. A preflight that bailed before claiming would leave the row
// `pending` with no worker and no error — the spinner-forever state `mark_failed`'s status
// guard was widened to prevent. Failing here goes through the caller's `mark_failed`, so the
// host gets the reason.
ensure_export_space(pool, event_id, export_path).await?;
// On error, mark THIS generation failed — a no-op if we've since been superseded (the
// caller in `spawn_export_jobs` does it, epoch-guarded). Temp artifacts are cleaned up
// here so a failing export can't leak them (which is what fills the disk in the first place).
@@ -658,6 +667,11 @@ async fn run_html_export(
return Ok(());
}
// See run_zip_export: reclaim the superseded generation first, then refuse at the door rather
// than ENOSPC mid-write.
prune_superseded_archives(pool, export_path, "Memories", event_id, epoch).await;
ensure_export_space(pool, event_id, export_path).await?;
let res = run_html_export_inner(
epoch,
event_id,
@@ -1161,6 +1175,182 @@ fn parse_gen_seq(name: &str, prefix: &str, suffix: &str) -> Option<i64> {
.ok()
}
/// Filenames a live (current-epoch, `done`) job row still points at — OFF LIMITS to every prune,
/// regardless of the epoch encoded in the name.
///
/// A ViewerOnly regeneration carries the finished ZIP forward by re-stamping its row to the new
/// epoch WITHOUT renaming the file, so `Gallery.<event>.<older>.zip` is still the served archive and
/// deleting it by filename-epoch would 404 the download.
async fn protected_files(pool: &PgPool, event_id: Uuid) -> Vec<String> {
sqlx::query_scalar::<_, String>(
"SELECT split_part(j.file_path, '/', -1) FROM export_job j
JOIN event e ON e.id = j.event_id
WHERE j.event_id = $1 AND j.epoch = e.export_epoch
AND j.status = 'done' AND j.file_path IS NOT NULL",
)
.bind(event_id)
.fetch_all(pool)
.await
.unwrap_or_default()
}
/// Reclaim superseded FINAL archives BEFORE this generation starts writing its own.
///
/// Peak disk usage used to be two full generations, because the only prune ran after the new archive
/// was written, renamed and finalised. That ordering reads as durability ("don't delete the good
/// keepsake before the replacement is safe") but it buys nothing: readiness is derived from
/// `job.epoch = event.export_epoch AND status = 'done'`, so the moment `invalidate_and_arm` bumps
/// the epoch the old archive is ALREADY unreachable — `GET /export/zip` 404s whether the file is on
/// disk or not. Keeping it only reserves gigabytes for a download nobody can perform, and for
/// `Affects::Both` (a takedown) it is content someone has explicitly asked to have removed. So a
/// rebuild reclaims first and peaks at one generation.
///
/// Narrower than [`prune_stale_export_files`] on purpose: FINAL archives only. Those are inert — a
/// superseded worker either already renamed its file (and will delete it itself when its guarded
/// `finalize_job` fails) or never will. `.tmp` files and `viewer_tmp_` staging dirs are NOT touched
/// here: a superseded worker can still be streaming into those, and at build START it is far more
/// likely to be alive than at finalize time.
async fn prune_superseded_archives(
pool: &PgPool,
exports_dir: &Path,
prefix: &str,
event_id: Uuid,
keep_seq: i64,
) {
let protected = protected_files(pool, event_id).await;
let final_prefix = format!("{prefix}.{event_id}.");
let mut rd = match tokio::fs::read_dir(exports_dir).await {
Ok(rd) => rd,
Err(_) => return,
};
let mut reclaimed = 0u64;
while let Ok(Some(entry)) = rd.next_entry().await {
let name = entry.file_name();
let name = name.to_string_lossy();
if !is_superseded_archive(&name, &final_prefix, keep_seq, &protected) {
continue;
}
let len = entry.metadata().await.map(|m| m.len()).unwrap_or(0);
if tokio::fs::remove_file(entry.path()).await.is_ok() {
reclaimed += len;
}
}
if reclaimed > 0 {
tracing::info!(
"reclaimed {reclaimed} bytes of superseded {prefix} archives before rebuilding \
event {event_id} @ epoch {keep_seq}"
);
}
}
/// Is `name` a FINAL archive of a strictly-older generation that no live row points at?
///
/// Pure so the two dangerous cases can be pinned without a filesystem: the carried-forward archive
/// (protected despite an older epoch in its name) and the in-flight `.tmp` (never matched at all).
fn is_superseded_archive(
name: &str,
final_prefix: &str,
keep_seq: i64,
protected: &[String],
) -> bool {
if protected.iter().any(|p| p == name) {
return false;
}
parse_gen_seq(name, final_prefix, ".zip").is_some_and(|n| n < keep_seq)
}
/// Bytes the media in this event's keepsake will occupy, as an UPPER BOUND per archive.
///
/// Both archives write their media entries `Compression::Stored`, so an archive is essentially a
/// byte-for-byte second copy of the originals: `Gallery.zip` always, and `Memories.zip` for every
/// video ([`MediaSource::Original`]) and every image at or under 5 MB. Images over 5 MB are
/// downscaled to 2000px first, which only makes this estimate more conservative — the direction we
/// want, since being wrong low means ENOSPC halfway through.
///
/// Matches [`query_uploads`]' visibility filter exactly, so hidden/banned uploads aren't counted.
async fn estimate_export_bytes(pool: &PgPool, event_id: Uuid) -> Result<u64> {
let (bytes,): (i64,) = sqlx::query_as(
"SELECT COALESCE(SUM(u.original_size_bytes), 0)::bigint
FROM upload u
JOIN \"user\" usr ON usr.id = u.user_id
WHERE u.event_id = $1 AND u.deleted_at IS NULL
AND usr.uploads_hidden = FALSE AND usr.is_banned = FALSE",
)
.bind(event_id)
.fetch_one(pool)
.await
.context("estimating the export size")?;
Ok(bytes.max(0) as u64)
}
/// Headroom multiplier over the raw media sum: ZIP central directory, per-entry headers, the
/// embedded viewer, and the HTML export's temp staging.
const EXPORT_SIZE_OVERHEAD_PCT: u64 = 110;
/// Bytes this export must have available, given the raw media sum and how many jobs are competing.
///
/// `armed` is the multiplier that keeps two concurrent workers honest. `spawn_export_jobs` starts
/// the ZIP and the HTML halves at the same instant and both are gallery-sized, so a worker that
/// reserved only for itself would see "it fits", its sibling would independently see the same, and
/// together they would not fit — which is precisely the ENOSPC this preflight exists to prevent.
/// Computed in `u128` and clamped, NOT with `saturating_mul`: saturating first and then dividing by
/// 100 quietly turns an overflow into a number ~100x too small, which is the one direction that
/// matters here — an under-estimate authorises the very write the preflight exists to refuse.
fn required_free_bytes(media_bytes: u64, armed: i64) -> u64 {
let needed = media_bytes as u128 * EXPORT_SIZE_OVERHEAD_PCT as u128 / 100
* armed.max(1).min(i64::from(u32::MAX)) as u128;
needed.min(u64::MAX as u128) as u64
}
/// Refuse to start an export that cannot fit, with a reason the host can act on.
///
/// Without this the failure mode is ENOSPC halfway through a multi-GB write, and the wreckage
/// outlives it: the epoch has already moved, so the job row is `failed` at the CURRENT generation
/// and `GET /export/zip` 404s, while the last good archive sits on disk unreferenced. The host's
/// only escape (`POST /host/export/rebuild`) needs the very space that isn't there. Failing at the
/// door instead leaves the disk untouched and puts a number in front of the operator.
///
/// Both halves are spawned concurrently and both are gallery-sized, so a worker must reserve for
/// its live sibling too — otherwise each independently sees "it fits", and together they don't.
async fn ensure_export_space(pool: &PgPool, event_id: Uuid, export_path: &Path) -> Result<()> {
let media_bytes = estimate_export_bytes(pool, event_id).await?;
// Every job armed at any epoch for this event that hasn't finished is competing for this disk.
let (armed,): (i64,) = sqlx::query_as(
"SELECT COUNT(*) FROM export_job
WHERE event_id = $1 AND status IN ('pending', 'running')",
)
.bind(event_id)
.fetch_one(pool)
.await
.context("counting armed export jobs")?;
let needed = required_free_bytes(media_bytes, armed);
// `None` = the mount couldn't be resolved. Fail OPEN, exactly as the upload quota does: refusing
// to build the keepsake because we can't read a number would be a worse failure than trying.
let Some(free) = crate::services::disk::free_bytes(export_path) else {
tracing::warn!("export preflight: disk snapshot unavailable; proceeding without the check");
return Ok(());
};
if free < needed {
let gb = |b: u64| b as f64 / 1_000_000_000.0;
tracing::error!(
needed,
free,
armed,
"export preflight: not enough free space to build the keepsake for event {event_id}"
);
anyhow::bail!(
"Nicht genug Speicherplatz für das Keepsake: benötigt ca. {:.1} GB, frei sind {:.1} GB. \
Bitte Speicher freigeben und das Keepsake anschließend neu erstellen.",
gb(needed),
gb(free)
);
}
Ok(())
}
/// Best-effort removal of stale per-generation export artifacts for one export type. Deletes
/// ONLY strictly-older generations (`n < keep_seq`) — never `keep_seq`'s own current file,
/// and never a NEWER generation that a concurrent re-release may already be producing (that
@@ -1177,20 +1367,7 @@ async fn prune_stale_export_files(
event_id: Uuid,
keep_seq: i64,
) {
// Files still referenced by a live (current-epoch) job row are OFF LIMITS regardless of the
// epoch in their name. A ViewerOnly regeneration carries the finished ZIP forward by re-stamping
// its row to the new epoch WITHOUT renaming the file — so `Gallery.<event>.<older>.zip` is still
// the served archive, and deleting it by filename-epoch would 404 the download.
let protected: Vec<String> = sqlx::query_scalar::<_, String>(
"SELECT split_part(j.file_path, '/', -1) FROM export_job j
JOIN event e ON e.id = j.event_id
WHERE j.event_id = $1 AND j.epoch = e.export_epoch
AND j.status = 'done' AND j.file_path IS NOT NULL",
)
.bind(event_id)
.fetch_all(pool)
.await
.unwrap_or_default();
let protected = protected_files(pool, event_id).await;
// EVERY shape is event-scoped. All events share one exports volume, so a name keyed only by
// generation would let event A's prune delete event B's live keepsake (and let two events
@@ -1504,3 +1681,131 @@ So geht's:\n\
Alles ist lokal auf deinem Gerät gespeichert.\n\
\n\
Viel Freude mit den Erinnerungen!\n";
// ── Tests ────────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
const EVT: &str = "11111111-1111-1111-1111-111111111111";
fn gallery_prefix() -> String {
format!("Gallery.{EVT}.")
}
#[test]
fn a_strictly_older_archive_is_reclaimed() {
// The whole point: at the start of a rebuild at epoch 5, generation 4's archive is dead
// weight — readiness is derived from `epoch = event.export_epoch`, so it is already
// unreachable — and its bytes are very often exactly the bytes the rebuild needs.
assert!(is_superseded_archive(
&format!("Gallery.{EVT}.4.zip"),
&gallery_prefix(),
5,
&[]
));
}
#[test]
fn our_own_and_newer_generations_are_never_touched() {
// `keep_seq` is OUR generation; a NEWER one belongs to a re-release that has already
// superseded us, and deleting it would let a lagging worker nuke a live keepsake.
for seq in [5, 6] {
assert!(
!is_superseded_archive(
&format!("Gallery.{EVT}.{seq}.zip"),
&gallery_prefix(),
5,
&[]
),
"generation {seq} must survive a prune keeping 5"
);
}
}
#[test]
fn a_carried_forward_archive_survives_despite_an_older_epoch_in_its_name() {
// THE dangerous case. A ViewerOnly regeneration (a moderated comment) re-stamps the
// finished ZIP's row to the new epoch WITHOUT renaming the file, so the SERVED archive
// legitimately carries an older generation number. Pruning it by filename-epoch would 404
// the photo download to change nothing in it. The protected set is what stops that, and
// moving the prune to build-start makes this case reachable far more often.
let carried = format!("Gallery.{EVT}.4.zip");
assert!(!is_superseded_archive(
&carried,
&gallery_prefix(),
5,
std::slice::from_ref(&carried)
));
}
#[test]
fn temps_and_staging_dirs_are_out_of_scope_for_the_early_prune() {
// A superseded worker may still be streaming into these, and at build START it is much
// more likely to be alive than at finalize time. Only inert FINAL archives are reclaimed
// here; `prune_stale_export_files` still handles the rest after we win.
for name in [
format!("Gallery.{EVT}.4.tmp"),
format!("viewer_tmp_{EVT}_4"),
format!("Memories.{EVT}.4.zip"),
] {
assert!(
!is_superseded_archive(&name, &gallery_prefix(), 5, &[]),
"{name} must not be reclaimed by the pre-build prune"
);
}
}
#[test]
fn another_events_archive_is_never_reclaimed() {
// All events share one exports volume, so the prefix carries the event id.
let other = "22222222-2222-2222-2222-222222222222";
assert!(!is_superseded_archive(
&format!("Gallery.{other}.4.zip"),
&gallery_prefix(),
5,
&[]
));
}
#[test]
fn unrelated_files_are_left_alone() {
for name in ["Gallery.zip", "notes.txt", "Gallery..4.zip"] {
assert!(!is_superseded_archive(name, &gallery_prefix(), 5, &[]));
}
}
#[test]
fn a_lone_armed_job_reserves_for_one_archive() {
// A ViewerOnly regeneration re-arms only the HTML half — reserving for two would refuse
// a rebuild that fits perfectly well.
assert_eq!(required_free_bytes(1_000, 1), 1_100);
}
#[test]
fn two_concurrent_halves_reserve_for_both() {
// The bug this exists to prevent: each worker independently sees "it fits", and together
// they don't. Both halves are gallery-sized, so the reservation must be for the pair.
assert_eq!(required_free_bytes(1_000, 2), 2_200);
}
#[test]
fn a_zero_count_still_reserves_for_one() {
// Defensive: a racing status transition must never yield a zero requirement, which would
// wave through an export of any size onto a full disk.
assert_eq!(required_free_bytes(1_000, 0), 1_100);
}
#[test]
fn an_empty_gallery_needs_nothing() {
assert_eq!(required_free_bytes(0, 2), 0);
}
#[test]
fn a_pathological_size_saturates_instead_of_wrapping() {
// u64 overflow would wrap to a TINY requirement and authorise the exact write we're
// guarding against — the failure mode must be "refuse", never "wrap and allow".
assert_eq!(required_free_bytes(u64::MAX, 2), u64::MAX);
}
}