`query_uploads` selects the rows the archives are built from; `estimate_export_bytes` sizes them for the disk preflight. They stated the same WHERE clause separately, and the direction of drift matters: an estimate that MISSES rows the archive writes under-reserves, which is precisely the ENOSPC the preflight exists to prevent. The integration test claimed to guard this and cannot. Both sides of `the_estimate_sums_exactly_the_rows_the_archive_will_contain` are `SRC:`-marked hand-copies in tests/common/mod.rs -- neither is production code -- so drift means production moved while both copies sat still, and the test goes on passing. The convention is sound for pinning behaviour; it is structurally incapable of detecting divergence from the thing it copies. So fix it where it can be fixed. One `export_visibility_where!()` fragment, `concat!`-ed into both queries at compile time (still `&'static str`, no allocation), with the `u`/`usr` alias contract stated. Divergence is now impossible by construction rather than watched for. The tests keep their value and lose the overclaim: the docstrings now say they pin WHICH uploads may be counted -- each excluded row in the fixture is excluded by a different predicate, so weakening any one of them still fails here -- and say plainly that they do not detect drift, with a pointer to what does. No behaviour change. The filters were verified identical before the hoist (`u.event_id = $1 AND u.deleted_at IS NULL AND usr.uploads_hidden = FALSE AND usr.is_banned = FALSE`); 99 backend tests still pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
164 lines
7.8 KiB
Rust
164 lines
7.8 KiB
Rust
//! DB-backed tests for the export disk preflight.
|
|
//!
|
|
//! The keepsake used to be built with NO free-space check at all, and the failure that produced was
|
|
//! not "the export failed" but "the deliverable is stuck and the escape hatch needs the space that
|
|
//! isn't there":
|
|
//!
|
|
//! 1. A takedown bumps the epoch and re-arms both halves.
|
|
//! 2. The ZIP hits ENOSPC partway through a multi-GB write.
|
|
//! 3. The job row is now `failed` at the CURRENT epoch, so readiness
|
|
//! (`epoch = event.export_epoch AND status = 'done'`) is false and `GET /export/zip` 404s —
|
|
//! while the last good archive sits on disk, unreferenced and unreachable.
|
|
//! 4. `POST /host/export/rebuild` re-arms the same doomed write.
|
|
//!
|
|
//! Two changes close it: reclaim the superseded generation BEFORE building (so peak usage is one
|
|
//! generation, not two) and refuse up front with a number the host can act on.
|
|
//!
|
|
//! What these tests pin is the ESTIMATE — the part that decides. The arithmetic on top of it lives
|
|
//! in `services/export.rs`'s unit tests; the filesystem selection lives in `is_superseded_archive`.
|
|
//!
|
|
//! ON DRIFT, precisely, because it is easy to overclaim here. The hazard is that `query_uploads`
|
|
//! (which selects the rows the archives are built from) and `estimate_export_bytes` (which sizes
|
|
//! them) could disagree — and an estimate missing rows the archive writes UNDER-reserves, the one
|
|
//! direction that reintroduces the ENOSPC. **These tests cannot catch that**, and neither can any
|
|
//! test in this harness: both sides here are `SRC:`-marked hand-copies in `tests/common/mod.rs`,
|
|
//! so if production moved and the copies didn't, they would sit still and keep passing.
|
|
//!
|
|
//! That is fixed where it can be — the two queries now share one `export_visibility_where!()`
|
|
//! fragment in `services/export.rs`, so they cannot diverge by construction. What is left for
|
|
//! these tests is what the convention is genuinely good at: pinning the BEHAVIOUR, so a change
|
|
//! that deliberately alters the filter has to come here and say so.
|
|
|
|
mod common;
|
|
|
|
use common::*;
|
|
use sqlx::PgPool;
|
|
|
|
/// The estimate must equal the sum over EXACTLY the rows `query_uploads` returns — computed from
|
|
/// that row set, not from a restatement of its WHERE clause.
|
|
///
|
|
/// PINS: which uploads the preflight is allowed to count. Each excluded row below is excluded by a
|
|
/// DIFFERENT predicate, so a change that drops or weakens any one of them fails here and has to be
|
|
/// argued for. (It does not detect production drifting away from these copies — see the file
|
|
/// header; `export_visibility_where!()` is what makes that impossible.)
|
|
#[sqlx::test]
|
|
async fn the_estimate_sums_exactly_the_rows_the_archive_will_contain(pool: PgPool) {
|
|
let event_id = seed_event(&pool, "wedding").await;
|
|
|
|
let visible = seed_user(&pool, event_id, "Anna").await;
|
|
let banned = seed_user(&pool, event_id, "Ben").await;
|
|
let hidden = seed_user(&pool, event_id, "Cara").await;
|
|
|
|
seed_upload(&pool, event_id, visible, 1_000, false).await;
|
|
seed_upload(&pool, event_id, visible, 2_500, false).await;
|
|
// Each of these is excluded from the archive by a DIFFERENT predicate.
|
|
seed_upload(&pool, event_id, visible, 9_000, true).await; // soft-deleted
|
|
seed_upload(&pool, event_id, banned, 9_000, false).await; // uploader banned
|
|
seed_upload(&pool, event_id, hidden, 9_000, false).await; // uploads hidden
|
|
|
|
set_user_moderation(&pool, banned, true, true).await;
|
|
set_user_moderation(&pool, hidden, false, true).await;
|
|
|
|
let rows = export_visible_uploads(&pool, event_id).await;
|
|
let expected: i64 = rows.iter().map(|(_, bytes)| bytes).sum();
|
|
|
|
assert_eq!(rows.len(), 2, "only Anna's two live uploads are archived");
|
|
assert_eq!(
|
|
estimate_export_bytes(&pool, event_id).await,
|
|
expected,
|
|
"the preflight must size the gallery the export will actually write"
|
|
);
|
|
assert_eq!(expected, 3_500);
|
|
}
|
|
|
|
/// An event with nothing to archive estimates zero rather than NULL.
|
|
///
|
|
/// PREVENTS: `SUM()` over no rows returning NULL and the decode blowing up — which would abort the
|
|
/// export with a type error instead of building an (entirely legitimate) empty keepsake.
|
|
#[sqlx::test]
|
|
async fn an_empty_gallery_estimates_zero_not_null(pool: PgPool) {
|
|
let event_id = seed_event(&pool, "wedding").await;
|
|
assert_eq!(estimate_export_bytes(&pool, event_id).await, 0);
|
|
|
|
// And with a user who has uploaded nothing.
|
|
seed_user(&pool, event_id, "Anna").await;
|
|
assert_eq!(estimate_export_bytes(&pool, event_id).await, 0);
|
|
}
|
|
|
|
/// A release arms both halves, so the preflight sees a count of 2 and reserves for the pair.
|
|
///
|
|
/// PREVENTS: the concurrency under-reservation. `spawn_export_jobs` starts the ZIP and HTML workers
|
|
/// at the same instant, and BOTH are gallery-sized (`Memories.zip` streams the original for every
|
|
/// video and every image at or under 5 MB, all `Compression::Stored`). A worker reserving only for
|
|
/// itself would see "it fits", its sibling would independently see the same, and together they
|
|
/// would ENOSPC — which is why `required_free_bytes` multiplies by this count.
|
|
#[sqlx::test]
|
|
async fn a_release_arms_both_halves_so_the_preflight_reserves_for_two(pool: PgPool) {
|
|
let event_id = seed_event(&pool, "wedding").await;
|
|
let user = seed_user(&pool, event_id, "Anna").await;
|
|
seed_upload(&pool, event_id, user, 1_000, false).await;
|
|
|
|
assert_eq!(
|
|
armed_job_count(&pool, event_id).await,
|
|
0,
|
|
"nothing is armed before the release"
|
|
);
|
|
|
|
let epoch = release_gallery(&pool, "wedding").await.expect("released");
|
|
assert_eq!(
|
|
armed_job_count(&pool, event_id).await,
|
|
2,
|
|
"a release arms zip AND html — both compete for the same disk"
|
|
);
|
|
|
|
// A worker that has claimed its half is still competing; `running` must keep counting.
|
|
assert!(claim_job(&pool, event_id, "zip", epoch).await);
|
|
assert_eq!(
|
|
armed_job_count(&pool, event_id).await,
|
|
2,
|
|
"claiming moves pending -> running, which must not drop out of the reservation"
|
|
);
|
|
|
|
// Only a FINISHED half stops competing.
|
|
assert!(finalize_job(&pool, event_id, "zip", epoch, "exports/Gallery.zip").await);
|
|
assert_eq!(
|
|
armed_job_count(&pool, event_id).await,
|
|
1,
|
|
"a done half no longer needs space reserved for it"
|
|
);
|
|
}
|
|
|
|
/// A ViewerOnly regeneration re-arms only the HTML half, so the preflight reserves for one.
|
|
///
|
|
/// PREVENTS: over-reservation refusing a rebuild that fits perfectly well. Moderating a comment
|
|
/// carries the finished ZIP forward untouched; demanding room for a second copy of it would fail
|
|
/// the one operation that needs no new gallery-sized write at all.
|
|
#[sqlx::test]
|
|
async fn a_viewer_only_regeneration_reserves_for_one_half(pool: PgPool) {
|
|
let event_id = seed_event(&pool, "wedding").await;
|
|
let user = seed_user(&pool, event_id, "Anna").await;
|
|
seed_upload(&pool, event_id, user, 1_000, false).await;
|
|
|
|
let epoch = release_gallery(&pool, "wedding").await.expect("released");
|
|
for t in ["zip", "html"] {
|
|
assert!(claim_job(&pool, event_id, t, epoch).await);
|
|
assert!(finalize_job(&pool, event_id, t, epoch, &format!("exports/{t}")).await);
|
|
}
|
|
assert_eq!(armed_job_count(&pool, event_id).await, 0);
|
|
|
|
// A moderated comment: bump the epoch, carry the ZIP forward, re-arm only the viewer.
|
|
let (_, _, next) = bump_epoch(&pool, "wedding").await.expect("bumped");
|
|
assert!(
|
|
carry_zip_forward(&pool, event_id, next).await,
|
|
"the finished ZIP is re-stamped, not rebuilt"
|
|
);
|
|
let mut conn = pool.acquire().await.expect("acquire");
|
|
enqueue_types_at_epoch(&mut conn, event_id, next, &["html"]).await;
|
|
|
|
assert_eq!(
|
|
armed_job_count(&pool, event_id).await,
|
|
1,
|
|
"only the viewer is being rebuilt, so only one archive's worth of space is needed"
|
|
);
|
|
}
|