`query_uploads` selects the rows the archives are built from; `estimate_export_bytes`
sizes them for the disk preflight. They stated the same WHERE clause separately, and
the direction of drift matters: an estimate that MISSES rows the archive writes
under-reserves, which is precisely the ENOSPC the preflight exists to prevent.
The integration test claimed to guard this and cannot. Both sides of
`the_estimate_sums_exactly_the_rows_the_archive_will_contain` are `SRC:`-marked
hand-copies in tests/common/mod.rs -- neither is production code -- so drift means
production moved while both copies sat still, and the test goes on passing. The
convention is sound for pinning behaviour; it is structurally incapable of detecting
divergence from the thing it copies.
So fix it where it can be fixed. One `export_visibility_where!()` fragment,
`concat!`-ed into both queries at compile time (still `&'static str`, no allocation),
with the `u`/`usr` alias contract stated. Divergence is now impossible by
construction rather than watched for.
The tests keep their value and lose the overclaim: the docstrings now say they pin
WHICH uploads may be counted -- each excluded row in the fixture is excluded by a
different predicate, so weakening any one of them still fails here -- and say plainly
that they do not detect drift, with a pointer to what does.
No behaviour change. The filters were verified identical before the hoist
(`u.event_id = $1 AND u.deleted_at IS NULL AND usr.uploads_hidden = FALSE AND
usr.is_banned = FALSE`); 99 backend tests still pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing in export.rs ever asked whether the keepsake would fit. Both archives write
their media `Compression::Stored`, so each is essentially a byte-for-byte second copy
of the originals -- Gallery.zip always, and Memories.zip for every video and every
image at or under 5 MB. On the documented CX33 (80 GB, all three volumes on one
filesystem) the upload quota's fixed point leaves ~40 GB free, and a release spawns
BOTH halves concurrently against it.
The failure is not "the export failed", it is "the deliverable is stuck":
1. ENOSPC lands partway through a multi-GB write.
2. The epoch has already moved, so the job row is `failed` at the CURRENT
generation and readiness (epoch = event.export_epoch AND status = 'done') is
false -- GET /export/zip 404s.
3. The last good archive sits on disk, unreferenced and unreachable.
4. POST /host/export/rebuild, the only escape, re-arms the same doomed write.
Three changes.
Reclaim before building. `prune_stale_export_files` ran only after the new archive
was written, renamed and finalised. That reads as durability but buys nothing: the
moment `invalidate_and_arm` bumps the epoch the old archive is ALREADY unreachable,
so keeping it reserves gigabytes for a download nobody can perform -- and for a
takedown it is content someone explicitly asked to have removed. Peak usage is now
one generation. Narrower than the post-finalize prune on purpose: final archives
only, never a `.tmp` or a `viewer_tmp_` dir, since a superseded worker can still be
streaming into those and at build START is far more likely to be alive.
Preflight the space. SUM(original_size_bytes) over exactly `query_uploads`'
visibility filter, +10% for ZIP overhead, multiplied by the number of armed jobs --
without that multiplier each of the two concurrent halves independently sees "it
fits" and together they don't. Runs AFTER claim_job, not before as reported: bailing
before the claim leaves the row `pending` with no worker and no error, the
spinner-forever state `mark_failed`'s status guard exists to prevent. Fails open when
the mount can't be read, exactly as the upload quota does.
Show the host the reason. /export/status returned {status, progress_pct} and nothing
else, so the host dashboard could only render "fehlgeschlagen" next to the retry
button. The message was written to the row and surfaced solely in the ADMIN job list
-- a different screen, possibly a different person. It now travels with the status,
and only on a failure, so a message left on a since-succeeded row can't appear beside
a green "ist bereit".
Tests: 10 unit (the u128 clamp caught a real bug in the first draft -- saturating_mul
then /100 turns an overflow into a number ~100x too small, the one direction that
authorises the write being guarded against; the carried-forward archive must survive
its own older epoch in the filename), 4 DB-backed (the estimate is asserted against
the row set the archive actually contains, not against a restatement of the WHERE
clause, so the two queries cannot drift), 3 e2e over the four-hop plumbing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All 40 backend tests were pure-function tests — not one line of SQL ran under `cargo test`,
even though the riskiest code in the repo is SQL. Add 12 DB-backed `#[sqlx::test]` tests
(fresh throwaway DB per test, real migrations) pinning the invariants that code is
load-bearing for, each mutation-verified to fail on broken code:
export_epoch.rs (8): epoch is post-increment on release (a worker born pre-increment is
inert); epoch strictly monotonic across reopen; a retired-epoch worker's writes are
no-ops; export_current is EXACTLY the invariant (8-case table); the ViewerOnly
carry-forward carries a done ZIP forward AND matches nothing when the ZIP is unfinished
(the af997a8 strand bug); boot recovery doesn't clobber a live ZIP.
upload_concurrency.rs (4): the atomic quota UPDATE stops two stale-snapshot uploads from
both committing (sequential AND concurrent-tx via EPQ); the FOR SHARE upload lock
serializes against release, blocking a photo from committing after the export snapshot.
Deleting FOR SHARE, or the carry-forward's `status='done'`, each makes a test fail.
Test isolation: TRUNCATE now also resets disk_cache and sse_tickets. The stale disk
reading was harmless only while quotas were globally off in e2e (they no longer are — see
the new quota spec), i.e. two holes were masking each other.
clippy: `cargo clippy --all-targets -- -D warnings` now passes (it did not). Deleted the
dead jobs.rs (an unused BackgroundJob sketch) and unused model methods; `#[allow(dead_code)]`
+ comment on the sqlx FromRow field-sets that ARE populated by the DB. No SQL or logic
changed. `cargo test` requires DATABASE_URL by design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
