fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(export): close all 13 findings from the multi-agent review

Browse Source 
A 6-lens multi-agent review (103 agents; every finding adversarially verified by three
refute-by-default skeptics) confirmed the epoch redesign is sound — the core invariant
holds — and found 13 real defects around it. All are fixed here.

The redesign's invariant was re-verified and stands: an accepted upload is always in the
keepsake, and a downloadable keepsake always reflects the most recent release.

But a WEAKER adjacent invariant did NOT hold — "a downloadable keepsake reflects the
current content" — and that is the theme of the biggest fixes.

CONTENT REMOVAL NOW ALWAYS REACHES THE KEEPSAKE
  Regeneration was wired only to host deletes. Ban, unban, guest self-delete and guest
  comment-delete did not trigger it — yet the export query filters `is_banned = FALSE`,
  so the pipeline already agreed that content must not be there. Ban someone for abusive
  content after release and every guest kept downloading an archive containing it.
  All five removal paths now invalidate and rebuild. `query_comments` also gained the
  banned/hidden filter it was missing entirely (the ZIP had it; the viewer did not).

  Each removal is now ATOMIC with its invalidation (one tx, models made executor-generic).
  Previously the delete committed in its own tx and the regeneration in a second: a dropped
  handler future between them left the taken-down photo permanently downloadable, and
  nothing could notice — the keepsake still looked complete and the host could no longer
  even find the upload to retry.

NO MORE TAKEDOWN STORM
  Every removal spawned two uncancellable full exports. A superseded worker was INERT, not
  STOPPED: it still ran every ffmpeg spawn and image resize and wrote a whole archive before
  discovering it had lost. Five deletes left ten workers alive, each holding a
  full-gallery-sized temp, in a 1 GB container.
  Now: a debounce before `claim_job` (a superseded worker fails its claim and does ZERO
  work, collapsing a burst into one export), plus `update_progress` — which already ran
  exactly the right liveness predicate and threw the answer away — returning it so both file
  loops bail the instant they are retired. Moderating a comment no longer rebuilds the
  multi-GB ZIP either: the ZIP is media-only, so it is carried forward, with prune taught to
  protect any file a live row still references.

AN ESCAPE HATCH
  A failed export was terminal at runtime: release_gallery refuses an already-released event,
  recovery only runs at boot, and there was no retry route — the only outs were restarting the
  container or reopening uploads to every guest. Added POST /host/export/rebuild. Also widened
  mark_failed's guard to `status IN ('running','pending')`: when claim_job itself ERRORED, the
  row was still `pending`, so the failure was never recorded and the job sat at 0% forever.

SILENT INVALIDATION
  Retiring the epoch 404s the download instantly, but nothing told the clients — guests kept
  seeing an enabled download button through the whole rebuild. Now broadcasts an invalidation.
  And the download was a top-level <a href>: a 404 (or a 429 from the per-IP export limit that
  everyone on the venue WiFi shares) NAVIGATED THE USER OUT OF THE PWA onto raw JSON. It now
  targets a hidden iframe, so an error response is harmless.

ALSO
  - The HTML export still had the exists()-then-open TOCTOU the ZIP path was fixed to remove,
    at three sites, two with a hard `?` that failed the ENTIRE viewer. It is reachable: the
    compression worker hard-deletes an original when a transcode fails and can still be running
    when the gallery is released.
  - Crash-orphaned .tmp files were immortal (prune deliberately skips .tmp). Swept at boot,
    where it is unambiguously safe.
  - export_status read the epoch in one statement and used it in the next; now one query.
  - DiskCache::snapshot IGNORED its path argument on a cache hit, returning whatever filesystem
    was measured last. Latent only because both callers pass media_path — it would have silently
    misreported the moment anyone measured the exports volume. Now keyed by path.
  - Corrected two comments that asserted safety properties the code does not have (claim_job
    does NOT prevent a retired-epoch claim — retirement is enforced at READ time; and a
    multi-replica reap is NOT contained, it can corrupt the keepsake). Added a rollback runbook
    to migration 014, the repo's first destructive migration.

TESTS
  The regression guard for the headline FOR SHARE fix was probabilistic — it could pass on
  broken code if the interleaving fell the wrong way. It is now STRUCTURAL: the upload is held
  open mid-body with its pre-flight check already passed, so the release provably lands inside
  the exact window. Verified 5/5 FAILING (201 instead of 403) with the fix reverted, 5/5
  passing with it.

Verified: backend 40 tests, svelte-check 0 errors, e2e typecheck clean,
e2e 160 passed / 1 skipped on chromium-desktop.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
fabi
2026-07-14 20:35:47 +02:00
parent 5affef47cf
commit 9666d74a46
14 changed files with 632 additions and 147 deletions
Download Patch File Download Diff File Expand all files Collapse all files

64
e2e/helpers/upload-client.ts
View File

@@ -46,3 +46,67 @@ export const JPEG_MAGIC = new Uint8Array([0xff, 0xd8, 0xff, 0xe0, 0x00, 0x10, 0x
export const PNG_MAGIC = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
/** ELF header — the magic bytes of a Linux executable. `infer` reports `application/x-executable`. */
export const ELF_MAGIC = new Uint8Array([0x7f, 0x45, 0x4c, 0x46]);
/**
* Upload whose request body PAUSES mid-stream until you release it.
*
* This exists to make the release-vs-upload race deterministic instead of hoped-for. The bug it
* guards is a TOCTOU: `upload` checks `uploads_locked_at` BEFORE reading the body (minutes, for a
* big video) and commits the row afterwards — so a release landing in between used to let the photo
* commit AFTER the export snapshot, leaving it in the live feed but permanently absent from the
* keepsake.
*
* Racing N normal uploads against a release can't prove anything: if they all happen to commit
* first (or all get rejected), the assertion passes on broken code too. Here the server is *stuck*
* mid-body with its pre-flight check already passed, so the release provably lands inside the
* window. Await `checkPassed`, fire the release, then `finish()`.
*/
export function uploadPausedMidStream(
token: string,
head: Buffer,
tail: Buffer,
opts: UploadOptions = {}
): { checkPassed: Promise<void>; finish: () => void; response: Promise<Response> } {
const boundary = '----eventsnapPausedBoundary' + Math.random().toString(16).slice(2);
const filename = opts.filename ?? 'paused.jpg';
const contentType = opts.contentType ?? 'image/jpeg';
const preamble = Buffer.from(
`--${boundary}\r\n` +
`Content-Disposition: form-data; name="file"; filename="${filename}"\r\n` +
`Content-Type: ${contentType}\r\n\r\n`
);
const epilogue = Buffer.from(`\r\n--${boundary}--\r\n`);
let releaseBody: () => void;
const gate = new Promise<void>((r) => (releaseBody = r));
let markCheckPassed: () => void;
const checkPassed = new Promise<void>((r) => (markCheckPassed = r));
const body = new ReadableStream<Uint8Array>({
async pull(controller) {
controller.enqueue(new Uint8Array(preamble));
controller.enqueue(new Uint8Array(head));
// The server has now read the part headers and the first bytes — which means it is PAST its
// pre-flight lock check and is sitting in the body loop. This is the window.
markCheckPassed();
await gate;
controller.enqueue(new Uint8Array(tail));
controller.enqueue(new Uint8Array(epilogue));
controller.close();
},
});
const response = fetch(`${BASE}/api/v1/upload`, {
method: 'POST',
headers: {
Authorization: `Bearer ${token}`,
'Content-Type': `multipart/form-data; boundary=${boundary}`,
},
body,
// Node's fetch requires this for a streaming request body.
duplex: 'half',
} as RequestInit & { duplex: 'half' });
return { checkPassed, finish: () => releaseBody(), response };
}

77
e2e/specs/10-flow-review/export-reopen-rerelease.spec.ts
View File

@@ -30,8 +30,9 @@
* generation and regenerates, rather than leaving the stale worker to finalize.
* - supersession (reopen): a reopen ALONE bumps the event epoch, so a worker holding the
* pre-reopen epoch is retired and can publish nothing.
* - acceptance: every upload the server ACCEPTED (201) is in the keepsake — the upload-path
* TOCTOU that was the real cause of the "stale keepsake" bug.
* - acceptance: an upload held open mid-body across a release is REJECTED, not silently dropped
* from the keepsake — the upload-path TOCTOU that was the real cause of the "stale keepsake" bug.
* Deterministic: the body is paused with the pre-flight check already passed.
* - takedown: deleting a photo AFTER release regenerates the keepsake without it.
*/
import { test, expect } from '../../fixtures/test';
@@ -41,7 +42,7 @@ import { writeFileSync, mkdtempSync, readFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { seedUpload } from '../../helpers/seed';
import { uploadRaw } from '../../helpers/upload-client';
import { uploadRaw, uploadPausedMidStream } from '../../helpers/upload-client';
import { SseListener } from '../../helpers/sse-listener';
const BASE = process.env.E2E_FRONTEND_URL ?? 'http://localhost:3101';
@@ -342,51 +343,51 @@ test.describe('Flow re-review — reopen→re-release export integrity + stale-k
expect((await downloadZipEntries(host.jwt)).length).toBe(1);
});
test('EVERY upload the server accepted is in the keepsake (upload-path release TOCTOU)', async ({
test('an upload STUCK MID-BODY when the release lands is rejected, not silently lost', async ({
host,
}) => {
// THE contract, and the bug that survived three rounds of fixes to the export state machine
// because it was never in the state machine:
// THE bug that survived three rounds of fixes to the export state machine — because it was never
// in the state machine. `upload` checked `uploads_locked_at` BEFORE streaming the body (minutes,
// for a 500 MB video) and committed the row without re-checking. A release landing mid-upload let
// the row commit AFTER the export snapshot: the photo showed up in the live feed and was
// PERMANENTLY missing from the keepsake, with nothing to regenerate it.
//
// `upload` checked `uploads_locked_at` BEFORE streaming the body (minutes, for a big video)
// and then committed the row WITHOUT re-checking. So a release landing mid-upload let the row
// commit AFTER the export snapshot: the photo appeared in the live feed and was PERMANENTLY
// missing from the keepsake. Nothing ever regenerated it.
//
// The invariant is absolute and does not depend on winning any race: if the server said 201,
// that photo is in the keepsake. If it said 403, it isn't. Race uploads against a release and
// hold the server to exactly that.
const bytes = readFileSync(join(process.cwd(), 'fixtures', 'media', 'sample.jpg'));
// One upload that is definitely committed before the race, so the keepsake is never trivially
// empty (the release can otherwise win every race and the assertion would prove nothing).
// This test is DETERMINISTIC, not a hoped-for race. The upload is held open mid-body with its
// pre-flight check already passed, so the release provably lands inside the exact window. The
// server must then reject it at commit time (`FOR SHARE` re-check) — anything else means a photo
// the server accepted is not in the archive.
await seedUpload(host.jwt, { caption: 'pre-race' });
const inflight = Array.from({ length: 10 }, (_, i) =>
uploadRaw(host.jwt, bytes, { filename: `race${i}.jpg`, contentType: 'image/jpeg' })
);
// Fire the release while those are in flight.
const releasing = post('/api/v1/host/gallery/release', host.jwt);
const img = readFileSync(join(process.cwd(), 'fixtures', 'media', 'sample.jpg'));
const split = Math.floor(img.length / 2);
const paused = uploadPausedMidStream(host.jwt, img.subarray(0, split), img.subarray(split), {
filename: 'stuck.jpg',
contentType: 'image/jpeg',
});
const results = await Promise.all(inflight);
await releasing;
// Wait until the server is genuinely mid-body (past its pre-flight lock check).
await paused.checkPassed;
const accepted = results.filter((r) => r.status >= 200 && r.status < 300).length;
const rejected = results.filter((r) => r.status === 403).length;
expect(accepted + rejected, 'every upload either succeeded or was locked out').toBe(
results.length
);
// Land the release squarely inside that window.
expect((await post('/api/v1/host/gallery/release', host.jwt)).status).toBe(204);
// Now let the body finish. The pre-flight check passed, so ONLY the commit-time re-check can
// save this photo from becoming a phantom.
paused.finish();
const res = await paused.response;
expect(
res.status,
'an upload whose body completed after the release MUST be rejected (403 uploads_locked). ' +
'A 201 here means the server accepted a photo it will never put in the keepsake — silent, ' +
'permanent data loss.'
).toBe(403);
expect((await res.json()).error).toBe('uploads_locked');
// And the keepsake holds exactly the one upload that was genuinely committed before the release.
await waitExportDone(host.jwt);
const listing = await downloadZipEntries(host.jwt);
// +1 for the pre-race upload.
expect(
listing.length,
`server accepted ${accepted} racing upload(s) (+1 pre-race) and locked out ${rejected}, but ` +
`the keepsake holds ${listing.length}. An accepted photo missing from the keepsake is ` +
`silent, permanent data loss — exactly the bug this guards.`
).toBe(accepted + 1);
expect(listing.length, `zip entries:\n${listing.join('\n')}`).toBe(1);
});
test('deleting a photo AFTER release regenerates the keepsake without it (takedown)', async ({