perf(backend): close config/upload/export hot paths + stability fixes

Performance:
- Cache the runtime `config` table in-memory (ConfigCache) with synchronous
  invalidation on every write (admin PATCH + test reseed). Was re-reading each
  key from Postgres on every request (~8 round-trips per upload).
- Stream uploads chunk-by-chunk to a temp file instead of buffering the whole
  body in RAM (peak was up to the per-class cap, e.g. 500 MB/video); only 512
  sniff-bytes are kept for magic-byte detection, then atomic rename into place.
- Cache the media-filesystem disk snapshot (DiskCache, 15s TTL) shared by the
  quota check and admin stats; drop the discarded System::refresh_all().
- HTML export streams video (and small-image) originals straight into the ZIP
  via a manifest instead of copying them to a temp dir first (removed the
  transient 2x disk usage) and drops the double directory scan.
- Auth extractor resolves session -> live user in one JOIN (was two queries),
  touching last_seen_at by token hash.

Stability:
- SSE: on broadcast lag, emit a `resync` event so the client runs a delta
  fetch instead of silently losing events; frontend reconciles adds, deletions,
  and (via an in-place refresh) like/comment counts on visible cards.
- Storage quota fails OPEN when the disk can't be read (was a 0-byte limit that
  locked out all uploads).
- Graceful shutdown drains in-flight requests on SIGTERM/SIGINT, bounded by a
  10s backstop so open SSE streams can't stall a deploy.
- Upload removes the persisted file if the DB transaction fails (no orphaned
  bytes with no row to reclaim them).

Tests:
- New pure select_disk() with 5 unit tests (longest-prefix, fallbacks, fail-open).
- New e2e export-video spec covering the HTML export's video-streaming branch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
fabi
2026-07-07 20:26:45 +02:00
parent 4cdb3ae14a
commit d6c91974eb
18 changed files with 745 additions and 214 deletions

View File

@@ -0,0 +1,64 @@
/**
* Covers the HTML export's VIDEO path (perf/stability fix P4). The export used to
* copy every original — including full-size videos — into a temp dir and then re-read
* them into the ZIP (transient 2× disk). It now streams video originals straight from
* disk into the archive. The image-only export specs never exercised that branch, so
* this drives a real video upload → export → and proves the video entry lands in
* Memories.zip (i.e. the streamed-from-original path works and the video isn't dropped).
*
* The fixture clip is <1s, so ffmpeg extracts no thumbnail frame — but exits 0, so the
* compression worker keeps the upload (it isn't auto-cleaned). That's the intended
* shape here: the full video is exported even when its thumbnail is absent.
*/
import { test, expect } from '../../fixtures/test';
import { uploadRaw } from '../../helpers/upload-client';
import { readFileSync } from 'node:fs';
import { join } from 'node:path';
const BASE = process.env.E2E_FRONTEND_URL ?? 'http://localhost:3101';
test.describe('Export — video streaming (P4)', () => {
test('a real video upload is streamed into Memories.zip (not dropped)', async ({ host }) => {
test.setTimeout(60_000);
const bearer = { Authorization: `Bearer ${host.jwt}` };
// Seed a real video upload owned by the host.
const mp4 = readFileSync(join(process.cwd(), 'fixtures', 'media', 'sample.mp4'));
const up = await uploadRaw(host.jwt, mp4, { filename: 'clip.mp4', contentType: 'video/mp4' });
expect(up.status).toBe(201);
const { id } = await up.json();
// Release the gallery → spawns the real zip + html export jobs.
const rel = await fetch(`${BASE}/api/v1/host/gallery/release`, { method: 'POST', headers: bearer });
expect(rel.status).toBe(204);
// Wait for the HTML (Memories.zip) job — that's the one whose media staging P4 changed.
await expect
.poll(
async () => {
const res = await fetch(`${BASE}/api/v1/export/status`, { headers: bearer });
return (await res.json()).html?.status;
},
{ timeout: 45_000, intervals: [500] }
)
.toBe('done');
// Download Memories.zip via the gated single-use ticket.
const ticketRes = await fetch(`${BASE}/api/v1/export/ticket`, { method: 'POST', headers: bearer });
const { ticket } = await ticketRes.json();
const dl = await fetch(`${BASE}/api/v1/export/html?ticket=${encodeURIComponent(ticket)}`);
expect(dl.status).toBe(200);
const bytes = new Uint8Array(await dl.arrayBuffer());
// Valid ZIP (PK local-file-header magic).
expect(Array.from(bytes.subarray(0, 2))).toEqual([0x50, 0x4b]);
// ZIP entry names are stored verbatim (uncompressed) in the archive, so the video's
// media entry name appears as plaintext bytes. Its presence means the streamed
// video entry was written; if the stream had failed, entry.close() would have
// errored and the job would never have reached 'done'.
const needle = `media/${id}.mp4`;
const haystack = new TextDecoder('latin1').decode(bytes);
expect(haystack.includes(needle), `Memories.zip must contain ${needle}`).toBe(true);
});
});