test(loadtest): 100-guest / 1000-image stress harness

HTTP-level load driver simulating ~100 guests uploading ~1000 images in bursts
over a window, plus SSE viewers and one real browser on /diashow. Correlates
upload→upload-processed (pipeline latency), waits for the compression backlog
to drain against DB ground truth, and emits per-status/latency metrics with
pass/fail flags. Includes a realistic-JPEG generator and a diashow-SSE
regression check (confirm-diashow-fix.mjs). Run artifacts are gitignored.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-18 17:29:20 +02:00
parent 3c3a7d0082
commit 9b8698f86b
6 changed files with 1065 additions and 0 deletions

112
e2e/loadtest/README.md Normal file
View File

@@ -0,0 +1,112 @@
# EventSnap load / stress test
Simulates a real event: **~100 guests** joining and uploading **~1000 images** in
bursts (1020 at a time) spread across a compressed time window, a pool of
**viewers** holding live SSE connections, and **one real browser** on `/diashow`
acting as the showcase display.
## Goal (this harness is tuned for it)
**Validate the shipping config.** We run at the real production defaults —
compression concurrency (`COMPRESSION_WORKER_CONCURRENCY`, default **2**), DB
pool (default **10**), quotas **on** — and answer: *does the app survive the
event, and how far behind real-time does the diashow fall?*
The headline metric is **pipeline latency**: time from an upload succeeding to
its preview being ready (`upload-processed` SSE event) — i.e. *how long until the
photo appears on the diashow*. A backlog that builds is fine; a backlog that
**never drains** is a fail for a live event.
## Methodology: what we change vs. shipping
We **only disable rate limits**. They're per-IP / per-user anti-abuse guards; a
synthetic test from one IP trips them in a way real guests (distinct IPs, phones)
never would — leaving them on would measure the limiter, not the pipeline.
Everything else (compression concurrency, DB pool, quotas) stays at the real
default so the result is honest.
> **Standalone finding to remember:** the shipping `upload_rate_per_hour` default
> is **10**. A real guest uploading a burst of 1020 photos would be throttled by
> the shipping config too. That's a genuine event-day issue worth surfacing
> separately from this pipeline test.
## Prereqs
- The isolated test stack up: `cd e2e && npm run stack:up` (Caddy on `:3101`,
`EVENTSNAP_TEST_MODE=1`, `/admin/__truncate` live).
- Node 24+ (global `fetch`/`FormData`/`Blob`), Python 3 + Pillow, Docker CLI
access (used for `docker stats` + `docker exec psql` ground-truth sampling).
- `@playwright/test` (already an e2e dep) for the diashow watcher.
## 1. Generate the image pool (once)
Realistic phone-sized JPEGs (~24 MB, 12 MP, high entropy). The driver reuses
this pool at random across all 1000 uploads — real load is byte size + decode
cost, not file uniqueness.
```bash
python3 e2e/loadtest/gen-images.py 40 # → /tmp/eventsnap-loadtest/photos
```
~40 images ≈ 120 MB pool; projects to **~34 GB** of originals for 1000 uploads
(previews/thumbnails add more). The generator prints the projection; check disk.
## 2. Smoke run first (~1 min)
Proves the wiring — join, upload, SSE correlation, drain, metrics — before the
real thing:
```bash
LT_GUESTS=5 LT_IMAGES=50 LT_WINDOW_SEC=60 node e2e/loadtest/driver.mjs
```
## 3. Full run (~15 min + drain)
Two terminals. Start the showcase display first, then the driver:
```bash
# terminal A — the showcase device
node e2e/loadtest/diashow-watch.mjs
# terminal B — 100 guests / 1000 images / 15-min window (defaults)
node e2e/loadtest/driver.mjs
```
The driver truncates event data first (`LT_TRUNCATE=0` to keep), disables rate
limits, joins guests, opens SSE, runs the burst schedule, then **waits for the
compression backlog to drain** before reporting.
## Output
- Console: live progress every 10 s, then a RESULTS block with pass/fail flags.
- `e2e/loadtest/results/run-<timestamp>.json`: full metrics — upload latency
percentiles, pipeline latency percentiles, drain time, per-status counts, SSE
reconnect/resync counts, and a `docker stats` + DB-connection time series.
- `e2e/loadtest/results/diashow/`: periodic screenshots of the live display.
## What the flags mean
| Flag | Meaning |
|------|---------|
| `✗ 5xx` | server errored under load — hard fail |
| `✗ 507` | quota rejected uploads — disk/quota misconfig for the event size |
| `✗ backlog did not drain` | compression can't keep up even after uploads stop — diashow never catches up |
| `⚠ pipeline p95 > 60s` | photos take >1 min to appear on the diashow at peak |
| `⚠ SSE resyncs` | live consumers lagged the broadcast channel |
## Knobs
All via env (see header of `driver.mjs`): `LT_GUESTS`, `LT_IMAGES`,
`LT_WINDOW_SEC`, `LT_BURST_MIN/MAX`, `LT_BURST_CONC`, `LT_VIEWERS`,
`LT_TRUNCATE`, `LT_DRAIN_TIMEOUT_SEC`, `LT_KEEP_RATELIMITS`, `LT_BASE`,
`LT_APP_CONTAINER`, `LT_DB_CONTAINER`.
To later answer *"what config should I deploy?"*, re-run with a rebuilt stack
that sets `COMPRESSION_WORKER_CONCURRENCY` higher (boot-time env var in
`docker-compose.test.yml`) and compare the pipeline-latency / drain numbers.
## Teardown
```bash
cd e2e && npm run stack:down # wipes volumes (media + db)
```