9b8698f86b
HTTP-level load driver simulating ~100 guests uploading ~1000 images in bursts over a window, plus SSE viewers and one real browser on /diashow. Correlates upload→upload-processed (pipeline latency), waits for the compression backlog to drain against DB ground truth, and emits per-status/latency metrics with pass/fail flags. Includes a realistic-JPEG generator and a diashow-SSE regression check (confirm-diashow-fix.mjs). Run artifacts are gitignored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
113 lines
4.5 KiB
Markdown
113 lines
4.5 KiB
Markdown
# EventSnap load / stress test
|
||
|
||
Simulates a real event: **~100 guests** joining and uploading **~1000 images** in
|
||
bursts (10–20 at a time) spread across a compressed time window, a pool of
|
||
**viewers** holding live SSE connections, and **one real browser** on `/diashow`
|
||
acting as the showcase display.
|
||
|
||
## Goal (this harness is tuned for it)
|
||
|
||
**Validate the shipping config.** We run at the real production defaults —
|
||
compression concurrency (`COMPRESSION_WORKER_CONCURRENCY`, default **2**), DB
|
||
pool (default **10**), quotas **on** — and answer: *does the app survive the
|
||
event, and how far behind real-time does the diashow fall?*
|
||
|
||
The headline metric is **pipeline latency**: time from an upload succeeding to
|
||
its preview being ready (`upload-processed` SSE event) — i.e. *how long until the
|
||
photo appears on the diashow*. A backlog that builds is fine; a backlog that
|
||
**never drains** is a fail for a live event.
|
||
|
||
## Methodology: what we change vs. shipping
|
||
|
||
We **only disable rate limits**. They're per-IP / per-user anti-abuse guards; a
|
||
synthetic test from one IP trips them in a way real guests (distinct IPs, phones)
|
||
never would — leaving them on would measure the limiter, not the pipeline.
|
||
Everything else (compression concurrency, DB pool, quotas) stays at the real
|
||
default so the result is honest.
|
||
|
||
> **Standalone finding to remember:** the shipping `upload_rate_per_hour` default
|
||
> is **10**. A real guest uploading a burst of 10–20 photos would be throttled by
|
||
> the shipping config too. That's a genuine event-day issue worth surfacing
|
||
> separately from this pipeline test.
|
||
|
||
## Prereqs
|
||
|
||
- The isolated test stack up: `cd e2e && npm run stack:up` (Caddy on `:3101`,
|
||
`EVENTSNAP_TEST_MODE=1`, `/admin/__truncate` live).
|
||
- Node 24+ (global `fetch`/`FormData`/`Blob`), Python 3 + Pillow, Docker CLI
|
||
access (used for `docker stats` + `docker exec psql` ground-truth sampling).
|
||
- `@playwright/test` (already an e2e dep) for the diashow watcher.
|
||
|
||
## 1. Generate the image pool (once)
|
||
|
||
Realistic phone-sized JPEGs (~2–4 MB, 12 MP, high entropy). The driver reuses
|
||
this pool at random across all 1000 uploads — real load is byte size + decode
|
||
cost, not file uniqueness.
|
||
|
||
```bash
|
||
python3 e2e/loadtest/gen-images.py 40 # → /tmp/eventsnap-loadtest/photos
|
||
```
|
||
|
||
~40 images ≈ 120 MB pool; projects to **~3–4 GB** of originals for 1000 uploads
|
||
(previews/thumbnails add more). The generator prints the projection; check disk.
|
||
|
||
## 2. Smoke run first (~1 min)
|
||
|
||
Proves the wiring — join, upload, SSE correlation, drain, metrics — before the
|
||
real thing:
|
||
|
||
```bash
|
||
LT_GUESTS=5 LT_IMAGES=50 LT_WINDOW_SEC=60 node e2e/loadtest/driver.mjs
|
||
```
|
||
|
||
## 3. Full run (~15 min + drain)
|
||
|
||
Two terminals. Start the showcase display first, then the driver:
|
||
|
||
```bash
|
||
# terminal A — the showcase device
|
||
node e2e/loadtest/diashow-watch.mjs
|
||
|
||
# terminal B — 100 guests / 1000 images / 15-min window (defaults)
|
||
node e2e/loadtest/driver.mjs
|
||
```
|
||
|
||
The driver truncates event data first (`LT_TRUNCATE=0` to keep), disables rate
|
||
limits, joins guests, opens SSE, runs the burst schedule, then **waits for the
|
||
compression backlog to drain** before reporting.
|
||
|
||
## Output
|
||
|
||
- Console: live progress every 10 s, then a RESULTS block with pass/fail flags.
|
||
- `e2e/loadtest/results/run-<timestamp>.json`: full metrics — upload latency
|
||
percentiles, pipeline latency percentiles, drain time, per-status counts, SSE
|
||
reconnect/resync counts, and a `docker stats` + DB-connection time series.
|
||
- `e2e/loadtest/results/diashow/`: periodic screenshots of the live display.
|
||
|
||
## What the flags mean
|
||
|
||
| Flag | Meaning |
|
||
|------|---------|
|
||
| `✗ 5xx` | server errored under load — hard fail |
|
||
| `✗ 507` | quota rejected uploads — disk/quota misconfig for the event size |
|
||
| `✗ backlog did not drain` | compression can't keep up even after uploads stop — diashow never catches up |
|
||
| `⚠ pipeline p95 > 60s` | photos take >1 min to appear on the diashow at peak |
|
||
| `⚠ SSE resyncs` | live consumers lagged the broadcast channel |
|
||
|
||
## Knobs
|
||
|
||
All via env (see header of `driver.mjs`): `LT_GUESTS`, `LT_IMAGES`,
|
||
`LT_WINDOW_SEC`, `LT_BURST_MIN/MAX`, `LT_BURST_CONC`, `LT_VIEWERS`,
|
||
`LT_TRUNCATE`, `LT_DRAIN_TIMEOUT_SEC`, `LT_KEEP_RATELIMITS`, `LT_BASE`,
|
||
`LT_APP_CONTAINER`, `LT_DB_CONTAINER`.
|
||
|
||
To later answer *"what config should I deploy?"*, re-run with a rebuilt stack
|
||
that sets `COMPRESSION_WORKER_CONCURRENCY` higher (boot-time env var in
|
||
`docker-compose.test.yml`) and compare the pipeline-latency / drain numbers.
|
||
|
||
## Teardown
|
||
|
||
```bash
|
||
cd e2e && npm run stack:down # wipes volumes (media + db)
|
||
```
|