Six independent operational defects, none of which needed a new feature to fix. Log rotation. Docker's json-file driver is unbounded by default, and those files land on the HOST filesystem — outside every deploy.resources.limits in the compose file, and on the same disk as postgres_data and media_data. A full disk stops Postgres writing WAL, which takes the event down. Capped at 10m x 3 per service. Log level. RUST_LOG was set in neither .env.example nor docker-compose.yml, so the code fallback WAS the production level — and it was `debug`, with tower_http=debug emitting a line per request and per response into that unrotated file. Now info, with tower_http=warn to state that those spans are diagnostics, not an access log. ffmpeg pipe deadlock. run_ffmpeg piped stdout and stderr and then called wait(), which drains neither. Once the ~64 KiB pipe buffer filled, ffmpeg blocked writing and wait() never returned — burning the full 120s timeout, twice per seek position, three times per compression attempt. And the timeout is an Err, so the end state was a soft-deleted upload: a guest's playable video destroyed by a poster-frame failure. Now stdout is null (nothing ever read it) and stderr is drained by wait_with_output, whose tail is logged on a non-zero exit. Note wait_with_output consumes the child, so the old kill-on-timeout is gone; kill_on_drop(true) already covers it. Readiness probe. /health never touched the pool, so the disk-full endgame above stayed green all the way down. Adds /health/ready (SELECT 1 under 2s) as a SECOND route — the compose healthcheck deliberately keeps pointing at /health, because caddy gates its startup on it and a DB-dependent probe would turn a Postgres blip into the reverse proxy refusing to start. api.ts request timeout. The abort timer was cleared in a finally around fetch(), which resolves on the response HEAD — leaving res.text() uncovered and no longer abortable. An upstream that sends headers then stalls the body hung the call forever. The timer now lives until the body is read, including the 204 path (which otherwise leaked a live 20s timer per no-content request). Upload XHR watchdog. The XHR had no timeout while processQueue held the isProcessing latch across it; on a half-open socket neither error nor abort ever fires, so the latch pinned and the queue wedged. Bounds SILENCE rather than total duration — a 500 MB video over a venue uplink legitimately runs 30+ minutes while making steady progress. Rejects as NetworkError, which is already the retryable branch, so a stalled upload now recovers like any network blip. IndexedDB failures. addToQueue called getDb() unguarded and handleSubmit had no catch, so a private-mode refusal or a QuotaExceededError on a large blob left a permanent "Wird hochgeladen…" spinner, no toast, and — for an in-app camera capture — the only copy of the photo gone. Now reported as 'failed', which keeps the staged files on screen and stays on the page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
150 lines
5.4 KiB
YAML
150 lines
5.4 KiB
YAML
# Docker's default json-file driver is UNBOUNDED. Those files land on the HOST
|
|
# filesystem, outside every `deploy.resources.limits` below — so the container memory
|
|
# caps do nothing to stop them. On a single-box deployment the host disk is also where
|
|
# the postgres_data and media_data volumes live, and a full disk stops Postgres writing
|
|
# WAL, which takes the whole event down. 4 services x 3 x 10m caps the worst case at
|
|
# ~120 MiB. Applied to every service via the anchor; a new service must opt in too.
|
|
x-logging: &default-logging
|
|
driver: json-file
|
|
options:
|
|
max-size: "10m"
|
|
max-file: "3"
|
|
|
|
services:
|
|
db:
|
|
image: postgres:16-alpine
|
|
restart: unless-stopped
|
|
logging: *default-logging
|
|
env_file: .env
|
|
environment:
|
|
POSTGRES_USER: ${POSTGRES_USER}
|
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
|
POSTGRES_DB: ${POSTGRES_DB}
|
|
volumes:
|
|
- postgres_data:/var/lib/postgresql/data
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
|
|
interval: 5s
|
|
timeout: 5s
|
|
retries: 10
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
# 1G, not 512M. DATABASE_MAX_CONNECTIONS defaults to 30 for a ~100-guest event
|
|
# (feed polling + SSE + uploads at once), and 30 backends plus Postgres 16's
|
|
# default shared_buffers leaves very little headroom at 512M. An OOM here does
|
|
# not degrade one feature — it takes the event down, because every request
|
|
# path touches the database. Memory is the cheaper knob than shrinking the
|
|
# pool back and reintroducing the queueing it was raised to fix.
|
|
#
|
|
# Raising DATABASE_MAX_CONNECTIONS further means raising this too.
|
|
memory: 1G
|
|
|
|
app:
|
|
build:
|
|
context: ./backend
|
|
dockerfile: Dockerfile
|
|
restart: unless-stopped
|
|
logging: *default-logging
|
|
env_file: .env
|
|
environment:
|
|
# Default to info. The code fallback in main.rs is info too, but a stock deploy
|
|
# sets RUST_LOG nowhere, and this is the layer an operator will actually find when
|
|
# they need to raise it for a single event ("RUST_LOG=eventsnap_backend=debug").
|
|
RUST_LOG: ${RUST_LOG:-info}
|
|
# Activates the production secret guard in config.rs — refuses to boot with
|
|
# placeholder JWT_SECRET / ADMIN_PASSWORD_HASH.
|
|
APP_ENV: production
|
|
# The media volume is mounted at /media (below), so the app MUST write there.
|
|
# Pin it here rather than trusting .env: if MEDIA_PATH in .env points elsewhere
|
|
# (e.g. a host path used for running the backend natively) the container can't
|
|
# create it and every upload 500s with EACCES. `environment` overrides `env_file`,
|
|
# so this is authoritative for the container.
|
|
MEDIA_PATH: /media
|
|
depends_on:
|
|
db:
|
|
condition: service_healthy
|
|
volumes:
|
|
- media_data:/media
|
|
# Export archives live OUTSIDE /media so the public media ServeDir can't
|
|
# serve them — downloads go only through the ticket-gated handler.
|
|
- exports_data:/exports
|
|
expose:
|
|
- "3000"
|
|
healthcheck:
|
|
# Use 127.0.0.1, NOT localhost: the app binds IPv4 (0.0.0.0) but `localhost`
|
|
# resolves to ::1 (IPv6) first inside the container, so a localhost probe gets
|
|
# "connection refused" and the container never turns healthy — which would leave
|
|
# Caddy (gated on `condition: service_healthy` below) blocked forever on boot.
|
|
test: ["CMD-SHELL", "wget -q -O- http://127.0.0.1:3000/health || exit 1"]
|
|
interval: 10s
|
|
timeout: 5s
|
|
retries: 5
|
|
start_period: 20s
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
# Bounds a runaway ffmpeg transcode (large uploads, 2 workers) so it can't
|
|
# OOM the single box and take down Postgres.
|
|
memory: 1G
|
|
|
|
frontend:
|
|
build:
|
|
context: ./frontend
|
|
dockerfile: Dockerfile
|
|
restart: unless-stopped
|
|
logging: *default-logging
|
|
env_file: .env
|
|
environment:
|
|
# adapter-node behind Caddy TLS needs the public origin for CSRF checks on
|
|
# POST form actions — without it they fail only in production.
|
|
ORIGIN: "https://${DOMAIN}"
|
|
depends_on:
|
|
- app
|
|
expose:
|
|
- "3001"
|
|
healthcheck:
|
|
# 127.0.0.1, not localhost — see the app healthcheck note above (IPv4 bind vs
|
|
# ::1 resolution would leave this container permanently unhealthy).
|
|
test: ["CMD-SHELL", "wget -q -O- http://127.0.0.1:3001/ >/dev/null 2>&1 || exit 1"]
|
|
interval: 10s
|
|
timeout: 5s
|
|
retries: 5
|
|
start_period: 15s
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 256M
|
|
|
|
caddy:
|
|
image: caddy:2-alpine
|
|
restart: unless-stopped
|
|
logging: *default-logging
|
|
environment:
|
|
# The Caddyfile's site address is `{$DOMAIN}`, read from THIS container's env.
|
|
# Without it, `{$DOMAIN}` expands to empty, the site block collapses, and Caddy
|
|
# serves nothing / fails to obtain a TLS cert. `env_file` alone wouldn't help —
|
|
# Caddy needs it in `environment`, and this keeps the Caddyfile the single source.
|
|
DOMAIN: ${DOMAIN}
|
|
ports:
|
|
- "80:80"
|
|
- "443:443"
|
|
volumes:
|
|
- ./Caddyfile:/etc/caddy/Caddyfile:ro
|
|
- caddy_data:/data
|
|
depends_on:
|
|
app:
|
|
condition: service_healthy
|
|
frontend:
|
|
condition: service_healthy
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 256M
|
|
|
|
volumes:
|
|
postgres_data:
|
|
media_data:
|
|
exports_data:
|
|
caddy_data:
|