From faf2e62a29f8ba3c2aa5e85676f242fa9e54fa48 Mon Sep 17 00:00:00 2001 From: fabi Date: Tue, 28 Jul 2026 22:34:01 +0200 Subject: [PATCH] fix(deploy): route /health in production, and actually apply Caddyfile changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects in the update procedure I wrote last round, both of which make a successful-looking deploy a lie. 1. The documented health check could never pass. `curl -fsS https://DOMAIN/health` 404s against a perfectly healthy production stack. The backend registers /health on its ROOT router, not under /api/v1, and the production Caddyfile proxies only /api/* and /media/* — so /health fell through to the SvelteKit catch-all, which has no such route and returns its 404 page. With -f, curl exits 22 and the `&& echo` never runs. My own gloss ("Anything other than ok means check the logs") then sent the operator chasing a phantom outage. e2e/Caddyfile.test has carried `reverse_proxy /health app:3000` since it was written — precisely because the catch-all would otherwise swallow it. Production never did. Per the fix-the-gap-not-the-doc call, production gets the same line, and /health joins the no-store matcher so a cached response can't report the last known state instead of the current one. Verified by running the production Caddyfile against the real backend: /health -> 200 "ok", Cache-Control: no-store, with /api/v1/event and / unaffected. 2. The sequence never reloaded Caddy, so a Caddyfile-only change was dropped. `--build` only rebuilds services with a `build:` section, and caddy is a pinned upstream image. Compose decides whether to recreate a container from its config hash, which covers the mount SPECIFICATION but not the mounted file's CONTENTS — so a git pull that changes ./Caddyfile produces no delta, Compose reports `Running`, and Caddy serves its old config indefinitely. Exit code 0 throughout. Round 1's iOS download fix (137c4ee) is exactly this shape: Caddyfile plus four e2e files, so 100% of its production effect is in that one file. Following the README to the letter deployed it, showed both image IDs changing, and left iOS downloads broken. Demonstrated rather than assumed — added a probe header to a Caddyfile, ran the old sequence (`up -d --build`): header absent, change silently dropped. Ran the new step 4 (`up -d --force-recreate caddy`): header served. `--force-recreate` rather than `restart` or `caddy reload` because the bind mount is resolved to an inode at container-create time and git pull replaces the file rather than editing in place, so a restart can re-read the stale content — the exact failure I hit in round 1 when `caddy reload` didn't pick up an edit. Also rewrites the "db and caddy are untouched … so data volumes survive" sentence. I wrote it as reassurance; "caddy is untouched" was the bug. Co-Authored-By: Claude Opus 5 (1M context) --- Caddyfile | 12 ++++++++++-- README.md | 30 +++++++++++++++++++++++++----- 2 files changed, 35 insertions(+), 7 deletions(-) diff --git a/Caddyfile b/Caddyfile index 138ea14..f401ee6 100644 --- a/Caddyfile +++ b/Caddyfile @@ -40,9 +40,10 @@ @media_api path /api/v1/upload/*/preview /api/v1/upload/*/thumbnail header @media_api Cache-Control "private, max-age=300" - # API — never cache, EXCEPT the gated image routes above. + # API and health — never cache, EXCEPT the gated image routes above. A cached health + # response would report the last known state rather than the current one. @api { - path /api/* + path /api/* /health not path /api/v1/upload/*/preview /api/v1/upload/*/thumbnail } header @api Cache-Control "no-store" @@ -58,6 +59,13 @@ reverse_proxy /api/* app:3000 reverse_proxy /media/* app:3000 + # The backend registers /health on its ROOT router, not under /api/v1, so it needs its + # own line — without it the catch-all below hands /health to SvelteKit, which has no + # such route and returns its 404 page. That made the documented post-deploy check + # (`curl -fsS https://DOMAIN/health`) fail 100% of the time on a perfectly healthy + # stack. e2e/Caddyfile.test has always carried this line; production never did. + reverse_proxy /health app:3000 + # Everything else goes to SvelteKit frontend reverse_proxy frontend:3001 } diff --git a/README.md b/README.md index 99d9ac0..33b890d 100644 --- a/README.md +++ b/README.md @@ -132,13 +132,16 @@ cd /path/to/eventsnap # 2. Fetch the new code. git pull -# 3. Rebuild and restart. --build is NOT optional. +# 3. Rebuild and restart the application services. --build is NOT optional. docker compose up -d --build -# 4. Confirm the app came back up. Anything other than "ok" means check the logs. +# 4. Apply any Caddyfile change. Step 3 does NOT do this — see the warning below. +docker compose up -d --force-recreate caddy + +# 5. Confirm the app came back up. Anything other than "ok" means check the logs. curl -fsS https://DOMAIN/health && echo -# 5. Confirm a NEW image was actually built. Note the IMAGE ID before you start and +# 6. Confirm a NEW image was actually built. Note the IMAGE ID before you start and # compare — it must have changed. (Ignore the CREATED column; it reports the base # layer's age, not this build's.) An unchanged ID means step 3 ran without --build # and you are still serving the old code. @@ -151,8 +154,25 @@ that a migration applied by a *newer* build is not removed by checking out an ol so rolling back code without restoring the database snapshot from step 1 leaves the schema ahead of the binary and the app refusing to boot. -Only the two application services rebuild; `db` and `caddy` are pinned upstream images and -are untouched, so data volumes and the TLS certificate survive. +> **Why step 4 exists.** `--build` only rebuilds services that have a `build:` section, and +> `caddy` is a pinned upstream image. Compose decides whether to recreate a container from its +> *config hash*, which covers the mount **specification** (`./Caddyfile:/etc/caddy/Caddyfile:ro`) +> but **not the file's contents** — so a `git pull` that changes `./Caddyfile` produces no +> delta, Compose reports `Running`, and Caddy keeps serving its old config indefinitely. Exit +> code 0 throughout. +> +> That is not hypothetical: the fix that made the keepsake download work on iOS +> (`137c4ee`) touched the Caddyfile and four e2e files and nothing else, so **all** of its +> production effect lives in that one file. Without step 4 you deploy it, watch both image IDs +> change, and iOS downloads stay broken. +> +> `--force-recreate` rather than `restart` or `caddy reload`: the bind mount is resolved to an +> **inode** when the container is created, and `git pull` replaces the file instead of editing +> it in place, so the container can still be bound to the old, now-unlinked inode. A restart +> then re-reads the stale content. Recreating the container re-resolves the path. + +`db` is never touched, and recreating `caddy` does not disturb the `caddy_data` volume, so the +TLS certificate and all data volumes survive. ### Generate required secrets