fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(deploy): route /health in production, and actually apply Caddyfile changes

Browse Source 
Two defects in the update procedure I wrote last round, both of which make a
successful-looking deploy a lie.

1. The documented health check could never pass.

`curl -fsS https://DOMAIN/health` 404s against a perfectly healthy production
stack. The backend registers /health on its ROOT router, not under /api/v1, and
the production Caddyfile proxies only /api/* and /media/* — so /health fell
through to the SvelteKit catch-all, which has no such route and returns its 404
page. With -f, curl exits 22 and the `&& echo` never runs. My own gloss
("Anything other than ok means check the logs") then sent the operator chasing a
phantom outage.

e2e/Caddyfile.test has carried `reverse_proxy /health app:3000` since it was
written — precisely because the catch-all would otherwise swallow it. Production
never did. Per the fix-the-gap-not-the-doc call, production gets the same line,
and /health joins the no-store matcher so a cached response can't report the last
known state instead of the current one. Verified by running the production
Caddyfile against the real backend: /health -> 200 "ok", Cache-Control: no-store,
with /api/v1/event and / unaffected.

2. The sequence never reloaded Caddy, so a Caddyfile-only change was dropped.

`--build` only rebuilds services with a `build:` section, and caddy is a pinned
upstream image. Compose decides whether to recreate a container from its config
hash, which covers the mount SPECIFICATION but not the mounted file's CONTENTS —
so a git pull that changes ./Caddyfile produces no delta, Compose reports
`Running`, and Caddy serves its old config indefinitely. Exit code 0 throughout.

Round 1's iOS download fix (137c4ee) is exactly this shape: Caddyfile plus four
e2e files, so 100% of its production effect is in that one file. Following the
README to the letter deployed it, showed both image IDs changing, and left iOS
downloads broken.

Demonstrated rather than assumed — added a probe header to a Caddyfile, ran the
old sequence (`up -d --build`): header absent, change silently dropped. Ran the
new step 4 (`up -d --force-recreate caddy`): header served.

`--force-recreate` rather than `restart` or `caddy reload` because the bind mount
is resolved to an inode at container-create time and git pull replaces the file
rather than editing in place, so a restart can re-read the stale content — the
exact failure I hit in round 1 when `caddy reload` didn't pick up an edit.

Also rewrites the "db and caddy are untouched … so data volumes survive" sentence.
I wrote it as reassurance; "caddy is untouched" was the bug.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
fabi
2026-07-28 22:34:01 +02:00
parent 6d5c488e14
commit faf2e62a29
2 changed files with 35 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
Caddyfile
View File

@@ -40,9 +40,10 @@
@media_api path /api/v1/upload/*/preview /api/v1/upload/*/thumbnail
header @media_api Cache-Control "private, max-age=300"
# API — never cache, EXCEPT the gated image routes above.
# API and health — never cache, EXCEPT the gated image routes above. A cached health
# response would report the last known state rather than the current one.
@api {
path /api/*
path /api/* /health
not path /api/v1/upload/*/preview /api/v1/upload/*/thumbnail
}
header @api Cache-Control "no-store"
@@ -58,6 +59,13 @@
reverse_proxy /api/* app:3000
reverse_proxy /media/* app:3000
# The backend registers /health on its ROOT router, not under /api/v1, so it needs its
# own line — without it the catch-all below hands /health to SvelteKit, which has no
# such route and returns its 404 page. That made the documented post-deploy check
# (`curl -fsS https://DOMAIN/health`) fail 100% of the time on a perfectly healthy
# stack. e2e/Caddyfile.test has always carried this line; production never did.
reverse_proxy /health app:3000
# Everything else goes to SvelteKit frontend
reverse_proxy frontend:3001
}

30
README.md
View File

@@ -132,13 +132,16 @@ cd /path/to/eventsnap
# 2. Fetch the new code.
git pull
# 3. Rebuild and restart. --build is NOT optional.
# 3. Rebuild and restart the application services. --build is NOT optional.
docker compose up -d --build
# 4. Confirm the app came back up. Anything other than "ok" means check the logs.
# 4. Apply any Caddyfile change. Step 3 does NOT do this — see the warning below.
docker compose up -d --force-recreate caddy
# 5. Confirm the app came back up. Anything other than "ok" means check the logs.
curl -fsS https://DOMAIN/health && echo
# 5. Confirm a NEW image was actually built. Note the IMAGE ID before you start and
# 6. Confirm a NEW image was actually built. Note the IMAGE ID before you start and
# compare — it must have changed. (Ignore the CREATED column; it reports the base
# layer's age, not this build's.) An unchanged ID means step 3 ran without --build
# and you are still serving the old code.
@@ -151,8 +154,25 @@ that a migration applied by a *newer* build is not removed by checking out an ol
so rolling back code without restoring the database snapshot from step 1 leaves the schema
ahead of the binary and the app refusing to boot.
Only the two application services rebuild; `db` and `caddy` are pinned upstream images and
are untouched, so data volumes and the TLS certificate survive.
> **Why step 4 exists.** `--build` only rebuilds services that have a `build:` section, and
> `caddy` is a pinned upstream image. Compose decides whether to recreate a container from its
> *config hash*, which covers the mount **specification** (`./Caddyfile:/etc/caddy/Caddyfile:ro`)
> but **not the file's contents** — so a `git pull` that changes `./Caddyfile` produces no
> delta, Compose reports `Running`, and Caddy keeps serving its old config indefinitely. Exit
> code 0 throughout.
>
> That is not hypothetical: the fix that made the keepsake download work on iOS
> (`137c4ee`) touched the Caddyfile and four e2e files and nothing else, so **all** of its
> production effect lives in that one file. Without step 4 you deploy it, watch both image IDs
> change, and iOS downloads stay broken.
>
> `--force-recreate` rather than `restart` or `caddy reload`: the bind mount is resolved to an
> **inode** when the container is created, and `git pull` replaces the file instead of editing
> it in place, so the container can still be bound to the old, now-unlinked inode. A restart
> then re-reads the stale content. Recreating the container re-resolves the path.
`db` is never touched, and recreating `caddy` does not disturb the `caddy_data` volume, so the
TLS certificate and all data volumes survive.
### Generate required secrets