fix(deploy): route /health in production, and actually apply Caddyfile changes
Two defects in the update procedure I wrote last round, both of which make a
successful-looking deploy a lie.
1. The documented health check could never pass.
`curl -fsS https://DOMAIN/health` 404s against a perfectly healthy production
stack. The backend registers /health on its ROOT router, not under /api/v1, and
the production Caddyfile proxies only /api/* and /media/* — so /health fell
through to the SvelteKit catch-all, which has no such route and returns its 404
page. With -f, curl exits 22 and the `&& echo` never runs. My own gloss
("Anything other than ok means check the logs") then sent the operator chasing a
phantom outage.
e2e/Caddyfile.test has carried `reverse_proxy /health app:3000` since it was
written — precisely because the catch-all would otherwise swallow it. Production
never did. Per the fix-the-gap-not-the-doc call, production gets the same line,
and /health joins the no-store matcher so a cached response can't report the last
known state instead of the current one. Verified by running the production
Caddyfile against the real backend: /health -> 200 "ok", Cache-Control: no-store,
with /api/v1/event and / unaffected.
2. The sequence never reloaded Caddy, so a Caddyfile-only change was dropped.
`--build` only rebuilds services with a `build:` section, and caddy is a pinned
upstream image. Compose decides whether to recreate a container from its config
hash, which covers the mount SPECIFICATION but not the mounted file's CONTENTS —
so a git pull that changes ./Caddyfile produces no delta, Compose reports
`Running`, and Caddy serves its old config indefinitely. Exit code 0 throughout.
Round 1's iOS download fix (137c4ee) is exactly this shape: Caddyfile plus four
e2e files, so 100% of its production effect is in that one file. Following the
README to the letter deployed it, showed both image IDs changing, and left iOS
downloads broken.
Demonstrated rather than assumed — added a probe header to a Caddyfile, ran the
old sequence (`up -d --build`): header absent, change silently dropped. Ran the
new step 4 (`up -d --force-recreate caddy`): header served.
`--force-recreate` rather than `restart` or `caddy reload` because the bind mount
is resolved to an inode at container-create time and git pull replaces the file
rather than editing in place, so a restart can re-read the stale content — the
exact failure I hit in round 1 when `caddy reload` didn't pick up an edit.
Also rewrites the "db and caddy are untouched … so data volumes survive" sentence.
I wrote it as reassurance; "caddy is untouched" was the bug.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
12
Caddyfile
12
Caddyfile
@@ -40,9 +40,10 @@
|
|||||||
@media_api path /api/v1/upload/*/preview /api/v1/upload/*/thumbnail
|
@media_api path /api/v1/upload/*/preview /api/v1/upload/*/thumbnail
|
||||||
header @media_api Cache-Control "private, max-age=300"
|
header @media_api Cache-Control "private, max-age=300"
|
||||||
|
|
||||||
# API — never cache, EXCEPT the gated image routes above.
|
# API and health — never cache, EXCEPT the gated image routes above. A cached health
|
||||||
|
# response would report the last known state rather than the current one.
|
||||||
@api {
|
@api {
|
||||||
path /api/*
|
path /api/* /health
|
||||||
not path /api/v1/upload/*/preview /api/v1/upload/*/thumbnail
|
not path /api/v1/upload/*/preview /api/v1/upload/*/thumbnail
|
||||||
}
|
}
|
||||||
header @api Cache-Control "no-store"
|
header @api Cache-Control "no-store"
|
||||||
@@ -58,6 +59,13 @@
|
|||||||
reverse_proxy /api/* app:3000
|
reverse_proxy /api/* app:3000
|
||||||
reverse_proxy /media/* app:3000
|
reverse_proxy /media/* app:3000
|
||||||
|
|
||||||
|
# The backend registers /health on its ROOT router, not under /api/v1, so it needs its
|
||||||
|
# own line — without it the catch-all below hands /health to SvelteKit, which has no
|
||||||
|
# such route and returns its 404 page. That made the documented post-deploy check
|
||||||
|
# (`curl -fsS https://DOMAIN/health`) fail 100% of the time on a perfectly healthy
|
||||||
|
# stack. e2e/Caddyfile.test has always carried this line; production never did.
|
||||||
|
reverse_proxy /health app:3000
|
||||||
|
|
||||||
# Everything else goes to SvelteKit frontend
|
# Everything else goes to SvelteKit frontend
|
||||||
reverse_proxy frontend:3001
|
reverse_proxy frontend:3001
|
||||||
}
|
}
|
||||||
|
|||||||
30
README.md
30
README.md
@@ -132,13 +132,16 @@ cd /path/to/eventsnap
|
|||||||
# 2. Fetch the new code.
|
# 2. Fetch the new code.
|
||||||
git pull
|
git pull
|
||||||
|
|
||||||
# 3. Rebuild and restart. --build is NOT optional.
|
# 3. Rebuild and restart the application services. --build is NOT optional.
|
||||||
docker compose up -d --build
|
docker compose up -d --build
|
||||||
|
|
||||||
# 4. Confirm the app came back up. Anything other than "ok" means check the logs.
|
# 4. Apply any Caddyfile change. Step 3 does NOT do this — see the warning below.
|
||||||
|
docker compose up -d --force-recreate caddy
|
||||||
|
|
||||||
|
# 5. Confirm the app came back up. Anything other than "ok" means check the logs.
|
||||||
curl -fsS https://DOMAIN/health && echo
|
curl -fsS https://DOMAIN/health && echo
|
||||||
|
|
||||||
# 5. Confirm a NEW image was actually built. Note the IMAGE ID before you start and
|
# 6. Confirm a NEW image was actually built. Note the IMAGE ID before you start and
|
||||||
# compare — it must have changed. (Ignore the CREATED column; it reports the base
|
# compare — it must have changed. (Ignore the CREATED column; it reports the base
|
||||||
# layer's age, not this build's.) An unchanged ID means step 3 ran without --build
|
# layer's age, not this build's.) An unchanged ID means step 3 ran without --build
|
||||||
# and you are still serving the old code.
|
# and you are still serving the old code.
|
||||||
@@ -151,8 +154,25 @@ that a migration applied by a *newer* build is not removed by checking out an ol
|
|||||||
so rolling back code without restoring the database snapshot from step 1 leaves the schema
|
so rolling back code without restoring the database snapshot from step 1 leaves the schema
|
||||||
ahead of the binary and the app refusing to boot.
|
ahead of the binary and the app refusing to boot.
|
||||||
|
|
||||||
Only the two application services rebuild; `db` and `caddy` are pinned upstream images and
|
> **Why step 4 exists.** `--build` only rebuilds services that have a `build:` section, and
|
||||||
are untouched, so data volumes and the TLS certificate survive.
|
> `caddy` is a pinned upstream image. Compose decides whether to recreate a container from its
|
||||||
|
> *config hash*, which covers the mount **specification** (`./Caddyfile:/etc/caddy/Caddyfile:ro`)
|
||||||
|
> but **not the file's contents** — so a `git pull` that changes `./Caddyfile` produces no
|
||||||
|
> delta, Compose reports `Running`, and Caddy keeps serving its old config indefinitely. Exit
|
||||||
|
> code 0 throughout.
|
||||||
|
>
|
||||||
|
> That is not hypothetical: the fix that made the keepsake download work on iOS
|
||||||
|
> (`137c4ee`) touched the Caddyfile and four e2e files and nothing else, so **all** of its
|
||||||
|
> production effect lives in that one file. Without step 4 you deploy it, watch both image IDs
|
||||||
|
> change, and iOS downloads stay broken.
|
||||||
|
>
|
||||||
|
> `--force-recreate` rather than `restart` or `caddy reload`: the bind mount is resolved to an
|
||||||
|
> **inode** when the container is created, and `git pull` replaces the file instead of editing
|
||||||
|
> it in place, so the container can still be bound to the old, now-unlinked inode. A restart
|
||||||
|
> then re-reads the stale content. Recreating the container re-resolves the path.
|
||||||
|
|
||||||
|
`db` is never touched, and recreating `caddy` does not disturb the `caddy_data` volume, so the
|
||||||
|
TLS certificate and all data volumes survive.
|
||||||
|
|
||||||
### Generate required secrets
|
### Generate required secrets
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user