From d270cfcf91c64758a67e6739835abb3a82c29b49 Mon Sep 17 00:00:00 2001 From: MechaCat02 Date: Mon, 17 Aug 2026 17:54:34 +0200 Subject: [PATCH] docs(release): the rollback section's facts were wrong, and it is the emergency path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §6 and §9 both asserted "latest existing tag is v0.12.0". It is v0.17.4 — twelve tags newer — and HEAD is 82 commits past that. Everything built on that premise was therefore wrong at the one moment nobody has time to check it. **The dangerous half.** §9 argued that a rollback to the previous release could not even get off the ground: "No v0.12.0 image was ever built or pushed either, so the pre-pull would fail with `manifest unknown` before you ever got that far." That presents the registry as a safety net. It is not one. Images for the earlier v0.17.x releases may well be resident — and after §9's own T-2 pre-pull step, on the server — in which case the rollback pulls cleanly, starts, and crash-loops on `VersionMissing` (16 migrations against a database carrying 31) behind a live Caddy. That is the exact permanent outage §9 exists to prevent, reached through the one door it said was closed. Rewritten to say what actually decides: the migration set, checked with `git ls-tree`, never the registry. A successful `docker pull` is not evidence. The heading changes from "no older image you can roll back to" to "no released tag is a VALID rollback target" — older images very probably do exist; what none of them has is a schema the live database can boot against. The twin-tag scheme below it was always the right answer and is unchanged. **The version half.** §6 told you to build and push `v0.17.5`... under the name `v0.13.0`, which has existed as a git tag since April and carries 6 migrations. That image would disagree with the tree of the same name, and any host still pinned to that tag silently swaps builds on its next pull. `.env.example` shipped `EVENTSNAP_VERSION=v0.13.0` too, and README used a third number (`v0.13.1`). All three now say v0.17.5 (+ the `-a` rollback twin). **And the commands now reproduce their own output.** The `git tag` invocation this commit first reached for returned `archive/pre-squash-20260729`, not v0.17.4 — the repo carries a non-release tag that sorts first. Both call sites now use `--list 'v*' --sort=-v:refname`, and all three commands in the §9 evidence block were run and produce exactly what the block claims. Verified: `git tag --list 'v*' --sort=-v:refname | head -1` → v0.17.4; `git ls-tree --name-only v0.17.4 backend/migrations/ | grep -c up.sql` → 16; `git rev-list --count v0.17.4..HEAD` → 82. Co-Authored-By: Claude Opus 5 (1M context) --- .env.example | 13 +++++---- DEPLOYMENT_RUNBOOK.md | 68 +++++++++++++++++++++++++++++-------------- README.md | 4 +-- 3 files changed, 55 insertions(+), 30 deletions(-) diff --git a/.env.example b/.env.example index 5fffd0a..b587787 100644 --- a/.env.example +++ b/.env.example @@ -13,15 +13,16 @@ DOMAIN=my-event.example.com # Always an immutable tag, never `latest`: rollback is `EVENTSNAP_VERSION=` # + `docker compose up -d`, which works offline if that image is still resident locally. # -# ⚠ THIS TAG DOES NOT EXIST YET. The newest git tag is v0.12.0; v0.13.0 is the release you -# cut for the event. Build and push it (plus its identical rollback twin v0.13.0-a) BEFORE +# ⚠ THIS TAG DOES NOT EXIST YET. The newest git tag is v0.17.4; v0.17.5 is the release you +# cut for the event. Build and push it (plus its identical rollback twin v0.17.5-a) BEFORE # the first `docker compose up -d` — see DEPLOYMENT_RUNBOOK.md §6 (build) and §9 (rollback). # Copying this file and starting the stack without that step fails with `manifest unknown`. # -# Do NOT "fix" this by dropping back to v0.12.0: no image was ever built for it, and a -# 6-migration tree booting against a 31-migration database returns VersionMissing and -# crash-loops forever behind a live Caddy. §9 covers this in full. -EVENTSNAP_VERSION=v0.13.0 +# Do NOT "fix" this by dropping back to an older release tag. v0.17.4 carries 16 migrations +# against a database that will hold 31, so it returns VersionMissing and crash-loops forever +# behind a live Caddy — and unlike an unbuilt tag, an image for it may well PULL cleanly, so a +# successful `docker pull` is not evidence that the tag is safe. §9 covers this in full. +EVENTSNAP_VERSION=v0.17.5 # ── App server ──────────────────────────────────────────────────────────────── APP_PORT=3000 diff --git a/DEPLOYMENT_RUNBOOK.md b/DEPLOYMENT_RUNBOOK.md index 666e19f..851b935 100644 --- a/DEPLOYMENT_RUNBOOK.md +++ b/DEPLOYMENT_RUNBOOK.md @@ -180,7 +180,7 @@ EVENT_NAME=<...> EVENT_SLUG=<...> # ── Image version (NEW — drives the image: tags in docker-compose.yml) ──── -EVENTSNAP_VERSION=v0.13.0 +EVENTSNAP_VERSION=v0.17.5 # ── Secrets — ALL of them, before the first `up -d` ─────────────────────── JWT_SECRET= @@ -421,8 +421,13 @@ the host). docker run --rm --platform linux/amd64 alpine uname -m # must print x86_64 cd /Users/fabianhammprivat/Projects/EventSnap -VERSION=v0.13.0 # latest existing tag is v0.12.0 — see §9 before reusing it -ROLLBACK=v0.13.0-a # the SAME source, tagged twice; §9 explains why +# Newest EXISTING tag is v0.17.4 (16 migrations); HEAD has 31, so v0.17.5 is the release you cut +# for the event. Confirm before building — `git tag --list 'v*' --sort=-v:refname | head -1`, and +# keep the `--list 'v*'` filter: an unfiltered listing puts the archive/ tag first. Never reuse an +# existing tag: the image name would then disagree with the tree of the same name, and any host +# still pinned to that tag silently swaps builds on its next pull. +VERSION=v0.17.5 +ROLLBACK=v0.17.5-a # the SAME source, tagged twice; §9 explains why SHA=$(git rev-parse --short HEAD) docker buildx create --name eventsnap --use 2>/dev/null || docker buildx use eventsnap @@ -572,25 +577,44 @@ Then, from a phone on cellular (not the office wifi): ## 9. Rollback -### There is no older image you can roll back to. Build the rollback target yourself. +### No released tag is a valid rollback target. Build the rollback target yourself. + + + Read this before the event, not during it. The obvious move — drop `EVENTSNAP_VERSION` back to the previous released tag — **takes the app down permanently** and looks like a crash loop with no explanation: ``` -$ git ls-tree --name-only v0.12.0 backend/migrations/ | wc -l -12 # 6 migrations. HEAD has 31. -$ git rev-list --count v0.12.0..HEAD -217 +# `--list 'v*'` is load-bearing: the repo also carries non-release tags +# (archive/pre-squash-20260729), and an unfiltered listing puts one of them first. +$ git tag --list 'v*' --sort=-v:refname | head -1 +v0.17.4 +$ git ls-tree --name-only v0.17.4 backend/migrations/ | grep -c up.sql +16 # 16 migrations. HEAD has 31. +$ git rev-list --count v0.17.4..HEAD +82 ``` -`db.rs` runs `sqlx::migrate!()` with no `set_ignore_missing`, so an image built from a 6-migration -tree, booting against a database that already carries versions 007–031, returns `VersionMissing`. +`db.rs` runs `sqlx::migrate!()` with no `set_ignore_missing`, so an image built from a 16-migration +tree, booting against a database that already carries versions 017–031, returns `VersionMissing`. `create_pool` errors, `main` exits 1, and `restart: unless-stopped` restarts it forever — with Caddy still routing traffic to it. (`014_export_epoch.up.sql` documents this failure mode; §0 restates it.) -No `v0.12.0` image was ever built or pushed either, so the pre-pull would fail with -`manifest unknown` before you ever got that far. + +**Do not treat a successful `docker pull` as evidence that a tag is safe.** This section used to +claim the previous release had never been built or pushed, so "the pre-pull would fail with +`manifest unknown` before you ever got that far" — i.e. that the registry itself would stop you. +Do not rely on that. Images for the earlier v0.17.x releases may well be resident in the registry +(and, after a §9 pre-pull, on the server), in which case the rollback pulls cleanly, starts, and +crash-loops on `VersionMissing` — the failure this section exists to prevent, reached through the +one door it used to say was closed. The migration set is the only thing that decides whether a tag +is safe, and it is checked by `git ls-tree`, never by the registry. + +Every existing tag is older than HEAD's migration set, so **no released tag is a valid rollback +target.** That is what the twin-tag scheme below is for. **So: at build time, tag the SAME frozen commit twice.** Two identical images, two names. The rollback then swaps to a binary that is bit-for-bit what you tested and carries the identical @@ -598,8 +622,8 @@ migration set, which makes it a genuine no-op rather than a gamble: ```bash # In §6, push both tags from the one build: -VERSION=v0.13.0 -ROLLBACK=v0.13.0-a # same source, different name — the rollback target +VERSION=v0.17.5 +ROLLBACK=v0.17.5-a # same source, different name — the rollback target docker buildx build --platform linux/amd64 \ -t registry.mc02.dev/eventsnap/app:$VERSION \ @@ -611,17 +635,17 @@ docker buildx build --platform linux/amd64 \ **Rolling back — ~30 seconds, no network:** ```bash -sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env +sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.17.5-a/' .env docker compose up -d app frontend ``` This only works offline if both are already resident. **Pre-pull all four at T‑2:** ```bash -docker pull registry.mc02.dev/eventsnap/app:v0.13.0 -docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0 -docker pull registry.mc02.dev/eventsnap/app:v0.13.0-a -docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0-a +docker pull registry.mc02.dev/eventsnap/app:v0.17.5 +docker pull registry.mc02.dev/eventsnap/frontend:v0.17.5 +docker pull registry.mc02.dev/eventsnap/app:v0.17.5-a +docker pull registry.mc02.dev/eventsnap/frontend:v0.17.5-a docker image ls | grep eventsnap # confirm all four ``` @@ -668,8 +692,8 @@ harness — `backend/scripts/rehearse-014.sh`. Run it once against a real dump b **Registry-down transport fallback** (layers are already compressed — do not add gzip): ```bash -docker save registry.mc02.dev/eventsnap/app:v0.13.0 \ - registry.mc02.dev/eventsnap/frontend:v0.13.0 | ssh root@SERVER 'docker load' +docker save registry.mc02.dev/eventsnap/app:v0.17.5 \ + registry.mc02.dev/eventsnap/frontend:v0.17.5 | ssh root@SERVER 'docker load' ``` --- @@ -943,7 +967,7 @@ docker compose restart app # Roll back to the identically-built sibling image — see §9 for what this can and cannot fix. # Do NOT substitute an older release tag here; it will crash-loop on the migration set. -sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env && docker compose up -d app frontend +sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.17.5-a/' .env && docker compose up -d app frontend # Disk check df -h /var/lib/docker diff --git a/README.md b/README.md index b065779..adb8864 100644 --- a/README.md +++ b/README.md @@ -161,7 +161,7 @@ Caddy automatically obtains a Let's Encrypt certificate on first start. The app # ── On your workstation: build and push the new tag ─────────────────────────── # Push the rollback twin at the same time, from the same source — see # DEPLOYMENT_RUNBOOK.md §9 for why an identical second tag is the rollback target. -VERSION=v0.13.1 +VERSION=v0.17.5 docker buildx build --platform linux/amd64 \ -t registry.mc02.dev/eventsnap/app:$VERSION \ -t registry.mc02.dev/eventsnap/app:$VERSION-a --push ./backend @@ -179,7 +179,7 @@ cd /path/to/eventsnap git pull # 3. Point the stack at the new tag. -sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.1/' .env +sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.17.5/' .env # 4. Pull explicitly, BEFORE restarting. A failure here (bad tag, registry down) leaves the # running stack untouched; letting `up -d` discover it takes the app down first.