fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(release): the rollback section's facts were wrong, and it is the emergency path

Browse Source 
§6 and §9 both asserted "latest existing tag is v0.12.0". It is v0.17.4 — twelve
tags newer — and HEAD is 82 commits past that. Everything built on that premise
was therefore wrong at the one moment nobody has time to check it.

**The dangerous half.** §9 argued that a rollback to the previous release could
not even get off the ground: "No v0.12.0 image was ever built or pushed either,
so the pre-pull would fail with `manifest unknown` before you ever got that far."
That presents the registry as a safety net. It is not one. Images for the earlier
v0.17.x releases may well be resident — and after §9's own T-2 pre-pull step, on
the server — in which case the rollback pulls cleanly, starts, and crash-loops on
`VersionMissing` (16 migrations against a database carrying 31) behind a live
Caddy. That is the exact permanent outage §9 exists to prevent, reached through
the one door it said was closed.

Rewritten to say what actually decides: the migration set, checked with
`git ls-tree`, never the registry. A successful `docker pull` is not evidence.
The heading changes from "no older image you can roll back to" to "no released
tag is a VALID rollback target" — older images very probably do exist; what none
of them has is a schema the live database can boot against. The twin-tag scheme
below it was always the right answer and is unchanged.

**The version half.** §6 told you to build and push `v0.17.5`... under the name
`v0.13.0`, which has existed as a git tag since April and carries 6 migrations.
That image would disagree with the tree of the same name, and any host still
pinned to that tag silently swaps builds on its next pull. `.env.example` shipped
`EVENTSNAP_VERSION=v0.13.0` too, and README used a third number (`v0.13.1`). All
three now say v0.17.5 (+ the `-a` rollback twin).

**And the commands now reproduce their own output.** The `git tag` invocation
this commit first reached for returned `archive/pre-squash-20260729`, not
v0.17.4 — the repo carries a non-release tag that sorts first. Both call sites
now use `--list 'v*' --sort=-v:refname`, and all three commands in the §9
evidence block were run and produce exactly what the block claims.

Verified: `git tag --list 'v*' --sort=-v:refname | head -1` → v0.17.4;
`git ls-tree --name-only v0.17.4 backend/migrations/ | grep -c up.sql` → 16;
`git rev-list --count v0.17.4..HEAD` → 82.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-08-17 17:54:34 +02:00
parent e645d78a6e
commit d270cfcf91
3 changed files with 55 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
.env.example
View File

@@ -13,15 +13,16 @@ DOMAIN=my-event.example.com
# Always an immutable tag, never `latest`: rollback is `EVENTSNAP_VERSION=<previous>` # Always an immutable tag, never `latest`: rollback is `EVENTSNAP_VERSION=<previous>`
# + `docker compose up -d`, which works offline if that image is still resident locally. # + `docker compose up -d`, which works offline if that image is still resident locally.
# #
# ⚠ THIS TAG DOES NOT EXIST YET. The newest git tag is v0.12.0; v0.13.0 is the release you # ⚠ THIS TAG DOES NOT EXIST YET. The newest git tag is v0.17.4; v0.17.5 is the release you
# cut for the event. Build and push it (plus its identical rollback twin v0.13.0-a) BEFORE # cut for the event. Build and push it (plus its identical rollback twin v0.17.5-a) BEFORE
# the first `docker compose up -d` — see DEPLOYMENT_RUNBOOK.md §6 (build) and §9 (rollback). # the first `docker compose up -d` — see DEPLOYMENT_RUNBOOK.md §6 (build) and §9 (rollback).
# Copying this file and starting the stack without that step fails with `manifest unknown`. # Copying this file and starting the stack without that step fails with `manifest unknown`.
# #
# Do NOT "fix" this by dropping back to v0.12.0: no image was ever built for it, and a # Do NOT "fix" this by dropping back to an older release tag. v0.17.4 carries 16 migrations
# 6-migration tree booting against a 31-migration database returns VersionMissing and # against a database that will hold 31, so it returns VersionMissing and crash-loops forever
# crash-loops forever behind a live Caddy. §9 covers this in full. # behind a live Caddy — and unlike an unbuilt tag, an image for it may well PULL cleanly, so a
EVENTSNAP_VERSION=v0.13.0 # successful `docker pull` is not evidence that the tag is safe. §9 covers this in full.
EVENTSNAP_VERSION=v0.17.5
# ── App server ──────────────────────────────────────────────────────────────── # ── App server ────────────────────────────────────────────────────────────────
APP_PORT=3000 APP_PORT=3000

68
DEPLOYMENT_RUNBOOK.md
View File

@@ -180,7 +180,7 @@ EVENT_NAME=<...>
EVENT_SLUG=<...> EVENT_SLUG=<...>
# ── Image version (NEW — drives the image: tags in docker-compose.yml) ──── # ── Image version (NEW — drives the image: tags in docker-compose.yml) ────
EVENTSNAP_VERSION=v0.13.0 EVENTSNAP_VERSION=v0.17.5
# ── Secrets — ALL of them, before the first `up -d` ─────────────────────── # ── Secrets — ALL of them, before the first `up -d` ───────────────────────
JWT_SECRET=<openssl rand -hex 64> JWT_SECRET=<openssl rand -hex 64>
@@ -421,8 +421,13 @@ the host).
docker run --rm --platform linux/amd64 alpine uname -m # must print x86_64 docker run --rm --platform linux/amd64 alpine uname -m # must print x86_64
cd /Users/fabianhammprivat/Projects/EventSnap cd /Users/fabianhammprivat/Projects/EventSnap
VERSION=v0.13.0 # latest existing tag is v0.12.0 — see §9 before reusing it # Newest EXISTING tag is v0.17.4 (16 migrations); HEAD has 31, so v0.17.5 is the release you cut
ROLLBACK=v0.13.0-a # the SAME source, tagged twice; §9 explains why # for the event. Confirm before building — `git tag --list 'v*' --sort=-v:refname | head -1`, and
# keep the `--list 'v*'` filter: an unfiltered listing puts the archive/ tag first. Never reuse an
# existing tag: the image name would then disagree with the tree of the same name, and any host
# still pinned to that tag silently swaps builds on its next pull.
VERSION=v0.17.5
ROLLBACK=v0.17.5-a # the SAME source, tagged twice; §9 explains why
SHA=$(git rev-parse --short HEAD) SHA=$(git rev-parse --short HEAD)
docker buildx create --name eventsnap --use 2>/dev/null || docker buildx use eventsnap docker buildx create --name eventsnap --use 2>/dev/null || docker buildx use eventsnap
@@ -572,25 +577,44 @@ Then, from a phone on cellular (not the office wifi):
## 9. Rollback ## 9. Rollback
### There is no older image you can roll back to. Build the rollback target yourself. ### No released tag is a valid rollback target. Build the rollback target yourself.
<!-- Heading deliberately says "no VALID target", not "no older image". Older images very
probably DO exist in the registry — what none of them has is a migration set the live
database can boot against, and that is the property that decides. -->
Read this before the event, not during it. The obvious move — drop `EVENTSNAP_VERSION` back to the Read this before the event, not during it. The obvious move — drop `EVENTSNAP_VERSION` back to the
previous released tag — **takes the app down permanently** and looks like a crash loop with no previous released tag — **takes the app down permanently** and looks like a crash loop with no
explanation: explanation:
``` ```
$ git ls-tree --name-only v0.12.0 backend/migrations/ | wc -l # `--list 'v*'` is load-bearing: the repo also carries non-release tags
12 # 6 migrations. HEAD has 31. # (archive/pre-squash-20260729), and an unfiltered listing puts one of them first.
$ git rev-list --count v0.12.0..HEAD $ git tag --list 'v*' --sort=-v:refname | head -1
217 v0.17.4
$ git ls-tree --name-only v0.17.4 backend/migrations/ | grep -c up.sql
16 # 16 migrations. HEAD has 31.
$ git rev-list --count v0.17.4..HEAD
82
``` ```
`db.rs` runs `sqlx::migrate!()` with no `set_ignore_missing`, so an image built from a 6-migration `db.rs` runs `sqlx::migrate!()` with no `set_ignore_missing`, so an image built from a 16-migration
tree, booting against a database that already carries versions 007–031, returns `VersionMissing`. tree, booting against a database that already carries versions 017–031, returns `VersionMissing`.
`create_pool` errors, `main` exits 1, and `restart: unless-stopped` restarts it forever — with Caddy `create_pool` errors, `main` exits 1, and `restart: unless-stopped` restarts it forever — with Caddy
still routing traffic to it. (`014_export_epoch.up.sql` documents this failure mode; §0 restates it.) still routing traffic to it. (`014_export_epoch.up.sql` documents this failure mode; §0 restates it.)
No `v0.12.0` image was ever built or pushed either, so the pre-pull would fail with
`manifest unknown` before you ever got that far. **Do not treat a successful `docker pull` as evidence that a tag is safe.** This section used to
claim the previous release had never been built or pushed, so "the pre-pull would fail with
`manifest unknown` before you ever got that far" — i.e. that the registry itself would stop you.
Do not rely on that. Images for the earlier v0.17.x releases may well be resident in the registry
(and, after a §9 pre-pull, on the server), in which case the rollback pulls cleanly, starts, and
crash-loops on `VersionMissing` — the failure this section exists to prevent, reached through the
one door it used to say was closed. The migration set is the only thing that decides whether a tag
is safe, and it is checked by `git ls-tree`, never by the registry.
Every existing tag is older than HEAD's migration set, so **no released tag is a valid rollback
target.** That is what the twin-tag scheme below is for.
**So: at build time, tag the SAME frozen commit twice.** Two identical images, two names. The **So: at build time, tag the SAME frozen commit twice.** Two identical images, two names. The
rollback then swaps to a binary that is bit-for-bit what you tested and carries the identical rollback then swaps to a binary that is bit-for-bit what you tested and carries the identical
@@ -598,8 +622,8 @@ migration set, which makes it a genuine no-op rather than a gamble:
```bash ```bash
# In §6, push both tags from the one build: # In §6, push both tags from the one build:
VERSION=v0.13.0 VERSION=v0.17.5
ROLLBACK=v0.13.0-a # same source, different name — the rollback target ROLLBACK=v0.17.5-a # same source, different name — the rollback target
docker buildx build --platform linux/amd64 \ docker buildx build --platform linux/amd64 \
-t registry.mc02.dev/eventsnap/app:$VERSION \ -t registry.mc02.dev/eventsnap/app:$VERSION \
@@ -611,17 +635,17 @@ docker buildx build --platform linux/amd64 \
**Rolling back — ~30 seconds, no network:** **Rolling back — ~30 seconds, no network:**
```bash ```bash
sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.17.5-a/' .env
docker compose up -d app frontend docker compose up -d app frontend
``` ```
This only works offline if both are already resident. **Pre-pull all four at T‑2:** This only works offline if both are already resident. **Pre-pull all four at T‑2:**
```bash ```bash
docker pull registry.mc02.dev/eventsnap/app:v0.13.0 docker pull registry.mc02.dev/eventsnap/app:v0.17.5
docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0 docker pull registry.mc02.dev/eventsnap/frontend:v0.17.5
docker pull registry.mc02.dev/eventsnap/app:v0.13.0-a docker pull registry.mc02.dev/eventsnap/app:v0.17.5-a
docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0-a docker pull registry.mc02.dev/eventsnap/frontend:v0.17.5-a
docker image ls | grep eventsnap # confirm all four docker image ls | grep eventsnap # confirm all four
``` ```
@@ -668,8 +692,8 @@ harness — `backend/scripts/rehearse-014.sh`. Run it once against a real dump b
**Registry-down transport fallback** (layers are already compressed — do not add gzip): **Registry-down transport fallback** (layers are already compressed — do not add gzip):
```bash ```bash
docker save registry.mc02.dev/eventsnap/app:v0.13.0 \ docker save registry.mc02.dev/eventsnap/app:v0.17.5 \
registry.mc02.dev/eventsnap/frontend:v0.13.0 | ssh root@SERVER 'docker load' registry.mc02.dev/eventsnap/frontend:v0.17.5 | ssh root@SERVER 'docker load'
``` ```
--- ---
@@ -943,7 +967,7 @@ docker compose restart app
# Roll back to the identically-built sibling image — see §9 for what this can and cannot fix. # Roll back to the identically-built sibling image — see §9 for what this can and cannot fix.
# Do NOT substitute an older release tag here; it will crash-loop on the migration set. # Do NOT substitute an older release tag here; it will crash-loop on the migration set.
sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env && docker compose up -d app frontend sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.17.5-a/' .env && docker compose up -d app frontend
# Disk check # Disk check
df -h /var/lib/docker df -h /var/lib/docker

4
README.md
View File

@@ -161,7 +161,7 @@ Caddy automatically obtains a Let's Encrypt certificate on first start. The app
# ── On your workstation: build and push the new tag ─────────────────────────── # ── On your workstation: build and push the new tag ───────────────────────────
# Push the rollback twin at the same time, from the same source — see # Push the rollback twin at the same time, from the same source — see
# DEPLOYMENT_RUNBOOK.md §9 for why an identical second tag is the rollback target. # DEPLOYMENT_RUNBOOK.md §9 for why an identical second tag is the rollback target.
VERSION=v0.13.1 VERSION=v0.17.5
docker buildx build --platform linux/amd64 \ docker buildx build --platform linux/amd64 \
-t registry.mc02.dev/eventsnap/app:$VERSION \ -t registry.mc02.dev/eventsnap/app:$VERSION \
-t registry.mc02.dev/eventsnap/app:$VERSION-a --push ./backend -t registry.mc02.dev/eventsnap/app:$VERSION-a --push ./backend
@@ -179,7 +179,7 @@ cd /path/to/eventsnap
git pull git pull
# 3. Point the stack at the new tag. # 3. Point the stack at the new tag.
sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.1/' .env sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.17.5/' .env
# 4. Pull explicitly, BEFORE restarting. A failure here (bad tag, registry down) leaves the # 4. Pull explicitly, BEFORE restarting. A failure here (bad tag, registry down) leaves the
# running stack untouched; letting `up -d` discover it takes the app down first. # running stack untouched; letting `up -d` discover it takes the app down first.