fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(release): the rollback section's facts were wrong, and it is the emergency path

Browse Source 
§6 and §9 both asserted "latest existing tag is v0.12.0". It is v0.17.4 — twelve
tags newer — and HEAD is 82 commits past that. Everything built on that premise
was therefore wrong at the one moment nobody has time to check it.

**The dangerous half.** §9 argued that a rollback to the previous release could
not even get off the ground: "No v0.12.0 image was ever built or pushed either,
so the pre-pull would fail with `manifest unknown` before you ever got that far."
That presents the registry as a safety net. It is not one. Images for the earlier
v0.17.x releases may well be resident — and after §9's own T-2 pre-pull step, on
the server — in which case the rollback pulls cleanly, starts, and crash-loops on
`VersionMissing` (16 migrations against a database carrying 31) behind a live
Caddy. That is the exact permanent outage §9 exists to prevent, reached through
the one door it said was closed.

Rewritten to say what actually decides: the migration set, checked with
`git ls-tree`, never the registry. A successful `docker pull` is not evidence.
The heading changes from "no older image you can roll back to" to "no released
tag is a VALID rollback target" — older images very probably do exist; what none
of them has is a schema the live database can boot against. The twin-tag scheme
below it was always the right answer and is unchanged.

**The version half.** §6 told you to build and push `v0.17.5`... under the name
`v0.13.0`, which has existed as a git tag since April and carries 6 migrations.
That image would disagree with the tree of the same name, and any host still
pinned to that tag silently swaps builds on its next pull. `.env.example` shipped
`EVENTSNAP_VERSION=v0.13.0` too, and README used a third number (`v0.13.1`). All
three now say v0.17.5 (+ the `-a` rollback twin).

**And the commands now reproduce their own output.** The `git tag` invocation
this commit first reached for returned `archive/pre-squash-20260729`, not
v0.17.4 — the repo carries a non-release tag that sorts first. Both call sites
now use `--list 'v*' --sort=-v:refname`, and all three commands in the §9
evidence block were run and produce exactly what the block claims.

Verified: `git tag --list 'v*' --sort=-v:refname | head -1` → v0.17.4;
`git ls-tree --name-only v0.17.4 backend/migrations/ | grep -c up.sql` → 16;
`git rev-list --count v0.17.4..HEAD` → 82.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-08-17 17:54:34 +02:00
parent e645d78a6e
commit d270cfcf91
3 changed files with 55 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

68
DEPLOYMENT_RUNBOOK.md
View File

@@ -180,7 +180,7 @@ EVENT_NAME=<...>
EVENT_SLUG=<...>
# ── Image version (NEW — drives the image: tags in docker-compose.yml) ────
EVENTSNAP_VERSION=v0.13.0
EVENTSNAP_VERSION=v0.17.5
# ── Secrets — ALL of them, before the first `up -d` ───────────────────────
JWT_SECRET=<openssl rand -hex 64>
@@ -421,8 +421,13 @@ the host).
docker run --rm --platform linux/amd64 alpine uname -m # must print x86_64
cd /Users/fabianhammprivat/Projects/EventSnap
VERSION=v0.13.0 # latest existing tag is v0.12.0 — see §9 before reusing it
ROLLBACK=v0.13.0-a # the SAME source, tagged twice; §9 explains why
# Newest EXISTING tag is v0.17.4 (16 migrations); HEAD has 31, so v0.17.5 is the release you cut
# for the event. Confirm before building — `git tag --list 'v*' --sort=-v:refname | head -1`, and
# keep the `--list 'v*'` filter: an unfiltered listing puts the archive/ tag first. Never reuse an
# existing tag: the image name would then disagree with the tree of the same name, and any host
# still pinned to that tag silently swaps builds on its next pull.
VERSION=v0.17.5
ROLLBACK=v0.17.5-a # the SAME source, tagged twice; §9 explains why
SHA=$(git rev-parse --short HEAD)
docker buildx create --name eventsnap --use 2>/dev/null || docker buildx use eventsnap
@@ -572,25 +577,44 @@ Then, from a phone on cellular (not the office wifi):
## 9. Rollback
### There is no older image you can roll back to. Build the rollback target yourself.
### No released tag is a valid rollback target. Build the rollback target yourself.
<!-- Heading deliberately says "no VALID target", not "no older image". Older images very
probably DO exist in the registry — what none of them has is a migration set the live
database can boot against, and that is the property that decides. -->
Read this before the event, not during it. The obvious move — drop `EVENTSNAP_VERSION` back to the
previous released tag — **takes the app down permanently** and looks like a crash loop with no
explanation:
```
$ git ls-tree --name-only v0.12.0 backend/migrations/ | wc -l
12 # 6 migrations. HEAD has 31.
$ git rev-list --count v0.12.0..HEAD
217
# `--list 'v*'` is load-bearing: the repo also carries non-release tags
# (archive/pre-squash-20260729), and an unfiltered listing puts one of them first.
$ git tag --list 'v*' --sort=-v:refname | head -1
v0.17.4
$ git ls-tree --name-only v0.17.4 backend/migrations/ | grep -c up.sql
16 # 16 migrations. HEAD has 31.
$ git rev-list --count v0.17.4..HEAD
82
```
`db.rs` runs `sqlx::migrate!()` with no `set_ignore_missing`, so an image built from a 6-migration
tree, booting against a database that already carries versions 007–031, returns `VersionMissing`.
`db.rs` runs `sqlx::migrate!()` with no `set_ignore_missing`, so an image built from a 16-migration
tree, booting against a database that already carries versions 017–031, returns `VersionMissing`.
`create_pool` errors, `main` exits 1, and `restart: unless-stopped` restarts it forever — with Caddy
still routing traffic to it. (`014_export_epoch.up.sql` documents this failure mode; §0 restates it.)
No `v0.12.0` image was ever built or pushed either, so the pre-pull would fail with
`manifest unknown` before you ever got that far.
**Do not treat a successful `docker pull` as evidence that a tag is safe.** This section used to
claim the previous release had never been built or pushed, so "the pre-pull would fail with
`manifest unknown` before you ever got that far" — i.e. that the registry itself would stop you.
Do not rely on that. Images for the earlier v0.17.x releases may well be resident in the registry
(and, after a §9 pre-pull, on the server), in which case the rollback pulls cleanly, starts, and
crash-loops on `VersionMissing` — the failure this section exists to prevent, reached through the
one door it used to say was closed. The migration set is the only thing that decides whether a tag
is safe, and it is checked by `git ls-tree`, never by the registry.
Every existing tag is older than HEAD's migration set, so **no released tag is a valid rollback
target.** That is what the twin-tag scheme below is for.
**So: at build time, tag the SAME frozen commit twice.** Two identical images, two names. The
rollback then swaps to a binary that is bit-for-bit what you tested and carries the identical
@@ -598,8 +622,8 @@ migration set, which makes it a genuine no-op rather than a gamble:
```bash
# In §6, push both tags from the one build:
VERSION=v0.13.0
ROLLBACK=v0.13.0-a # same source, different name — the rollback target
VERSION=v0.17.5
ROLLBACK=v0.17.5-a # same source, different name — the rollback target
docker buildx build --platform linux/amd64 \
-t registry.mc02.dev/eventsnap/app:$VERSION \
@@ -611,17 +635,17 @@ docker buildx build --platform linux/amd64 \
**Rolling back — ~30 seconds, no network:**
```bash
sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env
sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.17.5-a/' .env
docker compose up -d app frontend
```
This only works offline if both are already resident. **Pre-pull all four at T‑2:**
```bash
docker pull registry.mc02.dev/eventsnap/app:v0.13.0
docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0
docker pull registry.mc02.dev/eventsnap/app:v0.13.0-a
docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0-a
docker pull registry.mc02.dev/eventsnap/app:v0.17.5
docker pull registry.mc02.dev/eventsnap/frontend:v0.17.5
docker pull registry.mc02.dev/eventsnap/app:v0.17.5-a
docker pull registry.mc02.dev/eventsnap/frontend:v0.17.5-a
docker image ls | grep eventsnap # confirm all four
```
@@ -668,8 +692,8 @@ harness — `backend/scripts/rehearse-014.sh`. Run it once against a real dump b
**Registry-down transport fallback** (layers are already compressed — do not add gzip):
```bash
docker save registry.mc02.dev/eventsnap/app:v0.13.0 \
registry.mc02.dev/eventsnap/frontend:v0.13.0 | ssh root@SERVER 'docker load'
docker save registry.mc02.dev/eventsnap/app:v0.17.5 \
registry.mc02.dev/eventsnap/frontend:v0.17.5 | ssh root@SERVER 'docker load'
```
---
@@ -943,7 +967,7 @@ docker compose restart app
# Roll back to the identically-built sibling image — see §9 for what this can and cannot fix.
# Do NOT substitute an older release tag here; it will crash-loop on the migration set.
sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env && docker compose up -d app frontend
sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.17.5-a/' .env && docker compose up -d app frontend
# Disk check
df -h /var/lib/docker