fix(deploy): ship the swap ceilings, pin the last boot-fatal env var, and correct docs that misdirect
* memswap_limit is now IN docker-compose.yml on all four services. Compose sets Memory but leaves MemorySwap unset, and Docker then permits swap equal to the memory limit — so following §5's "add 2 GB of swap" silently DOUBLED every ceiling, to ~5 GiB on a 3.82 GiB box. Nothing OOMs; instead Postgres's working set becomes swap-eligible on a shared-tenancy SSD, turning a bounded OOM-kill that restarts in seconds into unbounded latency with no signal but "everything is slow". The runbook told the operator to hand-add it, which also broke §0's own gate that docker-compose.yml must be unmodified. Verified rather than assumed: service-level memswap_limit does compose with deploy.resources.limits.memory (docker inspect → Memory=1073741824 MemorySwap=1207959552). * DATABASE_MAX_CONNECTIONS pinned in compose. It is the one env var that is now boot-FATAL when unparseable — the right call, but it means a stray quote or a trailing inline comment in .env crash-loops the app behind a live Caddy. MEDIA_PATH, EXPORT_PATH and APP_PORT are pinned for weaker reasons. * .env.example's quota narrative was sized for a CX33: "~30 GB of a fresh 70 GB" on a box with 40 GB. And on THIS box the fixed point never binds at all — ~210 MB/guest is below the 500 MiB floor, so everyone gets the floor and the per-user quota stops bounding aggregate growth. What actually stops uploads is the keepsake preflight at ~8 GB of media. That paragraph is what an operator reads when a guest is blocked, and it pointed at the wrong knob. * The emergency card gains the one disk symptom that can appear mid-event, where `df -h` — its only disk instruction — actively misleads: the gate fires ~10 GB + 2.2x media BEFORE the disk is full, so df shows ~20 GB free at the moment uploads are being refused. * Two code comments that now assert the opposite of the code: claim_job promised that "the update_progress liveness check bails such a worker out early" — it cannot, its predicate is on the job row, which a reopen does not touch, so a mid-export reopen grinds the whole gallery to completion on a 2-vCPU box during the live event. And prune_superseded_archives still argued "deleted bytes cannot be rolled back" as an invariant, after the reclaim path was changed to prune even when that will not close the shortfall. Both now describe what the code does. * Smaller corrections: runbook §3's "two 48 MP photos ≈ 800 MB" scenario is unreachable (compression.rs takes an exclusive heavy permit, so they serialise) and contradicted .env.example; "all four healthy" is wrong since caddy has no healthcheck; a README line reference pointed at a comment added by the same commit that broke it.
This commit is contained in:
@@ -674,13 +674,16 @@ async fn ensure_export_space_reclaiming(
|
||||
"pruning the previous {prefix} keepsake will not free the full shortfall on its own; \
|
||||
reclaiming anyway — it is already unservable, and the sibling half frees the rest"
|
||||
);
|
||||
} else {
|
||||
// `else`, not a second unconditional line: both used to fire in the shortfall case, and
|
||||
// they read as contradicting each other ("will not free the shortfall" / "reclaiming it
|
||||
// first") to whoever is reading logs at 2am.
|
||||
tracing::warn!(
|
||||
deficit,
|
||||
reclaimable,
|
||||
"not enough room to rebuild {prefix} alongside the previous keepsake; reclaiming it first"
|
||||
);
|
||||
}
|
||||
|
||||
tracing::warn!(
|
||||
deficit,
|
||||
reclaimable,
|
||||
"not enough room to rebuild {prefix} alongside the previous keepsake; reclaiming it first"
|
||||
);
|
||||
prune_superseded_archives(pool, export_path, prefix, event_id, epoch).await;
|
||||
ensure_export_space(pool, event_id, export_path).await
|
||||
}
|
||||
@@ -1566,9 +1569,21 @@ async fn query_hashtags(pool: &PgPool, event_id: Uuid) -> Result<Vec<(Uuid, Stri
|
||||
/// `export_job` (that is the point of the design: one write retires everything). So after a reopen
|
||||
/// the row is still `pending` at our epoch and this claim SUCCEEDS: the worker will build an archive
|
||||
/// nobody can ever see, because retirement is enforced at READ time (`export_current` requires
|
||||
/// `j.epoch = e.export_epoch`), not at write time. That is wasted work, not incorrectness — and the
|
||||
/// `update_progress` liveness check bails such a worker out early. Do not "optimise" this into a
|
||||
/// cross-table check: that is exactly the unsound guard we removed.
|
||||
/// `j.epoch = e.export_epoch`), not at write time. That is wasted work, not incorrectness. Do not
|
||||
/// "optimise" this into a cross-table check: that is exactly the unsound guard we removed.
|
||||
///
|
||||
/// This used to claim that "the `update_progress` liveness check bails such a worker out early".
|
||||
/// IT DOES NOT, and it cannot: `update_progress`'s predicate is `epoch = ours AND status =
|
||||
/// 'running'` on the JOB ROW, which a reopen does not touch — so the check returns true on every
|
||||
/// tick and the worker grinds the whole gallery to completion, every ffmpeg poster and every
|
||||
/// Lanczos3 resize, before its `finalize_job` writes `done` at an epoch nothing reads.
|
||||
///
|
||||
/// The cost is real on a 2-vCPU box: a host reopening the event mid-export — the documented
|
||||
/// "oops, one more photo" path — leaves a full export burning CPU and the heavy-image semaphore
|
||||
/// DURING the live event, and lands a full-gallery-sized orphan that nothing reclaims until the
|
||||
/// next successful build at a higher epoch. Bounded and not corrupting, so it is left as is; but
|
||||
/// the mitigation the old comment promised was never there, and anyone sizing this box should
|
||||
/// know that.
|
||||
///
|
||||
/// Errors are distinguished from a lost claim: silently treating a pool timeout as "someone else
|
||||
/// owns it" left the row `pending` at 0% with no live worker and no error — a spinner forever.
|
||||
@@ -1669,9 +1684,23 @@ async fn protected_files(pool: &PgPool, event_id: Uuid) -> Vec<String> {
|
||||
/// a hung ffmpeg — left the event with NO archive at all, which is the one outcome the product
|
||||
/// exists to prevent, at the one moment nobody is watching.
|
||||
///
|
||||
/// The single exception is phase 2 of `ensure_export_space_reclaiming`, where the previous
|
||||
/// generation's bytes are the only way the rebuild can fit at all. Both call sites carry the full
|
||||
/// reasoning; do not "restore" a pre-build prune on the strength of this function's convenience.
|
||||
/// The single exception is phase 2 of `ensure_export_space_reclaiming`, and BE PRECISE ABOUT WHAT
|
||||
/// THAT EXCEPTION NOW COSTS, because the guarantee above is weaker than it reads. That phase used
|
||||
/// to prune only when the reclaimed bytes would actually close the shortfall. It no longer does:
|
||||
/// `reclaimable` is scoped to one prefix while `deficit` covers both halves plus the reserve, so on
|
||||
/// a tight disk each worker measured its own share as insufficient, neither pruned, and every "Neu
|
||||
/// erzeugen" refused identically — permanently stuck, with dead archives on the volume that nothing
|
||||
/// would reclaim and nothing could serve. It now prunes anyway and lets the re-check decide.
|
||||
///
|
||||
/// The trade that buys: a rebuild can now delete the last physical copy and THEN fail, which is
|
||||
/// precisely the "no archive at all" outcome this doc argues against. It is accepted because the
|
||||
/// refusal it replaces was unrecoverable — deterministic across retries — whereas this failure
|
||||
/// converges once the sibling worker frees its share. But "an epoch can be rolled back, deleted
|
||||
/// bytes cannot" is no longer a guarantee this module provides end to end, and a manual
|
||||
/// `UPDATE event SET export_epoch = <n>` can no longer rescue that case.
|
||||
///
|
||||
/// Both call sites carry the full reasoning; do not "restore" a pre-build prune on the strength of
|
||||
/// this function's convenience.
|
||||
///
|
||||
/// Narrower than [`prune_stale_export_files`] on purpose: FINAL archives only. Those are inert — a
|
||||
/// superseded worker either already renamed its file (and will delete it itself when its guarded
|
||||
|
||||
Reference in New Issue
Block a user