fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(deploy): close the POSTGRES_PASSWORD trap that broke a fresh deploy either way

Browse Source 
README step 2 said to set "DOMAIN, JWT_SECRET, ADMIN_PASSWORD_HASH, EVENT_NAME, etc."
-- POSTGRES_PASSWORD was not in that list. .env.example said to set it and keep it in
sync with DATABASE_URL. The two documents disagreed, and both readings ended badly.

Branch A, README verbatim: the stack came up GREEN and healthy on
CHANGE_ME_use_a_strong_password -- a database credential published in the public repo.
The production secret guard covered JWT_SECRET and ADMIN_PASSWORD_HASH, and nothing
anywhere looked at the Postgres password.

Branch B, .env.example verbatim: a permanent restart loop, "password authentication
failed for user eventsnap".

The guard's own design made Branch B near-certain. It stops the APP on the first
`docker compose up -d` -- but not the `db` service in that same command, which
initialises its data directory and bakes in whatever password was in .env at that
moment. POSTGRES_PASSWORD is honoured ONLY at initdb. So the intended recovery -- see
the refusal, fix your secrets, boot again -- was exactly the sequence that broke it.
Nothing in the error named the cause, and the remedy (`down -v`) is both unguessable
and the one command you must never run once real data exists.

Three changes, which have to ship together: the guard alone would just move operators
out of Branch A and into Branch B.

- The guard now rejects a placeholder DATABASE_URL in production (the password rides
  in that URL, which is what the app actually reads). Branch A can no longer boot.
- It reports EVERY unset secret in one message instead of returning on the first.
  Fixing two secrets used to cost two boot cycles, on a stack where Caddy waits on the
  unhealthy app throughout, and each avoidable cycle is another chance to reach for -v.
- A 28P01 handler in db.rs turns the unguessable failure into a self-explaining one:
  it names the initdb semantics, gives `down -v` with an explicit "deletes db + media +
  exports, no undo", and gives the ALTER ROLE alternative for when data already exists.

Docs: step 2 now names POSTGRES_PASSWORD and says every secret must be set BEFORE the
first up; the troubleshooting block covers the auth-failure loop and both remedies.

Also replaces htpasswd (apache2-utils -- not on a stock VPS) with
`docker run --rm caddy:2-alpine caddy hash-password`, an image the stack already pulls,
in README, .env.example and the guard's own message. Verified the output ($2a$14)
against the shipped $2y$12 example.

Verified on a real Postgres, not just in tests: Branch A refuses and names DATABASE_URL;
all three placeholders report in one boot; a volume initialised with one password and
connected to with another prints the diagnostic; an unrelated connect failure (dead
port) stays silent. 8 unit tests, including that non-prod ignores all of it so the e2e
stack is unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
fabi
2026-07-30 19:28:34 +02:00
parent 08d92b7531
commit 3bcb7c6a76
4 changed files with 265 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files

180
backend/src/config.rs
View File

@@ -20,21 +20,53 @@ fn looks_placeholder(s: &str) -> bool {
/// Enforce secret hygiene. In production every guard is hard-fail: a booting app
/// with a publicly-known signing key is worse than one that refuses to start.
/// Outside production the dev sentinel is tolerated (warned) so local dev is frictionless.
fn validate_secrets(is_prod: bool, jwt_secret: &str, admin_password_hash: &str) -> Result<()> {
///
/// EVERY failure is collected and reported together. Returning on the first one made fixing two
/// secrets cost two boot cycles — the operator rotates JWT_SECRET, restarts, and only then learns
/// about ADMIN_PASSWORD_HASH. Restarting this stack is not free (Caddy waits on the unhealthy app),
/// and each avoidable cycle is another chance to reach for `down -v`.
fn validate_secrets(
is_prod: bool,
jwt_secret: &str,
admin_password_hash: &str,
database_url: &str,
) -> Result<()> {
if is_prod {
let mut problems: Vec<&str> = Vec::new();
if looks_placeholder(jwt_secret) {
return Err(anyhow!(
"Refusing to start in production with a placeholder JWT_SECRET — \
rotate it (openssl rand -hex 64)."
));
}
if jwt_secret.len() < 32 {
return Err(anyhow!("JWT_SECRET must be at least 32 characters."));
problems.push(
"JWT_SECRET is still the .env.example placeholder — rotate it \
(openssl rand -hex 64).",
);
} else if jwt_secret.len() < 32 {
problems.push("JWT_SECRET must be at least 32 characters.");
}
if admin_password_hash.is_empty() || looks_placeholder(admin_password_hash) {
problems.push(
"ADMIN_PASSWORD_HASH is unset or still the .env.example placeholder — generate one \
(docker run --rm caddy:2-alpine caddy hash-password --plaintext '<password>').",
);
}
// The DATABASE_URL carries the Postgres password, so a placeholder here means the stack is
// running on `CHANGE_ME_use_a_strong_password` — a credential published in the repo. The
// app used to boot green on it, because this guard only ever covered the two secrets it
// was written for and nothing else looked at POSTGRES_PASSWORD at all.
//
// Read POSTGRES_PASSWORD's docs before changing this: it is applied ONLY at initdb, so the
// remedy is not "edit .env and restart" — see the 28P01 diagnostic in db.rs.
if looks_placeholder(database_url) {
problems.push(
"DATABASE_URL still carries the .env.example placeholder password — set a strong \
one (openssl rand -hex 24) in BOTH DATABASE_URL and POSTGRES_PASSWORD.",
);
}
if !problems.is_empty() {
return Err(anyhow!(
"Refusing to start in production without a real ADMIN_PASSWORD_HASH — \
generate one (htpasswd -bnBC 12 '' <password> | tr -d ':\\n')."
"Refusing to start in production — {} secret(s) still unset or placeholder:\n - {}\n\
ALL secrets must be set BEFORE the first `docker compose up -d`: Postgres bakes \
POSTGRES_PASSWORD into its data directory on first boot and ignores later changes.",
problems.len(),
problems.join("\n - ")
));
}
} else if jwt_secret == DEV_JWT_SECRET_SENTINEL {
@@ -89,11 +121,12 @@ impl AppConfig {
let jwt_secret = std::env::var("JWT_SECRET").context("JWT_SECRET must be set")?;
let admin_password_hash = std::env::var("ADMIN_PASSWORD_HASH").unwrap_or_default();
let database_url = std::env::var("DATABASE_URL").context("DATABASE_URL must be set")?;
validate_secrets(is_prod, &jwt_secret, &admin_password_hash)?;
validate_secrets(is_prod, &jwt_secret, &admin_password_hash, &database_url)?;
Ok(Self {
database_url: std::env::var("DATABASE_URL").context("DATABASE_URL must be set")?,
database_url,
jwt_secret,
session_expiry_days: std::env::var("SESSION_EXPIRY_DAYS")
.unwrap_or_else(|_| "30".to_string())
@@ -141,12 +174,18 @@ mod tests {
const REAL_SECRET: &str = "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2";
const REAL_HASH: &str = "$2y$12$abcdefghijklmnopqrstuv.wxyzABCDEFGHIJKLMNOPQRSTUVWXYZ012";
const REAL_DB_URL: &str = "postgres://eventsnap:7f3a9c1e5b2d8a4f@db:5432/eventsnap";
#[test]
fn prod_rejects_shipped_placeholder_secret() {
// The exact string shipped in `.env` — >32 chars, so it must be caught by
// the substring guard, not the length check.
let err = validate_secrets(true, "change_me_to_a_random_64_byte_hex_string", REAL_HASH);
let err = validate_secrets(
true,
"change_me_to_a_random_64_byte_hex_string",
REAL_HASH,
REAL_DB_URL,
);
assert!(
err.is_err(),
"placeholder JWT_SECRET must be rejected in prod"
@@ -155,29 +194,108 @@ mod tests {
#[test]
fn prod_rejects_dev_sentinel_and_short_secret() {
assert!(validate_secrets(true, DEV_JWT_SECRET_SENTINEL, REAL_HASH).is_err());
assert!(validate_secrets(true, "tooshort", REAL_HASH).is_err());
assert!(validate_secrets(true, DEV_JWT_SECRET_SENTINEL, REAL_HASH, REAL_DB_URL).is_err());
assert!(validate_secrets(true, "tooshort", REAL_HASH, REAL_DB_URL).is_err());
}
#[test]
fn prod_rejects_missing_or_placeholder_admin_hash() {
assert!(validate_secrets(true, REAL_SECRET, "").is_err());
assert!(validate_secrets(true, REAL_SECRET, "$2y$12$placeholder_replace_me").is_err());
assert!(validate_secrets(true, REAL_SECRET, "", REAL_DB_URL).is_err());
assert!(
validate_secrets(
true,
REAL_SECRET,
"$2y$12$placeholder_replace_me",
REAL_DB_URL
)
.is_err()
);
}
/// The stack used to come up GREEN on the database password published in the repo: this guard
/// covered the two secrets it was written for, and nothing anywhere looked at the Postgres
/// credential. README step 2 doesn't name POSTGRES_PASSWORD either, so following the
/// documented procedure verbatim shipped it.
#[test]
fn prod_rejects_the_shipped_placeholder_database_password() {
let shipped = "postgres://eventsnap:CHANGE_ME_use_a_strong_password@db:5432/eventsnap";
let err = validate_secrets(true, REAL_SECRET, REAL_HASH, shipped).unwrap_err();
assert!(
err.to_string().contains("DATABASE_URL"),
"the refusal must name DATABASE_URL, not just fail: {err}"
);
// And it must point at the initdb trap, or the operator edits .env, restarts, and lands
// in a permanent auth-failure loop instead.
assert!(
err.to_string().contains("POSTGRES_PASSWORD"),
"the refusal must name POSTGRES_PASSWORD as the other half: {err}"
);
}
/// Every problem in ONE message. Reporting them one per boot made fixing two secrets cost two
/// restart cycles, on a stack where Caddy waits on the unhealthy app the whole time.
#[test]
fn prod_reports_every_placeholder_at_once() {
let err = validate_secrets(
true,
"change_me_to_a_random_64_byte_hex_string",
"$2y$12$placeholder_replace_me",
"postgres://eventsnap:CHANGE_ME_use_a_strong_password@db:5432/eventsnap",
)
.unwrap_err()
.to_string();
for expected in ["JWT_SECRET", "ADMIN_PASSWORD_HASH", "DATABASE_URL"] {
assert!(err.contains(expected), "{expected} missing from: {err}");
}
assert!(
err.contains("3 secret(s)"),
"the count must match what is listed: {err}"
);
}
#[test]
fn prod_accepts_real_secrets() {
assert!(validate_secrets(true, REAL_SECRET, REAL_HASH).is_ok());
assert!(validate_secrets(true, REAL_SECRET, REAL_HASH, REAL_DB_URL).is_ok());
}
/// A real password that happens to contain no placeholder substring must pass — including one
/// with URL-ish punctuation, so the guard can't be mistaken for a URL validator.
#[test]
fn prod_accepts_a_real_database_url_with_awkward_punctuation() {
assert!(
validate_secrets(
true,
REAL_SECRET,
REAL_HASH,
"postgres://eventsnap:aB3%24xY9-_.qW@db:5432/eventsnap"
)
.is_ok()
);
}
/// The e2e stack runs without APP_ENV=production, so none of this applies there — but assert
/// it, because a guard that tripped in e2e would be found the hard way.
#[test]
fn non_prod_ignores_a_placeholder_database_url() {
assert!(
validate_secrets(
false,
REAL_SECRET,
"",
"postgres://eventsnap:CHANGE_ME_use_a_strong_password@db:5432/eventsnap"
)
.is_ok()
);
}
#[test]
fn non_prod_tolerates_dev_sentinel() {
assert!(validate_secrets(false, DEV_JWT_SECRET_SENTINEL, "").is_ok());
assert!(validate_secrets(false, DEV_JWT_SECRET_SENTINEL, "", REAL_DB_URL).is_ok());
}
#[test]
fn non_prod_still_rejects_short_non_sentinel_secret() {
assert!(validate_secrets(false, "tooshort", "").is_err());
assert!(validate_secrets(false, "tooshort", "", REAL_DB_URL).is_err());
}
#[test]
@@ -185,9 +303,23 @@ mod tests {
// looks_placeholder lowercases before matching — an upper/mixed-case
// placeholder must still be rejected in prod.
assert!(
validate_secrets(true, "CHANGE_ME_TO_A_RANDOM_64_BYTE_HEX_STRING", REAL_HASH).is_err()
validate_secrets(
true,
"CHANGE_ME_TO_A_RANDOM_64_BYTE_HEX_STRING",
REAL_HASH,
REAL_DB_URL
)
.is_err()
);
assert!(
validate_secrets(
true,
REAL_SECRET,
"$2Y$12$PLACEHOLDER_replace_me",
REAL_DB_URL
)
.is_err()
);
assert!(validate_secrets(true, REAL_SECRET, "$2Y$12$PLACEHOLDER_replace_me").is_err());
}
#[test]
@@ -197,7 +329,7 @@ mod tests {
const LEN_31: &str = "abcdefghijklmnopqrstuvwxyz01234";
assert_eq!(LEN_32.len(), 32);
assert_eq!(LEN_31.len(), 31);
assert!(validate_secrets(true, LEN_32, REAL_HASH).is_ok());
assert!(validate_secrets(true, LEN_31, REAL_HASH).is_err());
assert!(validate_secrets(true, LEN_32, REAL_HASH, REAL_DB_URL).is_ok());
assert!(validate_secrets(true, LEN_31, REAL_HASH, REAL_DB_URL).is_err());
}
}

52
backend/src/db.rs
View File

@@ -4,17 +4,65 @@ use sqlx::postgres::PgPoolOptions;
const DEFAULT_MAX_CONNECTIONS: u32 = 10;
/// SQLSTATE for `invalid_password`.
const PG_INVALID_PASSWORD: &str = "28P01";
/// Turn the one connect failure with an unguessable cause into a self-explaining one.
///
/// `POSTGRES_PASSWORD` is honoured ONLY when Postgres initialises its data directory. Change it in
/// `.env` afterwards and the app authenticates with the new password against a volume that still
/// holds the old one — a permanent restart loop whose only symptom is
/// `password authentication failed`.
///
/// The production secret guard makes that sequence NEARLY CERTAIN rather than rare: it stops the
/// app on the first `docker compose up -d`, but not the `db` service in that same command, which
/// initialises and bakes in whatever password was in `.env` at that moment. So the intended
/// recovery — see the refusal, fix your secrets, boot again — is exactly the sequence that breaks
/// it. Nothing in the error names the cause, and the remedy destroys data, so it is the last thing
/// an operator should guess at.
fn explain_auth_failure(err: &sqlx::Error) {
let is_auth_failure = match err {
sqlx::Error::Database(db) => db.code().as_deref() == Some(PG_INVALID_PASSWORD),
_ => false,
};
if !is_auth_failure {
return;
}
tracing::error!(
"Postgres rejected the credentials in DATABASE_URL (SQLSTATE {PG_INVALID_PASSWORD}).\n\
\n\
This almost always means POSTGRES_PASSWORD was changed AFTER the database volume was \
first created. Postgres applies that variable only when it initialises its data \
directory; editing .env and restarting does not change the stored password, so the two \
drift apart permanently.\n\
\n\
If the event has NOT started and you have no data worth keeping:\n\n \
docker compose down -v && docker compose up -d\n\n\
(-v DELETES the database, the uploaded media and the exports. There is no undo.)\n\
\n\
If you DO have data: restore the old password into DATABASE_URL instead, or change the \
stored one with ALTER ROLE inside the running db container. Never reach for -v to fix a \
login problem on a live event."
);
}
pub async fn create_pool(database_url: &str) -> Result<PgPool> {
let max_connections = std::env::var("DATABASE_MAX_CONNECTIONS")
.ok()
.and_then(|s| s.parse::<u32>().ok())
.unwrap_or(DEFAULT_MAX_CONNECTIONS);
let pool = PgPoolOptions::new()
let pool = match PgPoolOptions::new()
.max_connections(max_connections)
.connect(database_url)
.await
.context("failed to connect to database")?;
{
Ok(pool) => pool,
Err(e) => {
explain_auth_failure(&e);
return Err(e).context("failed to connect to database");
}
};
sqlx::migrate!()
.run(&pool)