Files
EventSnap/backend/src/main.rs
MechaCat02 61119be817 Merge branch 'fix/deploy-unattended-blockers' into main
Two independent lines of production hardening diverged at 7d0334b and attacked
overlapping problems. Neither was a superset, so this is a merge of substance
rather than a fast-forward: every conflict was resolved on the merits, and the
losing side's intent was re-checked against the winner rather than assumed.

MIGRATIONS. The branch's 021/022/023 collided with main's already-DEPLOYED
021_hashtag_counts_respect_bans and 022_client_upload_idempotency. Renumbered to
023/024/025 in a prior commit — main's versions are applied in production, so
their version numbers are immutable and the branch's had to move. Verified by
running the full sqlx::test suite, which applies the whole chain from scratch.

RESOLVED IN MAIN'S FAVOUR (the branch would have regressed these):
  * upload-queue.ts wholesale — the branch's copy has ZERO client_upload_id
    references, so taking it would have silently destroyed end-to-end upload
    idempotency, the one thing standing between a lost response and a duplicate
    photo charged twice against the guest's quota.
  * maintenance.rs supervisor — the branch replaced it with a bare tokio::spawn,
    where one panic silently stops session pruning, media reclaim, the temp
    sweep and both HashMap prunes, permanently and with no log line.
  * The decode-budget probe on spawn_blocking, not inline on the async runtime.
  * feed/+page.svelte's 8s debounce + jitter + max-wait + hidden-tab deferral,
    against the branch's naive 800ms — at 100 guests the branch's version walks
    straight into the per-user feed rate limit.
  * db.rs pool tuning, /uploaders, and the docker-compose deployment story.
  * ONE /health, still DB-backed. The branch's split (dependency-free liveness +
    DB-backed readiness) is defensible, but a constant-"ok" /health is the exact
    defect faea555 fixed and verified live, its motive (Caddy's boot gate) is
    already covered by app depends_on db: service_healthy, and the two handlers
    were the same SELECT 1 under two names.

TAKEN FROM THE BRANCH:
  * The large-PNG OOM guard and its bounded-retry counter (023). Together these
    turn a single upload that can OOM-kill a 1G container into a bounded failure
    instead of an infinite restart loop under `restart: unless-stopped`.
  * 024_feed_scalar_counts — the feed no longer aggregates the whole event per
    page. Pure SQL; column names, order and types are unchanged by design.
  * The admin-lockout fix: look the admin up BY ROLE, never by name. 025 also
    frees any guest already squatting on a reserved name.
  * PIN lockout tier ordering, bounded caption/hashtag reads, SSE ticket caps,
    PoolTimedOut -> 503 + Retry-After, and the ffmpeg stderr drain.
  * backfill_video_posters, which main lacked entirely.
  * TempFileGuard, plus sweep_orphan_originals wired into main's SUPERVISED loop
    (not the branch's bare one) — it reclaims final-named originals whose commit
    never happened, a class main's .tmp-only sweep structurally cannot see.
  * shouldAbortForStall, hand-ported into main's upload-queue.ts since that file
    was resolved to main. Widens the watchdog at loadend instead of disarming it,
    bounding a half-open socket at 2 minutes rather than handing the window to
    xhr.timeout (5-60 min) with the whole queue's `processing` latch held.

ALSO: RUST_LOG and EXPORT_PATH pinned in compose. The code fallback was
`debug` (a line per request, all night) and EXPORT_PATH was the one path with a
mount-shaped default that nothing validated.

Verified: cargo check --all-targets, cargo clippy (clean), 144/144 backend tests
against a live Postgres including upload_idempotency and upload_concurrency,
51/51 vitest, svelte-check 0 errors, eslint clean, vite build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 21:16:31 +02:00

402 lines
18 KiB
Rust

use anyhow::Result;
use axum::Router;
use axum::extract::DefaultBodyLimit;
use axum::routing::{delete, get, patch, post};
use tower_http::trace::TraceLayer;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
mod auth;
mod config;
mod db;
mod error;
mod handlers;
mod models;
mod services;
mod state;
use config::AppConfig;
use state::AppState;
/// Hard HTTP body cap for the upload endpoint (576 MiB). Backstop against
/// memory-exhaustion; precise per-class size limits are enforced in the handler.
const MAX_UPLOAD_BYTES: usize = 576 * 1024 * 1024;
#[tokio::main]
async fn main() -> Result<()> {
dotenvy::dotenv().ok();
tracing_subscriber::registry()
.with(
// `info`, not `debug`. A stock deploy sets RUST_LOG nowhere (it is absent from
// .env.example and was absent from docker-compose.yml), so this fallback IS the
// production level — and at `debug` the TraceLayer below emits a line per request
// AND per response, into a log file that had no rotation. `tower_http=warn`
// rather than `info` states the intent: those spans are diagnostics, not an
// access log, and a future `DefaultOnResponse::new().level(Level::INFO)` should
// not silently re-enable them.
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| "eventsnap_backend=info,tower_http=warn".into()),
)
.with(tracing_subscriber::fmt::layer())
.init();
let config = AppConfig::from_env()?;
let pool = db::create_pool(&config.database_url).await?;
// Reset any rows left mid-flight by a previous (possibly crashed) instance —
// stuck `compression_status='processing'` uploads and `status='running'` export
// jobs. Must run before the server starts taking requests so clients never see
// the half-state.
services::maintenance::startup_recovery(&pool).await;
let state = AppState::new(pool.clone(), config.clone());
// Regenerate image derivatives an older pipeline produced: the big-screen display for
// uploads processed before it existed (v0.17.x), and anything predating the current
// DERIVATIVES_REV (rev 1 applies the EXIF orientation, without which every portrait
// phone photo is stored sideways). Fire-and-forget behind the compression semaphore;
// originals are never touched, so a failure just retries on the next start.
state.compression.backfill_stale_derivatives().await;
// Re-extract poster frames for videos a restart interrupted. `startup_recovery` above
// marks their compression `failed` but nothing re-enqueued them, so `thumbnail_path`
// stayed NULL for the rest of the event. Shares the attempt budget with the image
// backfill, so a clip that genuinely yields no frame stops being retried.
state.compression.backfill_video_posters().await;
// Re-spawn exports for events that were released but whose keepsake never finished
// (crash mid-export). Needs the media/export paths + SSE sender, so it runs here
// rather than inside `startup_recovery`. Fire-and-forget: the workers run in the
// background; the HTTP server can start accepting requests meanwhile.
services::export::recover_exports(
pool.clone(),
config.media_path.clone(),
config.export_path.clone(),
config.comments_enabled,
state.sse_tx.clone(),
)
.await;
// Hourly background hygiene: prune expired sessions, evict cold rate-limiter
// keys. Keeps the DB and process from growing unboundedly over multi-day events.
services::maintenance::spawn_periodic_tasks(
pool,
state.rate_limiter.clone(),
state.sse_tickets.clone(),
config.media_path.clone(),
);
// Ensure media directories exist
tokio::fs::create_dir_all(&config.media_path).await.ok();
let api = Router::new()
// Auth
.route("/api/v1/event", get(handlers::public::get_public_event))
.route("/api/v1/join", post(auth::handlers::join))
.route("/api/v1/recover", post(auth::handlers::recover))
// Forgotten-PIN escape hatch: ask a host to reset it (unauthenticated, throttled).
.route(
"/api/v1/recover/request",
post(auth::handlers::request_pin_reset),
)
.route("/api/v1/admin/login", post(auth::handlers::admin_login))
.route("/api/v1/session", delete(auth::handlers::logout))
// "Sign out everywhere" — revoke all of the caller's sessions.
.route("/api/v1/sessions", delete(auth::handlers::logout_all))
// Upload — HTTP-level body cap as an OOM backstop. The handler still enforces
// the precise per-class limits from DB config (max_image/video_size_mb); this
// layer just stops a multi-GB body from being buffered into memory before that
// check runs. Sized generously above the default 500 MB video limit + multipart
// overhead — if an admin raises max_video_size_mb above this, bump MAX_UPLOAD_BYTES.
.route(
"/api/v1/upload",
post(handlers::upload::upload).route_layer(DefaultBodyLimit::max(MAX_UPLOAD_BYTES)),
)
.route(
"/api/v1/upload/{id}",
patch(handlers::upload::edit_upload).delete(handlers::upload::delete_upload),
)
.route(
"/api/v1/upload/{id}/original",
get(handlers::upload::get_original),
)
// Preview/thumbnail variants are gated the same way as originals (visibility
// check + direct /media block below) so moderation actually revokes access to
// the displayed images, not just the full-res download.
.route(
"/api/v1/upload/{id}/preview",
get(handlers::upload::get_preview),
)
.route(
"/api/v1/upload/{id}/display",
get(handlers::upload::get_display),
)
.route(
"/api/v1/upload/{id}/thumbnail",
get(handlers::upload::get_thumbnail),
)
// Current-user endpoints (live quota estimate, profile + privacy note bundle)
.route("/api/v1/me/context", get(handlers::me::get_context))
.route("/api/v1/me/quota", get(handlers::me::get_quota))
// Feed
.route("/api/v1/feed", get(handlers::feed::feed))
.route("/api/v1/feed/delta", get(handlers::feed::feed_delta))
.route("/api/v1/hashtags", get(handlers::feed::hashtags))
.route("/api/v1/uploaders", get(handlers::feed::uploaders))
// Social
.route(
"/api/v1/upload/{id}/like",
post(handlers::social::toggle_like),
)
.route(
"/api/v1/upload/{id}/comments",
get(handlers::social::list_comments).post(handlers::social::add_comment),
)
.route(
"/api/v1/comment/{id}",
delete(handlers::social::delete_comment),
)
// SSE
.route("/api/v1/stream", get(handlers::sse::stream))
.route("/api/v1/stream/ticket", post(handlers::sse::issue_ticket))
// Host Dashboard
.route("/api/v1/host/event", get(handlers::host::get_event_status))
.route(
"/api/v1/host/event/close",
post(handlers::host::close_event),
)
.route("/api/v1/host/event/open", post(handlers::host::open_event))
.route(
"/api/v1/host/gallery/release",
post(handlers::host::release_gallery),
)
// Escape hatch: force a keepsake rebuild. Without it a failed export is terminal at runtime
// (release_gallery refuses an already-released event; recovery only runs at boot).
.route(
"/api/v1/host/export/rebuild",
post(handlers::host::rebuild_export),
)
.route("/api/v1/host/users", get(handlers::host::list_users))
.route(
"/api/v1/host/users/{id}/ban",
post(handlers::host::ban_user),
)
.route(
"/api/v1/host/users/{id}/unban",
post(handlers::host::unban_user),
)
.route(
"/api/v1/host/users/{id}/role",
patch(handlers::host::set_role),
)
.route(
"/api/v1/host/users/{id}/pin-reset",
post(handlers::host::reset_user_pin),
)
.route(
"/api/v1/host/pin-reset-requests",
get(handlers::host::list_pin_reset_requests),
)
.route(
"/api/v1/host/pin-reset-requests/{id}",
delete(handlers::host::dismiss_pin_reset_request),
)
.route(
"/api/v1/host/upload/{id}",
delete(handlers::host::host_delete_upload),
)
.route(
"/api/v1/host/comment/{id}",
delete(handlers::host::host_delete_comment),
)
// Export (all authenticated users)
.route("/api/v1/export/status", get(handlers::admin::export_status))
.route(
"/api/v1/export/ticket",
post(handlers::admin::export_ticket),
)
.route("/api/v1/export/zip", get(handlers::admin::download_zip))
.route("/api/v1/export/html", get(handlers::admin::download_html))
// Admin Dashboard
.route("/api/v1/admin/stats", get(handlers::admin::get_stats))
.route(
"/api/v1/admin/config",
get(handlers::admin::get_config).patch(handlers::admin::patch_config),
)
.route(
"/api/v1/admin/export/jobs",
get(handlers::admin::get_export_jobs),
);
// Test-only route: a hard reset for the Playwright E2E harness. The handler
// is compiled in always, but the route is only attached when
// `EVENTSNAP_TEST_MODE=1`. In production the call returns 404 — the route
// simply isn't there.
let api = if handlers::test_admin::is_test_mode() {
tracing::warn!(
"EVENTSNAP_TEST_MODE=1 — registering /api/v1/admin/__truncate. \
DO NOT enable this in production."
);
api.route(
"/api/v1/admin/__truncate",
post(handlers::test_admin::truncate_all),
)
} else {
api
};
// NOTE: media is deliberately NOT served over HTTP.
//
// Files live under `media_path` so the compression worker and the export job can read
// them off disk, but nothing may pull them straight from `/media/**` — that bypasses
// the visibility checks (soft-delete + ban-hide) that make a host takedown stick.
// Every legitimate fetch goes through `/api/v1/upload/{id}/{original,preview,display,
// thumbnail}`, which filter via `find_visible_media`; those are the only media URLs the
// backend ever emits (see `handlers::feed`).
//
// This used to be a `ServeDir` on `/media` with four `nest_service` blockers on the
// subtrees above it. That was bypassable: axum routes on the RAW path while `ServeDir`
// percent-decodes afterwards, so `/media/%70reviews/{id}.jpg` missed every blocker,
// fell through to the `ServeDir`, and was decoded back to `previews/` on disk — serving
// a taken-down photo to anyone, unauthenticated. Any single escaped byte worked, in all
// four subtrees. Deleting the route removes the vector outright rather than racing the
// decoder; `/media/**` now 404s regardless of encoding.
let router = Router::new()
// ONE probe, and it touches the database. The merge of the unattended-blockers work
// brought a competing design — a dependency-free `/health` for the compose gate plus
// a DB-backed `/health/ready` for an external monitor. That split is defensible, and
// it was rejected deliberately:
//
// * `/health` returning a constant "ok" is the exact defect faea555 fixed and
// verified live (stop Postgres → 503 → start → 200, with no app restart). Every
// request path touches the database, so a constant probe reports healthy while
// the app is useless — the disk-full endgame stayed green all the way down.
// * The split's motive was that Caddy's `depends_on: app: service_healthy` would
// be blocked by a Postgres hiccup at boot. But `app` itself already gates on
// `db: service_healthy`, so the DB is up before this probe ever runs, and the
// healthcheck carries a 20s start_period plus 5 retries on top.
// * The two handlers were the same `SELECT 1` with the same 2s timeout under two
// names, so keeping both bought nothing.
//
// The external uptime monitor the runbook now calls for points at this route.
.route("/health", get(health))
.merge(api)
.layer(TraceLayer::new_for_http())
.with_state(state);
let listener = tokio::net::TcpListener::bind(("0.0.0.0", config.app_port)).await?;
tracing::info!("listening on {}", listener.local_addr()?);
// `into_make_service_with_connect_info` is required by the pre-auth handlers, which
// extract `ConnectInfo<SocketAddr>` to use the peer address as the rate-limit key when
// X-Forwarded-For is absent. Without it those extractors fail at runtime.
axum::serve(
listener,
router.into_make_service_with_connect_info::<std::net::SocketAddr>(),
)
.with_graceful_shutdown(shutdown_signal())
.await?;
Ok(())
}
/// How long `/health` waits for the database before calling the app unhealthy. Deliberately
/// short: the point is to answer "can this process actually serve a request right now", and a
/// probe that blocks for the acquire timeout is itself a symptom.
const HEALTH_DB_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(2);
/// Readiness probe — the Docker healthcheck, Caddy's `depends_on` gate, and the runbook's
/// event-day `curl` all hit this.
///
/// It used to return the literal string `"ok"` and touch nothing. Every request in the app needs
/// the database, so that answered a question nobody asked: the container reported healthy while
/// every real request 500'd, and with no operator watching during the event there was no signal
/// at all. Note what this does NOT buy: Compose's `restart: unless-stopped` does not react to
/// healthcheck state, so nothing restarts on a red probe — this is a diagnostic, and it is
/// deliberately not wired to automatic recovery, because the pool already heals itself across a
/// Postgres restart (sqlx revalidates on acquire) and an auto-restart would truncate every
/// in-flight upload to "fix" an outage that was about to clear on its own.
async fn health(
axum::extract::State(state): axum::extract::State<AppState>,
) -> impl axum::response::IntoResponse {
use axum::http::StatusCode;
match tokio::time::timeout(
HEALTH_DB_TIMEOUT,
sqlx::query("SELECT 1").execute(&state.pool),
)
.await
{
Ok(Ok(_)) => (StatusCode::OK, "ok"),
Ok(Err(e)) => {
tracing::error!(error = ?e, "health check: database query failed");
(StatusCode::SERVICE_UNAVAILABLE, "database unavailable")
}
Err(_) => {
tracing::error!(
timeout_s = HEALTH_DB_TIMEOUT.as_secs(),
"health check: database did not respond"
);
(StatusCode::SERVICE_UNAVAILABLE, "database timeout")
}
}
}
/// Hard cap on how long we wait for in-flight connections to drain after a shutdown
/// signal. Uploads (streamed to disk) finish in well under this; the cap exists because
/// long-lived SSE streams never end on their own and would otherwise keep the graceful
/// drain — and thus the process — pending until the orchestrator force-kills it.
const SHUTDOWN_GRACE: std::time::Duration = std::time::Duration::from_secs(10);
/// Resolves on SIGINT (Ctrl-C) or SIGTERM (container stop / deploy). Letting
/// `axum::serve` drain in-flight requests on this signal means a redeploy no longer
/// truncates uploads mid-flight; any background compression/export half-states that a
/// hard kill would leave are already reconciled by `startup_recovery` on the next boot.
///
/// Once the signal fires we also arm a detached backstop that force-exits after
/// [`SHUTDOWN_GRACE`]. Without it, open SSE streams (which have no natural end) would
/// hold the graceful drain open indefinitely; the backstop bounds shutdown regardless
/// of the orchestrator's own kill timeout. If the drain completes first, `main` returns
/// and the process exits before the timer ever fires.
async fn shutdown_signal() {
let ctrl_c = async {
let _ = tokio::signal::ctrl_c().await;
};
#[cfg(unix)]
let terminate = async {
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) {
Ok(mut sig) => {
sig.recv().await;
}
Err(e) => {
tracing::warn!(error = ?e, "failed to install SIGTERM handler");
// Never resolve — fall back to ctrl_c only.
std::future::pending::<()>().await;
}
}
};
#[cfg(not(unix))]
let terminate = std::future::pending::<()>();
tokio::select! {
_ = ctrl_c => {}
_ = terminate => {}
}
tracing::info!(
"shutdown signal received, draining in-flight requests (max {}s)",
SHUTDOWN_GRACE.as_secs()
);
// Backstop: if the graceful drain is still blocked after the grace window (almost
// always because SSE streams are still open), exit anyway so deploys aren't stalled.
tokio::spawn(async {
tokio::time::sleep(SHUTDOWN_GRACE).await;
tracing::warn!(
"graceful drain exceeded {}s (likely open SSE streams); forcing exit",
SHUTDOWN_GRACE.as_secs()
);
std::process::exit(0);
});
}