Two independent lines of production hardening diverged at7d0334band attacked overlapping problems. Neither was a superset, so this is a merge of substance rather than a fast-forward: every conflict was resolved on the merits, and the losing side's intent was re-checked against the winner rather than assumed. MIGRATIONS. The branch's 021/022/023 collided with main's already-DEPLOYED 021_hashtag_counts_respect_bans and 022_client_upload_idempotency. Renumbered to 023/024/025 in a prior commit — main's versions are applied in production, so their version numbers are immutable and the branch's had to move. Verified by running the full sqlx::test suite, which applies the whole chain from scratch. RESOLVED IN MAIN'S FAVOUR (the branch would have regressed these): * upload-queue.ts wholesale — the branch's copy has ZERO client_upload_id references, so taking it would have silently destroyed end-to-end upload idempotency, the one thing standing between a lost response and a duplicate photo charged twice against the guest's quota. * maintenance.rs supervisor — the branch replaced it with a bare tokio::spawn, where one panic silently stops session pruning, media reclaim, the temp sweep and both HashMap prunes, permanently and with no log line. * The decode-budget probe on spawn_blocking, not inline on the async runtime. * feed/+page.svelte's 8s debounce + jitter + max-wait + hidden-tab deferral, against the branch's naive 800ms — at 100 guests the branch's version walks straight into the per-user feed rate limit. * db.rs pool tuning, /uploaders, and the docker-compose deployment story. * ONE /health, still DB-backed. The branch's split (dependency-free liveness + DB-backed readiness) is defensible, but a constant-"ok" /health is the exact defectfaea555fixed and verified live, its motive (Caddy's boot gate) is already covered by app depends_on db: service_healthy, and the two handlers were the same SELECT 1 under two names. TAKEN FROM THE BRANCH: * The large-PNG OOM guard and its bounded-retry counter (023). Together these turn a single upload that can OOM-kill a 1G container into a bounded failure instead of an infinite restart loop under `restart: unless-stopped`. * 024_feed_scalar_counts — the feed no longer aggregates the whole event per page. Pure SQL; column names, order and types are unchanged by design. * The admin-lockout fix: look the admin up BY ROLE, never by name. 025 also frees any guest already squatting on a reserved name. * PIN lockout tier ordering, bounded caption/hashtag reads, SSE ticket caps, PoolTimedOut -> 503 + Retry-After, and the ffmpeg stderr drain. * backfill_video_posters, which main lacked entirely. * TempFileGuard, plus sweep_orphan_originals wired into main's SUPERVISED loop (not the branch's bare one) — it reclaims final-named originals whose commit never happened, a class main's .tmp-only sweep structurally cannot see. * shouldAbortForStall, hand-ported into main's upload-queue.ts since that file was resolved to main. Widens the watchdog at loadend instead of disarming it, bounding a half-open socket at 2 minutes rather than handing the window to xhr.timeout (5-60 min) with the whole queue's `processing` latch held. ALSO: RUST_LOG and EXPORT_PATH pinned in compose. The code fallback was `debug` (a line per request, all night) and EXPORT_PATH was the one path with a mount-shaped default that nothing validated. Verified: cargo check --all-targets, cargo clippy (clean), 144/144 backend tests against a live Postgres including upload_idempotency and upload_concurrency, 51/51 vitest, svelte-check 0 errors, eslint clean, vite build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
264 lines
12 KiB
Rust
264 lines
12 KiB
Rust
//! Poster-frame extraction, shared by the compression worker and the HTML export.
|
|
//!
|
|
//! Both used to spawn `ffmpeg` themselves with the same broken invocation:
|
|
//!
|
|
//! ```text
|
|
//! ffmpeg -i <src> -vframes 1 -ss 00:00:01 -vf scale=… -y <out>
|
|
//! ```
|
|
//!
|
|
//! `-ss` AFTER `-i` is an output-side seek. Against a clip of a second or less ffmpeg exits **0 and
|
|
//! writes nothing** — and both call sites gated on the exit status, so neither noticed. The worker
|
|
//! then wrote `thumbnail_path` for a file that was never created (404 in the live feed) and the
|
|
//! export listed the entry in `data.json` while the ZIP writer skipped it (a broken image tile in
|
|
//! the keepsake). Every server-side signal stayed green. Phones produce such clips constantly:
|
|
//! mis-taps, Live Photos, boomerangs.
|
|
//!
|
|
//! This module exists for the same reason `imaging.rs` does — that one was created when compression
|
|
//! and export duplicated decode logic, and it paid off immediately when the `max_alloc` fix landed
|
|
//! in both workers at once. Same duplication, same fix.
|
|
|
|
use std::path::Path;
|
|
use std::time::Duration;
|
|
|
|
use anyhow::{Context, Result};
|
|
|
|
/// A malformed video can hang `ffmpeg` indefinitely. In the compression worker that never releases
|
|
/// the semaphore permit and the pool eventually deadlocks; in the export worker it strands the job
|
|
/// at `running` so the keepsake never completes. `export.rs` had NO timeout at all before this
|
|
/// module — sharing the spawn fixes that too.
|
|
/// 45s, not the 120s this started at. The timeout is not a budget for honest work — a poster
|
|
/// frame from a phone clip takes well under a second, and `-ss` before `-i` means even a 500 MB
|
|
/// file seeks rather than scans. It is purely the ceiling on how long a pathological input may
|
|
/// hold a compression permit that guests' photos are queued behind, so it should be as tight as
|
|
/// it can be without ever cutting off real work.
|
|
const FFMPEG_TIMEOUT: Duration = Duration::from_secs(45);
|
|
|
|
/// Seek positions to try, in order.
|
|
///
|
|
/// One second first: the opening frame of a real video is often black, a fade-in, or motion-blurred
|
|
/// as the camera settles, so it makes a poor poster. Zero second as the fallback, which is what
|
|
/// makes short clips work — and it is genuinely required, not defensive. Moving `-ss` before `-i`
|
|
/// (an input-side seek) is necessary but NOT sufficient: seeking to 1 s in a 1.000 s clip is still
|
|
/// past the last frame, and ffmpeg still exits 0 having written nothing. Verified against the real
|
|
/// production image.
|
|
const SEEK_POSITIONS: &[&str] = &["00:00:01", "0"];
|
|
|
|
/// Extract one poster frame from `src` into `dest`, scaled to `width` px wide.
|
|
///
|
|
/// `Ok(false)` means the video yielded no frame — a normal outcome for a very short or unusual
|
|
/// clip, NOT an error. Callers must degrade (no poster) rather than fail the upload: treating this
|
|
/// as an error would soft-delete every sub-second video, turning a cosmetic defect into data loss.
|
|
///
|
|
/// `Err` is reserved for something genuinely wrong — a hang we had to kill, or a failure to spawn.
|
|
pub async fn extract_poster_frame(src: &Path, dest: &Path, width: u32) -> Result<bool> {
|
|
for seek in SEEK_POSITIONS {
|
|
// A stale file from a previous attempt would be indistinguishable from a fresh success.
|
|
let _ = tokio::fs::remove_file(dest).await;
|
|
|
|
run_ffmpeg(src, dest, width, seek).await?;
|
|
|
|
// THE CHECK BOTH CALL SITES WERE MISSING: ask the filesystem, not the exit status.
|
|
// Non-empty, because a zero-byte file is not a poster either.
|
|
if tokio::fs::metadata(dest)
|
|
.await
|
|
.map(|m| m.is_file() && m.len() > 0)
|
|
.unwrap_or(false)
|
|
{
|
|
return Ok(true);
|
|
}
|
|
}
|
|
|
|
// Leave nothing behind for a caller to mistake for a result.
|
|
let _ = tokio::fs::remove_file(dest).await;
|
|
Ok(false)
|
|
}
|
|
|
|
/// Run one ffmpeg attempt. A non-zero exit is NOT an error here — the artifact check above is the
|
|
/// authority, and a corrupt input that fails at 1 s may still yield a frame at 0.
|
|
async fn run_ffmpeg(src: &Path, dest: &Path, width: u32, seek: &str) -> Result<()> {
|
|
let child = tokio::process::Command::new("ffmpeg")
|
|
.args([
|
|
// BEFORE -i: an input-side seek. See SEEK_POSITIONS.
|
|
"-ss",
|
|
seek,
|
|
"-i",
|
|
src.to_str().unwrap_or_default(),
|
|
"-vframes",
|
|
"1",
|
|
"-vf",
|
|
&format!("scale={width}:-1"),
|
|
"-y",
|
|
dest.to_str().unwrap_or_default(),
|
|
])
|
|
// ffmpeg writes the poster to `dest` itself; nothing here ever reads stdout, so
|
|
// giving it a pipe only created something that could fill.
|
|
.stdout(std::process::Stdio::null())
|
|
// stderr IS piped — it is the only diagnostic when a clip yields no frame — but it
|
|
// must be DRAINED, which is the whole point of `wait_with_output` below.
|
|
.stderr(std::process::Stdio::piped())
|
|
.kill_on_drop(true)
|
|
.spawn()
|
|
.context("failed to spawn ffmpeg")?;
|
|
|
|
// `wait_with_output`, NOT `wait`. ffmpeg is verbose on stderr (banner, stream info,
|
|
// per-frame progress) and `wait()` reads neither pipe — so once the ~64 KiB pipe buffer
|
|
// filled, ffmpeg blocked writing, `wait()` never returned, and the call burned the full
|
|
// timeout. That is not merely slow: the timeout is an `Err`, so after 2 seek positions x
|
|
// 3 compression attempts the caller soft-deletes a perfectly playable video for a
|
|
// poster-frame failure. `wait_with_output` polls the pipe and the exit status together.
|
|
//
|
|
// It also CONSUMES the child, so the explicit `child.kill()` that used to sit on the
|
|
// timeout arm cannot exist here — and is not needed: `kill_on_drop(true)` is set above,
|
|
// and dropping the future on timeout drops the child with it.
|
|
let out = match tokio::time::timeout(FFMPEG_TIMEOUT, child.wait_with_output()).await {
|
|
Ok(res) => res.context("ffmpeg wait failed")?,
|
|
Err(_) => anyhow::bail!("ffmpeg timed out after {}s", FFMPEG_TIMEOUT.as_secs()),
|
|
};
|
|
|
|
// A non-zero exit is not an error (see the doc comment) — the artifact check in
|
|
// `extract_poster_frame` is the authority. Log the tail so a systematically failing
|
|
// format is diagnosable without turning it into data loss.
|
|
if !out.status.success() {
|
|
tracing::debug!(
|
|
seek,
|
|
status = ?out.status,
|
|
stderr = %tail_lines(&out.stderr, 10),
|
|
"ffmpeg exited non-zero; the artifact check decides"
|
|
);
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
/// Last `n` lines of a child's stderr, lossily decoded.
|
|
///
|
|
/// Bounded on purpose: ffmpeg's stderr is unbounded, and the reason we now drain it is that
|
|
/// unbounded output used to be a hazard. Emitting all of it into a log line — into container
|
|
/// logs that are themselves size-capped — would just move the problem.
|
|
fn tail_lines(bytes: &[u8], n: usize) -> String {
|
|
let text = String::from_utf8_lossy(bytes);
|
|
let lines: Vec<&str> = text.lines().filter(|l| !l.trim().is_empty()).collect();
|
|
lines[lines.len().saturating_sub(n)..].join(" | ")
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
/// Is there a usable `ffmpeg` on PATH?
|
|
///
|
|
/// The poster-frame path shells out, and `extract_poster_frame` documents `Err` as meaning
|
|
/// "a hang or a SPAWN failure" — which is exactly what a missing binary produces. So on a
|
|
/// machine without ffmpeg the test below stops exercising the case it names (missing INPUT)
|
|
/// and instead reports a code defect that isn't there. The runtime image installs ffmpeg
|
|
/// (see backend/Dockerfile), so this only ever skips on a bare developer machine.
|
|
fn ffmpeg_available() -> bool {
|
|
std::process::Command::new("ffmpeg")
|
|
.arg("-version")
|
|
.stdout(std::process::Stdio::null())
|
|
.stderr(std::process::Stdio::null())
|
|
.status()
|
|
.is_ok()
|
|
}
|
|
|
|
/// The order is the whole fix. `-ss` must precede `-i`, and 0 must be tried after 1 s.
|
|
#[test]
|
|
fn the_fallback_seek_exists_and_comes_last() {
|
|
assert_eq!(
|
|
SEEK_POSITIONS,
|
|
&["00:00:01", "0"],
|
|
"1s first for a better poster, 0 as the fallback that makes short clips work"
|
|
);
|
|
}
|
|
|
|
/// A missing input yields no frame rather than an error: the caller must degrade to "no
|
|
/// poster", never fail the upload. `Err` is reserved for a hang or a spawn failure.
|
|
#[tokio::test]
|
|
async fn a_missing_source_yields_no_frame_rather_than_an_error() {
|
|
if !ffmpeg_available() {
|
|
eprintln!(
|
|
"SKIP a_missing_source_yields_no_frame_rather_than_an_error: no ffmpeg on PATH. \
|
|
A missing binary is a spawn failure, which this function returns Err for by \
|
|
design, so the missing-INPUT case cannot be exercised here. Install ffmpeg to \
|
|
run it (the runtime image already has it)."
|
|
);
|
|
return;
|
|
}
|
|
let dir = std::env::temp_dir().join(format!("es-video-{}", std::process::id()));
|
|
std::fs::create_dir_all(&dir).unwrap();
|
|
let dest = dir.join("out.jpg");
|
|
|
|
let got = extract_poster_frame(Path::new("/nonexistent/clip.mp4"), &dest, 400).await;
|
|
|
|
match got {
|
|
Ok(false) => {}
|
|
other => panic!("expected Ok(false) for a missing input, got {other:?}"),
|
|
}
|
|
assert!(
|
|
!dest.exists(),
|
|
"a failed extraction must leave nothing a caller could mistake for a poster"
|
|
);
|
|
let _ = std::fs::remove_dir_all(&dir);
|
|
}
|
|
|
|
#[test]
|
|
fn the_stderr_tail_is_bounded_and_survives_invalid_utf8() {
|
|
let noisy: Vec<u8> = (0..500)
|
|
.map(|i| format!("line {i}\n"))
|
|
.collect::<String>()
|
|
.into_bytes();
|
|
let got = tail_lines(&noisy, 3);
|
|
assert_eq!(got, "line 497 | line 498 | line 499");
|
|
|
|
// ffmpeg emits filenames verbatim, so its stderr is not guaranteed to be UTF-8.
|
|
assert_eq!(tail_lines(&[b'o', b'k', 0xff], 5), "ok\u{fffd}");
|
|
assert_eq!(tail_lines(b"", 5), "");
|
|
}
|
|
|
|
/// A real extraction must finish in a small fraction of `FFMPEG_TIMEOUT`.
|
|
///
|
|
/// Wall-clock is the ONLY observable of the bug this guards: piping stderr and then
|
|
/// calling `wait()` (which drains nothing) blocks ffmpeg on a full pipe buffer until the
|
|
/// timeout fires, and the timeout is an `Err`, so the upload is soft-deleted. The
|
|
/// assertion is deliberately on elapsed time, not on the exit status.
|
|
///
|
|
/// Honest limitation: our fixture is quiet enough not to fill a 64 KiB pipe on its own,
|
|
/// so this catches a regression to `wait()` only in combination with a verbose input. It
|
|
/// is still worth pinning — a reverted drain plus any chatty clip is data loss.
|
|
#[tokio::test]
|
|
async fn a_real_clip_yields_a_poster_well_inside_the_timeout() {
|
|
if tokio::process::Command::new("ffmpeg")
|
|
.arg("-version")
|
|
.stdout(std::process::Stdio::null())
|
|
.stderr(std::process::Stdio::null())
|
|
.status()
|
|
.await
|
|
.is_err()
|
|
{
|
|
eprintln!("skipping: ffmpeg not on PATH");
|
|
return;
|
|
}
|
|
let src = Path::new("../e2e/fixtures/media/sample.mp4");
|
|
if !src.exists() {
|
|
eprintln!("skipping: {} missing", src.display());
|
|
return;
|
|
}
|
|
|
|
let dir = std::env::temp_dir().join(format!("es-video-ok-{}", std::process::id()));
|
|
std::fs::create_dir_all(&dir).unwrap();
|
|
let dest = dir.join("poster.jpg");
|
|
|
|
let started = std::time::Instant::now();
|
|
let got = extract_poster_frame(src, &dest, 400).await;
|
|
let elapsed = started.elapsed();
|
|
|
|
assert!(matches!(got, Ok(true)), "expected a poster, got {got:?}");
|
|
assert!(dest.metadata().unwrap().len() > 0);
|
|
assert!(
|
|
elapsed < FFMPEG_TIMEOUT / 4,
|
|
"extraction took {elapsed:?}; a drained stderr finishes in well under \
|
|
{FFMPEG_TIMEOUT:?} — this is the pipe-deadlock regression guard"
|
|
);
|
|
let _ = std::fs::remove_dir_all(&dir);
|
|
}
|
|
}
|