fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

perf(backend): close config/upload/export hot paths + stability fixes

Browse Source 
Performance:
- Cache the runtime `config` table in-memory (ConfigCache) with synchronous
  invalidation on every write (admin PATCH + test reseed). Was re-reading each
  key from Postgres on every request (~8 round-trips per upload).
- Stream uploads chunk-by-chunk to a temp file instead of buffering the whole
  body in RAM (peak was up to the per-class cap, e.g. 500 MB/video); only 512
  sniff-bytes are kept for magic-byte detection, then atomic rename into place.
- Cache the media-filesystem disk snapshot (DiskCache, 15s TTL) shared by the
  quota check and admin stats; drop the discarded System::refresh_all().
- HTML export streams video (and small-image) originals straight into the ZIP
  via a manifest instead of copying them to a temp dir first (removed the
  transient 2x disk usage) and drops the double directory scan.
- Auth extractor resolves session -> live user in one JOIN (was two queries),
  touching last_seen_at by token hash.

Stability:
- SSE: on broadcast lag, emit a `resync` event so the client runs a delta
  fetch instead of silently losing events; frontend reconciles adds, deletions,
  and (via an in-place refresh) like/comment counts on visible cards.
- Storage quota fails OPEN when the disk can't be read (was a 0-byte limit that
  locked out all uploads).
- Graceful shutdown drains in-flight requests on SIGTERM/SIGINT, bounded by a
  10s backstop so open SSE streams can't stall a deploy.
- Upload removes the persisted file if the DB transaction fails (no orphaned
  bytes with no row to reclaim them).

Tests:
- New pure select_disk() with 5 unit tests (longest-prefix, fallbacks, fail-open).
- New e2e export-video spec covering the HTML export's video-streaming branch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
fabi
2026-07-07 20:26:45 +02:00
parent 4cdb3ae14a
commit d6c91974eb
18 changed files with 745 additions and 214 deletions
Download Patch File Download Diff File Expand all files Collapse all files

96
backend/src/services/export.rs
View File

@@ -192,6 +192,23 @@ async fn run_zip_export(
// ── HTML viewer export ──────────────────────────────────────────────────────
/// Where a media entry's bytes come from at ZIP-writing time. Derived variants
/// (thumbnails, compressed images) are staged under the temp dir; original-fidelity
/// variants (videos, small images) are streamed straight from the source on disk so
/// the export never transiently doubles disk usage by copying large files to temp.
enum MediaSource {
Temp(PathBuf),
Original(PathBuf),
}
impl MediaSource {
fn path(&self) -> &Path {
match self {
MediaSource::Temp(p) | MediaSource::Original(p) => p,
}
}
}
async fn run_html_export(
event_id: Uuid,
event_name: &str,
@@ -221,6 +238,9 @@ async fn run_html_export(
// 3. Process media and build post data
let mut viewer_posts: Vec<ViewerPost> = Vec::new();
// (zip entry name under media/, where its bytes come from). Built here, streamed
// into the ZIP in step 5 — so we also know the exact file count without a rescan.
let mut media_manifest: Vec<(String, MediaSource)> = Vec::new();
for (i, row) in uploads.iter().enumerate() {
let src = media_path.join(&row.original_path);
@@ -231,8 +251,10 @@ async fn run_html_export(
let is_video = row.mime_type.starts_with("video/");
let id_str = row.id.to_string();
// Generate thumbnail and full variant
let (thumb_name, full_name) = if is_video {
// Generate thumbnail and full variant. `full_source` says where the full-res
// bytes come from at ZIP time — for videos and small images that's the original
// on disk (streamed directly, never copied to temp).
let (thumb_name, full_name, full_source) = if is_video {
let thumb = format!("{id_str}_thumb.jpg");
let full_ext = ext_from_path(&row.original_path);
let full = format!("{id_str}.{full_ext}");
@@ -259,14 +281,13 @@ async fn run_html_export(
Ok(output) if output.status.success() => {}
_ => {
tracing::warn!("ffmpeg thumbnail failed for upload {}, skipping thumb", row.id);
// Create empty thumb entry — viewer handles missing thumbs gracefully
// Missing thumb entry — viewer handles missing thumbs gracefully.
}
}
// Copy video as-is
tokio::fs::copy(&src, media_tmp.join(&full)).await?;
(thumb, full)
// Stream the video full-res straight from the original at ZIP time — no
// copy to temp (that used to transiently double disk usage per video).
(thumb, full, MediaSource::Original(src.clone()))
} else {
let thumb = format!("{id_str}_thumb.jpg");
let ext = ext_from_path(&row.original_path);
@@ -291,13 +312,12 @@ async fn run_html_export(
tracing::warn!("thumbnail generation failed for upload {}: {e:#}", row.id);
}
// Full variant: compress if >5MB, otherwise copy original
// Full variant: compress to temp if >5MB, otherwise stream the original
// as-is (no temp copy).
let src_meta = tokio::fs::metadata(&src).await?;
let full_path = media_tmp.join(&full);
if src_meta.len() > 5_000_000 {
// Resize to max 2000px
let full_source = if src_meta.len() > 5_000_000 {
let src_clone = src.clone();
let full_path = media_tmp.join(&full);
let full_path_clone = full_path.clone();
let compress_result = tokio::task::spawn_blocking(move || -> Result<()> {
@@ -311,17 +331,31 @@ async fn run_html_export(
})
.await?;
if let Err(e) = compress_result {
tracing::warn!("compression failed for upload {}, copying original: {e:#}", row.id);
tokio::fs::copy(&src, &full_path).await?;
match compress_result {
Ok(()) => MediaSource::Temp(full_path),
Err(e) => {
tracing::warn!(
"compression failed for upload {}, using original: {e:#}",
row.id
);
MediaSource::Original(src.clone())
}
}
} else {
tokio::fs::copy(&src, &full_path).await?;
}
MediaSource::Original(src.clone())
};
(thumb, full)
(thumb, full, full_source)
};
// Register this post's two media entries. Thumbnails always come from temp
// (they're freshly generated); the full variant's source was decided above.
media_manifest.push((
thumb_name.clone(),
MediaSource::Temp(media_tmp.join(&thumb_name)),
));
media_manifest.push((full_name.clone(), full_source));
// Build comments for this upload
let post_comments: Vec<ViewerComment> = comments
.iter()
@@ -409,27 +443,23 @@ async fn run_html_export(
update_progress(pool, event_id, "html", 78).await;
// Write media files from temp directory
let mut media_entries = tokio::fs::read_dir(&media_tmp).await?;
let mut file_count = 0u32;
// Write media files from the manifest built in step 3. Thumbnails and derived
// image variants stream from temp; video and small-image full variants stream
// straight from the original on disk. Sources that don't exist (e.g. a thumb
// whose ffmpeg step failed) are skipped — the viewer tolerates gaps.
let file_total = media_manifest.len().max(1) as f32;
let mut files_written = 0u32;
// Count files first
{
let mut counter = tokio::fs::read_dir(&media_tmp).await?;
while counter.next_entry().await?.is_some() {
file_count += 1;
for (name, source) in &media_manifest {
let path = source.path();
if !path.exists() {
continue;
}
}
let file_total = file_count.max(1) as f32;
while let Some(dir_entry) = media_entries.next_entry().await? {
let filename = dir_entry.file_name();
let entry_name = format!("media/{}", filename.to_string_lossy());
let entry_name = format!("media/{name}");
let builder = ZipEntryBuilder::new(entry_name.into(), Compression::Stored);
let mut zip_entry = zip.write_entry_stream(builder).await?;
let mut f = tokio::fs::File::open(dir_entry.path()).await?.compat();
let mut f = tokio::fs::File::open(path).await?.compat();
fcopy(&mut f, &mut zip_entry).await?;
zip_entry.close().await?;