fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

perf(backend): close config/upload/export hot paths + stability fixes

Browse Source 
Performance:
- Cache the runtime `config` table in-memory (ConfigCache) with synchronous
  invalidation on every write (admin PATCH + test reseed). Was re-reading each
  key from Postgres on every request (~8 round-trips per upload).
- Stream uploads chunk-by-chunk to a temp file instead of buffering the whole
  body in RAM (peak was up to the per-class cap, e.g. 500 MB/video); only 512
  sniff-bytes are kept for magic-byte detection, then atomic rename into place.
- Cache the media-filesystem disk snapshot (DiskCache, 15s TTL) shared by the
  quota check and admin stats; drop the discarded System::refresh_all().
- HTML export streams video (and small-image) originals straight into the ZIP
  via a manifest instead of copying them to a temp dir first (removed the
  transient 2x disk usage) and drops the double directory scan.
- Auth extractor resolves session -> live user in one JOIN (was two queries),
  touching last_seen_at by token hash.

Stability:
- SSE: on broadcast lag, emit a `resync` event so the client runs a delta
  fetch instead of silently losing events; frontend reconciles adds, deletions,
  and (via an in-place refresh) like/comment counts on visible cards.
- Storage quota fails OPEN when the disk can't be read (was a 0-byte limit that
  locked out all uploads).
- Graceful shutdown drains in-flight requests on SIGTERM/SIGINT, bounded by a
  10s backstop so open SSE streams can't stall a deploy.
- Upload removes the persisted file if the DB transaction fails (no orphaned
  bytes with no row to reclaim them).

Tests:
- New pure select_disk() with 5 unit tests (longest-prefix, fallbacks, fail-open).
- New e2e export-video spec covering the HTML export's video-streaming branch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
fabi
2026-07-07 20:26:45 +02:00
parent 4cdb3ae14a
commit d6c91974eb
18 changed files with 745 additions and 214 deletions
Download Patch File Download Diff File Expand all files Collapse all files

123
backend/src/services/config.rs
View File

@@ -1,46 +1,129 @@
//! Reads of the runtime-tunable `config` table.
//! Reads of the runtime-tunable `config` table, fronted by an in-memory cache.
//!
//! Each handler used to keep a small local copy of these helpers; consolidating them
//! here means one place to add a parser, one place to mock for tests, and one place to
//! find when a key changes. New keys do not require code changes — they're picked up
//! the next time someone calls `get_*`.
//! the next time the cache reloads.
//!
//! ## Why a cache
//!
//! The `config` table is effectively static during an event, yet it was the busiest
//! query in the system: every request re-read each key with its own `SELECT`
//! (an upload touched it ~8 times). Against the small connection pool that was the
//! throughput ceiling. [`ConfigCache`] loads the whole table once and serves reads
//! from memory.
//!
//! ## Consistency contract
//!
//! Correctness comes from **synchronous invalidation on every write**, not from the
//! TTL. The two runtime write paths — the admin `PATCH /admin/config` handler and the
//! test-mode truncate/reseed — both call [`ConfigCache::invalidate`] after committing,
//! so the *next* read reloads from the DB and sees the new value immediately. The
//! [`RELOAD_TTL`] is only a safety net for out-of-band changes (e.g. a migration or a
//! manual DB edit); it is deliberately short but never the primary mechanism.
//!
//! Values are read with a default fallback so the app still starts if a key is missing
//! (e.g. during a migration window). Production seeds keys via migrations 005 and 009.
use std::collections::HashMap;
use std::sync::{Arc, RwLock};
use std::time::{Duration, Instant};
use sqlx::PgPool;
async fn fetch_raw(pool: &PgPool, key: &str) -> Option<String> {
sqlx::query_as::<_, (String,)>("SELECT value FROM config WHERE key = $1")
.bind(key)
.fetch_optional(pool)
.await
.ok()
.flatten()
.map(|(v,)| v)
/// How long a loaded snapshot is trusted before the next read reloads it. This is a
/// backstop for out-of-band DB changes only — every in-process write invalidates the
/// cache synchronously, so tests that PATCH-then-assert never depend on this expiring.
const RELOAD_TTL: Duration = Duration::from_secs(30);
struct Snapshot {
values: HashMap<String, String>,
loaded_at: Instant,
}
pub async fn get_str(pool: &PgPool, key: &str, default: &str) -> String {
fetch_raw(pool, key).await.unwrap_or_else(|| default.to_string())
/// In-memory cache of the entire `config` table. Cheap to `clone` (shares the pool and
/// the `Arc`), so it lives in `AppState` and every handler reads through it.
#[derive(Clone)]
pub struct ConfigCache {
pool: PgPool,
inner: Arc<RwLock<Option<Snapshot>>>,
}
pub async fn get_i64(pool: &PgPool, key: &str, default: i64) -> i64 {
fetch_raw(pool, key).await.and_then(|v| v.parse().ok()).unwrap_or(default)
impl ConfigCache {
pub fn new(pool: PgPool) -> Self {
Self {
pool,
inner: Arc::new(RwLock::new(None)),
}
}
/// Drop the cached snapshot so the next read reloads the whole table from the DB.
/// Call this after any write to the `config` table (admin PATCH, test reseed).
pub fn invalidate(&self) {
*self.inner.write().unwrap() = None;
}
/// Return the fresh snapshot if one is loaded and still within [`RELOAD_TTL`].
fn fresh_snapshot(&self) -> Option<HashMap<String, String>> {
let guard = self.inner.read().unwrap();
match guard.as_ref() {
Some(snap) if snap.loaded_at.elapsed() < RELOAD_TTL => Some(snap.values.clone()),
_ => None,
}
}
/// Read one key, loading the whole table into the cache on a miss/expiry. On a DB
/// error we return `None` (callers fall back to their default) without poisoning
/// the cache.
async fn get_raw(&self, key: &str) -> Option<String> {
if let Some(values) = self.fresh_snapshot() {
return values.get(key).cloned();
}
// Cache miss or stale — reload the entire table in one query.
let rows: Vec<(String, String)> =
match sqlx::query_as::<_, (String, String)>("SELECT key, value FROM config")
.fetch_all(&self.pool)
.await
{
Ok(rows) => rows,
Err(e) => {
tracing::warn!(error = ?e, "config reload failed; using defaults for this read");
return None;
}
};
let values: HashMap<String, String> = rows.into_iter().collect();
let result = values.get(key).cloned();
*self.inner.write().unwrap() = Some(Snapshot {
values,
loaded_at: Instant::now(),
});
result
}
}
pub async fn get_usize(pool: &PgPool, key: &str, default: usize) -> usize {
fetch_raw(pool, key).await.and_then(|v| v.parse().ok()).unwrap_or(default)
pub async fn get_str(cache: &ConfigCache, key: &str, default: &str) -> String {
cache.get_raw(key).await.unwrap_or_else(|| default.to_string())
}
pub async fn get_f64(pool: &PgPool, key: &str, default: f64) -> f64 {
fetch_raw(pool, key).await.and_then(|v| v.parse().ok()).unwrap_or(default)
pub async fn get_i64(cache: &ConfigCache, key: &str, default: i64) -> i64 {
cache.get_raw(key).await.and_then(|v| v.parse().ok()).unwrap_or(default)
}
pub async fn get_usize(cache: &ConfigCache, key: &str, default: usize) -> usize {
cache.get_raw(key).await.and_then(|v| v.parse().ok()).unwrap_or(default)
}
pub async fn get_f64(cache: &ConfigCache, key: &str, default: f64) -> f64 {
cache.get_raw(key).await.and_then(|v| v.parse().ok()).unwrap_or(default)
}
/// Parses common truthy spellings used by both the migration seeds and the admin form.
/// Accepts `true/false`, `1/0`, `yes/no`, `on/off` — case-insensitive. Anything else
/// returns `default`.
pub async fn get_bool(pool: &PgPool, key: &str, default: bool) -> bool {
let Some(raw) = fetch_raw(pool, key).await else { return default };
pub async fn get_bool(cache: &ConfigCache, key: &str, default: bool) -> bool {
let Some(raw) = cache.get_raw(key).await else { return default };
match raw.trim().to_ascii_lowercase().as_str() {
"true" | "1" | "yes" | "on" => true,
"false" | "0" | "no" | "off" => false,