perf(admin): fix O(mangas x jobs) overview query that pinned Postgres (0.85.1)
All checks were successful
deploy / test-backend (push) Successful in 26m28s
deploy / test-frontend (push) Successful in 10m18s
deploy / build-and-push (push) Successful in 11m21s
deploy / deploy (push) Successful in 13s

The admin overview dashboard polls /admin/overview every 30s. Its
`manga_stats` aggregate evaluated MANGA_SYNC_STATE_CASE over the WHOLE
library (14.5k mangas) with no LIMIT, and that CASE runs a correlated
EXISTS over crawler_jobs. With no index on the `sync_chapter_list`
manga_id path, Postgres seqscanned all in-flight jobs *per manga* —
O(mangas x jobs). The disabled-analysis backlog (9.5k pending
`analyze_page` rows that can never match the sync-kind filter) inflated
every scan, so each call took 6-7 min. The 30s poll stacked ~10 of them
concurrently → load 11, Postgres at ~100%.

EXPLAIN ANALYZE on the live DB: the rewrite drops the query from 6-7 min
to ~1.5s (per-manga crawler_jobs seqscan → single index-backed pass).

Fixes:
- Rewrite `manga_stats` to collect the in-flight manga-id set in ONE pass
  over the sync jobs (index-backed), then classify via a hash semi-join —
  O(mangas + jobs), immune to the analyze_page backlog size.
- Add partial index crawler_jobs_sync_chapter_list_manga_idx (migration
  0029) covering the sync_chapter_list manga_id path; mirrors the existing
  sync_manga index (0020). Also speeds the per-row Mangas tab listing.
- statement_timeout backstop (5s) on the aggregate so a pathological plan
  can never pin a backend for minutes again.
- Single-flight + 10s TTL cache on the /admin/overview handler so
  concurrent pollers coalesce instead of stacking.
- Slow the dashboard poll 30s -> 60s (server-cached now anyway).

The in_progress rule in the rewrite is kept in lockstep with the first
arm of MANGA_SYNC_STATE_CASE (documented in both places).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-17 17:05:40 +02:00
parent cde4aca98b
commit 35664bccc7
7 changed files with 127 additions and 20 deletions

View File

@@ -9,11 +9,15 @@
//!
//! Admin-only (`RequireAdmin`, cookie-only).
use std::sync::OnceLock;
use std::time::{Duration, Instant};
use axum::extract::State;
use axum::routing::get;
use axum::{Json, Router};
use chrono::{DateTime, Utc};
use serde::Serialize;
use tokio::sync::Mutex;
use crate::app::AppState;
use crate::auth::extractor::RequireAdmin;
@@ -24,14 +28,14 @@ pub fn routes() -> Router<AppState> {
Router::new().route("/admin/overview", get(overview))
}
#[derive(Debug, Serialize)]
#[derive(Debug, Clone, Serialize)]
pub struct OverviewStats {
pub users: UsersOverview,
pub mangas: repo::admin_view::MangaStats,
pub analysis: AnalysisOverview,
}
#[derive(Debug, Serialize)]
#[derive(Debug, Clone, Serialize)]
pub struct UsersOverview {
pub total: i64,
pub admins: i64,
@@ -39,16 +43,42 @@ pub struct UsersOverview {
pub newest_created_at: Option<DateTime<Utc>>,
}
#[derive(Debug, Serialize)]
#[derive(Debug, Clone, Serialize)]
pub struct AnalysisOverview {
pub analyzed_pages: i64,
pub total_pages: i64,
}
/// Process-wide single-flight cache for the overview aggregate.
///
/// The admin dashboard polls this endpoint on a timer; without coalescing, a
/// slow DB (e.g. a deep crawl/analysis backlog) lets identical requests stack
/// and pin Postgres. Holding the async mutex across the recompute gives BOTH a
/// short-TTL cache and single-flight: concurrent pollers block on the lock,
/// then read the freshly cached value instead of each launching their own
/// aggregate. Resets on process restart, which is fine for a dashboard stat.
///
/// Unlike per-`AppState` shared state (e.g. `auth_limiter`), this is a single
/// production-instance cache, so a process global is adequate. It is
/// test-safe because the cache lives in the handler *body*: the two auth tests
/// are rejected at the `RequireAdmin` extractor before reaching it, and the
/// one aggregate test issues a single request. A future test that mutates data
/// and re-reads within the TTL would need to account for this.
static OVERVIEW_CACHE: OnceLock<Mutex<Option<(Instant, OverviewStats)>>> = OnceLock::new();
const OVERVIEW_TTL: Duration = Duration::from_secs(10);
async fn overview(
State(state): State<AppState>,
_admin: RequireAdmin,
) -> AppResult<Json<OverviewStats>> {
let cache = OVERVIEW_CACHE.get_or_init(|| Mutex::new(None));
let mut guard = cache.lock().await;
if let Some((at, cached)) = guard.as_ref() {
if at.elapsed() < OVERVIEW_TTL {
return Ok(Json(cached.clone()));
}
}
// Independent reads — run concurrently so latency is the slowest
// query, not their sum (mirrors the storage handler).
let (user_counts, user_newest, mangas, coverage) = tokio::try_join!(
@@ -65,7 +95,7 @@ async fn overview(
};
let (analyzed_pages, total_pages) = coverage;
Ok(Json(OverviewStats {
let stats = OverviewStats {
users: UsersOverview {
total,
admins,
@@ -77,5 +107,7 @@ async fn overview(
analyzed_pages,
total_pages,
},
}))
};
*guard = Some((Instant::now(), stats.clone()));
Ok(Json(stats))
}