feat(admin): crawler observability dashboard + reliability hardening (0.55.0)

Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.

Backend:

- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
  into status / control / dead_jobs / backlog modules. SSE stream
  composes in-memory status with DB-derived queue counts, memoizes
  the counts for 1s and debounces watch pokes for 250ms (~10x QPS
  reduction per subscriber). One-shot GET /admin/crawler shares the
  same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
  (409 Conflict on overlapping click); browser restart goes through
  the coordinated_restart gate (drain + relaunch + auto-clear of the
  sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
  validation, never logged, audit row carries SHA-256 fingerprint).
  Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
  scope=all requires confirm:true; DISTINCT ON dedup keeps the
  partial unique index from rejecting requeues for chapters with
  multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
  panics; last-writer-wins on cover so concurrent dispatches don't
  clobber each other's slot. Pure functions (should_stop /
  should_mark_clean_exit / should_abort_pass) with named regression
  tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
  per-job timeout, circuit breaker on consecutive failures, BrowserManager
  coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
  LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
  Pages stream through with peak memory ~one HTTP chunk + 64-byte
  sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
  and crawler_jobs_dead_idx, both ordered by updated_at DESC to
  match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
  on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
  admin_no_store_guard (Cache-Control: no-store on admin
  responses), audit rows carry per-scope target_id.

Frontend:

- /admin/crawler page decomposed into lib/components/crawler/
  (11 components: ProgressBar, SearchBar, CrawlerHero,
  CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
  MissingCoversTable, DeadJobsTable, RestartConfirmModal,
  RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
  orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
  pageshow (BFCache); after 5 consecutive errors probes the status
  endpoint so a 401 routes through the global on401Hook instead of
  infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
  AbortControllers; refresh after a control action only runs when
  the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
  sync_state locally (no full chapter-list refetch); proper
  aria-label. Requeue-all gets its own confirm modal; both confirm
  modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
  for Accept: text/event-stream; pure shouldBypassProxyTimeout
  helper covered by unit tests.

Config / docs:

- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
  CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
  CRAWLER_BROWSER_RESTART_THRESHOLD.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend/src/api/admin/crawler/backlog.rs

@@ -0,0 +1,67 @@
+//! GET /admin/crawler/active-jobs — paginated `pending|running` chapters
+//! GET /admin/crawler/covers — paginated mangas missing a cover
+//!
+//! These are pure DB-derived reads that drive two of the three
+//! backlog tables on the dashboard. The third (dead jobs) lives in
+//! the [`super::dead_jobs`] module because it also exposes a write.
+
+use axum::extract::{Query, State};
+use axum::routing::get;
+use axum::{Json, Router};
+use serde::Deserialize;
+
+use crate::app::AppState;
+use crate::auth::extractor::RequireAdmin;
+use crate::error::AppResult;
+use crate::repo;
+use crate::repo::crawler::{ActiveJob, MissingCoverRow};
+
+use super::default_limit;
+
+pub(super) fn routes() -> Router<AppState> {
+    Router::new()
+        .route("/admin/crawler/active-jobs", get(list_active_jobs))
+        .route("/admin/crawler/covers", get(list_covers))
+}
+
+/// Pagination + title-search params shared by the backlog list endpoints.
+#[derive(Debug, Deserialize, Default)]
+struct ListParams {
+    #[serde(default)]
+    search: Option<String>,
+    #[serde(default = "default_limit")]
+    limit: i64,
+    #[serde(default)]
+    offset: i64,
+}
+
+async fn list_active_jobs(
+    State(state): State<AppState>,
+    _admin: RequireAdmin,
+    Query(params): Query<ListParams>,
+) -> AppResult<Json<crate::api::pagination::PagedResponse<ActiveJob>>> {
+    let limit = params.limit.clamp(1, 200);
+    let offset = params.offset.max(0);
+    let search = params.search.filter(|s| !s.trim().is_empty());
+    let (items, total) =
+        repo::crawler::list_active_jobs(&state.db, search.as_deref(), limit, offset).await?;
+    Ok(Json(crate::api::pagination::PagedResponse::with_total(
+        items, limit, offset, total,
+    )))
+}
+
+async fn list_covers(
+    State(state): State<AppState>,
+    _admin: RequireAdmin,
+    Query(params): Query<ListParams>,
+) -> AppResult<Json<crate::api::pagination::PagedResponse<MissingCoverRow>>> {
+    let limit = params.limit.clamp(1, 200);
+    let offset = params.offset.max(0);
+    let search = params.search.filter(|s| !s.trim().is_empty());
+    let (items, total) =
+        repo::crawler::list_missing_cover_mangas(&state.db, search.as_deref(), limit, offset)
+            .await?;
+    Ok(Json(crate::api::pagination::PagedResponse::with_total(
+        items, limit, offset, total,
+    )))
+}

backend/src/api/admin/crawler/control.rs

@@ -0,0 +1,206 @@
+//! POST /admin/crawler/run — trigger an out-of-cycle metadata pass
+//! POST /admin/crawler/browser/restart — coordinated restart
+//! POST /admin/crawler/session — refresh PHPSESSID
+//! POST /admin/crawler/session/clear-expired — clear sticky expired flag
+//!
+//! All four mutate live in-process state on `CrawlerControl` (browser
+//! manager, session controller, manual-pass mutex). They each emit an
+//! `admin_audit` row and `status.poke()` so SSE subscribers see the
+//! change instantly without polling.
+
+use axum::extract::State;
+use axum::routing::post;
+use axum::{Json, Router};
+use serde::{Deserialize, Serialize};
+use serde_json::json;
+
+use crate::app::AppState;
+use crate::auth::extractor::RequireAdmin;
+use crate::error::{AppError, AppResult};
+use crate::repo;
+
+use super::require_crawler;
+
+pub(super) fn routes() -> Router<AppState> {
+    Router::new()
+        .route("/admin/crawler/run", post(run_now))
+        .route("/admin/crawler/browser/restart", post(restart_browser))
+        .route("/admin/crawler/session", post(update_session))
+        .route(
+            "/admin/crawler/session/clear-expired",
+            post(clear_session_expired),
+        )
+}
+
+#[derive(Debug, Serialize)]
+struct RunResponse {
+    started: bool,
+}
+
+async fn run_now(
+    State(state): State<AppState>,
+    admin: RequireAdmin,
+) -> AppResult<Json<RunResponse>> {
+    let c = require_crawler(&state)?;
+    let mp = c.metadata_pass.as_ref().ok_or_else(|| {
+        AppError::ServiceUnavailable("no source configured (CRAWLER_START_URL unset)".into())
+    })?;
+    // Operator-click dedup. A pass holds the lock for its entire run
+    // (minutes); a second click while it's in flight returns 409 instead
+    // of fanning a second pass onto the single browser lease. The daily
+    // cron does NOT take this lock — cron is single-fire by definition,
+    // and its own contention with a manual pass already serialises
+    // through the browser lease + advisory lock at a lower layer.
+    let pass_guard = c
+        .manual_pass_lock
+        .clone()
+        .try_lock_owned()
+        .map_err(|_| AppError::Conflict("manual metadata pass already running".into()))?;
+    let mp = std::sync::Arc::clone(mp);
+    // Fire-and-forget: the pass can run for minutes; the dashboard
+    // streams progress over SSE. The guard moves into the task so the
+    // lock is released only when the pass finishes (or the task panics).
+    tokio::spawn(async move {
+        let _pass_guard = pass_guard;
+        if let Err(e) = mp.run().await {
+            tracing::warn!(error = ?e, "manual metadata pass failed");
+        }
+    });
+    repo::admin_audit::insert(&state.db, admin.0.id, "crawler_run", "crawler", None, json!({}))
+        .await?;
+    Ok(Json(RunResponse { started: true }))
+}
+
+#[derive(Debug, Serialize)]
+struct RestartResponse {
+    ok: bool,
+    error: Option<String>,
+}
+
+async fn restart_browser(
+    State(state): State<AppState>,
+    admin: RequireAdmin,
+) -> AppResult<Json<RestartResponse>> {
+    let c = require_crawler(&state)?;
+    let result = c.browser_manager.coordinated_restart(c.drain_deadline).await;
+    // A successful coordinated_restart re-runs on_launch, which re-injects
+    // PHPSESSID and re-probes — i.e. the session is live. Drop the sticky
+    // `session_expired` flag so chapter workers stop idling without
+    // requiring a second click on "Clear expired".
+    if result.is_ok() {
+        c.session.clear_expired();
+    }
+    // Push the post-restart browser phase to live subscribers immediately.
+    c.status.poke();
+    repo::admin_audit::insert(
+        &state.db,
+        admin.0.id,
+        "crawler_browser_restart",
+        "crawler",
+        None,
+        json!({ "ok": result.is_ok() }),
+    )
+    .await?;
+    Ok(Json(match result {
+        Ok(()) => RestartResponse {
+            ok: true,
+            error: None,
+        },
+        Err(e) => RestartResponse {
+            ok: false,
+            error: Some(format!("{e:#}")),
+        },
+    }))
+}
+
+#[derive(Debug, Deserialize)]
+struct UpdateSessionRequest {
+    phpsessid: String,
+}
+
+#[derive(Debug, Serialize)]
+struct UpdateSessionResponse {
+    /// Whether the post-update browser relaunch + session probe succeeded.
+    valid: bool,
+    error: Option<String>,
+}
+
+async fn update_session(
+    State(state): State<AppState>,
+    admin: RequireAdmin,
+    Json(body): Json<UpdateSessionRequest>,
+) -> AppResult<Json<UpdateSessionResponse>> {
+    let c = require_crawler(&state)?;
+    // Fingerprint BEFORE move so the raw value never reaches tracing or
+    // the audit row. SHA256-prefix is opaque to anyone reading the audit
+    // log but deterministic enough to correlate two updates of the same
+    // session value.
+    let fingerprint = phpsessid_fingerprint(&body.phpsessid);
+    c.session
+        .update(&body.phpsessid)
+        .await
+        .map_err(|e| AppError::InvalidInput(format!("{e:#}")))?;
+    // Relaunch the browser so on_launch re-injects the new cookie and
+    // re-probes — the restart's success IS the session-validity signal.
+    let probe = c.browser_manager.coordinated_restart(c.drain_deadline).await;
+    // Session + browser state changed — push to live subscribers.
+    c.status.poke();
+    repo::admin_audit::insert(
+        &state.db,
+        admin.0.id,
+        "crawler_session_update",
+        "crawler",
+        None,
+        json!({ "valid": probe.is_ok(), "phpsessid_fingerprint": fingerprint }),
+    )
+    .await?;
+    Ok(Json(match probe {
+        Ok(()) => UpdateSessionResponse {
+            valid: true,
+            error: None,
+        },
+        Err(e) => UpdateSessionResponse {
+            valid: false,
+            error: Some(format!("{e:#}")),
+        },
+    }))
+}
+
+#[derive(Debug, Serialize)]
+struct ClearExpiredResponse {
+    cleared: bool,
+}
+
+async fn clear_session_expired(
+    State(state): State<AppState>,
+    admin: RequireAdmin,
+) -> AppResult<Json<ClearExpiredResponse>> {
+    let c = require_crawler(&state)?;
+    c.session.clear_expired();
+    // session.expired flipped — push to live subscribers.
+    c.status.poke();
+    repo::admin_audit::insert(
+        &state.db,
+        admin.0.id,
+        "crawler_session_clear_expired",
+        "crawler",
+        None,
+        json!({}),
+    )
+    .await?;
+    Ok(Json(ClearExpiredResponse { cleared: true }))
+}
+
+/// Opaque, short fingerprint of a PHPSESSID for the admin audit log.
+/// The first 8 hex chars of SHA-256 — enough to correlate two updates
+/// of the same value without revealing the raw cookie. Reading the audit
+/// row does not give an operator anything that can re-construct the
+/// session.
+fn phpsessid_fingerprint(sid: &str) -> String {
+    use sha2::{Digest, Sha256};
+    let mut h = Sha256::new();
+    h.update(sid.as_bytes());
+    let digest = h.finalize();
+    let hex: String = digest.iter().take(4).map(|b| format!("{b:02x}")).collect();
+    hex
+}

backend/src/api/admin/crawler/dead_jobs.rs

@@ -0,0 +1,121 @@
+//! GET /admin/crawler/dead-jobs — paginated dead-letter list
+//! POST /admin/crawler/dead-jobs/requeue — flip dead jobs back to pending
+//!
+//! Requeue scopes:
+//! - `all` (requires `confirm: true` so a careless click / CSRF bait
+//!   can't flip the whole pile in one shot)
+//! - `manga` (all dead jobs whose chapter belongs to a manga)
+//! - `chapter` (all dead jobs for a single chapter)
+//! - `job` (a single dead row by id)
+//!
+//! Each requeue emits an `admin_audit` row with the relevant
+//! `target_id` populated, so an operator review post-incident can pin
+//! exactly which scope was acted on.
+
+use axum::extract::{Query, State};
+use axum::routing::{get, post};
+use axum::{Json, Router};
+use serde::{Deserialize, Serialize};
+use serde_json::json;
+use uuid::Uuid;
+
+use crate::app::AppState;
+use crate::auth::extractor::RequireAdmin;
+use crate::error::{AppError, AppResult};
+use crate::repo;
+use crate::repo::crawler::{DeadJob, RequeueScope};
+
+use super::default_limit;
+
+pub(super) fn routes() -> Router<AppState> {
+    Router::new()
+        .route("/admin/crawler/dead-jobs", get(list_dead_jobs))
+        .route("/admin/crawler/dead-jobs/requeue", post(requeue_dead_jobs))
+}
+
+#[derive(Debug, Deserialize, Default)]
+struct DeadJobsParams {
+    #[serde(default)]
+    search: Option<String>,
+    #[serde(default = "default_limit")]
+    limit: i64,
+    #[serde(default)]
+    offset: i64,
+}
+
+async fn list_dead_jobs(
+    State(state): State<AppState>,
+    _admin: RequireAdmin,
+    Query(params): Query<DeadJobsParams>,
+) -> AppResult<Json<crate::api::pagination::PagedResponse<DeadJob>>> {
+    let limit = params.limit.clamp(1, 200);
+    let offset = params.offset.max(0);
+    let search = params.search.filter(|s| !s.trim().is_empty());
+    let (items, total) =
+        repo::crawler::list_dead_jobs(&state.db, search.as_deref(), limit, offset).await?;
+    Ok(Json(crate::api::pagination::PagedResponse::with_total(
+        items, limit, offset, total,
+    )))
+}
+
+#[derive(Debug, Deserialize)]
+#[serde(tag = "scope", rename_all = "snake_case")]
+enum RequeueRequest {
+    /// `confirm: true` is required so a careless click / CSRF bait can't
+    /// flip the entire dead pile in one shot. Narrow scopes don't need it.
+    All {
+        #[serde(default)]
+        confirm: bool,
+    },
+    Manga { manga_id: Uuid },
+    Chapter { chapter_id: Uuid },
+    Job { job_id: Uuid },
+}
+
+#[derive(Debug, Serialize)]
+struct RequeueResponse {
+    requeued: u64,
+}
+
+async fn requeue_dead_jobs(
+    State(state): State<AppState>,
+    admin: RequireAdmin,
+    Json(body): Json<RequeueRequest>,
+) -> AppResult<Json<RequeueResponse>> {
+    // Reject scope=all without an explicit confirm so a single click or
+    // CSRF bait can't flip the whole dead pile. Narrow scopes don't need
+    // the confirmation — the operator already named a specific target.
+    if let RequeueRequest::All { confirm: false } = &body {
+        return Err(AppError::InvalidInput(
+            "confirm: true is required for scope=all".into(),
+        ));
+    }
+    let (scope, target_kind, target_id) = match &body {
+        RequeueRequest::All { .. } => (RequeueScope::All, "crawler", None),
+        RequeueRequest::Manga { manga_id } => (RequeueScope::Manga(*manga_id), "manga", Some(*manga_id)),
+        RequeueRequest::Chapter { chapter_id } => {
+            (RequeueScope::Chapter(*chapter_id), "chapter", Some(*chapter_id))
+        }
+        RequeueRequest::Job { job_id } => (RequeueScope::Job(*job_id), "crawler_job", Some(*job_id)),
+    };
+    let requeued = repo::crawler::requeue_dead_jobs(&state.db, scope).await?;
+    repo::admin_audit::insert(
+        &state.db,
+        admin.0.id,
+        "crawler_dead_jobs_requeue",
+        target_kind,
+        target_id,
+        json!({ "requeued": requeued, "scope": scope_label(&body) }),
+    )
+    .await?;
+    Ok(Json(RequeueResponse { requeued }))
+}
+
+fn scope_label(r: &RequeueRequest) -> &'static str {
+    match r {
+        RequeueRequest::All { .. } => "all",
+        RequeueRequest::Manga { .. } => "manga",
+        RequeueRequest::Chapter { .. } => "chapter",
+        RequeueRequest::Job { .. } => "job",
+    }
+}

backend/src/api/admin/crawler/mod.rs

@@ -0,0 +1,49 @@
+//! Admin-only crawler observability + control endpoints.
+//!
+//! Mounted under `/api/v1/admin/crawler*`, cookie-only via `RequireAdmin`.
+//! All control endpoints return 503 when the crawler daemon is disabled
+//! (`AppState.crawler == None`). Reads compose the live in-process status
+//! ([`crate::crawler::status`]) with DB-derived queue counts and the
+//! session/browser flags.
+//!
+//! Split into four siblings for navigability — the surface area grew
+//! past the point where keeping it in one file made review harder:
+//! - [`status`] — SSE stream + composed status snapshot
+//! - [`control`] — run / restart / session endpoints
+//! - [`dead_jobs`] — dead-letter list + requeue
+//! - [`backlog`] — pending-chapters and missing-covers backlog reads
+
+mod backlog;
+mod control;
+mod dead_jobs;
+mod status;
+
+use axum::Router;
+
+use crate::app::{AppState, CrawlerControl};
+use crate::error::AppError;
+
+pub fn routes() -> Router<AppState> {
+    Router::new()
+        .merge(status::routes())
+        .merge(control::routes())
+        .merge(dead_jobs::routes())
+        .merge(backlog::routes())
+}
+
+/// Default page size for the backlog list endpoints.
+pub(super) fn default_limit() -> i64 {
+    50
+}
+
+/// Shared 503 helper: the daemon-only control + observability endpoints
+/// gate on `AppState.crawler == Some(_)`. Returns the same code and body
+/// across every caller so the frontend can rely on `service_unavailable`
+/// rather than each handler returning its own variant.
+pub(super) fn require_crawler(
+    state: &AppState,
+) -> Result<&std::sync::Arc<CrawlerControl>, AppError> {
+    state.crawler.as_ref().ok_or_else(|| {
+        AppError::ServiceUnavailable("crawler daemon is disabled".into())
+    })
+}

backend/src/api/admin/crawler/status.rs

@@ -0,0 +1,245 @@
+//! GET /admin/crawler — composed status snapshot
+//! GET /admin/crawler/stream — Server-Sent Events live status
+//!
+//! The composed response merges the in-process status surface
+//! ([`crate::crawler::status`]) with two DB-derived counts
+//! (job-state breakdown and missing-cover backlog) so the dashboard
+//! reads the same shape from one one-shot fetch and from each SSE
+//! frame. The streaming path debounces a burst of pokes into a single
+//! frame and memoizes the DB counts for the [`QUEUE_MEMO_TTL`] window
+//! to avoid hammering Postgres.
+
+use std::convert::Infallible;
+use std::time::Duration;
+
+use axum::extract::State;
+use axum::response::sse::{Event, KeepAlive, Sse};
+use axum::routing::get;
+use axum::{Json, Router};
+use futures_util::stream::Stream;
+use serde::Serialize;
+
+use crate::app::AppState;
+use crate::auth::extractor::RequireAdmin;
+use crate::crawler::browser_manager::RestartPhase;
+use crate::crawler::status::{ActiveChapter, CoverTarget, LastPass, Phase};
+use crate::error::AppResult;
+use crate::repo;
+
+/// Backstop recompose interval for the SSE stream. Phase/worker/session
+/// changes push instantly via the status `watch`; this only bounds the
+/// staleness of DB-derived queue counts and the browser phase when those
+/// change without an accompanying status poke.
+const SSE_BACKSTOP: Duration = Duration::from_secs(5);
+
+/// Coalesce a burst of status pokes (e.g. one per stored page during a
+/// chapter download) into a single SSE frame. After the first change
+/// fires we sleep this long and absorb any further changes that arrive
+/// in the window before emitting. Tighter than human-perceivable jitter
+/// (~16-100 ms is the rule of thumb for "instant").
+const SSE_DEBOUNCE: Duration = Duration::from_millis(250);
+
+/// Per-connection memo TTL for DB-derived queue counts. With a busy
+/// chapter pass bumping the watch up to several times per second, the
+/// memo collapses N stream wakeups into one round-trip per second
+/// — typically a ~10x reduction in steady-state DB QPS per subscriber.
+const QUEUE_MEMO_TTL: Duration = Duration::from_millis(1000);
+
+pub(super) fn routes() -> Router<AppState> {
+    Router::new()
+        .route("/admin/crawler", get(get_status))
+        .route("/admin/crawler/stream", get(stream_status))
+}
+
+#[derive(Debug, Serialize)]
+struct QueueCounts {
+    pending: i64,
+    running: i64,
+    dead: i64,
+}
+
+#[derive(Debug, Serialize)]
+struct SessionStatus {
+    /// Whether the sticky session-expired flag is set (chapter workers idle).
+    expired: bool,
+    /// Whether a PHPSESSID is currently configured at all.
+    configured: bool,
+}
+
+#[derive(Debug, Serialize)]
+struct CrawlerStatusResponse {
+    /// `"running"` | `"disabled"`.
+    daemon: &'static str,
+    phase: Option<Phase>,
+    /// Configured chapter-worker count (for "N busy / M workers").
+    worker_count: usize,
+    /// Chapters being crawled right now, with live page counts.
+    active_chapters: Vec<ActiveChapter>,
+    /// The cover being fetched right now, if any.
+    current_cover: Option<CoverTarget>,
+    /// Mangas still queued for a cover fetch.
+    covers_queued: i64,
+    last_pass: LastPass,
+    session: SessionStatus,
+    /// `"healthy"` | `"draining"` | `"restarting"` | `"down"`.
+    browser: &'static str,
+    queue: QueueCounts,
+}
+
+/// Per-stream memo of the two DB-derived counts shared by every SSE
+/// frame: crawler-job state breakdown and missing-cover backlog. Held
+/// across iterations of the unfold loop so a burst of status pokes
+/// emits one frame from cached counts. A fresh memo (`new`) always
+/// misses on first call, so `get_status` (one-shot) sees no stale data.
+struct QueueCountsMemo {
+    cached: Option<(std::time::Instant, (i64, i64, i64), i64)>,
+}
+
+impl QueueCountsMemo {
+    fn new() -> Self {
+        Self { cached: None }
+    }
+
+    async fn get(
+        &mut self,
+        db: &sqlx::PgPool,
+    ) -> sqlx::Result<((i64, i64, i64), i64)> {
+        if let Some((at, qc, cv)) = self.cached.as_ref() {
+            if at.elapsed() < QUEUE_MEMO_TTL {
+                return Ok((*qc, *cv));
+            }
+        }
+        let qc = repo::crawler::job_state_counts(db).await?;
+        let cv = repo::crawler::count_missing_covers(db).await?;
+        self.cached = Some((std::time::Instant::now(), qc, cv));
+        Ok((qc, cv))
+    }
+}
+
+fn browser_phase_str(p: RestartPhase) -> &'static str {
+    match p {
+        RestartPhase::Healthy => "healthy",
+        RestartPhase::Draining => "draining",
+        RestartPhase::Restarting => "restarting",
+    }
+}
+
+/// Compose a full status snapshot from the in-memory status, the
+/// browser/session flags, and DB queue-count queries (routed through
+/// a memo so a burst of pokes doesn't hammer Postgres). Shared by
+/// `get_status` (with a fresh per-call memo) and `stream_status`
+/// (with a per-stream memo held across iterations).
+async fn compose_status(
+    state: &AppState,
+    memo: &mut QueueCountsMemo,
+) -> AppResult<CrawlerStatusResponse> {
+    let ((pending, running, dead), covers_queued) = memo.get(&state.db).await?;
+    let queue = QueueCounts {
+        pending,
+        running,
+        dead,
+    };
+
+    Ok(match state.crawler.as_ref() {
+        None => CrawlerStatusResponse {
+            daemon: "disabled",
+            phase: None,
+            worker_count: 0,
+            active_chapters: Vec::new(),
+            current_cover: None,
+            covers_queued,
+            last_pass: LastPass::default(),
+            session: SessionStatus {
+                expired: false,
+                configured: false,
+            },
+            browser: "down",
+            queue,
+        },
+        Some(c) => {
+            let snap = c.status.snapshot().await;
+            CrawlerStatusResponse {
+                daemon: "running",
+                phase: Some(snap.phase),
+                worker_count: snap.worker_count,
+                active_chapters: snap.active_chapters,
+                current_cover: snap.current_cover,
+                covers_queued,
+                last_pass: snap.last_pass,
+                session: SessionStatus {
+                    expired: c.session.is_expired(),
+                    configured: c.session.current().await.is_some(),
+                },
+                browser: browser_phase_str(c.browser_manager.phase()),
+                queue,
+            }
+        }
+    })
+}
+
+async fn get_status(
+    State(state): State<AppState>,
+    _admin: RequireAdmin,
+) -> AppResult<Json<CrawlerStatusResponse>> {
+    // Fresh memo — one-shot calls always query the DB. The wrapper exists
+    // only so compose_status can share its signature with the streaming
+    // path; there is no caching across one-shot calls.
+    let mut memo = QueueCountsMemo::new();
+    Ok(Json(compose_status(&state, &mut memo).await?))
+}
+
+/// Push live status to the dashboard instead of polling. Emits a snapshot
+/// immediately on connect, then on every status change (instant, via the
+/// `watch` notifier) and on a [`SSE_BACKSTOP`] tick (to refresh DB queue
+/// counts / browser phase that change without a status poke). The browser
+/// opens this only while the crawler page is mounted and closes it on
+/// navigate-away, so the subscription is scoped to the active page.
+async fn stream_status(
+    State(state): State<AppState>,
+    _admin: RequireAdmin,
+) -> Sse<impl Stream<Item = Result<Event, Infallible>>> {
+    // Subscribe before the first emit so no change between the initial
+    // snapshot and the first await is lost.
+    let rx = state.crawler.as_ref().map(|c| c.status.subscribe());
+    let memo = QueueCountsMemo::new();
+
+    let stream = futures_util::stream::unfold(
+        (state, rx, memo, true),
+        |(state, mut rx, mut memo, first)| async move {
+            // After the first immediate emit, wait for a change or the
+            // backstop tick before recomposing. On a change, debounce a
+            // short window so a burst of pokes (one per stored page
+            // during a chapter download, etc.) coalesces into a single
+            // frame instead of hammering subscribers.
+            if !first {
+                match rx.as_mut() {
+                    Some(rx) => {
+                        tokio::select! {
+                            _ = rx.changed() => {
+                                tokio::time::sleep(SSE_DEBOUNCE).await;
+                                // Mark any pokes that arrived during the
+                                // debounce window as observed so the next
+                                // iteration only fires on NEW changes.
+                                rx.borrow_and_update();
+                            }
+                            _ = tokio::time::sleep(SSE_BACKSTOP) => {}
+                        }
+                    }
+                    None => tokio::time::sleep(SSE_BACKSTOP).await,
+                }
+            }
+            // Compose; on a transient DB error, emit a keep-alive comment
+            // rather than tearing down the stream.
+            let event = match compose_status(&state, &mut memo).await {
+                Ok(resp) => Event::default()
+                    .event("status")
+                    .json_data(&resp)
+                    .unwrap_or_else(|_| Event::default().comment("serialize error")),
+                Err(_) => Event::default().comment("status unavailable"),
+            };
+            Some((Ok(event), (state, rx, memo, false)))
+        },
+    );
+
+    Sse::new(stream).keep_alive(KeepAlive::default())
+}

backend/src/api/admin/mod.rs

@@ -4,6 +4,7 @@
 //! bot/API tokens cannot reach admin routes (see
 //! `crate::auth::extractor::RequireAdmin`).
 
+pub mod crawler;
 pub mod mangas;
 pub mod resync;
 pub mod system;
@@ -19,4 +20,5 @@ pub fn routes() -> Router<AppState> {
         .merge(mangas::routes())
         .merge(resync::routes())
         .merge(system::routes())
+        .merge(crawler::routes())
 }