fabi/Mangalord
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(admin): crawler observability dashboard + reliability hardening (0.55.0)

Browse Source 
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.

Backend:

- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
  into status / control / dead_jobs / backlog modules. SSE stream
  composes in-memory status with DB-derived queue counts, memoizes
  the counts for 1s and debounces watch pokes for 250ms (~10x QPS
  reduction per subscriber). One-shot GET /admin/crawler shares the
  same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
  (409 Conflict on overlapping click); browser restart goes through
  the coordinated_restart gate (drain + relaunch + auto-clear of the
  sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
  validation, never logged, audit row carries SHA-256 fingerprint).
  Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
  scope=all requires confirm:true; DISTINCT ON dedup keeps the
  partial unique index from rejecting requeues for chapters with
  multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
  panics; last-writer-wins on cover so concurrent dispatches don't
  clobber each other's slot. Pure functions (should_stop /
  should_mark_clean_exit / should_abort_pass) with named regression
  tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
  per-job timeout, circuit breaker on consecutive failures, BrowserManager
  coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
  LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
  Pages stream through with peak memory ~one HTTP chunk + 64-byte
  sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
  and crawler_jobs_dead_idx, both ordered by updated_at DESC to
  match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
  on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
  admin_no_store_guard (Cache-Control: no-store on admin
  responses), audit rows carry per-scope target_id.

Frontend:

- /admin/crawler page decomposed into lib/components/crawler/
  (11 components: ProgressBar, SearchBar, CrawlerHero,
  CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
  MissingCoversTable, DeadJobsTable, RestartConfirmModal,
  RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
  orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
  pageshow (BFCache); after 5 consecutive errors probes the status
  endpoint so a 401 routes through the global on401Hook instead of
  infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
  AbortControllers; refresh after a control action only runs when
  the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
  sync_state locally (no full chapter-list refetch); proper
  aria-label. Requeue-all gets its own confirm modal; both confirm
  modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
  for Accept: text/event-stream; pure shouldBypassProxyTimeout
  helper covered by unit tests.

Config / docs:

- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
  CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
  CRAWLER_BROWSER_RESTART_THRESHOLD.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-06 18:49:56 +02:00
parent 679abae736
commit d6ac648ac9
54 changed files with 6076 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files

279
backend/src/app.rs
View File

@@ -1,5 +1,5 @@
use std::sync::Arc;
use std::sync::atomic::AtomicBool;
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
use anyhow::Context;
use async_trait::async_trait;
@@ -46,6 +46,38 @@ pub struct AppState {
/// same wiring that builds the daemon's chapter dispatcher, so a
/// force resync uses the daemon's BrowserManager + rate limiters.
pub resync: Option<Arc<dyn ResyncService>>,
/// Crawler observability + control handle (live status, coordinated
/// browser restart, runtime session, manual run). `None` when the
/// daemon is disabled; admin handlers gate on `.is_some()` → 503.
pub crawler: Option<Arc<CrawlerControl>>,
/// Browser origins permitted to issue mutating requests to
/// `/api/v1/admin/*`. See [`crate::config::Config::admin_allowed_origins`]
/// for the policy. Cloned per-request into the CSRF middleware; the
/// `Arc` keeps the clone cheap. Empty list → check is skipped
/// (operator opt-out documented in `.env.example`).
pub admin_allowed_origins: Arc<Vec<String>>,
}
/// Shared handle the admin crawler endpoints use to observe and control
/// the running daemon. Bundled so the handlers take one optional field on
/// `AppState` rather than many.
pub struct CrawlerControl {
pub browser_manager: Arc<BrowserManager>,
pub session: Arc<crate::crawler::session_control::SessionController>,
pub status: crate::crawler::status::StatusHandle,
/// Used by the "run metadata pass now" endpoint; `None` when no
/// `CRAWLER_START_URL` is configured (cron disabled).
pub metadata_pass: Option<Arc<dyn MetadataPass>>,
/// Drain budget for a manually-triggered coordinated browser restart.
pub drain_deadline: std::time::Duration,
/// Held for the duration of a `/admin/crawler/run` pass so a second
/// click returns 409 instead of fanning N overlapping metadata passes
/// onto the single browser lease. The daemon's daily cron does NOT
/// take this lock — cron and operator-triggered are different
/// trust modes (cron is single-fire by definition; the lock only
/// dedups operator clicks). `Arc` so the spawned task can hold an
/// owned guard past the request boundary.
pub manual_pass_lock: Arc<tokio::sync::Mutex<()>>,
}
/// Bundle returned by [`build`]. The router is what `axum::serve` consumes;
@@ -80,12 +112,12 @@ pub async fn build(config: Config) -> anyhow::Result<AppHandle> {
let storage: Arc<dyn Storage> = Arc::new(LocalStorage::new(config.storage_dir.clone()));
let (daemon, resync) = if config.crawler.daemon_enabled {
let (daemon, resync, crawler) = if config.crawler.daemon_enabled {
let spawned = spawn_crawler_daemon(db.clone(), Arc::clone(&storage), &config.crawler).await?;
(Some(spawned.handle), Some(spawned.resync))
(Some(spawned.handle), Some(spawned.resync), Some(spawned.crawler))
} else {
tracing::info!("crawler daemon disabled (CRAWLER_DAEMON=false)");
(None, None)
(None, None, None)
};
let auth_limiter = Arc::new(AuthRateLimiter::new(config.auth.rate_limit));
@@ -96,6 +128,8 @@ pub async fn build(config: Config) -> anyhow::Result<AppHandle> {
upload: config.upload.clone(),
auth_limiter,
resync,
crawler,
admin_allowed_origins: Arc::new(config.admin_allowed_origins.clone()),
};
let router = router(state).layer(cors_layer(&config.cors_allowed_origins));
Ok(AppHandle { router, daemon })
@@ -108,6 +142,7 @@ pub async fn build(config: Config) -> anyhow::Result<AppHandle> {
struct SpawnedDaemon {
handle: daemon::DaemonHandle,
resync: Arc<dyn ResyncService>,
crawler: Arc<CrawlerControl>,
}
async fn spawn_crawler_daemon(
@@ -115,11 +150,17 @@ async fn spawn_crawler_daemon(
storage: Arc<dyn Storage>,
cfg: &CrawlerConfig,
) -> anyhow::Result<SpawnedDaemon> {
// Reqwest client with cookie jar pre-seeded so CDN image fetches
// include PHPSESSID. Same shape as bin/crawler.rs main().
// Reqwest client with a shared cookie jar so CDN image fetches include
// PHPSESSID. The same `Arc<Jar>` is held by the SessionController, so a
// runtime session refresh rewrites it in place. Initial value: a
// persisted runtime session (survives restart) takes precedence over
// CRAWLER_PHPSESSID env.
let cookie_jar = Arc::new(reqwest::cookie::Jar::default());
let initial_sid = crate::crawler::session_control::SessionController::load_persisted(&db)
.await
.or_else(|| cfg.phpsessid.clone());
if let (Some(sid), Some(domain), Some(start_url)) =
(&cfg.phpsessid, &cfg.cookie_domain, &cfg.start_url)
(&initial_sid, &cfg.cookie_domain, &cfg.start_url)
{
let cookie_str = format!("PHPSESSID={sid}; Domain={domain}; Path=/");
let seed_url = reqwest::Url::parse(start_url)
@@ -129,7 +170,7 @@ async fn spawn_crawler_daemon(
let mut http_builder = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(30))
.no_proxy()
.cookie_provider(cookie_jar);
.cookie_provider(Arc::clone(&cookie_jar));
if let Some(ua) = &cfg.user_agent {
http_builder = http_builder.user_agent(ua);
}
@@ -157,6 +198,23 @@ async fn spawn_crawler_daemon(
}
let tor_recircuit_max = cfg.tor_recircuit_max_attempts;
// Session controller + sticky session-expired flag. Created before the
// browser so the on_launch hook can read the *current* session value
// (rather than a value captured at startup), and so a runtime refresh
// updates the cookie everywhere.
let session_expired = Arc::new(AtomicBool::new(false));
let session_controller = crate::crawler::session_control::SessionController::new(
initial_sid,
Arc::clone(&cookie_jar),
cfg.cookie_domain.clone(),
cfg.start_url.clone(),
db.clone(),
Arc::clone(&session_expired),
);
// Live status surface, sized to the worker count.
let status = crate::crawler::status::StatusHandle::new(cfg.chapter_workers);
// Browser manager. on_launch re-injects PHPSESSID on every fresh
// chromium spawn so an idle teardown followed by re-launch stays
// authenticated without operator action.
@@ -165,18 +223,25 @@ async fn spawn_crawler_daemon(
let chromium_proxy = crate::crawler::url_utils::chromium_proxy_arg(proxy);
launch_opts.extra_args.push(format!("--proxy-server={chromium_proxy}"));
}
let on_launch = match (&cfg.phpsessid, &cfg.cookie_domain, &cfg.start_url) {
(Some(sid), Some(domain), Some(start_url)) => {
let sid = sid.clone();
let on_launch = match (&cfg.cookie_domain, &cfg.start_url) {
(Some(domain), Some(start_url)) => {
let domain = domain.clone();
let start_url = start_url.clone();
let tor_for_launch = tor.as_ref().map(Arc::clone);
let sc = Arc::clone(&session_controller);
let on_launch: browser_manager::OnLaunch = Arc::new(move |browser| {
let sid = sid.clone();
let domain = domain.clone();
let start_url = start_url.clone();
let tor_for_launch = tor_for_launch.as_ref().map(Arc::clone);
let sc = Arc::clone(&sc);
Box::pin(async move {
// Read the *current* session each launch so a runtime
// refresh is picked up on the next (re)launch. No session
// configured → run unauthenticated (metadata needs no auth).
let Some(sid) = sc.current().await else {
tracing::info!("on_launch: no session set — skipping inject + probe");
return Ok(());
};
session::inject_phpsessid(&browser, &sid, &domain)
.await
.context("on_launch: inject_phpsessid")?;
@@ -197,8 +262,6 @@ async fn spawn_crawler_daemon(
};
let browser_manager = BrowserManager::new(launch_opts, cfg.idle_timeout, on_launch);
let session_expired = Arc::new(AtomicBool::new(false));
let metadata_pass: Option<Arc<dyn MetadataPass>> = cfg.start_url.as_ref().map(|url| {
let m: Arc<dyn MetadataPass> = Arc::new(RealMetadataPass {
browser_manager: Arc::clone(&browser_manager),
@@ -210,6 +273,8 @@ async fn spawn_crawler_daemon(
manga_limit: cfg.manga_limit,
download_allowlist: cfg.download_allowlist.clone(),
max_image_bytes: cfg.max_image_bytes,
metadata_max_consecutive_failures: cfg.metadata_max_consecutive_failures,
status: status.clone(),
tor: tor.as_ref().map(Arc::clone),
});
m
@@ -223,6 +288,10 @@ async fn spawn_crawler_daemon(
rate: Arc::clone(&rate),
download_allowlist: cfg.download_allowlist.clone(),
max_image_bytes: cfg.max_image_bytes,
transient_failures: Arc::new(AtomicU32::new(0)),
restart_threshold: cfg.browser_restart_threshold,
drain_deadline: cfg.job_timeout,
status: status.clone(),
tor: tor.as_ref().map(Arc::clone),
});
@@ -260,20 +329,32 @@ async fn spawn_crawler_daemon(
db,
cancel,
DaemonConfig {
metadata_pass,
metadata_pass: metadata_pass.clone(),
dispatcher,
chapter_workers: cfg.chapter_workers,
daily_at: cfg.daily_at,
tz: cfg.tz,
retention_days: cfg.retention_days,
session_expired,
status: status.clone(),
job_timeout: cfg.job_timeout,
extra_tasks: vec![reaper_task, shutdown_task],
},
);
let crawler = Arc::new(CrawlerControl {
browser_manager: Arc::clone(&browser_manager),
session: session_controller,
status,
metadata_pass,
drain_deadline: cfg.job_timeout,
manual_pass_lock: Arc::new(tokio::sync::Mutex::new(())),
});
Ok(SpawnedDaemon {
handle: daemon_handle,
resync,
crawler,
})
}
@@ -292,6 +373,8 @@ struct RealMetadataPass {
manga_limit: usize,
download_allowlist: DownloadAllowlist,
max_image_bytes: usize,
metadata_max_consecutive_failures: u32,
status: crate::crawler::status::StatusHandle,
tor: Option<Arc<crate::crawler::tor::TorController>>,
}
@@ -309,6 +392,8 @@ impl MetadataPass for RealMetadataPass {
false,
&self.download_allowlist,
self.max_image_bytes,
self.metadata_max_consecutive_failures,
Some(&self.status),
self.tor.as_deref(),
)
.await;
@@ -321,7 +406,8 @@ impl MetadataPass for RealMetadataPass {
// errored — the early-stop walk can complete its work and bail
// late, and a transient browser failure shouldn't cancel the
// residual cover backlog. The backfill has its own per-call cap
// so a runaway error stream can't monopolise the tick.
// so a runaway error stream can't monopolise the tick. It sets the
// CoverBackfill{index,total} phase + current_cover per entry.
match pipeline::backfill_missing_covers(
&self.browser_manager,
&self.db,
@@ -331,6 +417,7 @@ impl MetadataPass for RealMetadataPass {
pipeline::COVER_BACKFILL_DEFAULT_MAX,
&self.download_allowlist,
self.max_image_bytes,
Some(&self.status),
self.tor.as_deref(),
)
.await
@@ -359,6 +446,16 @@ struct RealChapterDispatcher {
rate: Arc<HostRateLimiters>,
download_allowlist: DownloadAllowlist,
max_image_bytes: usize,
/// Consecutive transient chapter failures; resets on any success.
/// Drives the automatic coordinated browser restart.
transient_failures: Arc<std::sync::atomic::AtomicU32>,
/// Consecutive-failure count that triggers an auto restart.
restart_threshold: u32,
/// How long a coordinated restart waits for in-flight leases to drain.
drain_deadline: std::time::Duration,
/// Live status surface — the dispatcher registers each chapter it
/// crawls (with a realtime page count) here.
status: crate::crawler::status::StatusHandle,
tor: Option<Arc<crate::crawler::tor::TorController>>,
}
@@ -374,10 +471,21 @@ impl ChapterDispatcher for RealChapterDispatcher {
let row = repo::chapter::dispatch_target(&self.db, chapter_id)
.await
.context("look up chapter for dispatch")?;
let Some((manga_id, source_url)) = row else {
let Some((manga_id, source_url, manga_title, chapter_number)) = row else {
// Chapter (or its source row) is gone — ack done.
return Ok(SyncOutcome::Skipped);
};
// Register the chapter as crawling now (live status). The
// guard removes it on every exit path — success, panic, or
// the worker's outer-timeout drop.
let _active = self.status.begin_chapter(crate::crawler::status::ActiveChapter {
manga_id,
manga_title,
chapter_id,
chapter_number,
pages_done: 0,
pages_total: None,
});
let lease = self.browser_manager.acquire().await?;
let result = content::sync_chapter_content(
&lease,
@@ -392,14 +500,37 @@ impl ChapterDispatcher for RealChapterDispatcher {
&self.download_allowlist,
self.max_image_bytes,
self.tor.as_deref(),
Some(&self.status),
)
.await;
drop(lease);
match result {
Ok(outcome) => Ok(outcome),
Ok(outcome) => {
// Any successful dispatch (including a clean Skipped)
// means the browser is healthy — reset the streak.
self.transient_failures.store(0, Ordering::Release);
Ok(outcome)
}
Err(e) => {
let streak = self.transient_failures.fetch_add(1, Ordering::AcqRel) + 1;
if crate::crawler::nav::anyhow_looks_browser_dead(&e) {
// Hard browser-dead: lazy invalidate (next acquire
// relaunches). Reset the streak — we're recovering.
self.browser_manager.invalidate().await;
self.transient_failures.store(0, Ordering::Release);
} else if self.restart_threshold > 0 && streak >= self.restart_threshold {
// Persistent transients that TOR recircuit couldn't
// fix — proactively restart Chromium.
tracing::warn!(
streak,
threshold = self.restart_threshold,
"auto browser restart: consecutive transient chapter failures"
);
let _ = self
.browser_manager
.coordinated_restart(self.drain_deadline)
.await;
self.transient_failures.store(0, Ordering::Release);
}
Err(e)
}
@@ -419,6 +550,11 @@ pub fn router(state: AppState) -> Router {
let max_request_bytes = state.upload.max_request_bytes;
Router::new()
.nest("/api/v1", crate::api::routes())
.layer(middleware::from_fn(admin_no_store_guard))
.layer(middleware::from_fn_with_state(
state.clone(),
admin_csrf_guard,
))
.layer(middleware::from_fn_with_state(
state.clone(),
private_mode_guard,
@@ -428,6 +564,113 @@ pub fn router(state: AppState) -> Router {
.layer(TraceLayer::new_for_http())
}
/// Path prefix the admin-only middlewares scope themselves to. The router
/// already nests `/api/v1`, so callers see `/api/v1/admin/...`.
const ADMIN_PATH_PREFIX: &str = "/api/v1/admin/";
/// CSRF defence for cookie-authenticated admin mutations. The session
/// cookie is `SameSite=Lax`, which still permits top-level form-POSTs
/// from a malicious page — this middleware rejects such requests by
/// comparing the request's `Origin` (with `Referer` as fallback) against
/// the configured allowlist. Safe methods (`GET`/`HEAD`/`OPTIONS`) are
/// always allowed. Requests with neither `Origin` nor `Referer` are
/// allowed (non-browser callers like curl can't be a CSRF vector). When
/// the allowlist is empty the check is skipped entirely (operator
/// opt-out — documented in `.env.example`).
async fn admin_csrf_guard(
State(state): State<AppState>,
req: Request,
next: Next,
) -> Result<Response, AppError> {
if !req.uri().path().starts_with(ADMIN_PATH_PREFIX) {
return Ok(next.run(req).await);
}
if matches!(
*req.method(),
Method::GET | Method::HEAD | Method::OPTIONS
) {
return Ok(next.run(req).await);
}
if state.admin_allowed_origins.is_empty() {
return Ok(next.run(req).await);
}
let headers = req.headers();
let origin = headers.get("origin").and_then(|v| v.to_str().ok());
let referer = headers.get("referer").and_then(|v| v.to_str().ok());
// No Origin AND no Referer → server-to-server / curl / extension.
// Browsers always send one or the other on a cross-site POST.
let Some(candidate) = origin.or(referer) else {
return Ok(next.run(req).await);
};
if origin_in_allowlist(candidate, &state.admin_allowed_origins) {
return Ok(next.run(req).await);
}
tracing::warn!(
candidate = %truncate_for_log(candidate, 64),
path = %req.uri().path(),
"admin CSRF: rejecting mutation with disallowed origin"
);
Err(AppError::Forbidden)
}
/// Match `candidate` (an `Origin` value, or a `Referer` URL whose
/// origin we'll extract) against `allowed`. `Origin` is `scheme://host[:port]`
/// with no path; `Referer` is a full URL — compare by parsing both and
/// matching scheme + host + port.
fn origin_in_allowlist(candidate: &str, allowed: &[String]) -> bool {
let cand_origin = parse_origin(candidate);
let Some(cand) = cand_origin else { return false };
allowed
.iter()
.filter_map(|a| parse_origin(a))
.any(|a| a == cand)
}
/// Extract the origin (`scheme://host[:port]`) from an `Origin` header
/// value or a `Referer` URL. Returns `None` when the input doesn't parse
/// as a URL with a host.
fn parse_origin(raw: &str) -> Option<String> {
let url = reqwest::Url::parse(raw.trim()).ok()?;
let host = url.host_str()?;
let scheme = url.scheme();
let port_str = match (url.port(), scheme) {
(Some(p), "http") if p == 80 => String::new(),
(Some(p), "https") if p == 443 => String::new(),
(Some(p), _) => format!(":{p}"),
(None, _) => String::new(),
};
Some(format!("{scheme}://{host}{port_str}"))
}
fn truncate_for_log(s: &str, max: usize) -> &str {
let end = s
.char_indices()
.take(max)
.last()
.map(|(i, c)| i + c.len_utf8())
.unwrap_or(0);
&s[..end.min(s.len())]
}
/// Forbids intermediaries (CDN, browser bfcache, reverse proxy with a
/// permissive default) from caching admin responses. Defence-in-depth
/// for cookie-authenticated reads — even though the responses already
/// vary on cookie, a misconfigured cache layer in front of the
/// SvelteKit container could leak a logged-in admin's view to another
/// session. Headers added on response so the rest of the API is
/// unaffected.
async fn admin_no_store_guard(req: Request, next: Next) -> Response {
let is_admin_path = req.uri().path().starts_with(ADMIN_PATH_PREFIX);
let mut resp = next.run(req).await;
if is_admin_path {
resp.headers_mut().insert(
axum::http::header::CACHE_CONTROL,
HeaderValue::from_static("no-store"),
);
}
resp
}
/// Paths reachable anonymously even when `PRIVATE_MODE=true`. Login and
/// logout are needed for the auth flow itself; `/health` is reserved
/// for load-balancer probes; `/auth/config` lets the frontend decide