fabi/Mangalord
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(admin): crawler observability dashboard + reliability hardening (0.55.0)

Browse Source 
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.

Backend:

- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
  into status / control / dead_jobs / backlog modules. SSE stream
  composes in-memory status with DB-derived queue counts, memoizes
  the counts for 1s and debounces watch pokes for 250ms (~10x QPS
  reduction per subscriber). One-shot GET /admin/crawler shares the
  same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
  (409 Conflict on overlapping click); browser restart goes through
  the coordinated_restart gate (drain + relaunch + auto-clear of the
  sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
  validation, never logged, audit row carries SHA-256 fingerprint).
  Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
  scope=all requires confirm:true; DISTINCT ON dedup keeps the
  partial unique index from rejecting requeues for chapters with
  multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
  panics; last-writer-wins on cover so concurrent dispatches don't
  clobber each other's slot. Pure functions (should_stop /
  should_mark_clean_exit / should_abort_pass) with named regression
  tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
  per-job timeout, circuit breaker on consecutive failures, BrowserManager
  coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
  LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
  Pages stream through with peak memory ~one HTTP chunk + 64-byte
  sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
  and crawler_jobs_dead_idx, both ordered by updated_at DESC to
  match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
  on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
  admin_no_store_guard (Cache-Control: no-store on admin
  responses), audit rows carry per-scope target_id.

Frontend:

- /admin/crawler page decomposed into lib/components/crawler/
  (11 components: ProgressBar, SearchBar, CrawlerHero,
  CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
  MissingCoversTable, DeadJobsTable, RestartConfirmModal,
  RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
  orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
  pageshow (BFCache); after 5 consecutive errors probes the status
  endpoint so a 401 routes through the global on401Hook instead of
  infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
  AbortControllers; refresh after a control action only runs when
  the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
  sync_state locally (no full chapter-list refetch); proper
  aria-label. Requeue-all gets its own confirm modal; both confirm
  modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
  for Accept: text/event-stream; pure shouldBypassProxyTimeout
  helper covered by unit tests.

Config / docs:

- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
  CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
  CRAWLER_BROWSER_RESTART_THRESHOLD.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-06 18:49:56 +02:00
parent 679abae736
commit d6ac648ac9
54 changed files with 6076 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files

349
backend/src/crawler/content.rs
View File

@@ -18,9 +18,9 @@ use uuid::Uuid;
use crate::crawler::detect::PageError;
use crate::crawler::rate_limit::HostRateLimiters;
use crate::crawler::safety::{fetch_bytes_capped, looks_like_image, DownloadAllowlist};
use crate::crawler::safety::{fetch_stream, looks_like_image, DownloadAllowlist};
use crate::crawler::session::{self, ChapterProbe};
use crate::storage::Storage;
use crate::storage::{Storage, StorageError};
/// Parse the chapter page DOM and return the page images in `pageN`
/// order. Filters out the loader `<img class="loading">` and any
@@ -186,11 +186,17 @@ where
}
}
/// Fetch all images for one chapter and persist them atomically. On
/// any error after the first storage put, the DB transaction rolls
/// back so the chapter stays at `page_count = 0` and is retried on the
/// next run. Bytes already written to storage become orphans; a future
/// reaper sweeps them.
/// Fetch one chapter's images and persist them. Each image is
/// streamed straight to storage via `Storage::put_stream` after a
/// short prefix is peeked off the body for content-type sniffing —
/// peak memory per concurrent dispatch is one HTTP chunk plus the
/// sniff prefix, not a full multi-MB image. The per-image size cap
/// (`CRAWLER_MAX_IMAGE_BYTES`) is enforced inside the stream so a
/// server that omits Content-Length still can't exhaust memory. The
/// page rows + `page_count` are then written in one short transaction.
/// On any failure the chapter stays at `page_count = 0` (no partial
/// rows) and the blobs already written are deleted best-effort by
/// [`cleanup_orphans`], so a retry starts clean.
#[allow(clippy::too_many_arguments)]
pub async fn sync_chapter_content(
browser: &chromiumoxide::Browser,
@@ -205,6 +211,10 @@ pub async fn sync_chapter_content(
allowlist: &DownloadAllowlist,
max_image_bytes: usize,
tor: Option<&crate::crawler::tor::TorController>,
// Optional live-status sink for the realtime page counter. The daemon
// dispatcher passes the shared handle (the chapter has already been
// registered via `begin_chapter`); the CLI / admin resync pass `None`.
progress: Option<&crate::crawler::status::StatusHandle>,
) -> anyhow::Result<SyncOutcome> {
// Skip if already fetched, unless caller explicitly forces.
if !force_refetch {
@@ -260,56 +270,189 @@ pub async fn sync_chapter_content(
// Resolve image URLs against the chapter URL (they may be relative).
let base = reqwest::Url::parse(source_url).context("parse chapter URL")?;
// Fetch every image bytes-first into memory before writing
// anything. Lets us bail the whole chapter cleanly if any image
// fails — DB stays at page_count=0, no partial rows persisted.
let mut fetched: Vec<(i32, Vec<u8>, &'static str)> = Vec::with_capacity(images.len());
// Stream each image straight to storage as it's fetched, capping peak
// memory at a single image rather than the whole chapter. Track the
// keys written so they can be rolled back if a later page (or the
// final DB commit) fails — preserving the all-or-nothing guarantee
// without holding a DB transaction open across the network puts
// (which matters once `Storage` is backed by S3).
let total = images.len();
// Publish the now-known page total so the dashboard shows "0/N".
if let Some(p) = progress {
p.set_chapter_pages(chapter_id, 0, Some(total));
}
let mut written_keys: Vec<String> = Vec::with_capacity(total);
let mut stored: Vec<StoredPage> = Vec::with_capacity(total);
for img in &images {
let url = base.join(&img.url).with_context(|| {
format!("join image URL {} onto {source_url}", img.url)
})?;
rate.wait_for(url.as_str()).await?;
let bytes = fetch_bytes_capped(
match download_and_store_page(
storage,
http,
url.as_str(),
Some(source_url),
rate,
&base,
source_url,
manga_id,
chapter_id,
img,
allowlist,
max_image_bytes,
)
.await?
.to_vec();
// Reject any non-image response: the only valid output of an
// image URL is an image. `infer` returns None on truncated
// bytes too, which also wants to be a failure not a silent
// `.bin` extension.
if !looks_like_image(&bytes) {
anyhow::bail!(
"image URL {url} returned non-image bytes \
(first 16: {:?}); refusing to store as binary blob",
&bytes.get(..16.min(bytes.len()))
);
.await
{
Ok(page) => {
written_keys.push(page.storage_key.clone());
stored.push(page);
// Live page counter: push the climbing count to subscribers.
if let Some(p) = progress {
p.set_chapter_pages(chapter_id, stored.len(), Some(total));
}
}
Err(e) => {
cleanup_orphans(storage, &written_keys).await;
return Err(e);
}
}
let ext = infer::get(&bytes)
.map(|k| k.extension())
.expect("looks_like_image asserted infer succeeded");
fetched.push((img.page_number, bytes, ext));
}
// Atomic write: storage puts + page row inserts + page_count
// update, all in one transaction. If anything fails, rollback +
// the chapter is retried next run. Storage orphans the bytes; a
// reaper sweeps them later.
let mut tx = db.begin().await.context("open chapter sync tx")?;
for (page_number, bytes, ext) in &fetched {
let key = format!(
"mangas/{manga_id}/chapters/{chapter_id}/pages/{:04}.{ext}",
page_number
// Short transaction: page rows + page_count only, no network I/O. On
// failure, roll back the stored bytes so the chapter stays at
// page_count=0 and is retried cleanly next run.
if let Err(e) = persist_pages(db, chapter_id, &stored).await {
cleanup_orphans(storage, &written_keys).await;
return Err(e);
}
Ok(SyncOutcome::Fetched { pages: stored.len() })
}
/// A page image that has been written to storage and is awaiting its DB
/// row. Carries everything `persist_pages` needs.
pub(crate) struct StoredPage {
page_number: i32,
storage_key: String,
content_type: String,
}
/// Bytes accumulated for content-type sniffing. `infer` only needs the
/// first few bytes for image formats (the longest signature in our
/// allow-list is AVIF at 12 bytes), but we read up to this many so a
/// fragmented TCP frame still produces a confident sniff and the
/// "first 16 bytes" diagnostic in the error path is useful.
const SNIFF_PREFIX_BYTES: usize = 64;
/// Download a single page image, validate it's really an image, and
/// stream it to storage. Returns the storage key + content type. Does
/// not touch the DB — persistence is batched into one short transaction
/// afterward.
///
/// Streaming path: we peek the first [`SNIFF_PREFIX_BYTES`] from the
/// HTTP body to determine the file extension (and thus the storage
/// key), then re-emit those bytes followed by the rest of the response
/// stream via `Storage::put_stream`. Peak memory per concurrent
/// dispatch is one HTTP chunk (~16 KiB) plus the sniff prefix, not a
/// full multi-MB image. The per-image cap is enforced as bytes flow.
#[allow(clippy::too_many_arguments)]
async fn download_and_store_page(
storage: &dyn Storage,
http: &reqwest::Client,
rate: &HostRateLimiters,
base: &reqwest::Url,
source_url: &str,
manga_id: Uuid,
chapter_id: Uuid,
img: &ChapterImage,
allowlist: &DownloadAllowlist,
max_image_bytes: usize,
) -> anyhow::Result<StoredPage> {
use futures_util::StreamExt as _;
let url = base
.join(&img.url)
.with_context(|| format!("join image URL {} onto {source_url}", img.url))?;
rate.wait_for(url.as_str()).await?;
let resp = fetch_stream(http, url.as_str(), Some(source_url), allowlist).await?;
let mut body = resp.bytes_stream();
// Drain chunks until we have enough bytes to sniff confidently
// (or the body is shorter than the prefix). Enforces the per-image
// cap on the prefix accumulation too.
let mut prefix = bytes::BytesMut::new();
while prefix.len() < SNIFF_PREFIX_BYTES {
let Some(chunk) = body.next().await else { break };
let chunk = chunk
.with_context(|| format!("stream chunk for {url}"))?;
if prefix.len().saturating_add(chunk.len()) > max_image_bytes {
anyhow::bail!(
"image {url} exceeds {max_image_bytes}-byte cap (received >{}+{})",
prefix.len(),
chunk.len()
);
}
prefix.extend_from_slice(&chunk);
}
let prefix = prefix.freeze();
// Reject any non-image response: the only valid output of an image
// URL is an image. `infer` returns None on truncated bytes too,
// which is also a failure not a silent `.bin` extension.
if !looks_like_image(&prefix) {
anyhow::bail!(
"image URL {url} returned non-image bytes \
(first 16: {:?}); refusing to store as binary blob",
&prefix.get(..16.min(prefix.len()))
);
storage
.put(&key, bytes)
.await
.with_context(|| format!("put {key}"))?;
// (chapter_id, page_number) is unique — re-runs idempotent.
}
let ext = infer::get(&prefix)
.map(|k| k.extension())
.expect("looks_like_image asserted infer succeeded");
let key = format!(
"mangas/{manga_id}/chapters/{chapter_id}/pages/{:04}.{ext}",
img.page_number
);
// Build a single stream of (prefix + remaining body) and pipe it
// straight to storage. The cap is enforced via a running total in
// the stream adapter so a server that omits Content-Length still
// can't exhaust memory.
let prefix_stream = futures_util::stream::once(async move {
Ok::<bytes::Bytes, StorageError>(prefix)
});
let prefix_len = SNIFF_PREFIX_BYTES.min(max_image_bytes);
let mut remaining = max_image_bytes.saturating_sub(prefix_len);
let url_for_err = url.clone();
let rest_stream = body.map(move |frame| match frame {
Ok(chunk) => {
if chunk.len() > remaining {
return Err(StorageError::Io(std::io::Error::other(format!(
"image {url_for_err} exceeds {max_image_bytes}-byte cap"
))));
}
remaining -= chunk.len();
Ok(chunk)
}
Err(e) => Err(StorageError::Io(std::io::Error::other(format!(
"stream chunk for {url_for_err}: {e}"
)))),
});
let combined = prefix_stream.chain(rest_stream);
storage
.put_stream(&key, Box::pin(combined))
.await
.with_context(|| format!("put_stream {key}"))?;
Ok(StoredPage {
page_number: img.page_number,
storage_key: key,
content_type: format!("image/{ext}"),
})
}
/// Persist the page rows + chapter `page_count` in one short transaction.
/// `(chapter_id, page_number)` is unique so re-runs are idempotent.
pub(crate) async fn persist_pages(
db: &PgPool,
chapter_id: Uuid,
stored: &[StoredPage],
) -> anyhow::Result<()> {
let mut tx = db.begin().await.context("open chapter sync tx")?;
for page in stored {
sqlx::query(
"INSERT INTO pages (chapter_id, page_number, storage_key, content_type)
VALUES ($1, $2, $3, $4)
@@ -318,22 +461,36 @@ pub async fn sync_chapter_content(
content_type = EXCLUDED.content_type",
)
.bind(chapter_id)
.bind(page_number)
.bind(&key)
.bind(format!("image/{ext}"))
.bind(page.page_number)
.bind(&page.storage_key)
.bind(&page.content_type)
.execute(&mut *tx)
.await
.with_context(|| format!("insert page row {page_number}"))?;
.with_context(|| format!("insert page row {}", page.page_number))?;
}
sqlx::query("UPDATE chapters SET page_count = $1 WHERE id = $2")
.bind(fetched.len() as i32)
.bind(stored.len() as i32)
.bind(chapter_id)
.execute(&mut *tx)
.await
.context("update page_count")?;
tx.commit().await.context("commit chapter sync")?;
Ok(())
}
Ok(SyncOutcome::Fetched { pages: fetched.len() })
/// Best-effort delete of partially-written page blobs after a chapter sync
/// fails, so a retry doesn't accumulate orphans. Errors are logged, not
/// raised — a leftover blob is harmless and a future reaper can sweep it.
pub(crate) async fn cleanup_orphans(storage: &dyn Storage, keys: &[String]) {
for key in keys {
if let Err(e) = storage.delete(key).await {
tracing::warn!(
%key,
error = ?e,
"failed to delete orphaned page blob after chapter sync failure"
);
}
}
}
// Suppress unused-import warning for `session::registrable_domain`
@@ -347,6 +504,90 @@ fn _keep_session_in_scope() {
#[cfg(test)]
mod tests {
use super::*;
use crate::storage::LocalStorage;
#[tokio::test]
async fn cleanup_orphans_deletes_written_keys() {
let dir = tempfile::tempdir().unwrap();
let storage = LocalStorage::new(dir.path());
let keys = vec![
"mangas/m/chapters/c/pages/0001.jpg".to_string(),
"mangas/m/chapters/c/pages/0002.jpg".to_string(),
];
for k in &keys {
storage.put(k, b"\xff\xd8\xff\xe0 jpeg-ish").await.unwrap();
assert!(storage.exists(k).await.unwrap());
}
cleanup_orphans(&storage, &keys).await;
for k in &keys {
assert!(!storage.exists(k).await.unwrap(), "{k} should be deleted");
}
}
#[tokio::test]
async fn cleanup_orphans_tolerates_missing_keys() {
// A key that was never written (e.g. the put itself failed) must
// not make cleanup error — it's best-effort.
let dir = tempfile::tempdir().unwrap();
let storage = LocalStorage::new(dir.path());
cleanup_orphans(&storage, &["never/written.jpg".to_string()]).await;
}
#[sqlx::test(migrations = "./migrations")]
async fn persist_pages_inserts_rows_and_sets_page_count(pool: PgPool) {
let manga_id = Uuid::new_v4();
let chapter_id = Uuid::new_v4();
sqlx::query("INSERT INTO mangas (id, title) VALUES ($1, 'T')")
.bind(manga_id)
.execute(&pool)
.await
.unwrap();
sqlx::query("INSERT INTO chapters (id, manga_id, number) VALUES ($1, $2, 1)")
.bind(chapter_id)
.bind(manga_id)
.execute(&pool)
.await
.unwrap();
let stored = vec![
StoredPage {
page_number: 1,
storage_key: "k/0001.jpg".into(),
content_type: "image/jpeg".into(),
},
StoredPage {
page_number: 2,
storage_key: "k/0002.jpg".into(),
content_type: "image/jpeg".into(),
},
];
persist_pages(&pool, chapter_id, &stored).await.unwrap();
let page_count: i32 =
sqlx::query_scalar("SELECT page_count FROM chapters WHERE id = $1")
.bind(chapter_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(page_count, 2);
let rows: i64 =
sqlx::query_scalar("SELECT COUNT(*) FROM pages WHERE chapter_id = $1")
.bind(chapter_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(rows, 2);
// Idempotent re-run (force refetch path): same rows, page_count stable.
persist_pages(&pool, chapter_id, &stored).await.unwrap();
let rows2: i64 =
sqlx::query_scalar("SELECT COUNT(*) FROM pages WHERE chapter_id = $1")
.bind(chapter_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(rows2, 2, "re-run is idempotent via ON CONFLICT");
}
#[test]
fn parse_chapter_pages_skips_loader_and_sorts_by_id() {