feat(admin): crawler observability dashboard + reliability hardening (0.55.0)

Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.

Backend:

- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
  into status / control / dead_jobs / backlog modules. SSE stream
  composes in-memory status with DB-derived queue counts, memoizes
  the counts for 1s and debounces watch pokes for 250ms (~10x QPS
  reduction per subscriber). One-shot GET /admin/crawler shares the
  same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
  (409 Conflict on overlapping click); browser restart goes through
  the coordinated_restart gate (drain + relaunch + auto-clear of the
  sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
  validation, never logged, audit row carries SHA-256 fingerprint).
  Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
  scope=all requires confirm:true; DISTINCT ON dedup keeps the
  partial unique index from rejecting requeues for chapters with
  multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
  panics; last-writer-wins on cover so concurrent dispatches don't
  clobber each other's slot. Pure functions (should_stop /
  should_mark_clean_exit / should_abort_pass) with named regression
  tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
  per-job timeout, circuit breaker on consecutive failures, BrowserManager
  coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
  LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
  Pages stream through with peak memory ~one HTTP chunk + 64-byte
  sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
  and crawler_jobs_dead_idx, both ordered by updated_at DESC to
  match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
  on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
  admin_no_store_guard (Cache-Control: no-store on admin
  responses), audit rows carry per-scope target_id.

Frontend:

- /admin/crawler page decomposed into lib/components/crawler/
  (11 components: ProgressBar, SearchBar, CrawlerHero,
  CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
  MissingCoversTable, DeadJobsTable, RestartConfirmModal,
  RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
  orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
  pageshow (BFCache); after 5 consecutive errors probes the status
  endpoint so a 401 routes through the global on401Hook instead of
  infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
  AbortControllers; refresh after a control action only runs when
  the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
  sync_state locally (no full chapter-list refetch); proper
  aria-label. Requeue-all gets its own confirm modal; both confirm
  modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
  for Accept: text/event-stream; pure shouldBypassProxyTimeout
  helper covered by unit tests.

Config / docs:

- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
  CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
  CRAWLER_BROWSER_RESTART_THRESHOLD.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend/src/storage/local.rs

@@ -1,10 +1,12 @@
 use std::path::{Path, PathBuf};
 
 use async_trait::async_trait;
+use futures_util::StreamExt as _;
 use tokio::fs;
+use tokio::io::AsyncWriteExt as _;
 use tokio_util::io::ReaderStream;
 
-use super::{Storage, StorageError, StreamingFile};
+use super::{PutByteStream, Storage, StorageError, StreamingFile};
 
 pub struct LocalStorage {
     root: PathBuf,
@@ -45,6 +47,49 @@ impl Storage for LocalStorage {
         Ok(())
     }
 
+    async fn put_stream(
+        &self,
+        key: &str,
+        mut stream: PutByteStream<'_>,
+    ) -> Result<u64, StorageError> {
+        let path = self.resolve(key)?;
+        if let Some(parent) = path.parent() {
+            fs::create_dir_all(parent).await?;
+        }
+        // Atomic install via temp + rename. A failure mid-stream
+        // removes the temp so nothing is visible at `key`. The temp
+        // name uses a UUID suffix so concurrent puts of the same key
+        // (e.g. two workers racing a retry) don't clobber each
+        // other's in-progress file before the rename.
+        let tmp = path.with_extension(format!(
+            "{}.tmp.{}",
+            path.extension().and_then(|e| e.to_str()).unwrap_or(""),
+            uuid::Uuid::new_v4().simple()
+        ));
+        let mut written: u64 = 0;
+        let result: Result<(), StorageError> = async {
+            let mut f = fs::File::create(&tmp).await?;
+            while let Some(chunk) = stream.next().await {
+                let chunk = chunk?;
+                f.write_all(&chunk).await?;
+                written = written.saturating_add(chunk.len() as u64);
+            }
+            // fsync before rename so a power-loss can't leave a
+            // zero-byte file at the destination.
+            f.sync_all().await?;
+            fs::rename(&tmp, &path).await?;
+            Ok(())
+        }
+        .await;
+        if let Err(e) = result {
+            // Best-effort cleanup; ignore "no such file" if the temp
+            // was never created.
+            let _ = fs::remove_file(&tmp).await;
+            return Err(e);
+        }
+        Ok(written)
+    }
+
     async fn get(&self, key: &str) -> Result<Vec<u8>, StorageError> {
         let path = self.resolve(key)?;
         match fs::read(&path).await {
@@ -142,6 +187,52 @@ mod tests {
         ));
     }
 
+    #[tokio::test]
+    async fn put_stream_writes_full_body_and_removes_temp_on_error() {
+        use bytes::Bytes;
+        use futures_util::stream;
+
+        let dir = tempdir().unwrap();
+        let s = LocalStorage::new(dir.path());
+
+        // Success: a 3-chunk stream writes the concatenation and
+        // returns the right byte count.
+        let chunks: Vec<Result<Bytes, StorageError>> = vec![
+            Ok(Bytes::from_static(b"alpha-")),
+            Ok(Bytes::from_static(b"beta-")),
+            Ok(Bytes::from_static(b"gamma")),
+        ];
+        let bytes_written = s
+            .put_stream("streamed/ok.bin", Box::pin(stream::iter(chunks)))
+            .await
+            .unwrap();
+        assert_eq!(bytes_written, b"alpha-beta-gamma".len() as u64);
+        assert_eq!(s.get("streamed/ok.bin").await.unwrap(), b"alpha-beta-gamma");
+
+        // Failure mid-stream: nothing is visible at the destination
+        // and no .tmp file is left behind.
+        let chunks: Vec<Result<Bytes, StorageError>> = vec![
+            Ok(Bytes::from_static(b"good")),
+            Err(StorageError::Io(std::io::Error::other("boom"))),
+        ];
+        let err = s
+            .put_stream("streamed/bad.bin", Box::pin(stream::iter(chunks)))
+            .await
+            .unwrap_err();
+        assert!(matches!(err, StorageError::Io(_)));
+        assert!(matches!(
+            s.get("streamed/bad.bin").await,
+            Err(StorageError::NotFound)
+        ));
+        // No stray temp file in the streamed/ directory.
+        let entries: Vec<_> = std::fs::read_dir(dir.path().join("streamed"))
+            .unwrap()
+            .filter_map(|e| e.ok())
+            .map(|e| e.file_name().to_string_lossy().to_string())
+            .collect();
+        assert_eq!(entries, vec!["ok.bin"]);
+    }
+
     #[tokio::test]
     async fn get_stream_emits_multiple_chunks_for_large_files() {
         use futures_util::StreamExt as _;

backend/src/storage/mod.rs

@@ -31,6 +31,13 @@ pub enum StorageError {
 /// object-safe regardless of the concrete reader behind it.
 pub type ByteStream = Pin<Box<dyn Stream<Item = io::Result<Bytes>> + Send>>;
 
+/// Boxed byte stream accepted by `Storage::put_stream`. The item type
+/// is fallible so a producer (e.g. an HTTP body) can surface a transport
+/// error mid-stream without breaking the trait shape; the storage impl
+/// is responsible for not installing a partial blob on such an error.
+pub type PutByteStream<'a> =
+    Pin<Box<dyn Stream<Item = Result<Bytes, StorageError>> + Send + 'a>>;
+
 pub struct StreamingFile {
     pub stream: ByteStream,
     pub size_bytes: u64,
@@ -39,6 +46,34 @@ pub struct StreamingFile {
 #[async_trait]
 pub trait Storage: Send + Sync {
     async fn put(&self, key: &str, bytes: &[u8]) -> Result<(), StorageError>;
+    /// Stream a blob to storage without holding the entire body in
+    /// memory. The chapter-content download path uses this so peak
+    /// memory stays at one chunk per concurrent dispatch (not one full
+    /// page image). The contract is atomic: a stream that errors mid-way
+    /// must leave nothing visible at `key` — implementations should
+    /// write to a temp location and rename only on the successful
+    /// drain. Returns the total bytes written on success.
+    ///
+    /// The default implementation buffers the stream into memory and
+    /// calls `put`, so backends without a native streaming write still
+    /// satisfy the contract (at the cost of peak memory). LocalStorage
+    /// overrides this to do a temp-file rename; a future S3Storage
+    /// would override with a multi-part upload.
+    async fn put_stream(
+        &self,
+        key: &str,
+        mut stream: PutByteStream<'_>,
+    ) -> Result<u64, StorageError> {
+        use futures_util::StreamExt as _;
+        let mut buf: Vec<u8> = Vec::new();
+        while let Some(chunk) = stream.next().await {
+            let chunk = chunk?;
+            buf.extend_from_slice(&chunk);
+        }
+        let len = buf.len() as u64;
+        self.put(key, &buf).await?;
+        Ok(len)
+    }
     /// Reads the entire blob into memory. Convenient for small assets
     /// (covers, thumbnails). For pages and other large blobs, use
     /// `get_stream` so axum can pipe bytes straight to the client.