feat(admin): crawler observability dashboard + reliability hardening (0.55.0)
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.
Backend:
- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
into status / control / dead_jobs / backlog modules. SSE stream
composes in-memory status with DB-derived queue counts, memoizes
the counts for 1s and debounces watch pokes for 250ms (~10x QPS
reduction per subscriber). One-shot GET /admin/crawler shares the
same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
(409 Conflict on overlapping click); browser restart goes through
the coordinated_restart gate (drain + relaunch + auto-clear of the
sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
validation, never logged, audit row carries SHA-256 fingerprint).
Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
scope=all requires confirm:true; DISTINCT ON dedup keeps the
partial unique index from rejecting requeues for chapters with
multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
panics; last-writer-wins on cover so concurrent dispatches don't
clobber each other's slot. Pure functions (should_stop /
should_mark_clean_exit / should_abort_pass) with named regression
tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
per-job timeout, circuit breaker on consecutive failures, BrowserManager
coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
Pages stream through with peak memory ~one HTTP chunk + 64-byte
sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
and crawler_jobs_dead_idx, both ordered by updated_at DESC to
match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
admin_no_store_guard (Cache-Control: no-store on admin
responses), audit rows carry per-scope target_id.
Frontend:
- /admin/crawler page decomposed into lib/components/crawler/
(11 components: ProgressBar, SearchBar, CrawlerHero,
CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
MissingCoversTable, DeadJobsTable, RestartConfirmModal,
RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
pageshow (BFCache); after 5 consecutive errors probes the status
endpoint so a 401 routes through the global on401Hook instead of
infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
AbortControllers; refresh after a control action only runs when
the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
sync_state locally (no full chapter-list refetch); proper
aria-label. Requeue-all gets its own confirm modal; both confirm
modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
for Accept: text/event-stream; pure shouldBypassProxyTimeout
helper covered by unit tests.
Config / docs:
- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
CRAWLER_BROWSER_RESTART_THRESHOLD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,10 +1,12 @@
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use async_trait::async_trait;
|
||||
use futures_util::StreamExt as _;
|
||||
use tokio::fs;
|
||||
use tokio::io::AsyncWriteExt as _;
|
||||
use tokio_util::io::ReaderStream;
|
||||
|
||||
use super::{Storage, StorageError, StreamingFile};
|
||||
use super::{PutByteStream, Storage, StorageError, StreamingFile};
|
||||
|
||||
pub struct LocalStorage {
|
||||
root: PathBuf,
|
||||
@@ -45,6 +47,49 @@ impl Storage for LocalStorage {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn put_stream(
|
||||
&self,
|
||||
key: &str,
|
||||
mut stream: PutByteStream<'_>,
|
||||
) -> Result<u64, StorageError> {
|
||||
let path = self.resolve(key)?;
|
||||
if let Some(parent) = path.parent() {
|
||||
fs::create_dir_all(parent).await?;
|
||||
}
|
||||
// Atomic install via temp + rename. A failure mid-stream
|
||||
// removes the temp so nothing is visible at `key`. The temp
|
||||
// name uses a UUID suffix so concurrent puts of the same key
|
||||
// (e.g. two workers racing a retry) don't clobber each
|
||||
// other's in-progress file before the rename.
|
||||
let tmp = path.with_extension(format!(
|
||||
"{}.tmp.{}",
|
||||
path.extension().and_then(|e| e.to_str()).unwrap_or(""),
|
||||
uuid::Uuid::new_v4().simple()
|
||||
));
|
||||
let mut written: u64 = 0;
|
||||
let result: Result<(), StorageError> = async {
|
||||
let mut f = fs::File::create(&tmp).await?;
|
||||
while let Some(chunk) = stream.next().await {
|
||||
let chunk = chunk?;
|
||||
f.write_all(&chunk).await?;
|
||||
written = written.saturating_add(chunk.len() as u64);
|
||||
}
|
||||
// fsync before rename so a power-loss can't leave a
|
||||
// zero-byte file at the destination.
|
||||
f.sync_all().await?;
|
||||
fs::rename(&tmp, &path).await?;
|
||||
Ok(())
|
||||
}
|
||||
.await;
|
||||
if let Err(e) = result {
|
||||
// Best-effort cleanup; ignore "no such file" if the temp
|
||||
// was never created.
|
||||
let _ = fs::remove_file(&tmp).await;
|
||||
return Err(e);
|
||||
}
|
||||
Ok(written)
|
||||
}
|
||||
|
||||
async fn get(&self, key: &str) -> Result<Vec<u8>, StorageError> {
|
||||
let path = self.resolve(key)?;
|
||||
match fs::read(&path).await {
|
||||
@@ -142,6 +187,52 @@ mod tests {
|
||||
));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn put_stream_writes_full_body_and_removes_temp_on_error() {
|
||||
use bytes::Bytes;
|
||||
use futures_util::stream;
|
||||
|
||||
let dir = tempdir().unwrap();
|
||||
let s = LocalStorage::new(dir.path());
|
||||
|
||||
// Success: a 3-chunk stream writes the concatenation and
|
||||
// returns the right byte count.
|
||||
let chunks: Vec<Result<Bytes, StorageError>> = vec![
|
||||
Ok(Bytes::from_static(b"alpha-")),
|
||||
Ok(Bytes::from_static(b"beta-")),
|
||||
Ok(Bytes::from_static(b"gamma")),
|
||||
];
|
||||
let bytes_written = s
|
||||
.put_stream("streamed/ok.bin", Box::pin(stream::iter(chunks)))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(bytes_written, b"alpha-beta-gamma".len() as u64);
|
||||
assert_eq!(s.get("streamed/ok.bin").await.unwrap(), b"alpha-beta-gamma");
|
||||
|
||||
// Failure mid-stream: nothing is visible at the destination
|
||||
// and no .tmp file is left behind.
|
||||
let chunks: Vec<Result<Bytes, StorageError>> = vec![
|
||||
Ok(Bytes::from_static(b"good")),
|
||||
Err(StorageError::Io(std::io::Error::other("boom"))),
|
||||
];
|
||||
let err = s
|
||||
.put_stream("streamed/bad.bin", Box::pin(stream::iter(chunks)))
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, StorageError::Io(_)));
|
||||
assert!(matches!(
|
||||
s.get("streamed/bad.bin").await,
|
||||
Err(StorageError::NotFound)
|
||||
));
|
||||
// No stray temp file in the streamed/ directory.
|
||||
let entries: Vec<_> = std::fs::read_dir(dir.path().join("streamed"))
|
||||
.unwrap()
|
||||
.filter_map(|e| e.ok())
|
||||
.map(|e| e.file_name().to_string_lossy().to_string())
|
||||
.collect();
|
||||
assert_eq!(entries, vec!["ok.bin"]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn get_stream_emits_multiple_chunks_for_large_files() {
|
||||
use futures_util::StreamExt as _;
|
||||
|
||||
Reference in New Issue
Block a user