feat(admin): crawler observability dashboard + reliability hardening (0.55.0)
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.
Backend:
- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
into status / control / dead_jobs / backlog modules. SSE stream
composes in-memory status with DB-derived queue counts, memoizes
the counts for 1s and debounces watch pokes for 250ms (~10x QPS
reduction per subscriber). One-shot GET /admin/crawler shares the
same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
(409 Conflict on overlapping click); browser restart goes through
the coordinated_restart gate (drain + relaunch + auto-clear of the
sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
validation, never logged, audit row carries SHA-256 fingerprint).
Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
scope=all requires confirm:true; DISTINCT ON dedup keeps the
partial unique index from rejecting requeues for chapters with
multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
panics; last-writer-wins on cover so concurrent dispatches don't
clobber each other's slot. Pure functions (should_stop /
should_mark_clean_exit / should_abort_pass) with named regression
tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
per-job timeout, circuit breaker on consecutive failures, BrowserManager
coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
Pages stream through with peak memory ~one HTTP chunk + 64-byte
sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
and crawler_jobs_dead_idx, both ordered by updated_at DESC to
match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
admin_no_store_guard (Cache-Control: no-store on admin
responses), audit rows carry per-scope target_id.
Frontend:
- /admin/crawler page decomposed into lib/components/crawler/
(11 components: ProgressBar, SearchBar, CrawlerHero,
CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
MissingCoversTable, DeadJobsTable, RestartConfirmModal,
RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
pageshow (BFCache); after 5 consecutive errors probes the status
endpoint so a 401 routes through the global on401Hook instead of
infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
AbortControllers; refresh after a control action only runs when
the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
sync_state locally (no full chapter-list refetch); proper
aria-label. Requeue-all gets its own confirm modal; both confirm
modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
for Accept: text/event-stream; pure shouldBypassProxyTimeout
helper covered by unit tests.
Config / docs:
- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
CRAWLER_BROWSER_RESTART_THRESHOLD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
654
backend/tests/api_admin_crawler.rs
Normal file
654
backend/tests/api_admin_crawler.rs
Normal file
@@ -0,0 +1,654 @@
|
||||
//! Integration tests for the admin crawler observability/control API.
|
||||
//!
|
||||
//! The default test harness wires `AppState.crawler = None` (no daemon),
|
||||
//! so the *control* endpoints return 503 and the *read* endpoints that
|
||||
//! work off the DB (status shell, dead-jobs list/requeue) still function.
|
||||
//! This is exactly the production "daemon disabled" posture.
|
||||
|
||||
mod common;
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use axum::http::StatusCode;
|
||||
use axum::Router;
|
||||
use http_body_util::BodyExt;
|
||||
use serde_json::json;
|
||||
use sqlx::PgPool;
|
||||
use tower::ServiceExt;
|
||||
use uuid::Uuid;
|
||||
|
||||
use common::{
|
||||
body_json, get, get_with_cookie, harness, harness_with_admin_origins,
|
||||
post_json_with_cookie, post_json_with_cookie_origin, register_user,
|
||||
};
|
||||
|
||||
async fn seed_admin(pool: &PgPool, app: &Router) -> String {
|
||||
let (username, cookie) = register_user(app).await;
|
||||
let u = mangalord::repo::user::find_by_username(pool, &username)
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
mangalord::repo::user::set_is_admin_unchecked(pool, u.id, true)
|
||||
.await
|
||||
.unwrap();
|
||||
cookie
|
||||
}
|
||||
|
||||
async fn seed_dead_job(pool: &PgPool, title: &str) -> Uuid {
|
||||
let manga_id = Uuid::new_v4();
|
||||
let chapter_id = Uuid::new_v4();
|
||||
sqlx::query("INSERT INTO mangas (id, title) VALUES ($1, $2)")
|
||||
.bind(manga_id)
|
||||
.bind(title)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
sqlx::query("INSERT INTO chapters (id, manga_id, number) VALUES ($1, $2, 1)")
|
||||
.bind(chapter_id)
|
||||
.bind(manga_id)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
let job_id = Uuid::new_v4();
|
||||
sqlx::query(
|
||||
"INSERT INTO crawler_jobs (id, payload, state, attempts, last_error) \
|
||||
VALUES ($1, $2, 'dead', 5, 'boom')",
|
||||
)
|
||||
.bind(job_id)
|
||||
.bind(json!({
|
||||
"kind": "sync_chapter_content",
|
||||
"source_id": "target",
|
||||
"chapter_id": chapter_id,
|
||||
"source_chapter_key": "k",
|
||||
}))
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
job_id
|
||||
}
|
||||
|
||||
/// Seed a chapter-content job in a given state ('pending'/'running').
|
||||
async fn seed_job(pool: &PgPool, title: &str, state: &str) {
|
||||
let manga_id = Uuid::new_v4();
|
||||
let chapter_id = Uuid::new_v4();
|
||||
sqlx::query("INSERT INTO mangas (id, title) VALUES ($1, $2)")
|
||||
.bind(manga_id)
|
||||
.bind(title)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
sqlx::query("INSERT INTO chapters (id, manga_id, number) VALUES ($1, $2, 1)")
|
||||
.bind(chapter_id)
|
||||
.bind(manga_id)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
sqlx::query("INSERT INTO crawler_jobs (id, payload, state) VALUES ($1, $2, $3)")
|
||||
.bind(Uuid::new_v4())
|
||||
.bind(json!({
|
||||
"kind": "sync_chapter_content",
|
||||
"source_id": "target",
|
||||
"chapter_id": chapter_id,
|
||||
"source_chapter_key": "k",
|
||||
}))
|
||||
.bind(state)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
/// Seed a manga with no cover + a live source row (queued for cover fetch).
|
||||
async fn seed_missing_cover(pool: &PgPool, title: &str) {
|
||||
let manga_id = Uuid::new_v4();
|
||||
sqlx::query("INSERT INTO mangas (id, title, cover_image_path) VALUES ($1, $2, NULL)")
|
||||
.bind(manga_id)
|
||||
.bind(title)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
sqlx::query("INSERT INTO sources (id, name, base_url) VALUES ('target','T','http://x') ON CONFLICT DO NOTHING")
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
sqlx::query(
|
||||
"INSERT INTO manga_sources (source_id, source_manga_key, manga_id, source_url) \
|
||||
VALUES ('target', $1, $2, 'http://x/m')",
|
||||
)
|
||||
.bind(format!("k-{manga_id}"))
|
||||
.bind(manga_id)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn active_jobs_and_covers_lists_over_http(pool: PgPool) {
|
||||
seed_job(&pool, "Naruto", "pending").await;
|
||||
seed_job(&pool, "Bleach", "running").await;
|
||||
seed_missing_cover(&pool, "One Piece").await;
|
||||
let h = harness(pool.clone());
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
|
||||
// Queued/active chapters.
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(get_with_cookie("/api/v1/admin/crawler/active-jobs", &cookie))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
let body = body_json(resp).await;
|
||||
assert_eq!(body["page"]["total"], 2);
|
||||
|
||||
// Queued covers.
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(get_with_cookie("/api/v1/admin/crawler/covers", &cookie))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
let body = body_json(resp).await;
|
||||
assert_eq!(body["page"]["total"], 1);
|
||||
assert_eq!(body["items"][0]["manga_title"], "One Piece");
|
||||
|
||||
// Both are admin-gated.
|
||||
let (_u, plain) = register_user(&h.app).await;
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(get_with_cookie("/api/v1/admin/crawler/active-jobs", &plain))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn get_status_requires_admin(pool: PgPool) {
|
||||
let h = harness(pool);
|
||||
// Unauthenticated → 401.
|
||||
let resp = h.app.clone().oneshot(get("/api/v1/admin/crawler")).await.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
|
||||
|
||||
// Authenticated non-admin → 403.
|
||||
let (_u, cookie) = register_user(&h.app).await;
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(get_with_cookie("/api/v1/admin/crawler", &cookie))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn get_status_reports_disabled_daemon_with_queue_counts(pool: PgPool) {
|
||||
seed_dead_job(&pool, "Naruto").await;
|
||||
let h = harness(pool.clone());
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(get_with_cookie("/api/v1/admin/crawler", &cookie))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
let body = body_json(resp).await;
|
||||
assert_eq!(body["daemon"], "disabled");
|
||||
assert_eq!(body["queue"]["dead"], 1);
|
||||
assert_eq!(body["browser"], "down");
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn control_endpoints_return_503_when_daemon_disabled(pool: PgPool) {
|
||||
let h = harness(pool.clone());
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
for uri in [
|
||||
"/api/v1/admin/crawler/run",
|
||||
"/api/v1/admin/crawler/browser/restart",
|
||||
"/api/v1/admin/crawler/session/clear-expired",
|
||||
] {
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(post_json_with_cookie(uri, json!({}), &cookie))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
resp.status(),
|
||||
StatusCode::SERVICE_UNAVAILABLE,
|
||||
"{uri} should be 503 when daemon disabled"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn status_stream_requires_admin(pool: PgPool) {
|
||||
let h = harness(pool);
|
||||
// Unauthenticated → 401.
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(get("/api/v1/admin/crawler/stream"))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
|
||||
// Non-admin → 403.
|
||||
let (_u, cookie) = register_user(&h.app).await;
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(get_with_cookie("/api/v1/admin/crawler/stream", &cookie))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn status_stream_emits_initial_event(pool: PgPool) {
|
||||
let h = harness(pool.clone());
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(get_with_cookie("/api/v1/admin/crawler/stream", &cookie))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
let ct = resp
|
||||
.headers()
|
||||
.get(axum::http::header::CONTENT_TYPE)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.unwrap_or_default()
|
||||
.to_string();
|
||||
assert!(ct.starts_with("text/event-stream"), "content-type was {ct:?}");
|
||||
|
||||
// Accumulate frames (the immediate snapshot may arrive split across
|
||||
// frames) until the status payload appears, with an overall timeout so
|
||||
// the never-ending stream can't hang the test.
|
||||
let mut body = resp.into_body();
|
||||
let mut acc = String::new();
|
||||
let deadline = tokio::time::timeout(Duration::from_secs(5), async {
|
||||
loop {
|
||||
let Some(frame) = body.frame().await else { break };
|
||||
if let Ok(data) = frame.expect("frame ok").into_data() {
|
||||
acc.push_str(&String::from_utf8_lossy(&data));
|
||||
if acc.contains("\"daemon\"") {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
.await;
|
||||
assert!(deadline.is_ok(), "did not receive status within 5s; got: {acc:?}");
|
||||
assert!(acc.contains("\"daemon\""), "missing status payload: {acc}");
|
||||
assert!(acc.contains("status"), "missing SSE event name: {acc}");
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn mutating_endpoints_reject_non_admin(pool: PgPool) {
|
||||
let h = harness(pool);
|
||||
// A logged-in non-admin must be forbidden from a mutating endpoint.
|
||||
let (_u, cookie) = register_user(&h.app).await;
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(post_json_with_cookie(
|
||||
"/api/v1/admin/crawler/dead-jobs/requeue",
|
||||
json!({ "scope": "all" }),
|
||||
&cookie,
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// CSRF Origin/Referer allowlist (T2)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn csrf_rejects_mutation_with_cross_origin_header(pool: PgPool) {
|
||||
let h = harness_with_admin_origins(
|
||||
pool.clone(),
|
||||
vec!["http://localhost:3000".to_string()],
|
||||
);
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(post_json_with_cookie_origin(
|
||||
"/api/v1/admin/crawler/session/clear-expired",
|
||||
json!({}),
|
||||
&cookie,
|
||||
Some("https://evil.example.com"),
|
||||
None,
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn csrf_allows_mutation_with_allowed_origin(pool: PgPool) {
|
||||
let h = harness_with_admin_origins(
|
||||
pool.clone(),
|
||||
vec!["http://localhost:3000".to_string()],
|
||||
);
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(post_json_with_cookie_origin(
|
||||
"/api/v1/admin/crawler/session/clear-expired",
|
||||
json!({}),
|
||||
&cookie,
|
||||
Some("http://localhost:3000"),
|
||||
None,
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
// Daemon disabled in the harness → 503 from the handler; the CSRF
|
||||
// gate must have let the request through before that response.
|
||||
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn csrf_allows_mutation_without_origin_or_referer(pool: PgPool) {
|
||||
// curl/server-to-server callers send neither — they can't be a CSRF
|
||||
// vector since there's no third-party browser context.
|
||||
let h = harness_with_admin_origins(
|
||||
pool.clone(),
|
||||
vec!["http://localhost:3000".to_string()],
|
||||
);
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(post_json_with_cookie(
|
||||
"/api/v1/admin/crawler/session/clear-expired",
|
||||
json!({}),
|
||||
&cookie,
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn csrf_falls_back_to_referer_when_origin_missing(pool: PgPool) {
|
||||
let h = harness_with_admin_origins(
|
||||
pool.clone(),
|
||||
vec!["http://localhost:3000".to_string()],
|
||||
);
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(post_json_with_cookie_origin(
|
||||
"/api/v1/admin/crawler/session/clear-expired",
|
||||
json!({}),
|
||||
&cookie,
|
||||
None,
|
||||
Some("http://localhost:3000/admin/crawler"),
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn csrf_skipped_on_safe_methods(pool: PgPool) {
|
||||
let h = harness_with_admin_origins(
|
||||
pool.clone(),
|
||||
vec!["http://localhost:3000".to_string()],
|
||||
);
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
// GET from a hostile origin is fine — browsers can't mutate via GET.
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(common::get_with_cookie_origin(
|
||||
"/api/v1/admin/crawler",
|
||||
&cookie,
|
||||
Some("https://evil.example.com"),
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn csrf_disabled_when_allowlist_empty(pool: PgPool) {
|
||||
// Default harness has admin_allowed_origins = empty → operator
|
||||
// opt-out documented in .env.example. A cross-origin POST passes
|
||||
// through to the handler.
|
||||
let h = harness(pool.clone());
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(post_json_with_cookie_origin(
|
||||
"/api/v1/admin/crawler/session/clear-expired",
|
||||
json!({}),
|
||||
&cookie,
|
||||
Some("https://evil.example.com"),
|
||||
None,
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Cache-Control: no-store on admin responses (S3)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn admin_responses_have_no_store(pool: PgPool) {
|
||||
let h = harness(pool.clone());
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(get_with_cookie("/api/v1/admin/crawler", &cookie))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
let cc = resp
|
||||
.headers()
|
||||
.get(axum::http::header::CACHE_CONTROL)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.unwrap_or_default()
|
||||
.to_string();
|
||||
assert!(
|
||||
cc.contains("no-store"),
|
||||
"admin response missing Cache-Control: no-store (got {cc:?})"
|
||||
);
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn non_admin_responses_unaffected_by_no_store(pool: PgPool) {
|
||||
let h = harness(pool);
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(get("/api/v1/health"))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
let cc = resp
|
||||
.headers()
|
||||
.get(axum::http::header::CACHE_CONTROL)
|
||||
.and_then(|v| v.to_str().ok());
|
||||
assert!(
|
||||
cc.is_none() || !cc.unwrap_or_default().contains("no-store"),
|
||||
"non-admin response should not get no-store (got {cc:?})"
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// scope=all confirm:true guard (S1)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn requeue_all_without_confirm_rejected(pool: PgPool) {
|
||||
seed_dead_job(&pool, "X").await;
|
||||
let h = harness(pool.clone());
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(post_json_with_cookie(
|
||||
"/api/v1/admin/crawler/dead-jobs/requeue",
|
||||
json!({ "scope": "all" }),
|
||||
&cookie,
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::BAD_REQUEST);
|
||||
// The dead row must NOT have been touched.
|
||||
let state: String = sqlx::query_scalar("SELECT state FROM crawler_jobs LIMIT 1")
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(state, "dead");
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn requeue_all_with_confirm_flips_dead_pile(pool: PgPool) {
|
||||
seed_dead_job(&pool, "X").await;
|
||||
seed_dead_job(&pool, "Y").await;
|
||||
let h = harness(pool.clone());
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(post_json_with_cookie(
|
||||
"/api/v1/admin/crawler/dead-jobs/requeue",
|
||||
json!({ "scope": "all", "confirm": true }),
|
||||
&cookie,
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
let pending: i64 = sqlx::query_scalar(
|
||||
"SELECT COUNT(*) FROM crawler_jobs WHERE state = 'pending'",
|
||||
)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(pending, 2);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// admin_audit target_id + PHPSESSID fingerprint (S2)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn requeue_chapter_audit_records_chapter_target_id(pool: PgPool) {
|
||||
let job_id = seed_dead_job(&pool, "Bleach").await;
|
||||
let chapter_id: Uuid = sqlx::query_scalar(
|
||||
"SELECT (payload->>'chapter_id')::uuid FROM crawler_jobs WHERE id = $1",
|
||||
)
|
||||
.bind(job_id)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
let h = harness(pool.clone());
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(post_json_with_cookie(
|
||||
"/api/v1/admin/crawler/dead-jobs/requeue",
|
||||
json!({ "scope": "chapter", "chapter_id": chapter_id }),
|
||||
&cookie,
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
let (target_kind, target_id): (String, Option<Uuid>) = sqlx::query_as(
|
||||
"SELECT target_kind, target_id FROM admin_audit \
|
||||
WHERE action = 'crawler_dead_jobs_requeue' \
|
||||
ORDER BY created_at DESC LIMIT 1",
|
||||
)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(target_kind, "chapter");
|
||||
assert_eq!(target_id, Some(chapter_id));
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn requeue_all_audit_omits_target_id_but_logs_count(pool: PgPool) {
|
||||
seed_dead_job(&pool, "X").await;
|
||||
let h = harness(pool.clone());
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
let _ = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(post_json_with_cookie(
|
||||
"/api/v1/admin/crawler/dead-jobs/requeue",
|
||||
json!({ "scope": "all", "confirm": true }),
|
||||
&cookie,
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
let (target_kind, target_id, payload): (String, Option<Uuid>, serde_json::Value) =
|
||||
sqlx::query_as(
|
||||
"SELECT target_kind, target_id, payload FROM admin_audit \
|
||||
WHERE action = 'crawler_dead_jobs_requeue' \
|
||||
ORDER BY created_at DESC LIMIT 1",
|
||||
)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(target_kind, "crawler");
|
||||
assert_eq!(target_id, None);
|
||||
assert_eq!(payload["scope"], "all");
|
||||
assert_eq!(payload["requeued"], 1);
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn dead_jobs_list_and_requeue_over_http(pool: PgPool) {
|
||||
let job_id = seed_dead_job(&pool, "Bleach").await;
|
||||
let h = harness(pool.clone());
|
||||
let cookie = seed_admin(&pool, &h.app).await;
|
||||
|
||||
// List.
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(get_with_cookie("/api/v1/admin/crawler/dead-jobs", &cookie))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
let body = body_json(resp).await;
|
||||
assert_eq!(body["page"]["total"], 1);
|
||||
assert_eq!(body["items"][0]["manga_title"], "Bleach");
|
||||
|
||||
// Requeue the single job.
|
||||
let resp = h
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(post_json_with_cookie(
|
||||
"/api/v1/admin/crawler/dead-jobs/requeue",
|
||||
json!({ "scope": "job", "job_id": job_id }),
|
||||
&cookie,
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
let body = body_json(resp).await;
|
||||
assert_eq!(body["requeued"], 1);
|
||||
|
||||
let state: String = sqlx::query_scalar("SELECT state FROM crawler_jobs WHERE id = $1")
|
||||
.bind(job_id)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(state, "pending");
|
||||
}
|
||||
@@ -50,6 +50,8 @@ fn admin_test_router(pool: PgPool) -> (Router, TempDir) {
|
||||
upload: UploadConfig::default(),
|
||||
auth_limiter,
|
||||
resync: None,
|
||||
crawler: None,
|
||||
admin_allowed_origins: Arc::new(Vec::new()),
|
||||
};
|
||||
let app = Router::new()
|
||||
.nest("/api/v1", api::routes())
|
||||
|
||||
@@ -17,7 +17,7 @@ use tower::ServiceExt;
|
||||
use mangalord::app::{router, AppState};
|
||||
use mangalord::auth::rate_limit::AuthRateLimiter;
|
||||
use mangalord::config::{AuthConfig, UploadConfig};
|
||||
use mangalord::storage::{LocalStorage, Storage, StorageError, StreamingFile};
|
||||
use mangalord::storage::{LocalStorage, PutByteStream, Storage, StorageError, StreamingFile};
|
||||
|
||||
use async_trait::async_trait;
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
@@ -78,6 +78,10 @@ fn harness_with_auth_config(
|
||||
// handlers return 503 in this config. Tests that need a stub
|
||||
// resync service swap it in via `harness_with_resync`.
|
||||
resync: None,
|
||||
crawler: None,
|
||||
// Empty allowlist = CSRF check skipped. The CSRF-specific test
|
||||
// harness `harness_with_admin_origins` overrides this.
|
||||
admin_allowed_origins: Arc::new(Vec::new()),
|
||||
};
|
||||
Harness { app: router(state), _storage_dir: storage_dir }
|
||||
}
|
||||
@@ -152,6 +156,37 @@ pub fn harness_with_resync(
|
||||
},
|
||||
auth_limiter,
|
||||
resync: Some(resync),
|
||||
crawler: None,
|
||||
admin_allowed_origins: Arc::new(Vec::new()),
|
||||
};
|
||||
Harness {
|
||||
app: router(state),
|
||||
_storage_dir: storage_dir,
|
||||
}
|
||||
}
|
||||
|
||||
/// Like [`harness`] but configures an admin CSRF allowlist so the
|
||||
/// `/admin/*` mutating endpoints reject cross-origin browser POSTs.
|
||||
/// Used by the admin CSRF integration tests.
|
||||
pub fn harness_with_admin_origins(pool: PgPool, origins: Vec<String>) -> Harness {
|
||||
let storage_dir = tempfile::tempdir().expect("tempdir");
|
||||
let storage = Arc::new(LocalStorage::new(storage_dir.path()));
|
||||
let auth_limiter = Arc::new(AuthRateLimiter::new(Default::default()));
|
||||
let state = AppState {
|
||||
db: pool,
|
||||
storage,
|
||||
auth: AuthConfig {
|
||||
cookie_secure: false,
|
||||
..AuthConfig::default()
|
||||
},
|
||||
upload: UploadConfig {
|
||||
max_request_bytes: 4 * 1024 * 1024,
|
||||
max_file_bytes: 256 * 1024,
|
||||
},
|
||||
auth_limiter,
|
||||
resync: None,
|
||||
crawler: None,
|
||||
admin_allowed_origins: Arc::new(origins),
|
||||
};
|
||||
Harness {
|
||||
app: router(state),
|
||||
@@ -189,6 +224,22 @@ impl Storage for FailingStorage {
|
||||
}
|
||||
self.inner.put(key, bytes).await
|
||||
}
|
||||
async fn put_stream(
|
||||
&self,
|
||||
key: &str,
|
||||
stream: PutByteStream<'_>,
|
||||
) -> Result<u64, StorageError> {
|
||||
// Count put_stream towards the same fail-index so tests that
|
||||
// expect "the Nth put fails" don't care which entry point
|
||||
// the caller took.
|
||||
let n = self.counter.fetch_add(1, Ordering::SeqCst);
|
||||
if n == self.fail_on_put_index {
|
||||
return Err(StorageError::Io(std::io::Error::other(
|
||||
"FailingStorage: injected put_stream failure",
|
||||
)));
|
||||
}
|
||||
self.inner.put_stream(key, stream).await
|
||||
}
|
||||
async fn get(&self, key: &str) -> Result<Vec<u8>, StorageError> {
|
||||
self.inner.get(key).await
|
||||
}
|
||||
@@ -251,6 +302,44 @@ pub fn post_json_with_cookie(
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
/// Same as [`post_json_with_cookie`] but also attaches `Origin` (and
|
||||
/// optionally `Referer`) headers. Used by the admin CSRF tests to drive
|
||||
/// the cross-origin reject + allowed-origin accept paths.
|
||||
pub fn post_json_with_cookie_origin(
|
||||
uri: &str,
|
||||
body: serde_json::Value,
|
||||
cookie: &str,
|
||||
origin: Option<&str>,
|
||||
referer: Option<&str>,
|
||||
) -> Request<Body> {
|
||||
let mut b = Request::builder()
|
||||
.method("POST")
|
||||
.uri(uri)
|
||||
.header(header::CONTENT_TYPE, "application/json")
|
||||
.header(header::COOKIE, cookie);
|
||||
if let Some(o) = origin {
|
||||
b = b.header(header::ORIGIN, o);
|
||||
}
|
||||
if let Some(r) = referer {
|
||||
b = b.header(header::REFERER, r);
|
||||
}
|
||||
b.body(Body::from(body.to_string())).unwrap()
|
||||
}
|
||||
|
||||
pub fn get_with_cookie_origin(
|
||||
uri: &str,
|
||||
cookie: &str,
|
||||
origin: Option<&str>,
|
||||
) -> Request<Body> {
|
||||
let mut b = Request::builder()
|
||||
.uri(uri)
|
||||
.header(header::COOKIE, cookie);
|
||||
if let Some(o) = origin {
|
||||
b = b.header(header::ORIGIN, o);
|
||||
}
|
||||
b.body(Body::empty()).unwrap()
|
||||
}
|
||||
|
||||
pub fn post_json_with_bearer(
|
||||
uri: &str,
|
||||
body: serde_json::Value,
|
||||
|
||||
@@ -40,6 +40,8 @@ fn make_cfg(
|
||||
tz: Tz::UTC,
|
||||
retention_days: 7,
|
||||
session_expired,
|
||||
status: mangalord::crawler::status::StatusHandle::new(workers),
|
||||
job_timeout: Duration::from_secs(60),
|
||||
extra_tasks: Vec::new(),
|
||||
}
|
||||
}
|
||||
@@ -88,6 +90,52 @@ impl ChapterDispatcher for PanickingDispatcher {
|
||||
}
|
||||
}
|
||||
|
||||
/// Never completes — used to verify the worker's outer dispatch timeout.
|
||||
struct HangingDispatcher {
|
||||
seen: AtomicUsize,
|
||||
}
|
||||
#[async_trait::async_trait]
|
||||
impl ChapterDispatcher for HangingDispatcher {
|
||||
async fn dispatch(&self, _payload: JobPayload) -> anyhow::Result<SyncOutcome> {
|
||||
self.seen.fetch_add(1, Ordering::AcqRel);
|
||||
std::future::pending::<()>().await;
|
||||
unreachable!("hanging dispatcher never resolves");
|
||||
}
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn worker_times_out_a_hung_dispatch_and_acks_failed(pool: PgPool) {
|
||||
enqueue_chapter_job(&pool).await;
|
||||
let dispatcher = Arc::new(HangingDispatcher {
|
||||
seen: AtomicUsize::new(0),
|
||||
});
|
||||
let session_expired = Arc::new(std::sync::atomic::AtomicBool::new(false));
|
||||
let cancel = CancellationToken::new();
|
||||
let mut cfg = make_cfg(None, dispatcher.clone(), session_expired, 1);
|
||||
cfg.job_timeout = Duration::from_millis(300);
|
||||
let handle = daemon::spawn(pool.clone(), cancel.clone(), cfg);
|
||||
|
||||
// The hung job should time out and return to pending with backoff
|
||||
// (attempts=1 < max=5). Poll for the recorded error.
|
||||
let mut timed_out = false;
|
||||
for _ in 0..40 {
|
||||
let n: i64 = sqlx::query_scalar(
|
||||
"SELECT COUNT(*) FROM crawler_jobs WHERE last_error = 'dispatch timed out'",
|
||||
)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
if n == 1 {
|
||||
timed_out = true;
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
}
|
||||
handle.shutdown().await;
|
||||
assert!(timed_out, "hung dispatch must be acked failed with a timeout error");
|
||||
assert!(dispatcher.seen.load(Ordering::Acquire) >= 1);
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn workers_drain_jobs_through_dispatcher(pool: PgPool) {
|
||||
enqueue_chapter_job(&pool).await;
|
||||
|
||||
304
backend/tests/crawler_dead_jobs.rs
Normal file
304
backend/tests/crawler_dead_jobs.rs
Normal file
@@ -0,0 +1,304 @@
|
||||
//! Integration tests for the dead-letter admin queries in
|
||||
//! `repo::crawler`: listing dead jobs with manga/chapter context and the
|
||||
//! scoped requeue (all / per-manga / single) used by the admin dashboard.
|
||||
|
||||
use mangalord::repo::crawler::{self, RequeueScope};
|
||||
use serde_json::json;
|
||||
use sqlx::PgPool;
|
||||
use uuid::Uuid;
|
||||
|
||||
/// Seed a manga with no cover + a live source row (so it's "queued for a
|
||||
/// cover fetch"). Returns the manga id.
|
||||
async fn seed_missing_cover(pool: &PgPool, title: &str) -> Uuid {
|
||||
let manga_id = Uuid::new_v4();
|
||||
sqlx::query("INSERT INTO mangas (id, title, cover_image_path) VALUES ($1, $2, NULL)")
|
||||
.bind(manga_id)
|
||||
.bind(title)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
sqlx::query("INSERT INTO sources (id, name, base_url) VALUES ('target', 'T', 'http://x') ON CONFLICT DO NOTHING")
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
sqlx::query(
|
||||
"INSERT INTO manga_sources (source_id, source_manga_key, manga_id, source_url) \
|
||||
VALUES ('target', $1, $2, 'http://x/m')",
|
||||
)
|
||||
.bind(format!("k-{manga_id}"))
|
||||
.bind(manga_id)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
manga_id
|
||||
}
|
||||
|
||||
/// Seed a manga + chapter and return their ids.
|
||||
async fn seed_chapter(pool: &PgPool, title: &str, number: i32) -> (Uuid, Uuid) {
|
||||
let manga_id = Uuid::new_v4();
|
||||
let chapter_id = Uuid::new_v4();
|
||||
sqlx::query("INSERT INTO mangas (id, title) VALUES ($1, $2)")
|
||||
.bind(manga_id)
|
||||
.bind(title)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
sqlx::query("INSERT INTO chapters (id, manga_id, number) VALUES ($1, $2, $3)")
|
||||
.bind(chapter_id)
|
||||
.bind(manga_id)
|
||||
.bind(number)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
(manga_id, chapter_id)
|
||||
}
|
||||
|
||||
/// Insert a crawler_jobs row in a given state for a chapter-content job.
|
||||
async fn insert_job(pool: &PgPool, chapter_id: Uuid, state: &str, attempts: i32) -> Uuid {
|
||||
let id = Uuid::new_v4();
|
||||
let payload = json!({
|
||||
"kind": "sync_chapter_content",
|
||||
"source_id": "target",
|
||||
"chapter_id": chapter_id,
|
||||
"source_chapter_key": "k",
|
||||
});
|
||||
sqlx::query(
|
||||
"INSERT INTO crawler_jobs (id, payload, state, attempts, last_error) \
|
||||
VALUES ($1, $2, $3, $4, 'boom')",
|
||||
)
|
||||
.bind(id)
|
||||
.bind(payload)
|
||||
.bind(state)
|
||||
.bind(attempts)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
id
|
||||
}
|
||||
|
||||
async fn state_of(pool: &PgPool, id: Uuid) -> String {
|
||||
sqlx::query_scalar::<_, String>("SELECT state FROM crawler_jobs WHERE id = $1")
|
||||
.bind(id)
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn list_dead_jobs_returns_context_and_total(pool: PgPool) {
|
||||
let (_m, c1) = seed_chapter(&pool, "Naruto", 700).await;
|
||||
insert_job(&pool, c1, "dead", 5).await;
|
||||
// A non-dead job must not appear.
|
||||
let (_m2, c2) = seed_chapter(&pool, "Bleach", 1).await;
|
||||
insert_job(&pool, c2, "pending", 0).await;
|
||||
|
||||
let (items, total) = crawler::list_dead_jobs(&pool, None, 50, 0).await.unwrap();
|
||||
assert_eq!(total, 1);
|
||||
assert_eq!(items.len(), 1);
|
||||
let row = &items[0];
|
||||
assert_eq!(row.manga_title.as_deref(), Some("Naruto"));
|
||||
assert_eq!(row.chapter_number, Some(700));
|
||||
assert_eq!(row.attempts, 5);
|
||||
assert_eq!(row.last_error.as_deref(), Some("boom"));
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn list_dead_jobs_filters_by_title_search(pool: PgPool) {
|
||||
let (_m, c1) = seed_chapter(&pool, "Naruto", 700).await;
|
||||
insert_job(&pool, c1, "dead", 5).await;
|
||||
let (_m2, c2) = seed_chapter(&pool, "One Piece", 1).await;
|
||||
insert_job(&pool, c2, "dead", 5).await;
|
||||
|
||||
let (items, total) = crawler::list_dead_jobs(&pool, Some("piece"), 50, 0)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(total, 1);
|
||||
assert_eq!(items[0].manga_title.as_deref(), Some("One Piece"));
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn requeue_all_resets_dead_jobs_to_pending(pool: PgPool) {
|
||||
let (_m, c1) = seed_chapter(&pool, "A", 1).await;
|
||||
let (_m2, c2) = seed_chapter(&pool, "B", 1).await;
|
||||
let j1 = insert_job(&pool, c1, "dead", 5).await;
|
||||
let j2 = insert_job(&pool, c2, "dead", 5).await;
|
||||
|
||||
let n = crawler::requeue_dead_jobs(&pool, RequeueScope::All)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(n, 2);
|
||||
assert_eq!(state_of(&pool, j1).await, "pending");
|
||||
assert_eq!(state_of(&pool, j2).await, "pending");
|
||||
let attempts: i32 = sqlx::query_scalar("SELECT attempts FROM crawler_jobs WHERE id = $1")
|
||||
.bind(j1)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(attempts, 0, "attempts reset on requeue");
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn requeue_by_manga_scopes_to_that_manga(pool: PgPool) {
|
||||
let (m1, c1) = seed_chapter(&pool, "A", 1).await;
|
||||
let (_m2, c2) = seed_chapter(&pool, "B", 1).await;
|
||||
let j1 = insert_job(&pool, c1, "dead", 5).await;
|
||||
let j2 = insert_job(&pool, c2, "dead", 5).await;
|
||||
|
||||
let n = crawler::requeue_dead_jobs(&pool, RequeueScope::Manga(m1))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(n, 1);
|
||||
assert_eq!(state_of(&pool, j1).await, "pending");
|
||||
assert_eq!(state_of(&pool, j2).await, "dead", "other manga untouched");
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn requeue_by_chapter_scopes_to_that_chapter(pool: PgPool) {
|
||||
let (_m, c1) = seed_chapter(&pool, "A", 1).await;
|
||||
let (_m2, c2) = seed_chapter(&pool, "A", 2).await;
|
||||
let j1 = insert_job(&pool, c1, "dead", 5).await;
|
||||
let j2 = insert_job(&pool, c2, "dead", 5).await;
|
||||
|
||||
let n = crawler::requeue_dead_jobs(&pool, RequeueScope::Chapter(c1))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(n, 1);
|
||||
assert_eq!(state_of(&pool, j1).await, "pending");
|
||||
assert_eq!(state_of(&pool, j2).await, "dead", "other chapter untouched");
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn requeue_single_job(pool: PgPool) {
|
||||
let (_m, c1) = seed_chapter(&pool, "A", 1).await;
|
||||
let (_m2, c2) = seed_chapter(&pool, "B", 1).await;
|
||||
let j1 = insert_job(&pool, c1, "dead", 5).await;
|
||||
let j2 = insert_job(&pool, c2, "dead", 5).await;
|
||||
|
||||
let n = crawler::requeue_dead_jobs(&pool, RequeueScope::Job(j1))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(n, 1);
|
||||
assert_eq!(state_of(&pool, j1).await, "pending");
|
||||
assert_eq!(state_of(&pool, j2).await, "dead");
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn requeue_skips_dead_when_live_job_exists_for_same_chapter(pool: PgPool) {
|
||||
let (_m, c1) = seed_chapter(&pool, "A", 1).await;
|
||||
let dead = insert_job(&pool, c1, "dead", 5).await;
|
||||
// A live pending job for the SAME chapter already exists.
|
||||
insert_job(&pool, c1, "pending", 0).await;
|
||||
|
||||
let n = crawler::requeue_dead_jobs(&pool, RequeueScope::All)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(n, 0, "must not resurrect a dead job that has a live counterpart");
|
||||
assert_eq!(state_of(&pool, dead).await, "dead");
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn requeue_with_two_dead_jobs_for_one_chapter_revives_one_not_500(pool: PgPool) {
|
||||
// Regression: two dead jobs for the SAME chapter must not both flip to
|
||||
// pending in one statement — that would violate the partial unique
|
||||
// dedup index and abort the whole requeue.
|
||||
let (manga_id, c1) = seed_chapter(&pool, "A", 1).await;
|
||||
let older = insert_job(&pool, c1, "dead", 5).await;
|
||||
let newer = insert_job(&pool, c1, "dead", 5).await;
|
||||
// Make `newer` unambiguously newer.
|
||||
sqlx::query("UPDATE crawler_jobs SET updated_at = now() - interval '1 hour' WHERE id = $1")
|
||||
.bind(older)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
for scope in [RequeueScope::All, RequeueScope::Manga(manga_id), RequeueScope::Chapter(c1)] {
|
||||
// Reset to two-dead before each scope variant.
|
||||
sqlx::query("UPDATE crawler_jobs SET state = 'dead' WHERE id = ANY($1)")
|
||||
.bind(vec![older, newer])
|
||||
.execute(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
let n = crawler::requeue_dead_jobs(&pool, scope)
|
||||
.await
|
||||
.expect("requeue must not error on duplicate dead jobs");
|
||||
assert_eq!(n, 1, "exactly one dead job per chapter is revived");
|
||||
// The newest one is the survivor; the other stays dead.
|
||||
assert_eq!(state_of(&pool, newer).await, "pending");
|
||||
assert_eq!(state_of(&pool, older).await, "dead");
|
||||
}
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn list_active_jobs_returns_pending_and_running_running_first(pool: PgPool) {
|
||||
let (_m, c1) = seed_chapter(&pool, "Naruto", 700).await;
|
||||
let (_m2, c2) = seed_chapter(&pool, "Bleach", 10).await;
|
||||
insert_job(&pool, c1, "pending", 0).await;
|
||||
insert_job(&pool, c2, "running", 1).await;
|
||||
// A dead + a done job must NOT appear.
|
||||
let (_m3, c3) = seed_chapter(&pool, "Gone", 1).await;
|
||||
insert_job(&pool, c3, "dead", 5).await;
|
||||
|
||||
let (items, total) = crawler::list_active_jobs(&pool, None, 50, 0).await.unwrap();
|
||||
assert_eq!(total, 2);
|
||||
assert_eq!(items.len(), 2);
|
||||
// Running first.
|
||||
assert_eq!(items[0].state, "running");
|
||||
assert_eq!(items[0].manga_title.as_deref(), Some("Bleach"));
|
||||
assert_eq!(items[1].state, "pending");
|
||||
assert_eq!(items[1].chapter_number, Some(700));
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn list_active_jobs_filters_by_title(pool: PgPool) {
|
||||
let (_m, c1) = seed_chapter(&pool, "Naruto", 1).await;
|
||||
let (_m2, c2) = seed_chapter(&pool, "One Piece", 1).await;
|
||||
insert_job(&pool, c1, "pending", 0).await;
|
||||
insert_job(&pool, c2, "pending", 0).await;
|
||||
let (items, total) = crawler::list_active_jobs(&pool, Some("piece"), 50, 0)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(total, 1);
|
||||
assert_eq!(items[0].manga_title.as_deref(), Some("One Piece"));
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn missing_covers_count_and_list(pool: PgPool) {
|
||||
seed_missing_cover(&pool, "Naruto").await;
|
||||
seed_missing_cover(&pool, "Bleach").await;
|
||||
// A manga WITH a cover must not be counted.
|
||||
let with_cover = Uuid::new_v4();
|
||||
sqlx::query("INSERT INTO mangas (id, title, cover_image_path) VALUES ($1, 'Done', 'k.jpg')")
|
||||
.bind(with_cover)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(crawler::count_missing_covers(&pool).await.unwrap(), 2);
|
||||
|
||||
let (items, total) = crawler::list_missing_cover_mangas(&pool, None, 50, 0)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(total, 2);
|
||||
assert_eq!(items.len(), 2);
|
||||
|
||||
let (items, total) = crawler::list_missing_cover_mangas(&pool, Some("naru"), 50, 0)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(total, 1);
|
||||
assert_eq!(items[0].manga_title, "Naruto");
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn job_state_counts_groups_by_state(pool: PgPool) {
|
||||
let (_m, c1) = seed_chapter(&pool, "A", 1).await;
|
||||
let (_m2, c2) = seed_chapter(&pool, "B", 1).await;
|
||||
let (_m3, c3) = seed_chapter(&pool, "C", 1).await;
|
||||
insert_job(&pool, c1, "pending", 0).await;
|
||||
insert_job(&pool, c2, "dead", 5).await;
|
||||
insert_job(&pool, c3, "dead", 5).await;
|
||||
|
||||
let (pending, running, dead) = crawler::job_state_counts(&pool).await.unwrap();
|
||||
assert_eq!(pending, 1);
|
||||
assert_eq!(running, 0);
|
||||
assert_eq!(dead, 2);
|
||||
}
|
||||
@@ -185,6 +185,68 @@ async fn lease_marks_running_and_bumps_attempts_and_sets_leased_until(pool: PgPo
|
||||
assert!(leased_until > chrono::Utc::now());
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn renew_extends_leased_until_while_running(pool: PgPool) {
|
||||
let id = match jobs::enqueue(&pool, &chapter_content_payload(Uuid::new_v4()))
|
||||
.await
|
||||
.unwrap()
|
||||
{
|
||||
EnqueueResult::Inserted(id) => id,
|
||||
EnqueueResult::Skipped => unreachable!(),
|
||||
};
|
||||
|
||||
// Lease with a short window, then collapse leased_until to the recent
|
||||
// past so the renew is unambiguously an extension.
|
||||
let leases = jobs::lease(&pool, None, 1, Duration::from_secs(5))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(leases.len(), 1);
|
||||
sqlx::query("UPDATE crawler_jobs SET leased_until = now() - interval '1 second' WHERE id = $1")
|
||||
.bind(id)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let still_owned = jobs::renew(&pool, id, Duration::from_secs(120))
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(still_owned, "renew on a running job returns true");
|
||||
|
||||
let leased_until: chrono::DateTime<chrono::Utc> =
|
||||
sqlx::query_scalar("SELECT leased_until FROM crawler_jobs WHERE id = $1")
|
||||
.bind(id)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(
|
||||
leased_until > chrono::Utc::now() + chrono::Duration::seconds(60),
|
||||
"leased_until pushed ~120s into the future"
|
||||
);
|
||||
assert_eq!(job_state(&pool, id).await, "running");
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn renew_is_noop_once_job_no_longer_running(pool: PgPool) {
|
||||
let id = match jobs::enqueue(&pool, &chapter_content_payload(Uuid::new_v4()))
|
||||
.await
|
||||
.unwrap()
|
||||
{
|
||||
EnqueueResult::Inserted(id) => id,
|
||||
EnqueueResult::Skipped => unreachable!(),
|
||||
};
|
||||
let leases = jobs::lease(&pool, None, 1, Duration::from_secs(60))
|
||||
.await
|
||||
.unwrap();
|
||||
// Job completes — heartbeat should now see it's no longer ours.
|
||||
jobs::ack_done(&pool, leases[0].id).await.unwrap();
|
||||
|
||||
let still_owned = jobs::renew(&pool, id, Duration::from_secs(120))
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(!still_owned, "renew on a non-running job returns false");
|
||||
assert_eq!(job_state(&pool, id).await, "done");
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn lease_with_kind_filter_only_matches_that_kind(pool: PgPool) {
|
||||
let manga_id = match jobs::enqueue(&pool, &sync_manga_payload("foo"))
|
||||
|
||||
@@ -109,7 +109,7 @@ async fn dispatch_target_prefers_most_recent_live_source(pool: PgPool) {
|
||||
seed_chapter_with_two_live_sources(&pool).await;
|
||||
|
||||
let row = dispatch_target(&pool, chapter_id).await.unwrap();
|
||||
let (_manga_id, source_url) =
|
||||
let (_manga_id, source_url, _title, _number) =
|
||||
row.expect("two live sources should yield a dispatch target");
|
||||
assert_eq!(
|
||||
source_url, new_url,
|
||||
@@ -133,7 +133,7 @@ async fn dispatch_target_skips_dropped_sources(pool: PgPool) {
|
||||
.unwrap();
|
||||
|
||||
let row = dispatch_target(&pool, chapter_id).await.unwrap();
|
||||
let (_manga_id, source_url) =
|
||||
let (_manga_id, source_url, _title, _number) =
|
||||
row.expect("a single live source should still yield a dispatch target");
|
||||
assert!(
|
||||
source_url != new_url,
|
||||
|
||||
Reference in New Issue
Block a user