feat(analysis): enqueue page-analysis on upload/crawl + admin backfill endpoints

Wires the analyze_page job kind into the page-create paths and adds admin
controls, all gated on a new ANALYSIS_ENABLED flag (AppState.analysis_enabled,
config::AnalysisConfig):

- Chapter upload (api::chapters) enqueues one analyze_page job per page
  after the tx commits (best-effort; failures logged, never fatal).
- Crawler content sync (persist_pages → RETURNING id) enqueues per
  persisted page when the daemon dispatcher's analysis_enabled flag is set;
  threaded through sync_chapter_content (CLI/resync pass false).
- repo::page_analysis::enqueue_all_pages: bulk backfill via INSERT..SELECT,
  skipping done pages (only_unanalyzed) and existing pending jobs.
- New admin endpoints (RequireAdmin, 503 when disabled, audited):
  POST /admin/analysis/reenqueue and POST /admin/pages/:id/analyze (force).

Tests: upload enqueues N jobs / no-op when disabled; crawler persist_pages
enqueue gate; admin reenqueue (backfill, idempotent, only-unanalyzed, 503,
non-admin 403) and force-analyze (force flag, 404).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-13 18:39:03 +02:00
parent 63e1aa5484
commit 2bbf1595ff
15 changed files with 592 additions and 11 deletions

View File

@@ -31,6 +31,34 @@ pub async fn enqueue_for_page(pool: &PgPool, page_id: Uuid, force: bool) -> AppR
Ok(())
}
/// Bulk-enqueue `analyze_page` jobs for existing pages — the admin
/// backfill path for content uploaded/crawled before analysis was
/// enabled. When `only_unanalyzed` is true, pages that already have a
/// `done` analysis row are skipped. Pages with a pending/running
/// `analyze_page` job are always skipped so repeated calls don't pile up
/// duplicates. Returns the number of jobs enqueued.
pub async fn enqueue_all_pages(pool: &PgPool, only_unanalyzed: bool) -> AppResult<u64> {
let result = sqlx::query(
r#"
INSERT INTO crawler_jobs (payload)
SELECT jsonb_build_object('kind', 'analyze_page', 'page_id', p.id, 'force', false)
FROM pages p
WHERE ($1 = false OR NOT EXISTS (
SELECT 1 FROM page_analysis pa
WHERE pa.page_id = p.id AND pa.status = 'done'))
AND NOT EXISTS (
SELECT 1 FROM crawler_jobs j
WHERE j.payload->>'kind' = 'analyze_page'
AND j.payload->>'page_id' = p.id::text
AND j.state IN ('pending', 'running'))
"#,
)
.bind(only_unanalyzed)
.execute(pool)
.await?;
Ok(result.rows_affected())
}
/// Load a page's analysis row, if it has one.
pub async fn load(pool: &PgPool, page_id: Uuid) -> AppResult<Option<PageAnalysis>> {
let row = sqlx::query_as::<_, PageAnalysis>(