feat(analysis): enqueue page-analysis on upload/crawl + admin backfill endpoints
Wires the analyze_page job kind into the page-create paths and adds admin controls, all gated on a new ANALYSIS_ENABLED flag (AppState.analysis_enabled, config::AnalysisConfig): - Chapter upload (api::chapters) enqueues one analyze_page job per page after the tx commits (best-effort; failures logged, never fatal). - Crawler content sync (persist_pages → RETURNING id) enqueues per persisted page when the daemon dispatcher's analysis_enabled flag is set; threaded through sync_chapter_content (CLI/resync pass false). - repo::page_analysis::enqueue_all_pages: bulk backfill via INSERT..SELECT, skipping done pages (only_unanalyzed) and existing pending jobs. - New admin endpoints (RequireAdmin, 503 when disabled, audited): POST /admin/analysis/reenqueue and POST /admin/pages/:id/analyze (force). Tests: upload enqueues N jobs / no-op when disabled; crawler persist_pages enqueue gate; admin reenqueue (backfill, idempotent, only-unanalyzed, 503, non-admin 403) and force-analyze (force flag, 404). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -31,6 +31,34 @@ pub async fn enqueue_for_page(pool: &PgPool, page_id: Uuid, force: bool) -> AppR
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Bulk-enqueue `analyze_page` jobs for existing pages — the admin
|
||||
/// backfill path for content uploaded/crawled before analysis was
|
||||
/// enabled. When `only_unanalyzed` is true, pages that already have a
|
||||
/// `done` analysis row are skipped. Pages with a pending/running
|
||||
/// `analyze_page` job are always skipped so repeated calls don't pile up
|
||||
/// duplicates. Returns the number of jobs enqueued.
|
||||
pub async fn enqueue_all_pages(pool: &PgPool, only_unanalyzed: bool) -> AppResult<u64> {
|
||||
let result = sqlx::query(
|
||||
r#"
|
||||
INSERT INTO crawler_jobs (payload)
|
||||
SELECT jsonb_build_object('kind', 'analyze_page', 'page_id', p.id, 'force', false)
|
||||
FROM pages p
|
||||
WHERE ($1 = false OR NOT EXISTS (
|
||||
SELECT 1 FROM page_analysis pa
|
||||
WHERE pa.page_id = p.id AND pa.status = 'done'))
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM crawler_jobs j
|
||||
WHERE j.payload->>'kind' = 'analyze_page'
|
||||
AND j.payload->>'page_id' = p.id::text
|
||||
AND j.state IN ('pending', 'running'))
|
||||
"#,
|
||||
)
|
||||
.bind(only_unanalyzed)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
Ok(result.rows_affected())
|
||||
}
|
||||
|
||||
/// Load a page's analysis row, if it has one.
|
||||
pub async fn load(pool: &PgPool, page_id: Uuid) -> AppResult<Option<PageAnalysis>> {
|
||||
let row = sqlx::query_as::<_, PageAnalysis>(
|
||||
|
||||
Reference in New Issue
Block a user