diff --git a/HYBRID-OCR-VISION.md b/HYBRID-OCR-VISION.md new file mode 100644 index 0000000..80ac5d0 --- /dev/null +++ b/HYBRID-OCR-VISION.md @@ -0,0 +1,136 @@ +# Hybrid OCR→Vision analysis pipeline — design brief for the dev agent + +**Status: design only (not implemented).** Forward-looking model for a future +`ANALYSIS_BACKEND=hybrid`, on top of the shipped `ocr` (default) and `vision` +backends. Constraint that shapes it: `ocrs` and the vision LLM must **not** be +resident at the same time (Pi RAM ceiling). OCR has priority; vision is a +preemptible, only-when-OCR-is-drained background phase. + +## The key realization: this is the crawl mutex, again + +The repo already runs a **`vision-manager`** sidecar +([vision-manager/manager.sh](vision-manager/manager.sh), +[VISION-AUTOSCALE.md](VISION-AUTOSCALE.md)) that owns the `mangalord-vision` +(llama.cpp, ~4 GiB) container lifecycle via a scoped docker-socket-proxy — the +backend gets no Docker access. It already: + +- **Starts** vision when `pending_analysis()` > 0 (counts `analyze_page` jobs). +- **Idle-stops** it after `STOP_DEBOUNCE` (anti-thrash) once the queue drains. +- **Defers starting** vision while `crawl_running()` > 0 — the **"RAM mutex"**: + the crawler's browser and the LLM can't both fit, so vision waits for crawl. +- **Memory-yield**: SIGTERMs a running vision at a HIGH host-RAM watermark, + blocks starting at a LOW watermark (see [VISION-MEMORY-YIELD.md](VISION-MEMORY-YIELD.md)). + +The OCR↔vision relationship is the relationship that already exists between +crawl and vision. OCR is just a third RAM-competing, higher-priority workload +that vision must defer to. Extend the existing arbiter rather than build a new one. + +## Job model: split one queue into two + +Today there is one job kind, `analyze_page`. Split it: + +- **`ocr_page`** — the in-process ocrs pass (cheap, fast, no container). +- **`ground_page`** — the vision grounding pass (needs the LLM container). + +**Pipeline (hybrid mode):** upload → enqueue `ocr_page`. The OCR daemon runs +ocrs, writes `page_ocr_text` + `search_doc` (text searchable *immediately*), then +enqueues `ground_page` for that page. The grounding daemon later adds +tags/scene/safety. Each kind gets its own dedup index (mirror migration 0031). + +Two daemons, two queues. + +## Arbitration: OCR priority + memory exclusivity + +Responsibilities split cleanly between the in-backend daemons and the manager: + +**Backend — OCR daemon** (already built): leases `ocr_page`, always allowed to +run (ocrs is small/local). This is the priority phase. + +**Backend — grounding daemon:** leases `ground_page`, but gates leasing on +**both**: +1. `VisionReadiness` (the existing `/health` gate — vision container up), AND +2. **`ocr_backlog == 0`** (a cheap `count(ocr_page WHERE state IN pending,running)`). + +Gate #2 is the whole priority mechanism, in-process, no aborts: the instant OCR +work appears, the grounding daemon **finishes its current page** (the in-flight +lease completes normally) and then **stops leasing** new pages and parks — "it +finishes, then yields." It resumes only when OCR has fully drained. + +**vision-manager:** two one-line changes to the existing logic: +1. `pending_analysis()` counts **`ground_page`** (not `ocr_page`) — vision only + starts when there is *grounding* work. +2. Add an **OCR start-block mutex** identical to `RESPECT_CRAWL_MUTEX`: defer + starting vision while `ocr_page` backlog > 0. (`RESPECT_OCR_MUTEX=1`.) + +**Why this yields memory exclusivity:** while OCR backlog > 0, the grounding +daemon won't lease → vision goes idle → the manager's idle-debounce stops the +container (and never restarts it under the OCR mutex). Only ocrs (small) is +resident. When OCR drains, grounding resumes, `pending_analysis()` > 0 again, the +manager starts vision. Only the *big* consumer (the LLM container) is ever +mutually exclusive with active OCR — which is the actual RAM constraint. + +**The one overlap window:** if OCR work arrives mid-grounding-page, ocrs (small, +in-process) briefly coexists with the one in-flight vision page before the +grounding daemon parks. ocrs's footprint makes this a non-issue in practice. If a +deployment needs *hard* exclusivity even there, the OCR daemon can additionally +wait for `vision_running == false` before its first dispatch. No deadlock: OCR's +wait is transient (one grounding page, bounded by `job_timeout`), while vision's +deferral on OCR backlog is the persistent, priority-respecting side. + +**Anti-thrash:** the manager's `STOP_DEBOUNCE` already prevents a stray OCR page +from cold-cycling the 4 GiB model. Chapter uploads arrive as bursts, so OCR +drains a whole batch in one phase before vision resumes — the natural good case. + +## Data-model split (the real refactor) + +`persist_analysis` currently writes OCR + tags + scene + safety in one +transaction and derives `search_doc` from the OCR rows (+ scene weight C). Two +passes means splitting it, sharing the tsvector builder: + +- **`persist_ocr(page, lines)`** — writes `page_ocr_text`, sets + `search_doc` from OCR (A/B/D buckets), marks the page text-searchable. Status + reflects OCR completion. +- **`persist_grounding(page, tags, scene, safety)`** — writes auto-tags, + warnings, scene; **recomputes** `search_doc` to fold in the scene (weight C) on + top of the existing OCR buckets. Records grounding completion (e.g. a + `grounded_at` column or tags-present sentinel). + +This keeps text search live after the cheap OCR phase, with semantic search/tags +arriving after the (deferred) grounding phase. + +## Reusable pieces + +- OCR side: the shipped `OcrsEngine` + `OcrAnalyzeDispatcher` + ([backend/src/analysis/ocr.rs](backend/src/analysis/ocr.rs)) — retarget to + `ocr_page`, and on success enqueue `ground_page`. +- Vision side: factor `VisionClient::ground(image, mime, ocr_text)` out of + `analyze()`'s Pass-B block + ([backend/src/analysis/vision.rs](backend/src/analysis/vision.rs) ~240–261) — + pure refactor, existing tests cover it. The grounding daemon is the existing + analysis daemon retargeted to `ground_page` with the extra `ocr_backlog==0` + lease gate. +- Manager: `pending_analysis()` kind swap + `RESPECT_OCR_MUTEX` block, mirroring + the crawl-mutex branch already at [vision-manager/manager.sh](vision-manager/manager.sh). + +## Scope / sequencing + +Larger than an inline hybrid dispatcher: job-kind split + dedup migrations, +`persist_*` split + search_doc-builder extraction, pipeline enqueue (OCR→ground), +grounding daemon lease gate, and the manager mutex. No new privileged surface +(reuses the docker-socket-proxy). Suggested order: +1. Split `persist_analysis` → `persist_ocr` / `persist_grounding` (+ tests). +2. Split job kinds + dedup migrations; retarget the OCR daemon to `ocr_page`. +3. `VisionClient::ground()` refactor + grounding daemon (`ground_page`, gated on + `ocr_backlog==0`). +4. OCR→ground enqueue on OCR success (hybrid mode only). +5. vision-manager: `ground_page` counting + `RESPECT_OCR_MUTEX`. + +Pure-`ocr` (shipped default) and `vision` (legacy) backends are unaffected; this +is the `hybrid` backend's runtime model. + +## Open decision + +Hard exclusivity in the one overlap window (OCR daemon waits for `vision_running +== false` before its first dispatch) — needed only if ocrs's few-hundred-MB +footprint can't coexist with a single in-flight grounding page on the target +Pi's RAM. Default: don't gate (rely on the grounding daemon parking fast).