feat(ops): vision-manager sidecar to autoscale the llama.cpp container
Add a tiny privileged sidecar (vision-manager/) that polls the analysis backlog in Postgres and starts/stops the mangalord-vision container by name: start when analyze_page jobs are pending, stop after STOP_DEBOUNCE idle. It is the single owner of the vision lifecycle and the only component with Docker access — scoped through tecnativa/docker-socket-proxy (CONTAINERS+POST only) on an internal-only network, so the internet-facing backend never touches the socket. Both helpers sit behind `profiles: [ai]` (vanilla `compose up` is unaffected). The manager debounces the stop, gates the first request on GET /health==200 after a cold start, honours a crawl RAM-mutex on the 8 GiB box, and owns its own idle timer (leak-safe if the backend dies). Backlog query uses the real schema (state + payload->>'kind'); a read-only DB role (readonly-role.sql) keeps it off the backend creds. Bump 0.80.0 -> 0.81.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
129
VISION-AUTOSCALE.md
Normal file
129
VISION-AUTOSCALE.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# Vision auto start/stop — design brief for the dev agent
|
||||
|
||||
**Goal:** the `mangalord-vision` (llama.cpp) container should only run while there
|
||||
is analysis work, and stop (freeing its ~4 GiB) once the queue drains —
|
||||
*without* handing the internet-facing backend control of the host Docker daemon.
|
||||
|
||||
`llama-server` has **no idle-unload** (unlike Ollama). The only lever is the
|
||||
container lifecycle: start it when work appears, stop it when work is gone.
|
||||
|
||||
> **Status: implemented.** The sidecar lives in [vision-manager/](vision-manager/)
|
||||
> (`manager.sh` + `Dockerfile` + `readonly-role.sql`), wired into
|
||||
> [docker-compose.yml](docker-compose.yml) behind `profiles: [ai]` alongside a
|
||||
> scoped `docker-socket-proxy`. The companion **readiness gate** is in the
|
||||
> analysis worker: when `ANALYSIS_VISION_HEALTH_URL` is set the worker will not
|
||||
> lease a page until vision answers `GET /health` 2xx, so an idle-stopped or
|
||||
> still-loading vision never burns a job's retries or writes a `failed` row
|
||||
> (the gotcha called out below). Configure via the `ai`-profile vars in
|
||||
> [.env.example](.env.example).
|
||||
|
||||
## Chosen approach — Option 2: a "vision-manager" sidecar
|
||||
|
||||
A tiny, single-purpose container that:
|
||||
|
||||
1. **Watches the analysis backlog** in Postgres (read-only DB user).
|
||||
2. **Starts** `mangalord-vision` when there is pending work.
|
||||
3. **Stops** it after the backlog has been empty for a debounce window.
|
||||
|
||||
The backend (`mangalord-backend`) is **not modified and gets no Docker access** —
|
||||
all privilege is isolated in the manager. This is the whole point: a popped
|
||||
backend can't reach the Docker socket.
|
||||
|
||||
```
|
||||
┌────────────────┐ reads pending count ┌────────────┐
|
||||
│ vision-manager │ ───────────────────────▶│ postgres │
|
||||
│ (has socket) │ └────────────┘
|
||||
└──────┬─────────┘
|
||||
start/stop one │ (raw socket, OR scoped via docker-socket-proxy)
|
||||
container only ▼
|
||||
┌────────────────┐
|
||||
│ mangalord-vision│ (profiles: [ai], started by name)
|
||||
└────────────────┘
|
||||
```
|
||||
|
||||
### What to poll (concrete)
|
||||
|
||||
The analysis daemon consumes the **shared `crawler_jobs` queue, filtered by the
|
||||
analysis job kind**, and every unanalyzed page has a `page_analysis` row with
|
||||
`status='pending'`. Either is a valid "is there work?" signal:
|
||||
|
||||
- **Queue-accurate (what the manager uses):** the queue column is `state`
|
||||
(not `status`) and the kind lives in the JSONB payload — see
|
||||
[backend/migrations/0012_crawler.sql](backend/migrations/0012_crawler.sql):
|
||||
```sql
|
||||
SELECT count(*) FROM crawler_jobs
|
||||
WHERE payload->>'kind' = 'analyze_page'
|
||||
AND state IN ('pending','running');
|
||||
```
|
||||
- **Backlog-simple:** `SELECT count(*) FROM page_analysis WHERE status='pending'`.
|
||||
|
||||
Prefer a **read-only** DB role scoped to those tables. Don't reuse the backend's
|
||||
DB credentials.
|
||||
|
||||
### Lifecycle logic (sketch)
|
||||
|
||||
```
|
||||
loop every POLL_INTERVAL (e.g. 20s):
|
||||
pending = count_pending_work()
|
||||
if pending > 0 and vision is stopped:
|
||||
docker start mangalord-vision
|
||||
wait for GET /health == 200 (up to a few minutes — cold model load)
|
||||
reset idle timer
|
||||
if pending == 0 and vision is running:
|
||||
if idle for >= STOP_DEBOUNCE (e.g. 5–10 min):
|
||||
docker stop mangalord-vision
|
||||
```
|
||||
|
||||
## Recommendations
|
||||
|
||||
- **Scope the Docker access.** Best: run `tecnativa/docker-socket-proxy` on an
|
||||
internal-only network with everything disabled except container start/stop,
|
||||
and point the manager at the proxy. Acceptable for a tiny trusted manager:
|
||||
mount the raw socket *into the manager only* (never the backend).
|
||||
- **Keep the manager dumb and small.** ~100 lines. A shell loop with
|
||||
`docker`/`curl`, or a small Go/Rust binary. No web surface.
|
||||
- **Make it the single source of truth** for the vision lifecycle. Don't also
|
||||
have the backend or a cron poke the same container.
|
||||
- **Start by container name**, not `compose up` (the service is `profiles:[ai]`;
|
||||
name-based `docker start/stop` works and won't fight a separate `compose up`).
|
||||
- Optionally expose the toggle as a runtime setting ("auto-manage vision: on/off")
|
||||
so it can be disabled without redeploying.
|
||||
|
||||
## Pitfalls / gotchas (flag all of these)
|
||||
|
||||
- **Cold start is expensive (~minutes)** to mmap 3.2 GiB + warm up. So:
|
||||
- **Debounce the STOP** (5–10 min idle) — bursty uploads/re-analysis enqueue
|
||||
in waves; stopping the instant the queue hits zero thrashes the load cycle
|
||||
and loses more time than it saves.
|
||||
- **Gate the first request on `GET /health == 200`** after start, or the first
|
||||
jobs fail with connection-refused. (The backend already has request/job
|
||||
timeouts — 300s/1800s — but a multi-minute cold start can still exceed a
|
||||
single request timeout if a job is dispatched too eagerly.)
|
||||
- **Idempotency / single-flight:** starting an already-running container must be
|
||||
a no-op; never issue concurrent starts. One manager instance only.
|
||||
- **Leak safety / don't depend on a "drained" signal from the backend:** the
|
||||
manager's own idle timer must stop the container even if the backend crashes
|
||||
mid-batch. Conversely, if the manager dies, the container keeps running
|
||||
(harmless, just RAM) — consider a max-uptime backstop.
|
||||
- **RAM headroom on the 8 GiB Pi:** vision sits ~4.4 GiB; `mem_limit: 6g` is set
|
||||
on the service. The manager must not start vision while another big consumer
|
||||
(a crawl with Chromium) is running, or the box OOMs. Consider a simple mutual
|
||||
exclusion / total-RAM check.
|
||||
- **Health vs readiness:** `/health` returns 200 once the model is loaded; that
|
||||
is the readiness signal. Don't treat "container running" as "ready".
|
||||
- **Re-analysis storms:** an admin "re-enqueue all" can flood the queue; the
|
||||
manager will (correctly) start vision and keep it up for a long backfill —
|
||||
expected, but make sure the STOP debounce doesn't bounce it mid-backfill if
|
||||
the queue briefly empties between batches.
|
||||
- **Profile interaction:** `docker compose --profile ai up` / CI deploys name
|
||||
only the two mangalord services, so they won't start/stop vision — but a
|
||||
human running a full `compose --profile ai up -d` could. Document that the
|
||||
manager owns the lifecycle.
|
||||
|
||||
## Why not give the backend the socket directly
|
||||
|
||||
`mangalord-backend` is internet-facing (behind Caddy at manga.mc02.dev). Mounting
|
||||
the raw Docker socket there makes any backend RCE a full host takeover
|
||||
(postgres, gitea, vaultwarden, …). If the backend *must* drive it, use the
|
||||
docker-socket-proxy scoped to start/stop of the single container — never the raw
|
||||
socket. Option 2 sidesteps the question entirely by keeping the backend clean.
|
||||
Reference in New Issue
Block a user