feat(ops): vision-manager sidecar to autoscale the llama.cpp container

Add a tiny privileged sidecar (vision-manager/) that polls the analysis
backlog in Postgres and starts/stops the mangalord-vision container by name:
start when analyze_page jobs are pending, stop after STOP_DEBOUNCE idle.
It is the single owner of the vision lifecycle and the only component with
Docker access — scoped through tecnativa/docker-socket-proxy (CONTAINERS+POST
only) on an internal-only network, so the internet-facing backend never
touches the socket.

Both helpers sit behind `profiles: [ai]` (vanilla `compose up` is
unaffected). The manager debounces the stop, gates the first request on
GET /health==200 after a cold start, honours a crawl RAM-mutex on the 8 GiB
box, and owns its own idle timer (leak-safe if the backend dies). Backlog
query uses the real schema (state + payload->>'kind'); a read-only DB role
(readonly-role.sql) keeps it off the backend creds.

Bump 0.80.0 -> 0.81.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-14 15:02:46 +02:00
parent b86aa80c87
commit 64a9dceb67
9 changed files with 419 additions and 3 deletions

129
VISION-AUTOSCALE.md Normal file
View File

@@ -0,0 +1,129 @@
# Vision auto start/stop — design brief for the dev agent
**Goal:** the `mangalord-vision` (llama.cpp) container should only run while there
is analysis work, and stop (freeing its ~4 GiB) once the queue drains —
*without* handing the internet-facing backend control of the host Docker daemon.
`llama-server` has **no idle-unload** (unlike Ollama). The only lever is the
container lifecycle: start it when work appears, stop it when work is gone.
> **Status: implemented.** The sidecar lives in [vision-manager/](vision-manager/)
> (`manager.sh` + `Dockerfile` + `readonly-role.sql`), wired into
> [docker-compose.yml](docker-compose.yml) behind `profiles: [ai]` alongside a
> scoped `docker-socket-proxy`. The companion **readiness gate** is in the
> analysis worker: when `ANALYSIS_VISION_HEALTH_URL` is set the worker will not
> lease a page until vision answers `GET /health` 2xx, so an idle-stopped or
> still-loading vision never burns a job's retries or writes a `failed` row
> (the gotcha called out below). Configure via the `ai`-profile vars in
> [.env.example](.env.example).
## Chosen approach — Option 2: a "vision-manager" sidecar
A tiny, single-purpose container that:
1. **Watches the analysis backlog** in Postgres (read-only DB user).
2. **Starts** `mangalord-vision` when there is pending work.
3. **Stops** it after the backlog has been empty for a debounce window.
The backend (`mangalord-backend`) is **not modified and gets no Docker access**
all privilege is isolated in the manager. This is the whole point: a popped
backend can't reach the Docker socket.
```
┌────────────────┐ reads pending count ┌────────────┐
│ vision-manager │ ───────────────────────▶│ postgres │
│ (has socket) │ └────────────┘
└──────┬─────────┘
start/stop one │ (raw socket, OR scoped via docker-socket-proxy)
container only ▼
┌────────────────┐
│ mangalord-vision│ (profiles: [ai], started by name)
└────────────────┘
```
### What to poll (concrete)
The analysis daemon consumes the **shared `crawler_jobs` queue, filtered by the
analysis job kind**, and every unanalyzed page has a `page_analysis` row with
`status='pending'`. Either is a valid "is there work?" signal:
- **Queue-accurate (what the manager uses):** the queue column is `state`
(not `status`) and the kind lives in the JSONB payload — see
[backend/migrations/0012_crawler.sql](backend/migrations/0012_crawler.sql):
```sql
SELECT count(*) FROM crawler_jobs
WHERE payload->>'kind' = 'analyze_page'
AND state IN ('pending','running');
```
- **Backlog-simple:** `SELECT count(*) FROM page_analysis WHERE status='pending'`.
Prefer a **read-only** DB role scoped to those tables. Don't reuse the backend's
DB credentials.
### Lifecycle logic (sketch)
```
loop every POLL_INTERVAL (e.g. 20s):
pending = count_pending_work()
if pending > 0 and vision is stopped:
docker start mangalord-vision
wait for GET /health == 200 (up to a few minutes — cold model load)
reset idle timer
if pending == 0 and vision is running:
if idle for >= STOP_DEBOUNCE (e.g. 510 min):
docker stop mangalord-vision
```
## Recommendations
- **Scope the Docker access.** Best: run `tecnativa/docker-socket-proxy` on an
internal-only network with everything disabled except container start/stop,
and point the manager at the proxy. Acceptable for a tiny trusted manager:
mount the raw socket *into the manager only* (never the backend).
- **Keep the manager dumb and small.** ~100 lines. A shell loop with
`docker`/`curl`, or a small Go/Rust binary. No web surface.
- **Make it the single source of truth** for the vision lifecycle. Don't also
have the backend or a cron poke the same container.
- **Start by container name**, not `compose up` (the service is `profiles:[ai]`;
name-based `docker start/stop` works and won't fight a separate `compose up`).
- Optionally expose the toggle as a runtime setting ("auto-manage vision: on/off")
so it can be disabled without redeploying.
## Pitfalls / gotchas (flag all of these)
- **Cold start is expensive (~minutes)** to mmap 3.2 GiB + warm up. So:
- **Debounce the STOP** (510 min idle) — bursty uploads/re-analysis enqueue
in waves; stopping the instant the queue hits zero thrashes the load cycle
and loses more time than it saves.
- **Gate the first request on `GET /health == 200`** after start, or the first
jobs fail with connection-refused. (The backend already has request/job
timeouts — 300s/1800s — but a multi-minute cold start can still exceed a
single request timeout if a job is dispatched too eagerly.)
- **Idempotency / single-flight:** starting an already-running container must be
a no-op; never issue concurrent starts. One manager instance only.
- **Leak safety / don't depend on a "drained" signal from the backend:** the
manager's own idle timer must stop the container even if the backend crashes
mid-batch. Conversely, if the manager dies, the container keeps running
(harmless, just RAM) — consider a max-uptime backstop.
- **RAM headroom on the 8 GiB Pi:** vision sits ~4.4 GiB; `mem_limit: 6g` is set
on the service. The manager must not start vision while another big consumer
(a crawl with Chromium) is running, or the box OOMs. Consider a simple mutual
exclusion / total-RAM check.
- **Health vs readiness:** `/health` returns 200 once the model is loaded; that
is the readiness signal. Don't treat "container running" as "ready".
- **Re-analysis storms:** an admin "re-enqueue all" can flood the queue; the
manager will (correctly) start vision and keep it up for a long backfill —
expected, but make sure the STOP debounce doesn't bounce it mid-backfill if
the queue briefly empties between batches.
- **Profile interaction:** `docker compose --profile ai up` / CI deploys name
only the two mangalord services, so they won't start/stop vision — but a
human running a full `compose --profile ai up -d` could. Document that the
manager owns the lifecycle.
## Why not give the backend the socket directly
`mangalord-backend` is internet-facing (behind Caddy at manga.mc02.dev). Mounting
the raw Docker socket there makes any backend RCE a full host takeover
(postgres, gitea, vaultwarden, …). If the backend *must* drive it, use the
docker-socket-proxy scoped to start/stop of the single container — never the raw
socket. Option 2 sidesteps the question entirely by keeping the backend clean.