Mangalord/VISION-AUTOSCALE.md

Vision auto start/stop — design brief for the dev agent

Goal: the mangalord-vision (llama.cpp) container should only run while there is analysis work, and stop (freeing its ~4 GiB) once the queue drains — without handing the internet-facing backend control of the host Docker daemon.

llama-server has no idle-unload (unlike Ollama). The only lever is the container lifecycle: start it when work appears, stop it when work is gone.

Status: implemented. The sidecar lives in vision-manager/ (manager.sh + Dockerfile + readonly-role.sql), wired into docker-compose.yml behind profiles: [ai] alongside a scoped docker-socket-proxy. The companion readiness gate is in the analysis worker: when ANALYSIS_VISION_HEALTH_URL is set the worker will not lease a page until vision answers GET /health 2xx, so an idle-stopped or still-loading vision never burns a job's retries or writes a failed row (the gotcha called out below). Configure via the ai-profile vars in .env.example.

Operator prerequisite: the mangalord-vision container is defined outside this compose project (the manager only drives it by name). For the manager and backend to resolve it by name for the /health probe, attach it to this project's default network (<project>_default, e.g. mangalord_default) — or point VISION_HEALTH_URL / ANALYSIS_VISION_HEALTH_URL at an address that resolves. If the container is unreachable the manager silently treats it as "not running / not ready" and will loop trying to start a container it cannot see.

Chosen approach — Option 2: a "vision-manager" sidecar

A tiny, single-purpose container that:

1. Watches the analysis backlog in Postgres (read-only DB user).
2. Starts mangalord-vision when there is pending work.
3. Stops it after the backlog has been empty for a debounce window.

The backend (mangalord-backend) is not modified and gets no Docker access — all privilege is isolated in the manager. This is the whole point: a popped backend can't reach the Docker socket.

            ┌────────────────┐   reads pending count   ┌────────────┐
            │ vision-manager │ ───────────────────────▶│  postgres  │
            │  (has socket)  │                          └────────────┘
            └──────┬─────────┘
   start/stop one  │  (raw socket, OR scoped via docker-socket-proxy)
   container only  ▼
            ┌────────────────┐
            │ mangalord-vision│  (profiles: [ai], started by name)
            └────────────────┘

What to poll (concrete)

The analysis daemon consumes the shared crawler_jobs queue, filtered by the analysis job kind, and every unanalyzed page has a page_analysis row with status='pending'. Either is a valid "is there work?" signal:

• Queue-accurate (what the manager uses): the queue column is state (not status) and the kind lives in the JSONB payload — see backend/migrations/0012_crawler.sql:

  SELECT count(*) FROM crawler_jobs
   WHERE payload->>'kind' = 'analyze_page'
     AND state IN ('pending','running');
  

• Backlog-simple: SELECT count(*) FROM page_analysis WHERE status='pending'.

Prefer a read-only DB role scoped to those tables. Don't reuse the backend's DB credentials.

Lifecycle logic (sketch)

loop every POLL_INTERVAL (e.g. 20s):
    pending = count_pending_work()
    if pending > 0 and vision is stopped:
        docker start mangalord-vision
        wait for GET /health == 200 (up to a few minutes — cold model load)
        reset idle timer
    if pending == 0 and vision is running:
        if idle for >= STOP_DEBOUNCE (e.g. 5–10 min):
            docker stop mangalord-vision

Recommendations

• Scope the Docker access. Best: run tecnativa/docker-socket-proxy on an internal-only network with everything disabled except container start/stop, and point the manager at the proxy. Acceptable for a tiny trusted manager: mount the raw socket into the manager only (never the backend).
• Keep the manager dumb and small. ~100 lines. A shell loop with docker/curl, or a small Go/Rust binary. No web surface.
• Make it the single source of truth for the vision lifecycle. Don't also have the backend or a cron poke the same container.
• Start by container name, not compose up (the service is profiles:[ai]; name-based docker start/stop works and won't fight a separate compose up).
• Optionally expose the toggle as a runtime setting ("auto-manage vision: on/off") so it can be disabled without redeploying.

Pitfalls / gotchas (flag all of these)

• Cold start is expensive (~minutes) to mmap 3.2 GiB + warm up. So:
  • Debounce the STOP (5–10 min idle) — bursty uploads/re-analysis enqueue in waves; stopping the instant the queue hits zero thrashes the load cycle and loses more time than it saves.
  • Gate the first request on GET /health == 200 after start, or the first jobs fail with connection-refused. (The backend already has request/job timeouts — 300s/1800s — but a multi-minute cold start can still exceed a single request timeout if a job is dispatched too eagerly.)
• Idempotency / single-flight: starting an already-running container must be a no-op; never issue concurrent starts. One manager instance only.
• Leak safety / don't depend on a "drained" signal from the backend: the manager's own idle timer must stop the container even if the backend crashes mid-batch. Conversely, if the manager dies, the container keeps running (harmless, just RAM) — consider a max-uptime backstop.
• RAM headroom on the 8 GiB Pi: vision sits ~4.4 GiB; mem_limit: 6g is set on the service. The manager must not start vision while another big consumer (a crawl with Chromium) is running, or the box OOMs. Consider a simple mutual exclusion / total-RAM check.
• Health vs readiness: /health returns 200 once the model is loaded; that is the readiness signal. Don't treat "container running" as "ready".
• Re-analysis storms: an admin "re-enqueue all" can flood the queue; the manager will (correctly) start vision and keep it up for a long backfill — expected, but make sure the STOP debounce doesn't bounce it mid-backfill if the queue briefly empties between batches.
• Profile interaction: docker compose --profile ai up / CI deploys name only the two mangalord services, so they won't start/stop vision — but a human running a full compose --profile ai up -d could. Document that the manager owns the lifecycle.

Why not give the backend the socket directly

mangalord-backend is internet-facing (behind Caddy at manga.mc02.dev). Mounting the raw Docker socket there makes any backend RCE a full host takeover (postgres, gitea, vaultwarden, …). If the backend must drive it, use the docker-socket-proxy scoped to start/stop of the single container — never the raw socket. Option 2 sidesteps the question entirely by keeping the backend clean.