231 lines
11 KiB
Markdown
231 lines
11 KiB
Markdown
# Vision-manager — memory-pressure yield (design outline)
|
||
|
||
**Status:** implemented in `vision-manager/manager.sh` (feat/vision-mem-yield).
|
||
Verifications V1–V4 (§5) all passed against the current backend — expired
|
||
leases are reclaimed continuously on every claim poll (not boot-only), connection
|
||
errors requeue with backoff, the readiness gate parks before leasing, and admin
|
||
re-enqueue recovers any straggler — so no backend changes were required.
|
||
`RESPECT_CRAWL_MUTEX` is **kept** for now (retiring it is deferred until the
|
||
watermarks are tuned on the real box).
|
||
**Extends:** `VISION-AUTOSCALE.md` (the `vision-manager` sidecar) and
|
||
`vision-manager/manager.sh`.
|
||
**One-liner:** let the manager *proactively stop* `mangalord-vision` when host
|
||
memory gets dangerously tight, so a transient spike elsewhere (a Chromium crawl,
|
||
a CI build, a backend burst) can complete instead of the kernel OOM-killer
|
||
shooting something stateful (postgres / backend).
|
||
|
||
---
|
||
|
||
## 1. Motivation
|
||
|
||
`llama-server` has no idle-unload and pins ~4.4 GiB while up. On the 8 GiB Pi
|
||
the existing autoscaler already stops it when the **backlog** is empty and (new)
|
||
when **analysis is disabled**. Neither reacts to *actual memory pressure*:
|
||
|
||
- `mem_limit: 6g` on the vision service caps only **vision's own** growth. It
|
||
does nothing about **aggregate host** pressure across all services.
|
||
- The static `RESPECT_CRAWL_MUTEX` only **defers a start** while a crawl runs —
|
||
it can't stop a vision that's **already running**, and it's blind to every
|
||
non-crawl memory consumer (CI build, backend spike, a second crawl worker).
|
||
- The only backstop under true pressure today is the **kernel OOM killer**,
|
||
which may pick postgres or the backend (long-lived, stateful, painful) rather
|
||
than the one process that is cheap and safe to lose.
|
||
|
||
A manager-driven stop is the friendly version: it picks the **right,
|
||
restartable victim** and does it **gracefully (SIGTERM)** before the kernel does
|
||
something violent.
|
||
|
||
This is the **dynamic generalization of `RESPECT_CRAWL_MUTEX`**: "back off when
|
||
RAM is actually tight" strictly dominates "never co-run with a crawl." If this
|
||
ships, `RESPECT_CRAWL_MUTEX` can likely be retired.
|
||
|
||
---
|
||
|
||
## 2. Feasibility (confirmed)
|
||
|
||
`/proc/meminfo` inside the manager container reports **host-wide** memory (it is
|
||
not namespaced), so the manager already sees the truth with **zero new
|
||
privilege** — no socket-proxy `docker info`, no host mount. Verified live from
|
||
inside `vision-manager`:
|
||
|
||
```
|
||
MemTotal: 8256576 kB (7.87 GiB)
|
||
MemAvailable: 1754384 kB → ~79% used, 21% available (vision up + 560 queued)
|
||
```
|
||
|
||
**Use `MemAvailable`, never `MemFree`.** Linux spends "free" RAM on page cache by
|
||
design, so a `MemFree`-based "used %" sits near 90 % during normal operation and
|
||
would false-trigger constantly. `MemAvailable` already nets out reclaimable
|
||
cache. The honest metric is:
|
||
|
||
```
|
||
used_pct = 100 * (1 - MemAvailable / MemTotal)
|
||
```
|
||
|
||
Read it in bash with `awk` over `/proc/meminfo` (both values are in kB):
|
||
|
||
```sh
|
||
read_mem_used_pct() {
|
||
awk '/^MemTotal:/{t=$2} /^MemAvailable:/{a=$2}
|
||
END { if (t>0) printf "%d", 100*(1-a/t); else print "ERR" }' /proc/meminfo
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 3. Behaviour
|
||
|
||
A third gate in the existing poll loop, evaluated **before** the
|
||
backlog/enable start/stop logic:
|
||
|
||
1. **Stop gate (active).** If `used_pct >= HIGH_WATERMARK` (e.g. 92) **and**
|
||
vision is running → `docker stop mangalord-vision`, record a cooldown
|
||
deadline, log loudly. This is the new capability the crawl mutex never had.
|
||
2. **Start inhibit.** While `used_pct >= LOW_WATERMARK` (e.g. 80) **or** a
|
||
cooldown is in effect → refuse to start vision even if backlog > 0 (same
|
||
shape as the crawl-mutex deferral: log once per episode, not every tick).
|
||
3. **Resume.** Once `used_pct < LOW_WATERMARK` **and** the cooldown has elapsed,
|
||
normal autoscaler behaviour resumes (start on backlog, idle-stop, etc.).
|
||
|
||
**Hysteresis is mandatory** (two distinct watermarks). A single threshold
|
||
oscillates around the boundary.
|
||
|
||
**The cooldown is mandatory** and is the subtle part. If vision *itself* is the
|
||
4.4 GiB hog, stopping it drops pressure → manager sees backlog → restarts it →
|
||
pressure climbs → stop … infinite thrash, each cycle paying the ~minutes cold
|
||
reload. The cooldown (refuse-to-restart for N minutes after a pressure-stop,
|
||
*regardless of backlog*) is what gives the **other** workload — the one that
|
||
actually needed the RAM — room to finish before vision muscles back in. The
|
||
yield only makes sense when something else needs the memory; the cooldown
|
||
encodes that.
|
||
|
||
### Gate ordering in the loop
|
||
|
||
```
|
||
loop every POLL_INTERVAL:
|
||
used = read_mem_used_pct()
|
||
pending = pending_analysis()
|
||
if not analysis_enabled(): pending = 0 # existing gate
|
||
if mem_yield_active(used): pending = 0 # NEW: inhibit starts
|
||
if used >= HIGH_WATERMARK and vision running: # NEW: active stop
|
||
stop_vision(); start_cooldown(); continue
|
||
... existing start-on-backlog / idle-debounce-stop logic on `pending` ...
|
||
```
|
||
|
||
Note the symmetry with the analysis-disabled gate already in `manager.sh`:
|
||
forcing `pending = 0` reuses the existing idle/stop path for the *inhibit* case,
|
||
while the explicit `stop_vision()` handles the *active* case the idle path can't.
|
||
|
||
---
|
||
|
||
## 4. Proposed configuration (all env, matching manager.sh style)
|
||
|
||
| Var | Default | Meaning |
|
||
|---|---|---|
|
||
| `MEM_YIELD_ENABLED` | `1` | Master switch for the whole feature. |
|
||
| `MEM_HIGH_WATERMARK_PCT` | `92` | `used_pct ≥` this → active-stop a running vision. |
|
||
| `MEM_LOW_WATERMARK_PCT` | `80` | `used_pct ≥` this → inhibit starts (no restart until below). |
|
||
| `MEM_YIELD_COOLDOWN` | `300` | Seconds after a pressure-stop during which restart is refused regardless of backlog. |
|
||
| `MEM_POLL_INTERVAL` | `POLL_INTERVAL` | Optional faster cadence for the mem check (pressure can spike between 20 s ticks). |
|
||
|
||
Watermarks want tuning on the real box; 92/80 are starting points. Keep
|
||
`HIGH − LOW ≥ ~10` so the band is wider than normal jitter.
|
||
|
||
---
|
||
|
||
## 5. Required verifications (do these BEFORE building)
|
||
|
||
The queue is built for expendability — `crawler_jobs` is a lease queue:
|
||
|
||
```
|
||
state | attempts | max_attempts(=5) | leased_until | last_error | scheduled_at
|
||
```
|
||
|
||
A graceful `docker stop` (SIGTERM → clean llama-server exit) means at most the
|
||
**one** in-flight `analyze_page` job is affected; the readiness gate
|
||
(`ANALYSIS_VISION_HEALTH_URL`) then parks the worker so it won't hammer the dead
|
||
endpoint or burn the other queued jobs. But two things must be confirmed in
|
||
`backend/src/analysis/daemon.rs` (not visible from the DB schema alone):
|
||
|
||
- [ ] **V1 — reclaim cadence.** The `bugfix/crawler-recovery-hardening` branch
|
||
added *reclaim-orphaned at startup*. Confirm expired leases are **also**
|
||
reclaimed on the normal claim poll (i.e. the claim query includes something
|
||
like `OR (state='running' AND leased_until < now())`). If reclaim is
|
||
**boot-only**, a pressure-kill strands that one job as `running` with a dead
|
||
lease until the next backend restart → the page silently never analyses.
|
||
**This decides how aggressive we can safely be.**
|
||
- [ ] **V2 — error path requeue.** When the worker's HTTP call to vision is
|
||
refused (vision killed), confirm the job is returned to `pending`
|
||
(`attempts++`) and not marked permanently `failed` on a connection error.
|
||
A connection-refused/transport error should be *retryable*, distinct from a
|
||
model-returned bad response.
|
||
- [ ] **V3 — park does not deadlock.** Confirm that when the readiness gate
|
||
parks the worker, it is **not holding an un-renewed lease** that blocks the
|
||
job indefinitely (it should either release on park or let the lease expire +
|
||
be reclaimed per V1).
|
||
- [ ] **V4 — attempts accounting.** A pressure-kill is "our fault," yet still
|
||
burns one of the 5 attempts. Confirm repeated yields on the *same* in-flight
|
||
page can't quietly exhaust retries → permanent `failed`. Mitigations if it's a
|
||
concern: bump `max_attempts`, or rely on the admin re-enqueue (all/manga/
|
||
chapter) you already have to recover stragglers.
|
||
|
||
**Empirical check** (once V1–V3 look right): with analysis on and one job
|
||
`running`, `docker stop mangalord-vision`, then watch
|
||
`SELECT state, attempts, last_error FROM crawler_jobs WHERE state IN
|
||
('running','pending') AND payload->>'kind'='analyze_page'` — the killed job
|
||
should reappear as `pending` with `attempts` incremented, and resume when vision
|
||
restarts. This is the acceptance test for the whole feature.
|
||
|
||
---
|
||
|
||
## 6. Pitfalls
|
||
|
||
- **`MemFree` vs `MemAvailable`** — using `MemFree` makes it fire constantly
|
||
(page cache). Already addressed by §2, but it's the #1 way to get this wrong.
|
||
- **Thrash / feedback loop** — if vision is the hog, stop→restart oscillates.
|
||
The cooldown (§3) is the guard; without it the feature is worse than nothing.
|
||
- **Spikes between ticks** — a Chromium burst can blow past 92 % inside a 20 s
|
||
poll gap and the kernel OOM-kills before the manager's next read. The valve
|
||
*reduces* OOM risk, it doesn't *eliminate* it. Consider a shorter
|
||
`MEM_POLL_INTERVAL` for just the mem read, and treat this as defence-in-depth
|
||
alongside (not instead of) `mem_limit` and swap/zram headroom.
|
||
- **Measuring the wrong scope** — confirm `/proc/meminfo` in the container keeps
|
||
showing **host** totals under this kernel/cgroup setup (it does today:
|
||
MemTotal 8256576 kB == the Pi). If a future runtime namespaces it, the metric
|
||
silently becomes the container's tiny cgroup and the gate never/always fires.
|
||
- **Cold-start cost** — every pressure-stop pays ~minutes to mmap + warm 3.2 GiB
|
||
on the next start. Don't set the watermarks so tight that normal operation
|
||
trips them; the backlog will stall during each reload.
|
||
- **Interaction with the idle-stop and disable gate** — make sure the three
|
||
reasons vision can be down (idle-drained, analysis-disabled, memory-yield) are
|
||
**distinctly logged** (flip-only, like the existing gates) so a stop is never
|
||
misdiagnosed. A single muddy "stopped vision" line will cost debugging time.
|
||
- **Double-stop / start races** — `docker stop` then an immediate backlog-driven
|
||
`docker start` in the same or next tick. The cooldown plus checking
|
||
`vision_running` before acting prevents this; don't drop those guards.
|
||
- **OOM-killer still owns the tail** — `mem_limit: 6g` caps vision's own growth
|
||
but the host can still be pushed over by the *sum* of everything else. This
|
||
feature lowers the probability of a bad OOM kill; pairing it with a little
|
||
**swap or zram** gives the kernel a cushion to survive the gap between a spike
|
||
and the manager's reaction.
|
||
|
||
---
|
||
|
||
## 7. Recommendation
|
||
|
||
Worth building, as a **third gate** in the existing `manager.sh` poll loop:
|
||
`MemAvailable`-based metric, two watermarks (hysteresis), a restart cooldown,
|
||
an **active stop** (the capability the crawl mutex lacks), and distinct
|
||
flip-only logging. It supersedes `RESPECT_CRAWL_MUTEX` (which can then be
|
||
removed).
|
||
|
||
**Gate it on verification V1 first** — whether expired leases are reclaimed
|
||
continuously or only at boot decides whether a yielded job cleanly retries or
|
||
strands. If V1/V2 hold, this is a low-risk, high-value safety valve; if reclaim
|
||
is boot-only, fix that in the backend before enabling memory-yield, otherwise
|
||
each pressure-stop quietly drops a page until the next backend restart.
|
||
|
||
Keep the manager dumb: this is ~30 lines of bash (one `awk` reader, two
|
||
watermark comparisons, one cooldown timestamp) — no new dependencies, no new
|
||
privilege, no web surface.
|