feat(vision-manager): memory-pressure yield gate (+ retain analysis gate) (#8)

.env.example

@@ -218,3 +218,15 @@ VISION_MANAGER_DATABASE_URL=
 # VISION_START_HEALTH_TIMEOUT  Max seconds to wait for /health 200 after start. Default 300 (cold model load).
 # VISION_RESPECT_CRAWL_MUTEX   1 = don't start vision while a crawl runs (RAM mutex on the 8 GiB box). Default 1.
 # VISION_MAX_UPTIME         >0 = force-stop vision after N seconds running (backstop). Default 0 (disabled).
+#
+# Memory-pressure yield — the dynamic generalization of the crawl mutex. The
+# manager reads host-wide used% (from /proc/meminfo MemAvailable) and stops a
+# RUNNING vision when memory gets tight, so a spike elsewhere (a crawl, CI, a
+# backend burst) can finish instead of the kernel OOM-killer shooting something
+# stateful. Two watermarks give hysteresis; the cooldown prevents stop/restart
+# thrash when vision itself is the hog. See VISION-MEMORY-YIELD.md.
+# VISION_MEM_YIELD_ENABLED      1 = enable the gate, 0 = disable entirely. Default 1.
+# VISION_MEM_HIGH_WATERMARK_PCT used% >= this → actively stop a running vision. Default 92.
+# VISION_MEM_LOW_WATERMARK_PCT  used% >= this → inhibit starts (keep HIGH - LOW >= ~10 so the band beats jitter). Default 80.
+# VISION_MEM_YIELD_COOLDOWN     Seconds after a pressure-stop during which restart is refused regardless of backlog. Default 300.
+# VISION_MEM_POLL_INTERVAL      Mem sub-poll cadence, seconds (<= VISION_POLL_INTERVAL); catches spikes between backlog ticks. Default 5.

VISION-MEMORY-YIELD.md

@@ -0,0 +1,230 @@
+# Vision-manager — memory-pressure yield (design outline)
+
+**Status:** implemented in `vision-manager/manager.sh` (feat/vision-mem-yield).
+Verifications V1–V4 (§5) all passed against the current backend — expired
+leases are reclaimed continuously on every claim poll (not boot-only), connection
+errors requeue with backoff, the readiness gate parks before leasing, and admin
+re-enqueue recovers any straggler — so no backend changes were required.
+`RESPECT_CRAWL_MUTEX` is **kept** for now (retiring it is deferred until the
+watermarks are tuned on the real box).
+**Extends:** `VISION-AUTOSCALE.md` (the `vision-manager` sidecar) and
+`vision-manager/manager.sh`.
+**One-liner:** let the manager *proactively stop* `mangalord-vision` when host
+memory gets dangerously tight, so a transient spike elsewhere (a Chromium crawl,
+a CI build, a backend burst) can complete instead of the kernel OOM-killer
+shooting something stateful (postgres / backend).
+
+---
+
+## 1. Motivation
+
+`llama-server` has no idle-unload and pins ~4.4 GiB while up. On the 8 GiB Pi
+the existing autoscaler already stops it when the **backlog** is empty and (new)
+when **analysis is disabled**. Neither reacts to *actual memory pressure*:
+
+- `mem_limit: 6g` on the vision service caps only **vision's own** growth. It
+  does nothing about **aggregate host** pressure across all services.
+- The static `RESPECT_CRAWL_MUTEX` only **defers a start** while a crawl runs —
+  it can't stop a vision that's **already running**, and it's blind to every
+  non-crawl memory consumer (CI build, backend spike, a second crawl worker).
+- The only backstop under true pressure today is the **kernel OOM killer**,
+  which may pick postgres or the backend (long-lived, stateful, painful) rather
+  than the one process that is cheap and safe to lose.
+
+A manager-driven stop is the friendly version: it picks the **right,
+restartable victim** and does it **gracefully (SIGTERM)** before the kernel does
+something violent.
+
+This is the **dynamic generalization of `RESPECT_CRAWL_MUTEX`**: "back off when
+RAM is actually tight" strictly dominates "never co-run with a crawl." If this
+ships, `RESPECT_CRAWL_MUTEX` can likely be retired.
+
+---
+
+## 2. Feasibility (confirmed)
+
+`/proc/meminfo` inside the manager container reports **host-wide** memory (it is
+not namespaced), so the manager already sees the truth with **zero new
+privilege** — no socket-proxy `docker info`, no host mount. Verified live from
+inside `vision-manager`:
+
+```
+MemTotal:     8256576 kB   (7.87 GiB)
+MemAvailable: 1754384 kB   → ~79% used, 21% available   (vision up + 560 queued)
+```
+
+**Use `MemAvailable`, never `MemFree`.** Linux spends "free" RAM on page cache by
+design, so a `MemFree`-based "used %" sits near 90 % during normal operation and
+would false-trigger constantly. `MemAvailable` already nets out reclaimable
+cache. The honest metric is:
+
+```
+used_pct = 100 * (1 - MemAvailable / MemTotal)
+```
+
+Read it in bash with `awk` over `/proc/meminfo` (both values are in kB):
+
+```sh
+read_mem_used_pct() {
+  awk '/^MemTotal:/{t=$2} /^MemAvailable:/{a=$2}
+       END { if (t>0) printf "%d", 100*(1-a/t); else print "ERR" }' /proc/meminfo
+}
+```
+
+---
+
+## 3. Behaviour
+
+A third gate in the existing poll loop, evaluated **before** the
+backlog/enable start/stop logic:
+
+1. **Stop gate (active).** If `used_pct >= HIGH_WATERMARK` (e.g. 92) **and**
+   vision is running → `docker stop mangalord-vision`, record a cooldown
+   deadline, log loudly. This is the new capability the crawl mutex never had.
+2. **Start inhibit.** While `used_pct >= LOW_WATERMARK` (e.g. 80) **or** a
+   cooldown is in effect → refuse to start vision even if backlog > 0 (same
+   shape as the crawl-mutex deferral: log once per episode, not every tick).
+3. **Resume.** Once `used_pct < LOW_WATERMARK` **and** the cooldown has elapsed,
+   normal autoscaler behaviour resumes (start on backlog, idle-stop, etc.).
+
+**Hysteresis is mandatory** (two distinct watermarks). A single threshold
+oscillates around the boundary.
+
+**The cooldown is mandatory** and is the subtle part. If vision *itself* is the
+4.4 GiB hog, stopping it drops pressure → manager sees backlog → restarts it →
+pressure climbs → stop … infinite thrash, each cycle paying the ~minutes cold
+reload. The cooldown (refuse-to-restart for N minutes after a pressure-stop,
+*regardless of backlog*) is what gives the **other** workload — the one that
+actually needed the RAM — room to finish before vision muscles back in. The
+yield only makes sense when something else needs the memory; the cooldown
+encodes that.
+
+### Gate ordering in the loop
+
+```
+loop every POLL_INTERVAL:
+    used = read_mem_used_pct()
+    pending = pending_analysis()
+    if not analysis_enabled():        pending = 0          # existing gate
+    if mem_yield_active(used):         pending = 0          # NEW: inhibit starts
+    if used >= HIGH_WATERMARK and vision running:           # NEW: active stop
+        stop_vision(); start_cooldown(); continue
+    ... existing start-on-backlog / idle-debounce-stop logic on `pending` ...
+```
+
+Note the symmetry with the analysis-disabled gate already in `manager.sh`:
+forcing `pending = 0` reuses the existing idle/stop path for the *inhibit* case,
+while the explicit `stop_vision()` handles the *active* case the idle path can't.
+
+---
+
+## 4. Proposed configuration (all env, matching manager.sh style)
+
+| Var | Default | Meaning |
+|---|---|---|
+| `MEM_YIELD_ENABLED` | `1` | Master switch for the whole feature. |
+| `MEM_HIGH_WATERMARK_PCT` | `92` | `used_pct ≥` this → active-stop a running vision. |
+| `MEM_LOW_WATERMARK_PCT` | `80` | `used_pct ≥` this → inhibit starts (no restart until below). |
+| `MEM_YIELD_COOLDOWN` | `300` | Seconds after a pressure-stop during which restart is refused regardless of backlog. |
+| `MEM_POLL_INTERVAL` | `POLL_INTERVAL` | Optional faster cadence for the mem check (pressure can spike between 20 s ticks). |
+
+Watermarks want tuning on the real box; 92/80 are starting points. Keep
+`HIGH − LOW ≥ ~10` so the band is wider than normal jitter.
+
+---
+
+## 5. Required verifications (do these BEFORE building)
+
+The queue is built for expendability — `crawler_jobs` is a lease queue:
+
+```
+state | attempts | max_attempts(=5) | leased_until | last_error | scheduled_at
+```
+
+A graceful `docker stop` (SIGTERM → clean llama-server exit) means at most the
+**one** in-flight `analyze_page` job is affected; the readiness gate
+(`ANALYSIS_VISION_HEALTH_URL`) then parks the worker so it won't hammer the dead
+endpoint or burn the other queued jobs. But two things must be confirmed in
+`backend/src/analysis/daemon.rs` (not visible from the DB schema alone):
+
+- [ ] **V1 — reclaim cadence.** The `bugfix/crawler-recovery-hardening` branch
+  added *reclaim-orphaned at startup*. Confirm expired leases are **also**
+  reclaimed on the normal claim poll (i.e. the claim query includes something
+  like `OR (state='running' AND leased_until < now())`). If reclaim is
+  **boot-only**, a pressure-kill strands that one job as `running` with a dead
+  lease until the next backend restart → the page silently never analyses.
+  **This decides how aggressive we can safely be.**
+- [ ] **V2 — error path requeue.** When the worker's HTTP call to vision is
+  refused (vision killed), confirm the job is returned to `pending`
+  (`attempts++`) and not marked permanently `failed` on a connection error.
+  A connection-refused/transport error should be *retryable*, distinct from a
+  model-returned bad response.
+- [ ] **V3 — park does not deadlock.** Confirm that when the readiness gate
+  parks the worker, it is **not holding an un-renewed lease** that blocks the
+  job indefinitely (it should either release on park or let the lease expire +
+  be reclaimed per V1).
+- [ ] **V4 — attempts accounting.** A pressure-kill is "our fault," yet still
+  burns one of the 5 attempts. Confirm repeated yields on the *same* in-flight
+  page can't quietly exhaust retries → permanent `failed`. Mitigations if it's a
+  concern: bump `max_attempts`, or rely on the admin re-enqueue (all/manga/
+  chapter) you already have to recover stragglers.
+
+**Empirical check** (once V1–V3 look right): with analysis on and one job
+`running`, `docker stop mangalord-vision`, then watch
+`SELECT state, attempts, last_error FROM crawler_jobs WHERE state IN
+('running','pending') AND payload->>'kind'='analyze_page'` — the killed job
+should reappear as `pending` with `attempts` incremented, and resume when vision
+restarts. This is the acceptance test for the whole feature.
+
+---
+
+## 6. Pitfalls
+
+- **`MemFree` vs `MemAvailable`** — using `MemFree` makes it fire constantly
+  (page cache). Already addressed by §2, but it's the #1 way to get this wrong.
+- **Thrash / feedback loop** — if vision is the hog, stop→restart oscillates.
+  The cooldown (§3) is the guard; without it the feature is worse than nothing.
+- **Spikes between ticks** — a Chromium burst can blow past 92 % inside a 20 s
+  poll gap and the kernel OOM-kills before the manager's next read. The valve
+  *reduces* OOM risk, it doesn't *eliminate* it. Consider a shorter
+  `MEM_POLL_INTERVAL` for just the mem read, and treat this as defence-in-depth
+  alongside (not instead of) `mem_limit` and swap/zram headroom.
+- **Measuring the wrong scope** — confirm `/proc/meminfo` in the container keeps
+  showing **host** totals under this kernel/cgroup setup (it does today:
+  MemTotal 8256576 kB == the Pi). If a future runtime namespaces it, the metric
+  silently becomes the container's tiny cgroup and the gate never/always fires.
+- **Cold-start cost** — every pressure-stop pays ~minutes to mmap + warm 3.2 GiB
+  on the next start. Don't set the watermarks so tight that normal operation
+  trips them; the backlog will stall during each reload.
+- **Interaction with the idle-stop and disable gate** — make sure the three
+  reasons vision can be down (idle-drained, analysis-disabled, memory-yield) are
+  **distinctly logged** (flip-only, like the existing gates) so a stop is never
+  misdiagnosed. A single muddy "stopped vision" line will cost debugging time.
+- **Double-stop / start races** — `docker stop` then an immediate backlog-driven
+  `docker start` in the same or next tick. The cooldown plus checking
+  `vision_running` before acting prevents this; don't drop those guards.
+- **OOM-killer still owns the tail** — `mem_limit: 6g` caps vision's own growth
+  but the host can still be pushed over by the *sum* of everything else. This
+  feature lowers the probability of a bad OOM kill; pairing it with a little
+  **swap or zram** gives the kernel a cushion to survive the gap between a spike
+  and the manager's reaction.
+
+---
+
+## 7. Recommendation
+
+Worth building, as a **third gate** in the existing `manager.sh` poll loop:
+`MemAvailable`-based metric, two watermarks (hysteresis), a restart cooldown,
+an **active stop** (the capability the crawl mutex lacks), and distinct
+flip-only logging. It supersedes `RESPECT_CRAWL_MUTEX` (which can then be
+removed).
+
+**Gate it on verification V1 first** — whether expired leases are reclaimed
+continuously or only at boot decides whether a yielded job cleanly retries or
+strands. If V1/V2 hold, this is a low-risk, high-value safety valve; if reclaim
+is boot-only, fix that in the backend before enabling memory-yield, otherwise
+each pressure-stop quietly drops a page until the next backend restart.
+
+Keep the manager dumb: this is ~30 lines of bash (one `awk` reader, two
+watermark comparisons, one cooldown timestamp) — no new dependencies, no new
+privilege, no web surface.

docker-compose.yml

@@ -168,6 +168,14 @@ services:
       START_HEALTH_TIMEOUT: ${VISION_START_HEALTH_TIMEOUT:-300}
       RESPECT_CRAWL_MUTEX: ${VISION_RESPECT_CRAWL_MUTEX:-1}
       MAX_UPTIME: ${VISION_MAX_UPTIME:-0}
+      # Memory-pressure yield — stop vision when the HOST is short on RAM so a
+      # spike elsewhere can finish without the kernel OOM-killer. See
+      # VISION-MEMORY-YIELD.md.
+      MEM_YIELD_ENABLED: ${VISION_MEM_YIELD_ENABLED:-1}
+      MEM_HIGH_WATERMARK_PCT: ${VISION_MEM_HIGH_WATERMARK_PCT:-92}
+      MEM_LOW_WATERMARK_PCT: ${VISION_MEM_LOW_WATERMARK_PCT:-80}
+      MEM_YIELD_COOLDOWN: ${VISION_MEM_YIELD_COOLDOWN:-300}
+      MEM_POLL_INTERVAL: ${VISION_MEM_POLL_INTERVAL:-5}
     networks:
       - default          # reach postgres + the vision container by name
       - vision-internal  # reach the socket-proxy

vision-manager/manager.sh

@@ -30,6 +30,17 @@ MAX_UPTIME="${MAX_UPTIME:-0}"               # >0: force-stop after N idle-or-not
 # together OOM the 8 GiB box). Set to 0 on roomier hosts.
 RESPECT_CRAWL_MUTEX="${RESPECT_CRAWL_MUTEX:-1}"
 
+# ---- Memory-pressure yield (see VISION-MEMORY-YIELD.md) --------------------
+# Dynamic generalization of the crawl mutex: stop a RUNNING vision when the
+# HOST is actually short on RAM, so a transient spike elsewhere (a crawl, CI,
+# a backend burst) can finish instead of the kernel OOM-killer shooting
+# something stateful. Uses host-wide MemAvailable from /proc/meminfo.
+MEM_YIELD_ENABLED="${MEM_YIELD_ENABLED:-1}"             # 0 disables the whole gate
+MEM_HIGH_WATERMARK_PCT="${MEM_HIGH_WATERMARK_PCT:-92}"  # used% >= → active-stop a running vision
+MEM_LOW_WATERMARK_PCT="${MEM_LOW_WATERMARK_PCT:-80}"    # used% >= → inhibit starts (hysteresis)
+MEM_YIELD_COOLDOWN="${MEM_YIELD_COOLDOWN:-300}"         # s after a pressure-stop before restart allowed
+MEM_POLL_INTERVAL="${MEM_POLL_INTERVAL:-5}"             # mem sub-poll cadence, <= POLL_INTERVAL
+
 PSQL=(psql "$DATABASE_URL" -At -v ON_ERROR_STOP=1)
 
 log() { echo "[vision-manager] $(date -u +%FT%TZ) $*"; }
@@ -53,6 +64,32 @@ crawl_running() {
         AND state = 'running';" 2>/dev/null || echo "ERR"
 }
 
+# Is the analysis subsystem enabled in Admin → Settings? The runtime source of
+# truth is the `app_settings` row (key='analysis'), NOT the .env seed. When
+# analysis is OFF the worker drains nothing, so leftover pending jobs would
+# otherwise keep vision pinned forever — we must gate on this too, not just the
+# queue depth. Returns the literal 'true'/'false'; 'ERR' (or anything else) on
+# query failure is treated as enabled by the caller (fail-open: never yank
+# vision out from under a live analysis on a transient DB hiccup).
+analysis_enabled() {
+  "${PSQL[@]}" -c \
+    "SELECT CASE WHEN COALESCE((value->>'enabled')::boolean, false)
+                 THEN 'true' ELSE 'false' END
+       FROM app_settings WHERE key = 'analysis';" 2>/dev/null || echo "ERR"
+}
+
+# ---- Memory pressure -------------------------------------------------------
+# Host-wide used% from MemAvailable (NOT MemFree, which sits near "full" thanks
+# to page cache and would false-trigger constantly). /proc/meminfo in this
+# container reports HOST totals (it is not namespaced under this kernel).
+# Emits the "ERR" sentinel on a malformed read so callers can ignore it.
+read_mem_used_pct() {
+  # Require BOTH fields: a partial read, or a pre-3.14 kernel without
+  # MemAvailable, would otherwise leave `a` at 0 → used=100% → a false stop.
+  awk '/^MemTotal:/{t=$2} /^MemAvailable:/{a=$2}
+       END { if (t>0 && a!="") printf "%d", 100*(1-a/t); else print "ERR" }' /proc/meminfo
+}
+
 # ---- Docker helpers --------------------------------------------------------
 # `docker inspect .State.Running` → true/false; empty if the container is
 # absent (defined elsewhere and not yet created) — treated as not-running.
@@ -84,7 +121,8 @@ start_vision() {
 }
 
 stop_vision() {
-  log "stopping $VISION_CONTAINER (idle ${STOP_DEBOUNCE}s)"
+  local reason="${1:-idle ${STOP_DEBOUNCE}s}"
+  log "stopping $VISION_CONTAINER ($reason)"
   if docker stop "$VISION_CONTAINER" >/dev/null 2>&1; then
     return 0
   fi
@@ -92,12 +130,44 @@ stop_vision() {
   return 1
 }
 
-# ---- Main loop -------------------------------------------------------------
+# ---- Memory-yield gates ----------------------------------------------------
-log "started: container=$VISION_CONTAINER health=$VISION_HEALTH_URL poll=${POLL_INTERVAL}s debounce=${STOP_DEBOUNCE}s docker_host=$DOCKER_HOST"
+# Active stop: when the host is over the HIGH watermark and vision is running,
+# SIGTERM it (the friendly, right-victim alternative to an OOM kill) and arm a
+# cooldown. Cheap on the hot path — only inspects docker once used% is already
+# over the mark. Reads `mem_yield_until`/`idle_for`/`up_for` from the loop scope.
+mem_check_and_stop() {
+  [ "$MEM_YIELD_ENABLED" = "0" ] && return 0
+  local used; used="$(read_mem_used_pct)"
+  case "$used" in ''|*[!0-9]*) return 0 ;; esac   # ignore a malformed read
+  if [ "$used" -ge "$MEM_HIGH_WATERMARK_PCT" ] && [ "$(vision_running)" = "true" ]; then
+    if stop_vision "memory-yield: used ${used}% >= ${MEM_HIGH_WATERMARK_PCT}% high watermark"; then
+      mem_yield_until=$(( $(date +%s) + MEM_YIELD_COOLDOWN ))
+      idle_for=0
+      up_for=0
+    fi
+  fi
+}
 
-idle_for=0       # seconds the queue has been empty while vision is running
+# Start inhibit: echo 1 while a cooldown is in effect OR used% is at/above the
-up_for=0         # seconds vision has been running (for MAX_UPTIME backstop)
+# LOW watermark. The two distinct watermarks give the mandatory hysteresis; the
-crawl_deferred=0 # 1 while a start is held off by the crawl mutex (log on flip only)
+# cooldown gives whatever needed the RAM room to finish before vision restarts.
+mem_yield_active() {
+  [ "$MEM_YIELD_ENABLED" = "0" ] && { echo 0; return; }
+  [ "$(date +%s)" -lt "$mem_yield_until" ] && { echo 1; return; }
+  local used; used="$(read_mem_used_pct)"
+  case "$used" in ''|*[!0-9]*) echo 0; return ;; esac
+  if [ "$used" -ge "$MEM_LOW_WATERMARK_PCT" ]; then echo 1; else echo 0; fi
+}
+
+# ---- Main loop -------------------------------------------------------------
+log "started: container=$VISION_CONTAINER health=$VISION_HEALTH_URL poll=${POLL_INTERVAL}s debounce=${STOP_DEBOUNCE}s docker_host=$DOCKER_HOST mem_yield=$([ "$MEM_YIELD_ENABLED" = 0 ] && echo off || echo "${MEM_LOW_WATERMARK_PCT}/${MEM_HIGH_WATERMARK_PCT}% cooldown=${MEM_YIELD_COOLDOWN}s mempoll=${MEM_POLL_INTERVAL}s")"
+
+idle_for=0        # seconds the queue has been empty while vision is running
+up_for=0          # seconds vision has been running (for MAX_UPTIME backstop)
+crawl_deferred=0  # 1 while a start is held off by the crawl mutex (log on flip only)
+analysis_off=0    # 1 while analysis is disabled in settings (log on flip only)
+mem_yield_until=0 # epoch deadline of the active memory-yield cooldown (0 = none)
+mem_inhibited=0   # 1 while starts are inhibited by memory pressure (log on flip only)
 
 while true; do
   pending="$(pending_analysis)"
@@ -114,6 +184,38 @@ while true; do
       ;;
   esac
 
+  # Gate on the runtime enable flag. If analysis is OFF, force the backlog to 0
+  # so the normal idle path runs the STOP_DEBOUNCE timer and shuts vision down.
+  # We deliberately reuse the idle debounce (rather than stopping instantly) so
+  # a quick toggle off→on inside the debounce window keeps vision warm — no
+  # cold reload for a momentary flip. Fail-open: only a literal 'false' counts
+  # as disabled; 'ERR'/empty leaves the real queue depth in effect.
+  if [ "$(analysis_enabled)" = "false" ]; then
+    if [ "$analysis_off" != "1" ]; then
+      log "analysis disabled in settings — ignoring ${pending} queued job(s); idle timer will stop vision"
+      analysis_off=1
+    fi
+    pending=0
+  else
+    analysis_off=0
+  fi
+
+  # Memory-yield gates (before the backlog logic). The active stop handles a
+  # running vision the idle path can't; the inhibit forces pending=0 so the
+  # existing idle-debounce path keeps it down (same shape as the crawl mutex).
+  mem_check_and_stop
+  running="$(vision_running)"   # refresh: the active stop may have just flipped it
+  if [ "$(mem_yield_active)" = "1" ]; then
+    if [ "$mem_inhibited" != "1" ]; then
+      log "memory-yield: inhibiting vision start (used >= ${MEM_LOW_WATERMARK_PCT}% or cooldown active)"
+      mem_inhibited=1
+    fi
+    pending=0
+  elif [ "$mem_inhibited" = "1" ]; then
+    log "memory-yield: pressure cleared — start inhibit lifted"
+    mem_inhibited=0
+  fi
+
   if [ "$running" = "true" ]; then
     up_for=$((up_for + POLL_INTERVAL))
   else
@@ -160,12 +262,24 @@ while true; do
 
   # Backstop: bound how long vision can stay up regardless of debounce state.
   if [ "$MAX_UPTIME" -gt 0 ] && [ "$running" = "true" ] && [ "$up_for" -ge "$MAX_UPTIME" ]; then
-    log "MAX_UPTIME ${MAX_UPTIME}s reached; force-stopping $VISION_CONTAINER"
+    if stop_vision "MAX_UPTIME ${MAX_UPTIME}s reached"; then
-    if stop_vision; then
       idle_for=0
       up_for=0
     fi
   fi
 
-  sleep "$POLL_INTERVAL"
+  # Sleep until the next backlog poll, but re-check memory pressure on the
+  # faster MEM_POLL_INTERVAL sub-cadence so a spike between backlog ticks can
+  # trigger an active stop before the kernel OOM-killer does. Degenerates to a
+  # single sleep when MEM_POLL_INTERVAL >= POLL_INTERVAL.
+  slept=0
+  while [ "$slept" -lt "$POLL_INTERVAL" ]; do
+    step="$MEM_POLL_INTERVAL"
+    remaining=$((POLL_INTERVAL - slept))
+    [ "$step" -gt "$remaining" ] && step="$remaining"
+    [ "$step" -lt 1 ] && step=1
+    sleep "$step"
+    slept=$((slept + step))
+    mem_check_and_stop
+  done
 done

vision-manager/readonly-role.sql

@@ -26,7 +26,11 @@ SELECT 'CREATE ROLE vision_manager LOGIN'
 -- (Re)set the password every run so rotating it is just a re-apply.
 ALTER ROLE vision_manager LOGIN PASSWORD :'pw';
 
--- CONNECT to the current database, and read-only on the one table we poll.
+-- CONNECT to the current database, and read-only on the two tables we poll:
+-- crawler_jobs (backlog depth + crawl-in-flight) and app_settings (the runtime
+-- analysis-enabled flag — without this the manager can't tell that analysis was
+-- turned off and would keep vision pinned on stale queued jobs).
 GRANT CONNECT ON DATABASE :"DBNAME" TO vision_manager;
 GRANT USAGE ON SCHEMA public TO vision_manager;
 GRANT SELECT ON crawler_jobs TO vision_manager;
+GRANT SELECT ON app_settings TO vision_manager;