re: build a shared probe harness so the same lessons stop being re-learned

Four probes were written from a blank file and each re-learned the same lessons
by losing a run: that a flat run cannot be told from a frozen guest without a
stall witness, that results held to the end of a run are destroyed by a turn
timeout, that a roster count which is not the stage's member count means a
different stage loaded and must be discarded, and that a run's witness state has
to be read before its numbers. Writing each lesson down did not stop the next
probe repeating it, because each probe started from nothing.

probeharness.py makes them structural. Probe(baseline=N) discovers the roster,
rescans up to five times and refuses to start if the count never reaches the
baseline. The witness is calibrated on construction, sampled by tick() and
reported by status() and summary(), so a probe cannot forget it, and when no
witness is found it reports UNVALIDATED rather than zero stalls. emit() flushes
on every line. craft(), strengths(), alive() and heap() supply the
roster-to-craft link, per-record liveness and the raw heap, so a new probe
writes only its own logic.

Verified rather than asserted: deploy_probe.py reimplements the per-record
deployment watch on top of it in about forty lines against wave7_probe's
hundred and fifty, and its first live run was clean -- 116 roster records, 32
witnesses at 10/s, zero stalled samples, seven losses tracked, and the TSV
written incrementally. Nothing about the result is new, which is the point: the
harness reproduces a known-good measurement.

The existing probes are deliberately not ported. They work, and rewriting them
would risk changing results other documents cite. New probes should use the
harness; old ones should be ported when they next need a change.
This commit is contained in:
Sylpheed RE agent
2026-08-25 05:22:24 +00:00
parent a040cad0da
commit 3355ce9198
5 changed files with 347 additions and 0 deletions

View File

@@ -630,6 +630,18 @@ search cannot find a *schedule*.
that cannot be told from a freeze**; the recurring fix is the shared harness
noted earlier. Needs two HUD readings at *different* values in non-stalled
samples.
***(2026-08-25) SHARED PROBE HARNESS built and verified**
([`probe-harness.md`](probe-harness.md), `tools/re-capture/probeharness.py`).
Makes structural the four lessons that were re-learned in four separate probes:
**built-in stall witness** (says `UNVALIDATED` when absent rather than reporting
zero stalls), **`emit()` flushes every line** so a timeout cannot destroy
results, **baseline discard with rescan** (`Probe(baseline=116)` refuses to
start on a different stage), and `summary()` putting the witness first.
Also provides `craft()`/`strengths()`/`alive()`/`heap()` so a new probe writes
only its own logic. ✅ Verified: `deploy_probe.py` reimplements the deployment
watch in ~40 lines vs 150, first live run clean — 116 roster, 32 witnesses at
10/s, **0 stalled**, 7 losses, TSV written incrementally. ⚠️ Existing probes
deliberately **not** ported — they work and other docs cite their results.
* ~~🚧 BLOCKER: t=210/240 unreachable in one turn~~ — **superseded, see above**;
it rested on an untested assumption that a turn is one shell call. 595 s shell cap ~220 s boot (a ~190 s title movie that cannot be
tapped through) ~25 s startup = **~350 s observation ≈ 193 game-seconds**.

61
docs/re/probe-harness.md Normal file
View File

@@ -0,0 +1,61 @@
# A shared probe harness — so the same lessons stop being re-learned
Status: ✅ built and verified on a live run.
## Why
Four separate probes were written from a blank file, and each re-learned the same
lessons by losing a run:
| lesson | where it was learned | where it was re-learned |
|---|---|---|
| a flat run is indistinguishable from a frozen guest without a **stall witness** | `guest-stalls.md` | 3 further probes |
| **save incrementally** — a turn timeout destroys anything held to the end | `guest-stalls.md` | `ob_probe2.py`, four iterations later |
| a roster count that is not the stage's member count is **a different stage** — discard | `mission-per-record-strength.md` | — |
| **read the witness first**, before interpreting any number | `remaining-ob-hunt.md` | — |
Writing each lesson down did not stop the next probe from repeating it, because
each probe started from nothing. `tools/re-capture/probeharness.py` makes them
structural instead.
## What it provides
```python
p = Probe(baseline=116) # None = accept whatever loads
if not p.ok: sys.exit(p.why) # e.g. "DISCARD: roster settled at 42"
print(p.summary()) # witness count + rate, read this FIRST
p.log('t\tvalue') # opens a TSV, flushed on every emit
while p.tick(every=15, secs=300):
p.emit(...) # written and flushed immediately
print(p.status()) # '' | '*** GUEST STALLED ***' | UNVALIDATED
```
* `Probe(baseline=…)` discovers the roster, **rescans up to five times**, and
refuses to start if the count never reaches the baseline.
* The **witness is built in**: calibrated on construction, sampled by `tick()`,
reported by `status()` and `summary()`. A probe cannot forget it, and when no
witness is found it says `UNVALIDATED` rather than reporting zero stalls.
* `emit()` **flushes on every line** — results cannot be lost to a timeout.
* `craft()`, `strengths()`, `alive()` and `heap()` provide the roster→craft link,
per-record liveness and the raw heap, so a new probe writes only its own logic.
## ✅ Verified
`deploy_probe.py` reimplements the per-record deployment watch on top of it, in
about 40 lines against `wave7_probe.py`'s 150. First live run:
```
roster 116, definitions 14, 32 witnesses at 10/s, stalled samples 0
t= 249s deployed=40 up=0 down=1 (cum 0/7)
```
Baseline check passed, witness found and reporting, seven losses tracked,
`/tmp/deploy.tsv` written incrementally. Nothing about the result is new — that
is the point; the harness reproduces a known-good measurement.
## Not done
The existing probes (`wave6/7`, `ob_probe2`, `ob_by_hud`, `census`, `liveness`)
are **not** ported. They work, and rewriting them risks changing results that
other documents cite. New probes should use the harness; old ones should be
ported only when they next need a change.