diff --git a/docs/re/METHOD.md b/docs/re/METHOD.md index 3b313f33..2f576ffc 100644 --- a/docs/re/METHOD.md +++ b/docs/re/METHOD.md @@ -2776,3 +2776,27 @@ failure was process rather than analysis: ๐Ÿ“Œ **An uncommitted tool is one timed-out command away from being unrecoverable, and a tool edited during its own run is not the tool that ran.** Commit first, then run. Cheap, and it would have prevented all four. + +## A harness self-test โ€” and the hole it found in mine on the first run + +`sylpheed-port` closed this gap first, and named it precisely: their controls +asserted **failure-on-perturbation**, but nothing asserted that a **broken harness +reports broken**. Their test feeds the machinery a stub that cannot fail โ€” prints +"everything is fine", asserts nothing โ€” and requires it to be flagged. + +โœ… **Built the equivalent for `check_refuted.py`, and it found a real hole on its +first run**: a register that parses **no claims** reported **clean**, forever. That +is the same shape as their stub, sitting in the tool that guards the shared +register and whose clean runs both agents lean on. Fixed โ€” it now refuses rather +than passing. + +๐Ÿ“Œ Two details worth copying. **The self-test drives the real machinery as a +subprocess and reads its actual exit code**; their first version *reasoned* that +the control would flag the stub โ€” the error this whole thread is about, committed +inside the tool built to prevent it. And **their exit convention separates the two +failures that matter**: `0` fine ยท `1` a real check failed ยท `2` the harness is +broken and nothing it reported can be trusted. A single non-zero cannot say which. + +โš ๏ธ **Filed, not fixed**: `impossibility_scope.py`, `bgm_stem_coherence.py` and +`ring_row.py` have controls and **no harness self-test**. Shape known, fix cheap, +not done โ€” recorded so their absence is a stated gap rather than scenery.