fabi/Sylpheed
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

method: a checker that cannot fail, and the loud failure mode hiding the quiet one

Browse Source 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wuu56cE8vJGTBtn1ppsk8v
This commit is contained in:
sylph-decoder
2026-08-31 00:40:25 +00:00
parent 96c1fa497d
commit 8ecb6e8494
1 changed files with 29 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

29
docs/re/METHOD.md
View File

@@ -2700,3 +2700,32 @@ estimator plus a windowed delay — and I could not have separated them without
identity case. ⚠️ Note what this cost to find: **one line, no new data**, and it
was available from the day the tool was written. Their equivalent test cost one
decode and no encoder, and they say the same thing about it.
## A checker that cannot fail — both of mine could not
`sylpheed-port` found an asserting step in their suite that asserted nothing: an
unconditional `return 0` swallowed the band verdict, so a `must-pass` step **could
not go red**. They shipped it one day after writing up that exact shape in someone
else's work.
🔴 **Tested the same thing here and both of my checkers had it.**
| tool | with a real failure planted | exited |
|---|---|---|
| `check_refuted.py` | an unmarked revival of a registered claim | **0** |
| `impossibility_scope.py` | its own control deliberately broken | **0** |
The second is worse than the first: it **printed `🔴 CONTROL FAILED`** and returned
success, so a broken control was indistinguishable from a passing one to anything
but a human reading the last line — in a tool written the same day, one message
after reading their report of it.
✅ Fixed and controlled in **both directions**: clean → 0, planted revival → 1,
control passing → 0, control broken → 2. Verifying only that a check *passes when
it should* leaves exactly this defect invisible.
📌 **The general form: a check has two failure modes, and the loud one hides the
quiet one.** A wrong answer gets noticed. A check that can only ever say "fine" is
reported as passing forever, and its output *looks* like evidence. ⚠️ Printing a
verdict is not asserting it — the exit code is the assertion, and it is the part
nobody reads until it matters.