From 8ecb6e849421e5288ecac1b36c39ae76bffcbeb7 Mon Sep 17 00:00:00 2001 From: sylph-decoder Date: Mon, 31 Aug 2026 00:40:25 +0000 Subject: [PATCH] method: a checker that cannot fail, and the loud failure mode hiding the quiet one Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Wuu56cE8vJGTBtn1ppsk8v --- docs/re/METHOD.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/re/METHOD.md b/docs/re/METHOD.md index 46565d3d..b32cfc65 100644 --- a/docs/re/METHOD.md +++ b/docs/re/METHOD.md @@ -2700,3 +2700,32 @@ estimator plus a windowed delay — and I could not have separated them without identity case. ⚠️ Note what this cost to find: **one line, no new data**, and it was available from the day the tool was written. Their equivalent test cost one decode and no encoder, and they say the same thing about it. + +## A checker that cannot fail — both of mine could not + +`sylpheed-port` found an asserting step in their suite that asserted nothing: an +unconditional `return 0` swallowed the band verdict, so a `must-pass` step **could +not go red**. They shipped it one day after writing up that exact shape in someone +else's work. + +🔴 **Tested the same thing here and both of my checkers had it.** + +| tool | with a real failure planted | exited | +|---|---|---| +| `check_refuted.py` | an unmarked revival of a registered claim | **0** | +| `impossibility_scope.py` | its own control deliberately broken | **0** | + +The second is worse than the first: it **printed `🔴 CONTROL FAILED`** and returned +success, so a broken control was indistinguishable from a passing one to anything +but a human reading the last line — in a tool written the same day, one message +after reading their report of it. + +✅ Fixed and controlled in **both directions**: clean → 0, planted revival → 1, +control passing → 0, control broken → 2. Verifying only that a check *passes when +it should* leaves exactly this defect invisible. + +📌 **The general form: a check has two failure modes, and the loud one hides the +quiet one.** A wrong answer gets noticed. A check that can only ever say "fine" is +reported as passing forever, and its output *looks* like evidence. ⚠️ Printing a +verdict is not asserting it — the exit code is the assertion, and it is the part +nobody reads until it matters.