From d1a16336199c823035b4222dbf7ad9462a424890 Mon Sep 17 00:00:00 2001 From: Sylpheed port agent Date: Mon, 31 Aug 2026 01:33:42 +0000 Subject: [PATCH] port: assert the scan boundary I had hand-verified, and give audit-kinds a self-test check-claims --control plants a revival in docs/port/ and requires exit 1. That the plant lands INSIDE a scanned directory was a property I checked manually, one time, and wrote up -- the exact pattern I had criticised in this same tool one iteration earlier. A fifth case now plants the identical text OUTSIDE the scanned root and requires 0, so the pair asserts the boundary is real: same text, 1 inside and 0 outside. Either half alone is consistent with the tool scanning everything, or nothing. Five cases: clean 0, unmarked 1, marked 0, outside-root 0, empty register 2. audit-kinds has always reported what it found and was never asked whether it can find anything, while its clean runs are cited as evidence that fifteen labels are grounded. --selftest pushes three synthetic rows through the real classifier and reads its verdict: citing nothing must read BARE, a real path ok, a missing path DANGLING. Verified two-directionally -- an extractor stubbed to accept everything returns exit 2. Asserting in check-all. All four submenus are now measured to reset -- LOAD GAME, TUTORIAL and OPTIONS joining EXTRAS -- and the main menu remains the only screen that remembers. Three of the four are not in this export, so no authored value changes. NOT promoted to a rule, deliberately. 'Submenus reset' at 4/4 is better evidence than the 2/2 that made wrap a menu-wide rule, and adopting it would change nothing today because the only submenu this port ships is already measured. What it would do is pre-decide the next screen from a generalisation instead of a measurement -- the trap that nearly let a derived rule overwrite EXTRAS' measured opening item. The guard prints the 4/4 finding beside its per-screen values so the evidence is visible without being load-bearing. MISSION-SELECT-versus-top-item stays open: none of the three separates it, each opens on its own first item, and NEW GAME is untested. Remaining without a harness self-test: verify-transcode-fidelity. Every asserting check passes, 13 of them. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01N7FiFFFwbvG2uxdcEh8HyF --- docs/port/BLOCKED.md | 2 +- docs/port/DECISIONS.md | 63 ++++++++++++++++++++++++++++++++++++++- tools/port/audit-kinds | 50 +++++++++++++++++++++++++++++++ tools/port/check-all | 3 ++ tools/port/check-claims | 17 +++++++++++ tools/port/contract-check | 14 ++++++++- 6 files changed, 146 insertions(+), 3 deletions(-) diff --git a/docs/port/BLOCKED.md b/docs/port/BLOCKED.md index 732a8988..c2aca35c 100644 --- a/docs/port/BLOCKED.md +++ b/docs/port/BLOCKED.md @@ -151,7 +151,7 @@ HANDOFF. | Milestone | Needs | HANDOFF | State | |---|---|---|---| -| all — control harnesses that assert themselves | **nothing from anybody; TWO tools still lack it** | `d38adcf` | 🟡 **DONE FOR `contract-check` AND `check-claims`, not for the rest.** `check-claims --control` now executes four cases as subprocesses — clean 0, unmarked revival 1, marked revival 0, **empty register 2** — where before it had **no control machinery at all** and an empty register reported clean forever. Verified two-directionally: pointing the plant at an unscanned path makes the control report itself broken. Remaining: `audit-kinds`, `verify-transcode-fidelity`. Earlier text: 🟡 **DONE FOR `contract-check`, NOT for the rest.** `--selftest` feeds the machinery a stub that cannot fail and requires it to be flagged; exit codes separate **0** all good / **1** a real check failed / **2** the harness is broken. Asserting in `check-all`. ⚠️ `check-claims`, `audit-kinds` and `verify-transcode-fidelity` have controls and **no harness self-test** — the shape is known and the fix is cheap, and this row exists so the gap does not read as finished. 🔴 The self-test caught two defects while being written: a first version that *argued* the harness would flag the stub instead of measuring it, and a `src` selection that anchored anything outside one list at the wrong document, flagging the stub for a fabricated reason. | +| all — control harnesses that assert themselves | **nothing from anybody; ONE tool still lacks it** | `d38adcf` | 🟡 **DONE FOR `contract-check`, `check-claims` AND `audit-kinds`.** `audit-kinds --selftest` pushes three synthetic rows through the real classifier — citing nothing must read BARE, a real path ok, a missing path DANGLING — and returns **2** when stubbed to accept everything. `check-claims --control` gained a **fifth case**: the identical plant text *outside* the scanned root must give 0, so the boundary is asserted rather than hand-verified once. Remaining: `verify-transcode-fidelity`. Earlier text: 🟡 **DONE FOR `contract-check` AND `check-claims`, not for the rest.** `check-claims --control` now executes four cases as subprocesses — clean 0, unmarked revival 1, marked revival 0, **empty register 2** — where before it had **no control machinery at all** and an empty register reported clean forever. Verified two-directionally: pointing the plant at an unscanned path makes the control report itself broken. Remaining: `audit-kinds`, `verify-transcode-fidelity`. Earlier text: 🟡 **DONE FOR `contract-check`, NOT for the rest.** `--selftest` feeds the machinery a stub that cannot fail and requires it to be flagged; exit codes separate **0** all good / **1** a real check failed / **2** the harness is broken. Asserting in `check-all`. ⚠️ `check-claims`, `audit-kinds` and `verify-transcode-fidelity` have controls and **no harness self-test** — the shape is known and the fix is cheap, and this row exists so the gap does not read as finished. 🔴 The self-test caught two defects while being written: a first version that *argued* the harness would flag the stub instead of measuring it, and a `src` selection that anchored anything outside one list at the wrong document, flagging the stub for a fabricated reason. | ## Coverage hole in my own check, 2026-08-31 — derived from HANDOFF `0159527` diff --git a/docs/port/DECISIONS.md b/docs/port/DECISIONS.md index ea4a48d9..a3713c11 100644 --- a/docs/port/DECISIONS.md +++ b/docs/port/DECISIONS.md @@ -9,7 +9,7 @@ dies, which is what this file is for. -267 sections. Search this before re-deriving anything. +269 sections. Search this before re-deriving anything. * [P0 — the exporter, 2026-08-28](#p0--the-exporter-2026-08-28) * [P1 — Godot draws the screen, 2026-08-28](#p1--godot-draws-the-screen-2026-08-28) @@ -278,6 +278,8 @@ dies, which is what this file is for. * [Their two tools had the shape I shipped, and the general form is sharper now](#their-two-tools-had-the-shape-i-shipped-and-the-general-form-is-sharper-now) * [Closing the two-directional gap: the control harness now asserts itself](#closing-the-two-directional-gap-the-control-harness-now-asserts-itself) * [The register check had no executable control, and an empty register passed forever](#the-register-check-had-no-executable-control-and-an-empty-register-passed-forever) +* [Two harness gaps closed, and one of them was mine done by hand](#two-harness-gaps-closed-and-one-of-them-was-mine-done-by-hand) +* [All four submenus reset, and I am not promoting it to a rule](#all-four-submenus-reset-and-i-am-not-promoting-it-to-a-rule) ## P0 — the exporter, 2026-08-28 @@ -13683,3 +13685,62 @@ two-directional assertion — it can fail, and it fails for the right reason. ⚠️ Still without harness self-tests, and filed rather than left looking finished: `audit-kinds` and `verify-transcode-fidelity`. Same shape, cheap, not done. + +## Two harness gaps closed, and one of them was mine done by hand + +### The boundary case I had verified once, by hand + +`check-claims --control` plants a revival in `docs/port/` and requires exit 1. +That the plant lands **inside a scanned directory** was a property I checked +manually, one time, and wrote up — **the exact pattern I had criticised in this +same tool one iteration earlier.** + +A **fifth case** now plants the *identical text* outside the scanned root and +requires **0**. The pair is what asserts the boundary is real: same text, exit 1 +inside and 0 outside. **Either half alone is consistent with the tool scanning +everything, or nothing.** Five cases: clean 0, unmarked 1, marked 0, outside-root +0, empty register 2. + +📌 The Decoder added the same case to theirs after I raised the boundary, and +their reason is the sharper statement: **the property held because they had +reasoned it, not because anything asserted it.** Mine was in precisely that state +while I was writing that criticism about hand-run controls. + +### `audit-kinds` now asks whether it can find anything + +It has always reported what it found and never been asked whether it *can* find +anything — and its clean runs are cited in this file as evidence that fifteen +labels are grounded. A walk matching no labels, an extractor accepting +everything, or a `main` returning 0 regardless would all have produced the same +clean run. + +`--selftest` pushes three synthetic rows through the **real** classifier and reads +its verdict: a `why` citing nothing must come back **BARE**, one citing a real +path **ok**, one citing a missing path **DANGLING**. Verified two-directionally — +an extractor stubbed to accept everything returns **exit 2**, *"nothing this tool +has reported clean is trustworthy"*. Asserting in `check-all`. + +## All four submenus reset, and I am not promoting it to a rule + +Measured: **LOAD GAME, TUTORIAL and OPTIONS reset**, joining EXTRAS. **Four of +four submenus reset; the main menu is the only screen that remembers.** Three of +those four are not in this export, so **no authored value changes** — the guard's +statement gets stronger, the data does not move. + +🔴 **Not promoted to a rule, deliberately.** *"Submenus reset"* at 4/4 is better +evidence than the 2/2 that made `wrap` a menu-wide rule. Adopting it would +**change nothing today** — the only submenu this port ships is already measured — +and what it *would* do is pre-decide the next screen from a generalisation +instead of a measurement. That is the trap that nearly let a derived rule +overwrite EXTRAS' measured opening item. The guard prints the 4/4 finding beside +its per-screen values so the evidence is visible without being load-bearing. + +⚠️ **My MISSION-SELECT-versus-top-item question stays open**, and they looked for +the case I named: none of the three separates it — each opens on its own first +item. `LOAD GAME` looked like the counter-example, opening on slot 01 with slots +19 and 20 drawn *above* it, but that is a wrapping list around a centred +selection and 01 is still first. **NEW GAME is untested.** + +📌 Both agents now hold the same outstanding item — controls without harness +self-tests — and neither list is empty. Mine is down to +`verify-transcode-fidelity`. diff --git a/tools/port/audit-kinds b/tools/port/audit-kinds index 89f3e5c1..fa4fbca8 100755 --- a/tools/port/audit-kinds +++ b/tools/port/audit-kinds @@ -118,7 +118,57 @@ def walk(o, f, path, out): walk(x, f, path, out) +def selftest(): + """Does this audit notice a label that rests on nothing? + + 🔴 THE GAP: `audit-kinds` has always reported what it found and never been + asked whether it can find anything. A walk that matched no labels, a citation + extractor that accepted everything, or a `main` that returned 0 regardless + would all have produced the same clean run -- and clean runs from this tool + are cited in `DECISIONS.md` as evidence that fifteen labels are grounded. + + Three synthetic rows are pushed through the REAL classifier, and its verdict + is read rather than reasoned about: + + a `why` citing nothing -> must be BARE + a `why` citing a path that exists -> must be ok + a `why` citing a path that does not -> must be DANGLING + + Exit codes follow the convention the Decoder and I converged on: 0 all good, + 1 a real audit failure, **2 the harness is broken** and no clean run from it + means anything. + """ + paths = known_paths() + cases = [ + ("bare", "no citation of any kind here, just prose", "BARE"), + ("ok", "see tools/port/audit-kinds for the method", "ok"), + ("dangling", "see docs/port/NO-SUCH-FILE-XYZ.md", "DANGLING"), + ] + bad = 0 + for name, why, want in cases: + cites = citations(why) + if not cites: + got = "BARE" + else: + unresolved = [c for t, c in cites + if t == "path" and c not in paths and not os.path.exists(c)] + got = "DANGLING" if unresolved else "ok" + mark = "✅" if got == want else "🔴" + print(f" harness: a why that is {name:<9} -> {got:<8} (want {want:<8}) {mark}") + if got != want: + bad += 1 + print() + if bad: + print("🔴 the classifier cannot tell grounded labels from ungrounded ones.") + print(" Exit 2: nothing this tool has reported clean is trustworthy.") + return 2 + print("the classifier separates bare, dangling and grounded citations") + return 0 + + def main(): + if "--selftest" in sys.argv: + return selftest() rows = [] for f in sorted(glob.glob("authored/*.json")): walk(json.load(open(f)), f, "", rows) diff --git a/tools/port/check-all b/tools/port/check-all index d34915b8..a792b271 100755 --- a/tools/port/check-all +++ b/tools/port/check-all @@ -54,6 +54,9 @@ step modding-rules must-pass tools/port/check-modding # 2026-08-30 nothing checked what any of them rested on -- seven were resting on # a sibling `why` that argued a different claim. step authored-kinds must-pass tools/port/audit-kinds +# The classifier is asked whether it can tell grounded from ungrounded at all, +# rather than only what it found. Exit 2 = the harness is broken. +step kinds-harness must-pass tools/port/audit-kinds --selftest # Band levels are alignment-free and carry their own known negative on every run; # the difference-signal half of the same tool stays report-only and asserts # nothing. See docs/port/DECISIONS.md -- the waveform question is still open. diff --git a/tools/port/check-claims b/tools/port/check-claims index fe54d6eb..e3bb93b0 100755 --- a/tools/port/check-claims +++ b/tools/port/check-claims @@ -110,6 +110,23 @@ if [ "${1:-}" = "--control" ]; then fi done rm -f "$probe" + # 🔴 FIFTH CASE: the same text OUTSIDE the scanned root must give 0. + # + # Without it, "the plant is inside a scanned directory" is a property I + # verified BY HAND, once -- which is the exact pattern I had just finished + # criticising in this tool one iteration earlier. The pair is what asserts the + # boundary is real: identical text, exit 1 inside and 0 outside. Either half + # alone is consistent with the tool scanning everything, or nothing. + # + # The Decoder added this to theirs after I raised the boundary; the reason it + # was worth adding is that their property held *because they had reasoned it*, + # not because anything asserted it. Mine was in the same state. + outside="${TMPDIR:-/tmp}/claims-control-outside.md" + printf '%s\n' "$claim" > "$outside" + got=$(run) + rm -f "$outside" + if [ "$got" = "0" ]; then printf ' %-26s exit 0 ✅\n' "same text outside root" + else printf ' %-26s exit %s, wanted 0 🔴\n' "same text outside root" "$got"; ok=1; fi got=$(CLAIMS_REGISTER="" "$0" >/dev/null 2>&1; echo $?) if [ "$got" = "2" ]; then printf ' %-26s exit 2 ✅\n' "empty register" else printf ' %-26s exit %s, wanted 2 🔴\n' "empty register" "$got"; ok=1; fi diff --git a/tools/port/contract-check b/tools/port/contract-check index 615bd2f4..d1c5e5ce 100755 --- a/tools/port/contract-check +++ b/tools/port/contract-check @@ -398,8 +398,20 @@ def guard_focus_scope(_h): if isinstance(v, dict) and "focus_persists" not in v) ok = stated == {"main_menu": True, "extras": False} if ok: + # ✅ 2026-08-31: all FOUR submenus are now measured to reset -- EXTRAS, + # LOAD GAME, TUTORIAL and OPTIONS -- and the main menu remains the only + # screen that remembers. Three of those four are not in this export, so + # no authored value changes. + # + # 🔴 NOT PROMOTED TO A RULE, deliberately. "Submenus reset" at 4/4 is + # better evidence than the 2/2 that made `wrap` a rule -- and adopting it + # would change nothing today, because the only submenu this port ships is + # already measured. What it WOULD do is pre-decide the next screen from a + # generalisation instead of a measurement, which is the trap that nearly + # let a derived rule overwrite EXTRAS' measured opening item. print(f" {'focus_persists scope':<30} guard {stated} measured;" - f" {len(silent)} screen(s) silent = UNMEASURED, not 'resets'") + f" {len(silent)} screen(s) silent = UNMEASURED, not 'resets'" + f" [4/4 submenus reset disc-wide; not promoted to a rule]") else: FAIL += 1 print(f" {'focus_persists scope':<30} 🔴 GUARD {stated} -- a screen states"