port: assert the scan boundary I had hand-verified, and give audit-kinds a self-test
check-claims --control plants a revival in docs/port/ and requires exit 1. That the plant lands INSIDE a scanned directory was a property I checked manually, one time, and wrote up -- the exact pattern I had criticised in this same tool one iteration earlier. A fifth case now plants the identical text OUTSIDE the scanned root and requires 0, so the pair asserts the boundary is real: same text, 1 inside and 0 outside. Either half alone is consistent with the tool scanning everything, or nothing. Five cases: clean 0, unmarked 1, marked 0, outside-root 0, empty register 2. audit-kinds has always reported what it found and was never asked whether it can find anything, while its clean runs are cited as evidence that fifteen labels are grounded. --selftest pushes three synthetic rows through the real classifier and reads its verdict: citing nothing must read BARE, a real path ok, a missing path DANGLING. Verified two-directionally -- an extractor stubbed to accept everything returns exit 2. Asserting in check-all. All four submenus are now measured to reset -- LOAD GAME, TUTORIAL and OPTIONS joining EXTRAS -- and the main menu remains the only screen that remembers. Three of the four are not in this export, so no authored value changes. NOT promoted to a rule, deliberately. 'Submenus reset' at 4/4 is better evidence than the 2/2 that made wrap a menu-wide rule, and adopting it would change nothing today because the only submenu this port ships is already measured. What it would do is pre-decide the next screen from a generalisation instead of a measurement -- the trap that nearly let a derived rule overwrite EXTRAS' measured opening item. The guard prints the 4/4 finding beside its per-screen values so the evidence is visible without being load-bearing. MISSION-SELECT-versus-top-item stays open: none of the three separates it, each opens on its own first item, and NEW GAME is untested. Remaining without a harness self-test: verify-transcode-fidelity. Every asserting check passes, 13 of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N7FiFFFwbvG2uxdcEh8HyF
This commit is contained in:
@@ -118,7 +118,57 @@ def walk(o, f, path, out):
|
||||
walk(x, f, path, out)
|
||||
|
||||
|
||||
def selftest():
|
||||
"""Does this audit notice a label that rests on nothing?
|
||||
|
||||
🔴 THE GAP: `audit-kinds` has always reported what it found and never been
|
||||
asked whether it can find anything. A walk that matched no labels, a citation
|
||||
extractor that accepted everything, or a `main` that returned 0 regardless
|
||||
would all have produced the same clean run -- and clean runs from this tool
|
||||
are cited in `DECISIONS.md` as evidence that fifteen labels are grounded.
|
||||
|
||||
Three synthetic rows are pushed through the REAL classifier, and its verdict
|
||||
is read rather than reasoned about:
|
||||
|
||||
a `why` citing nothing -> must be BARE
|
||||
a `why` citing a path that exists -> must be ok
|
||||
a `why` citing a path that does not -> must be DANGLING
|
||||
|
||||
Exit codes follow the convention the Decoder and I converged on: 0 all good,
|
||||
1 a real audit failure, **2 the harness is broken** and no clean run from it
|
||||
means anything.
|
||||
"""
|
||||
paths = known_paths()
|
||||
cases = [
|
||||
("bare", "no citation of any kind here, just prose", "BARE"),
|
||||
("ok", "see tools/port/audit-kinds for the method", "ok"),
|
||||
("dangling", "see docs/port/NO-SUCH-FILE-XYZ.md", "DANGLING"),
|
||||
]
|
||||
bad = 0
|
||||
for name, why, want in cases:
|
||||
cites = citations(why)
|
||||
if not cites:
|
||||
got = "BARE"
|
||||
else:
|
||||
unresolved = [c for t, c in cites
|
||||
if t == "path" and c not in paths and not os.path.exists(c)]
|
||||
got = "DANGLING" if unresolved else "ok"
|
||||
mark = "✅" if got == want else "🔴"
|
||||
print(f" harness: a why that is {name:<9} -> {got:<8} (want {want:<8}) {mark}")
|
||||
if got != want:
|
||||
bad += 1
|
||||
print()
|
||||
if bad:
|
||||
print("🔴 the classifier cannot tell grounded labels from ungrounded ones.")
|
||||
print(" Exit 2: nothing this tool has reported clean is trustworthy.")
|
||||
return 2
|
||||
print("the classifier separates bare, dangling and grounded citations")
|
||||
return 0
|
||||
|
||||
|
||||
def main():
|
||||
if "--selftest" in sys.argv:
|
||||
return selftest()
|
||||
rows = []
|
||||
for f in sorted(glob.glob("authored/*.json")):
|
||||
walk(json.load(open(f)), f, "", rows)
|
||||
|
||||
@@ -54,6 +54,9 @@ step modding-rules must-pass tools/port/check-modding
|
||||
# 2026-08-30 nothing checked what any of them rested on -- seven were resting on
|
||||
# a sibling `why` that argued a different claim.
|
||||
step authored-kinds must-pass tools/port/audit-kinds
|
||||
# The classifier is asked whether it can tell grounded from ungrounded at all,
|
||||
# rather than only what it found. Exit 2 = the harness is broken.
|
||||
step kinds-harness must-pass tools/port/audit-kinds --selftest
|
||||
# Band levels are alignment-free and carry their own known negative on every run;
|
||||
# the difference-signal half of the same tool stays report-only and asserts
|
||||
# nothing. See docs/port/DECISIONS.md -- the waveform question is still open.
|
||||
|
||||
@@ -110,6 +110,23 @@ if [ "${1:-}" = "--control" ]; then
|
||||
fi
|
||||
done
|
||||
rm -f "$probe"
|
||||
# 🔴 FIFTH CASE: the same text OUTSIDE the scanned root must give 0.
|
||||
#
|
||||
# Without it, "the plant is inside a scanned directory" is a property I
|
||||
# verified BY HAND, once -- which is the exact pattern I had just finished
|
||||
# criticising in this tool one iteration earlier. The pair is what asserts the
|
||||
# boundary is real: identical text, exit 1 inside and 0 outside. Either half
|
||||
# alone is consistent with the tool scanning everything, or nothing.
|
||||
#
|
||||
# The Decoder added this to theirs after I raised the boundary; the reason it
|
||||
# was worth adding is that their property held *because they had reasoned it*,
|
||||
# not because anything asserted it. Mine was in the same state.
|
||||
outside="${TMPDIR:-/tmp}/claims-control-outside.md"
|
||||
printf '%s\n' "$claim" > "$outside"
|
||||
got=$(run)
|
||||
rm -f "$outside"
|
||||
if [ "$got" = "0" ]; then printf ' %-26s exit 0 ✅\n' "same text outside root"
|
||||
else printf ' %-26s exit %s, wanted 0 🔴\n' "same text outside root" "$got"; ok=1; fi
|
||||
got=$(CLAIMS_REGISTER="" "$0" >/dev/null 2>&1; echo $?)
|
||||
if [ "$got" = "2" ]; then printf ' %-26s exit 2 ✅\n' "empty register"
|
||||
else printf ' %-26s exit %s, wanted 2 🔴\n' "empty register" "$got"; ok=1; fi
|
||||
|
||||
@@ -398,8 +398,20 @@ def guard_focus_scope(_h):
|
||||
if isinstance(v, dict) and "focus_persists" not in v)
|
||||
ok = stated == {"main_menu": True, "extras": False}
|
||||
if ok:
|
||||
# ✅ 2026-08-31: all FOUR submenus are now measured to reset -- EXTRAS,
|
||||
# LOAD GAME, TUTORIAL and OPTIONS -- and the main menu remains the only
|
||||
# screen that remembers. Three of those four are not in this export, so
|
||||
# no authored value changes.
|
||||
#
|
||||
# 🔴 NOT PROMOTED TO A RULE, deliberately. "Submenus reset" at 4/4 is
|
||||
# better evidence than the 2/2 that made `wrap` a rule -- and adopting it
|
||||
# would change nothing today, because the only submenu this port ships is
|
||||
# already measured. What it WOULD do is pre-decide the next screen from a
|
||||
# generalisation instead of a measurement, which is the trap that nearly
|
||||
# let a derived rule overwrite EXTRAS' measured opening item.
|
||||
print(f" {'focus_persists scope':<30} guard {stated} measured;"
|
||||
f" {len(silent)} screen(s) silent = UNMEASURED, not 'resets'")
|
||||
f" {len(silent)} screen(s) silent = UNMEASURED, not 'resets'"
|
||||
f" [4/4 submenus reset disc-wide; not promoted to a rule]")
|
||||
else:
|
||||
FAIL += 1
|
||||
print(f" {'focus_persists scope':<30} 🔴 GUARD {stated} -- a screen states"
|
||||
|
||||
Reference in New Issue
Block a user