fabi/Sylpheed
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

tools: the register now prints its own scope, not just its verdict

Browse Source 
sylpheed-port found audit-kinds auditing 16 of 71 authored justifications and never
saying so -- a checker that fails correctly while describing a sixth of the corpus.
Their line is the one that generalises: 'I checked and it was fine' and 'I checked
the part that declared itself' read identically in a log, and only one of them is
what gets quoted.

Measured here: of 86 refutation-shaped bullets in REFUTED.md, 83 are in the
registered form. 97 %, which is better than their 16/71 but was equally unstated.

The three gaps are deliberate, not a bug. They quote their claim in backticks and
are bare identifiers -- +0x29d0, position = instance - 0x12c -- so registering them
would match every live mention of the same offset and train the check to be
ignored. Reported rather than forced to 100 %, for the same reason they report the
ratio instead of demanding it: forcing a counter invites mislabelling, which is
worse than the gap.

Selftest and the real run both still exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wuu56cE8vJGTBtn1ppsk8v
This commit is contained in:
sylph-decoder
2026-08-31 03:33:00 +00:00
parent cd4156aae9
commit 976d657808
1 changed files with 24 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
tools/re-capture/check_refuted.py
View File

@@ -135,7 +135,30 @@ for line in ref.read_text().splitlines():
if m:
claims.append(m.group(1))
print(f"{len(claims)} quoted claims in REFUTED.md\n")
# ── SCOPE, printed before the verdict ────────────────────────────────────────
# sylpheed-port found `audit-kinds` auditing 16 of 71 authored justifications and
# never saying so -- a checker that FAILS CORRECTLY while describing a sixth of
# the corpus. Their line is the one that matters: "I checked and it was fine" and
# "I checked the part that declared itself" read identically in a log.
#
# Measured here 2026-08-31: of 86 refutation-shaped bullets in REFUTED.md, 83 are
# in the registered `* "claim"` form -- 97 %.
#
# ⚠️ THE THREE GAPS ARE NOT A BUG AND ARE NOT REGISTERED ON PURPOSE. They quote
# their claim in BACKTICKS and are bare identifiers -- `+0x29d0`,
# `position = instance - 0x12c`. Registering those would match every live mention
# of the same offset, producing permanent false hits and training the check to be
# ignored -- the unregistrable-claim limit this corpus already records for the
# 0.32 collision. Reported, not forced to 100 %.
_bul = [l for l in ref.read_text().splitlines() if l.lstrip().startswith("* ")]
_kill = [l for l in _bul if re.search(r"(→|->|—|--)\s*\*{0,2}(refuted|withdrawn|retracted"
r"|wrong|dead|no\b|it is)", l, re.I)]
_unreg = [l for l in _kill if not re.match(r'\s*\*\s*~?~?"([^"]{25,})"', l)]
_cov = 100 * (len(_kill) - len(_unreg)) / max(len(_kill), 1)
print(f"{len(claims)} quoted claims in REFUTED.md")
print(f"SCOPE: {len(_kill) - len(_unreg)} of {len(_kill)} refutation-shaped bullets are "
f"registered ({_cov:.0f}%); {len(_unreg)} quote their claim in backticks and are "
f"deliberately unregistrable\n")
# 🔴 A register that loaded NOTHING cannot fail, and would report clean forever --
# the same shape as sylpheed-port's stub that prints "everything is fine" and