port: audit-kinds was auditing 16 of 71 authored justifications and never said so
Back to the port, and the finding came from reading data rather than a tool. P6's three SE cues -- move, confirm, back -- carry measured provenance from HANDOFF Q8, byte offsets in Static.slb and careful why text. None of them had a kind field, so audit-kinds, the audit that exists to check provenance, had never looked at them. Counting the corpus: 55 why fields with no kind against 16 with one. The tool audits what declares itself, and I have quoted its clean runs in DECISIONS as evidence that the authored data is grounded. That was a statement about 16 of 71. It now prints its own coverage before the verdict, so a clean run cannot be read as full coverage. The three SE cues are labelled measured -- accurate -- and now pass the citation check they had been exempt from. 19 of 71. Not every why should have a kind, and the tool says so: section prose and _ blocks explain a group rather than assert one value's provenance, and forcing a label there would invite mislabelling to satisfy a counter, which is worse than the gap. So it reports the ratio rather than demanding it be 1. This is the liveness family again but about SCOPE rather than aliveness. Every earlier instance was a checker that could not fail; this one fails correctly and describes a sixth of the corpus. 'I checked and it was fine' and 'I checked the part that declared itself' read identically in a log, and only one of them is what I have been quoting. Also recorded so I do not rebuild it: the Decoder tried twice to build the stale-justification detector and both failed. Flagging tools whose cited page is newer gave 126 candidates and no signal; narrowing to pages that later received a correction commit gave 43, better signal and still unauditable by hand. They sampled 3 before publishing and all three were false positives. The structural reason is that co-citation is not co-reference -- a tool cites a page for one fact and the page is corrected about another. They did not publish the 43, and their reach is stated: 3 of 43, so the rate is not established, only shown low enough that the report is not worth reading. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N7FiFFFwbvG2uxdcEh8HyF
This commit is contained in:
@@ -178,6 +178,44 @@ def selftest():
|
||||
return 0
|
||||
|
||||
|
||||
def coverage(files):
|
||||
"""How much of the authored corpus this audit can even see.
|
||||
|
||||
🔴 IT SEES 15 OF 70. Every `kind` label is checked for a citation, and a
|
||||
clean run has been quoted in `DECISIONS.md` as evidence that the authored
|
||||
data is grounded -- but a `why` with NO `kind` beside it is invisible to this
|
||||
walk entirely, and there are 55 of those against 15 labels.
|
||||
|
||||
Found by reading the data rather than the tool: `audio.json`'s three SE cues
|
||||
carry measured provenance from HANDOFF Q8 and no `kind` field, so the audit
|
||||
that exists to check provenance never looked at them.
|
||||
|
||||
⚠️ NOT every `why` should have a `kind`. Section prose and `_` blocks explain
|
||||
a group rather than assert one value's provenance, and forcing a label there
|
||||
would invite mislabelling to satisfy a counter. So this REPORTS the ratio
|
||||
rather than demanding it be 1 -- a clean run must not read as full coverage.
|
||||
"""
|
||||
labelled = orphan = 0
|
||||
for f in files:
|
||||
def walk(o):
|
||||
nonlocal labelled, orphan
|
||||
if isinstance(o, dict):
|
||||
for k, v in o.items():
|
||||
if k.endswith("_why") or k == "why":
|
||||
stem = k[:-4] if k.endswith("_why") else ""
|
||||
kk = (stem + "_kind") if stem else "kind"
|
||||
if kk in o:
|
||||
labelled += 1
|
||||
else:
|
||||
orphan += 1
|
||||
walk(v)
|
||||
elif isinstance(o, list):
|
||||
for x in o:
|
||||
walk(x)
|
||||
walk(json.load(open(f, encoding="utf-8")))
|
||||
return labelled, orphan
|
||||
|
||||
|
||||
def main():
|
||||
if "--selftest" in sys.argv:
|
||||
return selftest()
|
||||
@@ -238,6 +276,11 @@ def main():
|
||||
print(f" ⚠️ {len(variants)} spellings of the same label: {variants}")
|
||||
print(" A consumer comparing == 'measured' misses the others, and a")
|
||||
print(" label that fails to match reads as ABSENT, not as wrong.\n")
|
||||
lab, orph = coverage(sorted(glob.glob("authored/*.json")))
|
||||
print(f" COVERAGE: {lab} `why` field(s) carry a `kind` and were audited above;")
|
||||
print(f" {orph} carry NO `kind` and are INVISIBLE to this audit. A clean run")
|
||||
print(f" below is a statement about {lab} of {lab + orph} authored justifications.")
|
||||
print()
|
||||
print(f" {bare} bare or borrowed, {dangling} dangling, {len(rows) - bare - dangling} with resolving citations")
|
||||
print(" 🔴 A resolving citation is not a verified label. Nothing here reads")
|
||||
print(" the cited page to confirm it says what the `why` claims.")
|
||||
|
||||
Reference in New Issue
Block a user