docs: check that the docs' headline figures match their committed data
Nothing had ever verified that a number written in prose matches the reference data file committed beside it. The figure is written once from a run; the prose is edited around it afterwards and the data file is regenerated independently, so drift is silent. All 19 headline figures across four censuses -- the eff-bit census, the plateau census, the top-level rotation census and the eff-bit alpha test -- currently agree with their data files. The checker had to be numeric, and the first attempt is the reason it is a script rather than a grep: comparing strings reported almost every figure as a mismatch, because the data files write 14709 where the docs write "14 709" with a thin space, and the docs round 33.66 to 33.7. A consistency check that fails on formatting trains you to ignore it, so the tolerance is explicit: exact against the data, within 0.05 against the doc to allow rounding. Also ran the full disc-gated workspace suite (build-reborn test, which wires SYLPHEED_DISC -- without it the disc tests self-skip and green means almost nothing), covering this session's three decoder changes: rotation_deg on Keyframe, the scale-0 fix in blit/fill_quad, and the flags field on T8adImage. 122 passed / 0 failed across the four suites that had completed; the long disc-gated integration tests (records_roundtrip_disc, first_header_word_is_record0_hash) were still running and are not counted here.
This commit is contained in:
@@ -712,3 +712,12 @@ agent's loop prompt, i.e. nowhere durable. See [`README.md`](README.md) for the
|
||||
call, but only if the negative space is written down — additive blend, name,
|
||||
lifetime, premultiplied alpha — so the next attempt starts where this one ended
|
||||
rather than at the beginning.
|
||||
* **Nothing was checking that a doc's figures match its committed data.** A
|
||||
number is written once from a run and then lives in prose that gets edited
|
||||
around it; the data file beside it is regenerated independently. All 19
|
||||
headline figures across four censuses do currently agree
|
||||
(`tools/re-capture/doc_figure_check.py`), which is worth knowing rather than
|
||||
assuming — but the checker had to be written **numerically**, because the first
|
||||
version grepped for the doc's formatting (`14 709` with a thin space, `33.7`
|
||||
rounded from `33.66`) and reported almost every figure as a mismatch. A
|
||||
consistency check that fails on formatting will train you to ignore it.
|
||||
|
||||
48
tools/re-capture/doc_figure_check.py
Executable file
48
tools/re-capture/doc_figure_check.py
Executable file
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Do the headline figures in a doc match its committed reference data?
|
||||
|
||||
Numbers drift between a run and the prose written about it, and nothing was
|
||||
checking. This compares the two NUMERICALLY, which matters: a naive string
|
||||
grep reports every figure as a mismatch, because the data files write `14709`
|
||||
and the docs write `14 709` (thin space) and round `33.66` to `33.7`. That
|
||||
false-positive run is why this is a script and not a grep.
|
||||
|
||||
doc_figure_check.py # runs the built-in case list
|
||||
"""
|
||||
import re, sys
|
||||
|
||||
CASES = [
|
||||
("eff-bit-census.txt", "structures/ui-paint-order-key.md",
|
||||
[14709, 2338, 2657, 1399, 8315, 0.468, 0.144]),
|
||||
("plateau-census.txt", "structures/ui-resting-pose.md",
|
||||
[15493, 3807, 24.57, 50.2]),
|
||||
("rotation-toplevel-census.txt", "structures/ui-keyframe-rotation.md",
|
||||
[2152, 13.89]),
|
||||
("eff-bit-alpha-test.txt", "structures/ui-paint-order-key.md",
|
||||
[55.52, 33.66, 52.52, 30.17, 76.5, 64.1]),
|
||||
]
|
||||
NUM = re.compile(r"\d[\d ,]*\.?\d*")
|
||||
|
||||
def nums(text):
|
||||
out = set()
|
||||
for m in NUM.finditer(text):
|
||||
try: out.add(float(re.sub(r"[ ,]", "", m.group())))
|
||||
except ValueError: pass
|
||||
return out
|
||||
|
||||
def main():
|
||||
bad = 0
|
||||
for dfile, mfile, figs in CASES:
|
||||
D = nums(open(f"docs/re/data/{dfile}", encoding="utf-8").read())
|
||||
M = nums(open(f"docs/re/{mfile}", encoding="utf-8").read())
|
||||
for f in figs:
|
||||
in_d = any(abs(f - x) < 0.011 for x in D)
|
||||
in_m = any(abs(f - x) < 0.051 for x in M) # the doc may round
|
||||
if not (in_d and in_m):
|
||||
bad += 1
|
||||
print(f" CHECK {dfile} / {mfile}: {f} data:{in_d} doc:{in_m}")
|
||||
print(f"{sum(len(c[2]) for c in CASES)} figures checked, {bad} to look at")
|
||||
return 1 if bad else 0
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user