From a61c191939386a40c5f7bcf88cb446e8998863a7 Mon Sep 17 00:00:00 2001 From: sylph-decoder Date: Fri, 4 Sep 2026 13:20:09 +0000 Subject: [PATCH] re: out-of-sample test FAILS 3 of 6 -- plate-relative numbers withdrawn One fresh no-input boot against predictions registered before it. Sweep- internal quantities pass on a run that had no hand in deriving them: leaf period ratio 1.1753, size ratio 1.3009, pulse/sweep 0.0963. Those are now 3-for-3 across captures. The three plate-relative quantities fail, together and in the same direction: ptcopyright ramp ratio 0.550 vs 0.733 (0.75x) and sweep->plate separation 0.0996 vs 0.138-0.141 (0.71x). So the separation is not a constant -- three runs give 0.138, 0.141, 0.0996 -- and my published "2.2% agreement" was n=2. Withdrawn. The F6 conclusion survives qualitatively; the number does not. Also records that check_labels.py fails its first independent test. I validated it on the two captures that produced the labels, which is an instrument validated on its own training data. Two of four checks fire on f6c. I am not guessing which of labels/tolerances/run-variation is responsible. Caught mid-analysis that I had first run the checks against a log still being written; re-ran on the complete log and the numbers are identical, so the failures are not a truncation artefact. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Jc4pciRArGHfxGGhEbwp5t --- docs/re/f6-out-of-sample-RESULT.md | 57 ++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 docs/re/f6-out-of-sample-RESULT.md diff --git a/docs/re/f6-out-of-sample-RESULT.md b/docs/re/f6-out-of-sample-RESULT.md new file mode 100644 index 00000000..457ccbec --- /dev/null +++ b/docs/re/f6-out-of-sample-RESULT.md @@ -0,0 +1,57 @@ +# πŸ”΄ Out-of-sample: **three of six predictions failed** + +The predictions are in +[`f6-out-of-sample-prereg.md`](f6-out-of-sample-prereg.md), committed before the +capture. One fresh no-input boot, `f6c`. + +| # | quantity | predicted | `f6c` | | +|---|---|---|---|---| +| 1 | `pteff03a` / `pteff03` period | 1.200 Β±5 % | **1.1753** | βœ… | +| 2 | strip length ratio | 1.333 Β±6 % | **1.3009** | βœ… | +| 3 | pulse period / sweep loop | 0.100 Β±5 % | **0.0963** | βœ… | +| 4 | pulse amplitude vs declared curve | ≀3 levels | **8.73** | πŸ”΄ | +| 5 | `ptcopyright` ramp / parent ramp | 0.733 Β±8 % | **0.5500** | πŸ”΄ | +| 6 | sweep leads `ptbtn00f` | 0.138–0.141 loops | **0.0996** | πŸ”΄ | + +## What holds + +**Everything sweep-internal, across three captures.** The two leaves' period +ratio, their size ratio, and the pulse period as a fraction of the sweep loop all +reproduce on a run that had no hand in deriving them. Those stay βœ…. + +## What does not + +**The three plate-relative quantities.** And they fail *together and in the same +direction*: 5 lands at 0.75Γ— its prediction, 6 at 0.71Γ—. That is not noise β€” the +plate and `ptcopyright` both arrive earlier relative to the sweep in this run than +in the two that produced the numbers. + +πŸ”΄ **So the sweepβ†’plate separation is not a constant.** Three runs give +**0.138, 0.141, 0.0996**. My published "0.138 and 0.141, agreeing to 2.2 %" was +**n = 2**, and two runs agreeing is not reproducibility. The figure is withdrawn. + +⚠️ **The F6 conclusion survives; its number does not.** With no input the glow +still clearly precedes the plate β€” 0.0996 of a loop is a wide margin β€” and with β’Ά +the two still coincide. What is dead is any specific value for the lead. + +## πŸ”΄ And the check I built last iteration fails its first independent test + +`check_labels.py` was written to catch label drift, and I validated it on **the +two captures that produced the labels**. On the first capture it did not see, two +of its four checks fire. + +That is an instrument validated on its own training data β€” the same class as +every other error this session, one level up. **A detector tuned on n = 2 and +failing on n = 3 is not yet a detector**, and I presented it as one. + +I cannot yet say which of three causes is responsible, and I am not going to +guess: the labels could be wrong in `f6c`, the tolerances could be fitted, or the +game's plate timing could genuinely vary run to run. The diagnostic that would +separate them is more captures β€” `n = 3` is where this became visible and `n = 3` +is not enough to resolve it. + +## Reach + +Three captures, one screen, one emulator build. The sweep-internal results are +now 3-for-3; the plate-relative ones are 0-for-1 out of sample after 2-for-2 +in-sample.