fabi/Sylpheed
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

re: out-of-sample test FAILS 3 of 6 -- plate-relative numbers withdrawn

Browse Source 
One fresh no-input boot against predictions registered before it. Sweep-
internal quantities pass on a run that had no hand in deriving them: leaf
period ratio 1.1753, size ratio 1.3009, pulse/sweep 0.0963. Those are now
3-for-3 across captures.

The three plate-relative quantities fail, together and in the same
direction: ptcopyright ramp ratio 0.550 vs 0.733 (0.75x) and sweep->plate
separation 0.0996 vs 0.138-0.141 (0.71x). So the separation is not a
constant -- three runs give 0.138, 0.141, 0.0996 -- and my published "2.2%
agreement" was n=2. Withdrawn. The F6 conclusion survives qualitatively;
the number does not.

Also records that check_labels.py fails its first independent test. I
validated it on the two captures that produced the labels, which is an
instrument validated on its own training data. Two of four checks fire on
f6c. I am not guessing which of labels/tolerances/run-variation is
responsible.

Caught mid-analysis that I had first run the checks against a log still
being written; re-ran on the complete log and the numbers are identical, so
the failures are not a truncation artefact.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jc4pciRArGHfxGGhEbwp5t
This commit is contained in:
sylph-decoder
2026-09-04 13:20:09 +00:00
parent 8927d2f1ae
commit a61c191939
1 changed files with 57 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
docs/re/f6-out-of-sample-RESULT.md Normal file
View File

@@ -0,0 +1,57 @@
# 🔴 Out-of-sample: **three of six predictions failed**
The predictions are in
[`f6-out-of-sample-prereg.md`](f6-out-of-sample-prereg.md), committed before the
capture. One fresh no-input boot, `f6c`.
| # | quantity | predicted | `f6c` | |
|---|---|---|---|---|
| 1 | `pteff03a` / `pteff03` period | 1.200 ±5 % | **1.1753** | ✅ |
| 2 | strip length ratio | 1.333 ±6 % | **1.3009** | ✅ |
| 3 | pulse period / sweep loop | 0.100 ±5 % | **0.0963** | ✅ |
| 4 | pulse amplitude vs declared curve | ≤3 levels | **8.73** | 🔴 |
| 5 | `ptcopyright` ramp / parent ramp | 0.733 ±8 % | **0.5500** | 🔴 |
| 6 | sweep leads `ptbtn00f` | 0.138–0.141 loops | **0.0996** | 🔴 |
## What holds
**Everything sweep-internal, across three captures.** The two leaves' period
ratio, their size ratio, and the pulse period as a fraction of the sweep loop all
reproduce on a run that had no hand in deriving them. Those stay ✅.
## What does not
**The three plate-relative quantities.** And they fail *together and in the same
direction*: 5 lands at 0.75× its prediction, 6 at 0.71×. That is not noise — the
plate and `ptcopyright` both arrive earlier relative to the sweep in this run than
in the two that produced the numbers.
🔴 **So the sweep→plate separation is not a constant.** Three runs give
**0.138, 0.141, 0.0996**. My published "0.138 and 0.141, agreeing to 2.2 %" was
**n = 2**, and two runs agreeing is not reproducibility. The figure is withdrawn.
⚠️ **The F6 conclusion survives; its number does not.** With no input the glow
still clearly precedes the plate — 0.0996 of a loop is a wide margin — and with Ⓐ
the two still coincide. What is dead is any specific value for the lead.
## 🔴 And the check I built last iteration fails its first independent test
`check_labels.py` was written to catch label drift, and I validated it on **the
two captures that produced the labels**. On the first capture it did not see, two
of its four checks fire.
That is an instrument validated on its own training data — the same class as
every other error this session, one level up. **A detector tuned on n = 2 and
failing on n = 3 is not yet a detector**, and I presented it as one.
I cannot yet say which of three causes is responsible, and I am not going to
guess: the labels could be wrong in `f6c`, the tolerances could be fitted, or the
game's plate timing could genuinely vary run to run. The diagnostic that would
separate them is more captures — `n = 3` is where this became visible and `n = 3`
is not enough to resolve it.
## Reach
Three captures, one screen, one emulator build. The sweep-internal results are
now 3-for-3; the plate-relative ones are 0-for-1 out of sample after 2-for-2
in-sample.