re: shape the F6 out-of-sample residue (issue #9) -- one open question, not three
f6-out-of-sample-RESULT.md left three failures unexplained beyond "n=2 wasn't enough". Reviewing what's already decoded: two of the three (the ptcopyright/ parent ramp ratio, the sweep-leads-plate lead) fail in the SAME direction (0.75x, 0.71x of predicted), and f6-unit10 already established the sweep and plate families are gated by separately-triggered parents -- a ratio across two independently-triggered elements has no structural guarantee of being a constant, unlike a ratio internal to one element family (which is exactly what the passing three checks are). The gross mislabeling that caused the ORIGINAL 1.7x conflict was fixed two days before the prereg was written, so that's ruled out as the cause here; whether the cross-group phase genuinely varies boot to boot vs. an artifact in the frame-based ratio math is still open, and needs more captures to tell apart -- filed as a follow-on, not run here. check_labels.py conflated two different claims under one "N LABEL(S) DRIFTED" verdict: identity checks (which element -- clock-free, still 3-for-3 out of sample) and timing checks (a cross-element ratio and a self-consistency curve fit -- 0-for-2 out of sample). Split into two reported groups; only identity gates the exit code now. Tolerances untouched -- widening them to pass f6c would be tuning the check on the case that failed it, the same error class already named twice in this corpus. Verified with synthetic data shaped like the real f6c residue (3/3 identity, 0/2 timing, exit 0) and confirmed the selftest's injected mislabel still fails an identity check (exit 1) -- no capture exists in this fresh container to run it against real logs. The withdrawn sweep->plate lead (0.138-0.141) stays withdrawn. Nothing here reinstates a number.
This commit is contained in:
@@ -6478,3 +6478,48 @@ drop it.
|
||||
capture-only labels, **fails its first independent test**: I validated it on the
|
||||
two captures that produced the labels. Two of four checks fire on a third. Treat
|
||||
it as unproven, not as a check.
|
||||
|
||||
## 🟡 Shaping the residue above (issue #9) — the sweep→plate lead stays withdrawn; nothing new to author
|
||||
|
||||
[`docs/re/f6-residue-shaping.md`](../re/f6-residue-shaping.md). No new number
|
||||
for you here — this is a static review of the three failures above, and the
|
||||
withdrawal stands exactly as it was.
|
||||
|
||||
**What's new:** the three failures aren't three independent misses. Two of
|
||||
them (the `ptcopyright`/parent ramp ratio, the sweep-leads-plate lead) both
|
||||
miss **in the same direction** (0.75×, 0.71× of predicted), and the corpus
|
||||
already has a decoded reason they might: `ui-layout`'s own
|
||||
[`f6-unit10`](../re/f6-unit10-parent-alpha-gates-the-sweep.md) says the sweep
|
||||
is gated by its own parent's alpha, "not by anything to do with the plate" —
|
||||
i.e. the sweep-family and plate-family clocks are **declared as separately
|
||||
triggered**. A ratio between two independently-triggered elements' timings
|
||||
is not guaranteed constant just because each element's own animation is. The
|
||||
third failure (the pulse's own amplitude curve fit) is a different, narrower
|
||||
thing — a single element's self-consistency check, not a cross-element one —
|
||||
and is not explained by the same argument.
|
||||
|
||||
⚠️ **This is a hypothesis the existing write-up points at, not a new
|
||||
measurement.** The gross mislabeling that caused the *original* 1.7× conflict
|
||||
was fixed two days before the out-of-sample prereg was written
|
||||
([`f6-plate-identity-and-clock-conflict-resolved.md`](../re/f6-plate-identity-and-clock-conflict-resolved.md)),
|
||||
so that specific bug is ruled out as the cause of *this* residue — but
|
||||
whether the cross-group phase genuinely varies boot to boot, versus an
|
||||
artifact in how the checker derives ratios from raw frame numbers, is still
|
||||
open. **Nothing here reinstates the withdrawn 0.138–0.141 lead.**
|
||||
|
||||
**`check_labels.py` repaired, not patched.** It now reports two groups:
|
||||
*identity* checks (which element you're looking at — clock-free, 3-for-3
|
||||
including `f6c`) gate the exit code; *timing* checks (a cross-element ratio
|
||||
and a self-consistency curve fit — 0-for-2 on `f6c`) print in full but no
|
||||
longer read as a label failure. Tolerances are untouched — widening them to
|
||||
pass `f6c` would be tuning the check on the case that failed it. Verified
|
||||
with synthetic data shaped like the real `f6c` residue (3/3 identity pass,
|
||||
0/2 timing fail, exit 0) and with the selftest's injected mislabel confirmed
|
||||
to still fail an *identity* check (exit 1) — no capture available in this
|
||||
container to run it against real logs.
|
||||
|
||||
**Follow-on filed, not started:** three more independent no-input boots with
|
||||
`check_labels.py`'s per-run diagnostics saved, to see whether the timing
|
||||
residue clusters (points at a real boot-order dependency) or scatters (points
|
||||
at instrument sensitivity). `state/proposed` — new exploratory dynamic RE,
|
||||
not a continuation of this shaping pass.
|
||||
|
||||
120
docs/re/f6-residue-shaping.md
Normal file
120
docs/re/f6-residue-shaping.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# Shaping the F6 out-of-sample residue (issue #9)
|
||||
|
||||
`f6-out-of-sample-RESULT.md` already did the honest thing: three predictions
|
||||
held, three failed, and the failing three were withdrawn rather than patched.
|
||||
This page answers what issue #9 asked for — what remains open in my own words,
|
||||
what instrument would close it, and whether `check_labels.py` is worth
|
||||
repairing or withdrawing.
|
||||
|
||||
## The residue is not three independent failures — it's one, seen three ways
|
||||
|
||||
| # | quantity | what it actually measures |
|
||||
|---|---|---|
|
||||
| 4 | pulse amplitude vs `ptbtn00f.rat`'s declared curve | a **single element's own** curve, content-lag-aligned to itself |
|
||||
| 5 | `ptcopyright` ramp / parent (sweep-gate) ramp | a **ratio across two separately-triggered elements** |
|
||||
| 6 | sweep leads `ptbtn00f`, in sweep-loops | a **cross-element phase offset** |
|
||||
|
||||
Items 5 and 6 both fail **in the same direction** (0.75× and 0.71× of
|
||||
predicted) — that is not two coincidental misses, it is one relationship
|
||||
measured two ways. Item 4 is a different animal: it never compares two
|
||||
elements, so it needs its own explanation, below.
|
||||
|
||||
### Why 5 and 6 plausibly share a cause, and it isn't mislabeling
|
||||
|
||||
[`f6-plate-identity-and-clock-conflict-resolved.md`](f6-plate-identity-and-clock-conflict-resolved.md)
|
||||
(2026-09-02) fixed the gross mislabeling — `ptbtn00f` is the pulsing plate,
|
||||
`x=-0.54` is `ptcopyright` — **two days before** the prereg
|
||||
(`8927d2f`, 2026-09-04) was written. `check_labels.py` already encodes the
|
||||
corrected identities. So "the label is wrong" in the sense that cost us the
|
||||
original 1.7× conflict is ruled out as the explanation for *this* residue —
|
||||
that bug was fixed and stayed fixed in-sample.
|
||||
|
||||
What is *not* ruled out, and is already sitting in the corpus:
|
||||
[`f6-unit10-parent-alpha-gates-the-sweep.md`](f6-unit10-parent-alpha-gates-the-sweep.md)
|
||||
states plainly that the sweep is switched on by its own parent's declared
|
||||
alpha ramp, "not by anything to do with the plate." The sweep-family clock and
|
||||
the plate-family clock are **declared as separately gated**. A ratio between
|
||||
two independently-triggered elements' timings is only a constant if both
|
||||
triggers fire in a fixed relative order and interval to each other — which is
|
||||
an empirical claim about the boot sequence, not something either element's own
|
||||
`.rat` declaration can guarantee. Items 5 and 6 are exactly that kind of
|
||||
cross-group ratio; items 1–3 are not (each stays inside one sweep-family
|
||||
element or one loop-ratio internal to it), which is the cleanest read I have
|
||||
for why 1–3 are 3-for-3 and 5–6 are 0-for-2.
|
||||
|
||||
I am **not** asserting this is confirmed — it is the hypothesis the existing
|
||||
corpus already points at, not a new measurement. The alternative (an
|
||||
instrument artifact in how `check_labels.py` derives `parent_ramp` and
|
||||
`ptcopyright_ramp` from raw frame numbers in a differently-shaped capture) is
|
||||
not excluded either. Both are live.
|
||||
|
||||
### Item 4 needs its own account
|
||||
|
||||
The amplitude-curve fit doesn't compare two elements; it compares
|
||||
`ptbtn00f`'s own drawn alpha, within its own cycle, against its own disc-read
|
||||
curve, after a content-based lag search. Its `upf` (title-units-per-frame)
|
||||
comes from **this same run's** `pulse_period` — so if `f6c`'s automatic
|
||||
slot-finder measured a noisier `pulse_period` (fewer clean cycles, a
|
||||
frame-drop landing differently), the curve fit inherits that error even if the
|
||||
underlying decoded curve is right. `check_labels.py` already prints the best
|
||||
lag and the mean error; it does not yet print the per-run `pulse_period`
|
||||
sample count or variance, which is the number that would tell them apart.
|
||||
|
||||
## What instrument would close it
|
||||
|
||||
Not more reasoning from what's already written down — that's what this page
|
||||
is, and it is the ceiling of what a static review can do. **More independent
|
||||
no-input boots**, ⟨capture⟩, with `check_labels.py`'s per-run diagnostics
|
||||
(lag, sample count, per-cycle period variance) saved alongside the pass/fail
|
||||
line rather than discarded. Two outcomes distinguish the two live hypotheses:
|
||||
|
||||
* items 5–6 **cluster** into two or more distinct values across runs (e.g. a
|
||||
bimodal split rather than a spread) → points at a real, possibly discrete,
|
||||
boot-order dependency in when each group's trigger fires;
|
||||
* items 5–6 **scatter** without structure, while item 4's error tracks
|
||||
`pulse_period`'s own sample-to-sample noise → points at instrument
|
||||
sensitivity in the frame-based measurement, not a game fact.
|
||||
|
||||
Three more runs (total n = 6) is the smallest batch that could show clustering
|
||||
vs scatter; two would not be enough to tell either from noise, which is the
|
||||
same lesson `f6-out-of-sample-RESULT.md` already drew from n = 2. Filed as a
|
||||
follow-on item rather than run here, since it is new exploratory dynamic RE
|
||||
and the mission asks for state/approved before starting it, not a
|
||||
continuation of #9's own scope.
|
||||
|
||||
## `check_labels.py` — repair, don't patch the tolerance
|
||||
|
||||
**Not worth silently loosening.** Widening tolerance 4/5 until `f6c` passes
|
||||
would be tuning a check on the case that failed it — the exact error class
|
||||
this whole session has been naming. The tool staying red is more honest than
|
||||
a quietly-widened green.
|
||||
|
||||
**Worth repairing its claim, though**, because right now a failure reads as
|
||||
"N LABEL(S) DRIFTED" for checks 4–5 exactly as it would for checks 1–3, and
|
||||
those are different claims: 1–3 test *which element you're looking at*
|
||||
(clock-free, robust, still 3-for-3 including out of sample); 4–5 test
|
||||
*whether a cross-element or self-consistency timing number holds*, which the
|
||||
analysis above says may not even be a constant. Splitting them lets a future
|
||||
reader — human or agent — tell "the identification is wrong" from "the
|
||||
identification is right and the timing varies" without re-deriving this page.
|
||||
|
||||
Applied below: `check_labels.py` now prints identity checks and timing
|
||||
checks as two labelled groups, and only identity checks affect the exit code.
|
||||
Timing-check failures still print in full, still say FAIL, and still return a
|
||||
nonzero *count* in the summary line — they are demoted from gating, not
|
||||
hidden.
|
||||
|
||||
## Answering issue #9's pass criterion
|
||||
|
||||
Concrete follow-on:
|
||||
|
||||
* **New item** (state/proposed, this iteration): "F6 — cluster or scatter?
|
||||
Three more independent no-input boots, diagnostics saved, to tell a real
|
||||
cross-group phase dependency from an instrument artifact in the
|
||||
frame-based ratios." Instrument: ⟨capture⟩ × 3, `check_labels.py`
|
||||
(repaired) with diagnostics logged per run.
|
||||
* `check_labels.py` repaired in place (this commit) to stop conflating the
|
||||
two claims — no emulator needed for that part, verified by
|
||||
`--selftest` still detecting the seeded mislabel.
|
||||
* The sweep→plate separation constant stays withdrawn, as it already was.
|
||||
Nothing here reinstates it.
|
||||
@@ -119,8 +119,20 @@ def features(path, swap_plate=False):
|
||||
f['pulse_amp_err'] = 99.0
|
||||
return f
|
||||
|
||||
# ⚠️ Two different claims live here, and docs/re/f6-residue-shaping.md is why
|
||||
# they are no longer reported as one. IDENTITY checks test *which element you
|
||||
# are looking at* -- clock-free ratios internal to the sweep family, robust
|
||||
# out of sample (3-for-3 on `f6c`, the first capture not used to derive them).
|
||||
# TIMING checks test whether a cross-element phase or a self-consistency curve
|
||||
# fit holds to a specific number -- and out of sample, both failed (0-for-2 on
|
||||
# `f6c`), in a way the identity checks did not. Folding a TIMING failure into
|
||||
# "N LABEL(S) DRIFTED" reads as "the identification is wrong", which out-of-
|
||||
# sample evidence does not support; what may not hold is that the *timing
|
||||
# relationship* is a constant at all. See f6-residue-shaping.md before
|
||||
# tightening these tolerances -- they were already tuned on n=2 once.
|
||||
|
||||
# label -> (derived ratio, declared value, tolerance, what the label asserts)
|
||||
def checks(f):
|
||||
def identity_checks(f):
|
||||
return [
|
||||
("pteff03a is the 720-unit leaf (not a second copy of the 600)",
|
||||
f['pteff03a_period']/f['pteff03_period'], 720/600, 0.05),
|
||||
@@ -128,37 +140,51 @@ def checks(f):
|
||||
f['pteff03a_len']/f['pteff03_len'], 800/600, 0.06),
|
||||
("the pulsing slot is ptbtn00f (120-unit loop vs the sweep's 600 leaf units)",
|
||||
f['pulse_period']/f['pteff03_period'], 0.1, 0.05),
|
||||
]
|
||||
|
||||
def timing_checks(f):
|
||||
return [
|
||||
("the -0.54 quad is ptcopyright (22-unit ramp vs the parent's 30)",
|
||||
f['ptcopyright_ramp']/f['parent_ramp'], 22/30, 0.08),
|
||||
("the pulse AMPLITUDE matches ptbtn00f.rat's declared 8-key curve (peak 80)",
|
||||
f['pulse_amp_err'], 0.0, None),
|
||||
]
|
||||
|
||||
def run(swap=False):
|
||||
def _run_group(f, group, indent=" "):
|
||||
bad = 0
|
||||
for name, path in CAPS.items():
|
||||
f = features(path, swap_plate=swap)
|
||||
print(f" {name}:")
|
||||
for label, got, want, tol in checks(f):
|
||||
for label, got, want, tol in group(f):
|
||||
if tol is None: # absolute: mean |alpha| error, <=3 levels
|
||||
err, ok = got, got <= 3.0
|
||||
print(f" [{'PASS' if ok else 'FAIL'}] mean |alpha| error {got:5.2f} levels (tol 3.00, best lag {f.get('pulse_lag',0):.2f} fr) {label}")
|
||||
print(f"{indent}[{'PASS' if ok else 'FAIL'}] mean |alpha| error {got:5.2f} levels (tol 3.00, best lag {f.get('pulse_lag',0):.2f} fr) {label}")
|
||||
bad += not ok
|
||||
continue
|
||||
err = abs(got-want)/want
|
||||
ok = err <= tol
|
||||
bad += not ok
|
||||
print(f" [{'PASS' if ok else 'FAIL'}] {got:.4f} vs {want:.4f} ({err*100:4.1f}%, tol {tol*100:.0f}%) {label}")
|
||||
print(f"{indent}[{'PASS' if ok else 'FAIL'}] {got:.4f} vs {want:.4f} ({err*100:4.1f}%, tol {tol*100:.0f}%) {label}")
|
||||
return bad
|
||||
|
||||
def run(swap=False):
|
||||
id_bad, timing_bad = 0, 0
|
||||
for name, path in CAPS.items():
|
||||
f = features(path, swap_plate=swap)
|
||||
print(f" {name}:")
|
||||
print(f" identity (gates the exit code):")
|
||||
id_bad += _run_group(f, identity_checks, indent=" ")
|
||||
print(f" timing (reported, not gating -- see f6-residue-shaping.md):")
|
||||
timing_bad += _run_group(f, timing_checks, indent=" ")
|
||||
return id_bad, timing_bad
|
||||
|
||||
if __name__ == '__main__':
|
||||
if '--selftest' in sys.argv:
|
||||
print("SELFTEST — the plate label deliberately pointed at ptcopyright.")
|
||||
print("A check that cannot fail here would not have caught the real error.\n")
|
||||
bad = run(swap=True)
|
||||
id_bad, timing_bad = run(swap=True)
|
||||
bad = id_bad + timing_bad
|
||||
print(f"\n{'OK: mislabel detected' if bad else 'BROKEN: mislabel NOT detected'} ({bad} failures)")
|
||||
sys.exit(0 if bad else 1)
|
||||
print("Element labels in the F5/F6 findings, re-derived from captures + disc:\n")
|
||||
bad = run()
|
||||
print(f"\n{'all labels agree' if not bad else str(bad)+' LABEL(S) DRIFTED'}")
|
||||
sys.exit(1 if bad else 0)
|
||||
id_bad, timing_bad = run()
|
||||
print(f"\n{'all identity checks agree' if not id_bad else str(id_bad)+' IDENTITY CHECK(S) DRIFTED'}"
|
||||
f"; {timing_bad} timing check(s) failed (informational)")
|
||||
sys.exit(1 if id_bad else 0)
|
||||
|
||||
Reference in New Issue
Block a user