From 9405d71aaedb93f245b066219df7e7591fcd3e0f Mon Sep 17 00:00:00 2001 From: Sylpheed port agent Date: Mon, 31 Aug 2026 00:37:59 +0000 Subject: [PATCH] port: a hard negative found a coverage hole and two defects hiding each other The Decoder generalised my identity rule back at me -- a positive control that is merely 'high' hides the difference between an exact instrument and a lossy one -- and it landed on the band check I shipped yesterday. Its positive control was 0.29 and 0.66 dB, and small is not zero. Source against itself read 7.656 dB, larger than the number the check calls faithful: bands() applied the fold to one side only, correct for source-versus-transcode and wrong for source-versus-itself. The fold is per-side now and identity reads 0.000 dB exactly. The published 0.66 stands unchanged; what changed is that the instrument is known unbiased rather than assumed to be, and the scale's bottom is anchored. Same rule applied to the port's headline numbers: the image RMSE metric reads 0.0000 for a capture against itself and after a PNG round-trip, so 13.21 is real difference and not pipeline noise. verify-capture now asserts that before printing any row and refuses if it is not exact. Then their refutation attempt on 'band energies need no alignment'. It survives -- 1 s of misalignment costs 0.16 dB -- but 10 s costs 1.00 dB, so the claim is narrowed to robust, not free. Their real point: separation is material-dependent, two unrelated music banks separate by 5.28 dB where an unrelated movie gave me 19-20. A movie is an easy negative, so I built the hard one and it failed. A 6 kHz lowpass is caught on ADV at 4.27 dB, 2.8x, and NOT caught on S00A at 1.28 dB against a 1.5 dB threshold, because S00A's own 6-16 kHz content sits at -67 dB. A transcode that lost its whole top end would pass on S00A. Reported per asset as COVERED / NOT COVERED rather than asserted, and tracked in BLOCKED. Splitting the top band raised ADV from 2.58 to 4.27 dB. That is changing the instrument's resolution so it can see a failure it must see, driven by a control it failed -- the pass threshold is unchanged. Repairing it exposed two defects that had been hiding each other. return 0 was unconditional: making the difference path report-only swallowed the band verdict, so check-all's transcode-bands must-pass step could not fail -- an asserting step that asserts nothing, shipped by me one day after writing up the same shape in someone else's work. And the disqualified difference path was still voting on the exit code, so fixing the return turned the run red for the wrong reason. Neither would have surfaced without a control the tool could fail. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01N7FiFFFwbvG2uxdcEh8HyF --- docs/port/BLOCKED.md | 6 ++ docs/port/DECISIONS.md | 101 +++++++++++++++++++++++++- tools/port/verify-capture | 32 ++++++++ tools/port/verify-transcode-fidelity | 105 +++++++++++++++++++++++++-- 4 files changed, 235 insertions(+), 9 deletions(-) diff --git a/docs/port/BLOCKED.md b/docs/port/BLOCKED.md index c2809ef1..afaf8d92 100644 --- a/docs/port/BLOCKED.md +++ b/docs/port/BLOCKED.md @@ -147,6 +147,12 @@ HANDOFF. | ~~P1–P7 — the keyframe record layout~~ | ~~adopt the corrected pose/time pairing~~ | — | ✅ **ADOPTED 2026-08-29 by pinning `formats-pin-2026-08-29c`.** This row was wrong twice: it said the change *"cannot be taken yet"* and that it *"reaches the port only when that branch lands on `main`"*. **It arrives when the tag is pinned**, which is what MISSION §2's tagging rule exists for. ⚠️ And the knob I tested first, `SYLPHEED_KF_TIME_SHIFT`, is a **retired partial fix** that left pose 0 untimed — the real correction is the tagged crate's default, with the old reading behind `SYLPHEED_KF_TIME_LEGACY=1`. **The blast radius was far smaller than this row predicted**: under the correction *every pose is timed* (866 keyframes, 0 untimed), so `pose_at`'s synthetic-exit branch became dead code rather than wrong code and nothing needed re-deriving. Oracle: `publisher_logo` 1.00 %→**0.75 %**, `developer_logos` 0.39 %→**0.33 %**, `extras`' differing region collapsing from 736×525 to **398×295 at the sweep position**. 🔴 Open cost: `sylpheed-cli` builds from the workspace crate, so `verify-screen` compares two decoder eras until the tag reaches `main`. Revert to the path dependency then. | | ~~P7 / naming — the four unnamed builds~~ | ~~which locale and variant is each of entries 0, 1, 12, 15?~~ | — | ✅ **answered 2026-08-29** (`docs/re/ui-title-build-map.md`): all four are the loading screen, two variants — plain (7 elements) and dressed (10) — decoded from their own `pgloading_*` element names. ⚠️ **Not adopted as names yet, for two reasons the RE agent gave and one the port found.** Theirs: the executable names exactly two, and *which* bundle takes which name is 🟡 undecided, so `LOADING`/`LOADING2` must not go in an asset path; and locale is 🟡 — the English member of a pair is the one in the first half of `GP_TITLE.p00`, 8/8 structurally but only 3/3 where a capture can check, and the three pairs that matter are the three no capture can check. Mine: **the message gives the bundles as "0/1 and 10/11", which is the `is_build` ordinal, and `authored/screen_names.json` is keyed by PAK ENTRY** — in entry space 10 and 11 are `palogo_sqex` and `palogo_gamearts`, the splashes. See the refutation section in `DECISIONS.md`. | +## Coverage hole in my own check, 2026-08-31 — derived from HANDOFF `0159527` + +| Milestone | Needs | HANDOFF | State | +|---|---|---|---| +| P4/P7 — band check sensitivity | **nothing from anybody; my check does not cover `S00A`'s top end** | `0159527` | 🔴 **MEASURED GAP, reported per asset rather than hidden or asserted.** A 6 kHz-lowpassed source — a transcode that lost its whole top end — deviates **4.27 dB on `ADV` (covered, 2.8×)** and **1.28 dB on `S00A`, under the 1.5 dB pass threshold**, because `S00A`'s own 6–16 kHz content sits at −67 dB. **So that failure would pass on `S00A`.** Found by building the *hard* negative after the Decoder measured that unrelated music banks separate by 5.28 dB where an unrelated movie gave me 19–20 — a movie is an easy negative. Splitting the top band raised `ADV` from 2.58 to 4.27 dB; the pass threshold is unchanged. ⚠️ Not asserted, because a permanently red suite on a gap I cannot close today helps nobody; printed as **COVERED / NOT COVERED** per asset so it cannot become scenery. | + ## Open on my own side, 2026-08-30 — derived from HANDOFF `0159527` | Milestone | Needs | HANDOFF | State | diff --git a/docs/port/DECISIONS.md b/docs/port/DECISIONS.md index 455c3bc9..c0d23793 100644 --- a/docs/port/DECISIONS.md +++ b/docs/port/DECISIONS.md @@ -9,7 +9,7 @@ dies, which is what this file is for. -261 sections. Search this before re-deriving anything. +263 sections. Search this before re-deriving anything. * [P0 — the exporter, 2026-08-28](#p0--the-exporter-2026-08-28) * [P1 — Godot draws the screen, 2026-08-28](#p1--godot-draws-the-screen-2026-08-28) @@ -272,6 +272,8 @@ dies, which is what this file is for. * [🔴 My seek trap was over-general — the Decoder narrowed it](#my-seek-trap-was-over-general--the-decoder-narrowed-it) * [A capital letter hid a refuted claim in the file whose job is to say what is open](#a-capital-letter-hid-a-refuted-claim-in-the-file-whose-job-is-to-say-what-is-open) * [The difference path cannot verify a lossless encode — so nothing it says counts](#the-difference-path-cannot-verify-a-lossless-encode--so-nothing-it-says-counts) +* [The identity rule, turned back on my own newest tool — and it was biased](#the-identity-rule-turned-back-on-my-own-newest-tool--and-it-was-biased) +* [Their refutation attempt on my band check found a coverage hole and two defects](#their-refutation-attempt-on-my-band-check-found-a-coverage-hole-and-two-defects) ## P0 — the exporter, 2026-08-28 @@ -13440,3 +13442,100 @@ is unaffected and still asserts. 📌 What this changes about the milestone: **P4's waveform question is not merely open, it is open with a disqualified instrument.** That is worse than it looked yesterday and better than believing 8.73 dB meant something. + +## The identity rule, turned back on my own newest tool — and it was biased + +The Decoder ran my identity rule against their coherence estimator, it passed, and +they returned a sharper form of it: **a positive control that is merely "high" +hides the difference between an exact instrument and a lossy one.** Theirs read +0.94 for two reasons at once — a correct estimator *plus* a windowed delay — and +only the identity case could separate them. + +📌 **That lands on the band check I shipped yesterday and asserted in +`check-all`.** Its positive control was **0.29 and 0.66 dB** — small, and *small +is not zero*. A systematic bias would sit inside 0.66 dB looking like a pass. + +Adding source-against-itself: **7.656 dB.** Larger than the number the check calls +faithful. + +### The bias was in the control's construction, not the measurement + +`bands()` applies the fold to the **left side only**, which is correct for the +real comparison — a 5.1 source needs folding, an already-stereo transcode does +not. Applied to source-against-itself, that same asymmetry compares a folded +signal against a raw six-channel average. The fold is now per-side, and identity +reads **0.000 dB, exact**. + +✅ **The published 0.66 dB is unchanged** — that comparison was always +asymmetric-by-design and remains correct. What changed is that the instrument is +now **known unbiased** rather than assumed to be, and the check has three +separated populations instead of two: + +| | | +|---|---| +| identity | **0.000 dB — exact** | +| source vs its own transcode | 0.29 / 0.66 dB | +| source vs an unrelated movie | 19.10 / 20.02 dB | + +**The bottom of that scale is now anchored rather than inferred**, which is the +whole difference between "0.66 is small" and "0.66 is small *compared with zero, +measured*". + +📌 The general rule, now stated in the form that catches both our cases: **a +control that establishes only an upper bound on error cannot distinguish an exact +instrument from a slightly wrong one — and "slightly wrong" is the interesting +failure, because it passes.** Mine was one function argument. Theirs was one line. +Both were available from the day the tool was written. + +## Their refutation attempt on my band check found a coverage hole and two defects + +They tried to refute *"band energies need no alignment"*. **It survives** — 1 s of +misalignment costs 0.16 dB, well inside the pass band — but they narrowed it +correctly: at **10 s the cost reaches 1.00 dB**, because a fixed analysis window +covers different material once the shift is large. *"Needs no alignment"* was my +wording and it was too strong; the tool now says **robust to misalignment, not +free of it**. + +Their second point is the one that mattered: **the separation margin is +material-dependent.** Two unrelated music banks separate by only 5.28 dB where an +unrelated movie gave me 19–20. **A movie is a very easy negative.** So I built the +*hard* one — the failure this check exists to catch — and it failed. + +### 🔴 A 6 kHz-lowpassed source: `ADV` caught it, `S00A` does not + +| | worst band deviation | +|---|---| +| real transcodes | 0.29 / 0.67 dB | +| **6 kHz lowpass, `ADV`** | **4.27 dB — covered, 2.8×** | +| **6 kHz lowpass, `S00A`** | **1.28 dB — NOT COVERED, under the 1.5 dB threshold** | +| unrelated movie | 21.78 / 22.55 dB | + +**A transcode that lost everything above 6 kHz would pass this check on `S00A`**, +because `S00A`'s own 6–16 kHz content sits at −67 dB — removing it changes almost +nothing. The check's sensitivity is a property of the *material*, which is the +Decoder's negative-separation finding arriving on the positive side. + +📌 On the way, splitting the top band raised `ADV`'s detection from 2.58 to +4.27 dB. **That is changing the instrument's resolution so it can see a failure it +must see, driven by a control it failed — not loosening the pass threshold**, +which is unchanged. The distinction is the whole difference between fixing an +instrument and fitting one. + +⚠️ Reported per asset as **COVERED / NOT COVERED** rather than asserted: making +the suite permanently red on a gap I cannot close today helps nobody, and hiding +it turns a coverage hole into scenery. Tracked in `BLOCKED.md`. + +### 🔴 And repairing it exposed two defects that had been hiding each other + +* **`return 0` was unconditional.** Making the difference path report-only + swallowed the band verdict with it, so `check-all`'s `transcode-bands + must-pass` step **could not fail** — an asserting step that asserts nothing, + shipped by me, one day after I wrote up the same shape in someone else's work. + Band failures were being printed and discarded. +* **The disqualified difference path was still voting on the exit code.** Fixing + the return turned the run red for that reason rather than the real one. + +📌 **Two defects hiding each other**: with the return broken, the voting bug was +invisible; with the voting bug present, fixing the return produced a red run for +the wrong cause. Neither would have surfaced without building a control the tool +could fail — which is the argument for hard negatives in one line. diff --git a/tools/port/verify-capture b/tools/port/verify-capture index 2e1dbe3e..3d11a038 100755 --- a/tools/port/verify-capture +++ b/tools/port/verify-capture @@ -138,6 +138,38 @@ echo "RMSE is reported and is NOT a target: the capture carries the game's own" echo "tone ramp, so it has a floor. What finds a real defect is the DIFFERING" echo "REGION -- a missing or misplaced element is a large connected blob." echo +# 🔴 THE METRIC'S OWN ZERO, asserted before any row is printed. +# +# Every number below is "small is good", and this file already says the RMSE has +# a floor from the game's tone ramp. What was never established is the floor of +# the COMPARISON ITSELF. A control that only bounds error from above cannot tell +# an exact instrument from a slightly wrong one -- and slightly wrong is the +# failure that passes. The Decoder reached that form of it after their coherence +# estimator's positive control read 0.94 for two reasons at once. +# +# Measured here rather than assumed: a capture against itself, and against a PNG +# round-trip of itself, must both be EXACTLY 0. If they are not, the metric has a +# bias and no row below means what it says. +_ctl="" +for row in "${MAP[@]}"; do + IFS=: read -r _n _c _rest <<<"$row"; [ -f "$_c" ] && { _ctl="$_c"; break; } +done +if [ -n "$_ctl" ]; then + _rt="${TMPDIR:-/tmp}/verify-capture-rt.png"; convert "$_ctl" -quality 100 "$_rt" + for _pair in "$_ctl|$_ctl|identity" "$_ctl|$_rt|PNG round-trip"; do + IFS='|' read -r _a _b _lab <<<"$_pair" + _d=$(convert "$_a" "$_b" -metric RMSE -compare -format "%[distortion]" info: 2>&1 | tail -1) + _v=$(python3 -c "print('%.4f' % (float('$_d')*255))" 2>/dev/null || echo "?") + if [ "$_v" = "0.0000" ]; then + printf ' metric control, %-16s RMSE %s -- exact\n' "$_lab:" "$_v" + else + printf ' 🔴 metric control, %-13s RMSE %s -- NOT ZERO. The comparison is\n' "$_lab:" "$_v" + echo " biased and every row below is unreadable. Refusing." + exit 3 + fi + done + echo +fi printf '%-17s %-9s %-7s %-22s %s\n' screen raw-rmse diff region note for row in "${MAP[@]}"; do IFS=: read -r name cap pose forced capcrop <<<"$row" diff --git a/tools/port/verify-transcode-fidelity b/tools/port/verify-transcode-fidelity index 1d79415c..e7033422 100755 --- a/tools/port/verify-transcode-fidelity +++ b/tools/port/verify-transcode-fidelity @@ -153,7 +153,17 @@ def align(src, dst, af): return int(round(coarse * RATE)), c -BANDS = [(0, 500), (500, 2000), (2000, 6000), (6000, 16000)] +# 🔴 THE TOP BAND IS SPLIT BECAUSE THE NEAR-MISS CONTROL FAILED. With a single +# 6-16 kHz band, a 6 kHz-lowpassed source -- a transcode that lost its whole top +# end, the failure this check exists to catch -- deviated by only 2.58 dB and +# would have PASSED. The band was wide enough to average the loss away against +# the filter's transition region. +# +# ⚠️ This is changing the instrument's RESOLUTION so it can see a failure it must +# see, driven by a control it failed. It is NOT loosening the pass threshold for +# the real comparison, which is unchanged -- that would be tuning until the +# answer came out right, which is the thing this project keeps catching. +BANDS = [(0, 500), (500, 2000), (2000, 6000), (6000, 10000), (10000, 16000)] def band_db(path, af, lo, hi, seconds=25.0, skip=2.0): @@ -188,13 +198,28 @@ def band_db(path, af, lo, hi, seconds=25.0, skip=2.0): return float(m.group(1)) -def bands(src, dst, af, label): - """Per-band level, source against transcode. No alignment involved.""" +def bands(src, dst, af, label, af_dst=None): + """Per-band level, source against transcode. ROBUST to misalignment, not free of it. + + ⚠️ CLAIM NARROWED 2026-08-31 after the Decoder tried to refute it. It survives + -- **1 s of misalignment costs 0.16 dB**, well inside the 1.5 dB pass band -- + but it is **not literally alignment-free**: at **10 s the cost reaches 1.00 dB**, + because a fixed analysis window covers different material once the shift is + large relative to it. "Needs no alignment" was my wording and it was too + strong; the honest claim is robustness up to a few seconds. + + 🔴 THE FOLD IS PER-SIDE, and the identity control is what made that + necessary. `af` applies to the LEFT side only, which is correct for the real + comparison -- a 5.1 source needs folding, an already-stereo transcode does + not. Applying that same asymmetry to source-against-itself compares a folded + signal with a raw six-channel average and reports **7.656 dB on an + identity**, larger than the 0.66 dB this check calls a pass. + """ print(f" {label}") worst = 0.0 for lo, hi in BANDS: a = band_db(src, af, lo, hi) - b = band_db(dst, None, lo, hi) + b = band_db(dst, af_dst, lo, hi) if a is None or b is None: print(f" {lo:>5}-{hi:<5} Hz one side silent -- no comparison") continue @@ -283,7 +308,7 @@ def main(): fail += 1 continue af = downmix_of(man, name) - worst = bands(src, dst, af, f"{name} -- band energies (needs no alignment)") + worst = bands(src, dst, af, f"{name} -- band energies (robust to misalignment, not free of it)") verdict = "ok" if worst <= PASS_BAND_DB else "🔴 OUT OF TOLERANCE" print(f" worst band deviation {worst:.2f} dB {verdict}") if worst > PASS_BAND_DB: @@ -292,6 +317,59 @@ def main(): # that has only ever seen a faithful pair cannot be told from one that # compares a file with itself by accident -- and this tool has already # produced four confident wrong numbers on the other quantity. + # 🔴 THE IDENTITY CONTROL, added 2026-08-31 after the Decoder generalised + # my own rule back at me: **a positive control that is merely "high" + # hides the difference between an exact instrument and a lossy one.** + # This check's positive side was 0.29 and 0.66 dB -- small, and small is + # not zero. A systematic bias (the fold applied to one side only, a + # different window, a resampler difference) would sit inside 0.66 dB + # while looking like a pass. Source against itself must be EXACTLY 0.00 + # in every band, and anything else is the instrument, not the transcode. + ident = bands(src, src, af, " control: source vs ITSELF, must be exact", af_dst=af) + idv = "ok" if ident == 0.0 else f"🔴 {ident:.3f} dB on an identity -- the instrument is biased" + print(f" worst band deviation {ident:.3f} dB {idv}") + if ident != 0.0: + fail += 1 + # 🔴 A NEAR-MISS NEGATIVE, because an unrelated movie is an EASY one. + # The Decoder measured two unrelated music BANKS separating by just + # 5.28 dB where an unrelated movie gave me 19-20, so the margin against a + # hard negative is 8x, not 30x. The negative that matters is the failure + # this check exists to catch: a transcode that lost its top end. A 6 kHz + # lowpass of the source is that failure, constructed. + low = bands(src, src, af, " control: 6 kHz-lowpassed source, must be caught", + af_dst=(af + "," if af else "") + "lowpass=f=6000") + # Judged against THE CHECK'S OWN pass threshold, not an invented 3x. + # + # With the top band split this lands at 4.27 dB: it fails the 1.5 dB pass + # test, so the check does catch it -- but by 2.8x, against the 6.4x it + # has over the worst real transcode (0.67 dB). ⚠️ NOT COMFORTABLE, and + # said out loud rather than smoothed: a loss milder than a 6 kHz brick + # wall could sit between 0.67 and 1.5 and pass. The honest statement is + # that this check catches a SEVERE top-end loss and is not characterised + # for a mild one. + # + # The 3x bar it used to be judged against was mine and stricter than the + # check itself; using the check's own threshold is the principled + # criterion, and lowering the 3x to make a failing control pass would + # have been tuning. + # 🔴 REPORTED PER ASSET, NOT ASSERTED, and the reason is a measured gap + # rather than convenience. `ADV` catches the lowpass by 2.8x. **`S00A` + # does not catch it at all** -- 1.28 dB against a 1.5 dB threshold -- + # because its own 6-16 kHz content sits at -67 dB, so removing it changes + # almost nothing. The check's sensitivity is MATERIAL-DEPENDENT, which is + # the Decoder's finding about negative-separation arriving on the + # positive side. + # + # Asserting it would make the suite permanently red on a gap I cannot + # close today; hiding it would make a coverage hole into scenery. So it + # prints COVERED / NOT COVERED per asset and the gap is tracked in + # BLOCKED.md. The identity and unrelated-movie controls still assert. + if low > PASS_BAND_DB: + print(f" worst band deviation {low:.2f} dB COVERED, caught by" + f" {low / PASS_BAND_DB:.1f}x") + else: + print(f" worst band deviation {low:.2f} dB 🔴 NOT COVERED --" + f" a 6 kHz top-end loss on {name} would PASS this check") other = [v for v in man["videos"] if v["name"] != name] if other: osrc = os.path.join("export", other[0]["file"]) @@ -304,9 +382,11 @@ def main(): if af is None: print(f" {name:<28} ⚠️ no `-af` in the recorded command: the source" " is stereo, comparing without a fold") - down = compare(src, dst, af, name) - if down is None or down < PASS_DB: - fail += 1 + # Report-only: a disqualified path must not vote on the exit code. It + # did, which is why the run went red for the wrong reason the moment the + # return was fixed -- two defects hiding each other, and repairing one + # exposed the other rather than the run going quietly green. + compare(src, dst, af, name) if control: print(f" known negatives for {name}:") same = compare(src, src, af, " source vs itself") @@ -337,6 +417,15 @@ def main(): print(" What this run DOES establish is the trap list below, each reproduced") print(" here rather than reasoned about. See docs/port/DECISIONS.md.") print(" 🔴 It measures AUDIO only; `-q:v 8` was chosen on SSIM separately.") + # 🔴 THIS RETURN WAS UNCONDITIONAL `return 0` FOR A DAY. Making the difference + # path report-only swallowed the band verdict with it, so `check-all`'s + # `transcode-bands must-pass` step COULD NOT FAIL -- an asserting step that + # asserts nothing, which is the exact shape this project keeps finding in + # other people's work and had now shipped in mine. The band failures were + # being printed and discarded. + if fail: + print(f"\n🔴 {fail} band control failure(s)") + return 1 return 0