fabi/Sylpheed
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

port: a hard negative found a coverage hole and two defects hiding each other

Browse Source 
The Decoder generalised my identity rule back at me -- a positive control that is
merely 'high' hides the difference between an exact instrument and a lossy one --
and it landed on the band check I shipped yesterday. Its positive control was 0.29
and 0.66 dB, and small is not zero. Source against itself read 7.656 dB, larger
than the number the check calls faithful: bands() applied the fold to one side
only, correct for source-versus-transcode and wrong for source-versus-itself. The
fold is per-side now and identity reads 0.000 dB exactly. The published 0.66 stands
unchanged; what changed is that the instrument is known unbiased rather than
assumed to be, and the scale's bottom is anchored.

Same rule applied to the port's headline numbers: the image RMSE metric reads
0.0000 for a capture against itself and after a PNG round-trip, so 13.21 is real
difference and not pipeline noise. verify-capture now asserts that before printing
any row and refuses if it is not exact.

Then their refutation attempt on 'band energies need no alignment'. It survives --
1 s of misalignment costs 0.16 dB -- but 10 s costs 1.00 dB, so the claim is
narrowed to robust, not free. Their real point: separation is material-dependent,
two unrelated music banks separate by 5.28 dB where an unrelated movie gave me
19-20. A movie is an easy negative, so I built the hard one and it failed. A 6 kHz
lowpass is caught on ADV at 4.27 dB, 2.8x, and NOT caught on S00A at 1.28 dB
against a 1.5 dB threshold, because S00A's own 6-16 kHz content sits at -67 dB. A
transcode that lost its whole top end would pass on S00A. Reported per asset as
COVERED / NOT COVERED rather than asserted, and tracked in BLOCKED.

Splitting the top band raised ADV from 2.58 to 4.27 dB. That is changing the
instrument's resolution so it can see a failure it must see, driven by a control
it failed -- the pass threshold is unchanged.

Repairing it exposed two defects that had been hiding each other. return 0 was
unconditional: making the difference path report-only swallowed the band verdict,
so check-all's transcode-bands must-pass step could not fail -- an asserting step
that asserts nothing, shipped by me one day after writing up the same shape in
someone else's work. And the disqualified difference path was still voting on the
exit code, so fixing the return turned the run red for the wrong reason. Neither
would have surfaced without a control the tool could fail.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N7FiFFFwbvG2uxdcEh8HyF
This commit is contained in:
Sylpheed port agent
2026-08-31 00:37:59 +00:00
parent 35bf469172
commit aedcd35eef
4 changed files with 235 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
tools/port/verify-capture
View File

@@ -138,6 +138,38 @@ echo "RMSE is reported and is NOT a target: the capture carries the game's own"
echo "tone ramp, so it has a floor. What finds a real defect is the DIFFERING"
echo "REGION -- a missing or misplaced element is a large connected blob."
echo
# 🔴 THE METRIC'S OWN ZERO, asserted before any row is printed.
#
# Every number below is "small is good", and this file already says the RMSE has
# a floor from the game's tone ramp. What was never established is the floor of
# the COMPARISON ITSELF. A control that only bounds error from above cannot tell
# an exact instrument from a slightly wrong one -- and slightly wrong is the
# failure that passes. The Decoder reached that form of it after their coherence
# estimator's positive control read 0.94 for two reasons at once.
#
# Measured here rather than assumed: a capture against itself, and against a PNG
# round-trip of itself, must both be EXACTLY 0. If they are not, the metric has a
# bias and no row below means what it says.
_ctl=""
for row in "${MAP[@]}"; do
IFS=: read -r _n _c _rest <<<"$row"; [ -f "$_c" ] && { _ctl="$_c"; break; }
done
if [ -n "$_ctl" ]; then
_rt="${TMPDIR:-/tmp}/verify-capture-rt.png"; convert "$_ctl" -quality 100 "$_rt"
for _pair in "$_ctl|$_ctl|identity" "$_ctl|$_rt|PNG round-trip"; do
IFS='|' read -r _a _b _lab <<<"$_pair"
_d=$(convert "$_a" "$_b" -metric RMSE -compare -format "%[distortion]" info: 2>&1 | tail -1)
_v=$(python3 -c "print('%.4f' % (float('$_d')*255))" 2>/dev/null || echo "?")
if [ "$_v" = "0.0000" ]; then
printf ' metric control, %-16s RMSE %s -- exact\n' "$_lab:" "$_v"
else
printf ' 🔴 metric control, %-13s RMSE %s -- NOT ZERO. The comparison is\n' "$_lab:" "$_v"
echo " biased and every row below is unreadable. Refusing."
exit 3
fi
done
echo
fi
printf '%-17s %-9s %-7s %-22s %s\n' screen raw-rmse diff region note
for row in "${MAP[@]}"; do
IFS=: read -r name cap pose forced capcrop <<<"$row"

105
tools/port/verify-transcode-fidelity
View File

@@ -153,7 +153,17 @@ def align(src, dst, af):
return int(round(coarse * RATE)), c
BANDS = [(0, 500), (500, 2000), (2000, 6000), (6000, 16000)]
# 🔴 THE TOP BAND IS SPLIT BECAUSE THE NEAR-MISS CONTROL FAILED. With a single
# 6-16 kHz band, a 6 kHz-lowpassed source -- a transcode that lost its whole top
# end, the failure this check exists to catch -- deviated by only 2.58 dB and
# would have PASSED. The band was wide enough to average the loss away against
# the filter's transition region.
#
# ⚠️ This is changing the instrument's RESOLUTION so it can see a failure it must
# see, driven by a control it failed. It is NOT loosening the pass threshold for
# the real comparison, which is unchanged -- that would be tuning until the
# answer came out right, which is the thing this project keeps catching.
BANDS = [(0, 500), (500, 2000), (2000, 6000), (6000, 10000), (10000, 16000)]
def band_db(path, af, lo, hi, seconds=25.0, skip=2.0):
@@ -188,13 +198,28 @@ def band_db(path, af, lo, hi, seconds=25.0, skip=2.0):
return float(m.group(1))
def bands(src, dst, af, label):
"""Per-band level, source against transcode. No alignment involved."""
def bands(src, dst, af, label, af_dst=None):
"""Per-band level, source against transcode. ROBUST to misalignment, not free of it.
⚠️ CLAIM NARROWED 2026-08-31 after the Decoder tried to refute it. It survives
-- **1 s of misalignment costs 0.16 dB**, well inside the 1.5 dB pass band --
but it is **not literally alignment-free**: at **10 s the cost reaches 1.00 dB**,
because a fixed analysis window covers different material once the shift is
large relative to it. "Needs no alignment" was my wording and it was too
strong; the honest claim is robustness up to a few seconds.
🔴 THE FOLD IS PER-SIDE, and the identity control is what made that
necessary. `af` applies to the LEFT side only, which is correct for the real
comparison -- a 5.1 source needs folding, an already-stereo transcode does
not. Applying that same asymmetry to source-against-itself compares a folded
signal with a raw six-channel average and reports **7.656 dB on an
identity**, larger than the 0.66 dB this check calls a pass.
"""
print(f" {label}")
worst = 0.0
for lo, hi in BANDS:
a = band_db(src, af, lo, hi)
b = band_db(dst, None, lo, hi)
b = band_db(dst, af_dst, lo, hi)
if a is None or b is None:
print(f" {lo:>5}-{hi:<5} Hz one side silent -- no comparison")
continue
@@ -283,7 +308,7 @@ def main():
fail += 1
continue
af = downmix_of(man, name)
worst = bands(src, dst, af, f"{name} -- band energies (needs no alignment)")
worst = bands(src, dst, af, f"{name} -- band energies (robust to misalignment, not free of it)")
verdict = "ok" if worst <= PASS_BAND_DB else "🔴 OUT OF TOLERANCE"
print(f" worst band deviation {worst:.2f} dB {verdict}")
if worst > PASS_BAND_DB:
@@ -292,6 +317,59 @@ def main():
# that has only ever seen a faithful pair cannot be told from one that
# compares a file with itself by accident -- and this tool has already
# produced four confident wrong numbers on the other quantity.
# 🔴 THE IDENTITY CONTROL, added 2026-08-31 after the Decoder generalised
# my own rule back at me: **a positive control that is merely "high"
# hides the difference between an exact instrument and a lossy one.**
# This check's positive side was 0.29 and 0.66 dB -- small, and small is
# not zero. A systematic bias (the fold applied to one side only, a
# different window, a resampler difference) would sit inside 0.66 dB
# while looking like a pass. Source against itself must be EXACTLY 0.00
# in every band, and anything else is the instrument, not the transcode.
ident = bands(src, src, af, " control: source vs ITSELF, must be exact", af_dst=af)
idv = "ok" if ident == 0.0 else f"🔴 {ident:.3f} dB on an identity -- the instrument is biased"
print(f" worst band deviation {ident:.3f} dB {idv}")
if ident != 0.0:
fail += 1
# 🔴 A NEAR-MISS NEGATIVE, because an unrelated movie is an EASY one.
# The Decoder measured two unrelated music BANKS separating by just
# 5.28 dB where an unrelated movie gave me 19-20, so the margin against a
# hard negative is 8x, not 30x. The negative that matters is the failure
# this check exists to catch: a transcode that lost its top end. A 6 kHz
# lowpass of the source is that failure, constructed.
low = bands(src, src, af, " control: 6 kHz-lowpassed source, must be caught",
af_dst=(af + "," if af else "") + "lowpass=f=6000")
# Judged against THE CHECK'S OWN pass threshold, not an invented 3x.
#
# With the top band split this lands at 4.27 dB: it fails the 1.5 dB pass
# test, so the check does catch it -- but by 2.8x, against the 6.4x it
# has over the worst real transcode (0.67 dB). ⚠️ NOT COMFORTABLE, and
# said out loud rather than smoothed: a loss milder than a 6 kHz brick
# wall could sit between 0.67 and 1.5 and pass. The honest statement is
# that this check catches a SEVERE top-end loss and is not characterised
# for a mild one.
#
# The 3x bar it used to be judged against was mine and stricter than the
# check itself; using the check's own threshold is the principled
# criterion, and lowering the 3x to make a failing control pass would
# have been tuning.
# 🔴 REPORTED PER ASSET, NOT ASSERTED, and the reason is a measured gap
# rather than convenience. `ADV` catches the lowpass by 2.8x. **`S00A`
# does not catch it at all** -- 1.28 dB against a 1.5 dB threshold --
# because its own 6-16 kHz content sits at -67 dB, so removing it changes
# almost nothing. The check's sensitivity is MATERIAL-DEPENDENT, which is
# the Decoder's finding about negative-separation arriving on the
# positive side.
#
# Asserting it would make the suite permanently red on a gap I cannot
# close today; hiding it would make a coverage hole into scenery. So it
# prints COVERED / NOT COVERED per asset and the gap is tracked in
# BLOCKED.md. The identity and unrelated-movie controls still assert.
if low > PASS_BAND_DB:
print(f" worst band deviation {low:.2f} dB COVERED, caught by"
f" {low / PASS_BAND_DB:.1f}x")
else:
print(f" worst band deviation {low:.2f} dB 🔴 NOT COVERED --"
f" a 6 kHz top-end loss on {name} would PASS this check")
other = [v for v in man["videos"] if v["name"] != name]
if other:
osrc = os.path.join("export", other[0]["file"])
@@ -304,9 +382,11 @@ def main():
if af is None:
print(f" {name:<28} ⚠️ no `-af` in the recorded command: the source"
" is stereo, comparing without a fold")
down = compare(src, dst, af, name)
if down is None or down < PASS_DB:
fail += 1
# Report-only: a disqualified path must not vote on the exit code. It
# did, which is why the run went red for the wrong reason the moment the
# return was fixed -- two defects hiding each other, and repairing one
# exposed the other rather than the run going quietly green.
compare(src, dst, af, name)
if control:
print(f" known negatives for {name}:")
same = compare(src, src, af, " source vs itself")
@@ -337,6 +417,15 @@ def main():
print(" What this run DOES establish is the trap list below, each reproduced")
print(" here rather than reasoned about. See docs/port/DECISIONS.md.")
print(" 🔴 It measures AUDIO only; `-q:v 8` was chosen on SSIM separately.")
# 🔴 THIS RETURN WAS UNCONDITIONAL `return 0` FOR A DAY. Making the difference
# path report-only swallowed the band verdict with it, so `check-all`'s
# `transcode-bands must-pass` step COULD NOT FAIL -- an asserting step that
# asserts nothing, which is the exact shape this project keeps finding in
# other people's work and had now shipped in mine. The band failures were
# being printed and discarded.
if fail:
print(f"\n🔴 {fail} band control failure(s)")
return 1
return 0