port: verify-menu-audio could not FAIL -- the verdict never reached an exit code

It computed ok, printed a red line when a cue was silent, and the python had no
exit path at all, so it returned 0 every time while check-all registered it
must-pass. A cue could stop sounding and the suite would print the failure and
stay green.

This project's recurring defect one level up: not an instrument sitting below
the thing under test, but one that SEES the failure and does not report it.

The diagonal now exits. The no-op-silence and cue-order lines stay REPORTED --
both carry documented cross-run instability (whole-buffer recording shifts; a
0.15 margin this file's own comments show reaching 0.109 on a sounding cue), and
making either binding would produce red on correct audio.

--control feeds it a walk in which move never sounded, built from the tool's own
suppression machinery, and requires it to fail.
This commit is contained in:
Sylpheed port agent
2026-09-02 16:31:17 +00:00
parent efa085a668
commit 94506ec247

View File

@@ -1,7 +1,18 @@
#!/usr/bin/env bash
# Does the port actually MAKE SOUND on the P5 walk, and the RIGHT sound?
#
# tools/port/verify-menu-audio
# tools/port/verify-menu-audio # assert
# tools/port/verify-menu-audio --control # can it fail?
#
# 🔴 FOR WEEKS THIS COULD NOT FAIL. It computed the verdict, printed a red line
# when a cue was silent -- and the python had NO EXIT PATH, so it returned 0
# every time while `check-all` registered it `must-pass`. A cue could stop
# sounding and the suite would print the failure and stay green.
#
# That is this project's recurring defect one level up: not an instrument that
# sits below the thing under test, but an instrument that SEES the failure and
# does not report it. Ask of any check: what would this still report if the
# feature were absent -- AND what would it EXIT?
#
# This is the P6 gate check. P6's gate is "sound on the P5 gate", and until this
# existed the only evidence for it was that `audio.play("move")` appears in
@@ -29,6 +40,7 @@ set -euo pipefail
cd "${PROJECT_DIR:-/work}"
export DISPLAY="${DISPLAY:-:97}"
OUT="${OUT:-${TMPDIR:-/tmp}/verify-menu-audio}"
CONTROL=0; [ "${1:-}" = "--control" ] && CONTROL=1
mkdir -p "$OUT"
[ -d port/.godot ] || godot --headless --path port --import >/dev/null 2>&1
@@ -72,7 +84,18 @@ for c in move confirm back; do
exit 2; }
done
python3 - "$OUT" <<'PYEOF'
# THE CONTROL. Replace the walk with the run that already had `move` silenced, so
# the cue is genuinely missing from the baseline. Silencing it again can then
# remove nothing, the diagonal cannot drop, and the check MUST fail. Built from
# the tool's OWN suppression machinery rather than a second mechanism -- a
# control built a different way tests the control, not the check.
if [ $CONTROL -eq 1 ]; then
cp "$OUT/sup_move.wav" "$OUT/walk.wav"
echo "control: analysing a walk in which \`move\` never sounded"
fi
rc=0
python3 - "$OUT" <<'PYEOF' || rc=$?
import array, math, subprocess, sys
O = sys.argv[1]; SR = 44100
def dec(src, dst):
@@ -184,6 +207,14 @@ for row in ("move", "confirm", "back"):
print(" 🔴 silencing %s did not remove %s -- that cue is NOT SOUNDING" % (row, row))
print(" => %s" % ("all three cues SOUND: silencing each one collapses its own signal"
if ok else "at least one cue is not sounding"))
# 🔴 THE VERDICT EXITS. Everything below this line is REPORTED, not asserted, and
# deliberately so: the no-op-silence line and the cue-order line both carry
# DOCUMENTED cross-run instability (whole-buffer recording shifts; a 0.15 margin
# this file's own comments show going to 0.109 on a sounding cue). Making either
# binding would produce red on correct audio, which is how a suite gets ignored.
# The diagonal has no threshold to drift: silencing a cue either removes its own
# signal or it was never there.
VERDICT_FAILED = not ok
# 🔴 THE VERDICT IS THE DIAGONAL ONLY, and the first version of this asserted the
# off-diagonal too -- "silencing a cue must not move the others". That failed, and
# the material is why: `confirm` lands at 1.12 s and `back` at 1.21 s, 0.09 s apart
@@ -198,5 +229,18 @@ times = [t for _, t, hit in found if hit]
print("cue order vs script order : %s"
% ("CONSISTENT" if times == sorted(times) and len(times) == 3
else "check %s" % [(c, round(t, 2)) for c, t, _ in found]))
raise SystemExit(1 if VERDICT_FAILED else 0)
PYEOF
echo "artifacts in $OUT"
if [ $CONTROL -eq 1 ]; then
if [ $rc -eq 0 ]; then
echo
echo " 🔴 CONTROL FAILED -- the check passed a walk with \`move\` silenced, so it"
echo " cannot detect a cue that stops sounding."
exit 1
fi
echo
echo "the check rejects a run with a cue missing (rc=$rc)"
exit 0
fi
exit $rc