port: verify-menu-audio could not FAIL -- the verdict never reached an exit code
It computed ok, printed a red line when a cue was silent, and the python had no exit path at all, so it returned 0 every time while check-all registered it must-pass. A cue could stop sounding and the suite would print the failure and stay green. This project's recurring defect one level up: not an instrument sitting below the thing under test, but one that SEES the failure and does not report it. The diagonal now exits. The no-op-silence and cue-order lines stay REPORTED -- both carry documented cross-run instability (whole-buffer recording shifts; a 0.15 margin this file's own comments show reaching 0.109 on a sounding cue), and making either binding would produce red on correct audio. --control feeds it a walk in which move never sounded, built from the tool's own suppression machinery, and requires it to fail.
This commit is contained in:
@@ -1,7 +1,18 @@
|
||||
#!/usr/bin/env bash
|
||||
# Does the port actually MAKE SOUND on the P5 walk, and the RIGHT sound?
|
||||
#
|
||||
# tools/port/verify-menu-audio
|
||||
# tools/port/verify-menu-audio # assert
|
||||
# tools/port/verify-menu-audio --control # can it fail?
|
||||
#
|
||||
# 🔴 FOR WEEKS THIS COULD NOT FAIL. It computed the verdict, printed a red line
|
||||
# when a cue was silent -- and the python had NO EXIT PATH, so it returned 0
|
||||
# every time while `check-all` registered it `must-pass`. A cue could stop
|
||||
# sounding and the suite would print the failure and stay green.
|
||||
#
|
||||
# That is this project's recurring defect one level up: not an instrument that
|
||||
# sits below the thing under test, but an instrument that SEES the failure and
|
||||
# does not report it. Ask of any check: what would this still report if the
|
||||
# feature were absent -- AND what would it EXIT?
|
||||
#
|
||||
# This is the P6 gate check. P6's gate is "sound on the P5 gate", and until this
|
||||
# existed the only evidence for it was that `audio.play("move")` appears in
|
||||
@@ -29,6 +40,7 @@ set -euo pipefail
|
||||
cd "${PROJECT_DIR:-/work}"
|
||||
export DISPLAY="${DISPLAY:-:97}"
|
||||
OUT="${OUT:-${TMPDIR:-/tmp}/verify-menu-audio}"
|
||||
CONTROL=0; [ "${1:-}" = "--control" ] && CONTROL=1
|
||||
mkdir -p "$OUT"
|
||||
[ -d port/.godot ] || godot --headless --path port --import >/dev/null 2>&1
|
||||
|
||||
@@ -72,7 +84,18 @@ for c in move confirm back; do
|
||||
exit 2; }
|
||||
done
|
||||
|
||||
python3 - "$OUT" <<'PYEOF'
|
||||
# THE CONTROL. Replace the walk with the run that already had `move` silenced, so
|
||||
# the cue is genuinely missing from the baseline. Silencing it again can then
|
||||
# remove nothing, the diagonal cannot drop, and the check MUST fail. Built from
|
||||
# the tool's OWN suppression machinery rather than a second mechanism -- a
|
||||
# control built a different way tests the control, not the check.
|
||||
if [ $CONTROL -eq 1 ]; then
|
||||
cp "$OUT/sup_move.wav" "$OUT/walk.wav"
|
||||
echo "control: analysing a walk in which \`move\` never sounded"
|
||||
fi
|
||||
|
||||
rc=0
|
||||
python3 - "$OUT" <<'PYEOF' || rc=$?
|
||||
import array, math, subprocess, sys
|
||||
O = sys.argv[1]; SR = 44100
|
||||
def dec(src, dst):
|
||||
@@ -184,6 +207,14 @@ for row in ("move", "confirm", "back"):
|
||||
print(" 🔴 silencing %s did not remove %s -- that cue is NOT SOUNDING" % (row, row))
|
||||
print(" => %s" % ("all three cues SOUND: silencing each one collapses its own signal"
|
||||
if ok else "at least one cue is not sounding"))
|
||||
# 🔴 THE VERDICT EXITS. Everything below this line is REPORTED, not asserted, and
|
||||
# deliberately so: the no-op-silence line and the cue-order line both carry
|
||||
# DOCUMENTED cross-run instability (whole-buffer recording shifts; a 0.15 margin
|
||||
# this file's own comments show going to 0.109 on a sounding cue). Making either
|
||||
# binding would produce red on correct audio, which is how a suite gets ignored.
|
||||
# The diagonal has no threshold to drift: silencing a cue either removes its own
|
||||
# signal or it was never there.
|
||||
VERDICT_FAILED = not ok
|
||||
# 🔴 THE VERDICT IS THE DIAGONAL ONLY, and the first version of this asserted the
|
||||
# off-diagonal too -- "silencing a cue must not move the others". That failed, and
|
||||
# the material is why: `confirm` lands at 1.12 s and `back` at 1.21 s, 0.09 s apart
|
||||
@@ -198,5 +229,18 @@ times = [t for _, t, hit in found if hit]
|
||||
print("cue order vs script order : %s"
|
||||
% ("CONSISTENT" if times == sorted(times) and len(times) == 3
|
||||
else "check %s" % [(c, round(t, 2)) for c, t, _ in found]))
|
||||
raise SystemExit(1 if VERDICT_FAILED else 0)
|
||||
PYEOF
|
||||
echo "artifacts in $OUT"
|
||||
|
||||
if [ $CONTROL -eq 1 ]; then
|
||||
if [ $rc -eq 0 ]; then
|
||||
echo
|
||||
echo " 🔴 CONTROL FAILED -- the check passed a walk with \`move\` silenced, so it"
|
||||
echo " cannot detect a cue that stops sounding."
|
||||
exit 1
|
||||
fi
|
||||
echo
|
||||
echo "the check rejects a run with a cue missing (rc=$rc)"
|
||||
exit 0
|
||||
fi
|
||||
exit $rc
|
||||
|
||||
Reference in New Issue
Block a user