11b11ebd2d
The reference renderer was a revision behind for three consecutive diff runs and nothing said so. After the pin bump, `build-reference-cli` reported success at revb436e5aand `verify-screen` showed main_menu jumping 3/255 -> 72/255. The natural reading was that the port had regressed. It had not: the port was right and the REFERENCE was stale. The shared CARGO_TARGET_DIR still held a `sylpheed-cli` built from47f423f, cargo said "Finished in 0.13s" and left it there. A clean target dir built a binary resolving ptframe1 to (440,108) t=62; the shared one still said (620,108) t=16. The old check -- does `screen list` run -- cannot catch this. A stale binary runs perfectly. So: build into `$CARGO_TARGET_DIR/reference-cli/$rev`, keyed by the pin, where a new revision has no artifacts to reuse. Then verify the binary AGAINST export/: both come from the same pin, so if the CLI resolves ptframe1's rest differently from what the exporter wrote, the two halves of the verification are not the same revision and it fails loudly. It compares the two rather than asserting a literal, so it stays true when the pin moves again. docker/bin is baked into the image, so this needs an image rebuild to reach PATH; until then invoke the repo copy by path.
88 lines
4.2 KiB
Bash
Executable File
88 lines
4.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Diff Godot's drawing of an exported screen against `sylpheed-cli screen
|
|
# render` of the same build -- the P1 gate.
|
|
#
|
|
# tools/verify-screen # every screen in the manifest
|
|
# tools/verify-screen main_menu title # named screens
|
|
#
|
|
# Writes <screen>.godot.png, <screen>.ref.png and <screen>.diff.png into
|
|
# $OUT (default: a directory under /tmp) and prints, per screen, the largest
|
|
# per-channel difference anywhere in the frame.
|
|
#
|
|
# The two renderers are held to the same inputs on purpose:
|
|
#
|
|
# * the REFERENCE CLI is the one built by `build-reference-cli`, from the same
|
|
# `sylpheed-formats` revision the exporter is pinned to. /reborn's own
|
|
# target/ is a live mount of the other agent's checkout and moves mid-run; a
|
|
# pixel disagreement against a moving decoder proves nothing.
|
|
# * `--black` because Godot clears to black and the screen carries its own
|
|
# background. The CLI's default dim slate stands in for a 3D scene behind an
|
|
# in-mission screen, which is not this screen.
|
|
# * `--primitives --animated` because those are what make the CLI draw the same
|
|
# element set. `--focus` is NOT passed: nothing is focused at rest (HANDOFF
|
|
# Q5 measured initial focus as unstable boot to boot, so choosing one is
|
|
# P5's decision).
|
|
# * `--pose=rest` on the Godot side. Since P2 the port's DEFAULT is to play the
|
|
# timeline, and the settled timeline is deliberately NOT what `rest` says --
|
|
# the export's `rest` misses `ptframe1`/`ptframe2` on the main menu, and the
|
|
# running game shows them (docs/DECISIONS.md). Both renderers read `rest`
|
|
# through the same decoder, so asking for it here keeps this a test of the
|
|
# PORT against the reference. It is not the test of whether `rest` is right;
|
|
# that one is the oracle capture, and the port already departs from it.
|
|
#
|
|
# A difference here is not automatically the port's fault. Say which renderer is
|
|
# wrong and why -- do not tune until they match.
|
|
set -euo pipefail
|
|
cd "${PROJECT_DIR:-/work}"
|
|
|
|
# `reference-cli/`, not `release/`: the reference binary is built per pinned
|
|
# revision so a pin change cannot silently reuse the previous revision's build.
|
|
# See docker/bin/build-reference-cli.
|
|
CLI="${SYLPHEED_CLI:-${CARGO_TARGET_DIR:-/sylph-home/port/target-container}/reference-cli/sylpheed-cli}"
|
|
DISC="${SYLPHEED_DISC:-/disc}"
|
|
OUT="${OUT:-${TMPDIR:-/tmp}/verify-screen}"
|
|
export DISPLAY="${DISPLAY:-:97}"
|
|
|
|
[ -x "$CLI" ] || { echo "no reference CLI at $CLI -- run build-reference-cli" >&2; exit 2; }
|
|
[ -f export/manifest.json ] || { echo "no export/manifest.json -- run build-export --run" >&2; exit 2; }
|
|
mkdir -p "$OUT"
|
|
|
|
# Godot needs one scan to register the `class_name` globals; without it every
|
|
# script fails to parse and the run dies with no frame drawn.
|
|
[ -d port/.godot ] || godot --headless --path port --import >/dev/null 2>&1
|
|
|
|
screens=("$@")
|
|
if [ ${#screens[@]} -eq 0 ]; then
|
|
mapfile -t screens < <(python3 -c '
|
|
import json; print("\n".join(s["name"] for s in json.load(open("export/manifest.json"))["screens"]))')
|
|
fi
|
|
|
|
status=0
|
|
for name in "${screens[@]}"; do
|
|
build=$(python3 -c '
|
|
import json,sys
|
|
m=json.load(open("export/manifest.json"))
|
|
f=next(s["file"] for s in m["screens"] if s["name"]==sys.argv[1])
|
|
print(json.load(open("export/"+f))["source"]["build"])' "$name")
|
|
|
|
"$CLI" screen render "$DISC/dat/GP_TITLE.pak" "$OUT/$name.ref.png" \
|
|
--build "$build" --black --primitives --animated >/dev/null
|
|
|
|
godot --path port --resolution 1280x720 -- \
|
|
"--screen=$name" --pose=rest "--capture=$OUT/$name.godot.png" >"$OUT/$name.log" 2>&1
|
|
|
|
convert "$OUT/$name.godot.png" "$OUT/$name.ref.png" \
|
|
-compose difference -composite -colorspace Gray -auto-level "$OUT/$name.diff.png"
|
|
read -r max mean <<<"$(convert "$OUT/$name.godot.png" "$OUT/$name.ref.png" \
|
|
-compose difference -composite -format "%[fx:maxima*255] %[fx:mean*255]" info:)"
|
|
|
|
# 3/255 is what integer-truncating compositing in the CLI and float rounding
|
|
# in a GPU differ by. Anything above that is a placement, order or colour
|
|
# disagreement and needs a reason, not a threshold.
|
|
verdict=OK
|
|
awk "BEGIN{exit !($max > 3)}" && { verdict=DIFFERS; status=1; }
|
|
printf '%-16s build %-3s max %-5s mean %-8s %s\n' "$name" "$build" "$max" "${mean:0:6}" "$verdict"
|
|
done
|
|
echo "artifacts in $OUT"
|
|
exit $status
|