tools: verify-screen -- diff every exported screen against the reference renderer
The P1 gate is a pixel comparison, so it needs to be a command rather than a paragraph. `tools/verify-screen` renders each screen in the manifest both ways and reports the largest per-channel difference anywhere in the frame. Both renderers are held to the same inputs on purpose: * the reference CLI is the one `build-reference-cli` builds from the revision the exporter is PINNED to -- not `/reborn/target/`, which is a live mount of the other agent's checkout and moves mid-iteration. A pixel disagreement against a moving decoder has a free variable in it and proves nothing. * `--black`, because the screen carries its own background and the CLI's default dim slate stands in for a 3D scene that is not on this screen. * `--primitives --animated`, because the port draws every element at rest and those two flags are what make the CLI draw the same set. NOT `--focus`: nothing is focused at rest. The threshold is 3/255 -- what integer-truncating compositing in the CLI and float rounding on a GPU differ by. Above that is a placement, order or colour disagreement that needs a reason, and the script says DIFFERS rather than pretending a wider tolerance is a result.
This commit is contained in:
77
tools/verify-screen
Executable file
77
tools/verify-screen
Executable file
@@ -0,0 +1,77 @@
|
||||
#!/usr/bin/env bash
|
||||
# Diff Godot's drawing of an exported screen against `sylpheed-cli screen
|
||||
# render` of the same build -- the P1 gate.
|
||||
#
|
||||
# tools/verify-screen # every screen in the manifest
|
||||
# tools/verify-screen main_menu title # named screens
|
||||
#
|
||||
# Writes <screen>.godot.png, <screen>.ref.png and <screen>.diff.png into
|
||||
# $OUT (default: a directory under /tmp) and prints, per screen, the largest
|
||||
# per-channel difference anywhere in the frame.
|
||||
#
|
||||
# The two renderers are held to the same inputs on purpose:
|
||||
#
|
||||
# * the REFERENCE CLI is the one built by `build-reference-cli`, from the same
|
||||
# `sylpheed-formats` revision the exporter is pinned to. /reborn's own
|
||||
# target/ is a live mount of the other agent's checkout and moves mid-run; a
|
||||
# pixel disagreement against a moving decoder proves nothing.
|
||||
# * `--black` because Godot clears to black and the screen carries its own
|
||||
# background. The CLI's default dim slate stands in for a 3D scene behind an
|
||||
# in-mission screen, which is not this screen.
|
||||
# * `--primitives --animated` because the port draws every element at its
|
||||
# resting pose, and those two flags are what make the CLI draw the same set.
|
||||
# `--focus` is NOT passed: nothing is focused at rest (HANDOFF Q5 measured
|
||||
# initial focus as unstable boot to boot, so choosing one is P5's decision).
|
||||
#
|
||||
# A difference here is not automatically the port's fault. Say which renderer is
|
||||
# wrong and why -- do not tune until they match.
|
||||
set -euo pipefail
|
||||
cd "${PROJECT_DIR:-/work}"
|
||||
|
||||
CLI="${SYLPHEED_CLI:-${CARGO_TARGET_DIR:-/sylph-home/port/target-container}/release/sylpheed-cli}"
|
||||
DISC="${SYLPHEED_DISC:-/disc}"
|
||||
OUT="${OUT:-${TMPDIR:-/tmp}/verify-screen}"
|
||||
export DISPLAY="${DISPLAY:-:97}"
|
||||
|
||||
[ -x "$CLI" ] || { echo "no reference CLI at $CLI -- run build-reference-cli" >&2; exit 2; }
|
||||
[ -f export/manifest.json ] || { echo "no export/manifest.json -- run build-export --run" >&2; exit 2; }
|
||||
mkdir -p "$OUT"
|
||||
|
||||
# Godot needs one scan to register the `class_name` globals; without it every
|
||||
# script fails to parse and the run dies with no frame drawn.
|
||||
[ -d port/.godot ] || godot --headless --path port --import >/dev/null 2>&1
|
||||
|
||||
screens=("$@")
|
||||
if [ ${#screens[@]} -eq 0 ]; then
|
||||
mapfile -t screens < <(python3 -c '
|
||||
import json; print("\n".join(s["name"] for s in json.load(open("export/manifest.json"))["screens"]))')
|
||||
fi
|
||||
|
||||
status=0
|
||||
for name in "${screens[@]}"; do
|
||||
build=$(python3 -c '
|
||||
import json,sys
|
||||
m=json.load(open("export/manifest.json"))
|
||||
f=next(s["file"] for s in m["screens"] if s["name"]==sys.argv[1])
|
||||
print(json.load(open("export/"+f))["source"]["build"])' "$name")
|
||||
|
||||
"$CLI" screen render "$DISC/dat/GP_TITLE.pak" "$OUT/$name.ref.png" \
|
||||
--build "$build" --black --primitives --animated >/dev/null
|
||||
|
||||
godot --path port --resolution 1280x720 -- \
|
||||
"--screen=$name" "--capture=$OUT/$name.godot.png" >"$OUT/$name.log" 2>&1
|
||||
|
||||
convert "$OUT/$name.godot.png" "$OUT/$name.ref.png" \
|
||||
-compose difference -composite -colorspace Gray -auto-level "$OUT/$name.diff.png"
|
||||
read -r max mean <<<"$(convert "$OUT/$name.godot.png" "$OUT/$name.ref.png" \
|
||||
-compose difference -composite -format "%[fx:maxima*255] %[fx:mean*255]" info:)"
|
||||
|
||||
# 3/255 is what integer-truncating compositing in the CLI and float rounding
|
||||
# in a GPU differ by. Anything above that is a placement, order or colour
|
||||
# disagreement and needs a reason, not a threshold.
|
||||
verdict=OK
|
||||
awk "BEGIN{exit !($max > 3)}" && { verdict=DIFFERS; status=1; }
|
||||
printf '%-16s build %-3s max %-5s mean %-8s %s\n' "$name" "$build" "$max" "${mean:0:6}" "$verdict"
|
||||
done
|
||||
echo "artifacts in $OUT"
|
||||
exit $status
|
||||
Reference in New Issue
Block a user