Sylpheed/tools/port/verify-screen

#!/usr/bin/env bash
# Diff Godot's drawing of an exported screen against `sylpheed-cli screen
# render` of the same build.
#
# WHAT THIS IS, AND WHAT IT IS NOT.
#
# It is a CONSISTENCY check between two decoders that share their assumptions,
# and a REGRESSION detector: "did anything move since last commit". It is NOT a
# correctness check and agreement here is NOT evidence of correctness.
#
# `sylpheed-cli` is not the oracle. The oracle is the Xenia Canary capture and
# the game. Reborn is an explorer and extraction CLI for verifying decodes, and
# it can be wrong -- this corpus has been bitten three times by both renderers
# agreeing and both being wrong: pteff05 (the menu background, missing from
# both), scale-0, and rest(). Each time the capture caught it and neither
# renderer could have.
#
# 🔴 AND ITS FRAMES MUST NEVER BE SCORED AGAINST A CAPTURE. This script poses
# `--pose=rest`, deliberately -- both renderers read `rest` through the same
# decoder, which is what makes it a test of the PORT against the REFERENCE. It
# is NOT the pose the port ships, and on some screens the two are very far
# apart: `rest` for each `ptlogo_back2eff*` sparkle is the peak of its own
# 4-unit flash, so `--pose=rest` lights all of them at once, a frame the game
# never shows.
#
# I scored this script's `title_jp` frame against the oracle capture and
# concluded the port had drifted away from the game -- r +0.7462 against the
# reference's +0.8727. Posed as it SHIPS, the same block scores **+0.9994**.
# The conclusion was an artefact of the pose, and it was written up as a finding.
# Correctness questions go to `tools/verify-capture`, which poses as shipped.
#
# So: a DIFFERS row means "we moved apart, go find out which of us moved". It
# does not mean the port is wrong. Where a capture and this tool disagree, the
# capture wins. Use `tools/verify-capture` for the correctness question.
#
#   tools/verify-screen                 # every screen in the manifest
#   tools/verify-screen main_menu title # named screens
#
# Writes <screen>.godot.png, <screen>.ref.png and <screen>.diff.png into
# $OUT (default: a directory under /tmp) and prints, per screen, the largest
# per-channel difference anywhere in the frame.
#
# The two renderers are held to the same inputs on purpose:
#
#  * the COMPARISON CLI is the one built by `build-reference-cli`, from the same
#    `sylpheed-formats` revision the exporter is pinned to. /reborn's own
#    target/ is a live mount of the other agent's checkout and moves mid-run; a
#    pixel disagreement against a moving decoder proves nothing.
#  * `--black` because Godot clears to black and the screen carries its own
#    background. The CLI's default dim slate stands in for a 3D scene behind an
#    in-mission screen, which is not this screen.
#  * `--primitives --animated` because those are what make the CLI draw the same
#    element set. `--focus` is NOT passed: nothing is focused at rest (HANDOFF
#    Q5 measured initial focus as unstable boot to boot, so choosing one is
#    P5's decision).
#  * `--pose=rest` on the Godot side. Since P2 the port's DEFAULT is to play the
#    timeline, and the settled timeline is deliberately NOT what `rest` says --
#    the export's `rest` misses `ptframe1`/`ptframe2` on the main menu, and the
#    running game shows them (docs/DECISIONS.md). Both renderers read `rest`
#    through the same decoder, so asking for it here keeps this a test of the
#    PORT against the reference. It is not the test of whether `rest` is right;
#    that one is the oracle capture, and the port already departs from it.
#
# A difference here is not automatically the port's fault, and it is not
# automatically a fault at all. Say which renderer moved and why -- do not tune
# until they match.
set -euo pipefail
cd "${PROJECT_DIR:-/work}"

# THE REFERENCE IS THE WORKSPACE'S OWN `sylpheed-cli`, and that is a change.
#
# It used to be a binary built per PINNED REVISION into `reference-cli/<rev>/`,
# because `sylpheed-formats` was a git dependency and /reborn's target/ was a
# live mount of the other agent's checkout that moved mid-run. A pixel
# disagreement against a moving decoder has a free variable in it.
#
# The monorepo merge (`65cefa7`) removed that problem by construction:
# `crates/sylpheed-export/Cargo.toml` now says
# `sylpheed-formats = { path = "../sylpheed-formats" }`, so the exporter, this
# reference and the port all read ONE decoder -- the working tree's.
#
# 🔴 It also silently broke the old machinery, and this script did not notice.
# `build-reference-cli` greps Cargo.toml for `Syplheed-Reborn.git", rev = "..."`;
# that line no longer exists, so the script exits 1 and the binary at
# `reference-cli/sylpheed-cli` is whatever was last built before the merge --
# here, three hours older than the sources and from a revision nothing points
# at any more. Running the diff against it would have compared the port to a
# decoder from another era and called the result a regression check. This
# corpus has already been bitten by a stale reference renderer three times.
#
# So: build it from the workspace. `SYLPHEED_CLI` still overrides, for anyone
# who does want to pin one deliberately.
CLI="${SYLPHEED_CLI:-}"
if [ -z "$CLI" ]; then
  CLI="${CARGO_TARGET_DIR:-/sylph-home/port/target-container}/release/sylpheed-cli"
  cargo build --release -p sylpheed-cli >/dev/null 2>&1 || true
fi
DISC="${SYLPHEED_DISC:-/disc}"
OUT="${OUT:-${TMPDIR:-/tmp}/verify-screen}"
export DISPLAY="${DISPLAY:-:97}"

[ -x "$CLI" ] || { echo "no reference CLI at $CLI -- \`cargo build --release -p sylpheed-cli\` failed?" >&2; exit 2; }
[ -f export/manifest.json ] || { echo "no export/manifest.json -- run build-export --run" >&2; exit 2; }

# 🔴 THE REFERENCE BINARY IS NOT NECESSARILY THE ONE THIS SCRIPT BUILT.
#
# `CARGO_TARGET_DIR` is a SHARED `/sylph-home/port/target-container`. Two source
# trees -- this workspace and any worktree built with the same variable set --
# write one `release/sylpheed-cli`, and cargo fingerprints per source path, so
# each build reports "Finished" while the binary on disk belongs to whichever
# tree wrote last. `cargo build` here returns in 0.15 s and changes nothing.
#
# That is the hazard the header above says the monorepo removed. It did not; the
# shared target dir reintroduced it by another route. Measured 2026-08-30: a CLI
# built from this workspace is `rest t=70` (the stale record layout) while the
# binary actually sitting in the target dir was `rest t=12` (fixed) -- so this
# script was comparing the port against a decoder from a tree nobody had named.
#
# ⚠️ It happened to be the RIGHT era, which is worse than wrong: it agreed with
# the exporter's pin by luck, and one successful rebuild would have flipped it
# silently. `title_jp` differs by 74 507 px between the two eras.
#
# So the era is CHECKED, against the export the port actually reads, rather than
# assumed from having run `cargo build`.
ref_rest=$("$CLI" screen info "$DISC/dat/GP_TITLE.pak" --build 5 --all 2>/dev/null \
  | grep -i 'pteff00' | head -1 | sed -n 's/.*rest (0,0) t=\([0-9]*\).*/\1/p')
exp_rest=$(python3 -c '
import json
m=json.load(open("export/manifest.json"))
f=next(s["file"] for s in m["screens"] if s["name"]=="main_menu")
d=json.load(open("export/"+f))
print(int(next(e for e in d["elements"] if e.get("id")=="pteff00")["rest"]["t"]))')
if [ -n "$ref_rest" ] && [ "$ref_rest" != "$exp_rest" ]; then
  echo "🔴 the reference CLI and the export disagree on the decoder era:" >&2
  echo "   reference $CLI says pteff00 rest t=$ref_rest" >&2
  echo "   export/   (built by the pinned exporter) says rest t=$exp_rest" >&2
  echo "   Every row below would compare two decoder eras. Refusing." >&2
  exit 2
fi
mkdir -p "$OUT"

# Godot needs one scan to register the `class_name` globals; without it every
# script fails to parse and the run dies with no frame drawn.
[ -d port/.godot ] || godot --headless --path port --import >/dev/null 2>&1

screens=("$@")
if [ ${#screens[@]} -eq 0 ]; then
  mapfile -t screens < <(python3 -c '
import json; print("\n".join(s["name"] for s in json.load(open("export/manifest.json"))["screens"]))')
fi

status=0
for name in "${screens[@]}"; do
  build=$(python3 -c '
import json,sys
m=json.load(open("export/manifest.json"))
f=next(s["file"] for s in m["screens"] if s["name"]==sys.argv[1])
print(json.load(open("export/"+f))["source"]["build"])' "$name")

  # `--all` because the exporter now addresses by PAK ENTRY INDEX, which is the
  # numbering `--all` uses; without it the CLI enumerates only the 12 bundles
  # `is_build` accepts and `--build 10` would land on entry 12. `--all` widens
  # the list, it does not change how any one bundle composites.
  "$CLI" screen render "$DISC/dat/GP_TITLE.pak" "$OUT/$name.ref.png" \
    --build "$build" --all --black --primitives --animated >/dev/null

  # 🔴 THE REFERENCE RENDERER SILENTLY OMITS A `.tbm` BACKGROUND.
  #
  # The Decoder reached and captured the TUTORIAL screen and found that
  # `screen render` draws every OTHER element of a `.tbm`-bearing build and
  # leaves the background out, with no diagnostic: their render of GP_TUTORIAL
  # build 0 is the correct layout on pure black, 6.0-6.4 % inked against the
  # game's 99.7 %. `docs/re/structures/tbm-submenu-not-reached.md`, their branch.
  #
  # I confirmed the shape of it here with both controls: `screen info` reports
  # `pubase.tbm` on GP_TUTORIAL build 0 and no `.tbm` on any of the 16 builds in
  # my manifest. So this trap CANNOT fire today.
  #
  # ⚠️ That is a fact about today's manifest, not a property of this script, and
  # the failure it would cause is the expensive kind: the port draws a
  # background the reference does not, the row reads DIFFERS, and the header
  # above tells the reader to go find out which renderer moved. Neither did.
  # The row would be a real disagreement caused by a KNOWN omission on the
  # reference side, and nothing on screen would say so.
  #
  # So the row says so. This does not change the verdict or the bar -- it
  # attaches the provenance to the one row that would otherwise mislead.
  tbm=$("$CLI" screen info "$DISC/dat/GP_TITLE.pak" --build "$build" --all 2>/dev/null \
    | grep -ioc '\.tbm' || true)

  # 🔴 `--loop-phase=0` PINS THE PULSE, AND WITHOUT IT THIS SCRIPT WAS
  # NONDETERMINISTIC. `press_start` returned `over3` **5021, 8919, 5021** on
  # three identical runs: the plate's looping focus record rides `time_units`,
  # so the captured frame lands wherever the grab fell, while the reference
  # renderer cannot pulse at all.
  #
  # ⚠️ The port is NOT the thing that is wrong. A thing that pulses does not
  # stop because the screen has arrived, and the pulse is measured. What was
  # wrong is comparing a moving frame against a static one and calling the
  # difference a regression -- a detector that answers differently each run
  # teaches its reader to ignore it, which is worse than one that fails.
  #
  # So the phase is pinned HERE, in the harness, and nothing about playback
  # changes: `loop_phase_units` defaults to free-running everywhere else.
  # ⚠️ It is usually stable -- 3 of 4 control runs agreed -- which is exactly
  # why this survived: it looks deterministic most of the time.
  godot --path port --resolution 1280x720 -- \
    "--screen=$name" --pose=rest --loop-phase=0 "--capture=$OUT/$name.godot.png" >"$OUT/$name.log" 2>&1

  convert "$OUT/$name.godot.png" "$OUT/$name.ref.png" \
    -compose difference -composite -colorspace Gray -auto-level "$OUT/$name.diff.png"
  read -r max mean <<<"$(convert "$OUT/$name.godot.png" "$OUT/$name.ref.png" \
    -compose difference -composite -format "%[fx:maxima*255] %[fx:mean*255]" info:)"

  # HOW MANY pixels are over the bar, not just how far the worst one is. A
  # single `max` cannot tell 2 pixels from 25 444, and this run produced both:
  # `main_menu` trips the threshold on TWO pixels out of 921 600 while
  # `title_jp` trips it on 2.8 % of the frame. Reporting only the max made those
  # the same verdict, which is how a real disagreement hides behind a rounding
  # one. The bar itself is NOT raised -- tuning a threshold until things match
  # is the failure this script's own header warns about.
  over=$(convert "$OUT/$name.godot.png" "$OUT/$name.ref.png" \
    -compose difference -composite -colorspace Gray -threshold $((3*65535/255)) \
    -format "%[fx:int(mean*w*h)]" info:)

  # BOTH FRAMES BLANK IS NOT AGREEMENT, AND THIS SCRIPT USED TO SAY IT WAS.
  #
  # `build_12` and `build_15` -- the two dressed loading screens -- render as
  # pure black in BOTH renderers, mean 0 and max 0, so the difference is 0 and
  # the row read `max 0  over3 0  OK`. Two of the sixteen rows in the committed
  # baseline were comparing nothing against nothing and reporting the strongest
  # verdict this script has.
  #
  # That is worse than a missing test: it is a test that reports a pass. The
  # screens are black because `pgloading_eff00` is a full-frame opaque black
  # quad whose `rest.t` (38) sits inside its own opening black hold, and
  # `--pose=rest` freezes it there -- see docs/port/DECISIONS.md. Whether that
  # is the port's bug or the decoders' reading of `rest` is open; what is not
  # open is that a blank pair may not be scored.
  #
  # ✅ RESOLVED 2026-08-30, AND THE PARAGRAPH ABOVE IS NOW HISTORY. It was the
  # PAINT ORDER, not `rest`. `pgloading_eff00` carries `layer: null`,
  # `layer_source: none` -- the only elements in the export with neither a read
  # nor an implied key -- so without the forced-backdrop pass the first element
  # becomes `pgloading_loop5` and the opaque quad paints over everything. With
  # the pass, both screens render at max 214.5 in BOTH renderers (mean 1.949
  # port, 1.918 reference) and the rows read `OK` on a real comparison.
  #
  # ⚠️ The guard STAYS. It is not firing today, which is exactly when a guard
  # quietly rots -- and it was right when it was written: two of sixteen rows
  # were comparing nothing against nothing and reporting this script's
  # strongest verdict. Leaving the reasoning above intact is deliberate; a
  # reader who hits a blank pair tomorrow needs it.
  #
  # So blankness is checked FIRST and reported as its own verdict. It is not a
  # failure -- the port may legitimately have nothing to draw -- but it is not a
  # pass either, and `status` is left alone so an unrelated screen's DIFFERS is
  # still what fails the run.
  ink=$(convert "$OUT/$name.godot.png" "$OUT/$name.ref.png" \
    -evaluate-sequence max -colorspace Gray -format "%[fx:maxima*255]" info:)
  verdict=OK
  if awk "BEGIN{exit !($ink <= 0)}"; then
    verdict="BLANK -- both renderers drew nothing; this row proves nothing"
  else
    # 🔴 THE VERDICT USES `over3`, NOT `max` ALONE, AND FOR YEARS IT DID NOT.
    #
    # This script computed `over3` precisely because "a single `max` cannot tell
    # 2 pixels from 25 444" -- its own words, a few lines up -- and then decided
    # the verdict on `max` regardless. So `main_menu` (max 4, over3 **0**) read
    # DIFFERS while `extras` (max 3, over3 0) read OK: one unit on one pixel,
    # separating two frames that are pixel-for-pixel equivalent at the bar.
    #
    # ⚠️ This is NOT raising the bar, which this file rightly warns against. The
    # bar is still 3. What changes is that a frame with NO pixel over it gets a
    # verdict of its own instead of being lumped in with a real disagreement --
    # the distinction the statistic was added to make and was never given.
    if awk "BEGIN{exit !($over > 0)}"; then
      verdict=DIFFERS; status=1
    elif awk "BEGIN{exit !($max > 3)}"; then
      verdict="ROUNDING -- max $max but NO pixel over the bar"
    fi
  fi
  if [ "${tbm:-0}" -gt 0 ]; then
    verdict="$verdict  [build carries a .tbm: the REFERENCE omits that background, so a DIFFERS here is likely theirs]"
  fi
  printf '%-17s build %-3s max %-5s mean %-8s over3 %-7s %s\n' \
    "$name" "$build" "$max" "${mean:0:6}" "$over" "$verdict"
done
echo "artifacts in $OUT"
exit $status