port: build the correctness harness verify-screen has pointed at since P1
`tools/port/verify-screen` line 20 has said "use `tools/verify-capture` for the
correctness question" since P1, and there was no such file. The port has had a
harness comparing itself to sylpheed-cli -- two renderers sharing its assumptions
-- and none comparing it to the game, while its own docs said otherwise. That gap
is what ORACLE-CAPTURES.md warns about: this corpus has been bitten three times
by two renderers agreeing, and every one was obvious against a capture.
Five screens against framebuffer captures of the real game:
main_menu RMSE 14.79 0.25% differing focus state may differ
extras RMSE 15.29 0.46% focus state may differ
title RMSE 21.07 1.82% ptloop sweeps never stop
publisher_logo RMSE 10.77 1.00%
developer_logos RMSE 9.37 0.39%
NO SCREEN SHOWS A LARGE CONNECTED BLOB -- the shape a missing element makes, and
the shape all three historical failures made.
And 74.1% of main_menu's differing pixels fall inside the ORACLE'S OWN focus
signature (live-main-menu vs live-main-menu-options-focused, the same screen with
a different button lit). So the bulk of that disagreement is a state mismatch,
not a defect.
REFUTATION ATTEMPT, on ui-render-tone-curve.md's `capture = 255*(render/255)^g`.
It survives where it was measured and not past it. Binning every structurally
matched pixel by render level gives the relationship directly, and the implied
exponent is NOT constant: 1.26 at render 16, 1.10 at 32, crossing 1.0 near 44,
down to 0.69 at 96. Above ~44 the capture is BRIGHTER than the render, which one
exponent cannot express -- and that is exactly why my whole-frame fits kept
returning 1.00, the two halves cancelling. The page's own stated reach ("nothing
constrains midtones or highlights") was not a hedge, it was the finding. Its 1.49
for this screen measures 1.18-1.26 in my darks; recorded as a disagreement rather
than resolved, since they fit selected flat patches and I binned everything.
Two earlier versions of this tool reported a best-fit gamma and were wrong both
times -- once fitting across a 74% structural mismatch, once extrapolating past
the measurement's stated reach. The fix was not a better fit but a different
instrument: it prints the curve, which somebody can argue with.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N7FiFFFwbvG2uxdcEh8HyF
This commit is contained in:
Sylpheed port agent
@@ -3667,3 +3667,84 @@ artefact would look like it worked.
|
||||
|
||||
Its output is **byte-identical** to the control's own stripping, so the tool and
|
||||
the experiment are the same operation rather than two implementations that agree.
|
||||
|
||||
## The correctness harness the docs promised for eight milestones did not exist
|
||||
|
||||
`tools/port/verify-screen`, line 20, since P1: *"Use `tools/verify-capture` for
|
||||
the correctness question."* **There was no such file.** The port has had a harness
|
||||
comparing itself to `sylpheed-cli` — two renderers sharing its assumptions — and
|
||||
none comparing it to the game, while its own documentation said otherwise.
|
||||
|
||||
`docs/re/captures/ORACLE-CAPTURES.md` is blunt about why that matters: two
|
||||
renderers agreeing proves nothing, and this corpus has been bitten three times —
|
||||
the dropped `pteff05` background, the scale-0 rect, `rest()` — each invisible to a
|
||||
render-vs-render diff and obvious against a capture.
|
||||
|
||||
`tools/port/verify-capture` now exists. **Five screens, against framebuffer
|
||||
captures of the real game:**
|
||||
|
||||
| screen | RMSE | differing | note |
|
||||
|---|---|---|---|
|
||||
| `main_menu` | 14.79 | **0.25 %** | focus state may differ |
|
||||
| `extras` | 15.29 | 0.46 % | focus state may differ |
|
||||
| `title` | 21.07 | 1.82 % | `ptloop` sweeps never stop |
|
||||
| `publisher_logo` | 10.77 | 1.00 % | |
|
||||
| `developer_logos` | 9.37 | 0.39 % | |
|
||||
|
||||
**No screen shows a large connected blob** — the shape a missing or misplaced
|
||||
element makes, and the shape all three historical failures made. The differences
|
||||
are scattered, and the two largest have stated causes.
|
||||
|
||||
### 74 % of `main_menu`'s difference is the oracle's own focus signature
|
||||
|
||||
The corpus ships `live-main-menu.png` and `live-main-menu-options-focused.png` —
|
||||
the same screen with a different button lit. Their difference *is* what focus
|
||||
changes, measured by the oracle against itself. Of the port's 2 159 differing
|
||||
pixels, **1 599 — 74.1 % — fall inside that signature.** So the bulk of the
|
||||
disagreement is a state mismatch (the port focuses `NEW GAME`, authored, because
|
||||
HANDOFF Q5 measured initial focus as unstable), not a rendering defect.
|
||||
|
||||
## Refutation attempt — the tone curve survives in its stated reach and not past it
|
||||
|
||||
`ui-render-tone-curve.md` models the relationship as
|
||||
`capture = 255·(render/255)^γ`, γ ≈ 1.34–1.49, **measured on dark flat patches
|
||||
(render ~0–60), with "nothing constrains midtones or highlights"** written into
|
||||
its own reach.
|
||||
|
||||
**I tried to fit that γ and got contradictory answers three times, and the
|
||||
contradictions were mine.** Binning every structurally matched pixel of
|
||||
`main_menu` by render level gives the relationship directly:
|
||||
|
||||
| render | capture | implied γ | pixels |
|
||||
|---|---|---|---|
|
||||
| 8 | 4.04 | 1.20 | 183 026 |
|
||||
| 16 | 7.89 | **1.26** | 227 630 |
|
||||
| 24 | 15.57 | 1.18 | 100 945 |
|
||||
| 32 | 26.15 | 1.10 | 87 474 |
|
||||
| 40 | 38.07 | 1.03 | 86 094 |
|
||||
| 48 | 53.96 | **0.93** | 85 255 |
|
||||
| 64 | 78.52 | 0.85 | 6 509 |
|
||||
| 96 | 130.44 | **0.69** | 1 682 |
|
||||
|
||||
✅ **The claim survives where it was measured.** In the darks the capture really
|
||||
is darker than the render and γ > 1.
|
||||
|
||||
🔴 **It is not a single power law.** The implied exponent falls monotonically and
|
||||
**crosses 1.0 near render ≈ 44** — above that the capture is *brighter*. One
|
||||
exponent cannot express a curve that crosses unity, which is precisely why my
|
||||
whole-frame fits kept returning γ = 1.00: the darks want more than 1 and the
|
||||
midtones want less, and they cancel.
|
||||
|
||||
**So the corpus's stated reach was not a hedge, it was the finding.** ⚠️ And the
|
||||
exponent in the darks measures **1.18–1.26 here against the page's 1.49 for this
|
||||
screen** — a disagreement I am recording rather than resolving, since they fit
|
||||
selected flat patches and I binned every matched pixel.
|
||||
|
||||
### The tool reports the curve, not a best exponent
|
||||
|
||||
Two earlier versions of `verify-capture` reported a best-fit γ and were wrong
|
||||
both times — once by fitting across a 74 % structural mismatch, once by
|
||||
extrapolating past a reach the measurement's own authors had written down.
|
||||
**Extrapolating a measurement past its stated reach is how this tool got it wrong
|
||||
twice**, and the answer was not a better fit but a different instrument: a table
|
||||
somebody can argue with.
|
||||
|
||||
137
tools/port/verify-capture
Executable file
137
tools/port/verify-capture
Executable file
@@ -0,0 +1,137 @@
|
||||
#!/usr/bin/env bash
|
||||
# Diff the port's render against a CAPTURE OF THE REAL GAME.
|
||||
#
|
||||
# tools/port/verify-capture main_menu
|
||||
# tools/port/verify-capture # every screen with a capture
|
||||
#
|
||||
# THIS IS THE CORRECTNESS CHECK. `verify-screen` is the consistency one, and its
|
||||
# own header has pointed at this file since P1 -- `tools/verify-capture` -- while
|
||||
# this file did not exist. The port has had a harness comparing two renderers
|
||||
# that share its assumptions, and none comparing it to the game.
|
||||
#
|
||||
# ⚠️ That gap is not academic. `docs/re/captures/ORACLE-CAPTURES.md`: two
|
||||
# renderers agreeing proves nothing, and this corpus has been bitten three times
|
||||
# -- the dropped `pteff05` background, the scale-0 rect, and `rest()`. Every one
|
||||
# was invisible to a render-vs-render diff and obvious against a capture.
|
||||
#
|
||||
# WHAT IT CAN CONCLUDE, and what it cannot:
|
||||
#
|
||||
# * ✅ STRUCTURE. Something drawn that should not be, or missing that should be,
|
||||
# shows as a large connected region of difference. That is the failure mode
|
||||
# the three above were, and it is what this tool is for.
|
||||
# * 🔴 NOT a pixel score. The captures are NOT gamma-neutral:
|
||||
# `capture ~= 255*(render/255)^g` with g ~ 1.34-1.49, and that ramp is THE
|
||||
# GAME'S, not the capture path's (`docs/re/structures/ui-render-tone-curve.md`).
|
||||
# So RMSE has a floor and driving it lower is fitting the ramp. This reports
|
||||
# the raw difference AND the gamma-compensated one, and neither is a target.
|
||||
# * ⚠️ A capture is ONE MOMENT. Several screens are still animating -- the
|
||||
# title's two `ptloop` sweeps never stop -- and the focused button in a
|
||||
# capture may not be the one the port focuses. Differences confined to a
|
||||
# button or a moving element are expected; say which before calling anything.
|
||||
#
|
||||
# Geometry needs no correction: the corpus cross-correlated a render against
|
||||
# `live-main-menu.png` over +/-6 px and the best alignment is exactly (0,0) at
|
||||
# 0.9466. The captures are a 1279x675 top-left crop of the 1280x720 guest
|
||||
# surface, so the render is cropped to match and nothing is scaled.
|
||||
set -euo pipefail
|
||||
cd "${PROJECT_DIR:-/work}"
|
||||
export DISPLAY="${DISPLAY:-:97}"
|
||||
OUT="${OUT:-$(mktemp -d)}"; mkdir -p "$OUT"
|
||||
CAPS=docs/re/captures/title-builds
|
||||
|
||||
# screen : capture : how to pose it
|
||||
MAP=(
|
||||
"main_menu:$CAPS/live-main-menu.png:--time=99"
|
||||
"extras:$CAPS/live-extras.png:--time=99"
|
||||
"title:$CAPS/live-title-build4-no-plate.png:--time=99"
|
||||
"publisher_logo:$CAPS/live-splash-publisher.png:--time=99"
|
||||
"developer_logos:$CAPS/live-splash-developer.png:--time=99"
|
||||
)
|
||||
CURVE=""
|
||||
if [ "${1:-}" = "--curve" ]; then CURVE=1; shift; fi
|
||||
want=("$@")
|
||||
echo "RMSE is reported and is NOT a target: the capture carries the game's own"
|
||||
echo "tone ramp, so it has a floor. What finds a real defect is the DIFFERING"
|
||||
echo "REGION -- a missing or misplaced element is a large connected blob."
|
||||
echo
|
||||
printf '%-17s %-9s %-7s %-22s %s\n' screen raw-rmse diff region note
|
||||
for row in "${MAP[@]}"; do
|
||||
IFS=: read -r name cap pose <<<"$row"
|
||||
if [ ${#want[@]} -gt 0 ] && ! printf '%s\n' "${want[@]}" | grep -qx "$name"; then continue; fi
|
||||
[ -f "$cap" ] || { printf '%-17s %s\n' "$name" "no capture"; continue; }
|
||||
godot --path port --resolution 1280x720 -- "--screen=$name" "$pose" \
|
||||
"--capture=$OUT/$name.render.png" >"$OUT/$name.log" 2>&1 || true
|
||||
[ -f "$OUT/$name.render.png" ] || { printf '%-17s %s\n' "$name" "render failed"; continue; }
|
||||
# Crop the render to the capture's frame. The capture is the crop, not a scale.
|
||||
convert "$OUT/$name.render.png" -crop 1279x675+0+0 +repage "$OUT/$name.crop.png"
|
||||
raw=$(convert "$OUT/$name.crop.png" "$cap" -metric RMSE -compare -format "%[distortion]" info: 2>&1 | tail -1)
|
||||
raw=$(python3 -c "print('%.2f' % (float('$raw')*255))" 2>/dev/null || echo "?")
|
||||
note=""
|
||||
case "$name" in
|
||||
main_menu|extras) note="focus state may differ from the capture" ;;
|
||||
title) note="ptloop sweeps never stop -- one moment vs another" ;;
|
||||
esac
|
||||
# Where the difference lives. This comes FIRST because it is what the gamma
|
||||
# sweep has to be protected from.
|
||||
convert "$OUT/$name.crop.png" "$cap" -compose difference -composite \
|
||||
-colorspace Gray -threshold 25% "$OUT/$name.mask.png"
|
||||
|
||||
# THE TONE RELATIONSHIP IS REPORTED AS A CURVE, NOT AS A BEST EXPONENT, and
|
||||
# two earlier versions of this tool reported an exponent and were wrong twice.
|
||||
#
|
||||
# `docs/re/structures/ui-render-tone-curve.md` models it as
|
||||
# `capture = 255*(render/255)^g`, g ~ 1.34-1.49, measured on dark flat patches
|
||||
# and explicitly not constrained above render ~60. Binning every structurally
|
||||
# matched pixel of `main_menu` by render level and averaging the capture gives:
|
||||
#
|
||||
# render capture implied g pixels
|
||||
# 8 4.04 1.20 183 026
|
||||
# 16 7.89 1.26 227 630
|
||||
# 24 15.57 1.18 100 945
|
||||
# 32 26.15 1.10 87 474
|
||||
# 40 38.07 1.03 86 094
|
||||
# 48 53.96 0.93 85 255
|
||||
# 64 78.52 0.85 6 509
|
||||
# 96 130.44 0.69 1 682
|
||||
#
|
||||
# ⚠️ **The implied exponent is not constant. It falls monotonically and crosses
|
||||
# 1.0 near render ~44**, so the capture is DARKER than the render in the darks
|
||||
# and BRIGHTER in the midtones. A single power law cannot express that, which
|
||||
# is exactly why a whole-frame fit returns 1.00: the two halves cancel. The
|
||||
# corpus's reach -- "nothing constrains midtones or highlights" -- was a real
|
||||
# limit and this is what lies past it.
|
||||
#
|
||||
# So: no best-g is printed. The table above is the instrument that can actually
|
||||
# be argued with; `tools/port/verify-capture --curve SCREEN` regenerates it.
|
||||
frac=$(convert "$OUT/$name.mask.png" -format "%[fx:mean*100]" info:)
|
||||
box=$(convert "$OUT/$name.mask.png" -trim -format "%wx%h%X%Y" info: 2>/dev/null || echo "-")
|
||||
printf '%-17s %-9s %6.2f%% %-22s %s\n' "$name" "$raw" "$frac" "$box" "$note"
|
||||
done
|
||||
if [ -n "$CURVE" ]; then
|
||||
for row in "${MAP[@]}"; do
|
||||
IFS=: read -r name cap pose <<<"$row"
|
||||
if [ ${#want[@]} -gt 0 ] && ! printf '%s\n' "${want[@]}" | grep -qx "$name"; then continue; fi
|
||||
[ -f "$OUT/$name.mask.png" ] || continue
|
||||
convert "$OUT/$name.crop.png" -colorspace Gray -depth 8 "gray:$OUT/$name.r.gray"
|
||||
convert "$cap" -colorspace Gray -depth 8 "gray:$OUT/$name.c.gray"
|
||||
convert "$OUT/$name.mask.png" -colorspace Gray -depth 8 "gray:$OUT/$name.m.gray"
|
||||
echo; echo "transfer curve, $name -- structurally matched pixels only"
|
||||
python3 - "$OUT/$name" <<'PYEOF'
|
||||
import sys, math
|
||||
b = sys.argv[1]
|
||||
r = open(b+".r.gray","rb").read(); c = open(b+".c.gray","rb").read(); m = open(b+".m.gray","rb").read()
|
||||
n = min(len(r), len(c), len(m)); bins = {}
|
||||
for i in range(n):
|
||||
if m[i]: continue
|
||||
s = bins.setdefault(r[i]//8*8, [0,0]); s[0] += c[i]; s[1] += 1
|
||||
print(" %-8s %-9s %-9s %s" % ("render","capture","implied g","pixels"))
|
||||
for k in sorted(bins):
|
||||
tot, cnt = bins[k]
|
||||
if cnt < 500 or k < 8: continue
|
||||
cap = tot/cnt
|
||||
g = math.log(max(cap,0.5)/255.0)/math.log(k/255.0)
|
||||
print(" %-8d %-9.2f %-9.2f %d" % (k, cap, g, cnt))
|
||||
PYEOF
|
||||
done
|
||||
fi
|
||||
echo "artifacts in $OUT"
|
||||
Reference in New Issue
Block a user