One filter to select (004 -> 008, 35897 -> 7), a second on a transition it was
NOT selected by (008 -> 012, 7 -> 1), leaving exactly one address; then three
live paired readings, RAM/screenshot/RAM, all agreeing with the HUD.
The address is the same one run 1 reported. That does NOT reverse yesterday's
refutation and the entry says so explicitly: runs 2 and 3 read a hard 0 there on
an allocated page while the HUD counted, so "it is there in every run" stays
refuted. What is withdrawn is the stronger claim that the number was meaningless
- it recurs exactly, in 2 of the 4 runs measured, and run 3's amber candidate
0xbdb49668 sits one 64 KB page below it at the identical page offset 0x9668. The
practical rule is therefore: try 0xbdb59668, check it against the HUD, re-scan
when it reads 0.
Why run 3 failed and run 4 did not is also recorded, because it is a method
lesson rather than luck: the evidence was always in the first four minutes of the
stage, and the earlier runs simply could not look often enough - every HUD
reading cost a human round trip, so the 008 step went by between two of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Both hand attempts at this address managed exactly one filter and then lost the
counter, because every reading of the HUD costs a human round trip. ob_read.py
does it by template correlation over three fixed digit cells, and ob_hunt.py
uses that to run the whole method unattended: confirm the HUD on both sides of
the 0.9 s scan, then filter on each following transition, labelling the first as
the one it selected on and the rest as verification.
Two things measured rather than assumed:
* Normalisation is the point, not a nicety. The plate is translucent, so an
explosion turns it orange - on the 008 reference frame a cyan-stroke mask finds
6 of ~70 stroke columns and would silently read nothing. Per-cell mean/std
normalisation reads it correctly at 0.843.
* The accept rule is two-sided because the margin is narrow: over 12 cells from
four frames of known value the correct digit scores 0.843..1.000 and the best
WRONG digit reaches 0.789 (0 and 8 are similar outlines). So a floor of 0.80
AND a 0.05 margin over the runner-up, against a smallest observed correct
margin of 0.093. A bare threshold fitted between those two numbers would be
fitted to twelve samples.
Templates exist for 0 1 2 4 8 - the digits actually seen. Anything else reads as
"?" and callers must treat a "?" as unknown, never as a value. Rejects both
negatives tested: GAME OVER scores 0.05-0.18, the main menu 0.07-0.14.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
The launcher item closes: unattended boot to Stage 02 flight works again, and the
entry now names all three defects that were in the way, including the
wait_flight pixel that was outside the HUD.
The objective-counter item stays open but is sharper: a HUD-clean scan exists,
and the surviving candidate that is hard to dismiss is named with the odds
against it being chance. The blocker is now the pilot's survival, not the method.
Also withdraws yesterday's claim that the "Auto-Save is active. OK?" dialog does
not return - the run that appeared to show that was tapping A every six seconds
while it waited, so it cannot distinguish "absent" from "dismissed unseen".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
With the launcher repaired the scan could be run the way the method asks: the HUD
was confirmed at 004 immediately before AND after the 0.9 s scan, and the filter
was taken at a confirmed 012. That leaves 17 candidates from 39596.
One of them is 0xbdb49668 - the run-1 address minus exactly 0x10000, at the same
page offset 0x9668. Checked rather than admired: of all 39596 words holding 4 at
scan time, exactly TWO sit at that page offset, so a survivor landing there by
chance is a ~5e-5 event. The reading it suggests is testable - the counter sits
at a fixed offset in an allocation whose base moves by whole 64 KB pages.
Recorded as amber, not green, and the reason is written down: the later sample
that killed 10 of the 17 was taken during the GAME OVER flash, with no HUD on
screen, so the seven survivors are unrefuted rather than confirmed. The mission
ended because the escorted ACROPOLIS was lost at ~13 min while pilot.py sat in
EVADE/RETIRE; REMAINING OB held at 012 for the ten minutes before that, so there
was no transition to verify against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
It tested one named pixel, (450,640), described as "inside the SHIELD bar". It is
not inside anything. `screenshot` crops to the GAME SURFACE, 1279x675, while
those constants were chosen for a 1280x720 window: measured on a frame that was
unmistakably in flight (TIME 01:18.05, REMAINING OB 004), the SHIELD bar's green
rows are 590-605 and the ARMOR bar's are 650-670, and (450,640) sits in the gap
between them reading (64,66,116) - blue.
That is why the corpus already carried "wait_flight.sh reported NEVER REACHED
FLIGHT while the game was plainly in flight" as an unexplained note, and it is
exactly the failure screen_id.py's own header warns about. Reproduced live here:
the script sat for five minutes reporting nothing while tapping A into the
cockpit every six seconds.
Now classified by whole-image statistics - screen_id.py's "flight" class, green
fraction 1.3-1.5% against <0.1% on every menu - which no crop or scale can move.
Two consecutive frames are required so an explosion in a cutscene cannot pass,
and the A tapping (which dismisses the objective card) STOPS once flight is seen,
because in the cockpit A is a weapon press and not a "continue".
Tested against the running game before being believed: 0 with "IN FLIGHT at 19s"
on the same session the old test had been failing on, and then in the unattended
launch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Two separate reasons the scripted route stopped, both measured rather than guessed:
* LOAD -> READY ROOM is not 28 s. Both runs on 2026-08-23 overran it, so the next
press was eaten by the transition and the run ended up in OPTIONS once and
BRIEFINGS once. wait_screen.sh now waits for the screen, with an optional
--tap that clears a dialog the caller cannot know about (a freshly restored
profile inserts "Auto-Save is active. OK?" here).
* The READY ROOM is DRAWN long before it is USABLE: it comes up with a
"Preparing to Sortie" spinner and TAKE OFF greyed out. The two states differ by
1.7 units of blue whole-image, so screen_id.py cannot separate them and should
not try. take_off_armed.py tests the label instead: 0.0000 bright pixels while
preparing, 0.1633 once armed, on three captures from two runs. It carries its
own position check - the always-enabled BRIEFINGS label below reads 0.1027 in
all three, to four decimals, so if that reference is dark the boxes are off the
labels and the answer is "unknown", not a confident wrong one.
screen_id.py gains a "readyroom" class from the same measurements; nothing else
reclassifies.
Verified end to end and unattended: boot -> title -> LOAD GAME -> slot 01 ->
READY ROOM -> TAKE OFF -> "IN FLIGHT at 34s", pilot bound and engaging.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
0xbdb59668 reads 0 in two independent Stage 02 runs while the HUD counts
004 -> 008 -> 012. Not an unmapped read: SEEK_DATA at that offset returns the
offset itself and the next hole is 5 MB later, so it is an allocated,
zero-filled word. The address was a per-run artefact, exactly as that file
already suspected it might be; the method is the durable result.
Re-finding it in the new run also failed, and both failures are recorded because
they are the instructive part. Two candidates were produced and both died on the
corpus's own rule -- verify across a transition you did not select on:
0xbc2377dc went 12 -> 18 while the HUD stayed 012 and read 3 two minutes later,
and 0xbd295b04 was plain noise.
One correction to the method note in that file: the scan is not slow. Over the
live /dev/shm image it takes 0.9 s. The real trap is that REMAINING OB climbs
004 -> 012 within about four minutes as waves spawn, so a scan is only valid if
the HUD is confirmed to hold the same value immediately before AND after it --
which is why the earlier 4-then-8 intersection came back empty.
What blocked finishing: with pilot.py retired at hull 340/1500 nothing was
killing objectives and the counter sat at 012 for five minutes, so there was no
later transition to filter on. What the counter counts, and whether an OB-badged
entity carries a flag, is untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
The baseline binary, the profile, the Stage 02 save and the caches all live
outside both repos and had been wiped, and sylph-doctor reports 'all good'
without them. This is the recipe, verified by driving it to Stage 02 flight
rather than by inspecting files.
Three things worth having found:
* the baseline is an INCREMENTAL build, not a scratch one -- auto/upstream-baseline
differs from the instrumented branch in 202 files and zero submodule gitlinks,
so checking it out in the shared tree and rebuilding into the existing build dir
takes ~9 min at -j4. git worktree prune first; a dead worktree still claims the
branch.
* the baseline CANNOT make a profile (--create_profile_if_none is one of our 50
commits) and the Sign In dialog cannot be typed into, so bootstrap the profile
with the instrumented binary once and run the baseline against it.
* the Xbox content .header is NOT needed -- ContentManager::ListContent falls back
to the directory name when ReadContentHeaderFile fails, checked in the source.
The LOAD GAME screen then shows date, difficulty, flight time and clear ratio
anyway, which proves those come out of savedata itself. No header was fabricated.
Trap paid for once: keep the file name xenia_canary. A copy saved as
xenia_canary.baseline-... survives every pkill -x in the toolkit and holds the
run-canary lockfile while ps -C xenia_canary shows nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
ob_sample.py pairs a live read of a guest VA with a screenshot and a crop of the
HUD counter, and reads the word again AFTER the shot so a sample whose two reads
disagree can be thrown away instead of believed -- that race is what left the
first three-snapshot filter with zero survivors.
ob_session.sh is fly_session.sh plus that sampler, because the evidence this
needs is a transition, and an unattended craft is dead in about a minute. Its
header records that launch_mission.sh does not yet finish unattended on restored
state, so nobody discovers that halfway through a boot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
The mission's objective counter is a big-endian u32 at guest VA 0xbdb59668 in a
Stage 02 run on the upstream baseline. It was selected on an 18->24 transition
and then tracked 24->23->22 against the HUD on its own - four readings, two
changes it was not filtered on.
That last point is the whole discipline here, because the first attempt failed it.
An earlier differential over 19->18 also produced exactly one candidate,
0xbc22e83c, which matched the transition it was selected on and was still wrong:
read live it held 26 while the HUD showed 017. One matching transition is not
evidence.
A second trap is recorded too: a three-snapshot filter requiring 19 -> 19 -> 18
left ZERO survivors, because the value moves between the memory copy and the
screenshot that reads it. Filtering on the next DISTINCT value instead found the
counter on the first try.
ob_scan.py carries the method: scan one snapshot, then filter the candidate set
against live /dev/shm/xenia_memory_* at each new value, so only the first pass
needs a 4.8 GB copy.
Stated plainly as unsettled: the ADDRESS is from one run and cross-run stability
is untested, so the durable result is the method rather than the number. And what
the counter counts - whether every OB-badged entity is one of them, and whether
that badge is a flag in the entity object - is the follow-on the pilot actually
needs to CHOOSE targets rather than just know how many remain.
Built upstream/canary_experimental (a5a18f5c7, 82 commits ahead of our merge
base) clean, to answer whether the 0x82307128 mission crash is ours or theirs.
Two tooling gaps found building it: version.h is never generated (upstream's
xenia-build.py writes it, build-canary does not invoke it, and our tree only
builds because a stale copy from an old sylpheed-re build sits in the build dir),
and build-canary reports exit 0 on a FAILED build - only the missing binary gave
it away.
Pure upstream cannot be driven at all: --hid=file is ours and there is no uinput
here. So the honest baseline is upstream plus those three pad commits. That
baseline reproduced the LOST RESUME on its second boot - black screen, guest at
522% CPU, newest host thread at 00:00:00 CPU, zero kernel calls from its handle -
and upstream's threading_posix.cc still has the two-lock-scope publish. So pure
upstream is not a stable state; it carries a bug we had already fixed.
Adding ONLY the threading fix on top - still none of the RE instrumentation -
reaches Stage 02 FLIGHT: TIME 05:03.75, wingman Rhino 103, radio dialogue, and
REMAINING OB 020 with OB badges on targets. The run ended in a legitimate GAME
OVER (the unattended craft was shot down, as autopilot-memory-driven.md warns),
not a crash. Crash dumps for the whole run: 0, against 384 on the instrumented
branch at the briefing alone.
So the mission crash is not upstream's. Whether it is in our 50 commits or fixed
among upstream's 82 this run does not separate; that needs a bisect.
REMAINING OB is visible and moves (020->019->018->017) but is NOT found. A
three-snapshot BE-u32 filter left 0 survivors; a differential across widths gave
exactly one candidate, 0xbc22e83c, and it is REFUTED - read live it held 26 while
the HUD showed 017. The empty u32 filter is itself informative: the counter is
not a plain persistent big-endian word.
Finding REMAINING OB in RAM means getting to the escort stage, and
launch_mission.sh exists for exactly that. It carried three of the dead patterns
this session has been clearing: six vgamepad calls (the command no longer exists,
so the entire scripted route pressed nothing), the XUID E0300000EFBEA3D4 which
has no profile behind it and therefore opens the sign-in dialog that swallows
every keystroke, and the emulator's stdout going to /dev/null. wait_flight.sh had
a fourth, in the tap that skips the launch cinematic.
Fixed, the route works far past where it used to stop: boot -> title -> LOAD GAME
-> slot 01 -> READY ROOM -> TAKE OFF -> the Stage 02 briefing map, with ACROPOLIS
labelled on it.
And it stops there: 384 dumps at PC 0x82307128 and no flight HUD within 300s. The
briefing is still animating (two shots a minute apart differ by RMSE 2986), so
the emulator is not frozen - flight simply never starts.
One discriminator worth recording: the TUTORIAL reaches flight and has run
completely crash-free, while this path crashes during the briefing, which is
where the stage content loads. That is consistent with the cache-flush shape the
crash has always had, and points at WHAT is being loaded rather than at how far
the navigation gets.
REMAINING OB is therefore still not found - the mission was never entered.
The autopilot doc's own problem #2 is that the pilot ignores the mission
objective - its 300s run took no damage, killed one fighter, and watched
REMAINING OB RISE from 004 to 011. So this surveys where that knowledge is.
GP_TUTORIAL.pak yields the game's own list of what a player can do: BASIC
CONTROLS, HEADS-UP DISPLAY, RADAR, SUPPLY AND SPECIAL MOVES, RADIO ORDERS,
ADVANCED CONTROLS, with descriptions. Worth stating against what the pilot
actually does: pilot.py and navigator.py use move and attack only. Resupply,
special moves, radio orders to wingmen and advanced maneuvers are all unused, and
two of them bear directly on open problems - resupply is the untested RETIRE
mode, radio orders is an escort lever that flying skill does not substitute for.
tables.pak is a name-resolved config seam: 79 IDXD objects, 6275 distinct field
names, including DLG_MISSION_OBJECTIVE and DLG_STAGE_TITLE01..16 (so 16 stages).
GP_MAIN_GAME_<lang>.pak holds 131 weapon\ tables, 113 unit\ and 64 message\ -
the weapon ones being what problem #1 wants for real projectile speed and range.
But the objective TEXT is not text: DLG_MISSION_OBJECTIVE = pdscr099.prt, a
screen part, and the message tables resolve to face textures. All human-readable
text in this game is rendered sprite art, which is why the UI work has been
reading pixels all along. A per-mission objective table is an OCR job, not a
table dump - so the cheaper and more actionable route is the runtime REMAINING OB
counter, which is on the HUD and therefore in RAM.
The previous commit framed the empty capture as "the 3D draws never reach
CaptureShipDrawForRE" and proposed a static comparison to find out why. The
comparison was done and it refutes the framing.
The discriminator for ship geometry is stride=24 prim=4 with a large vcount -
xbg7-mesh.md records a real one as stride=24 vcount=10891 indices=18 prim=4. Not
"positions outside the screen rectangle", which is what I used and is a bad test:
a UI sprite placed at (137,308) passes it.
By the correct test this run's capture has exactly one prim=4 draw, a 6-vertex
full-screen quad, and the 2.9 MB file I was comparing against has NO prim=4 draws
at all. That file is a UI capture: 1303 of its 1582 draws are stride=24 prim=13,
which two other docs in this corpus already identify as the UI sprite shader. The
"1300 3D draws" it appeared to contain were UI sprite coordinates counted by the
bad test.
So the capture recorded exactly what was on screen. The tutorial's opening is an
empty starfield, the player's own ship and a HUD - no capital ship. And
ship-placement-runtime-capture.md has always stated the procedure: play into the
mission, frame the ship side-on, press F10.
What remains is therefore not a code question but a gameplay one: reach a real
mission and frame a capital ship. Worth stating that the original capture was
taken interactively on HW Vulkan, and this container runs lavapipe.
ship_capture_window.sh polls for the flight screen and presses F10 the moment it
appears rather than after a fixed sleep. One run gave three results.
First, the mission ran with ZERO crashes through t+152s - the first clean mission
run, where the three before it ended at 13243 / 11898 / 11497 - and it renders
and plays: player ship, starfield, full HUD, no dialog.
Second, the capture armed and wrote its file, so the mechanism works in-mission.
Third, and against expectation, the file holds NO ship geometry. 329KB, 181
deduped draws: 180 of them share a single vertex shader, all stride=28 vcount=3
prim=8 at full-screen coordinates, plus one full-screen quad, and not one draw
has positions outside the 1280x720 rectangle. The budget is not the limit -
kShipCaptureBudget is 8000 and only 181 distinct (vbase, WVP) pairs were seen -
and the scene was definitely drawing. The known-good capture from an earlier
session is 2.9 MB.
Fourth, a correction to the previous commit. It said the cache is "REFUTED as the
cure". Too strong: this run used the IDENTICAL complete cache as tut4 and
produced 0 crashes against tut4's 11497. What four runs support is that a
complete cache is not SUFFICIENT to prevent the storm and that run-to-run
variance dominates a 3-run comparison - not that the cache does nothing.
Next step is static: compare this capture's shape against the known-good one to
find why the 3D draws never reach CaptureShipDrawForRE.
This page's own reproduction - move the cache aside, get a throw; restore it, get
none - made "the cache is incomplete" the obvious explanation for the mission
crash. Three tutorial runs say otherwise.
tut2 subdir 6 MISSING 641 crashes by t+24s 13243 total
tut3 missing -> gained 6 in-run 2 crashes to t+80s 11898 total
tut4 6 present from the start 2 crashes to t+56s 11497 total
The missing entry was real: \aab216c3\6 was requested and absent, and the game
wrote it during tut3 (11 -> 12 files). tut4 then ran with a complete cache and
stormed anyway. Every dump across all three runs is PC 0x82307128, one address,
no others. So the cache may gate the boot-time throw this page documents; it does
not gate the mission one.
What the runs did give is an opening. Both post-cache runs show the same shape -
exactly 2 crashes, then nothing, for 56-80 seconds of rendering, advancing
mission - where the first run was already at 641 by t+24s. That is where the
second capital-ship capture has to happen: F10 armed INSIDE the window. Not a
guarantee, since the storm began at t+24s, ~t+96s and ~t+56s across the three
runs, but two of three offered most of a minute.
The 1663 refused resumes on thread F80001D8 looked like a second lost resume of
the kind fixed earlier today. They are not.
That thread DID execute - the log carries its XThread::Execute line, guest entry
82FFE6C0 - and then 137000 lines of silence. The silence proves nothing:
KeWaitForSingleObject and NtWaitForSingleObjectEx are declared kBlocking,
kHighFrequency, and PrintKernelCall skips every kHighFrequency export unless
--log_high_frequency_kernel_calls=true, which defaults to false. A thread parked
in a wait is invisible in these logs.
So the boring reading is the right one: the guest kicks a worker that is blocked
ON AN OBJECT, and Resume returns false because the thread is not SUSPENDED. That
is what a refused resume means, exactly as suspected before the count made it
look interesting.
Method note kept in the doc, because it is the reason one conclusion survives and
the other does not: the title-loader finding rested on 00:00:00 host CPU time
from ps -L and on the fix changing behaviour 5/5 against 1/5 - not on log
silence. In this codebase log silence alone is never sufficient evidence that a
thread is idle.
Also records a grep error of mine that briefly produced "zero kernel calls,
ever": the pattern ^[dikwF!]> missed the K> kernel prefix. Caught before it was
written down, and it changed the conclusion.
The backlog has said a second capital-ship capture needs the 0x82307128 crash
dealt with first, because the run died at SELECT DATA before any mission
existed. That is no longer where it stops. With the Canary threading fix making
the menu dependable, tutorial_launch.sh drives boot -> title -> menu -> TUTORIAL
and the mission LOADS AND RENDERS: flight HUD, "Go to the box on your screen",
warship counters, controller diagram.
It then freezes. 13243 crash dumps, every one at PC 0x82307128, guest thread 9,
Access Violation read at 0x10000000C, preceded by exactly ONE
RtlRaiseException(E06D7363) right after a HostPathDevice::ResolvePath() with
empty arguments - the cache-flush shape. Two screenshots 6s apart are identical
(RMSE 0), no new dumps accumulate, and the process still burns 400% CPU.
So: reachable, renders, unusable for a capture. The blocker moved rather than
lifted, which is worth recording precisely because the entry's stated reason is
now wrong.
One new lead, deliberately not claimed as a cause: the resume-refused diagnostic
added with the threading fix fires 1671 times on this path and 1663 of them are
the SAME thread, F80001D8, with 152 before the first crash - against about 7 on
the menu path. Resuming a non-suspended thread legitimately returns false and
"Resume to kick a worker" would produce exactly this, so it is a lead, not a bug.
It is cheap to settle: check whether that thread makes kernel calls between
refusals.
Cheapest next test named: this page already shows a COMPLETE on-disc cache
produces no throw at all, and the cache is 40MB with .partial/.cold-rebuilt
leftovers beside it. One boot to warm, one to re-test.
The three orders the derived rule was built from all live in GP_TITLE.pak, so
they cannot confirm it - the rule was fitted to them. These two are from
GP_SAVE_LOAD.pak, read off the running game now that the Canary threading fix
makes the main menu dependable.
The 9-element slot-list header composites EXACTLY as the sort predicts, on all 6
instances of it, and nothing about this screen was fed into the rule:
measured 7 8 0 1 2 3 4 5 6
derived 7 8 0 1 2 3 4 5 6
including TWO tied groups (0xb102 x2 and 0xb210 x5) that both come out in
declaration order, and the unkeyed pfeff00.prm fade quad last.
The 13-element save/load frame differs in exactly the two open questions and no
new ones: two unkeyed pfbase.tbm backgrounds paint FIRST where the sort puts the
keyless last - the splash's palogo_eff0.prm behaviour in a different file type,
so implied_layer_key now covers it - and the 0xb100 group of four paints
10,11,8,12 where declaration order is 8,10,11,12.
That second point is a SECOND screen with a mis-ordered tie, which is what the
question needed, and it immediately kills a candidate: 10 and 11 are kind=0x2002
while 8 and 12 are 0x0000, so "descending kind then declaration index"
reproduces 10,11,8,12 exactly - and then fails both title groups, where every
element of 0x8083 is kind 0 and where 0x80a0 would predict 2,3,4,5,0,1,7 against
a measured 0,2,4,7,1,3,5. Seven candidates refuted now.
16 disc tests green.
resume_reliability.sh counts, per boot, whether the title's loader thread became
the CALLER of a kernel call. Counting boots where the press reached the title:
before the fix 1 of 5 ran the loader; after it, 5 of 5. Three of the post-fix
boots were driven far enough to classify the screen and all three reached the
MAIN MENU, each with the same shape - title at 216-252s, 40 kernel calls by the
loader, 6 ResolvePath reads. The failing boots before had zero of both, every
time.
Caveat kept in the doc: pre-fix runs were interleaved with compiles and a race is
load-sensitive, so the counts corroborate the mechanism rather than carry it.
Also fixes a self-inflicted confound. Two runs came back 'title_at_s=none' with
the emulator log ending in 'Killed', which I first read as the title detector
being too strict. It was not: an earlier copy of the script was still running
because 'pkill -f resume_reliability.sh' had matched the shell running it rather
than the script, and each of that copy's iterations kills the emulator BY NAME -
outside run-canary's lockfile. The harness now takes its own lockfile and refuses
a second copy, and the note records that pkill -f matches your own shell.
resume_reliability.sh boots N times and records, per boot, whether the title's
loader thread was created and whether it ever became the CALLER of a kernel call.
That is the thing the threading fix targets, and it is binary; "did the menu
appear" is a worse test because it needs the game to get further and the attract
loop confuses it.
Two defects fixed on the way, both found by the harness disagreeing with itself:
* is_title.py's glyph count alone still fired at 113s and 173s, during the
intro movie, and the single press was wasted there. Measured, not guessed:
on those runs NO loader thread was created at all, and the title handler
always creates one - even on the boots where it then fails to run. The title
check now also requires the frame to be STATIC, reusing the RMSE between the
two grabs 0.6s apart that the movie branch already computes, and raises the
glyph threshold to 800 (a real title measures ~1450-1520, the SQUARE ENIX
logo 0).
* `grep -ac ... || echo 0` corrupted every CSV field it guarded: grep exits 1
on zero matches, so the fallback APPENDED a second value and the field came
out "0 0".
Retrospective before/after from the logs already on disc, using ResolvePath after
the resume (logged at every verbosity, so it is comparable across all runs):
before the fix 1 of 5 boots that created a loader thread went on to load assets;
after it, 2 of 2. n is small and a proper count is running.
The stalled loader thread is a lost wakeup in Xenia's POSIX threading, fixed on
the canary branch as a60fe7d11 and written up here. A thread created suspended
publishes state_ and suspend_count_ in two separate lock scopes, and Resume()
waits only for state_ before testing suspend_count_ == 0 - so a resumer in the
gap drops the resume and the thread waits forever. The Linux XThread::Resume
discards that false, which is why the guest saw success.
On the first clean boot after the fix the loader thread is the CALLER on 20
kernel-call lines and issues 4 ResolvePath reads. Every failed boot before it had
exactly zero of both.
Stated plainly as not shown: that boots now reach the menu RELIABLY. One post-fix
boot, and it is confounded by the harness.
Which is the second half. skip_intro.sh's title test has now been wrong twice in
opposite directions: originally one absolute pixel (625,618) - a 1280x720
coordinate against the 1279x675 game surface, so it read the copyright line and
timed out with the title on screen - and then my replacement, screen_id.py, which
is too loose and called the SQUARE ENIX publisher logo "title" 151s into a boot,
spending the script's single press there. is_title.py now counts the green (A)
glyph over the whole frame: geometry-independent and specific, measured at 0
pixels on the logo and 1520 on a real title.
Two findings, the second only visible because of the first.
1. The harness has always had kernel logging switched off. log_mask DISABLES
categories (Kernel=1, Apu=2, Cpu=4, Gpu=8), so the long-standing --log_mask=13
meant Kernel+Cpu+Gpu off; kernel calls also log at Debug while log_level
defaults to Info. Seeing one needs BOTH LOG_MASK=12 and LOG_LEVEL=3, and no
boot log this project has taken ever contained a kernel call. boot_menu.sh now
takes LOG_MASK / LOG_LEVEL / EXTRA_FLAGS. A whole boot at Debug with Kernel on
is 23 MB, so the default was costing far more than it saved.
2. With that on, a captured failure shows the (A) handler doing everything right:
XamUserGetXUID(0, 7, ...)
NtCreateEvent(...)
ExCreateThread(..., entry=821748F0, ..., 00000001)
ExCreateThread Active: Thread Initially Suspended,
XThreadF80000CC (1F) Stack: 70880000-70900000
NtResumeThread(F80000CC, ...)
and the thread then never executing. Measured two independent ways: it makes
ZERO kernel calls - it appears 13 times in the log and every one is as an
ARGUMENT, never as the calling thread, while five other threads make 31905
calls after the resume - and its host thread has 00:00:00 CPU time while the
process sits at 546% and has burned 37 minutes of CPU in 6:46 wall.
A spinning thread burns CPU. This one has not run at all.
So the chain is: press delivered -> handler runs -> thread created suspended ->
resumed -> never scheduled. Input, the cache-flush crash and the game's own logic
are all excluded. A lost resume is a race, which is the first explanation that
fits the ~1-in-3 success rate.
Not settled: where the resume is lost, and no successful boot has been captured
with kernel logging to compare against.
"The title screen ignores (A)" is withdrawn. First-divergence across three boots
of the same binary says otherwise.
A slot-(1F) guest thread is spawned BY the press: exactly once per run,
immediately after the keydown, same stack base 70880000-70900000 in both runs
that got one, and never at all in the run that never accepted a press - which
rules out a periodic worker starting around the same time.
prm6, reached the menu: (A) at line 6498, (1F) at 6500, 6 ResolvePath after
opt2, stuck on the title: (A) at line 1287, (1F) at 1288, 0 ResolvePath after
opt, stalled before title: no (A) ever, no (1F) thread at all
In the successful run the loader immediately reads six paths out of the on-disc
cache and the menu appears. In the failed run the same thread starts and performs
no file I/O ever again. Total ResolvePath for the three boots is 90/84/78 - the
successful run's extra six are exactly the ones after the press, so the boots are
otherwise identical in I/O.
Refuted as the cause: the cache-flush std::out_of_range. All four of today's runs
have zero GUEST-THROW, zero CRASH DUMP and zero Access Violation, and the guest
stays alive throughout with its keystroke-poll counter climbing past 15000.
Next probe is neither input nor the crash: what the (1F) thread waits on.
The tie-break needs a fourth measured screen with a tied group, and OPTIONS is
three d-pad steps from the main menu, avoiding the NEW GAME hang. It was not
reached.
Advancing past the title is intermittent: 1 success in 3 attempts today with the
same binary, profile and procedure.
Attempt 1 stalled before the title existed. The guest was alive - RSS steady at
598MB and the keystroke poll counter climbing past 15000 - but the screen was
black and screen_children.py found ONLY the splash object resident. So that
failure is not a presentation problem: the game never constructed the title.
Being able to separate "stalled before building the screen" from "built but not
presenting" is new, and it is what the reader is for.
Attempt 2 reached the title at 265s and refused four presses - (A) at 0.25s,
0.25s and 0.5s, START at 0.3s - every one of them logged as DELIVERED to the
guest. Third independent confirmation that input delivery is fine and the game
simply does not act on the press. Waiting longer does not help either: presses
were spread from 10s to about 4 minutes after the title appeared, while the
earlier successful run's press landed within seconds.
Records the log as evidence and states the next probe: it has to be guest-side
now, on whatever consumes the keystroke, since the input path is excluded.
With the layer key and the primitives' implied keys in place, the tie-break -
how the game orders elements sharing a key - is all that is left between the
derived order and ground truth. Three measured screens now constrain it.
On the menu and the splash every tied group comes out in declaration order,
which is what the stable sort already gives. The title is the only screen that
discriminates, and nothing predicts it: 0x8083 x5 paints eff1, eff2, eff5, eff3,
eff4, and 0x80a0 x7 paints logo1 x3, tm, logo2 x3.
Refuted: declaration order; RATC child order; first keyframe time (52, 56, 62,
58, 60 - the measured order is not sorted by them); resting keyframe time;
resting X or Y (938, 938, 64, 788, 447); and T8aD header words +00, +04, +0c and
+10, which are either identical within a group or unsorted.
Child order is worth its own line: a strict improvement over declaration order
(7 misplaced positions on the title instead of 9, and it recovers the logo
grouping) and exactly right on the other two screens. NOT adopted, because on the
only screen that can tell them apart it is still wrong.
Adds a test that measures what the residual costs instead of assuming it. Of the
3 disagreeing pairs of drawn elements across all three screens, all 3 have
overlapping bounding boxes and 2 share opaque pixels: ptlogo_back2eff5 against
eff3 (22568 px) and eff4 (32395 px). The third pair, ptlogo2 vs ptlogo_tm,
overlaps by two columns and shares NO opaque pixel - the wordmark is transparent
there. A bounding-box test called that a defect; reading the alpha says it is
not, which is why the test reads pixels. The set is pinned, so a change that
makes it worse fails.
15 disc tests green.
Two places checked for the key a .prm element sorts by, both empty:
* the declaration entry's four unread words are constant across every element
of all three measured screens (+28=0, +36=0xffffffff, +56=0, and +44 is a
button ordinal 1-5, 0xffffffff elsewhere);
* the bundle carries NO data for a primitive at all - the menu build declares
pteff00.prm, pteff02.prm and pteff05.t32 and has zero RATC children for any
of them, its 34 children being 21 T8aD sprites and 13 .rat records.
So the layer comes from the game code. But it is consistent, which is what makes
a per-name table a measurement rather than a fudge. Bracketing each unkeyed
element between the keys of its measured neighbours: pteff02.prm falls in
(0x8010, 0x8040) on BOTH screens it appears on, pteff00.prm is past the maximum
on both, and palogo_eff0.prm is below the minimum on the splash.
implied_layer_key records exactly those and nothing else; an unlisted primitive
keeps u32::MAX and still sorts last. With it, derived_paint_order produces the
same layer-key sequence as the order read off the running game on all three
measured screens - primitives included - and matches element-for-element on four
of the five bundle instances. The fifth is the title, differing only inside its
tied groups, which is a separate open question.
This does not make include_primitives safe by default: the 36 builds that come
out one colour are wiped by pzeff00.prm and pceff00.prm, never measured, hence
not in the table.
14 disc tests green.
The game receives the (A) press - measured, vk=5800 flags=0001, out of ~6500
polls. Input delivery was never broken; four harness bugs were, and with those
fixed the boot goes title -> one (A) -> main menu.
GP_DIALOG DIFFICULTY is still unreached (NEW GAME hits the standing black-screen
hang), but the main menu carries two primitives, which is what the question
needed. Records the slip in the instrumentation too: logging on XSUCCEEDED
counts X_ERROR_EMPTY as success and buried the two real keystrokes under 6499
empty polls.
Next step is now STATIC: a primitive's position is stable by role across three
measured screens but nothing in the file predicts it, so dump the full 60-byte
declaration entries - four words unread - and look for a word that sorts like
the measured orders.
Read off the running game with screen_children.py and identified by pivot
signature as GP_TITLE.pak ratc-index 8, the NEW GAME / LOAD GAME / TUTORIAL /
OPTIONS / EXTRAS screen:
paint order: 1 3 4 2 5 8 9 6 7 15 10 11 12 13 14 0
It is the first measured screen carrying TWO primitives, and they land in
different places, which is the point. pteff02.prm (the 25% dim) paints 4th,
beneath the whole UI; pteff00.prm (the transition fade, resting transparent)
paints last. Both match their positions on the title screen exactly. So a
primitive's place is per-element and stable by role - backdrop first, dim at
slot 4, fade last - and there are now three permutations to test a derivation
against rather than two.
Wired into measured_paint_order, keyed by element names so both language builds
get it. The English build composited with --primitives edge-correlates at 0.9591
at shift (0,0) against a framebuffer capture taken in the same session - a third
screen confirming paint order, resting pose, fade alpha and primitives at once,
against a capture this project had not seen before.
13 disc tests green.
Chasing why (A) does not advance the title turned up four harness bugs, one real
root cause, and a refutation of that root cause as the explanation.
FIXED, each verified:
* boot_menu.sh sent the emulator stdout to /dev/null, which is why none of the
rest was visible. It now keeps a log and prints its path.
* skip_intro.sh probed pixel (625,618) for the green (A) glyph - a 1280x720
coordinate. screenshot returns the 1279x675 game surface, where that point
is on the copyright line and reads (8,17,31). Symptom: a 600s TIMEOUT with
the title on screen. Now classified with screen_id.py; next boot printed
"TITLE at 239s -> A".
* skip_intro.sh no longer taps through the movies. Making that press real (the
vgamepad fix) was a REGRESSION: the boot then reached the title in 90s and
that title accepted nothing. menu_draw_capture.sh records the same finding
independently.
* boot_menu.sh signs in whichever profile exists instead of a hard-coded XUID.
ROOT CAUSE FOUND: naming a XUID with no profile behind it opens a sign-in
dialog, xam_dialogs_shown_ goes to 1, and IsUIActive() then discards every
XamInputGetKeystrokeEx. Measured 8,388,601 swallowed calls in one boot with the
pad log showing presses arriving normally throughout. Only B13EBABEBABEBABE
exists on disc; the script asked for E0300000EFBEA3D4. Signing in the real
profile takes the swallow count 8,388,601 -> 0.
AND REFUTED: that is not why the title is stuck. With a valid profile and zero
swallowed keystrokes, a single (A) on the title that ends the boot still does
nothing. The dialog bug was real and worth fixing; it is not the explanation.
Next probe is specific: --log_mask=13 records no kernel calls, so it is not even
known whether the game polls at the title. A log on the SUCCESSFUL
XamInputGetKeystrokeEx return, mirroring the one already on the swallow path,
answers it in one boot.
'skip_intro.sh: line 44: vgamepad: command not found' right after 'movie -> skip
A'. Every '-> skip A' and 'TITLE -> A' this harness has printed pressed nothing,
which is also why the intro movie ran to 434s: it was never skipped.
screen_children.py walks every resident screen object (vtable 0x820b30b4), reads
the element array at +0x08 and the child array at +0x30, and prints the paint
permutation with each pivot. Validated in one run off the live title: it
reproduces BOTH previously measured orders character-for-character - the splash
[0,2,4,6,1,3,5] and the 24-element title permutation - so the next screen it is
pointed at can be trusted.
The screen that matters was not reached. GP_DIALOG DIFFICULTY carries exactly
one primitive and would say whether the game paints it first or last, which is
the bit that decides where primitives belong. But (A) does not advance the title
screen at holds of 0.10, 0.25 or 0.40s, and neither does START, with the pad
verified end to end: pad.py writes /tmp/xenia_pad.txt, the emulator runs with
--hid=file --pad_file pointing at it, and the mtime updates on every press. The
title is unambiguously the interactive one - PRESS (A) BUTTON is rendered.
Recorded as a blocker rather than worked around. Next step is specific: boot with
the emulator stdout kept and read the [RE-INPUT] IsUIActive log already present
on the canary branch, which would say whether the keystroke is being swallowed
emulator-side.
Trying to read a third paint order off the running game turned up two bugs in
the capture harness, both of which fail in ways that look like the game
misbehaving rather than the script being wrong.
1. `--audio` is not a cvar in this tree, and eight boot scripts passed it.
Xenia calls ShowSimpleMessageBox from ParseLaunchArguments, BEFORE logging is
initialised, so the symptom is a 10x10 window, no log, no guest memory and a
dialog that blocks on XIfEvent forever - i.e. a hang deep in the emulator.
run-canary`s own header documents this exact trap; the scripts predate it.
Removed from all eight.
2. `vgamepad` no longer exists - the uinput pad was replaced by the --hid=file
driver and pad.py - but skip_intro.sh still called it. The script runs
without `set -e`, so the call failed silently and the title branch pressed
nothing while still exiting 0. A caller was told "TITLE -> A" with the game
sitting on the title screen. It now presses through pad.py and exits 6 if
that fails.
The first bug is fixed and verified: the boot now reaches the title screen with
PRESS (A) BUTTON. The second is fixed but does NOT unblock the title - see the
next commit.
fill_quad composites an untextured primitive as a solid rectangle of the
keyframes fade colour, pivot x 2 in size, placed and scaled exactly as a sprite
is. Behind ComposeOptions::include_primitives and screen render --primitives.
On the title screen, whose paint order is ground truth, it is measurably right:
mean luminance 76.30 -> 63.72 against the captures 64.58, i.e. from +18% to
-1.3%, and mean absolute difference 16.07 -> 13.08. The background was ~40% too
bright; pteff02.prm, a 25% black dim, was what was missing. The wordmark is not
dimmed by it because the measured order paints that quad at slot 4, beneath the
logo. Edge correlation moves 0.9538 -> 0.9480, which is not informative here: a
uniform dim scales gradients uniformly so a normalised edge score barely sees it.
OFF BY DEFAULT, and that is the finding. A primitive has no T8aD header, so no
layer key, and derived_paint_order sorts the keyless to the end. GP_DIALOGs
pzeff00.prm is a single keyframe of opaque black at full screen; painted last it
wipes the build. Of the 125 builds that draw a visible primitive, 36 come out
>99% one colour with the derived order.
No constant default works, because the two screens read off the running game
disagree: the splash paints its primitive FIRST (the black backdrop) while the
title paints one at slot 4 and another LAST (the fade-out). Declaration order
fails the title too. A disc test measures the damage rather than asserting the
feature works, so the number moves when the ordering is solved.
Also records a false alarm worth keeping: a first pass reported 36 GP_DIALOG
builds at "100% black", which was a crude near-black pixel threshold and not a
black screen - those dialogs are dimmed 50% and perfectly legible. The genuinely
wiped builds are a different set.
blit modulated by tint only, which is 0xffffffff on essentially every keyframe,
so the fade word was decoded, stored and then discarded. Applying it as an ARGB
modulate takes the title composite's edge correlation against the framebuffer
capture of the running game from 0.4597 to 0.9538, both at zero shift. The white
wordmark with its blue outline, the trademark, the copyright and the orange
exploding planet all appear -- the planet because a full-screen blue effect that
rests at alpha 0 had been painting over it at full opacity.
ARGB is measured: across a fade-in the high byte walks 0x00 -> 0x80 -> 0xc0 ->
0xe0 -> 0xff while the low three stay ffffff, and the low 24 bits are 0xffffff on
5276 of the disc's 5453 resting keyframes.
A modulate can only remove pixels, so the risk is a blank screen. Measured: no-op
on 4060 of 5200 sprite elements, partial on 453, hides 687 -- transient HUD
indicators that should not be lit at rest -- and NO build is left with nothing
visible. A disc test asserts that, and that the no-op share stays high, which
also guards the resting rule against drifting onto ramp frames.
Both changes are in one commit because the second is invisible without the first.
Applying fade erased the word PAUSE, which the running-game capture plainly
shows: pgptitle.rat has three runs of two identical keyframes -- pre-roll, hold,
exit -- and the 'later run wins' tie-break grabbed the exit. A group carries the
screen's entry animation AND its exit, so a run ending on the last keyframe is
now excluded unless it is the only one. The title correlation is unchanged.
Element::rest() picked the keyframe with the largest gap to the next keyframe's
time. That reads a keyframe as a value held until the next one; it is the start
of a ramp toward it. A long gap after keyframe k means the screen spends that
time arriving at k+1, so the settled pose is at the far end of the gap.
The title wordmark zooms in over five frames and holds at (184,193) at 100% from
t=251 to t=264. The old rule picked the frame before the long gap: (179,186) at
101%, still mid-zoom.
Measured against the framebuffer capture of the running title screen, which is a
1:1 crop so frame coordinates map directly (confirmed: the copyright line lands
on row 669 in the capture and in both composites). Edge-correlated over the
wordmark box:
plateau (landed) best 0.4597 at shift (0,0)
longest dwell (old) best 0.1511 at shift (+3,+8), 0.1268 at (0,0)
The old composite scores 3x lower and only peaks after being moved, by about the
(-5,-7) that picking kf4 instead of kf5 predicts.
It also fixes six title elements the old rule rested at alpha 0x00 where the
capture plainly shows them, and pteff00.prm - the full-screen fade quad painted
last - which rested at opaque black. That was the blocker on .prm compositing.
Adds tools/re-capture/align_to_capture.py, which is how this was scored, and
turns the .prm test that deliberately asserted the old defect into a guard on
the fix.
Not settled and now the next item: compose ignores the keyframe fade alpha
entirely (blit modulates by tint only), which is why choosing the wrong keyframe
was invisible until now.
369 .prm elements exist in the disc's screen builds and every composite is
missing them. Swept statically:
* none of the 369 has a RATC child of its own name — no payload, nothing to
draw, so .prm is a primitive and not a sprite;
* kind & 0x10 and a .prm name agree with ZERO exceptions in either direction
over all 965 builds, so the format marks primitives as a decoded field and a
port need not parse filenames;
* 361 of 369 are exactly 1280x720 at scale 100% in the corner, and their
keyframe 'fade' ARGB is overwhelmingly black at some alpha — these are the
fade-to-black, dim-behind-menu and flash layers, i.e. the PRMD dim-quad the
compositor's backdrop has been standing in for.
Refuted before believing: drawing them at Element::rest() is wrong. The title's
pteff00.prm is opaque -> transparent -> transparent -> opaque, a transition whose
resting pose is the transparent plateau; rest() picks by longest dwell and lands
on the opaque endpoint, which is painted LAST on that screen and would black out
the title. A test asserts that wrong answer deliberately so that fixing the
resting rule fails it and leads to the note.
No compositing change: the resting rule is not .prm-specific and has to be A/B'd
against the title framebuffer capture first.
The developer-logo splash declares its sprites directly and has no .rat layout
child, so is_build rejected it and no screen command could render it — despite
it being one of only two screens whose paint order has been read off the running
game, and the one where the layer key explains the whole permutation.
Adds ui_layout::is_composable (a declaration table plus at least one element
resolving to a T8aD the bundle carries) and an opt-in --all on screen
list/info/render. Measured on the disc: 2859 RATC bundles, 965 pass is_build,
2751 pass is_composable, and 0 pass is_build without passing it — a strict
superset. It is opt-in because the 1786 extra bundles are mostly two-element
fragments (a button and its glow), and because widening the default would
renumber --build for every pak, invalidating the build indices the corpus's
notes cite by number.
The splash now renders 6/7 elements, painting its glows first in the order
measured off the game; a disc test pins that order.
`compose` skips focused-state records by default, so whatever that flag matches
vanishes from every composite. It matched a trailing `f` in the name and nothing
else, which is not the convention — it is a letter.
Measured across the disc: 2458 elements match the suffix and only 54 of them
have the base element they would be the focused version of; all 54 are
`pgmenu_btnNNf.t32`. The other 2404, spread over 864 bundles, are `_eff` glow
layers whose names end in the same letter — `pb_name_eff.t32` alone accounts for
1122 of them, and `palogo_gamearts_eff.t32` is one the draw capture shows the
game painting, before its logo, with its own layer key.
Requiring the pair recovers 587 glows in the composable builds. GP_OPTIONS is
the plainest case: without `pbmwindow_eff.t32` the panel had no window at all,
just two floating brackets (captures/ui-layout/options-with{out,}-its-glow-layer.png).
Two other candidate rules were tried and refuted first. The `opt` link is not a
focus link: 221 elements are opt-targets, only 2 of them suffix-match, and the
targets include `pjnet_bg.rat` and `pv_loading_loop1.rat`. And the suffix alone,
as shown above, is unrelated to anything.
The layer-key order was adopted from two measured screens and then applied to
every build on the disc, so it owed a regression check against the screens the
corpus had already validated against the running game.
Rendered the tutorial PAUSE menu and the title main menu both ways and diffed:
3.8 % and 1.1 % of pixels differ, max delta 45/255 and 34/255, and the two
renders are indistinguishable in layout — the change is confined to blends where
translucent sprites overlap. No regression, but which order is more faithful on
those two screens is unsettled and recorded as such.
Adds a corpus-wide test asserting every composite's draw list is strictly
increasing in (layer key, declaration index), streaming one pak at a time so it
does not OOM alongside the other whole-disc tests. It reports the rule's reach:
341 of 965 builds are reordered, and it fails if that share collapses.
