Locating DeltaSaber at mission time works: 53 occurrences, all in the heap
(0xBC66..-0xBC6C..), none in the XEX image -- DeltaSaber_T.xpr, _Special,
_NoseGun, _Missile, _TwinGun.
But searching all of guest memory for a big-endian pointer to those addresses
returns ZERO references for every one tried. That is the format, not a search
bug: those strings live in the mission pak's IDXD string pool, and this corpus
decoded that container long ago -- records reference names by OFFSET into the
pool, never by absolute pointer. So there are no pointers to find, and "find the
name, follow what points at it" cannot work on pak data by construction.
This also narrows what the string counts proved. The title-vs-mission control
stands (DeltaSaber 0 -> 81), so mission-specific DATA is loaded, which is more
than "the tables load at boot". But these are asset-table strings, not live
entity objects, so they do not show entities have been spawned. Honest split:
the mission's pak data is loaded; whether entity objects exist is still
unmeasured, because both probes tried -- gworld's vtable constants and
name-pointer following -- are respectively stale and structurally inapplicable.
Remaining routes, untried: derive the entity vtable from CODE via sylpheed.db's
vptr_writes table, which exists for exactly this; or find the entity list from
the mission update function rather than from data.
Three iterations concluded "no units are loaded" from scan_vtable(DEF_VTABLE)
returning 0. Withdrawn: the scan was measuring the wrong thing.
Counting unit-name strings in guest memory with the TITLE as a control (same boot
recipe, no mission):
string title after take-off
DeltaSaber 0 81
rou_ 1,981 7,680
UN_ 19 167
e010 67 134
ADAN 1,450 1,921
DeltaSaber is the player's craft: absent at the title, present 81 times after
take-off. Every other count rises several-fold. The stage content is
unambiguously in memory.
The control is what makes this a result. Raw counts prove nothing on their own --
rou_ and ADAN are numerous at the title too, because the unit tables load at boot.
Only the title-vs-mission difference, and DeltaSaber's 0 -> 81 in particular,
separates "tables loaded" from "mission loaded". My first reading skipped the
control and over-claimed; the control was run before publishing.
Withdrawn as a consequence: "no units are loaded"; and the inference that the
take-off lands in a never-ending cutscene (the movie accesses are real, but the
conclusion rested on DEF_VTABLE=0). gworld.py's DEF_VTABLE 0x820AF844 and
INST_VTABLE 0x820AF030 do not locate entities in this build/state despite being
genuine vtables in sylpheed.db.
Next: derive the correct entity vtable. The UN_ strings reachable by search are
UN_NOSE/UN_MOUNT attachment names in the XEX image, not runtime records -- so use
a name that only exists at mission time. DeltaSaber is exactly that: find its
heap occurrences, find what points at them, read the referencing object's vtable.
wait_flight.sh carries the corpus's best discriminator, and it is a whole-image
statistic rather than a named pixel: the flight HUD paints green over the frame
at 1.3-1.5% green fraction, against <0.1% on every menu.
Measured on the post-take-off state: green=0.0022, mean (1.6, 45.6, 50.1). That
is 0.22% -- an order of magnitude above a menu, an order of magnitude below
flight, on a dark blue-green frame. screen_id.py files it under "menu" because
that is the closest bucket it has; the state is neither.
wait_flight.sh then tapped A for 280 seconds and never saw the HUD.
With the 12 \dat\movie accesses, the reading is consistent: this is the cutscene,
it does not end on A, and it does not reach the flight HUD within five minutes.
Open, cheapest first: how the movie path signals completion (the corpus already
decoded the movie handler for the intro video and found a software-decode
deadline); whether a mission can be started by a route that skips the launch
cinematic; or whether the cutscene simply needs far longer under lavapipe.
Oracle status overall: boot, menus and take-off are reliable and reproducible;
only the cutscene->mission step is unsolved -- a much smaller and better-defined
gap than the freeze it replaced.
Logging file and kernel activity through the whole navigation and counting names:
12 \dat\movie <- heavy, repeated movie access
3 \dat\sound
2 Stage_S01 <- the stage resource is referenced
1 XamShowDeviceSelectorUI
So after take-off the game plays the launch cutscene. Under software rendering a
720p decode is very slow -- this corpus already raised INSTRUCTIONS_PER_MS 100x
because the movie handler has a 2000 ms software-decode deadline -- which is why
the state persists for minutes.
It is unambiguously live, by the widest margin yet measured:
frozen menus this state
screen liveness 0.00% 0.22% 11.00%
guest churn 0.000% 0.190% 0.671%
and the progress counter advances on input (17 -> 101 across the skip attempts),
so the pad reaches the game.
But it does not skip: four rounds of START / A / B advanced the counter every
time and left DEF_VTABLE / INST_VTABLE at 0/0. The input registers, it just does
not end the sequence.
The remaining gap is now narrow and named: the take-off cutscene runs and is not
skippable by the obvious buttons. Either it must play out, or the right skip
input is unknown, or it waits on something the movie path never delivers.
With the emulator launched correctly and the boot verified by the progress
counter first (counter > 0 = title reached; a flaky boot never leaves 0),
nav_to_flight.sh ran the whole sequence -- main menu at 20 s, mask poke, stage
select, briefing skip, ready room, take-off -- where every previous attempt
stalled.
The game is alive at the end of it: screen liveness 2.13% against 0.00% when
frozen, guest churn 0.288-2.932% against 0.000%. An animating screen and a busy
guest at the point that used to be stone dead.
But no units. DEF_VTABLE (0x820AF844) has no hits on a direct byte search, with
a working control in the same run: searching for the poked cleared-stage mask
0x0001FFFE finds it at 0x828F40C0, exactly where nav_to_flight wrote it. So the
search is sound and the vtable is genuinely absent. Both vtable constants were
also checked against sylpheed.db's vtables table and are real vtables, so they
are not stale. Three minutes of waiting left DEF/INST at 0/0 with the counter
static at 17.
So the run reaches a live, animating state that is not a mission with spawned
units -- most likely the launch cinematic or a post-take-off screen. The freeze
is gone; what still separates this from unit spawn is the remaining work.
nav_to_flight.sh contains no run-canary invocation -- it drives an
already-running emulator -- so whether a profile is signed in depends entirely on
how the caller launched it. Every nav_to_flight run in this session was launched
with a bare run-canary, which is why they froze. Worth stating plainly because
the script prints "signing in profile B13EBABEBABEBABE", which comes from the
emulator's startup rather than from the script arranging it.
Refuted: "each screen loads its own GP_*.pak", a plausible route to a real screen
identity. With full file logging the only names that appear are at boot
(sound.p0x, tables.p00, hidden) and nothing new opens as the UI moves between
screens -- the paks are opened once and read by offset, so file I/O carries no
per-screen signal.
Also recorded: one launch this iteration never reached the title -- progress
counter stuck at 0, black screen, guest churn 0.001%. That is a DIFFERENT
failure from the sign-in freeze, which stalls after the main menu with the
counters already advanced, and it matches the boot flakiness
canary-scripted-input-traps.md already documents. The counters distinguish
them: sign-in freeze stalls past the title, a bad boot never leaves 0.
The names in menu-state-in-memory.md are wrong for at least two of the three
words, and the test that shows it is going backwards. Driving deep into the
menus, then three presses of B:
0x828F38AC "cursor" 36 -> 38 -> 40 -> 41
0x828F37B4 "misc" 971 -> 1067 -> 1068 -> 1068
0x828A690C "screen" 56 -> 56, 56, 56
A cursor returns when you go back. These only ever increase -- monotonic
counters, with 0x828F38AC advancing about 2 per input. Driving forward produced
1,3,4,5,6,8,10,12,25,29,32,33,50,53,56 for the "screen id", which is an identity
sequence only if the game has 56+ screens and never revisits one -- exactly what
a counter also looks like.
Withdrawn: 0x828A690C as a screen IDENTITY (1 title, 3 main menu, 4 extras). The
values are path-dependent; they matched across runs because the same key sequence
produces the same count, not because 3 means main menu.
Survives: all three advance if and only if the game responds, and are stable when
it does not. That is a real input-progress signal, reproducible across runs and
both GPU backends, and it is what made blind navigation work. Read it as "did
the game react?", never "which screen is this?".
It also retro-confirms the freeze diagnosis: with the sign-in fix the sequence
runs 3 -> 5, skipping 4 entirely, so "4 = extras" was never a screen -- it was the
count at which the game stopped responding. The counter reading explains both
observations.
Open: no mission reached. Counters at 56/41/1068, guest churn 0.190% (alive;
frozen is 0.000%), DEF_VTABLE and INST_VTABLE still 0. Without a real screen
identity, navigation is dead reckoning; finding a genuine state enum is next, and
the snapshot-and-diff method can be repeated with these counters excluded.
Every Xam UI dialog goes through xeXamDispatchDialog, and the calling guest
thread blocks on fence.Wait() until it is dismissed. With kernel logging on the
last call before the freeze is XamShowSigninUI(00000001, 00000001): the game
asks for a signed-in profile, Xenia opens a modal dialog, and nothing in a
scripted run ever dismisses an ImGui dialog.
That accounts for every symptom at once -- Main XThread futex-blocked at 0 ms CPU
rather than spinning, emulator alive, no guest progress, no faults, and
independence from both the GPU backend and the allocation outcome.
Fix: run-canary --logged_profile_slot_0_xuid=B13EBABEBABEBABE. Same route, one
variable changed:
no profile with profile
screen id 4, forever 4 -> 5 -> 6 -> 8 -> 9 -> 10
XamShowSigninUI called not called
alloc failures 1 0
guest throws 1 0
guest churn 0.000% 1.006%
XamShowDeviceSelectorUI is then requested but storage_selection_dialog defaults
to false, so it is not shown and does not block.
Why it took so long: --log_mask=13, used by every script in this corpus,
DISABLES kernel logging -- the one category that names the dialog. Without it
the freeze presents as a bare futex block, which is why it was attributed in
turn to a heap leak, rounding, MmQueryStatistics, a build regression, the route,
the savegame, shader compilation, an infinite guest loop and the software
rasterizer, each refuted in its own section.
Nearest miss worth recording: challenge-mission-gate.md reported a "Disc Read
Error" dialog on this failure. That is XamShowDirtyDiscErrorUI, which calls
exit(1) -- "This is death, and should never return" -- so it cannot be our
freeze, whose process stays alive. Checking that distinction is what led to the
dialog mechanism.
Open: no mission reached yet (DEF_VTABLE/INST_VTABLE still 0 at screen 10), so
the remaining screens are menus. Ordinary navigation, not a blocker.
Driving the null backend blind, using the memory screen-id as the guide, to the
SAME screen the rendered run froze on (screen 4):
guest churn 19 of 8,089,600 bytes in 4 s (0.000%)
Main XThread 0 ms CPU / 4 s, state S, wchan futex_do_wait
log AllocRange unable ... 134217728 bytes; guest throw
Frozen identically with no rasterizer at all.
The flaw is the comparison, not the data. The earlier "control" measured the
null-GPU run while it was still in the MENUS and the lavapipe run AT the freeze,
then reported the difference as a backend effect. The allocations I cited as
proof it had walked through the content load (114 -> 120) were menu-time
allocations. A control needs both arms verified to be in the same state -- and
the memory screen-id built this same session is exactly what makes that
checkable, which I failed to use on the arm that mattered.
Withdrawn: "the freeze requires the rasterizer", "the fault is in the host
rendering path", and the hardware-Vulkan blocker that followed from it. The
rendering question is no longer load-bearing.
Survives: the freeze happens on the first content load after the main menu, on
every route and BOTH GPU backends, with Main XThread futex-blocked at zero CPU
rather than spinning; and four runs froze with three different allocation
outcomes, so the 128 MB refusal is correlated but not necessary.
Also survives and is independently verified: the memory screen-id navigation,
now reproduced on a third run. It is what exposed this error.
Open again: what is Main XThread waiting on? Identify the futex and its holder,
carefully -- gdb perturbs this measurement.
Menu navigation here has always been screenshot-driven, which is unusable under
--gpu=null -- the only configuration where the game does not hit the
software-rasterizer freeze. Without a memory signal, the one backend that runs
is the one that cannot be steered.
Snapshotting 0x82800000+3 MB at each menu of a rendered run and keeping the
4-byte words that differ between screens and hold small integers leaves exactly
four of 786 432. One has the property that matters -- it changes on a screen
transition and holds steady when only the highlight moves:
0x828A690C screen id 1 title, 3 main menu, 4 extras
0x828F38AC menu cursor (second copy at 0x828F38BC)
0x828F37B4 per-menu value
Verified on a fresh --gpu=null run with no display at all, driving the same keys
blind: title 1/2/12, main menu 3/4/45, after 4x down 3/12/45, extras 4/14/49 --
4/4 exact against the rendered run, across two runs and two GPU backends. That
is the check that matters, since this corpus has already had to mark one runtime
address run-dependent.
tools/re-capture/menu_state.py reads them; `menu_state.py watch` prints on
change.
Open: the rest of the sequence into a mission. Blind driving reached extras
(screen 4) and a further A did not move it, so MISSION SELECT needs a cursor
move first. Screen ids beyond 4 are unmapped, and the rendered run freezes on
entering that screen -- so map ids up to the freeze, then step blind past it.
The cleanest control available, and it settles which side the bug is on. Xenia
has a null graphics backend. Same ISO, same route, same presses:
lavapipe --gpu=null
Main XThread CPU 0 ms / 3 s 3860 ms / 4 s
state / wchan S / futex_do_wait R / not blocked
guest memory churn ~0 (18 B of 8.2 MB) 22.6% of 6 MB in 4 s
allocations stall 114 -> 120, no failures
process CPU 265% (all llvmpipe) 330% (mostly guest)
With no rasterizer the guest does not freeze: it runs at close to a full core and
walks through the content load -- the exact point that hangs every rendered run.
So the fault is in the host rendering path, which explains why every game-side
hypothesis on that page was refuted in turn.
Blocked, and recorded as such rather than improvised around: deciding whether
lavapipe is deadlocked or merely taking hours over one pathological draw needs
hardware Vulkan, which this container does not have. Six minutes of nine
saturated llvmpipe threads with no frame is suggestive, not conclusive.
Amber workaround: --gpu=null gives a live guest, and everything the backlog needs
-- entity positions, flight model, world unit -- is read from guest memory, not
pixels. The gap is navigation: menu-walking is screenshot-driven, and blind A
presses advanced allocations 114 -> 120 but never reached a mission (DEF_VTABLE
and INST_VTABLE scans both 0). Next step is to drive navigation from guest
memory instead of the screen, which is engineering rather than mystery.
The previous section concluded the guest spins forever in sub_82457780.
Measured without gdb, that is false. Every observation behind it came from a
gdb-hosted run, and gdb intercepts every SIGSEGV -- which Xenia uses for guest
memory watches -- so it perturbs precisely what was being measured.
Same measurement, no debugger, per-thread /proc sampling parsed after the last
')':
Main XThread 0 ms CPU per 3 s, 0 faults, state S, futex_do_wait
GPU Commands 10 ms
WSI swapchain queue 130 ms
llvmpipe-0..9 ~1030 ms each per 6 s
Process at 265% of a core, essentially all in the software rasterizer. The
guest thread is BLOCKED on a futex using zero CPU -- not spinning, and not in
sub_82457780.
Under gdb that same thread appeared as the top CPU consumer (890 ms / 4 s) and
appeared to be in guest JIT code, then in xe::ExceptionHandlerCallback. That
was signal interception plus gdb re-stopping it between samples (state t,
wchan ptrace_stop).
Withdrawn: "an infinite copy loop in sub_82457780 is the freeze", and "Main
XThread is the top CPU consumer, so it is a spin not a block". The opposite
holds.
Survives: the emit_source_annotations technique is sound and did prove the PC
was 0x824578A0 while the guest ran under gdb, so the guest does pass through
that copy loop. The bne-exact-equality reading remains an accurate description
of the disassembly and a plausible hazard -- just not this freeze.
What the freeze actually is: guest blocked on a host futex at zero CPU, software
rasterizer saturated for six minutes without producing a frame, nothing
faulting, screen unchanged. That is the guest waiting on a GPU operation that
never completes, with lavapipe spinning -- a HOST RENDERING problem, which is
also why every game-side hypothesis this session was refuted in turn.
Method rule earned the hard way and recorded on the page: do not diagnose a
performance or liveness question under a debugger. Use it to read state at a
known stopping point, then re-measure timing and CPU detached.
The previous section withdrew this localisation for resting on a stale
PPCContext. It is now re-established by a sound method.
Xenia has a CPU cvar, emit_source_annotations, that makes MarkSourceOffset emit
"nop; nop; mov eax,<guest_address>; nop; nop" at EVERY guest instruction
boundary -- 90 90 B8 xx xx xx xx 90 90 in the code bytes. So the guest PC reads
straight out of memory around rip, with no context and no offset guessing.
Dumping backwards from rip gives a clean run of them -- 0x82457874, 78, 7C, 80,
84 -- and the annotation immediately before rip is 0x824578A0. That is
`sth r6, 0(r9)` inside sub_82457780: the exact instruction predicted from the
disassembly last time, now reached independently.
The loop advances r11 by 8 and terminates on `bne cr6` -- exact inequality. If
r31 - r7 is not a positive multiple of 8, the comparison never becomes equal and
the loop never exits.
And the thread is genuinely running: over 4 s, Main XThread consumed 890 ms of
CPU, the top of any thread, with the process at ~238%. A spin, not a block.
One bad metric recorded: I tried to show the loop marching through memory by
watching the last non-zero byte above 0x70200000. It never moved -- but that
statistic saturates in an already-written region, so it could not have moved.
Not evidence either way, though it briefly looked like a refutation.
Open and now narrow: what makes r31 - r7 non-congruent to 0 mod 8. r7 comes
from 0(r30), r31 is the container end, and r3 holds a fresh buffer from
0x824F7240 -- read those three from HOST registers at the freeze.
Went back to verify the instrumentation before building on it. Some does not
hold.
Verified: the trailing comments in ppc_context.h are not struct offsets (the
struct opens with eight cr unions and fpscr), but rsi+0x20 for r[32] is
nonetheless correct -- confirmed empirically because the f[32] doubles begin at
+0x140 and 32*8 back from there lands on +0x20.
Withdrawn: the context is STALE mid-function. The same dump shows r1 = 0, and a
guest stack pointer cannot be zero inside a function -- Xenia's JIT keeps live
guest registers in host registers and spills only at certain points. So
r13 = 0x82457864 does not show where the guest is executing, and identifying
sub_82457780 as the stuck function is withdrawn as unproven. The description of
what that code does (grow-and-copy, beq exit) is still accurate; it is just not
established that the guest is in it.
Three readings retracted for broken instrumentation:
* "rip identical across samples ⇒ no progress" -- gdb stops the process
between samples, and the JIT lays code out differently per run (a05be939 vs
a05bc839 for the same instruction).
* "zero minor faults on the stuck thread" and "utime=0" -- both parsed
/proc/<pid>/task/<tid>/stat with awk positional fields, but comm is
parenthesised AND contains spaces ("Main XThread (F"), so every index after
it was wrong. Printing state=XThread should have been the giveaway.
* the SIGSEGV fault-storm refutation rested on the same parse; downgraded to
unproven, though the process-wide ~300/s still argues against a tight loop.
Survives: the freeze reproduces without gdb; with the process stopped exactly
one thread is in userspace JIT code while a wchan census puts the rest in
futex_do_wait (55) or hrtimer_nanosleep (11); the instruction is a 16-bit store
through the membase to guest 0x701d0000, a stack-region address, first of a
four-store group; and that memory was unchanged over seconds of running time.
Next step is a host->guest code mapping from Xenia's own code cache rather than
inference from a stale context.
Ran the freeze under the corpus's gdb wrapper (ptrace_scope is 1, so a debugger
must launch rather than attach, and its handle lines are needed because Xenia
uses SIGSEGV for guest memory watches).
One thread of 80 is in guest code: thread 50, Main XThread, at rip a05be939 in
JIT output. All others are in a futex or clock_nanosleep.
Xenia's x64 backend keeps PPCContext in rsi, with r[32] at +0x20. The guest
GPRs there give r8 = a3ac0000 (the 64 MB buffer from the doubling sequence),
r12 = a3ac0a18 inside it, and r13 = 82457864 -- guest code, sub_82457780.
That address sits in a grow-and-copy: size = count*8 clamped to 0x1FFFFFFF, a
call to 0x824F7240, then a loop copying halfwords eight bytes at a time. The
host instruction it is stopped on is that copy's store:
mov %r12w,(%rdi,%rax,1) rdi = membase, rax = 0x701d0000
Sampled three times seconds apart with continue in between: rip, r13, r11 and
r31 identical every time. No progress.
The loop's exit test is beq -- equality, not >= -- so an inconsistent start/end
pair never terminates it. Recorded as amber: that is a reading of the
disassembly, not a demonstration.
Two of my own readings corrected:
* "the guest spins at ~400% CPU" was ps's CUMULATIVE AVERAGE since process
start, not an instantaneous rate. Per-thread sampling puts Main XThread
nowhere near the top.
* a SIGSEGV fault storm fitted the constant rip nicely and is refuted: 1500
minor faults in 5 s, zero major.
Caveat kept on the page: these are gdb-hosted observations and gdb intercepts
every SIGSEGV, so absolute timings are not the ungoverned ones. The freeze is
not a gdb artefact -- it reproduces in every non-gdb run.
Four runs now freeze at the first content load after the main menu with three
different allocation outcomes: stock (fails at 128 MB and throws), GPU-writeback
patched (no failure, stops at 32 MB), heavy XMA logging (no failure, stops at
16 MB), and no-savegame (no failure, no throw). The memory account on this page
is accurate but describes a symptom; everything treating the 128 MB refusal as
the freeze is superseded.
The savegame is not the variable. --content_root pointed at a profile-only tree
(the real content untouched) boots to the identical pools -- BC220000 Size
03A80000 and B50C0000 Size 070E0000, same addresses and sizes, same 43
allocations by the menu -- then freezes at the same press with zero failures.
Not slow shader compilation either. Every earlier run called it frozen within
~30 s; this one was left six minutes at 0.00% liveness with CPU still 399-479%.
Correction made mid-measurement: the allocation counter creeping 119->124 looked
like "guest running, display stuck". It is not. A broad sample -- 400 extents,
8.2 MB -- changed 18 bytes in 3 seconds. The guest is spinning, with one thread
allocating about once a minute. The earlier "8 MB slab unchanged" reading was
weak for the same reason in reverse: one arbitrary region proves little. Sample
broadly before calling a guest alive or dead, exactly as the screen needed two
frames rather than one.
Now excluded: the heap failure, the leak, rounding, MmQueryStatistics, a
heap-size knob, the exception cvar, a kernel-object wait, a build regression, the
route, the savegame, and shader compilation.
Read from a LIVE guest (main menu, liveness 98.59%) rather than a frozen one,
sampling a page per megabyte:
B50C0000 112.82 MB 89.7% zeros, header all zero
BC220000 58.45 MB 90.8% zeros, header = pointers into itself
Neither holds content at the main menu and neither matches a disc file's size.
They are pre-reserved pools claimed at boot and mostly untouched.
BC220000's first words -- bc220010, bc79c9c0, bcd24c00, bc220040, bd17cda0,
bc79ccc0, all inside [BC220000, BFCA0000) -- interleaved with ordinals 6,0,4,5,
0,1,2,3, are an intrusive free-list. That closes a loop from this morning: the
22 "Release failed because address is not a region start" messages were frees of
interior pointers into this exact block. The game suballocates from its own
pool and hands the sub-pointers to MmFreePhysicalMemory. Independent
confirmation that Xenia is right to refuse them.
It sharpens the contradiction rather than resolving it. The budget adds up --
~171 MB of pools, ~379 MB live at the menu, 128 MB wanted for a content load,
about 495 MB of a 512 MB console against Xenia's fixed 16 MB GPU reservation,
which is the measured ~15 MB shortfall. But the SAME emulator with the SAME
reservation ran a mission for 500 s on 08-10, and that run's evidence is entity
hull values sampled from guest memory at 2 Hz, not screenshots, so it is not the
frozen-frame illusion that fooled this session twice.
So the next thing to vary is the boot inputs -- profile, save data, config --
since B50C0000's request is 0x070D28B0, an odd computed size rather than a round
reservation, and so plausibly depends on something configurable.
Last iteration concluded the freeze is "entering MISSION SELECT" and made
avoiding that screen the next experiment. Ran it; the conclusion was too
narrow.
First, a liveness metric that actually separates the states: two frames five
seconds apart, percentage of pixels changed. The menu animates, so healthy is
99.80-99.97% and frozen is 0.00%. No navigation script needed, and no
classifier. Committed as tools/re-capture/route_liveness_probe.sh; this is what
should have been used from the first run.
Then the menu's FIRST item, NEW GAME, which never touches MISSION SELECT:
main menu 99.80% alive
after A on NEW GAME 7.33%
after the next A 0.00% -- frozen, and screen_id calls it "flight"
with the same 134217728-byte AllocRange failure in the log.
So the correct statement is broader: the game freezes on the first content load
after the main menu, whichever item is taken. MISSION SELECT was just the route
every earlier run used. "Avoid MISSION SELECT" is withdrawn -- there is nothing
to avoid, and that also puts the memory account back at the centre, since ~379
MB live plus a 128 MB content load fails on any route.
Worth repeating because it caught me twice: screen_id.py called a frozen frame
"flight" on a run that never left the menus. Liveness first, classification
second.
The whole page called this a stage-load/take-off freeze. Wrong, and the
evidence was in each run's own screenshots.
nav_to_flight.sh saves a shot per step. Compared CONSECUTIVELY:
01-extras -> 02-missionselect rmse 51.07, 99.42% changed
02-missionselect -> 03-selected rmse 0.00, 0.00% changed
03 -> 04 -> 05 -> 06 -> 07 -> 08-flight all 0.00
Shots 02 through 08 are pixel-identical. The screen stops at MISSION SELECT and
never changes. Every later step the script reports -- selected, readyroom,
takeoff, flight -- is it pressing buttons at a frozen image while screen_id.py
classifies a static frame.
Withdrawn as a result:
* "MISSION SELECT and the stage list worked, the failure came later during the
take-off load". It froze entering MISSION SELECT.
* challenge-mission-gate.md §5.6 already said entering MISSION SELECT fails on
a 128 MB request and shows "Disc Read Error". It was right; I contradicted
it because my navigation script claimed to have got further.
* Every "reached flight" in this session is false, including runs where
screen_id.py returned "flight".
Method note kept on the page: the first comparison ran every shot against shot 1
and returned rmse 51.07 / 99.42% for all seven, identical to two decimals across
seven different images. That is a tell, not a result. I then talked myself out
of it because md5sum showed eight distinct hashes -- which was PNG metadata.
Compare consecutive frames, and treat a suspiciously constant statistic as a bug
in the measurement.
Consequence for the oracle: the allocation failure happens while the display is
already frozen, so it is downstream, which fits the three runs that froze with
the allocation succeeding at 16, 32 and 128 MB. MISSION SELECT is the thing to
avoid -- every route this session used goes EXTRAS -> MISSION SELECT, and
newgame_path.sh / tutorial_launch.sh reach a mission by other menus.
This corpus has a 500-second Stage 02 flight from 2026-08-10, and the canary
tree gained the file-pad driver, the UI-draw capture, a threading_posix resume
fix and log_stuck_waits between then and now, with the running binary rebuilt on
08-24. So a regression was the obvious suspicion.
It is wrong. The container already keeps older builds under
/sylph-home/re/bin/, plus a host build from 08-17 that predates all of the 08-19
changes, and run-canary honours $XENIA_BIN -- so this cost no rebuild. The
08-17 binary, same route, same point: 1 alloc failure, 1 guest throw.
Identical.
So bisecting the emulator is not the way in, and the 08-10 run did not differ by
binary either. Whatever let it reach flight is in the route or the game state.
That sharpens the open question usefully. Both of the two largest live blocks --
112.88 MB at B50C0000 and 58.50 MB at BC220000 -- are allocated at boot, before
any menu, so 171 MB of the 379.5 MB is fixed regardless of route. The remaining
~208 MB is where a route difference could live, and the ledger can measure it:
capture live-bytes at the moment TAKE OFF is pressed for two navigation paths
and compare.
The canary branch already carries log_stuck_waits, written for this exact
question: it names the object a guest thread keeps timing out on, and is silent
on a healthy run because a wait that gets satisfied never builds a streak.
Enabled it and ran to the freeze on the stock build. Not one stuck-wait line,
alongside the usual AllocRange failure and guest throw. So no guest thread is
parked on a kernel object that never gets signalled; combined with 389% CPU
across running threads, the guest is spinning in its own code -- which is what
the throw-that-returns predicts, since execution resumes after the throw and
runs into code that assumed it would not.
Operational note worth having: --log_stuck_waits=true on the command line is
ACCEPTED but the config file value wins, and the first run silently logged
"log_stuck_waits = false" while I thought it was on. The startup dump prints
the effective value -- check the dump, not the flag. The flag is cheap and
silent so it is now left enabled in the container config.
Rebuilt canary twice to test the freeze, and the result demotes my own headline.
1. The game does not size anything from MmQueryStatistics. Xenia reports
kernel_pages = 1 MB under a comment admitting the numbers are guessed, and
the game really does call the export -- sub_82612420 converts
total_physical_pages and title.available_pages to bytes, and its caller holds
available-bytes in r23 while creating render surfaces. Patched it to 32 MB,
twice the shortfall, and rebuilt: 122 allocations, 484.5 MB ever, 379.5 MB
live in 84 blocks, failing at "free 28969/131072" -- byte-identical to stock.
A 32x change moved nothing. Refuted; reverted.
2. The ~19 MB the ledger could not see is Xenia's own startup reservation:
memory.cc:240 pins 16 MB of the parent heap for GPU writeback before the
guest runs, and vC0000000's parent IS the 512 MB physical heap. 16 MB is
more than the 14.84 MB shortfall.
3. Shrinking that reservation to 1 MB removes the failure completely -- zero
AllocRange failures, zero guest throws -- AND THE GAME STILL FREEZES.
Verified with a single emulator after killing the stale one: three frames at
rmse 0.00, an 8 MB guest slab unchanged over 3 s, 389% CPU on 3 running
threads. The ledger shows it dying EARLIER, stopping at the 32 MB doubling
step where stock reached 64 MB, so shrinking a live GPU region hangs the GPU
instead. Reverted.
So "the freeze is the refused 128 MB allocation" was too strong. The refusal is
real and Xenia's 16 MB reservation is the swing factor that decides it, but the
guest hangs without it too, at an earlier point. The refusal is one way this
stage load dies, not the cause.
Also recorded: --eh_dispatch, --mem_watch and --audio are all named in this
corpus's own scripts and docs and NONE exists in this build. Each is silently
rejected, which blocks boot rather than warning.
Built the allocation ledger to test the leak I proposed this morning, and it
refutes it.
* No leak. The books balance: 379.5 MB live in 84 blocks plus 113.2 MB free
is 492.7 MB of the console's 512, with ~19 MB in allocation paths the filter
did not capture. Nothing is missing that a leak would explain.
* The 22 "leaked" releases are correct refusals. All 22 failing frees are
interior pointers into ONE allocation -- the 58.5 MB block at BC220000, at
offsets from 5 to 30 MB. The game sub-allocates out of a physical pool and
frees the sub-blocks. BaseHeap::Release frees whole regions, so honouring
an interior pointer would free 58.5 MB, twenty-two times over. The fix I
was moving toward would have been a catastrophe.
* Rounding is not it either: 480.67 MB requested against 484.50 MB granted is
3.83 MB (0.8%), versus a 14.84 MB shortfall.
What actually happens is a doubling grow that holds both buffers: allocate 32 MB,
free the previous, allocate 64 MB, free the 32, then ask for 128 MB while still
holding the 64. That needs 192 MB live for one buffer on top of ~315 MB held
elsewhere, and comes up 14.84 MB short.
No configuration fixes it. There is no guest memory-size cvar; the 512 MB is
baked into the address map rather than a constant (the aliases at 0xA0000000 and
0xC0000000 are spaced exactly 0x20000000 apart, so growing the heap runs one
window into the next); and this tree has no eh_dispatch cvar, so
RtlRaiseException routes the guest's OOM throw to HandleCppException, which logs
and returns without unwinding. That fall-through is the freeze.
Also worth recording: log_mask DISABLES categories (Kernel=1, Apu=2, Cpu=4,
Gpu=8), so the --log_mask=13 used throughout this corpus has been running with
the kernel log switched off.
The oracle is NOT fixed. Ledger committed as data/heap-ledger-stage01.txt.
The previous write-up argued the leak from adjacency -- 22 failed releases next
to the failed allocation. memory.cc closes the loop outright:
* the "parent free N/M pages" in the error is
parent_heap_->unreserved_page_count() (memory.cc:1807);
* unreserved_page_count_ is incremented in exactly ONE place, memory.cc:1445,
inside BaseHeap::Release's page-table loop;
* the failing path returns at memory.cc:1399, at the top of that same
function, before the loop -- page table untouched, no free block inserted;
* and PhysicalHeap::Release delegates to parent_heap_->Release, so the release
that fails and the allocation that later comes up short are the same heap.
So every "address is not a region start" returns zero pages to the counter the
allocator consults, and those pages stay reserved for the life of the process.
That is control flow, not correlation.
Magnitude is still open and I am not claiming it: 512 - 113 = 399 MB missing
against only 22-23 failed releases would need ~18 MB average each, which is
implausible as the whole story. Leaked releases are a contributor, maybe not
the dominant one. An allocation ledger -- log every MmAllocatePhysicalMemoryEx
and MmFreePhysicalMemory with sizes and balance them -- would settle it, and is
a better use of a run than reproducing the freeze again.
Also recorded so nobody hunts for it: there is NO cvar for guest memory size.
memory.cc has only protect_zero / protect_on_release / scribble_heap and the
MMIO ones, and xboxkrnl_memory.cc says "We don't support separate devkit
memory, so just ignore this flag". 512 MB is hardcoded to the retail console,
so the freeze cannot be dodged by giving the emulator more -- a fix has to be
the release path itself.
"Allocation failed" invites blaming the box, so I measured the box. Nothing on
the host side is binding: /dev/shm is 2.0 G at 17% used with 1.7 G free (the
guest memory file is 4.5 G apparent but sparse, only 319 MB of real blocks),
host RAM has 12.3 G available of 15.9, and the container cgroup is at 2.9 G of a
7.0 G limit.
The log's own numbers say where it really is:
131072 pages x 4 KB = 512 MB <- the Xbox 360's unified memory, exactly
28969 pages x 4 KB = 113 MB free
0x08000000 = 128 MB requested
So "parent heap" is the emulated console's physical memory. The game runs a
real 512 MB console down to 113 MB free and then asks for 128 MB; the container
is not involved.
Mechanism, recorded as amber rather than settled: 23 and 22 occurrences of
"BaseHeap::Release failed because address is not a region start" in the two
runs, adjacent to the failure (lines 1115-1176 against a failure at 1179). A
release that cannot find its region start returns without freeing, so each leaks
guest physical pages. That is inference from adjacency and count -- I have not
sampled free pages over time, which is the test that would settle it.
And the release failing at all is an emulator-side bug, not a game one: the
guest is freeing at an address Xenia's heap does not recognise as a region base.
challenge-mission-gate.md §5.6 attributes the 128 MB heap failure to a careless
cleared-stage mask poke, concludes that poking only real story ids does not blow
the heap, and ends by asking for the control: repeat without the poke.
Ran it. nav_to_flight.sh gains SYLPH_NO_POKE=1, which skips the write; only
Stage 1 is selectable without it, so the control changes stage too, which makes
the agreement stronger rather than weaker.
poked 0x0001FFFE Stage 02 frozen, 128 MB request refused
control untouched Stage 01 frozen, 128 MB request refused
Both logs carry not merely the same error but the same numbers:
requested 134217728 bytes, parent free 28969/131072 pages
28969 in both, across two stages and two boots. So the poke does not cause it
and neither does the stage; the guest reproducibly arrives at a 128 MB request
with ~113 MB free. An identical free-page count across independent runs also
says the allocation pattern is deterministic -- not a race, not host pressure.
The control was verified three ways, because the first attempt was confounded:
two emulators were alive at once (the previous one survived a pkill). The mask
was read back as 0x0 from the live mapping, the log was confirmed to be this
run's, and the liveness test was repeated after killing the stale process so
exactly one emulator was running -- three frames at rmse 0.00, and an 8 MB slab
of guest RAM with 0 bytes changed over 3 s.
Reproduced on a Stage 02 run and root-caused. Two witnesses, both taken while
screen_id.py was calling the screen "flight": three screenshots over 8 s at rmse
0.00 with 0.00% of pixels changed, and a 4 MB slab of guest RAM with 0 bytes
changed over 2 s. The emulator is not deadlocked -- 399% CPU over 15 running
threads. It spins while the guest does nothing.
The log stops mid-stage-load on:
PhysicalHeap::AllocRange unable to alloc physical memory in parent heap
(requested 134217728 bytes, parent free 28969/131072 pages)
MmAllocatePhysicalMemoryEx: Allocation failed Size: 08000000
Guest attempted to throw a C++ exception!
128 MB requested against ~113 MB free. So it is not only fragmentation, which
is what "failed to find contiguous range" suggests on its own -- there was less
free memory in that heap than the request needed at all. Preceded by repeated
"BaseHeap::Release failed because address is not a region start", which
challenge-mission-gate.md already notes leaks the range; a leak that repeats
through a session supplies the mechanism the freeze's variable onset (27, 45,
83, 183 s) needs.
This refutes a standing claim. challenge-mission-gate.md §5.6 concludes that
poking only real story ids (0x0001FFFE) does not blow the heap. This run poked
exactly that and hit the same 128 MB failure. Bounded, though: there the
failure was on entering MISSION SELECT, here MISSION SELECT worked and the
failure came at the take-off load -- so the reading is that the poke value is
not what decides it. That page's own open question, repeat without the poke,
is now the load-bearing experiment and is still unrun.
Also recorded: a frozen game passes the screen classifier. A single-frame
statistic cannot distinguish flight from frozen-in-flight, which is why the
entity probes returned 0 definitions, 0 movers and 0 vtable hits with no sign
anything was wrong. A second frame costs nothing and is decisive.
Spent a session getting to flight. Each obstacle presents as "the emulator
died" and none of them is.
* --audio prevents boot. run-canary's header already says the flag is not a
cvar in this tree and that an unknown argument blocks in a message box
before logging starts. Measured anyway, because the corpus also holds runs
that passed it and booted: 3 trials each in BOTH orders, 67 565 bytes of log
without the flag and 209 with -- and 209 is run-canary's own banner, not one
line from xenia. Order was reversed on purpose; this corpus has a standing
lesson that an A/B from run order is noise. Eight scripts on branch
auto/idxd-unnamed-keys still pass it; main and this branch are clean, which
reconciles August's successes with today's failures.
* launch_mission.sh's skip_intro deadlocks. It calls the attract loop a
"movie" and refuses to tap, and waited out 600 s of unbroken movie verdicts
before timing out. nav_to_flight.sh, against the same running emulator,
reached the main menu in 12 s and flight in 2 min 20 s by tapping A at the
title. The "wait it out" premise is wrong: the loop does not end.
* "EMULATOR GONE at ~40 s" is this project's own Stop hook killing xenia when
a Claude turn ends. That is recorded further down this same file and I
rediscovered it over three boots because I did not look. Sequential tool
calls within one turn are fine; ending the turn is what kills it.
The world unit is still unmeasured. Flight was reached and the screen
classifier agrees, but entities2.py finds 0 unit definitions -- its committed VA
window does not match this run, the same run-dependent-address problem this file
documents for the OB counter. Next attempt must hunt the range.
An audit of BACKLOG.md turned up a class of error with a single root cause: the
README defines only the CONFIRMED/PROBABLE/HYPOTHESIS confidence scale, while
the pages actually use a second vocabulary -- and 🔴 appears 98 times without
ever being defined. It gets used for two different things, "refuted" and
"blocked", and three entries slid from one into the other.
README now defines ✅/🟡/❔/🔴/❌/🚧 and states the rule the corpus was missing:
🔴 never means "we have not run it yet". That is ❔ or 🚧. Its blocked sense is
only for a real limit of the box -- no push credentials, no hardware Vulkan, a
decision only the user can make -- and since the box can run the emulator,
script input, screenshot and read guest memory, "needs a run" is never blocked.
I made exactly this mistake on the world-unit item earlier today, which is what
prompted looking for others.
Fixed in BACKLOG.md:
* the elimination test, marked 🔴 UNRUN and in fact run and refuted nine
lines further down;
* the frozen capture, marked 🔴 STILL UNRUN and in fact taken eleven lines
down -- 🔴 wrong twice, since "the freeze did not happen this run" is a
scheduling outcome and not a refutation;
* a 🚧 STILL UNRUN item whose stated blocker (the boot-nav bug) is fixed;
* the objective-counter heading, which asserts 0xbdb59668 as the answer while
its own first body line refutes that address -- retitled to say what is
actually solved, the method;
* the paint-order "third measured permutation" question, answered inside its
own entry by a third, fourth and fifth screen;
* the UTF-16 endianness question -- resolved, and it is not a stale comment:
localization.rs both documents LE and decodes with u16::from_le_bytes, so
it is a code bug worth filing.
Also fixes the corpus's only dangling link (INDEX.md pointed at
structures/idxd-unnamed-keys.md, never written).
I wrote "blocked on the oracle" for the unit-to-metre conversion. That was a
mislabel: red is for what the container cannot do, and run-canary works here.
What the km-name sweep actually established is narrower -- no STATIC test can
settle it, because the disc has exactly one size-bearing asset name.
The run is well-supported by tooling that already exists: findplayer.py
recovers the player position triple from motion, the HUD prints the distance to
the selected target in the game own units, and the same separation read both
ways is the conversion. Recorded as amber with the experiment written out.
The census filtered pak entries whose own first four bytes are T8aD. A sprite
is usually a child of a RATC bundle, and a bundle entry's magic is RATC, so a
top-level magic filter cannot see one:
top-level T8aD entries (counted) 4 525 sprites, 45 keys
T8aD inside RATC bundles (missed) 16 659 sprites, 204 keys
both 21 184 sprites, 216 keys
171 of the 216 keys exist only inside bundles. The sharpest statement of the
error: that census never saw GP_TITLE.pak at all -- the pak holding both of the
screens this page's entire evidence comes from.
Retracted: "45 values", "the keys are pak-local", "each auxiliary pak occupies
its own narrow high-byte band". On the full population 68/216 keys (31%, not
9%) cross a pak family and the per-pak ranges overlap heavily -- GP_BUNK
0x8000-0xa110, GP_TITLE 0x8000-0xc150, GP_LEADERBOARD 0x8000-0xf100. The tidy
banding was an artifact of seeing one or two keys per pak. So the key looks
like a shared vocabulary, which is the opposite of what I published.
Survives, now on the full population: the field is a u16 at +0x0A (upper half
zero 21 184/21 184), and it is an enumeration (216 values for 21 184 sprites).
Three wrong numbers on this page now, all the same shape -- a statistic computed
over a population I had not checked was the population in question. Stated once
at the end of the section rather than three times: check the sampling frame
before the statistic.
The census filters pak entries whose own first four bytes are T8aD. The
sprites this page measures paint order on are children of a RATC bundle --
ui_layout.rs reaches them via ratc::parse, and a bundle entry's magic is RATC,
so a child T8aD never matches a top-level magic filter.
So the 45 keys may describe a population that only partly overlaps, or does not
overlap at all with, the one the page's two measured screens come from. I do
not yet know which; the comparison is running. Marking the section rather than
leaving the counts to be read as covering the screens' sprites.
Same failure shape as the 37/45 language-duplication note lower down the page:
a number computed over a population I had not checked was the population in
question. Recording it as such.
The page rested on twelve values from two screens. This walks all 4525 sprites
on the disc.
* The field is a u16 at +0x0A. The upper half of the 32-bit word the page
reads is zero in 4525/4525. Nothing above changes -- 0x00008100 sorts the
same as 0x8100 -- but a future value with the high half set would mean
something had been misread rather than that the layer got deeper.
* It is an enumeration: 45 values for 4525 sprites, one of which (0x8100)
covers 1188 of them.
* The reading worth trying -- a global layer vocabulary shared across the UI
-- is refuted. Only 4 of 45 keys cross a pak family and 33 of 45 live only
in GP_MAIN_GAME_2D; every other pak owns a narrow high-byte band (0x90-0x94
for the in-game overlays, 0xa4 mission log, 0xb1-0xb2 save/load). A screen
that owns one or two keys is not ordering itself with them.
That supports "group id in the high bits, order in the low bits", which is what
the page already suspected, but it does NOT test it: paint order has been
measured on two screens and both are inside GP_MAIN_GAME_2D, so there is no
ground truth to check the split against. Left amber.
The first number I got was 37/45 shared, which would have supported precisely
the wrong conclusion. It came from counting paks instead of pak families: the
six GP_MAIN_GAME_*2D paks are the same screens in six languages and their key
sets are byte-for-byte identical. Recorded on the page, because the shape
recurs -- a corpus with near-duplicate members manufactures agreement.
Two follow-ups on yesterday's^Wthis morning's CollisionSet write-up.
1. The _cmesh <-> render-model link, which I recorded as UNTESTED because
matching stems against .xbg object names covered 4 of 158. The disc keeps
only one build manifest, so that corpus was never going to answer it. The
right corpus is the GameResourceID field of the DefTables / GP_MAIN_GAME
records -- 480 distinct values. Against those, with a control that shuffles
the characters of each stem:
ship/mob stems prefixed by a real resource id 108/112 = 96.4%
same stems, characters shuffled (control) 0/112 = 0.0%
asteroid stems prefixed (expected none) 0/46
So a CollisionSet entry is <GameResourceID>[_<part>]_cmesh. The 0/46 on
asteroids matters as much as the 108/112: a test that fired on everything
would be the bound-check hazard again.
2. The world unit. Sweeping every pak for a name carrying a kilometre figure
returns mapmesh_box_500km.col/.rgn and nothing else -- 162 references, all to
that one pair. The reading rests on a single filename with no corroborating
instance anywhere in the data, so no static test can settle it; marking it
blocked on the oracle rather than leaving it as an open static question.
My objection's premise did survive: rou_e010 is a real GameResourceID and
e010_ADAN_Attacker_S is in the stage tables, so the 133-unit mesh does belong
to a craft the game calls an attacker. Whether the trailing _S means "small"
is a further guess (there are _EX4 / _HF / _HF_Wayne variants), so it stays
suggestive rather than evidence.
All 18 blobs are byte-identical: the per-stage naming is nominal, and every
stage points at one shared 1675148-byte library stored eighteen times. That
identical size was the reason to open the item, and it turned out to be the
answer to it.
Record layout: {u32 size, u32 name_len, char name[name_len], u32 nv, u32 nt,
f32[3] x nv, u32[3] x nt}, next record at off + 8 + size. The indices are u32
here where MCOL uses u16 -- two different serialisers in one archive.
What makes this a decode rather than a plausible reading: the walk consumes the
file to the byte over 158 variable-length records, with the size word predicted
from the two counts 158/158. A wrong field would desynchronise within a few
records and could not land exactly on the end. All indices in range 158/158;
98.24% of edges shared by exactly two triangles; 147/158 fully manifold.
158 meshes, 90 836 triangles: per-part ship proxies (_bdy/_brg/_eng/_wep/_sld,
the XBG7 sub-part vocabulary) plus 46 stage asteroid meshes whose prefixes are
exactly the stages that have an _AsteroidVolume_wp MCOL.
Two things this file makes me walk back:
* The "1 unit = 1 metre" reading from mapmesh_box_500km is downgraded to
amber. The 500000 arithmetic stands, but it implies that a craft the game's
own tables call "small" is 133 m and that rob_f002 is 447 km -- 89% of the
arena width. The format check survives; the interpretation has no
independent support.
* The _cmesh <-> render-model name link is recorded as UNTESTED, not
confirmed: only one .xbg build manifest survives on the disc, so matching
stems against object names covers 4 of 158, which is no coverage at all.
The names live outside MiscBin: they are the MapPath / MapMesh /
CollisionMeshes field values of the per-stage StageResource object (IDXD schema
3c9ae32e, in every GP_MAIN_GAME_<lang>.pak), and each hashes with the ordinary
pak name_hash straight to a TOC entry. 40/40 resolve, no collisions -- the 11
REGN as <stem>.rgn, the 11 MCOL as <stem>.col, and the 18 remaining blobs as
CollisionSet_S01..S16 / _Tutorial / _test.bin. The .pe string table at 651540
was the way in: MapMesh and MapPath sit adjacent there.
This upgrades the pairing claim. The first section of mcol-collision.md could
only say REGN and MCOL had matching *distributions* of bbox and cell size, and
flagged that as not an object-to-object link. A phase record names one .rgn and
one .col, and all 11/11 pairs share a stem and agree exactly on both.
The names also check the format work from outside it: mapmesh_box_500km.col is
the object decoded here as 8 vertices and 12 triangles spanning exactly
+-250000, and its name says that cube is 500 km across -- so one world unit is
one metre, and a wrong stride could not have produced a box that measures what
its own filename claims. 70 of the 87 phases use it: most stages' only
collision is the arena wall, and _AsteroidVolume_ names the rest.
Still open: the 18 CollisionSet_*.bin are named but not decoded (all exactly
1675148 bytes), and CMapColliderBridge in the RTTI names the runtime consumer
without following it into the code.
The 0x50 header word, which the first section of this page had dismissed as "a
large value", is two u16 counts: vertices and triangles. They give the two
remaining blocks their stride, and every derived length is exact in 11/11 --
len(0x54) == align16(12*nv), len(0x58) == align16(6*nt), and nt equals the
bounding-sphere count decoded last iteration.
Checks that cannot pass by accident:
* sphere i is the TIGHT bounding sphere of triangle i, 4768/4768, with
max|v-c|/r median 0.99990 (a fixed 1.0001 epsilon), against a 1.32%
random-triangle control;
* the mesh is watertight -- every edge shared by exactly two triangles,
7152/7152, zero degenerate triangles, zero unreferenced vertices;
* the two smallest objects are 8 vertices and 12 triangles whose positions
are the eight +-250000 corners of the map bbox: a bare bounding cube.
The cell lists are a correct broad phase: with an exact triangle/box SAT test
only 3 overlapping triangles in 18 577 entries are absent, so a query walking
one cell's list cannot miss a hit. The 730 conservative extras bracket the
builder's own test between exact-SAT and AABB, which retires the 18 unexplained
"sphere misses" from the previous commit as that same margin.
mcol_probe.py gains `mesh` and `obj`; `verify` now runs all three checks and its
output is recorded in docs/re/data/mcol-verify.txt.
The unexplained ~0.75 ratio left at the end of the last iteration was my own
stride. I had read the block as 12-byte points because REGN's vertex section
is 12 bytes, and never checked it: len(0x5C) is not a multiple of 12 in 5 of
the 11 objects, so that stride was never arithmetically possible.
At stride 16 the relation is exact in 11/11 -- max u16 == len(0x5C)/16 - 1 --
and the record reads as {centre f32[3], radius f32}. Powered test, since a
u16 is reached through a specific grid cell: the sphere it names reaches that
cell in 18 559/18 577 = 99.90%, against a 12.02% random-sphere control. Both
fields carry signal (centre alone 26.75%, radius shuffled 70.19%).
The converse -- is the list *exactly* the intersecting set? -- is 0.38%, which
is the expected direction: a bounding sphere is conservative, so membership
implies overlap but not the reverse. The tighter geometry is in 0x54/0x58,
still undecoded. 18 entries (0.10%) go the wrong way and are recorded as open.
tools/re-capture/regn_decode.py is copied unchanged from auto/regn-reader so
the probe's POF0 reader is the known-good one rather than a second copy.
Two tests. Counts modulo 3 are spread across all three residues (639/2063/1786),
so the u16 array is not a triangle list. And a B record is reached through a
specific cell, so a point it references should lie in that cell -- referenced
points score 0.79% against a 0.48% random-point control. Chance.
The contrast is the point. One section earlier the same u16 entries scored
18379/18379 (100%) on 'are these valid point indices'. I flagged that at the
time as the weak bound-check and recorded it as consistent rather than as a
finding. The caution was right: the powered version of the same question now
returns chance, and had the 100% been written up as the decode this page would
carry a confident false statement about MCOL's geometry.
Fifth appearance of the pattern across REGN and MCOL and the first time both
halves have been run on the same field, so the page now states it plainly: a
bound-check asks whether something could be an index, and the answer is set by
the size of the target collection rather than by the field's meaning.
Datum for the next attempt: the maximum u16 is consistently about 0.75x the
point count (923/1232, 1019/1360, 1163/1552, 59/80), too consistent to be
coincidence and not explained.
Two corrections got there. MCOL has 9020 relocated words and only 4488 are
A-record pointers; I assumed the rest sat at B+8 mirroring A, refuted 0/4488.
Measuring their offset from the nearest preceding B record gives B+4 for 4488
(99.0%) and 44 before the first B -- exactly the four header pointers times 11
objects. Nothing unaccounted for.
So B is {u32 count, pointer at +4}. The pointers advance by exactly twice the
count: 4477/4477 (100.00%) over all 11 objects. That is a packed u16 array with
no padding, and it is the load-bearing evidence -- an exact arithmetic identity
over 4477 consecutive pairs. The companion check that those u16s are valid point
indices is the same weak bound-check flagged earlier and is recorded as
consistent rather than as evidence.
Chain: position -> cell -> A {cell index, count 1, ->B, sphere} -> B {count n,
->u16[n]} -> n indices into the point block. Same shape as REGN's cell -> item
-> refs -> geometry.
Open: the 0x54 and 0x58 blocks, which this chain never reaches, and what the
indexed points form.
Filtering array A by the cell-index criteria and following each pointer: every A
record points at a distinct B record, 11/11, and every A count field is exactly
1, 11/11. Per object the A-record count is the number of occupied cells (110 to
575) and the counts sum to it exactly.
That is the same design REGN uses -- the corpus already records 'every occupied
cell has count exactly 1' there. Two sibling formats, one convention, and a
further independent confirmation of the A reading since the filter and the
cardinality are unrelated criteria.
Array B resisted, and both attempts failed in ways worth recording. The record
boundary was off by 8 again, producing records that start with the tail of the
previous structure -- the same mistake as the 0x74 check two iterations ago. And
the u16-index test had no power: the 0x5C block holds ~1232 points, so 'is this
u16 below the point count' passes for almost any small value, and duly reported
100% at seven offsets. Fourth time in these two formats that a bound-check
against a large collection produced a meaningless 100%.
Recorded what would have power instead: B records are 1:1 with occupied cells
and A carries that cell's bounding sphere, so a B field can be tested for
spatial consistency with that specific cell.
Separating the two interleaved arrays by address and re-running the same three
criteria: array A (2509 records) gives byte3==1 at 100.00%, a valid cell index
at 100.00%, and the sphere reaching that cell at 99.92%. Array B (6467) gives
30.65% and 30.60% -- a different record type, and the control showing A's 100%
is not what any 32-byte block would score.
So array A is the per-cell record: cell index (x,y,z), count, pointer into array
B, bounding sphere -- the same role REGN's section 3 plays. That confirms the
earlier 50% was the interleaving artifact and not a half-working reading.
The split was crude, first-half-by-address giving 2509 vs 6467 rather than an
even cut, and A still came out at 100%. A rough partition isolating a perfect
population is stronger than a careful one isolating a good-ish population.
Array B's layout is still unread.
Read as 8 big-endian words, word 0 as four bytes is (x,y,z,1) -- a 3-D cell
index, matching the object's 5x5x5 grid. Words 4-6 are a position and word 7 a
positive scalar (a bounding sphere); word 1 is a count, word 2 the relocated
pointer. Every record pointer lands in the same region, 8976/8976, each a fixed
distance on with the same stride -- so there are two parallel arrays, A and B.
The 50% is the tell. Three independent criteria -- byte 3 == 1, valid cell
index, sphere reaches that cell -- all land on 50.0%, which says half the
records are not this type rather than that the reading half-works. The POF0 slot
list interleaves both arrays and I was testing B's records against A's layout.
So array A is a per-cell record, the same role REGN's section 3 plays. Array B
is unread, and the criteria have not yet been re-run on A alone -- if the
reading is right they should go to 100%.
Used the known-good decoder rather than my own broken one. My version had three
errors: the delta stream starts at table+16 not +8, the tag bits are
0x40/0x80/0xC0 rather than 0/1/2, and slots carry the +16 fixup base. Sanity
check passes -- on REGN the tool returns header slots 0x70-0x84 exactly.
On MCOL, over all 11: the header-region relocated slots are exactly 0x54, 0x58,
0x5C and 0x74 (11/11); 0x5C resolves to 0x80, the first byte after the header
(11/11); and 0x74 resolves to eight bytes before the first array pointer (10/11).
So MCOL has four top-level pointers where REGN has six.
92.7% of gaps between consecutive relocated words are 32 bytes, in 7-127
contiguous runs per object, which with the 0x74 offset reads as arrays of
32-byte records each carrying a pointer at +8.
The four targets are two float blocks, a block of small ints / u16 pairs, and
counts followed by the record array -- the shape of a mesh, stated as a reading
of the shape since none of the blocks is decoded.
Open: the record layout, what the index block indexes, the one object where 0x74
does not land 8 before the array, and whether the 7.3% non-32 gaps are just run
boundaries.
MCOL sits beside REGN in hidden/MiscBin.pak, 11 of each, never decoded. Over all
11: POF0 at data_size+16 11/11, bbox pad words 1.0/1.0/0.0 11/11, and
extent == max-min 11/11. So the header prefix is the same shape as REGN's and
the POF0 mechanism applies, which means the chunk+0x10 base and the loader's own
pointer list -- the two things that cracked REGN -- are available here too.
The map parameters are not merely similar but identical in distribution: bboxes
2/6/3 at 250k/50k/25k and the 0x40 triple 2 at 50000 and 9 at 10000, matching
REGN exactly. Eleven maps, each with an MCOL and a REGN over the same volume at
the same cell size. Noted that this matches distributions, not a demonstrated
object-to-object pairing.
Everything past 0x40 diverges from REGN and is open. Also recorded that my own
POF0 delta decoder is wrong here -- eight leading zero deltas -- and that the
working one is regn_decode.py on auto/regn-reader, which should be used rather
than re-derived. Re-deriving it is the mistake I made.
The original reason for investigating REGN was that a mission's enemy count
rises and falls, so a scheduler with parameters must exist somewhere, and a
per-map uniform grid is what such a thing would be indexed by.
Now that it is decoded that reasoning is answered: REGN is a tetrahedral
navigation mesh -- vertices, faces carrying plane equations and adjacency,
tetrahedra with portal costs between face pairs, and a grid indexing which tets
fall in each cell. Every section is accounted for by that structure, and there is
no time field, no unit reference and no trigger anywhere in it.
So the wave-scheduler search should treat REGN as excluded rather than unread.
The page's original hedge was right to keep the reading provisional, but the
reasoning it hedged was a guess from shape, and the shape belonged to
pathfinding -- which is what pointed the whole investigation here.
The arrival timetable in Route_S<NN>.tbl, keyframed per squadron per phase with
t in seconds, remains the only located part of the mechanism.
Re-derived the other branch's central check with my own code, my own reading of
the record and my own control, rather than accepting the number.
At base chunk+0x10, with the plane at intra-record +0 and three u16 vertex
indices at +32: every one of 133573 faces has all three named vertices
satisfying its own plane equation. Random-vertex control 2782/400719 = 0.69%.
100% against 0.69% is not a fit.
That also settles the record boundary from my side: the u16s describing a plane
sit after it in the same 48-byte record, so my earlier 'four zeros at the start
of each record' was those integers seen 16 bytes out of position, one record
late.
Two implementations, two independent guesses at the intra-record layout, the
same 100%.
I recorded the static coupling search as exhausted and needing PE code. The PE
work was done on auto/regn-reader and it solved the whole thing: REGN is a
tetrahedral navigation mesh, reached via the POF0 fixup table -- the loader's own
list of which words are pointers, so nothing needed guessing. Six sections;
position -> cell -> 32-byte item -> tet refs -> tetrahedron, with section 2 a
face carrying a plane plus its 3 vertices and the two tets either side.
Controlled checks: face through 3 of 4 tet vertices 253722/253722 against a
0.07-2.2% random control; portal cost equals face-centroid distance
380460/380460.
Also records against myself that the base is chunk+0x10 and my offsets here were
16 bytes early, that the plane arithmetic survives only because those fields
landed on the same bytes, and that my points-in-bbox count was never evidence --
a shift inside a homogeneous f32 array yields other floats from the same array.
Leaving the wrong conclusion in the backlog would have told the next reader the
avenue was closed when it was the one that worked.
The other branch supplied concrete offsets: for 3506e972 its face record 0 and
its plane normal both begin at 0x1c700, and my chunk + offset_at_0x78 + 16 gives
0x1c700. Same bytes, different bookkeeping -- so the n.p+d result stands
unchanged and was never in dispute.
The base is chunk+0x10, on evidence with power: the loader does addi r3,r31,16;
at +0x10 the six POF0-relocated slots land exactly on 0x70-0x84, the six section
pointers, whereas at +0 they would relocate the u16 counts and leave two section
pointers unrelocated, which is non-functional; and section-0 record 0 reads as a
bbox corner at +0x10 and garbage at +0.
So my '13467/13467 points inside the bbox' was vacuous. Only 11 of 13467 read as
denormal at the wrong base -- the rest were still plausible coordinates, because
a 16-byte shift inside a packed array of f32 triples yields other floats from
the same array. Recorded the general form: a containment test cannot detect a
shift inside a homogeneous array, because the shifted values come from the same
distribution. For that class of error it is not a weak check, it is no check.
'Section 0 is a point list' happens to be right; the evidence I gave for it was
not evidence.
