Bounding the pointer scan to 0xBC000000-0xBD000000 (with a full-sweep fallback)
drops find_mission from a ~371MB walk to 0.7s. The run then went 694s with the
probe attached and NO freeze, against 3-of-3 frozen inside ~4 minutes with the
unbounded version. n=1, but the first probe-attached run to survive.
State encoding pinned to three points: 1 = not yet deployed, 2 = active,
4 = destroyed. ADN111 caught going 2 -> 4 at 433s while the active count fell
36 -> 27.
The phase ended at 694.9s WITHOUT the ordinal advancing, and every field matches
the branch read statically from sub_82260710: [phase+300]=2 (last-phase flag),
[mission+20]=0 (mission-over state), [phase+196]=1 (finished), [mission+40]=1
(unchanged). The static state machine is confirmed on the live oracle for the
mission-over half.
But this was a LOSS, not a clear: GAME OVER on screen, escort at 35.7%, pilot
DEAD at 676s, and two of the three objective squadrons still at state 2. So the
'destroy all three clears phase 1' prediction remains untested. What is
established is that the else-branch is the only route to phase 2 and needs
[phase+300] != 2 when the phase ends.
Five attempts, still no phase advance observed -- the obstacle is now keeping the
escort alive, not the freeze or the instrument.
A pilot-only run froze at t~150.7s (frozen.py: max_pixel_delta=0), found by
accident when the sweep-free experiment aborted at startup and the run flew with
no script probe attached. The tally is now 3-of-3 frozen with the probe versus
1-of-3 without, not 3-versus-0. Still a lean, but not the clean separation the
previous entry claimed, and marked down accordingly -- the fourth time a freeze
conclusion here has had to be softened by one more run.
The sweep-free test itself could not run: ScriptMission is re-allocated per run,
so the address from an earlier run (0xBC7A2A20) read back all zeros. The cheap
self-consistency check ([m+44] must equal the phase's [+244]) rejected it
instead of reporting garbage, which is the part that worked.
Names the cheaper replacement: a BOUNDED pointer scan. Every ScriptMission seen
so far sits in 0xBC79xxxx-0xBC7Axxxx, so ~32MB instead of ~371MB would cut the
sweep cost roughly tenfold. Not yet implemented.
phase_watch.py runs give a sharper tally than the earlier inducer test, because
the only difference between the groups is one tool: pilot alone went clean to
936s and 1064s (2 runs), while pilot + phase_watch froze at ~70s, ~253s and
~126s (3 runs). Same boot path, same pilot, same mission.
What phase_watch adds is find_mission(), which does TWO full sweeps of guest
memory at startup plus ~130 small reads every 5s. The sweeps are the same shape
as the heavy_read.py scans the earlier n=2 experiment implicated.
Still correlation, not cause -- 3 vs 2, and the earlier inducer test produced a
clean counter-example. But it is the best-supported version of 'the instrument
provokes the freeze' so far, and it is my instrument, which makes it actionable.
The cheap test is named: find_mission only needs to run once, so re-run the
watch with the sweeps replaced by an address from a prior run. Freezes stopping
implicates the sweeps; continuing implicates the per-sample reads.
Cost so far: three attempts at observing a phase advance, all truncated inside
four minutes.
ADN110/111/112 all carry first-keyframe time 170 on their _p1F routes, and in a
live run all three flipped from not-deployed to active at ~143s on the probe
clock. The probe zero sits roughly 10-20s into the mission, putting the arrival
at ~155-165s of mission time against a predicted 170.
That pins the unit as seconds: 170 frames at 30fps is 5.7s, so they would have
been active at the first sample and demonstrably were not. Three squadrons
sharing one timetable entry changing state in the same 5s window is not
coincidence. Flagged as a match rather than an exact measurement -- the probe
clock is not aligned to phase start.
Supersedes mission-arrival-watch.md's headline negative: 'no arrival has ever
been observed across six runs' was an instrument limit, not a fact about the
game. Its craft-counting analysis stays accurate -- craft counts conflate
deployment with attrition and cannot see an arrival at all.
Watching [ScriptMission+40] and the three phase-1 objective squadrons together:
all of ADN110/111/112 flip state 1 -> 2 at ~143s, while records in state 2 climb
24 -> 35 over four minutes.
So state 1 means 'not yet deployed' for these, not 'gone'. The built-in table's
'1/3/4 = gone/dead/invalid' shorthand is incomplete, and reading state != 2 as
destroyed would have been wrong exactly as flagged last iteration.
This also answers a much older question: mission-arrival-watch.md and the wave
work recorded '0 confirmed arrivals' across many runs by watching the CRAFT
population. The script's own unit table shows arrivals plainly -- eleven records
enter state 2 within four minutes. The old negative measured the wrong
structure; craft counts conflate deployment with attrition, the per-unit state
field does not.
Both attempts froze (at ~70s and ~253s), so no phase advance was reached. The
freeze witness caught both immediately, which is why the truncation is visible
instead of a silently flat line.
New harness tools/re-capture/phase_watch.py.
Unblocks the phase experiment, which was stuck because '38 enemies died' could
not say whether the right ones did. Chasing craft->squadron was the wrong angle:
the script VM keeps that table itself, indexed by the .ssb symbol-table-2 index.
Route: find the .ssb header in guest memory (0xAB840010 for a Stage 02 run),
code base = filebase + 0x24, scan for a word equal to it, then VALIDATE
arithmetically -- [ScriptMission+44] must equal filebase + symtab1 offset + 4.
Measured 0xAB874C94, predicted 0xAB874C94, exact. A second candidate that also
pointed at the code base failed that check and was discarded; without it either
would have looked plausible.
ScriptPhase+324 -> +4 is an array of 122 per-unit records -- exactly the size of
Stage 02's symbol table 2, an independent confirmation of the index space.
[ScriptMission+40] reads 1 in a phase-1 mission. The mirror at
[*(0x828F35F8)+236] that three earlier runs polled reads 0, because ChangePhase
only posts once the ordinal exceeds 1 -- so +40 is the real counter and is
reachable from /dev/shm with no debugger.
Flagged rather than asserted: the three objective squadrons read state=1 with a
LIVE object pointer in a mission where nothing has been shot, which does not fit
the built-in table's '1 = gone'. Reading state != 2 as destroyed would be a
plausible-but-wrong inference; the encoding needs pinning first.
New tool tools/re-capture/squadron_state.py, verified end to end against the
manual reading.
Checked liveness before investing this time (animating, max_delta=254) and the
run was real: 823 distinct speeds, hull 1500->1485, escort 100%->30.7%, ADAN
population 127->89 (38 destroyed) over ~1064s, no GUEST FROZEN banner. The phase
mirror stayed 0 throughout.
But this still does not test the prediction. The claim is that phase 1 clears
when ADN110/111/112 -- 27 specific Turrets -- die; what was measured is that 38
enemies of some kind died. SYLPH_PREFER biases target choice but the pilot went
to DEFEND and spent the run on e010_ADAN_Attacker_S protecting the escort.
The missing instrument is per-squadron liveness: nothing maps a live craft back
to its roster squadron, the same gap roster-to-craft-link.md records as unsolved.
So 'no advance after 38 kills' is compatible with both 'the condition is wrong'
and 'the right 27 were never killed'. The decisive experiment is blocked on
attribution, not on flying time.
Last iteration ended with 'what reads the flag file is unknown' after an offset
search failed. The framing was wrong: I was hunting an engine-side reader, but
the consumer is the script itself, through built-in 9 (read_freg), which loads
[phase+88][i] into the double result register.
Stage02.ssb calls read_freg 12 times -- the same count as set_flag (12) and
clear_flag (12) -- so the latch is symmetric and entirely inside the VM. That
closes the middle of the set_flag -> ... -> END_PHASE chain: one coroutine
latches, another reads it back and branches.
Also records an unconfirmed observation: op10 + op13 pairs with consecutive
small immediates and their own code offsets look like a case/branch dispatch.
Flagged as a pattern, not named -- neither handler has been read.
SYLPH_PREFER=Turret, escort at 100%, 1070s, phase field 0 throughout, 4029
Turret-targeting pilot samples. It looked like a clean negative -- 'hunting the
objective squadrons does not advance the phase' -- and it is worthless:
frozen.py reports max_pixel_delta=0, and the pilot's first sample at t=0.0
already has spd=0 with the same yaw/pitch/target/d=7186 it still had at 1070s.
The mission froze on entry to flight and nothing was ever shot, while screen_id
said 'flight' the whole time.
I was one step from writing this up as a fact about the game; running the freeze
test rather than trusting a plausible log is what caught it.
phase_probe.py now calls frozen.py every 60s and prints a GUEST FROZEN banner
inline, so a dead-world reading is labelled in the data instead of discovered
later. Note frozen.frozen() returns a TUPLE (is_frozen, max_delta) -- testing it
directly is always truthy and would have made the witness fire on every check.
Verified against the frozen guest.
Also recorded: the pilot log is itself a freeze witness -- identical
yaw/pitch/target across thousands of samples is a dead world, not patience.
Resolving the three squadrons the phase-1 script condition polls against the
roster: all three are phase-1 (route-name map), all fly UN_e007_ADAN_Turret, all
n=9, all on Formation_ADAN_Turret09_30 -- so the condition watches 27 Turrets.
Three layers agree independently: route names say phase 1, the roster says what
they fly and how many, and the script polls exactly those three before latching
set_flag(8).
Also confirms on the data that 'Turret' is a craft type, not ship-mounted AA:
UN_e007_ADAN_Turret is flown by AI_ADAN_CraftSquadron_Rookie/_Veteran in a
30-slot craft formation, never AI_Structure.
Polled the runtime phase mirror every 3s for 936s of hunting flight. It never
left 0, which the static work predicts rather than contradicts: ChangePhase is
only posted once the ordinal exceeds 1, so a phase-1 run never writes it. That
does make the mirror useless as a phase-1 readout -- it reads 0, not 1, and
[ScriptMission+40] needs a debugger.
The run ended in GAME OVER with the player's hull at full 1500, so a script
lose-branch fired rather than the player dying. The pilot log narrows it: the
tracked asset fell from 100.8% to 23.4% with ASSET-HIT events, while the pilot
avoided f101_TCAF_Acropolis 2588 times. 'Escort lost' is a measured candidate,
not a guess -- but the asset ended at 23.4%, not 0, so the threshold (or whether
another unit's loss ended it) is not established.
Two runs now (530s, 936s) and no phase advance observed. Since phase 1 polls
unit_state on ADN110/111/112 and the pilot hunts whatever is nearest rather than
the objective, biasing it at those squadrons is the next step.
Refuted: sub_8226D740 does NOT read the flag file. Its three addi rN,r31,120 are
fields of a local trigger-record buffer passed to sub_8226E220 as out-params;
the phase lives in r26/r29. Offsets 88 and 120 are too common image-wide (2091
and 5228 instructions) to isolate the register files by offset search, so what
reads the flags is still unknown and the set_flag -> END_PHASE chain has a gap.
Resolving symbol-table-2 indices turns the bytecode into mission logic. At
0xF524 Stage02.ssb polls unit_state on ADN110, ADN111 and ADN112, updates each
one's objective marker, then latches set_flag(8) -- exactly the
trigger/predicate/set_flag/END_PHASE shape predicted from the disassembly, now
observed in the mission's own code with names the roster tables already gave.
The 12 END_PHASE sites are outro sequences (wait_cmds_drained / fade_sound(3) /
builtin85(3) / wait_s(3) / END_PHASE / yield) -- the terminator, not the
decision.
Fixes a decode bug that hid every argument: the tracker only followed
local[i] = special[0], but the common form is an immediate written straight into
local[i] (k=01,03), so every unit predicate printed with NO arguments. The
disassembly looked complete while being empty exactly where it mattered.
Also records the live probe result: the phase mirror at [*(0x828F35F8)+236]
stayed 0 for ~530s of actively-hunting flight, no advance observed -- which is
what the static analysis predicts for phase 1, since ChangePhase only posts once
the ordinal exceeds 1.
Table at 0x8227226C is 147 big-endian absolute VAs (verified structurally: it
ends exactly where the first handler begins, all targets inside sub_82272220).
Arguments are not in the instruction -- every handler does c_str() on
[phase+20], a packed blob, which is what the local[] staging fills. Return 2 =
yield; five built-ins block by skipping the pc advance.
Recovered the ScriptPhase state layout: 32-entry float and flag register files,
int/double result registers, the timer block, and the runtime unit array at
+324 indexed by symbol-table-2 index -- a direct hook from bytecode call sites
to the two .ssb symbol tables.
Spot-checked two claims against the disassembly rather than trusting them: id 4
loads a DOUBLE into the thread countdown and returns 2 (wait_s), and id 24 reads
current/initial squadron member counts (squad_survival_pct). Both exact.
Counting Stage02.ssb: unit_state 255, hp_pct_test 167, dist_lt 92, unit_alive
71, unit_relation 52 -- and squad_survival_pct, group_ratio_pct and the two
global counters are NOT called at all. So Stage 02's phases are gated on named
units (destroyed / HP / proximity), never on an aggregate count, even though the
kill-counter primitives exist in the VM. That answers the standing 'next wave
after N kills or after an event?' question for this stage: specific units, not a
number.
isl.py now names the built-ins, so the run-up to the first END_PHASE reads
wait_cmds_drained / fade_sound(3) / builtin85(3) / wait_s(3) / END_PHASE.
Not settled: 3 handlers unresolved (55, 75, 105); the 1024-slot interpreter
command table is only partly recovered.
Resolver table 0x82271D74 gives four kinds: 0 global[i], 1 immediate,
2 special[i] ([phase+164]/[phase+168]), 3 local[i] ([phase+20+i]). Byte[0] is
the rvalue kind, byte[1] the lvalue kind, so the recurring instruction pair is
argument staging -- values land in local[] at offsets 0,4,8,0xC and the next
call consumes them. A built-in's arguments are not in its own instruction.
Fixed a decode that would have been believed: immediates in set.f are DOUBLES
carried as two words (op 1 stores with stfd). Reading the high word as a float
gives 2.125 where the script means 3.0.
isl.py now tracks staging and prints call arguments, so the run-up to the first
END PHASE in Stage02 reads as builtin=64(0x42,2,1,9,1,-1) / 120 / 59(3) / 85(3)
/ 4(3) / 6. Three built-ins taking 3 just before the phase ends look like a
wait-seconds family -- flagged as unconfirmed until the built-in table is read.
Read the encoding off the interpreter rather than guessing: instruction is a
big-endian u32 whose LOW byte is the opcode (25 of them, table 0x822635FC),
byte[2] is the instruction length -- every handler advances the pc by it -- and
bytes[0..1] are operand kinds. Op 12 is a jump whose operand is relative to the
code base [phase+232], which settles that offsets are code-base-relative for
this opcode. Op 19 is the built-in call: id in word@+4, and word@+8 is a
monotonically increasing STATEMENT id (0x245, 0x248, 0x24A, ...).
Confirmed by disassembling Stage02.ssb: the stream decodes cleanly from the code
base and routines terminate on ret exactly where expected.
Scanning the code region on the call encoding: 2846 call sites, 73 of the 147
built-ins used. The phase-control ones are located -- built-in 6 (end phase) at
12 sites, 62 at 3, 39 (mark last phase) at 8 -- so a phase has several exit
paths, as a mission with win and lose branches should.
New tool tools/re-capture/isl.py with --calls and --to (resync-into-target,
needed because instructions are variable-length so you cannot walk backwards).
Not settled: the 147 built-ins are uncharacterised, so this is structure without
meaning -- we can see THAT a phase ends, not WHAT was tested.
mission-phase-advance.md recorded the bytecode as not on the disc. It is; the
earlier grep failed because every pak entry is Z1+zlib, so no name is plaintext.
The loader resolves table KEYS, not a filename: GamePart name -> GP_SCRIPT ->
script.tbl (name_hash 0x75FE4656), whose SCRIPTS record is a 40-field manifest
mapping MISSION1..MISSION29 to StageNN.ssb. Verified: Stage02.ssb is 226,596
bytes, md5 aff69b5a..., byte-identical across all six language paks.
28 scripts exist; S17 is the missing one -- which agrees with the table sweep
(S17 in none of the five families) AND with the loader guard
sub_8225EC78 'if (n == 16 || n > 32) return', proving that mission number is
0-based. Three independent routes to the same conclusion.
Header decoded from ScriptMission::Load rather than guessed: version, code
offset 0x24, two symbol tables. Stage02 symtab1 = 326 symbols (197 message ids,
119 route names, 10 subobjectives), symtab2 = 122 (111 unit ids, 11 characters).
Two loops close: the 111 unit ids are exactly the 111 UnitGroup_S02 squadrons,
and the route names appear as SCRIPT SYMBOLS -- which is why nothing in the
executable parses Route_*_p<N>* names. The _pN convention is the script's.
Refuted: the .embsec_ sections are code (32,368 instructions, 108 functions),
not script; MiscBin.pak and DefTables.pak have zero hits.
Open: the 25 ISL opcodes and 147 built-ins are not decoded, and the
mission-level stream at +0x24 is only partly read.
Third run with the inducer on from flight start went 900s without freezing, so
the tally is 2 for / 1 against. Stated confound: the inducer was far weaker this
run (~47-50s per CPU pass vs ~4.2s in run 2, because two subagents were
saturating the box -- 22 passes in 15 min instead of ~150). That is consistent
with either reading, so it is not scored as a confirmation. What it does
establish is that the inducer is not sufficient, and that future runs must
report its measured rate rather than just that it was on.
Second healthy stability sample: 9 stable / 14 vary vs 12 / 13 in the first,
intersecting to only 8 threads. So the stable set is run-dependent and must be
measured within the run it is used in -- the same lesson as gdb thread numbering,
one level up.
The frozen half of the distribution experiment is still uncollected.
Closes the open question from mission-phase-membership.md. The static sweep
found no trigger because there is none in the data: each phase ends when its own
per-mission script says so.
[ScriptMission+40] is the 1/2/3 phase ordinal -- initialised at 0x822606B0 and
incremented at exactly one site, 0x822609F8-0x82260A00 (checked: only one stw to
40(rN) in the whole state machine). Its guard is [ScriptPhase+196] != 0, and
that flag has only two writers, vtable slots 0 and 1, reached only from built-ins
6 and 62 of the phase-script VM's 147-entry command table. Built-in 39 sets
[phase+300]=2, which ends the mission instead of advancing.
Refutes all four candidate triggers: no kill counter, timer, trigger volume or
message event appears on the path. Any such condition lives inside the script,
which is why three phases of one stage can have three different clear conditions.
CScriptInterpreter::ChangePhase (sub_822FF330, opcode 995) is the consumer, not
the cause; it writes a second phase mirror at [*(0x828F35F8)+236] -- a probe
target.
Not settled: the script bytecode is not on the disc under any obvious name.
Tooling correction: the .pe is not stale, it is a flat VA image (offset = VA -
0x82000000), verified 7/7 against the DB. Also: instructions.raw is an INTEGER,
and decoding it as a hex string silently compares nothing -- that nearly got
this correction recorded backwards.
Swept all 33 dat paks plus hidden/DefTables.pak: 7,750 IDXD objects, 190,782
records, 1,271,462 (name -> tag) pairs, 0 failures, 0 parse failures.
The key rule is stronger than first written: a record's key is tag_hash of its
own inline name, 190,782/190,782 -- no roster needed (roster-based recovery only
reaches ~30% of records). Verified independently here on UnitGroup_S02, 112/112.
The modulus is uniquely pinned, not just consistent: gcd of
poly256(name) - (tag & 0xFFFFFF) over all pairs is exactly 0x00FFFFDF, prime.
Case-sensitivity is load-bearing, not incidental: the disc has 17 name pairs
differing only in case (UNIT/Unit, TYPE/Type, STAGE/Stage, ...); name_hash
collides on all 17, tag_hash separates all 17.
Records the limit too: IXUD, the wide-string sibling, uses a DIFFERENT low-24
hash (0/19,808) though it keeps the same top-byte checksum (19,808/19,808). Its
offsets are counted in 16-bit chars (STR + 2*strsize == filesize) -- newly
decoded, all 534 parse. Its low bits are provably not a modular polynomial (gcd
= 1; exhaustive Barrett search over M in [2^20,2^25) finds nothing; fixed-position
deltas are non-constant), so it is lossy and needs the XEX.
Six captures across one healthy run: 12 thread states stable, 13 vary. Every
thread previously reported as a freeze signature is in the VARIES set --
including T68/T69, which I had kept as 'what reproduces across both freezes'.
They park and unpark during ordinary play, landing on the same objects they hold
when frozen, so seeing them parked while frozen is not evidence.
Net: no thread-level freeze signature has survived. Both frozen diffs are
consistent with healthy variation, and one-sample-per-state was never capable of
separating them.
Still standing: the 12 stable threads hold the same object in all six captures
and none of them moved in either frozen capture -- so 'not a whole-emulator
stall' survives, now resting on the stable set being undisturbed rather than on
a count of unchanged threads.
Also noted: gdb thread numbers are not comparable across runs (this run has
T132-T142, earlier runs had T104-T106), so future cross-run work must key on the
object address or guest thread id.
Route_S<NN>.tbl names every route Route_<squadron>_p<N><kind> -- 120/120 for
S02 -- so the squadron-to-phase map was on the disc all along, in a table read
until now only for fly-in geometry. 108/111 S02 squadrons map; the 3 misses are
typos in the route table and all are phase 2.
That resolves the 387-vs-300 gap after four failed attempts, by refuting its
premise: there were never 300 craft. Phase 1's member-level n multiset
{1x25, 2x1, 4x4, 9x12} matches the measured craft-per-record fan-in
{2x24, 4x1, 8x4, 18x12} bucket for bucket at exactly 2x, so 300 = 2*(151-1),
the -1 being the documented 41-vs-42 off-by-one.
The 2x is the probe double-counting, not real units: a 2x reading needs 16
Delta Sabers for 8 named pilots, and breaks the formation capacity bound in 20
of 37 phase-1 squadrons (16 of them on Formation_1_only, one slot) where the 1x
reading has 0 violations. All three measured craft totals being even is a
further tell.
Refuted: UnitGroup has no phase/spawn/delay field (1019 = sum(Count)*4 + 5*111,
every slot accounted for); DisableInterval is No for all 111 S02 squadrons;
stage\EnumSquadron_Test.tbl does not exist.
Confirms tag_hash independently: UnitGroup record key == tag_hash(squadron id),
1160/1160.
Still open: what advances a phase -- no static table encodes a trigger.
Closes the 4-byte record key. tag_hash is name_hash's shape -- byte-sum
checksum in the top byte over a 24-bit modular polynomial -- with two different
constants: modulus 0x00FFFFDF (2^24-33, prime) instead of 0x00FFF9D7, and no
lowercasing, so tags are case-sensitive. name_hash explains 0 of 8643.
Recovered from the tables rather than the executable: every inline field name
is a known (name -> tag) pair, and comparing names differing in one character
gives the per-position weights 1, 0x100, 0x10000, 0x21, 0x2100, ... -- a byte
leaving bit 24 re-enters as 33, i.e. reduction mod 2^24-33. Holds where it is
easy to get wrong (distance 8 and 9 carry correctly).
A record's key is the tag of its own name: FormationSet rosters 362/362,
UnitGroup rosters 281/281, S02 squadron names 111/111 -- so records can be
addressed by name without reading the roster first.
Implemented in Python (unitgroup.tag_hash) and Rust
(sylpheed_formats::hash::tag_hash) with 3 new unit tests carrying disc-derived
vectors; cargo test -p sylpheed-formats --lib hash is 8/8 green.
Not settled: the guest routine is unlocated, so this uses exact modular
arithmetic where the game may use a Barrett step without final fixup.
Run 2 put the inducer on from flight start and froze ~96s in, against 670s
clean with it off -- n=2, contrast sharp, confounder (elapsed mission time)
still untouched.
Withdraws last iteration's 'T74/T75 move off a semaphore onto an event' as the
signature to chase: it does not reproduce. In run 2 they are on XEvent while
HEALTHY and stay there. The healthy state varies between instants, so a
one-sample-per-state diff cannot separate a freeze transition from ordinary
variation -- I read a difference of samples as a difference of states.
Reproduces across both: T68 and T69 go from not-waiting to waiting, T69 on a
semaphore and T68 on an event both times. And 21 of 24 threads unchanged in run
2 (17 of 24 in run 1), so 'not a whole-emulator stall' now has two independent
captures behind it.
Next: repeat the capture several times within one healthy run to establish which
thread states are stable before reading any frozen diff.
FormationSet_S<NN>.tbl records are slot lists -- 1 + 8*FrameCount fields,
exactly. Resolving every squadron's FormationID and comparing gives
sum(n) <= FrameCount holding 1159/1160 across all 28 stages, 0 unresolved, with
539 filling the formation exactly. The single violation is a debug leftover
(S20, AI_Test / MessageSet_test, Formation_1_only with n=2) and is recorded.
The old 'n is not the _NN suffix of FormationID' observation was right but drew
the wrong conclusion: the suffix IS FrameCount, so n=9 against _30 just means 9
units in 9 of 30 slots.
Also: FormationID does not hash into its table (0/16). FormationSet carries a
name roster record -- no FrameCount, fields are (tag, name, '') with the tags
being the record keys -- the same convention as Enumerate_Squadrons. Second
occurrence of 'keys are resolved by an in-table roster, not by hashing'.
Does not close the 387-vs-300 gap, and the key derivation stays open.
The 'S17-S23 have no stage record' gap was an artefact of enumerating by the
literal 'Stage_S<NN>'. Tutorial records omit it -- they name no per-stage .xpr
and pull AIParams/weapons/strings/subobjectives/nameplate/collision from a
shared _Tutorial set -- so all six were skipped. stagetbl.py Stage_S18 returns a
full six-record definition and always would have.
Counting distinct *_S<NN> names across all 1119 decompressed entries: UnitGroup
and Route cover 28 stages (S01-S16, S18-S23, S24-S29); Stage literal and
AIParams cover 22; SUBObjectiveSettings 16 (story only). S17 appears in none --
it is not a stage that lost its data, it does not exist.
Tutorial records carry the same Phase_1/2/3 structure as story stages, so the
tutorial is not a special mission type at the data layer.
Refuted en route: GP_TUTORIAL.pak does not hold the config -- 2 RATC entries,
zero IDXD, like GP_CHALLENGE.pak.
Caught the freeze by waiting for the event (frozen.py + in_flight) instead of
sleeping a guessed interval; freeze_waitobj.sh splits into boot/watch so the
wait is not capped by one Bash call. Verified hard: a frame minutes later is
byte-identical to the capture.
Healthy vs frozen, same run: 20 -> 24 wait frames, XEvent 19 -> 23,
XSemaphore 8 -> 7. The signature is per-thread -- 17 of 24 threads sit on the
exact object they were on, four previously-running threads park, and T74/T75
move off a semaphore onto an event. So the freeze is not a whole-emulator stall.
Also corrects the previous entry's test: screen_id reads 'flight' during a
freeze by design, which is why frozen.py exists. Re-testing the saved frames
says that run was genuinely healthy, but it was right by luck.
heavy_read.py added to test whether the instrument provokes the freeze: I/O is
free (371 MB in 0.1s, page cache), the cost is Python-level CPU. One data point
-- 670s clean, then frozen 54s after the inducer started -- recorded as n=1, not
as causation.
23 wait frames, 30 objects, nothing unresolved -- the second deref turns every
former miss into a resolved object, as predicted. XEvent 20 / XSemaphore 9 /
XTimer 1; every WaitMultiple thread waits on a pair, and 78/79/80 and 64/65 are
worker groups sharing a handle.
%ebp does not survive as the count -- WaitMultiple reuses it at 8fc158 -- so the
array is bounded by reading until an entry stops resolving instead.
The frozen capture is still not taken: screen_id reads 'flight' at the second
capture and out to ~470s, so the mission never black-screened. The diff in the
data file is two healthy captures and is recorded as such.
The 8 threads whose [rbx] did not resolve to a vtable were never in
XObject::Wait. The backtrace grep matched WaitMultiple as a substring, and
there %rbx is the XObject** array (mov %rsi,%rbx) with the count in %ebp, so
[rbx] is objects[0] -- an object pointer, needing a second deref -- not a
vtable. The unwind restored rbx correctly for all 18.
freeze_waitobj.sh now takes the function and frame index from the backtrace and
applies the matching read, and captures twice in one run (healthy and after the
~270s black-screen) so the comparison is within-run. waitobj_report.py tabulates
both and diffs them, discarding any value info symbol cannot resolve.
Re-extracting the same gdb capture per thread rather than by grep qualifies the
previous entry. Of eighteen threads whose frame 3 is XObject::Wait, eight have
[rbx] equal to the XEvent vtable plus sixteen, two equal to the XSemaphore
vtable plus sixteen, and eight hold a pointer into the mmap region that is not a
vtable at all. So the waits are on two distinct kernel types, XEvent and
XSemaphore, and the earlier claim that the object is an XEvent was right for the
majority but not the whole picture.
The eight non-vtable readings are the method checking itself rather than a
failure. A polymorphic object's first word is always a vtable pointer, so those
values are simply not this: rbx could not be restored for those frames and the
unwind returned whatever the register held. A reading counts only if [rbx] lands
in the binary's vtable range and resolves to a "vtable for" symbol; anything
else is discarded rather than interpreted. Ten of eighteen resolve and the rest
are honestly unknown.
That also settles the previous entry's worry that the 0x7ffc and 0x7ffd
addresses looked like stack. They are the shared mmap region, which holds thread
stacks and large allocations alike, so the vtable check rather than the address
range is what separates an object from a stack slot.
The follow-up run that would have added /proc/<pid>/maps classification and a
wider object dump never booted -- EMULATOR GONE at 0s, skip_intro exit 4 --
most likely a stale emulator or lockfile from the preceding gdb session, whose
process tree is parented differently and escaped the usual cleanup. So the map
classification, the multi-word object dump and the frozen-state capture are all
still unrun.
Executed the wait-object read on a live run under gdb. For every thread whose
frame 3 is XObject::Wait, rbx gives a pointer whose first quadword is
0x5555562db8f0, which is the PIE base plus 0xd878f0, and the symbol table has
vtable for xe::kernel::XEvent at 0xd878e0. The stored pointer is the vtable
symbol plus sixteen, since offset-to-top and RTTI come first, so it matches
exactly. The same vtable appears on every sampled waiting thread while the this
pointers differ, meaning many threads waiting on different XEvent instances. The
whole chain needs no DWARF and no rebuild, as the static groundwork predicted.
Two caveats, both recorded rather than smoothed over.
This is a healthy-play snapshot, not the freeze. The capture landed at 195 s of
flight with screen_id reporting flight and a non-black mean, because boot under
gdb costs about 300 s and the entire experiment has to fit inside one call --
a timeout kills the process group and takes the emulator with it, which lost an
earlier attempt outright. So this describes what threads wait on during normal
play, which is the control the frozen capture never had, but it is not the
frozen case.
And the this addresses look like host stack rather than heap. Either xenia
places these objects somewhere unusual, or rbx at frame 3 is not Wait's this
after the unwind and the vtable match is coincidence. An exact plus-sixteen
match on a known symbol is hard to get by accident, but the address range is not
what was expected. The settling check is to dump a few words at rbx and see
whether they look like an XEvent -- vtable, KernelState pointer, handle and type
fields -- or like saved registers.
Pure static analysis, no run spent. XObject::Wait's prologue does mov %rdi,%rbx
at 8fbc9c, so the this pointer lives in a callee-saved register rather than a
stack slot. And the binary carries full unwind information: .eh_frame with
127231 FDEs, which survives in Release builds because C++ exceptions need it,
including an FDE covering 8fbc90 to 8fbde2 that tracks rbx explicitly.
Together those mean that from a thread frozen deep in pthread_cond_wait, moving
to the XObject::Wait frame and reading rbx yields the XObject being waited on --
gdb reconstructs callee-saved registers during the unwind from .eh_frame alone,
with no DWARF involved. Reading the first quadword at that pointer gives the
vtable, and vtable symbols are in the symtab, so the object's concrete type is
identifiable too.
This revises the previous entry, which listed route 1 as per-frame archaeology
that must be redone whenever the binary changes, and route 2, a RelWithDebInfo
rebuild, as what would make the question easy. Route 1 is neither expensive nor
fragile: two gdb commands per thread, no rebuild, and the oracle stays
byte-identical to the binary every other measurement in this corpus was taken
against.
Not yet executed on a frozen run, which is the next step and is now a small one.
Two corrections to the plan the freeze file leaves for the next pass, both cheap
to establish and both worth having before a run is spent discovering them.
The export in the frozen frame is confirmed. The shim frame carries its ordinal
as a template parameter, 176 decimal, which is 0xB0, and xboxkrnl_table.inc line
190 maps that to KeWaitForSingleObject. The identification was previously read
off frame names; it is now pinned by ordinal.
The file's stated next step does not work on this binary. It says the wait
target is one info-args-equivalent away now that a debugger can be attached, but
readelf shows zero debug sections and only a symtab of 27807 entries. The
Release build is not stripped, which is why frames carry function names, but it
has no debug info at all, so gdb can name a frame and nothing inside it: no info
args, no p *this, no member access. The earlier note conflated "not stripped"
with "has debug info".
Two routes remain and their costs are now recorded. A prologue-guided stack read
can recover XObject::Wait's spilled this pointer from a known stack slot, needing
no rebuild and keeping the oracle byte-identical to the one every other
measurement in this corpus used, at the price of per-frame archaeology that must
be redone if the binary changes. Alternatively a RelWithDebInfo build via
build-canary makes this and every future freeze question straightforward, at the
price of a full compile and a binary that differs from the measured one.
Neither is attempted here. Recorded so the next pass picks with the costs
visible.
One new data point on the in-mission freeze. The refuted resume-spin lead rested
on refused resumes being normal -- thousands during gameplay, more in a healthy
run than a frozen one. Today's freeze log contains zero of them across 1147
lines, and the mission still froze at about 267 s with a black screen, against
2447 in an older log. So the warning is not even necessary for a freeze, let
alone sufficient, which closes the lead from the other side.
Also refuted today, before this file was found: the burst of BaseHeap::Release
failures at the end of the short log looks like a freeze signature and is not.
In the longer log the same failures span lines 1044 to 5210 and the log
continues for 2700 lines afterwards; they begin at mission load in both runs and
are routine.
The uncomfortable part is that both were already settled in
mission-freeze-resume-spin.md, as was the 0xbdb59668 address that a previous
iteration rediscovered independently. That is twice in one session that existing
work was redone.
The cause is mechanical rather than forgetfulness. docs/re/INDEX.md listed 20 of
43 notes and none of the recent ones, so searching the index for prior work on
the freeze returned nothing -- the corpus was searched, but the search was
blind. INDEX.md now carries a generated table of every note under docs/re/, 59
entries with title and status, and states outright that it should be searched
before starting an investigation. Regenerating it is a few lines of Python and
should be redone whenever notes are added.
A two-segment OB watch made the mechanism visible. Segment 1 ran clean, with the
witness reporting zero stalled samples of fifty and OB steady at 4 for 250
seconds. Segment 2, attaching to the same still-alive emulator, got "NOT IN A
MISSION (no unit definitions)".
The emulator was alive; the mission was gone. The pilot's hull reads 1500 in
every line, so the player never died. Its log stops at t=267 s with consecutive
byte-identical lines, the freeze signature. And the screen is now entirely
black, mean (0,0,0). This is the pre-existing mission-end black-screen freeze
already recorded in the corpus, caught in the act. Segment 1's clean witness is
not a contradiction, since it sampled to about 250 s and the freeze came at 267.
That corrects an earlier claim. mission-arrival-watch.md records the blocker as
removed because the emulator survives between calls, which is true, but it was
stated as though observation could be extended indefinitely within a turn. The
emulator surviving is not the mission surviving. The binding limit is the game's
own black-screen at roughly four and a half minutes and no amount of chaining
crosses it. The earlier chained run that reached 435 s of cumulative
verified-live observation did so across two segments of one mission, which fits:
its second segment froze at t=135 s.
The consequence is worth stating plainly. Any experiment needing more than about
four and a half minutes of one continuous mission is not doable on this build,
chained or not, and that includes clearing sixteen marked fighters at two kills
per five minutes. The freeze is therefore now the highest-value target in its own
right, since fixing it would unblock the arrival question, the phase-advance
question and the OB series together. It is also a long-standing known defect
rather than anything this work introduced.
ob_watch.py verifies the address against a confidence-gated HUD reading before
reporting anything, and two consecutive fresh launches settle the question:
0xbdb59668 held 3165285888 against a HUD of 4 on one, and exactly 4 on the next.
The address is not stable across launches, the old note that it recurs in about
five runs of seven was right, and the gate did its job by refusing to report a
series from an address that did not describe that run. The watcher now hunts the
address on the current run when confirmation fails, using the same intersection
method, so it no longer depends on a lucky launch.
The HUD reader is also gated now. ob_read returns a best and second score per
digit and those were printed but never checked, which is how one misread
poisoned an intersection and produced a wrong refutation of big-endian u32. A
reading is accepted only if every digit scores at least 0.80 with a margin of at
least 0.05, the rule ob_read's own docstring states.
The measurement itself is a negative. With a clean witness, zero stalled samples
of fifty, OB held at 4 for 250 seconds while the pilot targeted e010 for 1964
ticks and fired on 1635 of them. Constant fire at the marked attackers and not
one decrement, so it destroyed none, which matches the roughly two marked kills
per five minutes measured earlier. The fire rate itself rose from 4.6 % of ticks
in an earlier diagnosis to 83 % here without producing more kills.
Recorded as unreproduced rather than explained away: the run that found the
address saw the counter rise 4, 8, 12 over five minutes, and that reading was
confirmed against the HUD. This run was flat over a comparable window. Both
observations are sound and they disagree, so the rise is not a stable property
of the mission's first five minutes and presumably depends on progress this run
never reached.
Ported onto the shared harness, the HUD changed from 4 to 8 and the intersection
collapsed in a single step: one u32be survivor at 0xbdb59668, with the u16be and
u8 hits at 0xbdb5966a and 0xbdb5966b being the low half and low byte of that same
word.
Verified live rather than asserted. Reading screenshot and memory together three
times, the one legible HUD frame showed 012 against mem@0xbdb59668 = 12; the
other two frames were unreadable rather than mismatched. This independently
rediscovers the address the earliest sessions found by digit-transition hunting,
by a completely different method.
It also corrects an earlier conclusion. A previous run intersected HUD readings
of 4 and then 11, got zero survivors in every encoding, and that was written up
as eliminating big-endian u32 for the whole region. This run shows u32be holds
the counter, so the refutation was wrong. The likely cause is the input:
ob_digits.png has templates for 0 1 2 4 8 only, so values containing other
digits are misread rather than rejected, and "11" was probably one of those. A
single bad reading poisons an intersection permanently, because it removes the
true address and nothing later can restore it. The lesson is that an
intersection method needs individually verifiable inputs -- the reader's
confidence scores were printed but never gated on.
One observation reopens the arrival question in a useful way: the counter
increases, 4 then 8 then 12 across about five minutes, measured in memory so not
a digit misread. A count of remaining marked targets that rises means targets
are being added during the mission. That does not contradict the deployment
finding, since the roster is fixed at load, but it does mean the game marks new
objective targets as the mission proceeds. Watching this one address across a
whole mission is now the obvious next experiment and costs almost nothing.
Four probes were written from a blank file and each re-learned the same lessons
by losing a run: that a flat run cannot be told from a frozen guest without a
stall witness, that results held to the end of a run are destroyed by a turn
timeout, that a roster count which is not the stage's member count means a
different stage loaded and must be discarded, and that a run's witness state has
to be read before its numbers. Writing each lesson down did not stop the next
probe repeating it, because each probe started from nothing.
probeharness.py makes them structural. Probe(baseline=N) discovers the roster,
rescans up to five times and refuses to start if the count never reaches the
baseline. The witness is calibrated on construction, sampled by tick() and
reported by status() and summary(), so a probe cannot forget it, and when no
witness is found it reports UNVALIDATED rather than zero stalls. emit() flushes
on every line. craft(), strengths(), alive() and heap() supply the
roster-to-craft link, per-record liveness and the raw heap, so a new probe
writes only its own logic.
Verified rather than asserted: deploy_probe.py reimplements the per-record
deployment watch on top of it in about forty lines against wave7_probe's
hundred and fifty, and its first live run was clean -- 116 roster records, 32
witnesses at 10/s, zero stalled samples, seven losses tracked, and the TSV
written incrementally. Nothing about the result is new, which is the point: the
harness reproduces a known-good measurement.
The existing probes are deliberately not ported. They work, and rewriting them
would risk changing results other documents cite. New probes should use the
harness; old ones should be ported when they next need a change.
ob_by_hud.py now scans seven readings of the same bytes and keeps a separate
candidate set for each, as byte offsets: u32 big and little endian, u16 both
endiannesses at both alignments, and u8. The big-endian u32 reading had been
refuted, so widening rather than assuming is the point.
u32le is much the tightest at 154 candidates against u32be's 4452. That is a
hint about the encoding rather than a result, since a rarer bit pattern narrows
faster regardless of meaning.
The run is inconclusive. The HUD read 4 at every sample, so there was no second
value to collapse the sets against, and from t=136 the candidate counts are
byte-identical across five samples in all seven encodings, which is what a
frozen guest looks like -- nothing in 32 MB changed at all. The probe had no
stall witness, so the run cannot prove it either way. One is added now.
Worth stating plainly: this is the fourth probe written without a witness and
the third whose flat output could not be distinguished from a freeze. Each time
the fix gets applied to that one script. The durable fix is the shared probe
harness already noted in this file, and the lesson recurring four times is
itself the argument for building it.
What the hunt needs is unchanged: two HUD readings at different values in
non-stalled samples. The counter moves on kills, which lands back on the combat
limit, though the earlier 4 to 11 observation shows it does move.
The correlation route is gated on marked-fighter kills, which the pilot manages
at about two per five minutes. ob_read.py already reads the counter off the
screen, so ob_by_hud.py matches the displayed value against memory directly and
needs no kills at all: screenshot, read the digits, keep heap words equal to that
value, intersect across readings.
Four readings at value 4 narrowed 6156 candidates to 4312, the expected slow
drift. Then the HUD read 11 and the intersection collapsed to zero. A word
holding this counter must equal 4 at the first four samples and 11 at the last,
and none does, so within the entity heap read as big-endian u32 the counter does
not exist. It may be u16, u8, little-endian, or outside that region. Both
previous hunts assumed big-endian u32 there, so this eliminates the assumption
rather than merely failing to find anything.
The displayed value also went up, from 4 to 11 over about 340 seconds. A pure
countdown of remaining marked targets should not rise, and the deployment work
says phase 1 gains no new participants. Three readings are possible and none is
tested: the cell being read is not REMAINING OB, the digits are misread, or the
counter genuinely counts something that can increase. The two clean readings
scored 0.95 to 0.98 against their templates, but 4 and 11 use only digits that
are in the strip, which is exactly the selection effect that would hide a wrong
reading -- the template set covers 0 1 2 4 8 only, and most samples came back
unreadable.
Next is widening the scan to u16 and u8 and to little-endian, and beyond the
entity heap, which is a change to one function and costs no combat. Extending
ob_digits.png with the missing digits would also raise the sample yield, since
only two of eleven readings in a 480 s run were usable.
With the value filter in place, an attach watching turret losses -- frequent,
unlike marked-fighter kills -- narrowed from 374 candidates to 2 to 0 across
three events. That is exactly how a correlation search should behave, and it
ends in a refutation rather than fizzling out: no plain u32 in the entity heap
decrements consistently with turret kills.
Witness checked first, per the rule adopted last iteration: 12 of 25 samples
stalled, but all four kill events fall in the early non-stalled stretch between
t=45 and t=177, so the events themselves are sound.
The negative fits the objective text rather than contradicting it. Stage 02
phase 1 asks to shoot down all invading enemy fighters, and turrets are not
fighters, so a counter of remaining marked targets should not move when a turret
dies. It also rules out the cheaper reading that REMAINING OB is a general kill
or enemy-remaining tally: it ignored ten turret deaths.
The method is proven and the search region is right, since 0xbdb59668 where the
counter was originally seen lies inside the scanned range. What is missing is
unchanged -- two or three e010 kill events in non-stalled samples -- against the
same two obstacles, the freeze rate and a pilot managing about two
marked-fighter kills per five minutes. One clean 220 s run this iteration
produced zero.
