39 Commits

Author SHA1 Message Date
Claude (auto-RE)
bbfeb1c387 re: ship-placement generalisation — static audit + first Stage-02 F10 capture (WIP)
Working the BACKLOG item "capital ships assemble wrong in the viewer".

- ship_audit over all 22 stage containers: only ONE outlier ship
  (f002_bdy_05, Stage_S03/S27, dist 6540 vs cluster spread 1071), so static
  assembly is not grossly broken class-wide. Recurring MULTIKEY joint tracks on
  f104/f105/f106/e102 are the standing hypothesis for class-specific error —
  e106, the one validated ship, has none.
- new tools/re-capture/ship_capture_session.sh: one blocking session that boots
  Stage 02 and fires N F10 ship-captures with screenshots. Boot to in-flight was
  24 s; 3 of 5 presses produced logs (2964/3111/3668 draws).
- NEGATIVE, unexplained: correlate_capture matched ZERO parts for f101/f105/
  f106/e105. Documented with the collected facts and the next step (invert the
  match: largest capture vcounts -> which decoded part has that count).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 05:16:20 +00:00
claude-re
695351dfc4 docs/re: mark the 'targeting is automatic' section superseded
It concluded no target-select input exists because no single press cycled a
target; select is A pressed twice. Kept with a correction banner rather than
deleted — a probe that never performs the action will 'prove' the action does
not exist, which is the reusable warning.
2026-07-30 20:48:15 +00:00
claude-re
4821ba7fea pilot: target select is A pressed TWICE — from the HUD tutorial, not from probing
HEADS-UP DISPLAY tutorial, verbatim: 'Press A twice to target the enemy closest
to the center of the screen.' A double tap, which is exactly why every button
sweep in flight-controls-runtime.md found nothing and why I concluded targeting
was automatic — each sweep tapped once. It also explains the missiles:
GuidanceType 5 guides to the GAME's selection and the loop had never made one,
so 98 launches guided to nothing.

Wired in: double-tap A when the committed contact is already within 14 deg of
the nose, so the game's choice and ours are the same object. One run: 8 kills
from 66 missiles (12% per missile) against the previous 9 from 101 (8.9%). The
absolute count is inside run variance and the efficiency gain is one sample, so
neither is claimed as decisive — it needs repeat runs.

Also documents that expository tutorials self-advance while interactive ones
stall (BASIC CONTROLS waits forever on 'Go to the box'), and that captions need
cropping across many frames because they type out.
2026-07-30 20:47:54 +00:00
claude-re
58f421d896 re-capture: read the ADVANCED CONTROLS tutorial; its moves regress the pilot, so they ship off
tutorial_capture.sh plays a tutorial and photographs what it teaches. ADVANCED
CONTROLS states three mechanics the key-config screen only named:
  B + LS            Side Roll / 180 Degree Turn / Level Off
  B + A together    face the target (snap turn)
  LT + RT together  'sets your fighter's speed to that of the target ... works
                    well when you are trying to get behind an enemy. Once
                    behind an enemy, this also helps you attack them.'

Wired both usable ones in and measured, one run each, everything else equal:
  commitment only ............ 101 missiles, 364 fire frames,  9 kills
  + match(4500) + snap-face ....  9 missiles,  28 fire frames,  0 kills
  + match(1200) + snap-face ...  57 missiles, 225 fire frames,  2 kills

So both are a net regression as applied, and both now default to OFF. Matching a
target's speed while still 5 km behind means never closing (the pilot sat at
272 u/s all run) — the tutorial scopes it to being already in the saddle. The
B+A snap turn reorients mid-pursuit and destroys the dwell commitment buys.

The code and thresholds stay so a future session can re-enable and A/B them over
SEVERAL runs; one run per config is inside this stage's spawn variance.
2026-07-30 19:50:25 +00:00
claude-re
9f41fe08e9 pilot: target commitment takes kills from 2 to 9; and the target pointer does not exist
The loop re-scored every contact every tick, so the nose chased whichever
fighter scored best that instant and aim error wandered 10-40 deg through a
pass. A missile lock is time-on-target, so constant switching is the one thing
guaranteed to prevent a kill. Commit to a contact until it dies, passes 6000,
sits >90 deg off the nose for 2.5 s, or 14 s elapse.

Same guns, same ballistics, same escort weighting, same missile cadence:
kills 0000 (five gun-only runs) -> 0002 (missiles) -> 0009 (commitment), with
101 missiles vs 98, and the largest hostile-population fall of any run
(134->97). Own hull untouched. The ACROPOLIS still ended at 76.6%, so this is
lethality, not the mission outcome.

Also records a negative result so it is not re-attempted: the selected target is
NOT a raw entity pointer. Three searches came up empty — a +-0x1400 window of
the player object, a full-RAM sweep of every entity-pointer word tapped through
each button (only thread-stack slots churn, which is frame noise), and a delta
tally over all 150 entities of the kind that found the definition pointer at
+0x130. The selection must be a handle, an index, or in a subsystem outside the
entity object.
2026-07-30 19:19:23 +00:00
claude-re
95ac545b2b docs/re: the game's own in-flight action list, off the disc
Decoded dat/GP_OPTIONS.pak (po_keys_btn* sprites): the OPTIONS key-config screen
lists every bindable in-flight action. Two of them change the plan.

'Change Target' exists — so target selection IS an input, and the earlier probe
that swept LB/X/B/A/LS/RS found nothing only because it watched the ammo
counters, which those actions do not touch.

'Padlock Mode Toggle' is a view/aim lock onto the selected target: the aim-dwell
problem solved by a game mechanic rather than by tuning a PD controller, and
presumably how a human holds a contact long enough to lock a missile.

Confirms our measured bindings (Use Nose Weapon = RB, Use Main Weapon = Y,
Accelerate/Decelerate = RT/LT, Radar Map Toggle = d-pad) and adds Special Move,
Maneuver, Resupply and Change Main Weapon (which would reach ASMissile, Power
5000). Also records that CONTROL SETTINGS carries a Control Type preset plus
yaw/pitch/roll sensitivity, so the mapping is not fixed and any stick
calibration is only valid for the save's current profile.

Tutorial menu (GP_TUTORIAL.pak) names the six lessons: BASIC CONTROLS,
HEADS-UP DISPLAY, RADAR, SUPPLY AND SPECIAL MOVES, RADIO ORDERS, ADVANCED
CONTROLS.
2026-07-30 18:54:16 +00:00
claude-re
ab8f5307ff docs/re: in-flight control mapping, and why lethality is an aim-dwell problem
Probed rather than assumed: hold each pad input and photograph the HUD ammo
counters. RB = nose gun (NOSE BM 6000->5956 in 4 s, ~11 rounds/s, HEAT rises),
Y = main mount (MAIN MPM 300->299), d-pad = tactical map overlay, and
LB/X/B/A/LS/RS move neither counter.

No target-cycle input exists. The green TARGET marker is already up with nothing
pressed, so the game selects for us and a guided missile's lock is a
time-on-target mechanic, not a button. That closes the lethality question: it is
not target choice (escort weighting), not ballistics (now from the confirmed
Shell records), and not the mapping — the steering loop simply never holds the
nose on one contact long enough to lock. Aim dwell is the next lever.

Also records two things the reimplementation needs: two weapons with separate
ammo pools and HUD counters, and a HEAT bar that fills while the gun fires
(cap and cool-down not yet measured).
2026-07-30 18:49:16 +00:00
claude-re
76f463b611 pilot: RB fires, Y is the main mount, and guided missiles produce the first kills
fire_probe.sh holds each pad input in flight and photographs the HUD ammo
counters. RB moves NOSE BM 06000 -> 05956 in 4 s (~11 rounds/s, HEAT rises);
Y moves MAIN MPM 00300 -> 00299; nothing else moves either counter. So the
control mapping is measured rather than assumed, and 'we never shoot' is dead:
we shoot and miss.

The disc data says to stop shooting: Shell_TCAF_DeltaSaber_Missile_P is Power
200, GuidanceType 5 (guided), MaximumRange 5000, versus the nose gun's Power 15
unguided — one missile is worth ~14 gun hits on a 500 HP fighter and it steers
itself. Launching them (press Y, release a tick later, >=2 s apart) produced
YOU KILLED: WARPLANES 0002 — the first non-zero kill counter of the series,
against 0000 in all five gun-only runs, with hostiles down 134 -> 104.

Still only 2 kills per 98 missiles (~2%). Likely cause: the game expects a lock
before launch and an unlocked missile is wasted. Reading the lock state out of
RAM is the next step.
2026-07-30 18:40:39 +00:00
claude-re
3f6efadf9e pilot: real ballistics from the solved Shell records — and the metric that says it did not help
Shell_TCAF_DeltaSaber_*_P: Velocity 8000, LifeTime 0.5 s, MaximumRange 4000
(self-consistent: 8000 x 0.5 = 4000), all confirmed. Two things were wrong:
lead computed flight time as d / OUR speed (400-2000 u/s, so every shot was led
4-16x too far), and FIRE_RANGE was 5000 — past where the shells expire.

Both fixed. But the HUD's own kill counters read 0000/0000 at the end of EVERY
run including the nearest-fighter baseline, so the pilot kills nothing in any
configuration and 'fraction of frames firing' was never measuring lethality.
No improvement is claimed.

One clean negative result kept: gating on the target's angular half-size alone
(2.7 deg at 2584 units) is far tighter than the steering loop can hold the nose
— firing collapsed to 1 frame in 2639. Angular size is a floor on the firing
cone, never a cap.

Next: the HUD carries a live ammo count, so holding fire and watching it settles
'we never shoot' vs 'we shoot and miss' in a single run.
2026-07-30 18:16:26 +00:00
claude-re
402985adbf pilot: escort-weighted targeting (DEFEND), plus the capital-ship keep-out it needed
While the asset is losing hull, target what is pressing IT — ranked by distance
to the asset minus credit for closing on it — instead of what is nearest to us.
Trigger and ranking both read the live hull (pos+0x154), so nothing is inferred.
DEFEND engaged 1.9 s after the asset's first hit and held 54% of a 330 s run.

It did NOT measurably save the asset: over the window two runs share, the
policies are equal to within noise (t=239: 23218 vs 23038). Two reasons, both
recorded rather than papered over: the runs are not comparable past that window
(spawn timing differs and the hostile count GREW 134->166 in one, fell 147->118
in the other), and the real bottleneck is lethality — the guns are on for 12% of
combat frames because the target is outside the 9 deg cone the rest of the time.

Also corrects a single-run claim in the previous commit: the asset is NOT
reliably safe for the first ~170 s. A second run had first damage at t=70 s. The
stage does not replay identically; only 'the loss is slow' survives.

Fixes a fatal bug the new mode exposed: DEFEND flies at the asset, which sits
inside the friendly formation, and the first escort run went hull 1500 -> DEAD in
one tick at 2026 units/s, 0.6 s from a friendly destroyer that avoidance thought
it would clear by 365 units — the ship's radius is 2000. Keep-out applied only to
hostile turrets. Every entity above BIG_RADIUS now gets its own radius + 800 of
physical keep-out with braking inside it, whatever its faction.
2026-07-30 17:49:38 +00:00
claude-re
c277e42c92 docs/re: the hull anchor is class-wide — the escort objective is scoreable live
own_state.py found hull = position + 0x154 for the PLAYER. Stage 02 is an
escort and is lost when the ACROPOLIS sinks, so scoring it needs someone
else's hull. Measured over 240 s of Stage 02: at t=0 pos+0x154 equals each
entity's own definition HP across 7 classes and 5 distinct HP values (turret
100, fighter 500, destroyer 10000, cruiser 30000, Acropolis 25000). Nothing
read above its HP; the five that read slightly below were already under fire
when the player launched. It falls with damage (780 events), goes negative at
death, and the object then leaves the heap.

UN_f101_TCAF_Acropolis: HP 25000, radius 1400, measured 25000 -> 23038 over
240 s with the attack starting only at t~170 s (~600 HP/min) — so the earlier
GAME OVER was not a fast loss, it was an undefended one.

Also: REMAINING OB reads 012 while 118 ADAN entities are alive, so it counts
objectives, not hostiles; its address is still unknown.
2026-07-30 17:21:05 +00:00
claude-re
8ee3bfbcef re-capture: stop detaching the display and the emulator
ensure_display() and the run-canary launch used 'setsid nohup', to make them
outlive the shell that started them. That is what has been killing every long
run: a setsid'd process belongs to no supervised tree, and both Xvfb and xenia
were reaped a couple of minutes in — the 'they die on their own every few
minutes' note in the project memory. Measured: a bare Xvfb with no emulator
running exited 0 (a clean shutdown, not a crash and not the OOM killer) at the
exact moment a turn ended, and a session whose display was setsid'd from inside
a tracked task died the same way 12 s after launch.

Run launch_mission.sh as one tracked background task and keep Xvfb, openbox and
xenia as its children. Adds an exit-status wrapper so a future death reports the
server's own exit code (128+N for signal N) instead of being inferred, and
clears a stale X lock before starting.
2026-07-30 17:12:00 +00:00
claude-re
91fb022caf re-capture: a dead display must not look like a playing movie
skip_intro.sh compared two screenshots to detect the intro movie. When the X
server died 3 min into a run, `screenshot` failed silently and left both PNGs
at their previous contents — two stale files, whose RMSE is a constant non-zero
number, i.e. exactly the signature of a changing screen. The loop then reported
"movie -> skip A" every 5 s for the whole 600 s timeout with no emulator and no
display alive, and the session wasted 10 minutes before saying BOOT FAILED.

Both waiters now verify, every iteration, that the screenshot was actually
written, that the display answers xdpyinfo, and that a non-zombie xenia_canary
exists — with distinct exit codes (3 display, 4 emulator, 5 capture) so the
session log names the cause instead of timing out.

Also adds mission_state.py + escort_session.sh: read pos+0x154 against each
definition's HP for EVERY entity, to test whether the hull anchor is a property
of the entity class rather than of the player object (the escort question).
2026-07-30 17:06:08 +00:00
claude-re
93d6534efe docs/re: open a backlog, first item = capital ships assemble wrong in the viewer
The runtime-capture write-up declares static ship assembly exact, but its test
covers exactly one ship (e106). Records what is suspect (engine-cluster rig,
the X-reflect twin heuristic, cross-id turrets), and that the F10 capture is
already the oracle to settle it on a second class.
2026-07-30 17:01:32 +00:00
41c34dc3ca re: surviving is not winning -- Stage 02 is an escort
The second 240 s flight ended on GAME OVER with our hull untouched at
1500/1500. Nothing shot us down; the ACROPOLIS was sunk while the
pilot pursued an attacker two kilometres away. "Nearest hostile
fighter" is the wrong objective function for this stage.

The fix is available with what is already solved: the protected ship's
hull is readable with the same anchor as ours (position+0x154, maximum
= its own definition HP), so target priority can be "closing on the
asset" and the escort can be scored live.

Also records that a frozen log tail is what mission-end looks like
from outside the emulator, not a wedge.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 05:43:42 +00:00
2e9903d0fc re: the autopilot now survives, and kills
Three measurements, then a pilot built on them.

* Hull is position+0x154. Found by anchoring on a field the definition
  already had solved (HP = 1500) rather than scanning for a value that
  falls: an undamaged craft must contain its own definition's number.
  Confirmed by the trace across a death -- 30/60/90 per hit, negative
  at 0, GAME OVER on screen.
* RT accelerates, LT brakes, and the throttle is a persistent setting
  (488 -> 1510 -> 174 units/s, measured as displacement per second of
  the craft's own position, so no speed field was needed). This
  overturns the earlier "RT is not the throttle", which came from
  assuming the control and hunting for a field.
* Shield is probably position+0x430 (== definition MaxValue 400), not
  yet confirmed live -- nothing had damaged it.

pilot.py is a state machine on damage (ENGAGE / EVADE / RETIRE) that
treats turrets as keep-out zones instead of targets. It flew Stage 02
for 300 s with the hull untouched at 1500/1500 and took the first
confirmed kill (WARPLANES 0001); every run the day before was dead
inside 35 s.

Two bugs the live run exposed and this fixes: gating the guns on the
commanded direction keeps them cold whenever avoidance is steering
(gate on the target instead), and an orbit-plus-brake rule made it
circle one attacker for 40 s outside its own firing cone.

Also: boot to in-flight is now ~100 s unattended, because
wait_flight.sh waits for the HUD's own shield bar instead of a fixed
75 s sleep that lavapipe does not honour; entities2.py picks the
attitude block by matching the measured flight path (taking the first
orthonormal block gave a bone/camera frame); and the entity-heap scan
is numpy instead of a per-word Python loop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 05:38:00 +00:00
4623387f6c re: the memory-driven autopilot now flies, tracks and shoots
Three findings turned the previous dead end into a working loop, each checked
against something independent rather than assumed:

  * a live entity's definition pointer sits at position + 0x130, so one heap
    scan types every craft in the scene -- which is what separates ~20 real
    combatants from ~30000 moving particles. The result is coherent: wingmen,
    enemy turrets and attackers, friendly capital ships, one Player.
  * orientation is a 3x3 at position - 0x70 stored with a 16-BYTE ROW STRIDE
    (a 4x4 whose translation row is the position). The earlier search for nine
    contiguous floats could not find this by construction, which is why the
    first pass wrongly concluded there was no transform. Confirmed by its row 2
    matching measured direction of travel at cos = +1.000.
  * RB is the fire button, established by consequence: of RB/LB/A/B/X/Y/RT/LT
    it is the only one that makes the nose-ammo counter in RAM fall.

Control is PD on the aiming error with the derivative from body angular
velocity, and target selection weighted by off-boresight angle -- pure
nearest-first kept picking targets 90 deg off the nose, whose bearing rate then
outran the turn rate and held the craft outside its firing cone at a steady 27
deg pitch error.

Observed: distance to target closing monotonically, yaw error driven from -8 deg
to ~0, fire=1 once inside the cone, ammo counter falling.

Not solved: survival. There is no evasion, no shield/armour awareness and no
throttle control, so it flies a straight pursuit into defended space and is
shot down; every long run has ended in GAME OVER. Mission completion needs
those, plus objective-aware target priority.
2026-07-29 19:32:37 +00:00
72365b217a re: memory-driven autopilot -- infrastructure, and an honest account of what is not solved
Reads the live world out of guest RAM and drives the pad from it. Working:
loop-rate memory reads, whole-RAM float scanning with numpy (1270 orthonormal
3x3 blocks in 6.2 s), entity enumeration by unit type (116 live instances in
Stage 02), pad control written straight into the vgamepad FIFO (the CLI spawns
a process per command and its tap/hold sleep inside the server, so neither is
usable in a control loop), unattended mission entry, and the Hangar loadout --
the "Recommended" control is AUTO SELECT, which at 5 % progress is a no-op
because only two weapons are developed and both are already mounted.

Not working, and the reason the craft is not yet flown: the class 0x820af030
is NOT the live entity. It has one object per spawned thing and carries the
unit-ID string, which is why it looked like the entity list, but every one of
its 384 words is constant across a 29 s in-flight capture. No transform lives
in it or one pointer hop from it. Input correlation (hard left yaw vs hard
right, looking for a turn axis that reverses) does find self-like objects at
cos = -0.99, but they cluster in what looks like a camera volume rather than
the craft, and with no definition pointer near them the trick of learning one
entity's layout and applying it to the rest has nothing to anchor on -- so the
33418 moving triples in a firefight cannot be split into enemies, friendlies
and bullets, and there is nothing to aim at.

Two dead ends are recorded so they are not repeated: RT is not the throttle
(the two-state speed scan therefore found nothing), and comparing orientation
matrices 2 s apart is outside the small-angle regime, which is what produced
"angular velocities" of 30000.

Also corrects the claim in unit-struct-runtime.md that 0x820af030 holds live
state. The definition class 0x820af844 and every value derived from it are
unaffected.

autopilot2.py (a PD controller using body angular velocity from consecutive
rotation matrices) is committed but has never had a valid config to run
against, and is marked as untested.
2026-07-29 19:04:33 +00:00
22d2518847 re: don't let zombie emulators fool the liveness check
Container PID 1 is `sleep infinity` and never reaps, so every emulator this
script kills stays as a <defunct> entry that `pgrep -x` still matches -- after
a few captures the script always believed one was running. Same trap as the
Xvfb check. Ask ps for the process state and skip anything in Z.
2026-07-29 18:24:18 +00:00
1ee2d5e406 re: widen unit coverage to all six tutorials, and prove the fields are run-invariant
Captures the five remaining tutorials (grab_tutorial.sh, one cold boot each)
and re-solves over seven snapshots from seven separate emulator runs. Coverage
18 -> 21 units, confirmed fields 22 -> 27 (Size_Y, MassScore, MinimumVelocity,
AV_PitchPlus_Min, AV_PitchMinus_Min join the zero-contradiction set).

The multi-run union exposed something the single-run check could not: the same
unit's object is NOT byte-identical between runs. unit_runtime.py --crosscheck
pins down why -- exactly 15 words differ, 13 of them holding guest heap
pointers, and none of them is a solved or interpolated field offset. So every
value reported is run-invariant, which is a stronger statement than the
within-run identity check that came before it.

The two non-pointer stragglers are a real caveat, now documented rather than
smoothed over: +0x2c8 reads 8000.0 for UN_f001_TCAF_DeltaSaber_T_Ttrl in two
tutorials and 10000.0 in a third, with a 0/1 flag at +0x2d0. A couple of words
in the object are set per stage, so it is mostly but not entirely the parsed
table. Unidentified, marked NEEDS-HUMAN.

Coverage honesty: all six tutorials together add only 3 units the missions do
not already have -- they reuse one training box, one drone and the player
craft. Further coverage needs story progress, not more tutorials.

Two navigation facts encoded in grab_tutorial.sh, both learned by breaking
them: the main menu is not input-ready for ~10 s after the title tap and early
d-pad presses are dropped (which sends the A to NEW GAME); and NEW GAME is not
a shortcut to Stage 01 -- it gates on DIFFICULTY then plays the prologue. A
NEW GAME excursion to the READY ROOM leaves game01/savedata byte-identical,
verified by diff against a backup.
2026-07-29 18:22:47 +00:00
2196ac112d re: unit definitions are parsed at stage load, not on wave spawn
Counted the definition objects at three points in Stage 02 -- just after
take-off, ~12 min into combat, and after GAME OVER: 14 objects, same 14 IDs
each time. Capturing a stage therefore costs one load and one snapshot; the
mission does not have to be played or survived.
2026-07-29 17:31:45 +00:00
2fa4c33d1a re: solve the runtime Unit (craft/vessel) struct from live guest memory
Craft stats were the last parked Route-B target: the Hangar exposes only a
weight class, and in menus the flight-model object is not instantiated at all.
It is instantiated in-mission, so this reads it there.

The definition class is discovered rather than assumed (unit_discover.py):
scanning for pointers to the disc ID strings and tallying the word behind each
pointer site picks out vtable 0x820af844 -- one object per unit ID, name-record
pointer at +0x04, string at +0x10, exactly the Weapon shape. The neighbouring
class 0x820af030 is the spawned entity, not the definition; the two are told
apart by one-object-per-ID and by 14/14 objects being byte-identical across two
snapshots 12 minutes and one firefight apart.

22 fields bind with zero contradictions on >=3 distinct values. Beyond that,
the Maneuver sub-record turns out to be laid out in schema declaration order,
4 bytes per field, base 0x9c with a two-slot gap after AA_Roll_Min -- 29
independently solved anchors fit two exact bases with no conflicts. That rule
pins five fields NO unit table ever values (PitchDragFactor, RollDragFactor,
DragFactorThreshold, ArterBurner_Acc, DecPitchFactor); PitchDrag == RollDrag ==
the disc's YawDrag on 18/18 units corroborates the interpolation.

Angles are float32 radians at runtime and degrees on disc.

Coverage is 18 of 110 units: unlike weapons, unit definitions are instantiated
per stage, so it grows by visiting missions. The AI-behaviour tail of Maneuver
is explicitly NOT resolved and marked tentative in the CSV.

Captured from Xenia Canary (BASIC CONTROLS tutorial + Stage 02 from save 01).
2026-07-29 17:27:53 +00:00
6a41525c41 re: generalize the two capture tools beyond weapons
idxd_tokens took the sub-record type list as an argument already but still
hard-filtered entries to those declaring a `Weapon_*` id, so it could not be
pointed at another schema. Filter on the requested type names instead -- it now
splits `EnumUnit` (Generic/Maneuver/Shield/Explosion/Frame/Turret/...) too.

gmem: honour $GMEM_FILE. `cp --sparse=always /dev/shm/xenia_memory_* snap.bin`
takes ~2 s and reads identically, which matters because a running Canary pegs
every core under lavapipe and makes repeated live reads stall unpredictably.

Groundwork for the craft (UNIT) stats. Not a finding yet: in menus only the
player craft's *name* is resident -- the flight-model object is not instantiated
until a mission loads, so that needs an in-mission snapshot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 06:05:13 +00:00
9daac6f1f0 re: read the defaulted weapon stats out of live guest memory
Canary backs the whole guest address space with one shared-memory file, so the
game's RAM is readable from the host while it runs -- no debugger, no emulator
patch. That turns the parked Route-B question (fields the IDXD omits because
they sit at their default) from "unreachable" into a table lookup.

gmem.py maps guest VAs into /dev/shm/xenia_memory_* via Xenia's fixed table and
searches only the allocated extents, so a full-RAM scan is ~0.2 s.

weapon_runtime.py finds the parsed objects by scanning for their vtable, then
*solves* the struct layout instead of guessing it: every (field, offset,
encoding) triple is scored against the disc records, and a binding is accepted
only with zero contradictions -- and marked confirmed only when >=10 records
agree on >=3 distinct values, because a field whose samples are all one number
matches any offset holding that constant.

Result: Weapon (0xc0) and Shell (0x200) mapped, 4393 values the disc does not
carry, for all 126 weapons at 5 % save progress. Cross-checks hold -- the
Arsenal DATA SHEET read 4 for wep_05/wep_60 Max. Lock Ons and the memory read
agrees; the in-flight HUD's 06000/00300 match LoadingCount; all 252 objects are
byte-identical across a screen change, so this is definition data, not state.

Two findings fell out: `IsCharging = Yes` switches LoadingCount and
TriggerShotCount to float32 (it partitions the 8 float-encoded records exactly),
and two .tbl entries declare Shell_TCAF_Ship_AAGun with conflicting Power -- the
runtime keeps the one that omits it.

Where the defaulted values *come from* (the IDXD's undecoded binary node region
vs code defaults) is not settled here; they vary per weapon, so they are not one
constructor constant.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 05:29:34 +00:00
e3f3d55d29 re: ui layout generalizes — the title main menu rebuilds too
Second screen, independent of the pause work: GP_TITLE.pak's ptbtn01..05.rat give
X=542 for all five items and Y=162/242/322/402/482, an 80px pitch where the pause
menu used 70. Measured against the main-menu screenshot the sprite tops sit at a
constant +46px for all five, which is exactly the 45px of Xenia window chrome plus
one — so the record's Y is the sprite's top edge in the guest framebuffer, to the
pixel, on a screen the format was not derived from.

GP_TITLE also splits by sub-screen the way the pause pak splits by context:
ptbtn00 (PRESS A BUTTON), ptbtn01..05 (main menu), ptbtn11..13 (EXTRAS submenu),
pgloading_* (loading screen).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 19:44:29 +00:00
9a86a09f8d re: ui — the RATC bundle header is the screen's element declaration table
u32 count at 0x14, then fixed 60-byte entries of [name | 4 flag words | pivotX |
pivotY | 0]. The table lists both .t32 sprites and .rat records, so it is the
screen's element list.

Verified: for all 7 .t32 entries in the tutorial pause bundle the declared pivot
is exactly half the decoded texture's dimensions, 7/7 with no mismatch. That also
settles what the pair at 0x50/0x54 of a .rat record is — a .rat simply opens with
one of these declarations.

The language split is visible inside one bundle: the English build's .t32
declarations carry correct English pivots while its .rat declarations carry
Japanese-derived ones, so the sprite table is regenerated per language and the
layout layer is inherited from the Japanese master.

Adds tools/re-capture/ratc_decls.py (parse the table, check pivots against the
decoded textures).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 19:40:57 +00:00
aaa4ea60b3 re: ui .rat — pivot field, language-independence, PRMD, and a tightened claim
Follow-up decoding on the layout record:

- 0x50/0x54 is the sprite PIVOT: exactly half the texture size on 4 of 5 plain
  sprites. It is a rotation/scale centre, not a draw offset — X/Y remains the
  top-left, which is what actually reproduces the screenshot.
- The records are byte-identical across the English and Japanese bundles: layout
  is authored once and only the sprites are swapped. That explains pgpbtn01's
  pivot implying a 226px texture that neither shipped language has.
- loop1.rat is NOT the screen composition — it is a looping sprite animation
  (3-name table + ~30 keyframes at one position). The eff*/deli*/msg draw list is
  therefore still unfound, and that gap is now stated plainly.
- The 380-byte top-level entries are PRMD primitives: a colour plus four explicit
  corner coordinates for a full-screen quad — the pause dim overlay. The format
  is tag-driven ('opt ', 'PRMD', 'end '), not a fixed struct.

Also demotes one piece of evidence. The screenshot is of the TUTORIAL pause menu,
so only pgp_ttrl_* can be checked against it; the in-mission records mapping onto
the same screenshot under a constant offset works only because both builds share
the 70px pitch and proves nothing. In-mission coordinates are now marked
unverified.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 19:38:04 +00:00
621b9dd6e9 re: crack the UI layout record — the retail UI rebuilds from the disc
Each UI sprite <name>.t32 ships with a <name>.rat companion, and that companion
is the layout record: big-endian u32, a 1280x720 design space at 0x18, and a
placement block of scaleX/scaleY/tint/X/Y. Animated elements repeat the block
with a trailing time field (keyframes); an 'opt ' tag carries a length-prefixed
link to the focused-state record.

Verified in that order, records first: across the four pause buttons exactly one
field varies, stepping 268/338/408/478 at a constant 70px pitch with X fixed at
226; the three items visible in a screenshot of the running game measure
346.0/415.5/485.5, mapping onto those numbers under one constant offset to
within 0.5px. The tutorial build's records are outright framebuffer coordinates,
and compositing its sprites there reproduces the screenshot pixel-accurately.

Also records two mistakes the A/B caught, since a static-only reading would have
shipped both: texture width does not identify a language (Japanese fits English
widths), and the in-mission and tutorial pause menus are separate sprite sets
rather than one set re-packed into slots.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 19:05:21 +00:00
07eff85819 re: in-flight HUD — independent confirmation of the ammo mapping
The tutorials need no save and are ~1 min from a cold boot, so they are the
cheapest route into a live flight scene. The HUD prints the equipped weapons as
NOSE BM 06000 / MAIN MPM 00300 — matching wep_01's LoadingCount 6000 and wep_02's
300, with the weapons identified from the HANGAR loadout rather than from the
ammo number. That makes it a genuinely independent confirmation of
Ammo Capacity == LoadingCount, from a different renderer in a different mode.

Also inventories the HUD elements (heat gauges, shield/armor pools, target armor
gauge, per-weapon RANGE markers) and notes that the RANGE gauge draws each
weapon's MaximumRange on a scale — a route to a real number where the Arsenal
panel only gives a letter class.

Adds autopilot.py (waypoint-arrow chaser). It detects reliably but oscillates;
recorded as such rather than as a working tool.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:50:46 +00:00
c3b5f3f401 re: state the real (non-circular) evidence for Ammo Capacity == LoadingCount
The 'N/N exact matches' framing was circular: each UI entry was identified BY its
ammo value. Replaced with what actually carries the claim — tab-uniqueness of the
forced match, the independent Max. Lock Ons agreement on two of them, and one
entry (STILETTO BG I) identified from the HANGAR loadout rather than from ammo.
Also downgrades LIGHT MACHINE GUN MG I to PROBABLE: three guns share ammo 3000.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:09:03 +00:00
ba33c533da re: runtime DATA SHEET reads the live weapon record (Route B, first capture)
Drove the retail game headless (Canary + lavapipe, vgamepad) to the Ready Room
Arsenal and read the Gallery-Mode DATA SHEET for every weapon the 5%-progress
save reveals.

Two UI->field mappings are CONFIRMED against weapons whose values ARE on disc:
  Ammo Capacity  == LoadingCount      (9/9 exact matches)
  Max. Lock Ons  == TriggerShotCount  (FALCON 9AM 12 == wep_02; BUZZARD 22 == wep_04)

That makes the panel an oracle for fields the disc leaves defaulted, and it is
shown for undeveloped weapons too, so no points need spending. Recovered
TriggerShotCount for wep_05 and wep_60 (both 4, on-disc absent), and bracketed
the defaulted Power / MaximumRange via the Range/Damage letter classes.

Ruled out first: the defaults are not in hidden/DefTables.pak (no WEAPON-schema
objects there) nor in the localized GP_MAIN_GAME_* duplicates.

Also adds the two analysis examples that produce the shopping list of defaulted
fields, the screenshot harness used to drive the menus, and the evidence PNGs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:01:25 +00:00
MechaCat02
c067761e56 docs: handoff — static ship placement exact, capture-verified (2026-07-26)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 19:54:55 +02:00
MechaCat02
ea32e77e45 ship: engine placement CONFIRMED by 43-instance capture; exhaust markers added
capture_verify example: clusters every part's ship-relative transform across
the new multi-snapshot, multi-instance F10 captures (5 snapshots, 43 e106
instances, all angles). Verdict: the dominant clusters match the static
assembly to ~1 unit on EVERY part — including BOTH engine nacelles at
(+-131, -133, -131) — so the "engines inside the hull" appearance is the
game's own placement: the engine geometry sits recessed in the aft hull, and
the visible "thrusters" in-game are exhaust FX drawn at the GN_Jet/GN_SJet
frames (Z ~ -570, past the stern).

To close that perception gap the viewer now draws simple exhaust cones at the
game's own jet frames (ship::exhaust_frames; part of the external-parts
toggle). The jet frames flip Z, so the cone apex is authored at +Z and lands
trailing aft — verified in the offline render.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 19:44:12 +02:00
MechaCat02
31f2637e6c viewer,ship: external parts ON by default + Ships catalog = composite ships only
Fixes the two things the user saw in the Ships browser:
- "Can't find the thrusters": show_external defaulted OFF, hiding the engine
  rig / bridge / turrets entirely. Externals are EXACT since the node-TRS fix,
  so the toggle now defaults ON (relabelled; off = bare hull bodies).
- "Some parts very far away": families without a composite scene graph (fighter
  morph sets like f002 whose detached parts are authored far from the origin,
  shared turret/prop part families) fell into the stack-at-origin fallback and
  rendered as piles with stray pieces. ShipModel now carries has_composite and
  the catalog lists only real assembled ships.

Offline audit across ALL stages (ship_audit example): the only far-away
outliers were composite-less f002 morph parts (now filtered); every capital
ship assembles coherently (e108 places its 7 turret instances, f105 its
mirrored shield generators). All primary composites are single-key tracks —
multikey exists only in animation composites (e901 attacks, missile-open),
which assemble_ship does not select.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 19:15:34 +02:00
MechaCat02
737b61242e ship: STATIC assembly is now EXACT — node-TRS off-by-one cracked via the capture
The runtime capture served its true purpose: as the answer key that exposed the
static encoding. The XBG7 node joint table (node+0x44) holds NINE keyframe
channels [TX TY TZ RY RX RZ SX SY SZ] behind EIGHT pointer slots shifted one
track forward: channel k+1 = f64@(ptr[k]+8), and channel 0 -- TX, which no
pointer names -- sits at ptr[0]-0x48 (tracks are 0x50-byte records, value at
+8). The old 8-slot read took TY as X, RY as Y (usually 0 -- why both hulls
stacked on the centreline) and never saw TX at all (the +-264 hull offsets).
Euler order is Ry(ch3)*Rx(ch4)*Rz(ch5) with angles applied directly, pinned by
the captured engine nacelle Rx(-15)*Rz(30) decomposition. mesh.rs gains
read_trs9/node_rotation; scene_world_nodes and node_transforms both fixed
(saber fin overrides unaffected).

assemble_ship rewritten fully static and exact:
- every composite-node instance placed (alias duplicates collapsed, real
  instances kept) including cross-id turret mounts (rou_e303_wep_01_root x2)
- the e_rou_<id>_eng cluster rig mounts at GN_Engine_01 (two mirrored nacelles
  +-131 with -+30-degree cant + centre engine) -- world = frame o rig_local
- brg/sld at their exact GN frames (frames were always exact)
- shared-geometry twin pairs at +-TX: X-reflect the instance whose offset
  opposes the geometry's dominant side (viewer reverses winding on det<0)
- squeezed node names resolve (wep02 -> wep_02_01)

Ground-truth gate: ship::tests::static_assembly_matches_runtime_capture asserts
static == the baked e106 capture for all 8 parts (T<1.0, R<0.02) plus both
nacelles, both turrets, and the starboard-hull mirror. Viewer now uses pure
static assembly; ship_capture + the baked table remain as the verification
oracle. Renders show a coherent, bilaterally-symmetric destroyer.

81 formats-lib + all disc tests green; viewer builds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 19:00:49 +02:00
MechaCat02
015e49ac8a ship: diagnostics — baked-table renderer + evidence the placement IS static
ship_render example: assemble a ship from the baked capture table (or --static)
and write orthographic top/side PPM renders + per-part world bounds — the
offline eye for placement bugs.

Findings recorded (docs to follow with the fix):
- e106 baked eng_01 is CROSS-INSTANCE contamination: the F10 capture de-dups by
  vertex-buffer address alone, so for a part drawn by several fleet ships only
  the FIRST instance's transform survives; eng_01's 30-degree rotation and
  below-hull position belong to a different destroyer. Fleet formation made it
  reproduce across captures, defeating the cross-validation.
- The static composite DOES carry the exact placement: BE-f64 +-264.0 (the
  lateral hull pair offset the static assembler loses) and -164.044 (the bridge
  Z we captured as -164.037) sit in e_rou_e106's node joint tables. The static
  decode is incomplete, not the data — full static assembly is achievable, with
  captures demoted to a verification oracle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 18:39:34 +02:00
MechaCat02
d9442a2106 ship_capture: bake e106 COMPLETE from the 2026-07-26 F10 capture — all 8 parts
The second F10 capture (ship at _m LOD distance) closes e106 entirely: all 8
parts placed including the bridge both earlier captures missed, cross-validating
the doc's 7 known translations to 0.1 units and adding brg_01 at the exact
centreline (0, 153.3, -164.0).

Matching hardened by what the real capture taught us:
- LOD-aware: each part tries every variant vcount (base/_m/_l/_d) — a distant
  ship draws its LOD copies, same local frame.
- Position-validated, SET-based, against the UNION of a part's variants: buffer
  order != decode order, and one draw's vcount equalled the _m count while its
  buffer held the FULL 1633-vert geometry. A coincidental vcount (foreign
  51-vert mesh vs the bridge) is rejected by geometry.
- Runtime mirror handled: twins share one file geometry; the engine uploads the
  starboard copy X-reflected. Mirror-validated draws bake diag(-1,1,1) and the
  viewer reverses triangle winding for det<0 so front faces stay outward.
- Near-axis rotation entries snapped to exact 0/+-1 for a clean table; eng_01
  keeps its genuine 30-degree nacelle rotation.

data/ship_placements.txt now ships e106 (viewer renders via the captured table);
embedded_e106_is_complete locks the baked data. part_pos + capture_match helper
examples added. 10 ship_capture tests; 80 formats-lib tests green; viewer builds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 18:26:23 +02:00
MechaCat02
4efc0278b1 ship_capture: ingest the draw-logger format + quantify the static gaps
The correlator now accepts BOTH capture formats: the F10 ship-capture snapshot
and the draw-logger format (mission_draws.log / xenia_re_draws.log) via
parse_drawlog — group the ship shader's draws by vertex-buffer base, vcount =
size_words/stride_words, keep the first c0..c2 WVP. So a capital ship seen in a
normal instrumented run bakes without a dedicated F10 pass. correlate_capture
auto-detects the format. Shared normalize_wvp between both parsers.

Also add examples/ship_dump.rs (offline static-placement inspector) and record
the measured static-assembler gaps for e106 in the doc: bdy_01/bdy_02 overlap
(runtime port/starboard mirror at X=+-264), eng_02 and wep_02_01 never placed.
These confirm exact placement is runtime-only — nothing more to squeeze
statically; the capture-driven table is the only path. The draw logs present on
this box are the player fighter, so no capital-ship table can be baked offline.

7 ship_capture tests (adds draw-log parse + correlate); 77 formats-lib green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:57:08 +02:00
MechaCat02
c6ca5ce9e7 ship: bake pipeline — correlator as a tested module + viewer prefers captured table
Promote the F10 ship-capture correlator from an example into a reusable, unit-
tested library module (ship_capture): parse_capture (log -> per-draw rigid
WorldView from the c0..c2 WVP constants), correlate (match parts to draws by
vertex count, express each in the reference part's frame), and a checked-in
placement-table format (serialize_table/parse_table, embedded via
embedded_placement from data/ship_placements.txt).

build_ship_model now prefers a ship's captured placement over the static
assemble_ship when a table entry exists (empty table -> unchanged fallback).
correlate_capture example refactored onto the module + gains --emit to print a
table block. Doc updated: bake plumbing done; remaining is running real F10
captures to populate the table (needs the emulator).

5 new ship_capture tests (parse/normalize/correlate/table round-trip); 76
formats-lib tests + viewer build green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:48:00 +02:00
117 changed files with 21572 additions and 268 deletions

1
.gitignore vendored
View File

@@ -20,3 +20,4 @@ Thumbs.db
# Trunk build output
dist/
__pycache__/

View File

@@ -0,0 +1,21 @@
# Capital-ship part placements — runtime-captured ground truth (see
# docs/re/ship-placement-runtime-capture.md and src/ship_capture.rs).
#
# Populate with:
# SYLPHEED_ISO=... cargo run --release --example correlate_capture -- \
# <capture.log> <Stage_SNN> <ship_id> [ref_part_substr] --emit
# then paste the emitted `ship …` block(s) below. The viewer prefers a ship's
# entry here over the static assembler when present.
#
# Per placement line: <part> <R00 R01 R02 R10 R11 R12 R20 R21 R22> <T0 T1 T2>
#
ship e106 ref=e106_bdy_04
e106_bdy_01 1 0 0 0 1 0 0 0 1 -263.9948 -150.43242 1164.8667
e106_bdy_02 -1 0 0 0 1 0 0 0 1 264.0088 -150.41339 1164.8651
e106_bdy_03 1 0 0 0 1 0 0 0 1 0.0029247368 -34.517372 1075.878
e106_bdy_04 1 0 0 0 1 0 0 0 1 0 0 0
e106_brg_01 1 0 0 0 1 0 0 0 1 -0.005471501 153.31795 -164.03748
e106_eng_01 0.8659965 -0.50002277 0 0.4829627 0.83651507 0.25885975 -0.12941553 -0.22417325 0.96592265 131.2386 -133.06625 -132.06075
e106_eng_02 1 0 0 0 1 0 0 0 1 0.0013684053 -49.088192 -139.47827
e106_wep_02_01 1 0 0 0 1 0 0 0 1 0.000040663195 49.126797 1009.06744

View File

@@ -0,0 +1,43 @@
//! Identify which stage + ship a capture log came from: match the capture's
//! draw vertex counts against every resource (incl. LODs) of every Stage_SNN.xpr.
//! cargo run --release --example capture_match -- <capture.log> <resource3d dir>
use sylpheed_formats::mesh::{xbg7_resource_names, Xbg7Model};
use sylpheed_formats::ship_capture::{parse_capture, parse_drawlog};
use std::collections::{BTreeMap, HashSet};
fn main() {
let a: Vec<String> = std::env::args().collect();
let text = std::fs::read_to_string(&a[1]).unwrap();
let mut draws = parse_capture(&text);
if draws.is_empty() { draws = parse_drawlog(&text); }
let caps: HashSet<u32> = draws.iter().map(|d| d.vcount).filter(|v| *v >= 100).collect();
eprintln!("{} draws, {} distinct vcounts>=100", draws.len(), caps.len());
let mut entries: Vec<_> = std::fs::read_dir(&a[2]).unwrap().filter_map(|e| e.ok())
.filter(|e| { let n = e.file_name().to_string_lossy().to_string(); n.starts_with("Stage_") && n.ends_with(".xpr") })
.map(|e| e.path()).collect();
entries.sort();
for path in entries {
let bytes = std::fs::read(&path).unwrap();
let names = xbg7_resource_names(&bytes);
let want: HashSet<String> = names.iter().cloned().collect();
let models = Xbg7Model::models_named(&bytes, &want, &|| false);
// resource -> vcount; group hits by ship-id prefix
let mut hits: BTreeMap<String, Vec<(String, u32)>> = BTreeMap::new();
for m in &models {
let vc: usize = m.meshes.iter().map(|s| s.positions.len()).sum();
if vc >= 100 && caps.contains(&(vc as u32)) {
let id = m.name.get(..4).unwrap_or("?").to_string();
hits.entry(id).or_default().push((m.name.clone(), vc as u32));
}
}
let total: usize = hits.values().map(|v| v.len()).sum();
if total >= 3 {
println!("== {} : {} matching resources ==", path.file_name().unwrap().to_string_lossy(), total);
for (id, v) in &hits {
let s: Vec<String> = v.iter().map(|(n, c)| format!("{n}({c})")).collect();
println!(" {id}: {}", s.join(" "));
}
}
}
}

View File

@@ -0,0 +1,192 @@
//! Verify static ship assembly against multi-snapshot, multi-instance F10
//! captures: cluster every part's ship-relative transform across ALL ships in
//! ALL snapshots, and compare the dominant clusters with `assemble_ship`.
//!
//! cargo run --release --example capture_verify -- <stage.xpr> <ship_id> <log> [log…]
//!
//! Every draw is position-validated against the ship's parts (any LOD, set
//! based). Each `bdy_04`(-LOD) draw acts as a ship reference; every validated
//! part draw within ship range contributes `rel = WV_ref⁻¹ ∘ WV_part`. The true
//! per-part placements appear as tight clusters repeated once PER SHIP per
//! snapshot; cross-ship pairings scatter and stay below the cluster threshold.
use std::collections::HashSet;
use sylpheed_formats::mesh::{xbg7_resource_names, Xbg7Model};
use sylpheed_formats::ship::{assemble_ship, is_base_part, ship_id_of};
use sylpheed_formats::ship_capture::{parse_capture, CapturedDraw};
type M3 = [[f64; 3]; 3];
fn transpose(m: &M3) -> M3 {
[
[m[0][0], m[1][0], m[2][0]],
[m[0][1], m[1][1], m[2][1]],
[m[0][2], m[1][2], m[2][2]],
]
}
fn mmul(a: &M3, b: &M3) -> M3 {
let mut o = [[0.0; 3]; 3];
for i in 0..3 {
for j in 0..3 {
o[i][j] = (0..3).map(|k| a[i][k] * b[k][j]).sum();
}
}
o
}
fn mv(m: &M3, v: [f64; 3]) -> [f64; 3] {
[
m[0][0] * v[0] + m[0][1] * v[1] + m[0][2] * v[2],
m[1][0] * v[0] + m[1][1] * v[1] + m[1][2] * v[2],
m[2][0] * v[0] + m[2][1] * v[1] + m[2][2] * v[2],
]
}
fn main() {
let a: Vec<String> = std::env::args().collect();
let bytes = std::fs::read(&a[1]).unwrap();
let id = &a[2];
// Part position sets (base + LODs share the local frame) + per-variant vcounts.
let names = xbg7_resource_names(&bytes);
let base_parts: Vec<String> = names
.iter()
.filter(|n| is_base_part(n) && ship_id_of(n) == Some(id.as_str()))
.cloned()
.collect();
let mut want: HashSet<String> = base_parts.iter().cloned().collect();
for p in &base_parts {
for suf in ["_m", "_l", "_d"] {
let c = format!("{p}{suf}");
if names.contains(&c) {
want.insert(c);
}
}
}
let models = Xbg7Model::models_named(&bytes, &want, &|| false);
struct Part {
base: String,
vcounts: Vec<u32>,
pos: Vec<[f32; 3]>, // union of variants
}
let mut parts: Vec<Part> = Vec::new();
for p in &base_parts {
let mut vcounts = Vec::new();
let mut pos = Vec::new();
for cand in [p.clone(), format!("{p}_m"), format!("{p}_l"), format!("{p}_d")] {
if let Some(m) = models.iter().find(|m| m.name == cand) {
let mp: Vec<[f32; 3]> =
m.meshes.iter().flat_map(|s| s.positions.iter().copied()).collect();
vcounts.push(mp.len() as u32);
pos.extend(mp);
}
}
parts.push(Part { base: p.clone(), vcounts, pos });
}
// Validate a draw against a part (direct or X-mirrored), like ship_capture.
let validate = |d: &CapturedDraw, p: &Part| -> Option<bool> {
if !p.vcounts.contains(&d.vcount) || d.pos.is_empty() {
return None;
}
let near = |a: &[f32; 3], b: &[f32; 3]| {
(a[0] - b[0]).abs() <= 1e-2 && (a[1] - b[1]).abs() <= 1e-2 && (a[2] - b[2]).abs() <= 1e-2
};
let (mut direct, mut mirror) = (0usize, 0usize);
for q in &d.pos {
if p.pos.iter().any(|v| near(q, v)) {
direct += 1;
}
let qm = [-q[0], q[1], q[2]];
if p.pos.iter().any(|v| near(&qm, v)) {
mirror += 1;
}
}
let need = d.pos.len().div_ceil(2);
if direct >= need && direct >= mirror {
Some(false)
} else if mirror >= need {
Some(true)
} else {
None
}
};
// Collect rel samples per part across all logs.
let mut samples: std::collections::BTreeMap<String, Vec<[f64; 3]>> = Default::default();
for log in &a[3..] {
let text = std::fs::read_to_string(log).unwrap();
let draws = parse_capture(&text);
// label validated draws
let mut labeled: Vec<(usize, usize, bool)> = Vec::new(); // (draw, part, mirrored)
for (di, d) in draws.iter().enumerate() {
for (pi, p) in parts.iter().enumerate() {
if let Some(mir) = validate(d, p) {
labeled.push((di, pi, mir));
break;
}
}
}
let refs: Vec<usize> = labeled
.iter()
.filter(|(_, pi, _)| parts[*pi].base.ends_with("bdy_04"))
.map(|(di, ..)| *di)
.collect();
for &ri in &refs {
let rf = &draws[ri];
let rt = transpose(&rf.r);
for &(di, pi, _mir) in &labeled {
let d = &draws[di];
let dt = [d.t[0] - rf.t[0], d.t[1] - rf.t[1], d.t[2] - rf.t[2]];
let rel = mv(&rt, dt);
if rel.iter().map(|v| v * v).sum::<f64>().sqrt() < 2600.0 {
// also require the rel rotation be finite-sane
let _rm = mmul(&rt, &d.r);
samples.entry(parts[pi].base.clone()).or_default().push(rel);
}
}
}
eprintln!("{log}: {} validated draws, {} bdy_04 refs", labeled.len(), refs.len());
}
// Cluster per part (greedy, 8-unit radius), print clusters with ≥3 samples.
println!("\n== dominant ship-relative placements (clusters ≥3 samples) ==");
for (part, mut v) in samples {
v.sort_by(|x, y| x[0].partial_cmp(&y[0]).unwrap());
let mut clusters: Vec<([f64; 3], usize)> = Vec::new();
for s in &v {
match clusters.iter_mut().find(|(c, _)| {
(c[0] - s[0]).abs() < 8.0 && (c[1] - s[1]).abs() < 8.0 && (c[2] - s[2]).abs() < 8.0
}) {
Some((c, n)) => {
for k in 0..3 {
c[k] = (c[k] * *n as f64 + s[k]) / (*n as f64 + 1.0);
}
*n += 1;
}
None => clusters.push((*s, 1)),
}
}
clusters.sort_by(|x, y| y.1.cmp(&x.1));
let tops: Vec<String> = clusters
.iter()
.filter(|(_, n)| *n >= 3)
.take(6)
.map(|(c, n)| format!("[{:7.1} {:7.1} {:7.1}]×{n}", c[0], c[1], c[2]))
.collect();
println!(" {part:20} {}", tops.join(" "));
}
// Static assembly rel bdy_04 for comparison.
println!("\n== static assemble_ship rel bdy_04 ==");
let placed = assemble_ship(&bytes, id, true);
if let Some(r4) = placed.iter().find(|p| p.resource == format!("{id}_bdy_04")) {
for p in &placed {
println!(
" {:20} [{:7.1} {:7.1} {:7.1}]",
p.resource,
p.t[0] - r4.t[0],
p.t[1] - r4.t[1],
p.t[2] - r4.t[2]
);
}
}
}

View File

@@ -1,139 +1,62 @@
//! Recover exact capital-ship part placement from a Canary `xenia_ship_capture.log`
//! (F10 snapshot: per-draw guest vertex-buffer + up to 48 vertex-shader float
//! constants).
//! Recover exact capital-ship part placement from a Canary capture log
//! and (optionally) emit a checked-in placement-table block.
//!
//! FINDING (2026-07-24): the captured vertex BUFFER holds LOCAL coordinates —
//! byte-identical to the `.xpr` — so capital-ship parts are placed entirely in the
//! **vertex shader**. The per-part transform is in the VS constants: `c0..c2` are
//! the **WorldViewProjection** matrix rows. Their norms are `(sx, sy, 1)` = the
//! projection x/y scales (sy/sx = 16:9 aspect); dividing each row by its norm
//! yields the rigid **WorldView** (verified: rows orthonormal, det +1). The camera
//! View cancels when we express every part relative to a reference part:
//! rel_p = WV_ref⁻¹ · WV_p (a pure ship-space rigid transform)
//! Applying `rel_p` to part `p`'s local geometry assembles the ship exactly, in the
//! reference part's frame — ground truth, no static guessing.
//! The correlation math lives in [`sylpheed_formats::ship_capture`]; this example
//! is the CLI wrapper: it reads the log, decodes the ship's parts from the disc to
//! get their vertex counts + leading positions (the match keys), correlates, and
//! prints the result. With `--emit` it prints the `ship …` block to paste into
//! `crates/sylpheed-formats/data/ship_placements.txt`.
//!
//! Matching is **LOD-aware**: a ship on screen at distance is drawn with its
//! `_m`/`_l` LOD copies, which live in the same local frame as the base part — so
//! each base part tries its own vcount first, then its LOD variants', and the
//! recovered transform is recorded under the base name. Same-vcount twins
//! (mirrored port/starboard hulls) are routed by the draw's position dump.
//!
//! Usage:
//! SYLPHEED_ISO=... cargo run --release --example correlate_capture -- \
//! <capture.log> <Stage_SNN> <ship_id> [ref_part_substr]
//! e.g. `... /tmp/xenia_ship_capture.log Stage_S01 e106 bdy_04`
//! <capture.log> <Stage_SNN> <ship_id> [ref_part_substr] [--emit]
//! e.g. SYLPHEED_ISO="/home/fabi/RE Project Sylpheed/Project Sylpheed - Arc of
//! Deception (USA, Europe) (En,Ja).iso" \
//! cargo run --release --example correlate_capture -- \
//! xenia_ship_capture.log Stage_S01 e106 bdy_04 --emit
use sylpheed_formats::mesh::{xbg7_resource_names, Xbg7Model};
use sylpheed_formats::ship::{is_base_part, ship_id_of};
use sylpheed_formats::ship_capture::{
correlate, parse_capture, parse_drawlog, serialize_table, PartKey,
};
use sylpheed_formats::xiso::open_iso;
use std::collections::HashSet;
use std::path::Path;
type M3 = [[f64; 3]; 3];
struct Draw {
vbase: u32,
vcount: u32,
/// Rigid WorldView: R (rows) + T (view-space translation).
r: M3,
t: [f64; 3],
ok: bool,
}
fn parse(text: &str) -> Vec<Draw> {
let mut out = Vec::new();
let mut vbase = 0u32;
let mut vcount = 0u32;
let mut consts: Vec<(usize, [f64; 4])> = Vec::new();
let mut flush = |vbase: u32, vcount: u32, consts: &[(usize, [f64; 4])], out: &mut Vec<Draw>| {
if vbase == 0 {
return;
}
// c0..c2 = WVP rows; normalise each to unit → rigid WorldView.
let get = |i: usize| consts.iter().find(|(k, _)| *k == i).map(|(_, v)| *v);
let (Some(c0), Some(c1), Some(c2)) = (get(0), get(1), get(2)) else {
out.push(Draw { vbase, vcount, r: [[0.0; 3]; 3], t: [0.0; 3], ok: false });
return;
};
let rows = [c0, c1, c2];
let mut r = [[0.0; 3]; 3];
let mut t = [0.0; 3];
let mut ok = true;
for (i, row) in rows.iter().enumerate() {
let n = (row[0] * row[0] + row[1] * row[1] + row[2] * row[2]).sqrt();
if n < 1e-6 {
ok = false;
break;
}
r[i] = [row[0] / n, row[1] / n, row[2] / n];
t[i] = row[3] / n;
}
out.push(Draw { vbase, vcount, r, t, ok });
};
for line in text.lines() {
let l = line.trim();
if let Some(rest) = l.strip_prefix("DRAW ") {
flush(vbase, vcount, &consts, &mut out);
consts.clear();
let f = |k: &str| rest.split_whitespace().find_map(|t| t.strip_prefix(k));
vbase = f("vbase=0x").and_then(|s| u32::from_str_radix(s, 16).ok()).unwrap_or(0);
vcount = f("vcount=").and_then(|s| s.parse().ok()).unwrap_or(0);
} else if l.starts_with("vsconst") {
for cap in l.split("c").skip(1) {
// "<i>=(x,y,z,w) ..."
if let Some((idx, rest)) = cap.split_once('=') {
if let Ok(i) = idx.trim().parse::<usize>() {
let nums: Vec<f64> = rest
.trim_start_matches('(')
.split(')')
.next()
.unwrap_or("")
.split(',')
.filter_map(|x| x.trim().parse().ok())
.collect();
if nums.len() == 4 {
consts.push((i, [nums[0], nums[1], nums[2], nums[3]]));
}
}
}
}
}
}
flush(vbase, vcount, &consts, &mut out);
out
}
fn mt_apply(r: &M3, t: &[f64; 3], v: [f64; 3]) -> [f64; 3] {
[
r[0][0] * v[0] + r[0][1] * v[1] + r[0][2] * v[2] + t[0],
r[1][0] * v[0] + r[1][1] * v[1] + r[1][2] * v[2] + t[1],
r[2][0] * v[0] + r[2][1] * v[1] + r[2][2] * v[2] + t[2],
]
}
fn transpose(m: &M3) -> M3 {
[
[m[0][0], m[1][0], m[2][0]],
[m[0][1], m[1][1], m[2][1]],
[m[0][2], m[1][2], m[2][2]],
]
}
fn mmul(a: &M3, b: &M3) -> M3 {
let mut o = [[0.0; 3]; 3];
for i in 0..3 {
for j in 0..3 {
o[i][j] = (0..3).map(|k| a[i][k] * b[k][j]).sum();
}
}
o
}
fn main() {
let args: Vec<String> = std::env::args().collect();
if args.len() < 4 {
eprintln!("usage: correlate_capture <capture.log> <Stage_SNN> <ship_id> [ref_part_substr]");
let positional: Vec<&String> = args[1..].iter().filter(|a| !a.starts_with("--")).collect();
let emit = args.iter().any(|a| a == "--emit");
if positional.len() < 3 {
eprintln!(
"usage: correlate_capture <capture.log> <Stage_SNN> <ship_id> [ref_part_substr] [--emit]"
);
std::process::exit(2);
}
let (log, stage, id) = (&args[1], &args[2], &args[3]);
let ref_sub = args.get(4).map(|s| s.as_str()).unwrap_or("bdy_04");
let (log, stage, id) = (positional[0], positional[1], positional[2]);
let ref_sub = positional.get(3).map(|s| s.as_str()).unwrap_or("bdy_04");
let iso = std::env::var("SYLPHEED_ISO").expect("SYLPHEED_ISO");
let draws = parse(&std::fs::read_to_string(log).expect("read log"));
eprintln!("parsed {} draws ({} with WorldView)", draws.len(), draws.iter().filter(|d| d.ok).count());
// Accept either the F10 ship-capture format or the draw-logger format;
// auto-detect by trying the F10 parser first.
let text = std::fs::read_to_string(log).expect("read log");
let mut draws = parse_capture(&text);
if draws.is_empty() {
draws = parse_drawlog(&text);
eprintln!("parsed {} draws (draw-logger format)", draws.len());
} else {
eprintln!("parsed {} draws (F10 capture format)", draws.len());
}
// Decode the ship's base parts AND their LOD copies (vcount + leading
// positions are the match keys; LODs share the base part's local frame).
let bytes = {
let rt = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap();
rt.block_on(async {
@@ -142,66 +65,72 @@ fn main() {
})
};
let names = xbg7_resource_names(&bytes);
let parts: Vec<String> =
names.iter().filter(|n| is_base_part(n) && ship_id_of(n) == Some(id.as_str())).cloned().collect();
let want: HashSet<String> = parts.iter().cloned().collect();
let models = Xbg7Model::models_named(&bytes, &want, &|| false);
// Match each part to a captured draw by vertex count; keep its WorldView.
struct Placed {
part: String,
r: M3,
t: [f64; 3],
local: Vec<[f64; 3]>,
}
let mut placed: Vec<Placed> = Vec::new();
let mut used: HashSet<u32> = HashSet::new();
for part in &parts {
let Some(m) = models.iter().find(|m| &m.name == part) else { continue };
let local: Vec<[f64; 3]> =
m.meshes.iter().flat_map(|s| s.positions.iter()).map(|p| [p[0] as f64, p[1] as f64, p[2] as f64]).collect();
let vc = local.len() as u32;
if let Some(d) = draws.iter().find(|d| d.ok && d.vcount == vc && !used.contains(&d.vbase)) {
used.insert(d.vbase);
placed.push(Placed { part: part.clone(), r: d.r, t: d.t, local });
} else {
println!("{part:20} (no matching captured draw — occluded/culled?)");
}
}
// Reference part → ship frame. rel_p = WV_ref⁻¹ · WV_p (camera cancels).
let Some(rf) = placed.iter().find(|p| p.part.contains(ref_sub)).or(placed.first()) else {
eprintln!("no parts placed");
return;
};
let rt_ref = transpose(&rf.r);
let ref_t = rf.t;
println!("\nreference = {} → ship-relative placement (M rows, T):", rf.part);
let mut lo = [f64::MAX; 3];
let mut hi = [f64::MIN; 3];
for p in &placed {
// rel R = Rᵀ_ref · R_p ; rel T = Rᵀ_ref · (T_p T_ref)
let rel_r = mmul(&rt_ref, &p.r);
let dt = [p.t[0] - ref_t[0], p.t[1] - ref_t[1], p.t[2] - ref_t[2]];
let rel_t = mt_apply(&rt_ref, &[0.0; 3], dt);
// Assembled centroid (validation) + overall extent.
let mut c = [0.0; 3];
for v in &p.local {
let w = mt_apply(&rel_r, &rel_t, *v);
for k in 0..3 {
c[k] += w[k];
lo[k] = lo[k].min(w[k]);
hi[k] = hi[k].max(w[k]);
let base_parts: Vec<String> = names
.iter()
.filter(|n| is_base_part(n) && ship_id_of(n) == Some(id.as_str()))
.cloned()
.collect();
// Candidate resources per base part: itself + `_m`/`_l`/`_d` LOD copies.
let mut want: HashSet<String> = base_parts.iter().cloned().collect();
for p in &base_parts {
for suf in ["_m", "_l", "_d"] {
let cand = format!("{p}{suf}");
if names.contains(&cand) {
want.insert(cand);
}
}
let n = p.local.len().max(1) as f64;
println!(
" {:18} T=[{:8.1}{:8.1}{:8.1}] centroid=[{:8.1}{:8.1}{:8.1}]",
p.part, rel_t[0], rel_t[1], rel_t[2], c[0] / n, c[1] / n, c[2] / n
);
}
println!(
"\nassembled extent = [{:.0} {:.0} {:.0}] ({} parts placed)",
hi[0] - lo[0], hi[1] - lo[1], hi[2] - lo[2], placed.len()
);
let models = Xbg7Model::models_named(&bytes, &want, &|| false);
let positions_of = |name: &str| -> Option<Vec<[f32; 3]>> {
let m = models.iter().find(|m| m.name == name)?;
Some(m.meshes.iter().flat_map(|s| s.positions.iter().copied()).collect())
};
// One PartKey per (part, variant-vcount present in the capture) — correlate
// takes the first that validates. Validation uses the UNION of the part's
// variant position sets: a draw's `vcount` can match one LOD while its
// buffer holds another variant's geometry (seen on the real e106: the
// 558-vcount draw carried the FULL 1633-vert bdy_04 buffer), and every
// variant is the same part in the same local frame.
let mut keys: Vec<PartKey> = Vec::new();
for part in &base_parts {
let variants =
[part.clone(), format!("{part}_m"), format!("{part}_l"), format!("{part}_d")];
let union: Vec<[f32; 3]> =
variants.iter().filter_map(|v| positions_of(v)).flatten().collect();
let mut any = false;
for cand in &variants {
if let Some(pos) = positions_of(cand) {
let vcount = pos.len() as u32;
if draws.iter().any(|d| d.vcount == vcount) {
let lod = if cand == part { "full" } else { cand.rsplit('_').next().unwrap_or("?") };
eprintln!(" {part:20} try vcount={vcount:6} [{lod}]");
keys.push(PartKey { part: part.clone(), vcount, ref_pos: union.clone() });
any = true;
}
}
}
if !any {
eprintln!(" {part:20} — no draw matches any LOD (culled/off-screen?)");
}
}
let Some(ship) = correlate(id, &draws, &keys, ref_sub) else {
eprintln!("no parts matched a captured draw");
return;
};
eprintln!("\nreference = {} → ship-relative placement:", ship.reference);
for p in &ship.parts {
eprintln!(" {:18} T=[{:9.1}{:9.1}{:9.1}]", p.part, p.t[0], p.t[1], p.t[2]);
}
for part in &base_parts {
if !ship.parts.iter().any(|p| &p.part == part) {
eprintln!(" {part:18} — NOT placed (culled, or its only vcount hit failed position validation)");
}
}
if emit {
print!("{}", serialize_table(std::slice::from_ref(&ship)));
}
}

View File

@@ -0,0 +1,50 @@
//! Scratch analysis: for one IDXD key, list every object that DECLARES it and
//! whether it carries a value on disc or is left at the title-code default.
//!
//! Run: cargo run -p sylpheed-formats --example default_owners -- <KEY> [KEY...]
use sylpheed_formats::idxd::IdxdObject;
use sylpheed_formats::pak::PakArchive;
fn is_number(s: &str) -> bool {
let s = s.strip_suffix(['f', 'F']).unwrap_or(s);
let t = s.strip_prefix(['-', '+']).unwrap_or(s);
!t.is_empty() && t.chars().all(|c| c.is_ascii_digit() || c == '.')
}
fn is_key(s: &str) -> bool {
!s.is_empty()
&& s.chars().all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
&& s.chars().next().is_some_and(|c| c.is_ascii_alphabetic() || c == '_')
&& !is_number(s)
}
fn is_value(s: &str) -> bool {
is_number(s) || !is_key(s)
}
fn main() {
let root = std::env::var("SYLPHEED_DISC")
.unwrap_or_else(|_| "/home/fabi/RE - Project Sylpheed/sylph_extract".into());
let wanted: Vec<String> = std::env::args().skip(1).collect();
let arc = PakArchive::open(std::path::Path::new(&root).join("dat/GP_MAIN_GAME_E.pak")).unwrap();
for e in arc.entries() {
let Ok(bytes) = arc.read(e) else { continue };
let Ok(obj) = IdxdObject::parse(&bytes) else { continue };
let toks = obj.tokens().to_vec();
let mut hits: Vec<String> = vec![];
for (i, t) in toks.iter().enumerate() {
if !wanted.iter().any(|w| w == t) {
continue;
}
let valued = i > 0 && is_value(&toks[i - 1]);
hits.push(if valued {
format!("{t}={}", toks[i - 1])
} else {
format!("{t}=<DEFAULT>")
});
}
if !hits.is_empty() {
println!("0x{:08x} {:<44} {}", obj.schema_hash, obj.identity(), hits.join(" "));
}
}
}

View File

@@ -0,0 +1,135 @@
//! Scratch analysis: which IDXD fields are DECLARED but left at their default on
//! disc, per schema. Those defaults live in title code, so they can only be read
//! from the running game — this prints the shopping list for that dynamic capture.
//!
//! Run: cargo run -p sylpheed-formats --example defaulted_fields -- <disc-root>
use std::collections::{BTreeMap, BTreeSet};
use sylpheed_formats::idxd::IdxdObject;
use sylpheed_formats::pak::PakArchive;
fn is_number(s: &str) -> bool {
let s = s.strip_suffix(['f', 'F']).unwrap_or(s);
let t = s.strip_prefix(['-', '+']).unwrap_or(s);
!t.is_empty() && t.chars().all(|c| c.is_ascii_digit() || c == '.')
}
/// Same shape-test the parser uses: an identifier-looking token that is not a
/// value literal is a field-name key.
fn is_key(s: &str) -> bool {
!s.is_empty()
&& s.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
&& s.chars().next().is_some_and(|c| c.is_ascii_alphabetic() || c == '_')
&& !is_number(s)
}
fn is_value(s: &str) -> bool {
is_number(s) || !is_key(s)
}
fn main() {
let root = std::env::args()
.nth(1)
.unwrap_or_else(|| "/home/fabi/RE - Project Sylpheed/sylph_extract".into());
let paks: Vec<String> = std::env::args().skip(2).collect();
let paks = if paks.is_empty() {
vec![
"dat/GP_MAIN_GAME_E.pak".to_string(),
"dat/GP_HANGAR_ARSENAL.pak".to_string(),
]
} else {
paks
};
for pak_rel in &paks {
let path = std::path::Path::new(&root).join(pak_rel);
let Ok(arc) = PakArchive::open(&path) else {
eprintln!("-- skip {pak_rel} (open failed)");
continue;
};
println!("\n================ {pak_rel} ================");
// schema -> key -> (n_set, n_defaulted, distinct values, example owners)
type Stat = (usize, usize, BTreeSet<String>, Vec<String>);
let mut per_schema: BTreeMap<u32, (usize, BTreeMap<String, Stat>)> = BTreeMap::new();
for e in arc.entries() {
let Ok(bytes) = arc.read(e) else { continue };
let Ok(obj) = IdxdObject::parse(&bytes) else {
continue;
};
let ident = obj.identity();
let toks = obj.tokens().to_vec();
let entry = per_schema.entry(obj.schema_hash).or_default();
entry.0 += 1;
// Track which keys this object declares, and whether each is valued.
let mut seen_here: BTreeMap<String, Option<String>> = BTreeMap::new();
for (i, t) in toks.iter().enumerate() {
if !is_key(t) {
continue;
}
let prev = if i == 0 { None } else { Some(&toks[i - 1]) };
let valued = prev.map(|p| is_value(p)).unwrap_or(false);
let v = if valued { Some(toks[i - 1].clone()) } else { None };
seen_here.entry(t.clone()).or_insert(v);
}
for (k, v) in seen_here {
let s = entry.1.entry(k).or_default();
match v {
Some(val) => {
s.0 += 1;
if s.2.len() < 12 {
s.2.insert(val);
}
}
None => {
s.1 += 1;
if s.3.len() < 6 {
s.3.push(ident.clone());
}
}
}
}
}
for (schema, (n_obj, keys)) in per_schema {
if n_obj < 2 {
continue;
}
let name = match schema {
0x0426_e81d => "PLAYER",
0x6ab4_825a => "WEAPON",
0x43fa_a517 => "UNIT",
0x3c5b_0549 => "VESSEL",
0xbd86_d41c => "CHARACTER",
0x3c9a_e32e => "STAGE",
0xb412_e6d8 => "MESSAGE",
_ => "?",
};
let defaulted: Vec<_> = keys
.iter()
.filter(|(_, s)| s.1 > 0)
.collect();
if defaulted.is_empty() {
continue;
}
println!("\n--- schema 0x{schema:08x} {name} ({n_obj} objects) ---");
println!(
"{:<30} {:>5} {:>5} {}",
"KEY", "set", "dflt", "values seen (≤12) | owners defaulting"
);
for (k, (n_set, n_def, vals, owners)) in defaulted {
let vv: Vec<&str> = vals.iter().map(|s| s.as_str()).collect();
println!(
"{:<30} {:>5} {:>5} {} | {}",
k,
n_set,
n_def,
vv.join(","),
owners.join(",")
);
}
}
}
}

View File

@@ -0,0 +1,108 @@
//! RE diagnostic: emit every `weapon\Weapon_*.tbl` in a pak as machine-readable
//! records, split at the sub-object boundaries the schema declares.
//!
//! An IDXD `.tbl` for a weapon holds `count` sub-records — a `Weapon` and its
//! `Shell` — flattened into one string pool. `sylpheed-cli pak dump` shows them
//! merged, which is ambiguous (both declare `ID`/`Name`). The title's schema
//! name pool gives the split point: the pool contains the literal type names
//! (`Weapon`, `Shell`), and each sub-record's fields follow its type name.
//!
//! Here we split on a type-name token appearing as a *key* position, so each
//! record is emitted with its own ID and field set:
//!
//! ```text
//! REC <tbl-hash> <index> <TypeName> <ID>
//! F <key> <value>
//! ```
//!
//! Run: cargo run --release -p sylpheed-formats --example idxd_tokens -- <pak> [types...]
use sylpheed_formats::idxd::IdxdObject;
use sylpheed_formats::pak::PakArchive;
fn is_number(s: &str) -> bool {
let s = s.strip_suffix(['f', 'F']).unwrap_or(s);
let t = s.strip_prefix(['-', '+']).unwrap_or(s);
!t.is_empty() && t.chars().all(|c| c.is_ascii_digit() || c == '.')
}
/// `Yes`/`No`-style scalar literals are values, not field names, even though
/// they are identifier-shaped.
fn is_enum_value(s: &str) -> bool {
matches!(s, "Yes" | "No" | "YES" | "NO" | "On" | "Off" | "ON" | "OFF")
}
fn is_key_like(s: &str) -> bool {
!s.is_empty()
&& s.chars().all(|c| c.is_ascii_alphanumeric() || c == '_')
&& s.chars().next().is_some_and(|c| c.is_ascii_alphabetic() || c == '_')
&& !is_number(s)
&& !is_enum_value(s)
}
fn main() {
let mut args = std::env::args().skip(1);
let pak_path = args.next().expect("usage: idxd_tokens <pak> [TypeName...]");
let types: Vec<String> = {
let v: Vec<String> = args.collect();
if v.is_empty() {
vec!["Weapon".into(), "Shell".into()]
} else {
v
}
};
let pak = PakArchive::open(&pak_path).expect("open pak");
for entry in pak.entries() {
let Ok(bytes) = pak.read(entry) else { continue };
let Ok(obj) = IdxdObject::parse(&bytes) else { continue };
let toks = obj.tokens();
// Only entries that actually declare one of the requested sub-record
// types (the pool-start heuristic can prefix one stray byte, so match a
// short suffix rather than equality -- same rule as the splitter below).
if !toks
.iter()
.any(|t| types.iter().any(|ty| t == ty || (t.ends_with(ty.as_str()) && t.len() <= ty.len() + 2)))
{
continue;
}
if std::env::var("SYLPH_RAW").is_ok() {
println!("RAW {:08x}", entry.name_hash);
for t in toks {
println!("T {t}");
}
}
// Sub-record boundaries: a bare type-name token. The first one is the
// schema's own declaration (`EnumWeapon`-style header lives in title
// code, not here), so we take every occurrence in order.
let mut bounds: Vec<(usize, &str)> = Vec::new();
for (i, t) in toks.iter().enumerate() {
// The pool-start heuristic can glue one stray binary byte onto the
// first token ("YWeapon"), so match a short suffix, not equality.
if let Some(ty) = types
.iter()
.find(|ty| t == *ty || (t.ends_with(ty.as_str()) && t.len() <= ty.len() + 2))
{
bounds.push((i, ty.as_str()));
}
}
for (n, &(start, ty)) in bounds.iter().enumerate() {
let end = bounds.get(n + 1).map(|b| b.0).unwrap_or(toks.len());
let slice = &toks[start..end];
// The record's own ID value is the token straight after its type
// name (`Shell` -> `Shell_DSaber_P_wep_01_Beam`). The `ID`/`Name`
// *keys* are interned once in the pool, so only the first record
// shows them -- position-of-key lookup would miss the rest.
let id = slice.get(1).map(|s| s.as_str()).unwrap_or("?");
println!("REC {:08x} {n} {ty} {id}", entry.name_hash);
for i in 1..slice.len() {
let (k, v) = (slice[i].as_str(), slice[i - 1].as_str());
if is_key_like(k) && (!is_key_like(v) || is_number(v)) {
println!("F {k} {v}");
}
}
}
}
}

View File

@@ -0,0 +1,89 @@
//! RE diagnostic: raw-dump a composite scene graph's node records — name, table
//! pointers, and the FULL joint-table slot values (not just the 8 the decoder
//! currently reads) — to derive the true TRS slot layout against captured
//! ground truth.
//! cargo run --release --example node_dump -- <Stage.xpr path> <composite name> [slots]
use sylpheed_formats::mesh::xbg7_descriptor_range;
fn be32(d: &[u8], o: usize) -> u32 {
if o + 4 > d.len() {
return 0;
}
u32::from_be_bytes([d[o], d[o + 1], d[o + 2], d[o + 3]])
}
fn be_f64(d: &[u8], o: usize) -> f64 {
if o + 8 > d.len() {
return f64::NAN;
}
f64::from_be_bytes(d[o..o + 8].try_into().unwrap())
}
fn main() {
let a: Vec<String> = std::env::args().collect();
let bytes = std::fs::read(&a[1]).unwrap();
let comp = &a[2];
let nslots: usize = a.get(3).and_then(|s| s.parse().ok()).unwrap_or(12);
let (desc, desc_end) = xbg7_descriptor_range(&bytes, comp).expect("composite found");
let d = &bytes[desc..desc_end];
println!("descriptor [{desc:#x}..{desc_end:#x}) len {:#x}", d.len());
let mut i = 0usize;
while i + 3 <= d.len() {
let starts = (i + 4 <= d.len() && &d[i..i + 4] == b"rou_")
|| (i + 3 <= d.len() && &d[i..i + 3] == b"GN_");
if !starts {
i += 1;
continue;
}
let mut j = i;
while j < d.len() && d[j] != 0 && d[j].is_ascii_graphic() {
j += 1;
}
let name = std::str::from_utf8(&d[i..j]).unwrap_or("?").to_string();
if name.ends_with("_col") || name.ends_with("_spc") || name.ends_with("_gls") || name.ends_with("_lum") {
i = j.max(i + 1);
continue;
}
// find the 0xFFFFFFFF end marker
let mut ff = i;
let scan_end = (i + 0x90).min(d.len().saturating_sub(4));
while ff < scan_end && be32(d, ff) != 0xFFFF_FFFF {
ff += 4;
}
if ff >= scan_end || ff < 0x10 {
i = j.max(i + 1);
continue;
}
let t44 = be32(d, ff - 0x10) as usize;
let t48 = be32(d, ff - 0x0C) as usize;
let child = be32(d, ff - 0x08) as usize;
let sib = be32(d, ff - 0x04) as usize;
// Dump the record's u32s between name end and marker for structure context.
let rec_u32: Vec<String> = ((j + 1)..ff)
.step_by(4)
.map(|o| format!("{:08X}", be32(d, o)))
.collect();
println!("\nNODE {name} @{i:#x} ff@{ff:#x} t44={t44:#x} t48={t48:#x} child={child:#x} sib={sib:#x}");
println!(" rec: {}", rec_u32.join(" "));
if t44 != 0 && t44 < d.len() {
// Joint table: nslots pointer slots. Each slot record appears to be a
// keyframe track: a value ALSO sits 0x48 BEFORE the pointed address
// (the rest-pose key A), while the decoder currently reads ptr+8 (B).
for k in 0..nslots {
let p = be32(d, t44 + k * 4) as usize;
if p == 0 || p + 24 > d.len() {
continue;
}
let a = if p >= 0x48 { be_f64(d, p - 0x48) } else { f64::NAN };
let b = be_f64(d, p + 8);
let head = if p >= 0x50 { be32(d, p - 0x50) } else { 0 };
let t_a = if p >= 0x50 { be_f64(d, p - 0x50) } else { f64::NAN };
let t_b = be_f64(d, p);
println!(
" slot{k:2} @{p:#x} head={head:08X}: A={a:12.5} (t={t_a:8.3}) B={b:12.5} (t={t_b:8.3})"
);
}
}
i = j.max(i + 1);
}
}

View File

@@ -0,0 +1,20 @@
//! Print a resource's first N decoded positions (buffer order), for matching a
//! capture draw's `pos:` dump to a part:
//! cargo run --release --example part_pos -- <xpr path> <resource> [n]
use std::collections::HashSet;
use sylpheed_formats::mesh::Xbg7Model;
fn main() {
let a: Vec<String> = std::env::args().collect();
let bytes = std::fs::read(&a[1]).unwrap();
let n: usize = a.get(3).and_then(|s| s.parse().ok()).unwrap_or(4);
let want: HashSet<String> = [a[2].clone()].into();
for m in Xbg7Model::models_named(&bytes, &want, &|| false) {
let vc: usize = m.meshes.iter().map(|s| s.positions.len()).sum();
print!("{} vtx={}:", m.name, vc);
for p in m.meshes.iter().flat_map(|s| &s.positions).take(n) {
print!(" ({:.4},{:.4},{:.4})", p[0], p[1], p[2]);
}
println!();
}
}

View File

@@ -0,0 +1,141 @@
//! Audit static ship assembly across ALL stages: flag parts placed far outside
//! their ship's cluster (placement bugs) and joint tracks with more than one
//! keyframe (which the single-key TRS read does not model).
//! cargo run --release --example ship_audit -- <resource3d dir>
use std::collections::HashSet;
use sylpheed_formats::mesh::{xbg7_descriptor_range, xbg7_resource_names, Xbg7Model};
use sylpheed_formats::ship::{assemble_ship, ships_in_container};
fn be32(d: &[u8], o: usize) -> u32 {
if o + 4 > d.len() {
return 0;
}
u32::from_be_bytes([d[o], d[o + 1], d[o + 2], d[o + 3]])
}
/// Count joint-track records whose key count != 1 in a composite's descriptor.
fn multikey_tracks(bytes: &[u8], comp: &str) -> usize {
let Some((desc, desc_end)) = xbg7_descriptor_range(bytes, comp) else { return 0 };
let d = &bytes[desc..desc_end];
let mut n = 0usize;
let mut i = 0usize;
while i + 4 <= d.len() {
let starts = &d[i..i.min(d.len() - 4) + 4] == b"rou_"
|| (i + 3 <= d.len() && &d[i..i + 3] == b"GN_");
if !starts {
i += 1;
continue;
}
let mut j = i;
while j < d.len() && d[j] != 0 && d[j].is_ascii_graphic() {
j += 1;
}
let mut ff = i;
let scan_end = (i + 0x90).min(d.len().saturating_sub(4));
while ff < scan_end && be32(d, ff) != 0xFFFF_FFFF {
ff += 4;
}
if ff < scan_end && ff >= 0x10 {
let t44 = be32(d, ff - 0x10) as usize;
if t44 != 0 && t44 + 32 <= d.len() {
for k in 0..8 {
let p = be32(d, t44 + k * 4) as usize;
if p >= 0x50 && p + 16 <= d.len() {
let head = be32(d, p - 0x50);
if head != 1 && head != 0 {
n += 1;
}
}
}
}
}
i = j.max(i + 1);
}
n
}
fn main() {
let dir = std::env::args().nth(1).unwrap();
let mut entries: Vec<_> = std::fs::read_dir(&dir)
.unwrap()
.filter_map(|e| e.ok())
.map(|e| e.path())
.filter(|p| {
let n = p.file_name().unwrap_or_default().to_string_lossy().to_string();
n.starts_with("Stage_") && n.ends_with(".xpr")
})
.collect();
entries.sort();
for path in entries {
let stage = path.file_name().unwrap().to_string_lossy().to_string();
let bytes = std::fs::read(&path).unwrap();
let names = xbg7_resource_names(&bytes);
// Multi-key animation tracks per composite (breaks the single-key read).
for n in names.iter().filter(|n| n.contains("rou_") && !n.contains("break")) {
let mk = multikey_tracks(&bytes, n);
if mk > 0 {
println!("MULTIKEY {stage} {n}: {mk} tracks");
}
}
for ship in ships_in_container(&bytes) {
if ship.parts.len() < 2 {
continue;
}
let placed = assemble_ship(&bytes, &ship.id, true);
if placed.len() < 2 {
continue;
}
let want: HashSet<String> = placed.iter().map(|p| p.resource.clone()).collect();
let models = Xbg7Model::models_named(&bytes, &want, &|| false);
// Per-placement world centroid.
let mut cents: Vec<(String, [f32; 3])> = Vec::new();
for p in &placed {
let Some(m) = models.iter().find(|m| m.name == p.resource) else { continue };
let (mut c, mut n) = ([0.0f32; 3], 0f32);
for sub in &m.meshes {
for v in &sub.positions {
let w = p.apply(*v);
c[0] += w[0];
c[1] += w[1];
c[2] += w[2];
n += 1.0;
}
}
if n > 0.0 {
cents.push((p.resource.clone(), [c[0] / n, c[1] / n, c[2] / n]));
}
}
if cents.len() < 2 {
continue;
}
// Ship cluster = median centroid; flag parts > 3× the median spread.
let med = |k: usize| -> f32 {
let mut v: Vec<f32> = cents.iter().map(|(_, c)| c[k]).collect();
v.sort_by(|a, b| a.partial_cmp(b).unwrap());
v[v.len() / 2]
};
let m = [med(0), med(1), med(2)];
let dists: Vec<f32> = cents
.iter()
.map(|(_, c)| {
((c[0] - m[0]).powi(2) + (c[1] - m[1]).powi(2) + (c[2] - m[2]).powi(2)).sqrt()
})
.collect();
let mut sd: Vec<f32> = dists.clone();
sd.sort_by(|a, b| a.partial_cmp(b).unwrap());
let spread = sd[sd.len() / 2].max(50.0);
for ((res, c), dist) in cents.iter().zip(&dists) {
if *dist > 6.0 * spread && *dist > 800.0 {
println!(
"OUTLIER {stage} {}: {res} centroid=[{:.0} {:.0} {:.0}] dist={:.0} (spread {:.0})",
ship.id, c[0], c[1], c[2], dist, spread
);
}
}
}
}
println!("audit done");
}

View File

@@ -0,0 +1,50 @@
//! Dump a ship's static placement + scene-graph nodes for a stage container FILE
//! (works offline from an extracted disc, no ISO needed).
//! cargo run --release --example ship_dump -- <Stage_SNN.xpr path> <ship_id>
use sylpheed_formats::mesh::{scene_world_nodes, xbg7_resource_names, Xbg7Model};
use sylpheed_formats::ship::{assemble_ship, is_base_part, ship_id_of};
use std::collections::HashSet;
fn main() {
let a: Vec<String> = std::env::args().collect();
let (path, id) = (&a[1], &a[2]);
let bytes = std::fs::read(path).unwrap();
let names = xbg7_resource_names(&bytes);
// Base parts + their vertex counts.
let want: HashSet<String> = names.iter().filter(|n| is_base_part(n) && ship_id_of(n)==Some(id.as_str())).cloned().collect();
let models = Xbg7Model::models_named(&bytes, &want, &|| false);
println!("== base parts ({}) ==", want.len());
for m in &models {
let vc: usize = m.meshes.iter().map(|s| s.positions.len()).sum();
println!(" {:20} vtx={}", m.name, vc);
}
// Composites present for this id.
println!("== composites ==");
for n in &names {
if n.contains(&format!("rou_{id}")) && ship_id_of(n).is_none() {
let nodes = scene_world_nodes(&bytes, n);
println!(" {:24} {} nodes", n, nodes.len());
}
}
// assemble_ship result: centroids + extent.
for ext in [false, true] {
let placed = assemble_ship(&bytes, id, ext);
println!("== assemble_ship(external={ext}) -> {} parts ==", placed.len());
let mut lo=[f32::MAX;3]; let mut hi=[f32::MIN;3];
for p in &placed {
let src = models.iter().find(|m| m.name==p.resource);
let (mut c, mut n) = ([0.0f32;3], 0f32);
if let Some(src)=src { for sub in &src.meshes { for v in &sub.positions {
let w=p.apply(*v); for k in 0..3 { c[k]+=w[k]; lo[k]=lo[k].min(w[k]); hi[k]=hi[k].max(w[k]); } n+=1.0; }}}
let n=n.max(1.0);
println!(" {:20} T=[{:8.1}{:8.1}{:8.1}] centroid=[{:8.1}{:8.1}{:8.1}]",
p.resource, p.t[0],p.t[1],p.t[2], c[0]/n,c[1]/n,c[2]/n);
}
if placed.iter().any(|p| models.iter().any(|m| m.name==p.resource)) {
println!(" EXTENT=[{:.0} {:.0} {:.0}]", hi[0]-lo[0], hi[1]-lo[1], hi[2]-lo[2]);
}
}
}

View File

@@ -0,0 +1,141 @@
//! Diagnostic: assemble a ship from the BAKED capture table (or the static
//! scene graph with --static) and render orthographic PNGs + print per-part
//! world bounds, to see exactly what the viewer shows.
//! cargo run --release --example ship_render -- <Stage_SNN.xpr path> <ship_id> <out_prefix> [--static]
use std::collections::HashSet;
use sylpheed_formats::mesh::{ScenePart, Xbg7Model};
use sylpheed_formats::ship::assemble_ship;
use sylpheed_formats::ship_capture::{embedded_placement, to_scene_parts};
fn main() {
let a: Vec<String> = std::env::args().collect();
let use_static = a.iter().any(|s| s == "--static");
let (path, id, out) = (&a[1], &a[2], &a[3]);
let bytes = std::fs::read(path).unwrap();
let placed: Vec<ScenePart> = if use_static {
assemble_ship(&bytes, id, true)
} else {
let cap = embedded_placement(id).expect("ship not in baked table");
to_scene_parts(&cap)
};
let want: HashSet<String> = placed.iter().map(|p| p.resource.clone()).collect();
let models = Xbg7Model::models_named(&bytes, &want, &|| false);
// World triangles + per-part bounds.
let mut tris: Vec<[[f32; 3]; 3]> = Vec::new();
for p in &placed {
let Some(m) = models.iter().find(|m| m.name == p.resource) else { continue };
let mut lo = [f32::MAX; 3];
let mut hi = [f32::MIN; 3];
for sub in &m.meshes {
let w: Vec<[f32; 3]> = sub.positions.iter().map(|v| p.apply(*v)).collect();
for v in &w {
for k in 0..3 {
lo[k] = lo[k].min(v[k]);
hi[k] = hi[k].max(v[k]);
}
}
for t in sub.indices.chunks_exact(3) {
tris.push([w[t[0] as usize], w[t[1] as usize], w[t[2] as usize]]);
}
}
println!(
"{:20} X[{:8.1},{:8.1}] Y[{:8.1},{:8.1}] Z[{:8.1},{:8.1}]",
p.resource, lo[0], hi[0], lo[1], hi[1], lo[2], hi[2]
);
}
// Exhaust cones at the GN_Jet/GN_SJet frames (the game's visible thrusters).
for f in sylpheed_formats::ship::exhaust_frames(&bytes, id) {
const SEG: usize = 12;
let (r, len) = (22.0f32, 140.0f32);
let ring: Vec<[f32; 3]> = (0..SEG)
.map(|s| {
let a = s as f32 / SEG as f32 * std::f32::consts::TAU;
f.apply([a.cos() * r, a.sin() * r, 0.0])
})
.collect();
let apex = f.apply([0.0, 0.0, len]);
for s in 0..SEG {
tris.push([apex, ring[(s + 1) % SEG], ring[s]]);
}
println!(" exhaust frame {:16} T=[{:8.1}{:8.1}{:8.1}]", f.resource, f.t[0], f.t[1], f.t[2]);
}
println!("total {} tris from {} placements", tris.len(), placed.len());
// Orthographic z-buffered flat renders: top (X/Z, look down Y) and side (Z/Y, look down X).
for (name, ax, ay, az) in [("top", 0usize, 2usize, 1usize), ("side", 2usize, 1usize, 0usize)] {
let size = 900usize;
let (mut lo, mut hi) = ([f32::MAX; 3], [f32::MIN; 3]);
for t in &tris {
for v in t {
for k in 0..3 {
lo[k] = lo[k].min(v[k]);
hi[k] = hi[k].max(v[k]);
}
}
}
let cx = (lo[ax] + hi[ax]) * 0.5;
let cy = (lo[ay] + hi[ay]) * 0.5;
let ext = (hi[ax] - lo[ax]).max(hi[ay] - lo[ay]) * 0.55;
let scale = (size as f32 * 0.5) / ext;
let mut img = vec![0u8; size * size * 3];
let mut zbuf = vec![f32::MIN; size * size];
for t in &tris {
// screen coords
let p: Vec<[f32; 3]> = t
.iter()
.map(|v| {
[
(v[ax] - cx) * scale + size as f32 * 0.5,
(cy - v[ay]) * scale + size as f32 * 0.5,
v[az],
]
})
.collect();
// flat shade by triangle normal Z-ish
let e1 = [t[1][0] - t[0][0], t[1][1] - t[0][1], t[1][2] - t[0][2]];
let e2 = [t[2][0] - t[0][0], t[2][1] - t[0][1], t[2][2] - t[0][2]];
let n = [
e1[1] * e2[2] - e1[2] * e2[1],
e1[2] * e2[0] - e1[0] * e2[2],
e1[0] * e2[1] - e1[1] * e2[0],
];
let nl = (n[0] * n[0] + n[1] * n[1] + n[2] * n[2]).sqrt().max(1e-6);
let lum = (n[az].abs() / nl * 200.0 + 40.0) as u8;
// bbox raster
let minx = p.iter().map(|v| v[0]).fold(f32::MAX, f32::min).max(0.0) as usize;
let maxx = (p.iter().map(|v| v[0]).fold(f32::MIN, f32::max).min(size as f32 - 1.0)) as usize;
let miny = p.iter().map(|v| v[1]).fold(f32::MAX, f32::min).max(0.0) as usize;
let maxy = (p.iter().map(|v| v[1]).fold(f32::MIN, f32::max).min(size as f32 - 1.0)) as usize;
let det = (p[1][0] - p[0][0]) * (p[2][1] - p[0][1]) - (p[2][0] - p[0][0]) * (p[1][1] - p[0][1]);
if det.abs() < 1e-6 {
continue;
}
for y in miny..=maxy {
for x in minx..=maxx {
let (fx, fy) = (x as f32 + 0.5, y as f32 + 0.5);
let w0 = ((p[1][0] - fx) * (p[2][1] - fy) - (p[2][0] - fx) * (p[1][1] - fy)) / det;
let w1 = ((p[2][0] - fx) * (p[0][1] - fy) - (p[0][0] - fx) * (p[2][1] - fy)) / det;
let w2 = 1.0 - w0 - w1;
if w0 < 0.0 || w1 < 0.0 || w2 < 0.0 {
continue;
}
let z = w0 * p[0][2] + w1 * p[1][2] + w2 * p[2][2];
let idx = y * size + x;
if z > zbuf[idx] {
zbuf[idx] = z;
img[idx * 3] = lum;
img[idx * 3 + 1] = lum;
img[idx * 3 + 2] = lum;
}
}
}
}
let file = format!("{out}_{name}.ppm");
let mut ppm = format!("P6\n{size} {size}\n255\n").into_bytes();
ppm.extend_from_slice(&img);
std::fs::write(&file, ppm).unwrap();
println!("wrote {file}");
}
}

View File

@@ -0,0 +1,55 @@
//! Scratch analysis: inventory one UI screen's pak — every entry, and for RATC
//! entries the children they bundle. Optionally write an entry's decompressed
//! bytes out for hex inspection.
//!
//! Run: cargo run -p sylpheed-formats --example ui_screen -- <PAK> [--dump 0xHASH out.bin]
use sylpheed_formats::pak::{self, PakArchive};
use sylpheed_formats::ratc;
fn main() {
let mut args = std::env::args().skip(1);
let pak = args.next().expect("usage: ui_screen <pak> [--dump 0xHASH out.bin]");
let arc = PakArchive::open(&pak).expect("open pak");
let rest: Vec<String> = args.collect();
if rest.first().map(|s| s == "--dump").unwrap_or(false) {
let h = u32::from_str_radix(rest[1].trim_start_matches("0x"), 16).unwrap();
let bytes = arc.read_by_hash(h).expect("entry present").expect("decompress");
std::fs::write(&rest[2], &bytes).unwrap();
println!("wrote {} bytes to {}", bytes.len(), rest[2]);
return;
}
if rest.first().map(|s| s == "--child").unwrap_or(false) {
// --child 0xHASH <child-name> <out>: carve one RATC child out by its
// listed offset/size, so a 165-byte layout record can be hexdumped alone.
let h = u32::from_str_radix(rest[1].trim_start_matches("0x"), 16).unwrap();
let bytes = arc.read_by_hash(h).expect("entry present").expect("decompress");
let kids = ratc::parse(&bytes).expect("ratc");
let k = kids.iter().find(|k| k.name == rest[2]).expect("child not found");
std::fs::write(&rest[3], &bytes[k.offset..k.offset + k.size]).unwrap();
println!("wrote {} B ({} @ {:#x}) to {}", k.size, k.name, k.offset, rest[3]);
return;
}
println!("{pak}: {} entries", arc.len());
for e in arc.entries() {
let Ok(bytes) = arc.read(e) else {
println!(" {:08x} <decompress failed>", e.name_hash);
continue;
};
let label = pak::inner_format_label(&bytes);
println!(" {:08x} {:>9} B {}", e.name_hash, bytes.len(), label);
if ratc::is_ratc(&bytes) {
if let Some(kids) = ratc::parse(&bytes) {
for k in &kids {
println!(" - {:<40} {:>9} B {}", k.name, k.size, k.kind);
}
if kids.is_empty() {
println!(" (no children found)");
}
}
}
}
}

View File

@@ -68,6 +68,9 @@ pub mod localization;
// Whole-ship assembly from XBG7 part families (capital ships as split parts).
pub mod ship;
// Exact capital-ship placement from a runtime F10 ship-capture (ground truth).
pub mod ship_capture;
/// Re-export the most commonly used types at the crate root.
pub use font::FontInfo;
pub use idxd::{IdxdError, IdxdObject};

View File

@@ -1403,6 +1403,13 @@ pub fn submesh_albedos(bytes: &[u8], resource_name: &str) -> Vec<(usize, usize,
}
/// Locate the descriptor byte range of the named XBG7 resource in an XPR2 file.
/// Diagnostic access to a resource's raw XBG7 descriptor byte range — used by
/// the node-graph reverse-engineering examples. Not part of the decode API.
#[doc(hidden)]
pub fn xbg7_descriptor_range(bytes: &[u8], resource_name: &str) -> Option<(usize, usize)> {
xbg7_descriptor(bytes, resource_name)
}
fn xbg7_descriptor(bytes: &[u8], resource_name: &str) -> Option<(usize, usize)> {
if bytes.len() < 16 || &bytes[..4] != b"XPR2" {
return None;
@@ -1602,12 +1609,58 @@ fn rot_x(a: f32) -> M3 {
let (s, c) = a.sin_cos();
[[1.0, 0.0, 0.0], [0.0, c, -s], [0.0, s, c]]
}
/// Rotation about Y (vertical) — mixes X and Z (yaw).
fn rot_y(a: f32) -> M3 {
let (s, c) = a.sin_cos();
[[c, 0.0, s], [0.0, 1.0, 0.0], [-s, 0.0, c]]
}
/// Rotation about Z (fore/aft) — mixes X and Y (V-tail cant / roll).
fn rot_z(a: f32) -> M3 {
let (s, c) = a.sin_cos();
[[c, -s, 0.0], [s, c, 0.0], [0.0, 0.0, 1.0]]
}
/// Read a node's **9-channel TRS** `[TX TY TZ RY RX RZ SX SY SZ]` from its joint
/// table at `t44`.
///
/// The table holds **8 pointer slots**, but the node has **nine** keyframe
/// tracks (3 translation, 3 rotation, 3 scale), each a 0x50-byte record with its
/// value at `+8`. The pointers are shifted one record forward: `ptr[k]` points at
/// track `k+1`'s record, so channel `k+1`'s value is `f64 @ ptr[k]+8` and channel
/// 0 (TX!) — which no pointer names — sits one record *before* the first, at
/// `ptr[0] 0x48`.
///
/// This off-by-one is what hid every lateral offset: the old 8-slot read took
/// TY as "X", RY as "Y" (usually 0 — why hulls stacked on the centreline) and
/// never saw TX at all (the ±264 hull pair offset). Derived 2026-07-26 from the
/// e106 F10 runtime capture and verified against the captured transforms of all
/// 8 destroyer parts (see docs/re/ship-placement-runtime-capture.md).
fn read_trs9(d: &[u8], t44: usize) -> [f64; 9] {
let mut v = [0.0f64; 9];
let p0 = be32(d, t44) as usize;
if p0 >= 0x48 && p0 + 16 <= d.len() {
v[0] = be_f64(d, p0 - 0x48);
}
for k in 0..8 {
let p = be32(d, t44 + k * 4) as usize;
if p != 0 && p + 16 <= d.len() {
v[k + 1] = be_f64(d, p + 8);
}
}
v
}
/// Compose a node's local rotation from its three Euler channels
/// `(ch3, ch4, ch5) = (RY, RX, RZ)` — **`Ry·Rx·Rz`**, angles applied directly.
/// Convention pinned by the e106 engine-rig runtime capture: the nacelle node
/// stores `(RX, RZ) = (15°, +30°)` and the captured WorldView rotation equals
/// `Rx(15°)·Rz(30°)` exactly (decomposed element-for-element). The older saber
/// analysis script recovered the transpose of this convention; the fin override
/// [`saber_measured`] keeps those parts pinned either way.
fn node_rotation(ry: f64, rx: f64, rz: f64) -> M3 {
m3_mul(m3_mul(rot_y(ry as f32), rot_x(rx as f32)), rot_z(rz as f32))
}
/// The DeltaSaber fin-assembly part names — the structural signature that marks a
/// DeltaSaber-family model (`_T`/`_W`/`_A` = f001/f002/f004; same layout, slightly
/// different vertex counts) so the measured placements below apply to all three.
@@ -1685,18 +1738,9 @@ pub fn node_transforms(bytes: &[u8], resource_name: &str) -> Vec<NodePlacement>
let head_end = d.len().min(0x1684);
let prefix = format!("rou_{resource_name}_");
// 8-value TRS from a node's `+0x44` joint table (8 pointers, each f64 @ +8);
// returns identity-safe zeros when the table is absent.
let read_trs = |t44: usize| -> [f64; 8] {
let mut v = [0.0f64; 8];
for (k, slot) in v.iter_mut().enumerate() {
let p = be32(d, t44 + k * 4) as usize;
if p != 0 && p + 16 <= d.len() {
*slot = be_f64(d, p + 8);
}
}
v
};
// 9-channel TRS `[TX TY TZ RY RX RZ SX SY SZ]` from the node's `+0x44` joint
// table (see [`read_trs9`] — the 8 pointers are shifted one track forward).
let read_trs = |t44: usize| read_trs9(d, t44);
// Phase 1: collect node records in document order, each with its local
// affine and the offset where its subtree ends (its next sibling).
@@ -1752,18 +1796,18 @@ pub fn node_transforms(bytes: &[u8], resource_name: &str) -> Vec<NodePlacement>
let trs = if t44 != 0 && t44 + 32 <= d.len() {
read_trs(t44)
} else {
[0.0; 8]
[0.0; 9]
};
if std::env::var("XNODEDUMP").is_ok() {
eprintln!(
"NODE {name} vtx={vtx} t44={t44:#x} t48={t48:#x} ff@{ff:#x} sib@{sib_ptr:#x} TRS=[{:.4} {:.4} {:.4} | {:.4} {:.4} | {:.4} {:.4} {:.4}]",
trs[0], trs[1], trs[2], trs[3], trs[4], trs[5], trs[6], trs[7]
"NODE {name} vtx={vtx} t44={t44:#x} t48={t48:#x} ff@{ff:#x} sib@{sib_ptr:#x} TRS=[{:.4} {:.4} {:.4} | {:.4} {:.4} {:.4} | {:.4} {:.4} {:.4}]",
trs[0], trs[1], trs[2], trs[3], trs[4], trs[5], trs[6], trs[7], trs[8]
);
}
// Local transform: translation (t0=X, t2=up→Y, t1=fore/aft→Z), and a
// rotation Rz(r1)·Rx(r0) (r1 = V-tail cant about fore/aft, r0 = pitch).
let local_t = [trs[0] as f32, trs[2] as f32, trs[1] as f32];
let local_m = m3_mul(rot_z(trs[4] as f32), rot_x(trs[3] as f32));
// Local transform: translation (TX, TY, TZ) + Euler rotation (see
// [`node_rotation`]).
let local_t = [trs[0] as f32, trs[1] as f32, trs[2] as f32];
let local_m = node_rotation(trs[3], trs[4], trs[5]);
// Next sibling: its name starts 4 bytes before the pointer, and there a
// real node record begins with "rou_". Everything between here and there
// is this node's subtree.
@@ -1882,16 +1926,7 @@ pub fn scene_world_nodes(bytes: &[u8], composite_name: &str) -> Vec<ScenePart> {
let d = &bytes[desc..desc_end];
let head_end = d.len();
let read_trs = |t44: usize| -> [f64; 8] {
let mut v = [0.0f64; 8];
for (k, slot) in v.iter_mut().enumerate() {
let p = be32(d, t44 + k * 4) as usize;
if p != 0 && p + 16 <= d.len() {
*slot = be_f64(d, p + 8);
}
}
v
};
let read_trs = |t44: usize| read_trs9(d, t44);
// A node record begins with a name starting `rou_` or `GN_`.
let node_name_at = |i: usize| -> Option<(String, usize)> {
@@ -1944,13 +1979,13 @@ pub fn scene_world_nodes(bytes: &[u8], composite_name: &str) -> Vec<ScenePart> {
let trs = if t44 != 0 && t44 + 32 <= d.len() {
read_trs(t44)
} else {
[0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0]
};
let local_t = [trs[0] as f32, trs[2] as f32, trs[1] as f32];
let local_m = m3_mul(rot_z(trs[4] as f32), rot_x(trs[3] as f32));
// Slots 57 are per-axis scale (a hardpoint part ships at e.g. 0.5).
let local_t = [trs[0] as f32, trs[1] as f32, trs[2] as f32];
let local_m = node_rotation(trs[3], trs[4], trs[5]);
// Channels 68 are per-axis scale (a hardpoint part ships at e.g. 0.5).
let sc = |v: f64| if v.abs() < 1e-6 { 1.0 } else { v as f32 };
let local_s = [sc(trs[5]), sc(trs[6]), sc(trs[7])];
let local_s = [sc(trs[6]), sc(trs[7]), sc(trs[8])];
// Next sibling begins 4 bytes before its pointer, at a `rou_`/`GN_` name.
let sib_end = sib_ptr.checked_sub(4).filter(|&s| {
s > i && s + 4 <= head_end && (&d[s..s + 4] == b"rou_" || &d[s..s + 3] == b"GN_")

View File

@@ -74,6 +74,10 @@ pub struct ShipModel {
pub faction: Faction,
/// Base geometry resource names to draw together, in directory order.
pub parts: Vec<String>,
/// Whether a composite scene graph (`e_rou_<id>`) exists — i.e. this is a
/// real assembled ship. Families without one (fighter morph sets, shared
/// turret/prop parts) have no placement data and would stack at the origin.
pub has_composite: bool,
}
/// The `[a-z]NNN` id prefix of a resource name, if it has one (`e106_bdy_01`
@@ -146,7 +150,13 @@ pub fn ships_in_container(bytes: &[u8]) -> Vec<ShipModel> {
let id = ship_id_of(n).unwrap().to_string();
let entry = ships.entry(id.clone()).or_insert_with(|| {
ids.push(id.clone());
ShipModel { id: id.clone(), faction: Faction::from_id(&id), parts: Vec::new() }
let has_composite = !composites_for(&names, &id).is_empty();
ShipModel {
id: id.clone(),
faction: Faction::from_id(&id),
parts: Vec::new(),
has_composite,
}
});
entry.parts.push(n.clone());
}
@@ -160,22 +170,45 @@ pub fn ships_in_container(bytes: &[u8]) -> Vec<ShipModel> {
/// nothing and only pose their children.
fn resolve_hull_resource(node: &str, resources: &HashSet<String>) -> Option<String> {
let stem = node.rsplit_once("rou_").map(|(_, s)| s).unwrap_or(node);
if resources.contains(stem) {
return Some(stem.to_string());
let try_stem = |s: &str| -> Option<String> {
if resources.contains(s) {
return Some(s.to_string());
}
// Squeezed digit run (`wep02` names the `wep_02_01` resource): insert an
// underscore before the digits and try the `_01` first-part too.
if let Some(pos) = s.rfind(|c: char| c.is_ascii_digit()) {
let mut start = pos;
while start > 0 && s.as_bytes()[start - 1].is_ascii_digit() {
start -= 1;
}
if start > 0 && s.as_bytes()[start - 1] != b'_' {
let split = format!("{}_{}", &s[..start], &s[start..]);
if resources.contains(&split) {
return Some(split);
}
let first = format!("{split}_01");
if resources.contains(&first) {
return Some(first);
}
}
}
None
};
if let Some(r) = try_stem(stem) {
return Some(r);
}
for suf in ["_root", "_mov"] {
if let Some(s) = stem.strip_suffix(suf) {
if resources.contains(s) {
return Some(s.to_string());
if let Some(r) = try_stem(s) {
return Some(r);
}
}
}
// Trailing `_NN` instance index (`eng_01_1` → `eng_01`).
if let Some(pos) = stem.rfind('_') {
if !stem[pos + 1..].is_empty() && stem[pos + 1..].bytes().all(|c| c.is_ascii_digit()) {
let s = &stem[..pos];
if resources.contains(s) {
return Some(s.to_string());
if let Some(r) = try_stem(&stem[..pos]) {
return Some(r);
}
}
}
@@ -210,31 +243,31 @@ fn composites_for<'a>(names: &'a [String], id: &str) -> Vec<&'a String> {
/// Reconstruct a whole ship as a list of geometry-resource placements in a shared
/// ship-local frame — the placement the game builds at runtime.
///
/// Two tiers, matching how the game splits a ship:
/// 1. **Hull** — `rou_<id>_*` nodes in the primary composite scene graph name the
/// hull body resources and carry their world transform. This tier is exact: the
/// bodies (and any engine posed by a `rou_` node) land where the game puts them.
/// 2. **External hardpoints** — `GN_*` frame nodes are the Vessel mounts; each
/// unplaced bridge / shield-generator / engine part (`<id>_brg_*`, `_sld_*`,
/// `_eng_*`) is placed at its matching frame by category + index. This tier is
/// APPROXIMATE: a `GN_*` frame is the mount *pivot* (often on the centreline),
/// while the part's outboard/rotational offset lives in a detail sub-rig or in
/// runtime code that this static pass does not recover — so external parts land
/// near, not exactly, where they belong. Gated by `include_external`.
/// Fully static and EXACT (validated part-for-part against an e106 runtime
/// capture — see `docs/re/ship-placement-runtime-capture.md`):
/// 1. **Composite nodes** — every `rou_*` node in the primary composite
/// (`e_rou_<id>`) places a geometry resource, INCLUDING repeated instances
/// and cross-id turret mounts (`rou_e303_wep_01_root` ×2 on the e106 hull).
/// 2. **Detail rigs** — `e_rou_<id>_eng` is the engine cluster rig; its nodes
/// (e.g. two `eng_01` nacelle instances + `eng_02`) mount at the primary
/// composite's `GN_Engine_01` frame: `world = frame ∘ rig_local`.
/// 3. **GN hardpoints** — remaining unplaced `brg`/`sld` parts sit exactly at
/// their `GN_Bridge_NN` / `GN_ShieldG_NN` frame (frames carry full TRS).
/// 4. **Mirrored twins** — a port/starboard pair (`bdy_01`/`bdy_02`) shares one
/// geometry; the instance whose lateral offset points the geometry's dominant
/// side inboard is drawn X-reflected (capture-verified; det < 0 → the caller
/// reverses winding).
///
/// Turret placement (shared `e3NN`/`e4NN` gun models mounted at many `GN_*Gun*`
/// frames) needs the per-mount Vessel recipe and is not resolved at all.
/// `include_external` keeps tiers 24 (off = bare composite-node hull).
pub fn assemble_ship(bytes: &[u8], id: &str, include_external: bool) -> Vec<ScenePart> {
let names = xbg7_resource_names(bytes);
let resources: HashSet<String> = names.iter().cloned().collect();
// The PRIMARY composite carries the real ship-space layout (hull bodies +
// every `GN_*` hardpoint frame). Sibling composites (`e_rou_<id>_eng`,
// `_wep_*`) are LOCAL detail rigs — their nodes are in their own frame, not
// ship-space, so folding them in floats parts mid-ship. Pick the composite
// with the most nodes; that is the assembled hull.
// The PRIMARY composite carries the ship-space layout (hull bodies, turret
// mounts + every `GN_*` hardpoint frame): the one with the most nodes.
let scenes: Vec<(String, Vec<ScenePart>)> = composites_for(&names, id)
.into_iter()
.filter(|c| !c.ends_with("_joint"))
.map(|c| (c.clone(), scene_world_nodes(bytes, c)))
.collect();
let Some((_, nodes)) = scenes.iter().max_by_key(|(_, n)| n.len()) else {
@@ -252,24 +285,54 @@ pub fn assemble_ship(bytes: &[u8], id: &str, include_external: bool) -> Vec<Scen
// GN hardpoint frames, world-space, from the primary composite.
let mut frames: Vec<ScenePart> = Vec::new();
// Tier 1: every composite node placement, instances included. Cross-id
// resources (shared e303/e4NN turret models the composite mounts) count too.
for node in nodes {
if node.resource.starts_with("GN_") {
frames.push(node.clone());
} else if let Some(res) = resolve_hull_resource(&node.resource, &resources) {
// Only the ship's own parts. A composite also mounts shared, cross-id
// turret models (`e303_wep_*` on an `e106` hull); those need the
// per-mount Vessel recipe and are placed separately.
if ship_id_of(&res) != Some(id) {
continue;
if ship_id_of(&res) != Some(id) && !include_external {
continue; // hull-only view: own parts only
}
if placed_res.insert(res.clone()) {
// Distinct INSTANCES keep their own placement; a nested alias of the
// same resource at the same transform (`bdy_01_mov` + its identity
// `bdy_01` child) collapses to one draw.
let dup = placed.iter().any(|p| {
p.resource == res
&& (p.t[0] - node.t[0]).abs() < 0.01
&& (p.t[1] - node.t[1]).abs() < 0.01
&& (p.t[2] - node.t[2]).abs() < 0.01
});
if !dup {
placed_res.insert(res.clone());
placed.push(ScenePart { resource: res, m: node.m, t: node.t, s: node.s });
}
}
}
// Place unplaced external parts at their Vessel hardpoint frame (APPROXIMATE —
// see the doc comment). Skipped for a clean, exact hull-only render.
// Tier 2: the engine cluster rig, mounted at GN_Engine_01. The rig's nodes
// are in the FRAME's local space (two mirrored `eng_01` nacelles + centre
// `eng_02` for e106) — capture-verified to 0.1 units.
if include_external {
let rig_name = format!("e_rou_{id}_eng");
if names.contains(&rig_name) {
if let Some(frame) = frames.iter().find(|f| f.resource.starts_with("GN_Engine")) {
for rn in scene_world_nodes(bytes, &rig_name) {
let Some(res) = resolve_hull_resource(&rn.resource, &resources) else {
continue;
};
// world = frame ∘ rig_local
let m = m3_mul_pub(frame.m, rn.m);
let rt = m3_vec_pub(frame.m, rn.t);
let t = [rt[0] + frame.t[0], rt[1] + frame.t[1], rt[2] + frame.t[2]];
placed_res.insert(res.clone());
placed.push(ScenePart { resource: res, m, t, s: rn.s });
}
}
}
}
// Tier 3: remaining external parts at their exact GN frame (full TRS).
const CATS: [(&str, &str); 3] = [("brg", "Bridge"), ("sld", "ShieldG"), ("eng", "Engine")];
for part in names
.iter()
@@ -294,6 +357,13 @@ pub fn assemble_ship(bytes: &[u8], id: &str, include_external: bool) -> Vec<Scen
}
}
// Tier 4: mirror one of a shared-geometry twin pair. The engine uploads the
// second of a port/starboard pair X-reflected (capture-verified on
// bdy_01/bdy_02): for two same-geometry resources placed at ±TX, the
// instance whose TX sign matches the geometry's dominant-X side draws
// mirrored (so the pair ends up symmetric instead of twice the same side).
apply_twin_mirrors(bytes, &mut placed);
// Fallback for a ship with no composite scene graph (a simple prop): draw its
// base parts untransformed rather than nothing.
if placed.is_empty() {
@@ -306,6 +376,112 @@ pub fn assemble_ship(bytes: &[u8], id: &str, include_external: bool) -> Vec<Scen
placed
}
/// The ship's exhaust mount frames (`GN_Jet_*` / `GN_SJet_*`), world-space, from
/// the primary composite. These are where the game renders the engine exhaust
/// FX — the visible "thrusters". The engine GEOMETRY itself sits recessed in
/// the hull (capture-verified across 43 e106 instances: the game draws
/// `eng_01`/`eng_02` exactly where [`assemble_ship`] puts them); without the
/// FX the assembled ship reads engine-less, so a viewer can draw exhaust
/// markers at these frames.
pub fn exhaust_frames(bytes: &[u8], id: &str) -> Vec<ScenePart> {
let names = xbg7_resource_names(bytes);
let scenes: Vec<Vec<ScenePart>> = composites_for(&names, id)
.into_iter()
.filter(|c| !c.ends_with("_joint"))
.map(|c| scene_world_nodes(bytes, c))
.collect();
let Some(nodes) = scenes.into_iter().max_by_key(|n| n.len()) else {
return Vec::new();
};
nodes
.into_iter()
.filter(|n| n.resource.starts_with("GN_Jet") || n.resource.starts_with("GN_SJet"))
.collect()
}
/// Detect shared-geometry port/starboard twin RESOURCES placed at ±TX and mark
/// the dominant-side instance as an X-reflection (negate the matrix X column).
/// Twins are `<stem>_01`/`<stem>_02` pairs with equal vertex data; the reflected
/// one is the instance whose TX sign equals the geometry's mean-X sign (the
/// un-reflected drawing already covers that side's opposite).
fn apply_twin_mirrors(bytes: &[u8], placed: &mut [ScenePart]) {
use crate::mesh::Xbg7Model;
// Candidate pairs: resources differing only in a trailing _01/_02, both
// placed exactly once, at opposite-sign TX of equal magnitude.
let mut pairs: Vec<(usize, usize)> = Vec::new();
for i in 0..placed.len() {
let a = &placed[i];
let Some(stem) = a.resource.strip_suffix("_01") else { continue };
let twin = format!("{stem}_02");
let Some(j) = placed.iter().position(|p| p.resource == twin) else { continue };
let b = &placed[j];
if (a.t[0] + b.t[0]).abs() < 0.5 && a.t[0].abs() > 1.0 {
pairs.push((i, j));
}
}
if pairs.is_empty() {
return;
}
let want: HashSet<String> = pairs
.iter()
.flat_map(|&(i, j)| [placed[i].resource.clone(), placed[j].resource.clone()])
.collect();
let models = Xbg7Model::models_named(bytes, &want, &|| false);
for (i, j) in pairs {
let get = |r: &str| models.iter().find(|m| m.name == r);
let (Some(ma), Some(mb)) = (get(&placed[i].resource), get(&placed[j].resource)) else {
continue;
};
let first = |m: &Xbg7Model| -> Vec<[f32; 3]> {
m.meshes.iter().flat_map(|s| s.positions.iter().copied()).take(16).collect()
};
let (fa, fb) = (first(ma), first(mb));
if fa.len() != fb.len()
|| !fa.iter().zip(&fb).all(|(p, q)| {
(p[0] - q[0]).abs() < 1e-3 && (p[1] - q[1]).abs() < 1e-3 && (p[2] - q[2]).abs() < 1e-3
})
{
continue; // distinct (already-mirrored) geometries — nothing to do
}
let mean_x: f32 = ma
.meshes
.iter()
.flat_map(|s| s.positions.iter())
.map(|p| p[0])
.sum::<f32>()
/ ma.meshes.iter().map(|s| s.positions.len()).sum::<usize>().max(1) as f32;
if mean_x.abs() < 1e-3 {
continue; // symmetric geometry — mirroring is a no-op
}
// Reflect the instance whose lateral offset OPPOSES the geometry's
// dominant side (drawn plain it would fold onto the centreline; the
// capture shows the engine reflects exactly this one — for e106 the
// 264 port instance draws the file data plain, the +264 one mirrored).
let k = if (placed[i].t[0] > 0.0) == (mean_x > 0.0) { j } else { i };
for row in &mut placed[k].m {
row[0] = -row[0];
}
}
}
// Small helpers mirroring mesh.rs's private affine ops.
fn m3_mul_pub(a: [[f32; 3]; 3], b: [[f32; 3]; 3]) -> [[f32; 3]; 3] {
let mut o = [[0.0f32; 3]; 3];
for r in 0..3 {
for c in 0..3 {
o[r][c] = a[r][0] * b[0][c] + a[r][1] * b[1][c] + a[r][2] * b[2][c];
}
}
o
}
fn m3_vec_pub(a: [[f32; 3]; 3], v: [f32; 3]) -> [f32; 3] {
[
a[0][0] * v[0] + a[0][1] * v[1] + a[0][2] * v[2],
a[1][0] * v[0] + a[1][1] * v[1] + a[1][2] * v[2],
a[2][0] * v[0] + a[2][1] * v[1] + a[2][2] * v[2],
]
}
#[cfg(test)]
mod tests {
use super::*;
@@ -363,6 +539,85 @@ mod tests {
assert!(e108.parts.iter().all(|p| is_base_part(p)));
}
// Disc-gated GROUND-TRUTH test: the fully-static assembly must reproduce the
// runtime F10 capture (the baked e106 table) part-for-part — translations
// AND the engine-rig rotation — plus the multi-instance parts the capture's
// vbase-dedup could not see (two nacelles, two turrets).
#[test]
fn static_assembly_matches_runtime_capture() {
let Ok(iso) = std::env::var("SYLPHEED_ISO") else {
eprintln!("SYLPHEED_ISO unset — skipping");
return;
};
let bytes = {
use crate::xiso::open_iso;
let rt = tokio::runtime::Builder::new_current_thread().build().unwrap();
rt.block_on(async {
let mut r = open_iso(std::path::Path::new(&iso)).await.unwrap();
r.read_file("hidden/resource3d/Stage_S01.xpr").await.unwrap()
})
};
let placed = assemble_ship(&bytes, "e106", true);
let cap = crate::ship_capture::embedded_placement("e106").expect("e106 baked");
// Express static placements relative to bdy_04 (the capture's reference).
let r4 = placed
.iter()
.find(|p| p.resource == "e106_bdy_04")
.expect("bdy_04 placed")
.clone();
let rel = |p: &crate::mesh::ScenePart| -> [f32; 3] {
[p.t[0] - r4.t[0], p.t[1] - r4.t[1], p.t[2] - r4.t[2]]
};
for want in &cap.parts {
let best = placed
.iter()
.filter(|p| p.resource == want.part)
.map(|p| {
let t = rel(p);
let d = (t[0] - want.t[0]).abs()
+ (t[1] - want.t[1]).abs()
+ (t[2] - want.t[2]).abs();
(d, p)
})
.min_by(|a, b| a.0.partial_cmp(&b.0).unwrap())
.unwrap_or_else(|| panic!("{} not placed statically", want.part));
assert!(
best.0 < 1.0,
"{}: static rel-T {:?} != captured {:?} (Δ={:.2})",
want.part,
rel(best.1),
want.t,
best.0
);
// Rotations must match too (the engine rig is the interesting case).
let m = &best.1.m;
for r in 0..3 {
for c in 0..3 {
assert!(
(m[r][c] - want.m[r][c]).abs() < 0.02,
"{}: static M row{r} {:?} != captured {:?}",
want.part,
m[r],
want.m[r]
);
}
}
}
// Multi-instance coverage the capture couldn't see (vbase dedup).
let count = |res: &str| placed.iter().filter(|p| p.resource == res).count();
assert_eq!(count("e106_eng_01"), 2, "both engine nacelles placed");
assert_eq!(count("e303_wep_01"), 2, "both shared turrets placed");
// The mirrored starboard hull reflects (det < 0), the port one doesn't.
let det = |m: &[[f32; 3]; 3]| {
m[0][0] * (m[1][1] * m[2][2] - m[1][2] * m[2][1])
- m[0][1] * (m[1][0] * m[2][2] - m[1][2] * m[2][0])
+ m[0][2] * (m[1][0] * m[2][1] - m[1][1] * m[2][0])
};
let one = |res: &str| placed.iter().find(|p| p.resource == res).unwrap();
assert!(det(&one("e106_bdy_02").m) < 0.0, "starboard hull mirrored");
assert!(det(&one("e106_bdy_01").m) > 0.0, "port hull plain");
}
// Disc-gated: the scene graph must SPREAD a ship's parts, not stack them at
// the origin. Reconstruct the ADAN frigate e106 and assert its bridge sits
// clearly aft of its forward hull body (a real ship layout, not a pile).

View File

@@ -0,0 +1,694 @@
//! Exact capital-ship part placement from a Canary **F10 ship-capture** log —
//! the runtime ground truth that static [`crate::ship::assemble_ship`] only
//! approximates for external parts.
//!
//! ## Why a capture is needed
//!
//! A captured part's vertex BUFFER holds **local** coordinates (byte-identical to
//! the `.xpr`), so capital-ship parts are placed **entirely in the vertex shader**
//! — the buffer carries no placement. The per-part transform lives in the ship
//! shader's **vertex float constants**: `c0..c2` are the **WorldViewProjection**
//! matrix rows. Each row's norm is `(sx, sy, 1)` (the projection x/y scales, with
//! `sy/sx ≈ 1.78 = 16:9`); dividing a row by its norm yields the rigid
//! **WorldView** row (verified orthonormal, `det = +1`) and `row[3]/norm` is that
//! axis' view-space translation.
//!
//! The camera View cancels when every part is expressed relative to a **reference
//! part**: `rel_p = WV_ref⁻¹ · WV_p = (Rᵀ_ref·R_p , Rᵀ_ref·(T_p T_ref))` — a pure
//! ship-space rigid transform. That is what [`correlate`] emits and what the
//! checked-in [placement table](parse_table) stores, so the viewer can assemble a
//! ship exactly without re-capturing.
//!
//! ## Pipeline
//!
//! 1. Canary F10 → `xenia_ship_capture.log` (per-draw `vbase`/`vcount` + the first
//! 48 VS float4 constants). [`parse_capture`] → [`CapturedDraw`]s.
//! 2. [`correlate`] matches each ship base part to a draw **by vertex count**
//! (unique per part) and expresses it in the reference part's frame →
//! [`ShipPlacement`].
//! 3. [`serialize_table`]/[`parse_table`] persist it as a checked-in data file
//! (`data/ship_placements.txt`, embedded via [`embedded_placement`]); the viewer
//! prefers it over the static assembler when present.
use crate::mesh::ScenePart;
type M3 = [[f64; 3]; 3];
/// One captured draw's rigid **WorldView**: rotation rows `r` + view-space
/// translation `t`, recovered from the `c0..c2` WVP constants.
#[derive(Debug, Clone, PartialEq)]
pub struct CapturedDraw {
/// Guest vertex-buffer base address (the draw's identity for de-duping).
pub vbase: u32,
/// Vertex count — the key that matches a draw to a decoded part.
pub vcount: u32,
/// WorldView rotation rows (orthonormal).
pub r: M3,
/// WorldView view-space translation.
pub t: [f64; 3],
/// First few LOCAL vertex positions dumped with the draw (buffer order).
/// Used to disambiguate same-vcount twins (mirrored port/starboard parts).
pub pos: Vec<[f32; 3]>,
}
/// A ship part to match against the capture. `part` is the **base** part name
/// (what goes in the placement table); `vcount`/`ref_pos` come from whichever
/// resource variant is being tried (base or an `_m`/`_l` LOD copy — a LOD is the
/// same part in the same local frame, so its captured transform is the part's).
#[derive(Debug, Clone, PartialEq)]
pub struct PartKey {
/// Base part name for the table, e.g. `e106_bdy_01`.
pub part: String,
/// The tried resource's vertex count (the draw match key).
pub vcount: u32,
/// The tried resource's decoded positions (any order — validation is
/// set-based), used to validate a vcount hit and to route mirrored twins.
/// Empty = match by vcount alone.
pub ref_pos: Vec<[f32; 3]>,
}
/// A part's ship-relative rigid placement (in the reference part's frame).
#[derive(Debug, Clone, PartialEq)]
pub struct PartPlacement {
/// Geometry resource name, e.g. `e106_eng_01`.
pub part: String,
/// Rotation rows.
pub m: [[f32; 3]; 3],
/// Ship-relative translation.
pub t: [f32; 3],
}
/// A whole ship's captured placement: every part in a shared ship-local frame.
#[derive(Debug, Clone, PartialEq)]
pub struct ShipPlacement {
/// Ship family id, e.g. `e106`.
pub id: String,
/// The reference part whose frame the placements are expressed in.
pub reference: String,
/// Ship-relative placement of each matched part.
pub parts: Vec<PartPlacement>,
}
/// Parse a Canary ship-capture log into per-draw rigid WorldView transforms.
///
/// Only draws that carry the `c0..c2` constants are returned (a culled/occluded
/// part produces no draw and is simply absent). Robust to the exact spacing of
/// the `DRAW …` / `vsconst …` lines.
pub fn parse_capture(text: &str) -> Vec<CapturedDraw> {
let mut out = Vec::new();
let mut vbase = 0u32;
let mut vcount = 0u32;
let mut pos: Vec<[f32; 3]> = Vec::new();
let mut consts: Vec<(usize, [f64; 4])> = Vec::new();
let flush = |vbase: u32,
vcount: u32,
pos: &mut Vec<[f32; 3]>,
consts: &[(usize, [f64; 4])],
out: &mut Vec<CapturedDraw>| {
let pos = std::mem::take(pos);
if vbase == 0 {
return;
}
let get = |i: usize| consts.iter().find(|(k, _)| *k == i).map(|(_, v)| *v);
let (Some(c0), Some(c1), Some(c2)) = (get(0), get(1), get(2)) else {
return; // no WorldView for this draw — skip it
};
if let Some((r, t)) = normalize_wvp([c0, c1, c2]) {
out.push(CapturedDraw { vbase, vcount, r, t, pos });
}
};
for line in text.lines() {
let l = line.trim();
if let Some(rest) = l.strip_prefix("DRAW ") {
flush(vbase, vcount, &mut pos, &consts, &mut out);
consts.clear();
let f = |k: &str| rest.split_whitespace().find_map(|t| t.strip_prefix(k));
vbase = f("vbase=0x").and_then(|s| u32::from_str_radix(s, 16).ok()).unwrap_or(0);
vcount = f("vcount=").and_then(|s| s.parse().ok()).unwrap_or(0);
} else if l.starts_with("pos:") || l.starts_with("positions:") {
pos = parse_pos_line(l, 8);
} else if l.starts_with("vsconst") {
for cap in l.split('c').skip(1) {
let Some((idx, rest)) = cap.split_once('=') else { continue };
let Ok(i) = idx.trim().parse::<usize>() else { continue };
let nums: Vec<f64> = rest
.trim_start_matches('(')
.split(')')
.next()
.unwrap_or("")
.split(',')
.filter_map(|x| x.trim().parse().ok())
.collect();
if nums.len() == 4 {
consts.push((i, [nums[0], nums[1], nums[2], nums[3]]));
}
}
}
}
flush(vbase, vcount, &mut pos, &consts, &mut out);
out
}
/// Parse a `pos: (x,y,z) (x,y,z) …` dump line into up to `max` positions.
fn parse_pos_line(l: &str, max: usize) -> Vec<[f32; 3]> {
let mut out = Vec::new();
for group in l.split('(').skip(1) {
let Some(inner) = group.split(')').next() else { continue };
let nums: Vec<f32> = inner.split(',').filter_map(|x| x.trim().parse().ok()).collect();
if nums.len() == 3 {
out.push([nums[0], nums[1], nums[2]]);
if out.len() >= max {
break;
}
}
}
out
}
/// The vertex shader capital ships (and the player fighter) are drawn with — the
/// `c0..c2` WVP-row layout [`parse_capture`]/[`parse_drawlog`] rely on.
pub const SHIP_VS_HASH: &str = "0xC7F781F4C1D58054";
/// Parse the **draw-logger** format (`xenia_re_draws.log` / `mission_draws.log`,
/// the `--log_draws` cvar) into per-part rigid transforms — the alternative to the
/// F10 [`parse_capture`] snapshot. That log is what a normal instrumented run
/// already produces, so a capital ship seen in-mission can be baked without a
/// dedicated F10 capture.
///
/// A capital ship's parts each own a **distinct vertex buffer**, so we group by
/// the `stream … base=…` address, take its vertex count as `size_words /
/// stride_words`, and keep the first `c0..c2` WVP seen for that buffer. Only draws
/// with `vs=`[`SHIP_VS_HASH`] are kept (the ship shader), so HUD/skybox draws are
/// ignored. (The player fighter shares ONE buffer across its fin draws and so
/// collapses to a single entry here — fine, capital ships are the target.)
pub fn parse_drawlog(text: &str) -> Vec<CapturedDraw> {
let mut out = Vec::new();
let mut seen: std::collections::HashSet<u32> = std::collections::HashSet::new();
let mut is_ship = false;
let mut base = 0u32;
let mut stride = 0u32;
let mut size = 0u32;
let mut pos: Vec<[f32; 3]> = Vec::new();
let mut consts: Vec<(usize, [f64; 4])> = Vec::new();
let mut flush = |base: u32,
size: u32,
stride: u32,
pos: &mut Vec<[f32; 3]>,
consts: &[(usize, [f64; 4])],
seen: &mut std::collections::HashSet<u32>,
out: &mut Vec<CapturedDraw>| {
let pos = std::mem::take(pos);
if base == 0 || stride == 0 || !seen.insert(base) {
return;
}
let get = |i: usize| consts.iter().find(|(k, _)| *k == i).map(|(_, v)| *v);
let (Some(c0), Some(c1), Some(c2)) = (get(0), get(1), get(2)) else { return };
if let Some((r, t)) = normalize_wvp([c0, c1, c2]) {
out.push(CapturedDraw { vbase: base, vcount: size / stride, r, t, pos });
}
};
for line in text.lines() {
let l = line.trim();
if let Some(rest) = l.strip_prefix("DRAW ") {
flush(base, size, stride, &mut pos, &consts, &mut seen, &mut out);
consts.clear();
base = 0;
stride = 0;
size = 0;
is_ship = rest.contains(&format!("vs={SHIP_VS_HASH}"));
} else if is_ship && l.starts_with("positions:") {
pos = parse_pos_line(l, 8);
} else if is_ship && l.starts_with("stream ") {
let f = |k: &str| l.split_whitespace().find_map(|t| t.strip_prefix(k));
if let Some(b) = f("base=0x").and_then(|s| u32::from_str_radix(s, 16).ok()) {
base = b;
}
stride = f("stride_words=").and_then(|s| s.parse().ok()).unwrap_or(stride);
size = f("size_words=").and_then(|s| s.parse().ok()).unwrap_or(size);
} else if is_ship && l.starts_with('c') {
// `c<idx> x y z w` — the vsconst rows (space-separated).
let mut it = l.splitn(2, char::is_whitespace);
let Some(tag) = it.next() else { continue };
let Ok(i) = tag[1..].parse::<usize>() else { continue };
let nums: Vec<f64> = it.next().unwrap_or("").split_whitespace().filter_map(|x| x.parse().ok()).collect();
if nums.len() >= 4 {
consts.push((i, [nums[0], nums[1], nums[2], nums[3]]));
}
}
}
flush(base, size, stride, &mut pos, &consts, &mut seen, &mut out);
out
}
/// Normalize the three `c0..c2` WVP rows to a rigid WorldView `(R rows, T)` by
/// dividing each row by its (projection-scale) norm. `None` if any row is
/// degenerate.
fn normalize_wvp(rows: [[f64; 4]; 3]) -> Option<(M3, [f64; 3])> {
let mut r = [[0.0; 3]; 3];
let mut t = [0.0; 3];
for (i, row) in rows.iter().enumerate() {
let n = (row[0] * row[0] + row[1] * row[1] + row[2] * row[2]).sqrt();
if n < 1e-6 {
return None;
}
r[i] = [row[0] / n, row[1] / n, row[2] / n];
t[i] = row[3] / n;
}
Some((r, t))
}
/// Validate a vcount hit by the draw's dumped positions against the part's
/// decoded position SET (order-independent — decode order can differ from buffer
/// order). Returns `Some((hits, mirrored))` when at least half the dumped
/// positions are found among the part's vertices, either directly or **all
/// X-negated** — the engine uploads the second of a mirrored port/starboard pair
/// as an X-reflection of the shared file geometry, so the guest buffer disagrees
/// in X sign with every decoded copy. `None` = the dump belongs to a different
/// model (a coincidental vcount).
fn pos_validate(draw: &CapturedDraw, ref_pos: &[[f32; 3]]) -> Option<(usize, bool)> {
if draw.pos.is_empty() || ref_pos.is_empty() {
return Some((0, false)); // no data to validate with — accept neutrally
}
let near = |a: &[f32; 3], b: &[f32; 3]| {
(a[0] - b[0]).abs() <= 1e-2 && (a[1] - b[1]).abs() <= 1e-2 && (a[2] - b[2]).abs() <= 1e-2
};
let mut direct = 0usize;
let mut mirror = 0usize;
for p in &draw.pos {
if ref_pos.iter().any(|v| near(p, v)) {
direct += 1;
}
let pm = [-p[0], p[1], p[2]];
if ref_pos.iter().any(|v| near(&pm, v)) {
mirror += 1;
}
}
let need = draw.pos.len().div_ceil(2);
if direct >= need && direct >= mirror {
Some((direct, false))
} else if mirror >= need {
Some((mirror, true))
} else {
None
}
}
/// Correlate captured draws to a ship's parts and express each in the reference
/// part's frame.
///
/// Each [`PartKey`]'s `vcount` is the match key; when several draws share the
/// vcount (mirrored port/starboard twins) the draw whose dumped positions match
/// the part's `ref_pos` validates best is chosen. `ref_sub` selects the reference part by
/// substring (e.g. `bdy_04`); the first matched part is used if none contains it.
/// Parts with no matching captured draw (culled at that camera angle) are
/// omitted. Returns `None` if nothing matched.
pub fn correlate(
id: &str,
draws: &[CapturedDraw],
parts: &[PartKey],
ref_sub: &str,
) -> Option<ShipPlacement> {
let mut matched: Vec<(String, M3, [f64; 3], bool)> = Vec::new();
let mut used: std::collections::HashSet<u32> = std::collections::HashSet::new();
for key in parts {
// Several PartKeys may carry the same part (one per LOD-variant vcount);
// the first that validates wins, the rest are skipped.
if matched.iter().any(|(p, ..)| p == &key.part) {
continue;
}
// A vcount hit alone can be a coincidence (small LODs share counts across
// unrelated models — a 51-vert draw once matched the bridge but was a
// different mesh). Candidates failing position validation are REJECTED;
// among validated candidates the best hit count wins (routes twins), and
// a mirror-validated match records the X-reflection.
let mut best: Option<(&CapturedDraw, usize, bool)> = None;
for d in draws.iter().filter(|d| d.vcount == key.vcount && !used.contains(&d.vbase)) {
let Some((score, mirrored)) = pos_validate(d, &key.ref_pos) else {
continue; // positions disagree — not this part
};
if best.map_or(true, |(_, s, _)| score > s) {
best = Some((d, score, mirrored));
}
}
if let Some((d, _, mirrored)) = best {
used.insert(d.vbase);
matched.push((key.part.clone(), d.r, d.t, mirrored));
}
}
let ref_idx = matched.iter().position(|(p, ..)| p.contains(ref_sub)).unwrap_or(0);
let (ref_part, ref_r, ref_t, _) = matched.get(ref_idx)?.clone();
let rt_ref = transpose(&ref_r);
let parts_out = matched
.iter()
.map(|(part, r, t, mirrored)| {
let mut rel_r = mmul(&rt_ref, r);
let dt = [t[0] - ref_t[0], t[1] - ref_t[1], t[2] - ref_t[2]];
let rel_t = mat_vec(&rt_ref, dt);
// The captured WorldView transforms the *uploaded* buffer; for the
// mirrored twin that buffer is the X-reflection of the file geometry,
// so the file-local placement is R·diag(1,1,1) — negate column 0.
if *mirrored {
for row in &mut rel_r {
row[0] = -row[0];
}
}
PartPlacement {
part: part.clone(),
m: snap_m3(&rel_r),
t: [rel_t[0] as f32, rel_t[1] as f32, rel_t[2] as f32],
}
})
.collect();
Some(ShipPlacement { id: id.to_string(), reference: ref_part, parts: parts_out })
}
/// Snap near-axis rotation entries (float noise from the WV products) to exact
/// 0/±1 so the checked-in table is clean; real rotations are untouched.
fn snap_m3(m: &M3) -> [[f32; 3]; 3] {
let snap = |v: f64| -> f32 {
if v.abs() < 5e-4 {
0.0
} else if (v - 1.0).abs() < 5e-4 {
1.0
} else if (v + 1.0).abs() < 5e-4 {
-1.0
} else {
v as f32
}
};
[
[snap(m[0][0]), snap(m[0][1]), snap(m[0][2])],
[snap(m[1][0]), snap(m[1][1]), snap(m[1][2])],
[snap(m[2][0]), snap(m[2][1]), snap(m[2][2])],
]
}
/// Convert a captured placement into viewer [`ScenePart`]s (rigid, unit scale).
pub fn to_scene_parts(ship: &ShipPlacement) -> Vec<ScenePart> {
ship.parts
.iter()
.map(|p| ScenePart { resource: p.part.clone(), m: p.m, t: p.t, s: [1.0, 1.0, 1.0] })
.collect()
}
/// The checked-in placement table, embedded at build time. Empty until captures
/// are baked in with `correlate_capture --emit`.
const EMBEDDED_TABLE: &str = include_str!("../data/ship_placements.txt");
/// The captured placement for ship `id` from the embedded table, if baked in.
pub fn embedded_placement(id: &str) -> Option<ShipPlacement> {
parse_table(EMBEDDED_TABLE).into_iter().find(|s| s.id == id)
}
/// Serialize a placement table to the checked-in text format (see [`parse_table`]).
pub fn serialize_table(ships: &[ShipPlacement]) -> String {
let mut s = String::new();
s.push_str("# Capital-ship part placements — runtime-captured ground truth.\n");
s.push_str("# Generated by: cargo run --release --example correlate_capture -- \\\n");
s.push_str("# <capture.log> <Stage_SNN> <ship_id> [ref_part_substr] --emit\n");
s.push_str("# Per line: <part> <R00 R01 R02 R10 R11 R12 R20 R21 R22> <T0 T1 T2>\n");
for ship in ships {
s.push_str(&format!("\nship {} ref={}\n", ship.id, ship.reference));
for p in &ship.parts {
s.push_str(&format!(
" {} {} {} {} {} {} {} {} {} {} {} {} {}\n",
p.part,
p.m[0][0], p.m[0][1], p.m[0][2],
p.m[1][0], p.m[1][1], p.m[1][2],
p.m[2][0], p.m[2][1], p.m[2][2],
p.t[0], p.t[1], p.t[2],
));
}
}
s
}
/// Parse the checked-in placement table. `#` comments and blank lines are ignored;
/// a `ship <id> ref=<part>` line starts a block, and each following
/// `<part> <9 rotation floats> <3 translation floats>` line is one placement.
pub fn parse_table(text: &str) -> Vec<ShipPlacement> {
let mut ships: Vec<ShipPlacement> = Vec::new();
for line in text.lines() {
let l = line.trim();
if l.is_empty() || l.starts_with('#') {
continue;
}
if let Some(rest) = l.strip_prefix("ship ") {
let mut it = rest.split_whitespace();
let id = it.next().unwrap_or("").to_string();
let reference = it
.next()
.and_then(|s| s.strip_prefix("ref="))
.unwrap_or("")
.to_string();
ships.push(ShipPlacement { id, reference, parts: Vec::new() });
} else if let Some(ship) = ships.last_mut() {
let mut it = l.split_whitespace();
let part = it.next().unwrap_or("").to_string();
let nums: Vec<f32> = it.filter_map(|x| x.parse().ok()).collect();
if part.is_empty() || nums.len() != 12 {
continue;
}
ship.parts.push(PartPlacement {
part,
m: [
[nums[0], nums[1], nums[2]],
[nums[3], nums[4], nums[5]],
[nums[6], nums[7], nums[8]],
],
t: [nums[9], nums[10], nums[11]],
});
}
}
ships
}
fn transpose(m: &M3) -> M3 {
[
[m[0][0], m[1][0], m[2][0]],
[m[0][1], m[1][1], m[2][1]],
[m[0][2], m[1][2], m[2][2]],
]
}
fn mmul(a: &M3, b: &M3) -> M3 {
let mut o = [[0.0; 3]; 3];
for i in 0..3 {
for j in 0..3 {
o[i][j] = (0..3).map(|k| a[i][k] * b[k][j]).sum();
}
}
o
}
fn mat_vec(m: &M3, v: [f64; 3]) -> [f64; 3] {
[
m[0][0] * v[0] + m[0][1] * v[1] + m[0][2] * v[2],
m[1][0] * v[0] + m[1][1] * v[1] + m[1][2] * v[2],
m[2][0] * v[0] + m[2][1] * v[1] + m[2][2] * v[2],
]
}
#[cfg(test)]
mod tests {
use super::*;
/// A `vsconst` line for an already-unit WorldView (norm 1) so R = rows and
/// T = row[3]: identity rotation, translation `t`.
fn draw_line(vbase: u32, vcount: u32, t: [f64; 3]) -> String {
format!(
"DRAW vbase=0x{vbase:X} stride=32 vcount={vcount} indices=0 prim=tri vs=0x1\n \
vsconst base=0: c0=(1,0,0,{}) c1=(0,1,0,{}) c2=(0,0,1,{})\n",
t[0], t[1], t[2]
)
}
fn key(part: &str, vcount: u32) -> PartKey {
PartKey { part: part.to_string(), vcount, ref_pos: Vec::new() }
}
#[test]
fn parse_recovers_worldview() {
let log = draw_line(0x1000, 3, [10.0, 0.0, 0.0]);
let draws = parse_capture(&log);
assert_eq!(draws.len(), 1);
assert_eq!(draws[0].vcount, 3);
assert_eq!(draws[0].t, [10.0, 0.0, 0.0]);
assert_eq!(draws[0].r, [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]]);
}
#[test]
fn parse_normalizes_projection_scale() {
// A row scaled by the projection sx=2 must normalize back to unit, and its
// translation divides by the same norm.
let log = "DRAW vbase=0x2000 vcount=4\n \
vsconst base=0: c0=(2,0,0,20) c1=(0,2,0,0) c2=(0,0,1,0)\n";
let d = &parse_capture(log)[0];
assert!((d.r[0][0] - 1.0).abs() < 1e-9);
assert!((d.t[0] - 10.0).abs() < 1e-9, "20/2 = 10");
}
#[test]
fn correlate_expresses_parts_in_reference_frame() {
// Two parts, identity rotation: ref at view (10,0,0), other at (10,0,50).
// Relative to ref, the other must sit at (0,0,50).
let log = format!(
"{}{}",
draw_line(0x1000, 3, [10.0, 0.0, 0.0]),
draw_line(0x2000, 4, [10.0, 0.0, 50.0])
);
let draws = parse_capture(&log);
let parts = vec![key("e106_bdy_04", 3), key("e106_eng_01", 4)];
let ship = correlate("e106", &draws, &parts, "bdy_04").unwrap();
assert_eq!(ship.reference, "e106_bdy_04");
let refp = ship.parts.iter().find(|p| p.part == "e106_bdy_04").unwrap();
assert_eq!(refp.t, [0.0, 0.0, 0.0]);
let eng = ship.parts.iter().find(|p| p.part == "e106_eng_01").unwrap();
assert_eq!(eng.t, [0.0, 0.0, 50.0]);
}
#[test]
fn drawlog_format_parses_capital_ship_parts() {
// Two capital-ship parts, each its own vertex buffer (distinct base),
// stride 6 → vcount = size_words/6. Identity WVP with known translations.
let log = format!(
"DRAW prim=4 indices=6 src=0 index[...] vs={SHIP_VS_HASH}\n \
stream fc=0 base=0x12A60228 stride_words=6 size_words=9798 endian=2 type=3\n \
vsconst:\n c0 1.0 0.0 0.0 10.0\n c1 0.0 1.0 0.0 0.0\n c2 0.0 0.0 1.0 0.0\n\
DRAW prim=4 indices=6 src=0 index[...] vs={SHIP_VS_HASH}\n \
stream fc=0 base=0x12ABF2CC stride_words=6 size_words=4890 endian=2 type=3\n \
vsconst:\n c0 1.0 0.0 0.0 10.0\n c1 0.0 1.0 0.0 0.0\n c2 0.0 0.0 1.0 50.0\n\
DRAW prim=4 indices=3 src=0 index[...] vs=0xDEADBEEF00000000\n \
stream fc=0 base=0x99990000 stride_words=6 size_words=18 endian=2 type=3\n \
vsconst:\n c0 1.0 0.0 0.0 0.0\n c1 0.0 1.0 0.0 0.0\n c2 0.0 0.0 1.0 0.0\n"
);
let draws = parse_drawlog(&log);
// Two ship-shader buffers; the non-ship shader draw is ignored.
assert_eq!(draws.len(), 2);
assert_eq!(draws[0].vcount, 1633); // 9798/6
assert_eq!(draws[1].vcount, 815); // 4890/6
// Correlate: part B sits 50 along Z from reference part A.
let parts = vec![key("e106_bdy_04", 1633), key("e106_bdy_03", 815)];
let ship = correlate("e106", &draws, &parts, "bdy_04").unwrap();
let b = ship.parts.iter().find(|p| p.part == "e106_bdy_03").unwrap();
assert_eq!(b.t, [0.0, 0.0, 50.0]);
}
/// The real e106 bdy_01/bdy_02 case: both twin resources decode to
/// IDENTICAL file geometry; the engine uploads the second instance as an
/// X-reflection, so one captured buffer disagrees in X sign with the file.
/// Both draws must be placed, and the mirrored one must bake the X-flip
/// (negated first matrix column) so file-local geometry lands port-side.
#[test]
fn runtime_mirrored_twin_placed_with_reflection() {
let log = "DRAW vbase=0x1000 stride=24 vcount=426 indices=21 prim=4 vs=0x1\n \
pos: (134.4215,133.8319,-118.1757) (178.8384,85.3847,238.0463)\n \
vsconst base=0: c0=(1,0,0,264) c1=(0,1,0,0) c2=(0,0,1,0)\n\
DRAW vbase=0x2000 stride=24 vcount=426 indices=21 prim=4 vs=0x1\n \
pos: (-134.4215,133.8319,-118.1757) (-178.8384,85.3847,238.0463)\n \
vsconst base=0: c0=(1,0,0,-264) c1=(0,1,0,0) c2=(0,0,1,0)\n\
DRAW vbase=0x3000 stride=24 vcount=558 indices=9 prim=4 vs=0x1\n \
vsconst base=0: c0=(1,0,0,0) c1=(0,1,0,0) c2=(0,0,1,0)\n";
let draws = parse_capture(log);
assert_eq!(draws.len(), 3);
assert_eq!(draws[0].pos.len(), 2);
// Both twins carry the SAME (file) positions — +X side geometry.
let file_pos = vec![[134.4215, 133.8319, -118.1757], [178.8384, 85.3847, 238.0463]];
let parts = vec![
PartKey { part: "e106_bdy_01".to_string(), vcount: 426, ref_pos: file_pos.clone() },
PartKey { part: "e106_bdy_02".to_string(), vcount: 426, ref_pos: file_pos },
key("e106_bdy_04", 558),
];
let ship = correlate("e106", &draws, &parts, "bdy_04").unwrap();
assert_eq!(ship.parts.len(), 3, "both twins + reference placed");
let get = |p: &str| ship.parts.iter().find(|x| x.part == p).unwrap().clone();
// bdy_01 validated directly → the +264 draw, identity rotation.
let a = get("e106_bdy_01");
assert_eq!(a.t, [264.0, 0.0, 0.0]);
assert_eq!(a.m[0][0], 1.0);
// bdy_02 validated as the MIRROR → the 264 draw, X-flip baked in.
let b = get("e106_bdy_02");
assert_eq!(b.t, [-264.0, 0.0, 0.0]);
assert_eq!(b.m[0][0], -1.0, "mirrored twin must negate the X column");
assert_eq!(b.m[1][1], 1.0);
}
/// A draw whose vcount matches but whose position dump disagrees must be
/// REJECTED, not placed — the real false-positive: a foreign 51-vert model
/// matched `e106_brg_01_l` by count alone and put the bridge 2 km off-hull.
#[test]
fn vcount_coincidence_rejected_by_positions() {
let log = "DRAW vbase=0x1000 stride=24 vcount=51 indices=36 prim=4 vs=0x1\n \
pos: (-0.0000,46.2359,-12.9454) (-0.0000,-6.1936,264.8687)\n \
vsconst base=0: c0=(1,0,0,1975) c1=(0,1,0,0) c2=(0,0,1,0)\n\
DRAW vbase=0x3000 stride=24 vcount=558 indices=9 prim=4 vs=0x1\n \
vsconst base=0: c0=(1,0,0,0) c1=(0,1,0,0) c2=(0,0,1,0)\n";
let draws = parse_capture(log);
let parts = vec![
PartKey {
part: "e106_brg_01".to_string(),
vcount: 51,
// The REAL bridge LOD's vertices — disagree with the dump.
ref_pos: vec![[35.2480, 26.0376, 49.2504], [6.0, 55.9632, 41.1370]],
},
key("e106_bdy_04", 558),
];
let ship = correlate("e106", &draws, &parts, "bdy_04").unwrap();
assert!(
!ship.parts.iter().any(|p| p.part == "e106_brg_01"),
"coincidental vcount match must not place the bridge"
);
assert_eq!(ship.parts.len(), 1);
}
#[test]
fn table_round_trips() {
let ship = ShipPlacement {
id: "e106".to_string(),
reference: "e106_bdy_04".to_string(),
parts: vec![
PartPlacement {
part: "e106_bdy_04".to_string(),
m: [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]],
t: [0.0, 0.0, 0.0],
},
PartPlacement {
part: "e106_eng_01".to_string(),
m: [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]],
t: [131.0, -133.0, -132.0],
},
],
};
let text = serialize_table(std::slice::from_ref(&ship));
let back = parse_table(&text);
assert_eq!(back, vec![ship]);
}
#[test]
fn embedded_table_parses() {
// The checked-in data file must always parse (even if empty).
let _ = parse_table(EMBEDDED_TABLE);
}
/// The baked e106 capture (2026-07-26 F10, Stage_S01): all 8 parts placed,
/// port/starboard pair symmetric with the mirror on bdy_02, bridge on the
/// centreline. Guards the checked-in data against accidental edits.
#[test]
fn embedded_e106_is_complete() {
let e106 = embedded_placement("e106").expect("e106 baked in");
assert_eq!(e106.reference, "e106_bdy_04");
assert_eq!(e106.parts.len(), 8, "all 8 e106 parts placed");
let get = |p: &str| e106.parts.iter().find(|x| x.part == p).unwrap();
// Port/starboard hull pair: X = ∓264, the starboard copy mirrored.
assert!((get("e106_bdy_01").t[0] + 264.0).abs() < 0.1);
assert!((get("e106_bdy_02").t[0] - 264.0).abs() < 0.1);
assert_eq!(get("e106_bdy_02").m[0][0], -1.0);
assert_eq!(get("e106_bdy_01").m[0][0], 1.0);
// Bridge: centreline, above and aft of the hull reference.
let brg = get("e106_brg_01");
assert!(brg.t[0].abs() < 0.1 && brg.t[1] > 150.0 && brg.t[2] < -160.0);
}
}

View File

@@ -613,7 +613,7 @@ pub struct ShipRow {
}
/// The Ships browser state — capital ships assembled from XBG7 part families.
#[derive(Resource, Default)]
#[derive(Resource)]
pub struct ShipBrowser {
pub open: bool,
pub loading: bool,
@@ -624,11 +624,27 @@ pub struct ShipBrowser {
pub rows: Vec<ShipRow>,
/// Family id of the ship currently rendered (for row highlight).
pub selected: Option<String>,
/// Also place the approximate external parts (bridge / shield gen / engines at
/// their `GN_*` hardpoint frames). Off by default → the exact hull only.
/// Also place the external parts: bridge / shield generators / the engine
/// cluster rig / cross-mounted turrets. EXACT (capture-validated) since the
/// node-TRS fix, so ON by default; off = bare hull bodies only.
pub show_external: bool,
}
impl Default for ShipBrowser {
fn default() -> Self {
Self {
open: false,
loading: false,
loaded: false,
filter: String::new(),
faction: String::new(),
rows: Vec::new(),
selected: None,
show_external: true,
}
}
}
/// Ask the loader to scan the stage containers and build the ship catalog.
#[derive(Event, Default)]
pub struct RequestShipCatalog;
@@ -3630,6 +3646,40 @@ fn handle_voice_request(
});
}
/// A simple exhaust-plume cone: base ring at the frame origin opening toward
/// **Z** (the jet frames face aft), apex trailing behind. Stands in for the
/// game's engine-exhaust FX so assembled ships visually read as powered.
#[cfg(not(target_arch = "wasm32"))]
fn exhaust_cone_mesh() -> sylpheed_formats::mesh::GameMesh {
const SEG: usize = 12;
const R: f32 = 22.0;
const LEN: f32 = 140.0;
let mut positions = Vec::with_capacity(SEG + 2);
let mut normals = Vec::with_capacity(SEG + 2);
positions.push([0.0, 0.0, LEN]); // apex — the jet frame flips Z, sending it aft
normals.push([0.0, 0.0, -1.0]);
for s in 0..SEG {
let a = s as f32 / SEG as f32 * std::f32::consts::TAU;
positions.push([a.cos() * R, a.sin() * R, 0.0]);
normals.push([a.cos(), a.sin(), 0.3]);
}
positions.push([0.0, 0.0, 0.0]); // base centre (cap)
normals.push([0.0, 0.0, 1.0]);
let mut indices = Vec::new();
for s in 0..SEG as u32 {
let (a, b) = (1 + s, 1 + (s + 1) % SEG as u32);
indices.extend_from_slice(&[0, b, a]); // side
indices.extend_from_slice(&[(SEG + 1) as u32, a, b]); // cap
}
sylpheed_formats::mesh::GameMesh {
positions,
normals,
uvs: Vec::new(),
indices,
name: Some("exhaust".to_string()),
}
}
/// Duration (seconds) of a PCM WAV from its `fmt`/`data` chunks.
#[cfg(not(target_arch = "wasm32"))]
fn wav_duration(path: &Path) -> Option<f32> {
@@ -3860,8 +3910,11 @@ fn build_ship_catalog(source: &SourceKind) -> Vec<ShipRow> {
};
let label = format!("S{n:02}");
for sm in ships_in_container(&bytes) {
// Skip single-piece props / debris — not "whole ships".
if sm.parts.len() < 2 {
// Skip single-piece props / debris — not "whole ships" — and any
// family with no composite scene graph (fighter morph sets, shared
// turret parts): those have no placement data and would render as a
// pile at the origin with stray far-away pieces.
if sm.parts.len() < 2 || !sm.has_composite {
continue;
}
let v = vmap.get(&sm.id);
@@ -3952,8 +4005,10 @@ fn build_ship_model(
};
let should_cancel = || cancel.load(Ordering::Relaxed);
// Scene-graph placement: which resources to draw and where (bridge fore,
// engines aft, hardpoints on their frames).
// Placement: fully static scene-graph assembly — exact, validated
// part-for-part (translations + rotations + instances + mirror) against the
// e106 runtime capture (`ship::tests::static_assembly_matches_runtime_capture`).
// The captured table in ship_capture stays as the verification oracle only.
let placed = sylpheed_formats::ship::assemble_ship(&bytes, id, external);
if placed.is_empty() {
return PreparedXpr::Info(format!("No parts found for {label}."));
@@ -3986,6 +4041,12 @@ fn build_ship_model(
let Some(src) = base.iter().find(|m| m.name == p.resource) else {
continue;
};
// A captured mirror placement (det < 0 — the engine X-reflects the
// second of a port/starboard pair) flips triangle winding; reverse each
// triangle's index order so front faces stay outward.
let det = p.m[0][0] * (p.m[1][1] * p.m[2][2] - p.m[1][2] * p.m[2][1])
- p.m[0][1] * (p.m[1][0] * p.m[2][2] - p.m[1][2] * p.m[2][0])
+ p.m[0][2] * (p.m[1][0] * p.m[2][1] - p.m[1][1] * p.m[2][0]);
let mut m = src.clone();
for sub in &mut m.meshes {
for v in &mut sub.positions {
@@ -3994,6 +4055,11 @@ fn build_ship_model(
for nrm in &mut sub.normals {
*nrm = rot(&p.m, nrm);
}
if det < 0.0 {
for tri in sub.indices.chunks_exact_mut(3) {
tri.swap(1, 2);
}
}
}
models.push(m);
}
@@ -4001,6 +4067,26 @@ fn build_ship_model(
return PreparedXpr::Info(format!("No geometry decoded for {label}."));
}
// Exhaust markers: the game's visible "thrusters" are FX drawn at the
// GN_Jet/GN_SJet frames (the engine GEOMETRY is recessed in the hull —
// capture-verified). Draw a simple cone at each frame so the assembled ship
// reads correctly.
if external {
for (i, f) in sylpheed_formats::ship::exhaust_frames(&bytes, id).iter().enumerate() {
let mut cone = exhaust_cone_mesh();
for v in &mut cone.positions {
*v = f.apply(*v);
}
for nrm in &mut cone.normals {
*nrm = rot(&f.m, nrm);
}
models.push(Xbg7Model {
name: format!("__exhaust_{i}"),
meshes: vec![cone],
});
}
}
// Assemble, then relabel with the ship's display name.
match prepare_ship(&bytes, &models) {
PreparedXpr::Model {

View File

@@ -1590,7 +1590,7 @@ fn draw_ships_ui(
}
});
ui.horizontal(|ui| {
ui.checkbox(&mut ships.show_external, "Show external parts");
ui.checkbox(&mut ships.show_external, "External parts (bridge/engines/turrets)");
ui.label(
egui::RichText::new("(bridge / shield gens / engines — approximate placement)")
.weak()

View File

@@ -0,0 +1,71 @@
# Handoff — capital-ship placement: static assembly EXACT (2026-07-26)
Branch `feature/ipfb-idxd-parser`, commits `c6ca5ce..ea32e77`. Canary fork
branch `capture-ship-placement`, tip `877163792`.
## Outcome
**Capital ships assemble exactly from the ISO alone — no captures needed.**
Runtime captures served as the oracle that cracked the static encoding and now
remain as verification tooling only.
## The two format discoveries (docs/re/ship-placement-runtime-capture.md)
1. **Node-TRS off-by-one** (`mesh.rs::read_trs9`): an XBG7 scene-graph node's
joint table holds NINE keyframe channels `[TX TY TZ RY RX RZ SX SY SZ]`
behind EIGHT pointer slots shifted one track forward — channel k+1 =
`f64@(ptr[k]+8)`, channel 0 (TX) = `f64@(ptr[0]0x48)`. The old read never
saw TX (why hulls stacked on the centreline) and misassigned TY/RY.
2. **Euler order** (`mesh.rs::node_rotation`): `Ry(ch3)·Rx(ch4)·Rz(ch5)`,
angles direct — pinned by decomposing the captured engine-nacelle rotation
`Rx(15°)·Rz(30°)` element-for-element.
## assemble_ship (ship.rs) — fully static, capture-validated
- Every composite-node instance (alias duplicates collapsed), including
cross-id turret mounts (`rou_e303_wep_01_root` ×2 on e106; e108 places 7).
- Engine cluster rig `e_rou_<id>_eng` mounts at `GN_Engine_01`
(two mirrored nacelles + centre engine).
- `brg`/`sld` at their exact GN frames (frames were always exact).
- Shared-geometry ±TX twins: the instance whose offset opposes the geometry's
dominant side draws X-reflected (viewer reverses winding on det<0).
- `exhaust_frames()`: the `GN_Jet`/`GN_SJet` frames — the game renders its
engine-exhaust FX there; the viewer draws marker cones at them (the engine
GEOMETRY is recessed in the hull by design; without the FX ships read
engine-less — that was the "engines inside the hull" report).
## Validation chain (all green)
- `ship::tests::static_assembly_matches_runtime_capture` (SYLPHEED_ISO): static
== the baked e106 capture, all 8 parts, T<1.0 R<0.02, nacelles ×2, turrets
×2, starboard mirror.
- `examples/capture_verify.rs` over the user's 5 multi-angle snapshots
(43 e106 instances): dominant clusters match static to ~1 unit on every part
incl. both nacelles. Run:
`cargo run --release --example capture_verify -- <Stage.xpr> e106 <logs…>`
- Offline audit (`examples/ship_audit.rs`) across all stages: no outliers among
composite ships; composite-less families (fighter morph sets, turret/prop
parts) are filtered from the viewer's Ships catalog.
- 81 formats-lib + all disc tests; viewer builds.
## Canary instrumentation (branch `capture-ship-placement`)
F10 → `xenia_ship_capture_NN.log` per press (multi-angle in one run), de-duped
by (vertex-buffer, c0..c2 WVP hash) so every instance of every part is
recorded. Launch:
```
cd xenia-canary/build/bin/Linux/Release
./xenia_canary "<…/Project Sylpheed - Arc of Deception (USA, Europe) (En,Ja).iso>"
```
## Open / next
- Verify remaining capital ships against the existing 5 snapshots (they contain
every on-screen ship): e105 (its engine block assembles dorsally — the one
placement to confirm), f105/f106/f101, e102. Same capture_verify invocation
with the other ship id.
- AA-gun models at `GN_AAGun*` frames need the Vessel recipe binding.
- SJet (maneuvering-vernier) marker orientation is approximate; main aft plumes
verified.
- `_mov` node animation keys (hull-opening sequences) — rest pose rendered now.
- Viewer GUI verification by the user (exhaust cones, catalog cleanup).

46
docs/re/BACKLOG.md Normal file
View File

@@ -0,0 +1,46 @@
# RE backlog
Open items that are *not* being worked right now. Each entry says what is wrong or
unknown, what evidence exists, and what the first step would be. Move an item into
`INDEX.md` (with a `structures/…md` or a parser + test) once it is actually settled.
---
## Capital ships assemble wrong in the viewer
**Reported:** 2026-07-30, by the user. **Status:** ❔ open, not investigated.
The reborn viewer builds capital ships from the split XBG7 parts via
`sylpheed-formats::ship::assemble_ship`, and they come out **wrong** — parts in the
wrong place / wrong orientation.
**Why this is a real finding and not a known limitation:** the RE write-up
[`ship-placement-runtime-capture.md`](ship-placement-runtime-capture.md) declares
static assembly ✅ **exact** as of 2026-07-26 — 9-channel joint tables
`[TX TY TZ RY RX RZ SX SY SZ]`, Euler `Ry·Rx·Rz`, with
`ship::tests::static_assembly_matches_runtime_capture` asserting static == runtime
capture (T < 1.0, R < 0.02). So either the viewer is not using that path, or the
claim generalises worse than the test suggests.
**The likely gap:** that test is **one ship** — the `e106` destroyer, 8 parts plus
two nacelles, two turrets and the hull mirror. Nothing pins the other classes.
Rules that were derived from `e106` and could easily be `e106`-specific:
- the engine cluster rig mounted at `GN_Engine_01` (two mirrored nacelles + centre);
- "X-reflect the shared-geometry twin whose lateral offset opposes the geometry's
dominant side" — a heuristic, not a decoded flag;
- cross-id turret instancing (×2).
**First step (the oracle already exists):** re-run the runtime capture on a *different*
capital ship and diff static vs captured, exactly as `e106` was done — F10 in the
`capture-ship-placement` build of `xenia-canary-native` dumps the ship shader's
`c0..c2` WorldViewProjection rows per part; `WV_ref⁻¹ · WV_p` is the ship-space rigid
transform, which is ground truth. Pick a class whose rig differs from `e106`
(different engine count, a ship with no `sld`, a carrier). Then extend
`static_assembly_matches_runtime_capture` into a per-ship table so a regression in one
class cannot hide behind `e106` passing.
**Also worth ruling out first, cheaply:** that the viewer's own transform stack (scale,
handedness, node-instance recursion) is not re-breaking a correct assembly — compare
the viewer's placement against `assemble_ship`'s output directly before blaming the
format layer.

View File

@@ -21,6 +21,20 @@ Promote to a prose `structures/…md` file when a format needs behavioural notes
| Fonts (ttf/otf/ttc) | ✅ | `sylpheed-formats/src/font.rs` | standard OpenType, parsed via ttf-parser |
| XBG7 mesh | 🟡/❔ | `sylpheed-formats/src/mesh.rs` + `tests/mesh_disc.rs` ([xbg7](structures/xbg7-mesh.md)) | weapons/props: declaration-driven variable stride (36 models), GPU-confirmed. **Stage containers: 5662 sub-models across 22 stages** via content-anchored grouped pools (`stage_models`). Quantized hero bodies (DeltaSaber `f004`) still declined |
| Capital-ship part placement | 🟡 | `sylpheed-formats/src/ship.rs` (static) + [runtime capture](ship-placement-runtime-capture.md) | hull placement static-exact; external parts approximate statically. **Runtime capture** (Canary F10 → VS-constant WorldView) gives ground truth — validated on `e106` destroyer; not yet baked into the viewer |
| Weapon fields defaulted on disc | ✅ | [runtime struct](structures/weapon-struct-runtime.md) · [DATA SHEET route](weapon-datasheet-runtime.md) | **Solved.** Canary maps guest RAM into `/dev/shm`, so the parsed `Weapon`/`Shell` objects are readable live; their layout is solved against disc ground truth (zero contradictions over 100+ records). All 126 weapons, exact numbers, no story progress needed — [4 393 values](captures/weapon-runtime-fields.csv) the disc does not carry. Supersedes the letter-bucket limit of the DATA SHEET route, which now serves as the independent cross-check |
| Unit (craft/vessel) fields defaulted on disc | ✅/🟡 | [runtime struct](structures/unit-struct-runtime.md) | The parsed `unit\UN_*.tbl` definition object, vtable `0x820af844`, ≥`0x380` bytes, one per unit — **discovered, not assumed** (`unit_discover.py`), and distinguished from the spawned-entity class `0x820af030` by being one-per-ID and byte-constant within a run. Across runs only pointer words move — `--crosscheck` proves **no reported field offset is run-dependent** (two words, `+0x2c8`/`+0x2d0`, are stage-dependent and remain unidentified). 27 fields ✅ (21 units, 7 runs); the `Maneuver` block is **schema declaration order, 4 bytes/field, base `0x9c` with a two-slot gap after `AA_Roll_Min`** (29 anchors, 0 conflicts), which also pins 5 fields *no* disc record ever values. Angles are **radians at runtime, degrees on disc**. Unlike weapons, unit definitions are instantiated **per stage**, so coverage (21/110) grows by visiting missions — [values](captures/unit-runtime-fields.csv) |
| UI screen layout (`.rat`) | ✅/🟡 | [ui-rat-layout](structures/ui-rat-layout.md) | One pak per UI screen; each RATC = one (context × language) build; every `<name>.t32` sprite has a `<name>.rat` **layout record** (BE u32; 1280×720 design space; scale/tint/X/Y, keyframes for animated elements, `opt ` link to the focused state). **The tutorial PAUSE menu and the title main menu both rebuild pixel-accurately from the disc.** `loop1.rat` (screen-level draw order) not yet decoded |
## Runtime / dynamic-capture technique
| Technique | Conf. | Spec | Notes |
|-----------|-------|------|-------|
| Live guest-memory read | ✅ | [`tools/re-capture/gmem.py`](../../tools/re-capture/gmem.py) | Canary backs the guest address space with `/dev/shm/xenia_memory_*`; guest VAs map in through Xenia's fixed table. Full-RAM search ~0.2 s (sparse, `SEEK_DATA`). No debugger, no emulator patch, game keeps running |
| IDXD object layout solver | ✅ | [`tools/re-capture/weapon_runtime.py`](../../tools/re-capture/weapon_runtime.py) | Scan RAM for a class's vtable → enumerate its objects → brute-force `(field, offset, encoding)` against the disc records. Accepts a binding only on **zero** contradictions. Generalizes to any IDXD-backed definition |
| Live entity state, anchored on the definition | ✅ | [`tools/re-capture/own_state.py`](../../tools/re-capture/own_state.py) · [autopilot](autopilot-memory-driven.md) | An undamaged craft holds its definition's own numbers, so a *solved definition field* locates the matching live field without a value scan: definition `HP` (1500) → **hull at `position+0x154`**, confirmed by a trace across a death (30/60/90 per hit, negative at 0). Reusable for any live counter whose maximum the definition carries |
| Mission / escort state, every entity's hull | ✅ | [`tools/re-capture/mission_state.py`](../../tools/re-capture/mission_state.py) · [escort state](mission-escort-state.md) | `hull = position + 0x154` is a property of the **entity class**, not of the player object: at t=0 it equals each entity's own definition `HP` across 7 classes and 5 distinct HP values (turret 100, fighter 500, destroyer 10000, cruiser 30000, **ACROPOLIS 25000**), falls under fire (780 damage events in 240 s), goes negative at death, and the object then leaves the heap. So an escort objective is scoreable live — `UN_f101_TCAF_Acropolis` measured at 25000 → 23038 over 240 s, attack starting only at t≈170 s. `REMAINING OB` counts objectives, not hostiles (012 on the HUD vs 118 live ADAN); its address is still ❔ |
| In-flight control mapping | ✅/🟡 | [`tools/re-capture/fire_probe.sh`](../../tools/re-capture/fire_probe.sh) · [controls](flight-controls-runtime.md) | Measured by holding each pad input and photographing the HUD ammo counters: **`RB` = nose gun** (6000→5956 in 4 s, ~11 rounds/s, HEAT rises), **`Y` = main mount** (missiles, 300→299), d-pad = **tactical map** overlay, nothing else moves a counter. No target-cycle input exists — the `TARGET` marker is present with nothing pressed, so targeting is automatic and a missile lock is **time-on-target**. That, not target choice or ballistics, is what caps lethality at 2 kills per 98 missiles |
| Input → dynamics calibration | ✅ | [`tools/re-capture/ctrl_probe.py`](../../tools/re-capture/ctrl_probe.py) · [`binq.py`](../../tools/re-capture/binq.py) | Hold each pad input in turn and measure the craft's speed as displacement/s of its own position triple — no speed field needed first. Settled the throttle: **`RT` accelerates, `LT` brakes, and the setting persists** (488 → 1510 → 174 units/s), overturning an earlier field-scan conclusion |
## Functions / code paths

View File

@@ -0,0 +1,299 @@
# Memory-driven autopilot — build log and current state
**Status: 🟢 IT FLIES, KILLS AND SURVIVES — but it loses the mission anyway.**
Updated 2026-07-30. `pilot.py` flew Stage 02 for **300 s with the hull untouched
at 1500/1500** and scored the first confirmed autopilot kill (`YOU KILLED
WARPLANES 0001` on the HUD, screenshots `shots/pilot1-*.png`); the scene's
hostile count fell from 134 to 111 over the run. The day before, every run was
dead inside 35 s. What is still missing is the *end* of a mission: the objective
counter (`REMAINING OB`) rises as new waves spawn, and nothing yet tracks which
targets actually close it out — and the second run proved the point the hard
way: `GAME OVER` with the hull at 1500/1500, because Stage 02 is an **escort**
and the ACROPOLIS was sunk while the pilot chased fighters two kilometres away.
## 2026-07-30 — the numbers survival needs
Three things the loop was missing were measured this session, each by
consequence rather than by reading a field and hoping.
### Hull is `position + 0x154` ✅ CONFIRMED
The unit definition already had `HP` solved at `+0x054`
([unit-struct-runtime](structures/unit-struct-runtime.md)); the Delta Saber's is
**1500**. A craft that has taken no damage must therefore *contain that number*,
which turns "find the HP field" into a two-float lookup rather than a value scan
(`own_state.py`). It appears once in the entity object, at `pos+0x154`, and the
trace across a death settles it (`ctrl_probe.py` capture, `binq.py trace`):
```
t phase hull
0.00 base 1500.00 <- == definition HP
23.25 rest_A 1380.00 <- first hit, -120
26.68 … 27.18 B 1320 … 930 <- seven hits in 0.5 s
31.93 X 150.00
35.21 Y -30.00 <- goes negative
35.26 Y -180.00 -> GAME OVER on screen
```
Damage arrives in 30/60/90-point steps and the field goes *negative* at death,
so it is the raw hull counter, not a clamped display value. **1500 hull lost in
12 s** of sitting in a turret's line of fire is the whole reason every earlier
run died.
### Shield is `position + 0x430` 🟡 PROBABLE — not yet confirmed live
Same anchor trick: the definition's shield `MaxValue` is **400** and
`ChargeSpeed` **25**, and the entity object holds `400.0` at `+0x430`, `+0x434`
and `+0x438`, with `25.0` at `+0x448`. Which of the three is the *current* value
is unproven — the capture that spanned the death used a ±0x400 window and
cropped them out. `ctrl_probe.py` now samples ±0x800.
### `RT` accelerates, `LT` brakes, and the throttle is a *setting* ✅ CONFIRMED
`ctrl_probe.py` holds each input in turn and measures the craft's own speed as
displacement per second from the position triple, so no speed field is needed.
Distance flown / phase duration, one 3 s hold each, sticks neutral:
| phase | speed (units/s) | | phase | speed (units/s) |
|---|---|---|---|---|
| base (no input) | 488 | | A | 287 |
| **RT** | **1510** | | B | 139 |
| rest after RT | 1056 | | X | 125 |
| **LT** | **174** | | Y | (dying) |
| rest after LT | 428 | | LB / LS / RS / RY / RX / dpad | no effect |
RT triples the speed, LT cuts it to a third, and **the braked state persists**:
after the LT phase the craft sat at 125140 units/s with the sticks and triggers
neutral for the remaining 40 s, and nothing but RT brought it back. So these are
a throttle setting, not a momentary boost — which also means a control loop must
send only the *changes*.
This **corrects** the earlier note in this file ("`RT` is *not* the throttle,
and no button tested is"). That conclusion came from `findspeed.py`, which
assumed the control and went looking for a *field* that rose; measuring the
speed directly reverses it.
### Two method corrections
* **Pick the attitude block by the flight path, not by address order.** The
player object contains **20** orthonormal 3×3 blocks (identity frames, bone
or camera frames), and `pos-0x70` and `pos-0x30` hold the *same* matrix.
Taking `found[0]` wrote a config with `rot_delta = -0x764` and a nonsense
forward axis; `entities2.py self` now scores every block against the measured
direction of travel and picks the best (`cos = +1.000`, row 2, sign +1).
* **The entity-heap scan has to be numpy.** A per-word Python loop over the
16 MB entity region costs seconds per scan, which is the whole budget of a
10 Hz control loop; `np.isin` over a `>u4` view is milliseconds.
### Stage 02 as an autopilot testbed (from the in-flight HUD)
`OBJECTIVE: shoot down all invading enemy fighters while watching out for
attacks on the ACROPOLIS` · `DEFEAT: your fighter is shot down, or the ACROPOLIS
is sunk` · `HINT: you can resupply at the ACROPOLIS`. The HUD shows
**`REMAINING OB 004`** — only four objective targets — so this mission is
winnable by an autopilot that survives. It also shows separate **SHIELD** and
**ARMOR** bars (matching a 400-point shield over 1500 hull), `A/B 7,635`
afterburner, and `NOSE BM 06000` / `MAIN MPM 00300` ammo.
## What the loop did before that, observed
```
[ 82.5] tgt=e007_ADAN_Turret d=3384 yaw= -7.3 pit=+14.6 stick=(-0.13,-0.34) fire=0
[ 85.7] tgt=e007_ADAN_Turret d=2797 yaw= +1.9 pit=+27.2 stick=(+0.20,-0.59) fire=0
[117.3] tgt=e007_ADAN_Turret d=4211 yaw= +5.7 pit= +9.0 stick=(+0.25,-0.40) fire=1
```
Distance closes monotonically, yaw error is driven from 8° to ~0, and once both
errors are inside the firing cone it holds RB and the ammo counter falls. A
rescan reports the scene as e.g. `136 entities {'TCAF': 16, 'ADAN': 120}`.
**The chain that made it work** — each link checked, not assumed:
1. **Entity typing.** A live entity's definition pointer sits at
**position + 0x130**. One heap scan then yields every craft *with its unit
type*, which is what separates 20-odd real combatants from ~30 000 moving
particles. Verified by the result being coherent: wingmen, enemy turrets and
attackers, friendly capital ships, and exactly one `…_Player`.
2. **Orientation.** A 3×3 rotation at **position 0x70**, stored with a
**16-byte row stride** (a 4×4 transform whose translation row *is* the
position). An earlier search for nine *contiguous* floats structurally could
not find this, which is why the first pass concluded "no transform". The
binding is confirmed independently: its row 2 matches the craft's measured
direction of travel with **cos = +1.000**.
3. **The fire button is RB** — established by consequence, not by guessing:
of RB/LB/A/B/X/Y/RT/LT, pressing RB is the only one that makes the nose-ammo
counter in RAM fall (5958 → 5940). (This entry also claimed `RT` is *not* the
throttle — **wrong**, see the 2026-07-30 measurement above.)
4. **Control.** PD on the aiming error with the derivative taken from the
craft's own body angular velocity (from two consecutive rotation matrices),
and target selection weighted by off-boresight angle
(`score = d·(1 + 3·(θ/π)²)`) rather than pure nearest — closing on a target
90° off the nose only raises the bearing rate, which is what held the first
run outside its firing cone at a steady ~27° pitch error.
## Goal
Fly and fight a mission by reading the game's own world state out of guest RAM
and driving the pad from it — the RE payoff being an oracle for the
reimplementation's flight model and AI, and a way to reach missions the save
cannot otherwise reach (unit coverage for
[unit-struct-runtime](structures/unit-struct-runtime.md) is capped at 21/110
because definitions load **per stage**).
## What works (verified)
| Piece | Tool | Evidence |
|---|---|---|
| Live guest-RAM reads at loop rate | `gworld.py` | `/dev/shm` file opened once, `pread` per tick; a whole-RAM scan is ~6 s, a targeted read is microseconds |
| Whole-RAM float scanning | numpy over `SEEK_DATA` extents | 1 270 orthonormal 3×3 blocks located in 6.2 s |
| Entity enumeration by unit type | `gworld.py entities` | 116 live instances in Stage 02, typed by unit ID, incl. exactly one `…_Player` |
| Pad control at loop rate | `flight_probe.Pad` | writes command lines straight into the vgamepad FIFO; the `vgamepad` CLI spawns a process per command and its `tap`/`hold` sleep *inside* the server, so neither is usable in a control loop |
| Unattended mission entry | `launch_mission.sh` | title → LOAD GAME → slot 01 → READY ROOM → TAKE OFF → in flight, repeatable |
| Hangar loadout | `launch_mission.sh --hangar` | the "Recommended" control is **AUTO SELECT — "Mount most suitable weapons"**, already the default cursor position. At 5 % progress it is a no-op: only two weapons are developed, and they are already mounted |
| Finding moving objects | `findplayer.py` | position triples recovered from motion alone — straight-line, constant-speed filter over K whole-RAM samples |
## What is NOT solved
**Survival, and therefore mission completion.** The loop has no evasion, no
shield/armour awareness and no throttle control, so it flies a straight pursuit
into defended space and is eventually shot down — every long run so far has
ended in GAME OVER. Completing a mission needs, at least: reading own
shield/armour, breaking off when hit, and prioritising the mission's actual
objective targets over the nearest turret.
### Superseded (kept because the reasoning still matters)
The notes below were written before the chain above worked. They remain true as
statements about `0x820af030`, which is *not* the live entity —
1. **The `0x820af030` class is not the live entity.** It has one object per
spawned thing and carries the unit-ID string, so it looked like the entity
list — but over a 29 s in-flight capture **all 384 words of it are
constant** (`whatchanges.py`). It is a static spawn record. The earlier
claim in `unit-struct-runtime.md` that this class is "the spawned entity
instance — live state" is **wrong in the second half**: it is per-spawn, but
it is not live state. The parts of that document that depend on the
*definition* class `0x820af844` are unaffected.
2. **No transform in or one hop from that object**: no orthonormal 3×3, and no
unit quaternion, within `0x2000` of it or behind any of its 121 pointers.
3. **Input correlation finds *a* self-object, but not obviously the craft.**
Holding hard-left then hard-right yaw and looking for a position whose turn
axis reverses (`findself.py`, `selfstate.py`) gives clean hits
(`cos ≈ 0.99`), but they cluster at `0x40009xxx` in what looks like an
8-corner box with ±45 000 coordinates — a camera/skybox volume that follows
the player, not the craft. Its speed (≈359/s) is suspiciously close to the
HUD's 350, which supports "follows the player" but is not proof of identity.
4. **Entity typing is unavailable**: no definition pointer within ±0x800 of the
self-position, so the trick of learning one object's layout and applying it
to all the others has nothing to anchor on. Without typing, the 33 418
moving triples in a firefight cannot be separated into enemies, friendlies
and bullets, so there is nothing to aim at.
## Dead ends, recorded so they are not re-run
* **Speed-scan for the player object** (`findspeed.py`, the classic two-state
value scan: coast → boost → coast). Sound method, but **`RT` is not the
throttle** — 5 864 floats matched the cruise speed and none rose. The control
actually bound to acceleration was never established, and the run that would
have established it ended in GAME OVER.
* **Comparing orientation matrices 2 s apart.** At a real turn rate that is far
outside the small-angle regime, so the skew part of `A·Bᵀ` is not the rotation
vector and the "angular velocity" comes out as ~30 000. Sample incrementally
(6 Hz) and re-check orthonormality on every read — blocks found by a scan get
overwritten between the scan and the read.
## The second run lost the mission **without being hit** (2026-07-30)
A second 240 s flight, with the two fixes above, ended on the `GAME OVER`
screen — while the hull read **1500/1500 on the last live tick**. Nothing shot
us down. The other defeat condition fired: *the ACROPOLIS is sunk*. The HUD had
been showing a red `WARNING` banner for a while, and the pilot spent the whole
run pursuing an `e010_ADAN_Attacker_S` two kilometres away.
So surviving is necessary and not sufficient, and "nearest hostile fighter" is
the wrong objective function for this stage. **The mission is an escort.** What
follows:
* **Prioritise hostiles by their distance to the protected asset, not to us.**
The attackers worth killing are the ones closing on the ACROPOLIS.
* **The protected asset's health is readable with the same anchor as ours** —
hull at `position + 0x154`, its maximum being its own definition's `HP`. That
gives a live "are we winning" signal for the escort, and it should drive the
target choice directly.
* A frozen tail in the log (identical position, speed and target for the last
five seconds) is what mission-end looks like from the outside, **not** an
emulator wedge. Worth knowing before diagnosing the wrong thing.
* Practical: do **not** pipe a long run's log through `tail` — that discards
everything but the end, and the interesting part of this run is gone.
## After survival, the blocker is lethality (2026-07-30)
The 300 s run took **no damage at all** and killed **one** warplane, spending
~800 rounds of nose ammo (`06000``05193`) to do it, while `REMAINING OB` rose
from `004` to `011` as fresh waves spawned. So attrition at this rate never
finishes the mission, and the ranking of open problems has changed:
1. **Hit rate.** It opens fire at 25 km with a 9° cone and a crude lead
(`p + v·d/speed`, no projectile speed). The `Shell` records in
[weapon-struct-runtime](structures/weapon-struct-runtime.md) carry the real
projectile speed and `MaximumRange` per weapon — the lead and the firing
range should come from *those*, not from constants.
2. **Which targets count.** `REMAINING OB` is the mission's own objective
counter and it is on screen, so it is in RAM; finding it turns "shoot
whatever is nearest" into "shoot what closes the mission". Objective-marked
entities also draw an `OB` badge in the HUD, so the flag is likely a word in
the entity object.
3. **Confirming the shield word** — needs a run that actually takes damage; the
pilot is now good enough at avoiding that to make it awkward, so drive
straight at a turret on purpose with `--dry` steering disabled.
4. **Does the ACROPOLIS repair?** RETIRE mode has never triggered (the hull
never fell), so the resupply hint is still untested.
## The next step that unblocks the most (superseded — kept for the reasoning)
**Update 2026-07-30: this is no longer the blocker.** Entity typing via the
definition pointer already solved target selection, so the game's own target
pointer is now a convenience rather than a prerequisite. It would still be the
cheapest route to problem 2 above (objective targets), because whatever the HUD
locks on to is what the game itself considers a target.
**Find the game's own target pointer instead of typing entities ourselves.**
The HUD has a lock-on system (a `TARGET` marker and a target-cycle button), so
a global almost certainly holds a pointer to the currently-targeted entity.
Reading that gives an enemy's live object address directly — which yields both
target selection *and* the entity layout (position offset within it), i.e. it
collapses problems 3 and 4 into one. It is also cheap to find: cycle the target
with the pad and watch which pointer-shaped global changes in step.
## Operational notes
* An unattended craft **dies** — the ship flies straight into a firefight, and
two long scans were invalidated by a GAME OVER mid-run. Any scan longer than
~30 s needs either a survivable holding pattern or a fresh mission.
* `Xvfb` and the emulator die on their own every few minutes here, cleanly
(exit 0), cause unidentified. Everything that must not be interrupted is run
as **one background task** that starts the display, the emulator, the
navigation and the measurement together, so nothing has to survive between
tool calls.
* `pgrep` cannot be used for liveness in this container: PID 1 is
`sleep infinity` and never reaps, so dead processes linger as `<defunct>` and
still match by name. Use `ps -o stat=` and skip `Z`, or `xdpyinfo` for X.
* numpy is not installed system-wide; `pip install --break-system-packages
numpy` puts it in `/sylph-home/.local`, which is only on `sys.path` when
`HOME=/sylph-home`. Scripts run with `HOME=/sylph-home/re` need
`PYTHONPATH=/sylph-home/.local/lib/python3.12/site-packages`.
## Files
`pilot.py` (**the survival loop**) · `ctrl_probe.py` (input → speed calibration,
plus a per-tick window of the player object) · `binq.py` (query that capture) ·
`own_state.py` (definition-anchored hull/shield lookup) · `fly_session.sh`
(boot → mission → bind → fly, one task) · `wait_flight.sh` (wait for the real
HUD instead of a fixed sleep) · `navigator.py` (drift-aware steering + CPA
avoidance, reused by the pilot) ·
`gworld.py` (live reader + entity list) · `flight_probe.py` (scripted inputs +
sampling, and the `Pad` FIFO client) · `flight_analyze.py` · `whatchanges.py`
(encoding-agnostic "which words are live") · `findplayer.py` · `findself.py` ·
`findrot_global.py` · `findspeed.py` · `liveents.py` · `selfstate.py` (the
whole chain → JSON) · `autopilot2.py` (PD controller; **untested — it has never
had a valid config to run against**) · `launch_mission.sh`.

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.7 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

File diff suppressed because it is too large Load Diff

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.9 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.7 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 215 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 195 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 233 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 218 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 185 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 156 KiB

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

Binary file not shown.

After

Width:  |  Height:  |  Size: 881 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 543 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 626 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 884 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 890 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 188 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 167 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 220 KiB

File diff suppressed because it is too large Load Diff

Binary file not shown.

After

Width:  |  Height:  |  Size: 98 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 104 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 117 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 112 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 53 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 53 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 45 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 102 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 159 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 53 KiB

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,104 @@
# In-flight control mapping — measured, not assumed
**Status:** ✅ for the weapon bindings (ammo counters move), 🟡 for the rest (HUD
observation only). Probes: `tools/re-capture/fire_probe.sh` (hold each input, photograph
the ammo counters) and `lock_probe.sh` (tap each, watch the reticle). Stage 02, in flight.
Evidence: [`captures/fire-probe-ammo-counters.png`](captures/fire-probe-ammo-counters.png).
| input | effect | confidence |
|---|---|---|
| **`RB`** | **Nose gun.** `NOSE BM` 06000 → 05956 in a 4 s hold ≈ **11 rounds/s**; `HEAT` bar rises | ✅ |
| **`Y`** | **Main mount** (missiles). `MAIN MPM` 00300 → 00299 per tap | ✅ |
| **`RT` / `LT`** | Throttle up / brake, a *persistent* setting (488 → 1510 → 174 u/s) | ✅ (earlier session) |
| **d-pad** | **Tactical map** overlay (grid with contact blips) — not target cycling | 🟡 |
| `LB`, `X`, `B`, `A`, `LS`, `RS` | No change to either ammo counter | ✅ (as "not a weapon") |
## ~~Targeting appears to be automatic~~ — WRONG, corrected below
> **Superseded.** This section concluded targeting was automatic because no input
> cycled a target. It is wrong: the HUD tutorial states target select is **Ⓐ pressed
> twice**, and every sweep here tapped once. Kept because the reasoning is a useful
> warning — a probe that never performs the action will "prove" the action does not
> exist. The rest of the section's measurements stand.
No *single* press cycled a target. The green `TARGET` marker is already present in
idle frames with nothing pressed, which I read as the game selecting for us.
That fits the measurements end to end:
- the guns fire fine (11 rounds/s) but the kill counters read `0000` after five gun-only
runs → **we shoot and miss**;
- guided missiles (`Missile_P`, Power 200, `GuidanceType` 5) got the first kills,
`WARPLANES 0002`, but only **2 per 98 launches**;
- the pilot's own log shows aim error wandering between ~10° and ~40° for most of a
pass.
At the time I concluded the bottleneck was aim dwell. Partly right — target
**commitment** did take kills 2 → 9 — but the larger cause was simply that no target was
ever selected, so the guided missiles had nothing to guide to.
## The game's own action list (from the OPTIONS key-config screen)
Decoded from `dat/GP_OPTIONS.pak` (`po_keys_btn*` sprites) — this is the authoritative
set of bindable in-flight actions, straight off the disc, no probing required:
| # | Action | Our mapping |
|---|---|---|
| 1 | Aircraft Control | LX/LY ✅ |
| 2 | View Point Control | RX/RY (unused by the pilot) |
| 3 / 4 | Left / Right Yaw Control | — (separate from pitch/roll!) |
| 5 / 6 | Accelerate / Decelerate | `RT` / `LT` ✅ |
| 7 | **Use Main Weapon** | `Y` ✅ |
| 8 | **Use Nose Weapon** | `RB` ✅ |
| 9 | Special Move | ❔ |
| 10 | Maneuver | ❔ |
| 11 | Resupply | ❔ |
| 12 | **Change Target** | ❔ — **this is the target-select the loop needs** |
| 13 | Change Main Weapon | ❔ (would reach `ASMissile`, Power 5000) |
| 14 | **Padlock Mode Toggle** | ❔ — **the aim-dwell mechanism** |
| 15 | Radar Map Toggle | d-pad 🟡 (matches the observed map overlay) |
Two entries change the plan outright:
- **`Change Target` exists**, so target selection *is* an input after all. The earlier
probe swept `LB/X/B/A/LS/RS` and found no ammo change — consistent with those being
exactly these non-weapon actions. The probe simply watched the wrong indicator.
- **`Padlock Mode Toggle`** is a view/aim lock onto the selected target. That is the
aim-dwell problem solved *by a game mechanic* rather than by tuning a PD controller —
and it is why a human player can hold a contact long enough to lock a missile.
Also note `CONTROL SETTINGS` carries a **`Control Type`** preset plus **Yaw / Pitch /
Roll Sensitivity** and a separate **`Throttle`** option: the mapping is not fixed, and
the craft's response to a given stick deflection is configurable. Any calibration done
against one profile (e.g. the `ctrl_probe.py` throttle numbers) is only valid for the
save's current settings.
## What the tutorials state outright
`tutorial_capture.sh <index> <secs> <tag>` plays one lesson and photographs it. Captions
use a typewriter effect, so crop `900x125+160+40` from many frames to read a full
sentence. Lessons that require the player to *do* something stall (BASIC CONTROLS sits
on "Go to the box on your screen" forever with nobody flying); the expository ones run
on their own.
- **HEADS-UP DISPLAY (index 1):** *"Enemies are displayed with **red markers** and allies
with **blue markers**." · "Targeting an enemy displays an Armor Gauge…" ·* **"Press Ⓐ
twice to target the enemy closest to the center of the screen."**
- **ADVANCED CONTROLS (index 5):** `B`+`LS` = Side Roll / 180 Degree Turn / Level Off ·
`B`+`A` together = face the target · `LT`+`RT` together = *"sets your fighter's speed
to that of the target… works well when you are trying to get behind an enemy. Once
behind an enemy, this also helps you attack them."*
**`Change Target` is Ⓐ pressed TWICE** — a double tap. That is why every button sweep in
this document found nothing and why I wrongly concluded targeting was automatic: each
sweep tapped once. It also explains the missiles — `GuidanceType 5` needs the *game's*
selection, and the loop had never made one, so 98 launches guided to nothing.
## Notes for the reimplementation
- Two independent weapons with separate ammo pools and separate HUD counters:
`NOSE BM` (gun, 6000) and `MAIN MPM` (missiles, 300).
- The gun has a **HEAT** bar that fills while firing — a sustained-fire limit the
reimplementation needs; its cap and cool-down rate are not measured yet.
- The tactical map is a full-screen overlay bound to the d-pad and does not pause flight
(the craft kept taking fire with it open).

View File

@@ -0,0 +1,225 @@
# Escort / mission state from guest RAM — every entity's hull
**Status:** ✅ CONFIRMED (2026-07-30). Capture: `tools/re-capture/mission_state.py`,
session `tools/re-capture/escort_session.sh`, Stage 02 from save slot 01, 240 s of
flight, 240 samples at 1 Hz → [`captures/mission-state-stage02.jsonl`](captures/mission-state-stage02.jsonl).
Screenshot evidence: [`captures/escort-stage02-hud.png`](captures/escort-stage02-hud.png).
## The question
`own_state.py` found the **player's** hull by anchoring on a solved definition
field — an undamaged craft carries its definition's `HP` (+0x054), so the live
counter is the copy of that number that falls. Result: `hull = position + 0x154`
([autopilot](autopilot-memory-driven.md)).
Stage 02 is an **escort**, and it is lost when the ACROPOLIS sinks, not when the
player dies: a 240 s run hit `GAME OVER` with our own hull at 1500/1500. Scoring
that objective needs *someone else's* hull. So: is `+0x154` a property of the
**entity class**, or of the player object?
## Finding — it is class-wide
At the first sample of the run, before this session's fighting had touched them,
`pos+0x154` equals the entity's own definition `HP` across **seven classes and
five distinct HP values**:
| Class | radius | definition `HP` | `pos+0x154` at t=0 |
|---|---|---|---|
| `UN_e007_ADAN_Turret` | 22 | 100 | 100.0 (all 60 instances) |
| `UN_e010_ADAN_Attacker_S` | 100 | 500 | 500.0 (all 19) |
| `UN_f106_TCAF_Destroyer` | 2000 | 10000 | 10000.0 |
| `UN_e106_ADAN_Destroyer` | 2100 | 10000 | 10000.0 |
| `UN_f105_TCAF_Cruiser` | 3800 | 30000 | 30000.0 |
| `UN_e105_ADAN_Cruiser` | 3800 | 30000 | 30000.0 |
| **`UN_f101_TCAF_Acropolis`** | 1400 | **25000** | **25000.0** |
Measured directly by `mission_state.py scan` at the start of three separate runs:
**146/150, 147/150 and 147/150 entities** hold exactly their definition's `HP` at
`pos+0x154`. The handful that do not sit *slightly below* it (9800/10000,
29933.3/30000, 9725/10000, …) — the battle is already in progress when the player
launches, so those ships have already been shot at. **Nothing read above its `HP`,
and nothing read an unrelated number**, which is what a coincidental offset would
produce.
The value behaves like a live counter, not a copy of the definition:
- it **falls under fire** — 780 distinct damage events were logged across the run;
- it **goes negative at death** and the entity then disappears from the heap
(`UN_f106_TCAF_Destroyer``-30.0` of 10000, another → `-0.0`, a third GONE);
- the drops match what the HUD draws — the screenshot shows the ACROPOLIS and the
destroyer *CHARON* each with their own health bar, CHARON's already red.
So **`hull = position + 0x154` for every entity**, and the escort objective is
directly scoreable: read the protected ship's hull, normalise by its definition's
`HP`, done. No new anchor, no value scan.
## The escort asset, measured
`UN_f101_TCAF_Acropolis`, one instance, `HP` 25000, collision radius 1400.
Its hull over the 240 s run (pilot chasing the nearest hostile fighter, the
current `pilot.py` behaviour):
```
t= 0..150s 25000.0 untouched
t= 180.1s 24779.5
t= 210.1s 24149.5
t= 239.1s 23038.2 -1961.8 total, ≈ -600 HP/min once it starts
```
**⚠️ Onset is NOT a fixed schedule — corrected by a later run.** From this run alone
it looked like the asset is safe for the first ~170 s. A second run put the first
damage at **t = 70 s**, and its hostile population *grew* (134 → 166 ADAN) where this
one's shrank (147 → 118). So the stage is not replaying identically, and "the asset
is untouched early" is a property of one run, not of Stage 02. What survives the
second run is the weaker, still useful claim: **the loss is slow** — a few hundred to
~1400 HP/min against 25000, so tens of minutes to sink. The earlier `GAME OVER`
therefore was not a fast loss; it was an undefended one.
## Also captured
- Hostile population fell 147 → 118 over the run (the pilot fired on 435 of 1913
engage frames; most of the remainder it was manoeuvring with the target outside
the 9° firing cone).
- Two friendly destroyers were lost while the pilot was elsewhere.
- The HUD's `REMAINING OB` read **012** at t≈240 s while 118 ADAN entities were
alive, so that counter is **objectives, not hostiles** — its RAM address is still
unknown (❔ open).
## Escort-weighted targeting — implemented, and what it did NOT fix
`pilot.py` gained a **DEFEND** mode (2026-07-30): while the asset is losing hull,
target the hostiles pressing *it* — ranked by distance to the asset minus credit for
closing on it — instead of the ones nearest to us. Trigger and ranking both read the
live hull, so nothing is inferred.
It works mechanically: DEFEND engaged **1.9 s after the asset's first hit** in one run
(t=167.0), and held for 54 % of a 330 s run. **But it did not measurably save the
asset.** Over the window the two policies share, they are the same to within noise:
| t (s) | nearest-fighter | escort-weighted |
|---|---|---|
| 120 | 25000.0 | 24910.0 |
| 180 | 24779.5 | 24460.0 |
| 239 | 23038.2 | 23218.0 |
Two honest reasons it cannot yet be scored better than "no worse":
1. **The runs are not comparable past that window** — different spawn timing and, in
the escort-weighted run, a hostile population that *grew* 134 → 166 while the
baseline's fell 147 → 118.
2. **Lethality is the real bottleneck, not target choice.** The guns are on for only
**12 % of combat frames** (320 of 2630); the rest of the time the target is outside
the 9° firing cone while the loop manoeuvres. Choosing a better target does little
when most passes do not shoot.
**One bug found and fixed by the first escort run** (worth keeping as a pattern): the
new mode flies *at* the asset, which sits inside the friendly formation, and the run
ended `hull 1500 -> DEAD` in a single tick at 2026 units/s, 0.6 s from a friendly
destroyer the avoidance expected to clear by 365 units — against a hull of radius
2000. Keep-out had been applied only to hostile turrets. Every entity above
`BIG_RADIUS` now gets a physical keep-out of **its own radius + 800**, with braking
inside it, whatever its faction; the next run survived its full 330 s untouched.
## Ballistics from the disc data — and the measurement that invalidates the metric
The solved `Shell` records give the player's guns exactly
(`Shell_TCAF_DeltaSaber_{NoseGun,Gun,Beam}_P`, all ✅ CONFIRMED):
**`Velocity` 8000**, **`LifeTime` 0.5 s**, **`MaximumRange` 4000** — self-consistent,
since 8000 × 0.5 = 4000 — plus shell `Radius` 2030 and `Power` 15/30/40.
Two things in `pilot.py` were plainly wrong against those numbers, and both are fixed:
- **Lead used our own speed as the shell speed.** Flight time was `d / max(our_speed,
300)`, i.e. 4002000 u/s instead of 8000 — every shot led **416× too far ahead**.
- **`FIRE_RANGE` was 5000**, past the range at which the shells expire.
**But the outcome metric says none of this has been shown to help.** The HUD's own
counters — `YOU KILLED: WARSHIPS` / `WARPLANES` — read **0000 / 0000 at the end of
every run**, including the nearest-fighter baseline. The pilot is not killing
anything in any configuration, so "fraction of frames with the guns on" (12 % → 5 % →
1 frame in 2639 as the firing gate was varied) was never measuring lethality. The
corrections above are right on the physics and fix demonstrably wrong code; **they are
not evidence of improvement**, and none is claimed.
The firing gate itself produced one clean result worth keeping: gating on the target's
angular half-size **alone** (2.7° at 2584 units for a fighter) is far tighter than the
steering loop can hold the nose, and firing collapsed to 1 frame in 2639. Angular size
belongs in the gate as a **floor** that opens it up close, never as a cap.
### Why nothing died — settled by probe, then fixed
`fire_probe.sh` holds each pad input in turn in flight and photographs the HUD ammo
counters. Result:
| input | `NOSE BM` | `MAIN MPM` |
|---|---|---|
| idle | 06000 | 00300 |
| **RB** | **05956** (44 in 4 s, HEAT rises) | 00300 |
| **Y** | 05951 | **00299** (1) |
| LB / X / B / A / RT / LT | no change | no change |
So **`RB` is the nose gun (~11 rounds/s) and `Y` is the main mount** — measured, not
assumed — and the "we never shoot" hypothesis is dead: **we shoot and miss.**
Which is what the disc data says to stop doing. `Shell_TCAF_DeltaSaber_Missile_P` is
**Power 200, `GuidanceType` 5 (guided), `MaximumRange` 5000**, against the nose gun's
**Power 15, unguided**. One missile is worth ~14 gun hits on a 500 HP fighter *and it
steers itself* — the accuracy problem solved rather than tuned. (`ASMissile_P` is
Power **5000**, the anti-ship option.)
Adding missile launches to the pilot (press `Y`, release a tick later, ≥2 s apart)
produced **the first kills of the whole series: `YOU KILLED: WARPLANES 0002`**, versus
`0000` in all five gun-only runs, with hostiles down 134 → 104 (the largest fall yet).
**Still poor, and stated as such: 98 missiles for 2 kills (~2 %).** The likely cause is
that the game expects a *lock* — holding the target in the reticle before launch — and
an unlocked launch is wasted. Reading the lock state (or the lock timer) out of RAM is
the next step, and it is the same anchoring trick as everything else here.
## Target commitment — the change that actually moved kills
The pilot re-scored every contact every tick, so the nose chased whichever fighter was
momentarily best-scoring and the aim error wandered 1040° through a pass. Since a
missile lock is time-on-target, constant switching is the one thing guaranteed to
prevent a kill. **Commitment**: stay on the chosen contact until it dies, gets beyond
6000, sits >90° off the nose for 2.5 s, or 14 s elapse.
Nothing else changed — same guns, same ballistics, same escort weighting, same missile
cadence:
| run | kills (`WARPLANES`) | missiles | hostiles |
|---|---|---|---|
| gun-only × 5 | **0000** | 0 | 147→118 … 134→166 |
| + guided missiles | **0002** | 98 | 134→104 |
| + **target commitment** | **0009** | 101 | **134→97** |
4.5× the kills for the same ammunition, and the largest fall in hostile population of
any run. Our own hull finished untouched at 1500/1500.
**The escort is still not saved** — the ACROPOLIS finished at 76.6 % — so this improves
lethality, not the mission outcome, and the two should not be conflated.
### Negative result: the selected target is not a raw entity pointer
Worth recording so it is not re-attempted. `target_probe.py` looked for the selection
three ways: (1) every word in a ±0x1400 window of the player object that points at a
live entity — **none**; (2) every word in *all* of RAM holding an entity pointer, tapped
through each button — only thread-stack slots (`0x70xx_xxxx`) churned, which is frame
noise, not selection; (3) a delta tally over all 150 entities looking for a repeated
offset holding a pointer to *another* entity, the same trick that found the definition
pointer at `+0x130` — **zero candidates**.
So neither the player nor the AI ships keep a raw pointer to their target near their
transform. The selection is a handle, an index, or lives in a targeting subsystem
outside the entity object.
## Reimplementation notes
- Defeat conditions for an escort stage are readable as: protected-asset
`hull ≤ 0`, or player `hull ≤ 0`.
- Every unit's effective HP is the definition's `HP`, confirmed live for 7 classes —
the same field the [unit struct](structures/unit-struct-runtime.md) already solves
statically, so disc data and runtime agree.
- Entity removal on death is observable (the object leaves the heap), which gives a
clean lifetime signal for anything modelling spawn/despawn.

View File

@@ -0,0 +1,100 @@
# Capital-ship placement — does the `e106` result generalise? (WIP, 2026-07-31)
**Status:** 🚧 **WIP, time-boxed session.** Two results so far: a static audit across all
22 stage containers (done, below) and a first in-mission F10 capture run in Stage 02
(done — three capture logs, but **no capital-ship part correlated**; see "Open").
Context: [`BACKLOG.md`](BACKLOG.md) — "Capital ships assemble wrong in the viewer",
reported 2026-07-30. The oracle and the correlator already exist
([`ship-placement-runtime-capture.md`](ship-placement-runtime-capture.md)); the open
question is whether the rules derived from the one validated ship (`e106`, Stage_S01)
hold for other classes.
## 1. Static audit across all stages (offline, reproducible)
```
cargo run --release --example ship_audit -- ../sylph_extract/hidden/resource3d
```
87 lines of output, of which:
- **Only two OUTLIER lines, and they are the same ship twice**:
`Stage_S03`/`Stage_S27`, `f002_bdy_05` centroid `[-1398 6251 918]`, `dist=6540`
vs a cluster spread of `1071`. Every other assembled ship in every other stage
has all parts inside its own cluster.
→ **The user-visible breakage is NOT a gross static-placement outlier for most
classes.** Whatever is wrong in the viewer is either subtler than "part flung far
away" (wrong rotation, wrong mirror, missing part) or lives in the viewer, not in
`assemble_ship`. `f002_bdy_05` is a genuine, separate, reproducible static bug.
- **MULTIKEY**: joint tracks with more than one keyframe, which `read_trs9`'s
single-key read does not model. Recurring rigs: `e_rou_f104` (3), `e_rou_f105` (2),
`e_rou_f106` (2), `e_rou_e102` (6), `e_rou_e108_Missile_open` (2), `e_rou_e501` (1),
and the `e901` boss with 215 tracks per pose. Several of these (`Missile_open`, the
`e901_attack*` poses) are obviously *animation* and harmless for a static pose; the
plain hull rigs `f104`/`f105`/`f106`/`e102` are **not** obviously animation and are
the best hypothesis for a class-specific assembly error. ❔ **HYPOTHESIS — not
verified.** `e106`, the one validated ship, has **no** multikey tracks, which is
exactly how a rule that only works for single-key rigs could have passed unnoticed.
Raw audit output is reproducible with the command above (not checked in; it is
deterministic from the disc).
## 2. First in-mission capture run (Stage 02)
New tool: [`tools/re-capture/ship_capture_session.sh`](../../tools/re-capture/ship_capture_session.sh)
— one blocking session (per the session-lifetime rule): boot → Stage 02 in flight →
N× {screenshot, F10, small yaw}. Each F10 writes its own
`xenia_ship_capture_NN.log` next to the binary.
Run 2026-07-31, 5 presses requested:
- **Boot to in-flight took 24 s** (`skip_intro.sh` skipped the movie at 1 s and 6 s,
title at 11 s, HUD shield bar at 24 s) — much faster than the ~100 s in the notes.
- **3 of 5 F10 presses produced a log** (`_01``_03`, 2964 / 3111 / 3668 draws).
Logs (810 MB each) and the screenshots are at `/sylph-home/re/shipcap/`; not
committed for size. One screenshot is checked in as
[`captures/shipcap-stage02-launch.png`](captures/shipcap-stage02-launch.png).
- The screenshot confirms the capture frames are real in-mission combat frames
(HUD live, `REMAINING OB 004`, ACROPOLIS + a Destroyer labelled on screen, a
capital-ship hull filling the bottom of the frame).
### Result: no correlation yet ❌
```
correlate_capture xenia_ship_capture_03.log Stage_S02 <id> bdy_01
```
for `f101` (ACROPOLIS), `f105`, `f106`, `e105` reports *"no draw matches any LOD
(culled/off-screen?)"* for essentially every part — only two speculative LOD tries
(`f101_bdy_03` vcount 90 `[l]`, `e105_wep_01` vcount 60 `[l]`) and **zero accepted
matches**.
That is a **negative result, and it is not yet explained**. Facts collected:
- The capture is not empty or degenerate: 3668 draws in `_03`, top shaders
`0xDA51B0745ABF85D2` (1258), `0xE0BAFB4F520FE441` (1091), `0xEEA84C59D7F95371` (770).
None is the `e106` ship-shader hash from the 2026-07-26 capture; the F10 path does
not filter by hash, so this alone is not the cause.
- Large vertex counts *are* present (3024, 2772, 1736, 1612, 1240 …), so capital-ship-
sized geometry is being drawn.
Candidate explanations, **untested**:
1. the Stage-02 capital ships on screen are drawn from LOD/damage variants
(`_d00`, `_m`, `_l`) whose vcounts the correlator's variant list does not cover;
2. the position-validation step rejects otherwise-correct vcount hits (the capture
dumps ≤64 positions — a set-membership test against the wrong variant fails);
3. the ships in view at launch are drawn by a *different* draw path than `e106` in
Stage_S01 (e.g. instanced/batched), so no single draw equals one part.
**First step next session:** take the largest few vcounts in the capture and ask which
decoded part in `Stage_S02.xpr` has that count (invert the match), instead of asking
per-part whether a draw exists. That distinguishes (1)/(2) from (3) immediately.
## Honest summary
- ✅ Static assembly is **not** grossly broken across stages — 1 outlier ship
(`f002_bdy_05`), reproducible.
- 🟡 A concrete, testable hypothesis for class-specific breakage exists (multikey joint
tracks on `f104`/`f105`/`f106`/`e102`; `e106` has none).
- ❌ The runtime oracle did **not** reproduce on a second ship class yet. The capture
pipeline works end-to-end (boot → F10 → logs); the correlation step is where it
stops. **NEEDS-HUMAN / next session**, do not assume the `e106` rules generalise.

View File

@@ -1,7 +1,20 @@
# Capital-ship part placement — runtime capture (ground truth)
**Status:** pipeline working end-to-end for one ship (`e106` ADAN Destroyer);
correlator recovers exact ship-relative transforms. Not yet baked into the viewer.
**Status:** ✅✅ **STATIC ASSEMBLY IS EXACT — no captures needed anymore
(2026-07-26).** The capture became the oracle that cracked the static encoding:
the node joint tables are **9 keyframe channels `[TX TY TZ RY RX RZ SX SY SZ]`
with the 8 table pointers shifted one track forward** (channel 0 = TX sits at
`ptr[0]0x48`; the old 8-slot read took TY as X, never saw TX — why hulls
stacked on the centreline). Euler order `Ry·Rx·Rz`, angles direct
(`mesh.rs::read_trs9` / `node_rotation`). `assemble_ship` now places every
composite-node instance (cross-id turrets ×2), mounts the `e_rou_<id>_eng`
cluster rig at `GN_Engine_01` (two mirrored nacelles + centre engine), puts
`brg`/`sld` at their exact GN frames, and X-reflects the shared-geometry twin
whose lateral offset opposes the geometry's dominant side.
`ship::tests::static_assembly_matches_runtime_capture` asserts static ==
captured for all 8 e106 parts (T < 1.0, R < 0.02) + both nacelles + both
turrets + the hull mirror. The viewer uses pure static assembly; the baked
table + correlator remain as the verification oracle only.
## Problem
@@ -56,16 +69,54 @@ Branch **`capture-ship-placement`** in the `xenia-canary-native` worktree (off t
- Run: `run-canary-native.sh` (HW Vulkan, interactive). Play into the mission,
frame the ship side-on, press F10.
## Correlator
## Correlator (now a library module)
`sylpheed-formats/examples/correlate_capture.rs`:
The correlation math lives in **`sylpheed-formats::ship_capture`** (unit-tested with
synthetic captures): `parse_capture` → `CapturedDraw`s, `correlate(id, draws,
parts, ref_sub)` → a `ShipPlacement` (each part in the reference frame), and
`serialize_table`/`parse_table` for the checked-in text table.
Two capture formats are accepted (auto-detected): the F10 ship-capture snapshot
([`parse_capture`]) and the **draw-logger** format `mission_draws.log`/
`xenia_re_draws.log` (`parse_drawlog` — groups the ship shader `SHIP_VS_HASH`'s
draws by vertex-buffer `base`, `vcount = size_words/stride_words`). So a capital
ship seen in a normal instrumented run can be baked without a dedicated F10 pass.
### Matching rules (learned from the real 2026-07-26 capture)
- **LOD-aware**: a distant ship draws its `_m`/`_l` copies — same part, same
local frame, different vcount. Each part tries every variant vcount.
- **Position-validated**: a vcount hit alone can be a coincidence (a foreign
51-vert mesh nearly hijacked the bridge slot). Every candidate draw's dumped
positions must be found in the part's decoded position set or it is rejected.
Validation is **set-based** (buffer order ≠ decode order) against the **union**
of the part's variants — the real capture had a draw whose `vcount` equalled
the `_m` count while its buffer held the FULL-detail geometry.
- **Runtime mirror**: a port/starboard pair shares one file geometry; the engine
uploads the second instance **X-reflected**. A draw that validates only under
X-negation is accepted as the mirror, and the placement bakes `diag(1,1,1)`
(the viewer reverses triangle winding for det < 0).
- The mission ship shader can be a different hash per lighting variant
(`0x3A0829CC71516789` in this capture) — the F10 path doesn't filter by hash,
it validates by geometry instead.
`examples/correlate_capture.rs` is the CLI wrapper:
```
SYLPHEED_ISO=… cargo run --release --example correlate_capture -- \
<capture.log> <Stage_SNN> <ship_id> [ref_part_substr]
<capture.log> <Stage_SNN> <ship_id> [ref_part_substr] [--emit]
```
Parses the log, decodes the ship's base parts, matches each part to a draw by
**vertex count** (unique per part), recovers WorldView from `c0..c2`, and prints
each part's ship-relative transform (reference-part frame) + assembled extent.
Decodes the ship's base parts (for their vertex counts = the match key), correlates,
prints each part's ship-relative transform, and with `--emit` prints the `ship …`
block to paste into the placement table.
## Baked table + viewer preference
`crates/sylpheed-formats/data/ship_placements.txt` is the checked-in placement
table (embedded via `ship_capture::embedded_placement`). The viewer's
`build_ship_model` uses a ship's captured entry when present, else falls back to
the static `assemble_ship` (`external` toggle). **The table is currently empty** —
a real F10 capture log is needed to bake in `e106` (and others); paste the
`--emit` block into that file and rebuild.
## Validated result — `e106` ADAN Destroyer (Mission 1 / Stage_S01, side view)
@@ -93,13 +144,37 @@ not the 1894-tall tower the static Y put out.
**Missing:** `brg_01` (bridge, vcount 202) was **culled/occluded** at that camera
angle (0 draws) — needs a second capture framing the superstructure.
## What the static assembler gets wrong (measured, `e106` in Stage_S01)
Decoding the real container (`examples/ship_dump.rs`) shows the static
`assemble_ship` is **incomplete** — the capture is required to fix all of:
- **`bdy_01` / `bdy_02` overlap**: both get the *same* composite transform
(T = (116, 0, 857)); the real ship is a **port/starboard pair** at X = ∓264
(the mirror is applied at runtime, like the DeltaSaber fins in `saber_measured`).
- **`eng_02` and `wep_02_01` are never placed**: they aren't `rou_` hull nodes and
aren't in the external category list, so they drop out entirely.
- Only `brg_01` + `eng_01` get the approximate `GN_*`-frame treatment.
So even the *hull* tier is not fully correct, and no static signal supplies the
missing offsets (verified: the part vertex buffers are pure local coords). The
capture-driven table is the only path to correctness — there is nothing more to
squeeze statically.
> Offline status: the draw logs on this box are the **player fighter**, not a
> capital ship, so no capital-ship table can be baked here yet. Capture `e106`
> (Stage_S01, side-on) on the emulator box — F10 **or** just save
> `mission_draws.log` with the ship on screen — then `correlate_capture … --emit`.
## Next steps
1. **Bridge:** re-capture with the bridge visible → fills `brg_01`.
2. **Bake:** promote the correlator to a module; emit a per-ship placement table
(`ship_id → [(part, R, T)]`) as a checked-in data file; have the viewer's
`build_ship_model` prefer the captured table over `assemble_ship` when present.
3. **Other ships:** same capture for the cruiser (`e105`, Stage_S02), ACROPOLIS
(`f101`), TCAF cruiser/destroyer (`f105`/`f106`). One side-on F10 each.
1. ~~**Bake:** promote the correlator to a module + checked-in table + viewer
preference.~~ **DONE** (`ship_capture` module, `data/ship_placements.txt`,
`build_ship_model` prefers it). Remaining: run the captures below and paste the
`--emit` blocks in — needs the emulator (F10), can't be done offline.
2. **e106 + bridge:** re-capture `e106` (Stage_S01) framing the superstructure so
`brg_01` (culled last time) is included, then `--emit` → bake.
3. **Other ships:** one side-on F10 capture each for the cruiser (`e105`,
Stage_S02), ACROPOLIS (`f101`), TCAF cruiser/destroyer (`f105`/`f106`).
4. **Turrets:** the shared `e3NN`/`e4NN` gun models appear as their own draws in the
capture too — correlate them to place turrets (static never resolved these).

View File

@@ -0,0 +1,182 @@
# `.rat` — the UI element layout / animation record
**Status:**`CONFIRMED` for placement (2026-07-28). The retail UI can be
**reassembled from the disc**: the tutorial PAUSE menu rebuilds pixel-accurately from its
sprites placed at the coordinates in their `.rat` records — no fitting, no manual nudging.
![real vs rebuilt](captures/ui-layout/pause-tutorial-real-vs-rebuilt.png)
*Left: the running game (Canary screenshot). Right: rebuilt from `GP_PAUSE_MENU.pak` alone.
The remaining differences are the animated frame/glow sprites (`*eff*`) that were not
placed, and the live 3D background.*
## Screen composition
The UI is **one pak per screen**`GP_TITLE`, `GP_PAUSE_MENU`, `GP_READY_ROOM`,
`GP_SAVE_LOAD`, `GP_MISSION_SELECT`, `GP_OPTIONS`, `GP_SYSTEM`, `GP_TUTORIAL`,
`GP_STAGE_CLEAR`, `GP_MOVIE_THEATER`, `GP_DEBRIEFING_PILOTLOG`, `GP_LEADERBOARD`,
`GP_MISSION_LOG`, `GP_BUNK`, `GP_DIALOG`, `GP_GAMEOVER`, `GP_CHALLENGE`,
`GP_HANGAR_ARSENAL`.
Inside a screen pak, each top-level [RATC](../INDEX.md) bundle is **one (context × language)
build of that screen**. Its own header is the screen's **element declaration table**, and
the elements themselves follow as children:
| child | what it is |
|---|---|
| `<name>.t32` | the sprite ([T8aD](texture-color-k8888.md)) |
| `<name>.rat` | that sprite's **layout record** (this document) |
| `<screen>loop1.rat` | a looping sprite animation (see below) |
### The bundle header — element declaration table
```
0x14 u32 entry count
0x20 entry[count], 60 bytes each:
+0 char[28] element name, NUL-padded ("pgp_ttrl_eff10.t32", "pgp_ttrl_btn10.rat")
+28 u32 ×4 flags (0xffffffff / 0xffffffff / 0 / 0xffffffff on every entry seen)
+48 u32 pivot X
+52 u32 pivot Y
+56 u32 0
```
The table lists **both** sprites and `.rat` records — it is the screen's element list.
`pgp_ttrl` declares 11: six `eff*`, `msg`, and four `.rat`s (`title`, `btn10..12`).
**Verified:** for all 7 `.t32` entries the declared pivot is *exactly* half the decoded
texture's dimensions — `eff10` 408×120 → 204,60; `eff21` 428×360 → 214,180; `msg` 381×38 →
190,19; and so on, 7/7 with no mismatch (`tools/re-capture/ratc_decls.py`).
`GP_PAUSE_MENU.pak`'s six bundles are `{in-mission, tutorial} × {English, Japanese}`, with
the two in-mission builds each present twice at identical size. The purpose of that
duplicate is **NEEDS-HUMAN** (resolution or aspect variant?), and where the other four
shipped languages live is likewise unresolved — this pak holds only EN and JP.
Naming is transparent: `pgp` = pause screen, `pgp_ttrl_` = its tutorial context, `btnNN` =
menu item, `btnNNf` = that item's **focused** sprite, `eff*` = frame/glow decoration,
`deli*` = the item divider, `title`, `msg`.
## Record layout
Big-endian u32 throughout (Xbox 360), and **tag-driven**: 4-char ASCII tags (`opt `,
`PRMD`, `end `) mark sections, so a record is a stream of blocks rather than a fixed struct.
A minimal record (a static button) is 165 bytes:
```
0x00 "RATC" magic — a RATC bundle reused as a data record
0x08 u32 payload size
0x14 u32 entry count (loop1.rat: 3, matching its 3 sprite names)
0x18 u32 design width = 1280
0x1c u32 design height = 720
0x20 char[16] the sprite this record places, e.g. "pgpbtn00.t32"
0x50 u32 pivot X = texture width / 2
0x54 u32 pivot Y = texture height / 2
...
── placement block ──
u32 scale X = 100 (percent)
u32 scale Y = 100
u32 tint = 0xffffffff (RGBA, white = untinted)
u32 X ← top-left position
u32 Y ←
u32 time (keyframe records only)
...
"opt " u32 len char[len] link to another record, e.g. "pgpbtn00f.rat"
```
- **X/Y is the sprite's top-left**, not its centre: compositing at these coordinates
reproduces the screenshot, which drawing centred on them would not.
- **The pivot at 0x50/0x54 is half the texture size** — 4 of the 5 plain sprites match
exactly (`pgpbtn00` 86×42 → 43,21; `pgpbtn05` 221×42 → 110,21; `pgptitle` 202×73 →
101,36; `pgpbtn04` 172×43 → 86,22). It is a rotation/scale centre, not a draw offset.
- **Animated elements are a keyframe list.** `pgptitle.rat` (752 B) is the same placement
block repeated with a varying trailing `time` field — the PAUSE title's fly-in.
- **`opt `** carries a length-prefixed record name. On `pgpbtn00.rat` it points at
`pgpbtn00f.rat`, i.e. *normal state → focused state*. Focused records place their sprite
42 px left and 8 px up of the base, because the focused art includes the selection ring
that hangs off the left edge; their pivot is a constant (21,25) rather than half-size.
- `<screen>loop1.rat` is **not** a screen composition — it is a looping sprite animation:
a 3-name table (`pgpeff34/35/36.t32`) plus ~30 keyframes all at one position.
- The small (380 B) top-level entries are **`PRMD` primitives**, not sprites: a colour and
four explicit corner coordinates `(0,0) (1280,0) (0,720) (1280,720)` — the full-screen
quad that dims the scene behind the pause menu — terminated by `end `.
### The records are language-independent
`pgpbtn00.rat` is **byte-identical** in the English and Japanese bundles. The layout is
authored once and only the `.t32` sprites are swapped, which has two consequences:
- The baked pivot belongs to *whichever build the record was authored from*, not to the
sprite actually shipped beside it. That is why `pgpbtn01`'s pivot (113 → a 226 px wide
texture) matches neither the English sprite (207) nor the Japanese one (148).
- The split is visible inside a single bundle: in the **English** tutorial build, every
`.t32` declaration carries the correct English pivot (7/7), while the `.rat` declarations
carry Japanese-derived ones (`btn10` → 43,21 = 86/2, the *Japanese* sprite). So the
`.t32` table is regenerated per language and the `.rat` layer is inherited from the
Japanese master.
- **Do not infer anything about a language from a texture size** — see the traps below.
## Evidence
Positions were read out of the records and checked against a screenshot of the running
game, twice, in that order — the records were never fitted to the picture.
1. **Differential.** Across `pgpbtn00/01/04/05.rat`, exactly one field varies and it steps
`268 → 338 → 408 → 478` — a constant 70 px pitch — while the field before it is 226 in
all four. A vertical menu: constant X, evenly spaced Y.
2. **Absolute placement — the decisive test.** The *tutorial* build's records give
546/288, 546/358, 546/428 and 540/119. Compositing its sprites at exactly those numbers,
with no offset and no fitting, reproduces the screenshot (image above).
3. **Pivot.** 4 of 5 plain sprites carry exactly half their texture's dimensions at
0x50/0x54 (above).
> A caution on step 2, because the first pass here got it subtly wrong: the screenshot is
> of the **tutorial** pause menu, so only the `pgp_ttrl_*` records can be checked against
> it. The in-mission records (X = 226) also map onto the same screenshot under a single
> constant offset — but that only works because both builds share the 70 px pitch, and it
> proves nothing. The in-mission coordinates remain **unverified**: confirming them needs a
> screenshot of a pause during an actual mission.
## It generalizes — the title screen
The same method run against `GP_TITLE.pak` reproduces the **main menu**, which is a
different screen with a different item count and a different pitch:
![main menu real vs rebuilt](captures/ui-layout/title-mainmenu-real-vs-rebuilt.png)
`ptbtn01..05.rat` give X = 542 for all five and Y = 162 / 242 / 322 / 402 / 482 — an
**80 px** pitch, where the pause menu used 70. Measured against the screenshot, the sprite
tops land at a constant **+46 px** for all five (one reads 45, a 1-px edge-detection
wobble), and 46 is exactly the 45 px of Xenia window chrome plus one. So the record's Y is
the sprite's top edge in the guest framebuffer, to the pixel, on a second screen.
`GP_TITLE.pak` also splits by sub-screen the way the pause pak splits by context:
`ptbtn00` alone (the `PRESS Ⓐ BUTTON` prompt), `ptbtn01..05` (main menu), `ptbtn11..13`
(the EXTRAS submenu), plus `pgloading_*` for the loading screen.
## Two traps this caught
Both were mistakes made during this analysis, caught by comparing against the real game —
recording them because a static-only reading would have shipped them:
- **Texture width does not identify a language.** English `RESUME` (166 px) and Japanese
`再開` (86 px) differ hugely, but Japanese `通信ログ` (148 px) is within a few px of an
English label. The first language assignment made here was wrong; rendering the sprites
is the only reliable check. (The records being language-independent makes this worse:
a record's baked pivot implies a texture width that matches *no* shipped sprite.)
- **The in-mission and tutorial pause menus are different sprite sets, not one set
re-packed.** In-mission is `RESUME / RADIO LOG / OPTIONS / BACK TO TITLE` (4 items,
`pgpbtnNN`); the tutorial is `RESUME / OPTIONS / BACK TO MENU` (3 items,
`pgp_ttrl_btn1N`). Matching the 3-item screenshot against the 4-item set suggests a
runtime slot-packing rule that does not exist.
## Next
- **The `eff*` / `deli*` / `msg` placements are still missing** — those sprites have no
`.rat` of their own, and `loop1.rat` turned out to be an animation, not a composition.
So the screen's draw list lives somewhere not yet found (the parent RATC's own header
region, or title code). That is the gap between the rebuild above and a complete screen.
- The same method should now unroll the other screens directly; `GP_HANGAR_ARSENAL.pak`
(789 T8aD + 510 RATC) is the big one, and the ARSENAL `DATA SHEET` panel documented in
[weapon-datasheet-runtime.md](../weapon-datasheet-runtime.md) is a ready-made oracle for it.
- Tooling: `crates/sylpheed-formats/examples/ui_screen.rs` (inventory a screen pak, carve a
named RATC child), `sylpheed-cli pak textures` (decode every sprite).

View File

@@ -0,0 +1,274 @@
# Runtime `Unit` struct (craft / vessel definitions) — read from live guest memory
**Confidence: ✅ CONFIRMED** for the 27 fields marked ✅ below (each binding is
reproduced by 1021 independent disc records, on ≥3 distinct values, with
**zero** contradictions), plus the Maneuver block's declaration-order layout
(29 anchors, two bases, no conflicts). 🟡 PROBABLE for the fields interpolated
between confirmed anchors. 🟡/❔ for the thin single-value bindings, which are
listed but must not be trusted yet.
Captured 2026-07-29 from Xenia Canary running the retail disc in the sylph-re
container: **all six tutorials** and **Stage 02 "Declaration of War"** loaded
from save slot 01. 21 of the disc's 110 units, over 7 snapshots from 7 separate
emulator runs.
## Why this exists
Craft stats were the one Route-B target the previous two passes could not
reach. The Hangar exposes only `Gross Weight` (a class, not a number), and
[the menu route](../weapon-datasheet-runtime.md) has no surface at all for the
flight model. From the memory side, menus were equally useless: **only the
player craft's name string is resident there — the definition objects do not
exist until a mission loads.**
They do exist in-mission. This documents their layout and the values the disc
leaves defaulted.
## Finding the class (discovered, not assumed)
[`tools/re-capture/unit_discover.py`](../../../tools/re-capture/unit_discover.py)
takes no vtable as input. It locates every disc unit-ID string in a memory
snapshot, finds every aligned word pointing at `string_va - d` for a range of
`d`, and tallies the word at `pointer_site - k` across **distinct** unit IDs.
One `(d, k, word)` combination wins by a wide margin:
| δ (name-record → string) | ID pointer at | word | distinct units |
|---|---|---|---|
| `0x10` | object `+0x04` | `0x820af030` | 4 |
| `0x10` | object `+0x04` | `0x820af844` | 4 |
— i.e. exactly the `Weapon` shape: the object holds a name-record pointer at
`+0x04`, and the ID string sits at `name_record + 0x10`.
The two vtables are **two different things**, and telling them apart matters:
| vtable | what it is | evidence |
|---|---|---|
| `0x820af030` | **spawned entity record** — one per spawned thing, but **not live state** (see below) | 12 objects for 4 IDs; the same ID appears many times (one per box in the scene); irregular spacing |
| `0x820af844` | **parsed definition** — the `.tbl` | exactly one object per distinct unit ID; minimum spacing `0x380` |
**Correction (2026-07-29, from the autopilot work):** `0x820af030` was
described here as holding live state. It does not — across a 29 s in-flight
capture **all 384 words of it are constant**. It is one record per spawned
thing, but the flying entity's transform is somewhere else entirely. See
[autopilot-memory-driven](../autopilot-memory-driven.md). Nothing below depends
on it; the definition class `0x820af844` is unaffected.
Only `0x820af844` is used. It is the runtime image of the `.tbl`:
* **Within one run** it is byte-identical across two snapshots taken ~12 minutes
apart with combat in between (14/14 objects, 0 differing bytes) — definition
data, not live state.
* **Across runs** the same unit is *not* byte-identical, and that had to be
explained rather than waved away. `unit_runtime.py --crosscheck` compares
every unit that appears in more than one snapshot (5 of them, over 7 runs):
exactly **15 words differ**, and 13 of them hold guest pointers
(`0x8xxxxxxx`/`0xbxxxxxxx` — heap addresses, which move per process).
**No solved or interpolated field offset is among the 15** — every value
reported here is run-invariant.
* The two non-pointer stragglers, `+0x2c8` and `+0x2d0`, are **stage-dependent**:
for one and the same unit (`UN_f001_TCAF_DeltaSaber_T_Ttrl`) `+0x2c8` reads
`8000.0` in two tutorials and `10000.0` in a third, with `+0x2d0` a 0/1 flag
beside it. So the object is *mostly* but not *entirely* the parsed table —
a couple of words are set per stage. Unidentified; **NEEDS-HUMAN**.
One `.tbl`**one** object. A unit table is several sub-records (`Generic`,
`Maneuver`, `Shield`, `Explosion`, `Mass`, `Effect`, `SE`, `Turret_00N`), and
they are all flattened into that single ≥`0x380`-byte object — unlike weapons,
where `Weapon` and `Shell` are separate arrays.
## Solving the layout
Same discipline as
[`weapon_runtime.py`](../../../tools/re-capture/weapon_runtime.py): score every
`(field, byte-offset, encoding)` triple against the disc records and accept a
binding only on **zero contradictions**, requiring ≥3 distinct values so a
field whose samples are all one number cannot match any offset holding that
constant.
```bash
python3 tools/re-capture/unit_runtime.py unit_tokens.txt snap_a.bin snap_b.bin --csv
```
Snapshots are **unioned** — each mission instantiates only the units in its own
stage, so coverage grows by visiting stages. `cp --sparse=always` a copy of
`/dev/shm/xenia_memory_*` first (~2 s); the running emulator pegs every core
under lavapipe and makes repeated live reads flaky.
### Encoding note — angles are radians at runtime
Every `AV_*` / `AA_*` / `*Bank*` / `Turn_AngularVelocity` field is stored as
**float32 radians**, while the disc writes **degrees**. The solver needed a
`rad` encoding (`degrees(f32)`) to bind them at all; 18 units agree on
`AV_PitchPlus_Max` alone. A reimplementation reading the `.tbl` must convert.
## Confirmed layout
✅ = ≥10 disc records agree on ≥3 distinct values, zero contradict.
| offset | enc | field | agree | distinct |
|---|---|---|---:|---:|
| `+0x030` | f32 | `Size_X` | 21 | 13 |
| `+0x034` | f32 | `Size_Y` | 12 | 7 |
| `+0x038` | f32 | `Size_Z` | 19 | 13 |
| `+0x040` | f32 | `Color_R` | 21 | 5 |
| `+0x044` | f32 | `Color_G` | 19 | 6 |
| `+0x048` | f32 | `Color_B` | 17 | 5 |
| `+0x050` | f32 | `Size_Radius` | 12 | 10 |
| `+0x054` | f32 | `HP` | 19 | 10 |
| `+0x074` | f32 | `ResistanceToOptics` | 11 | 3 |
| `+0x08c` | f32 | `ScorePoint` | 21 | 9 |
| `+0x094` | f32 | `MassScore` | 10 | 7 |
| `+0x09c` | f32 | `MinimumVelocity` | 11 | 3 |
| `+0x0a0` | f32 | `MaximumVelocity` | 19 | 8 |
| `+0x0a4` | f32 | `CruisingVelocity` | 17 | 7 |
| `+0x0a8` | f32 | `Acceleration` | 18 | 5 |
| `+0x0ac` | f32 | `Deceleration` | 17 | 5 |
| `+0x0b0` | rad | `AV_PitchPlus_Max` | 18 | 7 |
| `+0x0b4` | rad | `AV_PitchPlus_Min` | 10 | 6 |
| `+0x0c4` | rad | `AV_PitchMinus_Min` | 10 | 5 |
| `+0x0f8` | f32 | `SideThrustVelocity_Max` | 14 | 3 |
| `+0x238` | f32 | `MaxValue` (Shield) | 13 | 6 |
| `+0x244` | f32 | `ChargeSpeed` (Shield) | 13 | 6 |
| `+0x270` | f32 | `DestroyMotionTime` | 19 | 7 |
| `+0x2a0` | f32 | `RadarRange` | 18 | 9 |
| `+0x2a4` | f32 | `FCSRange` | 12 | 8 |
| `+0x2b4` | f32 | `AttackVesselPoint` | 14 | 8 |
| `+0x2bc` | f32 | `DefencePoint` | 12 | 7 |
`HQRatio +0x058`, `ShieldRatio +0x05c`,
`ThrusterRatio +0x060`, `ResistanceToShell +0x078`,
`ResistanceToExplosion +0x07c`, `ResistanceToPlayer +0x080`,
`ResistanceParalyze +0x084`, `BridgeCount +0x070` (i32),
`DryMass +0x274`, `GrossMass +0x278`,
`LowerHPThresholdRatio +0x298`, `AttackCraftPoint +0x2b8` bind with zero
contradictions on fewer records or fewer distinct values — 🟡 PROBABLE. The
full solver output is in
[`captures/unit-runtime-fields.csv`](../captures/unit-runtime-fields.csv)
(1 827 values, `conf` column).
## The Maneuver block is laid out in schema declaration order
This is the strongest structural result and it is independent of any single
field's agreement count.
[`schema_order.py`](../../../tools/re-capture/schema_order.py) merges the
`Maneuver` field-name order from **all 113 unit tables** by topological sort
over their pairwise "k[i] precedes k[i+1]" constraints. The merge is acyclic and
**every one of the 113 tables is a subsequence of the merged 102-field order**
so that order is the schema's.
Against it, the solved offsets fall into two exact runs:
| declaration indices | offset rule | anchors that fit |
|---|---|---|
| 0 … 20 (`MinimumVelocity``AA_Roll_Min`) | `0x09c + 4·i` | 21 / 21 |
| 21 … 32 (`SideThrustVelocity_Max``AccPitchFactor`) | `0x0a4 + 4·i` | 8 / 8 |
One 4-byte slot per field, with a **two-slot gap after `AA_Roll_Min`**
(`0x0f0``0x0f4`, purpose unknown). 29 independently-derived anchors, zero
conflicts, across a 0x9c0x125 span.
### Fields NO disc record ever values — 🟡 PROBABLE
Five `Maneuver` fields are declared by the schema but left at their default by
*every* one of the 110 unit tables, so no amount of disc analysis can ever
reach them. The declaration-order rule pins them between confirmed anchors
(`YawDragFactor +0x104``ArterBurner_Vc +0x114`, and
`ReverseThrust_Vc +0x118``ReverseThrust_Acc +0x120`):
| offset | field | craft (`f00*`, `e0*`) | capital ships (`*1**`, `e2*`) | inert (`SchlosBase`, `Box`) |
|---|---|---:|---:|---:|
| `+0x108` | `PitchDragFactor` | 3 | 2 | 0 |
| `+0x10c` | `RollDragFactor` | 3 | 2 | 0 |
| `+0x110` | `DragFactorThreshold` | 0.5 | 0.5 | 0 |
| `+0x11c` | `ArterBurner_Acc` | 2 | 2 | 0 |
| `+0x128` | `DecPitchFactor` | 30 | 30 | 0 |
Corroboration beyond the interpolation, checked over all 18 units:
* `PitchDragFactor == RollDragFactor == YawDragFactor` holds **18/18** — and
`YawDragFactor` is *disc-supplied* (3.0 for craft, 2.0 for warships), so two
interpolated offsets reproduce a known number, per unit, every time.
* `DecPitchFactor == AccPitchFactor` holds **17/18**. The exception is
`UN_e015_ADAN_Puppy` (`AccPitchFactor` 1, `DecPitchFactor` 0.5) — which
defaults *both* on disc, so it is two independent fields that happen to be
set equal elsewhere, not a broken binding.
`UN_e015_ADAN_Puppy` also breaks the craft/warship bucketing above for
`DecPitchFactor` (0.5, not 30); the per-unit values are in the CSV.
The tail of the `Maneuver` block (the AI-behaviour fields — `SideRoll_*`,
`BarrelRoll_*`, `TurnAttack_*`, `HoldPosition_*`, `Slalom_*`, `Through_*`,
`SolidCutoff_*`, and the `AxisMode` / `AB_*` sub-block) is **NOT resolved**.
Those fields are declared by only a handful of tables and almost always with a
single distinct value, so the solver's bindings there are coincidences: it
placed `Slalom_CutoffRatio` at `+0x00c` and `TurnAttack_DoubleTimeMin` at
`+0x110`, both of which the declaration-order rule contradicts. They are marked
`tentative` in the CSV. **NEEDS-HUMAN / needs more coverage** — more stages
would give those fields distinct values and settle it.
## The player craft, `UN_f001_TCAF_DeltaSaber_T_Player`
30 of its fields are defaulted on disc. Notable recovered values:
| field | value | note |
|---|---|---|
| `Size_Radius` | 10 | ✅ |
| `FCSRange` | 500000 | ✅ — same as `RadarRange` |
| `ChargeSpeed` (shield) | 25 | ✅ |
| `ResistanceToOptics` | 1 | ✅ |
| `HQRatio` / `ShieldRatio` / `ThrusterRatio` | 1 / 1 / 1 | 🟡 |
| `ResistanceToShell` / `ResistanceToExplosion` | 1 / 1 | 🟡 |
| `DryMass` | 100 | 🟡 (`GrossMass` 250 is on disc) |
| `LowerHPThresholdRatio` | 0.3 | 🟡 |
| `ChargeDelay_Break` | 10 | 🟡 |
| `MassScore` | 0 | 🟡 |
Every AI-behaviour field the solver bound reads **0** for the player craft,
which is the expected shape (the player is not AI-driven) — but see the caveat
above: those offsets are not settled, so treat the zeros as consistent, not
proven.
The `…Ratio` family that the Route-B target list parked is **1.0 for almost
every unit**, with real exceptions that only the runtime shows:
`UN_e105_ADAN_Cruiser` `HQRatio` = 0.2, `UN_bf001_TCAF_SchlosBase` and
`UN_e106_ADAN_Destroyer` `ThrusterRatio` = 0.2,
`UN_n001_TTRL_Box` `ShieldRatio` = 0.3.
## Coverage and how to extend it
21 of 110 units. Unlike weapons — where one snapshot held all 126 — **unit
definitions are instantiated per stage**, so coverage is bounded by the stages
reachable from the save (slot 01 is at 5 %, Stage 02). `unit_runtime.py` unions
any number of snapshots and re-solves, and more units directly promote the 🟡
bindings to ✅ by adding distinct values — the six tutorials took the confirmed
set from 22 fields to 27.
The tutorials are nearly exhausted as a source: all six together contribute only
3 units the missions do not already have (`UN_f001_TCAF_DeltaSaber_T_Ttrl`,
`UN_e015_ADAN_Puppy_2`, `UN_f001_TCAF_DeltaSaber_T_Player_Ttrl2`) — they reuse
one training box, one target drone and the player craft. **Real coverage now
needs real missions**, i.e. story progress on the save.
`tools/re-capture/grab_tutorial.sh` captures one tutorial per invocation
(cold boot → menu → Nth entry → snapshot, ~2.5 min). It cold-boots for each
because backing out of a loaded mission via PAUSE → BACK TO MENU wedges the
emulator. Two timing facts it encodes, both learned the hard way: the main menu
is **not input-ready for ~10 s** after the title tap, and d-pad presses before
that are silently dropped — which sends the A to `NEW GAME` instead of
`TUTORIAL`. And `NEW GAME` is not a cheap way to reach Stage 01: it gates on a
DIFFICULTY menu and then plays the prologue movie.
A NEW GAME excursion as far as the READY ROOM leaves
`535107D4/00000001/game01/savedata` byte-identical — only the profile `.gpd`
achievement files change — so it does not endanger the 5 % save. Verified by
diff against a backup, not assumed.
**A stage's whole unit set is parsed at load, not as waves spawn** — checked by
counting the definition objects at three points in Stage 02: immediately after
take-off, ~12 minutes in mid-combat, and after GAME OVER. 14 objects, the same
14 IDs, every time. So capturing a stage costs one load and one snapshot; there
is no need to play it, and no need to survive it.
Stages captured so far: `Ttrl` (BASIC CONTROLS), Stage 02.

View File

@@ -0,0 +1,294 @@
# Runtime `Weapon` / `Shell` structs — read from live guest memory
**Confidence: ✅ CONFIRMED** for the fields marked ✅ below (each binding is
reproduced by 10125 independent disc records with **zero** contradictions);
🟡 for the thin ones. Captured 2026-07-29 from Xenia Canary running the retail
disc, save slot 01 (Stage 02, 5 % progress), READY ROOM and ARSENAL.
## Why this exists
`weapon\Weapon_*.tbl` is an [IDXD](../INDEX.md) record whose string pool **omits
every field left at its default**. That parked a long list of stats as
unreadable — the `…Ratio` / `…Count` family, `Power`, `MaximumRange`. The
[Arsenal DATA SHEET route](../weapon-datasheet-runtime.md) recovered a few of
them but only as letter buckets (Range/Damage as `A``E`), and only for the 9
weapons unlocked at 5 % progress.
This reads the values **directly out of the running game's memory** instead.
All 126 weapons, all fields, exact numbers, in one pass — and it needs no story
progress, because the definitions are parsed at load time whether or not the
player has unlocked the weapon.
## The lever: Canary maps guest RAM into `/dev/shm`
Xenia Canary backs the entire guest address space with one shared-memory file,
`/dev/shm/xenia_memory_<id>`. It is a plain file: **the guest's RAM is readable
from the host with `open`/`seek`/`read`, live, no debugger and no emulator
patch.** Guest VAs map into it through Xenia's fixed table (`memory.cc`), which
[`tools/re-capture/gmem.py`](../../../tools/re-capture/gmem.py) implements:
```bash
python3 tools/re-capture/gmem.py find "Weapon_DSaber_P_wep_01" # search all of RAM
python3 tools/re-capture/gmem.py words 0xbccce500 48 # dump as BE u32/f32
```
The file is sparse (~212 MB resident of 4.5 GB), and the scan uses
`SEEK_DATA`/`SEEK_HOLE`, so a full-RAM search costs ~0.2 s.
## Finding the objects
1. The title's **schema field-name pool** is in the XEX at `0x82086a30`, in
declaration order: `EnumWeapon`, type `Weapon` (`ID`, `Name`, `TargetType`,
`CartridgeModelName`), then type `Shell` (`ID`, `Name`, `MovementType`, …
`Explosion_MaxDamageRadius`). This is exactly the on-disc field order. No
pointer to these strings exists anywhere in RAM — PPC builds the addresses
with `lis`/`ori` immediate pairs — so the schema descriptor cannot be found
by pointer-chasing; the layout has to be solved instead (below).
2. Each parsed record becomes **two C++ objects**, a `Weapon` and its `Shell`,
each in its own contiguous array, each identified by its **vtable pointer**:
| class | vtable VA | stride | count |
|-------|-----------|--------|-------|
| `Weapon` | `0x820af548` | `0xc0` | 126 |
| `Shell` | `0x820af58c` | `0x200` | 126 |
The vtable VAs are static (XEX `.data`); the array base addresses are heap
and are **not** assumed — the tool locates every object by scanning RAM for
the vtable word.
3. Object `+0x04` points at a 0x40-byte **name record**; the ID string sits at
`+0x10` inside it. That is what keys each object back to its disc record.
`Weapon``Shell` pairing is by ID (`Weapon_X``Shell_X`) — the two arrays
are in **different orders**, so index-pairing would be wrong.
## Solving the layout (the part that makes this evidence, not guesswork)
[`tools/re-capture/weapon_runtime.py`](../../../tools/re-capture/weapon_runtime.py)
brute-forces every `(field, byte offset, encoding)` triple and scores it against
the disc: how many records does this offset *reproduce*, and how many does it
*contradict*? A binding is accepted only with **zero contradictions**, and is
marked ✅ only when ≥10 records agree on ≥3 distinct values.
That threshold matters: a field whose disc samples are all the same number
matches any offset holding that constant, so agreement count alone is not
evidence — the number of **distinct** values pinned down is. Two fields landing
on one offset is impossible in a real struct, so collisions are resolved to the
better-evidenced field and the loser is reported as unsolved. Bindings that
contradict more than 20 % of their samples are discarded outright rather than
reported on their agreeing subset.
Encodings found: `f32`, `u32`, `deg` (**angles are stored in radians**;
`SprayAngle`, `AngularVelocity` and `SplitCone` are degrees on disc), and `cnt`
— see the next section.
### `IsCharging` switches the counter encoding ✅
`LoadingCount` and `TriggerShotCount` at `+0x28` / `+0x44` are **int32 on 118
records and float32 on 8**. The 8 are exactly the records with
`IsCharging = Yes` — an exact partition, found by testing every disc field/value
pair against the float-encoded set. A charging weapon drains its magazine
continuously, so its counter needs a fraction.
Any reimplementation must branch on `IsCharging` when reading these two fields;
reading them as int unconditionally yields `1125515264` instead of `150`.
## Struct maps
See [`docs/re/captures/weapon-runtime-fields.csv`](../captures/weapon-runtime-fields.csv)
for the full machine-readable table — 7 182 rows, every confirmed field × every
one of the 126 records, tagged `disc` or `defaulted-on-disc`. **4 393 of those
values were not readable from the disc at all.**
### `Weapon` (`0xc0` bytes)
| offset | enc | field | agree | distinct | conf |
|--------|-----|-------|------:|---------:|------|
| `+0x010` | u32 | `ReticleType` | 62 | 13 | ✅ |
| `+0x01c` | u32 | `SpecialWeaponType` | 73 | 6 | ✅ |
| `+0x028` | cnt | `LoadingCount` | 121 | 33 | ✅ |
| `+0x02c` | f32 | `Interval` | 125 | 28 | ✅ |
| `+0x030` | f32 | `ReadyInterval` | 120 | 10 | ✅ |
| `+0x034` | f32 | `Heating` | 114 | 30 | ✅ |
| `+0x038` | f32 | `Cooling` | 106 | 21 | ✅ |
| `+0x03c` | f32 | `Mass` | 122 | 42 | ✅ |
| `+0x040` | f32 | `HitRatio` | 102 | 5 | ✅ |
| `+0x044` | cnt | `TriggerShotCount` | 114 | 17 | ✅ |
| `+0x048` | f32 | `TriggerShotInterval` | 99 | 11 | ✅ |
| `+0x050` | deg | `SprayAngle` | 96 | 17 | ✅ |
| `+0x06c` | f32 | `LockIntervalSingle` | 61 | 9 | ✅ |
| `+0x070` | f32 | `LockIntervalMulti` | 28 | 9 | ✅ |
| `+0x09c` | f32 | `MaximumCharging` | 3 | 3 | 🟡 |
Also identified structurally, not by the solver: `+0x00` vtable, `+0x04` name
record, `+0x0c` `TargetType` as a bitmask (`Vessel|Craft|Structure` = 7),
`+0x14`/`+0x18` name hashes, `+0x7c`/`+0x80` muzzle-flash FX name record + hash.
Unsolved (no consistent offset): `Cracker_ShotInterval`, `Cracker_SubShotCount`,
`MinimumCharging`, `MultiTargetCount`.
### `Shell` (`0x200` bytes)
| offset | enc | field | agree | distinct | conf |
|--------|-----|-------|------:|---------:|------|
| `+0x00c` | cnt | `DeleteFadeSpeed` | 39 | 2 | 🟡 |
| `+0x010` | cnt | `GuidanceType` | 9 | 4 | 🟡 |
| `+0x014` | cnt | `SpiralType` | 3 | 1 | 🟡 |
| `+0x020` | f32 | `Length` | 107 | 3 | ✅ |
| `+0x024` | f32 | `Volume` | 19 | 3 | ✅ |
| `+0x028` | f32 | `ShellMass` | 110 | 34 | ✅ |
| `+0x03c` | f32 | `HP` | 28 | 2 | 🟡 |
| `+0x040` | f32 | `LifeTime` | 94 | 43 | ✅ |
| `+0x044` | f32 | `FadeInTime` | 13 | 2 | 🟡 |
| `+0x048` | f32 | `FadeOutTime` | 7 | 1 | 🟡 |
| `+0x04c` | f32 | `Velocity` | 106 | 15 | ✅ |
| `+0x050` | f32 | `MinimumVelocity` | 11 | 2 | 🟡 |
| `+0x054` | f32 | `MaximumVelocity` | 29 | 7 | ✅ |
| `+0x058` | deg | `AngularVelocity` | 49 | 19 | ✅ |
| `+0x05c` | f32 | `Acceleration` | 9 | 4 | 🟡 |
| `+0x064` | f32 | `BeginGuidanceTimeAdjust` | 24 | 2 | 🟡 |
| `+0x068` | f32 | `EndGuidanceTime` | 19 | 4 | ✅ |
| `+0x070` | f32 | `Spiral_BeginTime` | 2 | 2 | 🟡 |
| `+0x074` | f32 | `Spiral_BeginTimeAdjust` | 25 | 2 | 🟡 |
| `+0x08c` | f32 | `MinimumRange` | 43 | 7 | ✅ |
| `+0x090` | f32 | `MaximumRange` | 117 | 21 | ✅ |
| `+0x0a0` | f32 | `Color_R` | 18 | 4 | ✅ |
| `+0x0a4` | f32 | `Color_G` | 121 | 3 | ✅ |
| `+0x0a8` | f32 | `Color_B` | 100 | 2 | 🟡 |
| `+0x0b4` | f32 | `Radius` | 89 | 10 | ✅ |
| `+0x0b8` | f32 | `Power` | 101 | 37 | ✅ |
| `+0x0bc` | f32 | `FailedDamageRatio` | 2 | 2 | 🟡 |
| `+0x0c0` | f32 | `PlayerLaserPower` | 11 | 6 | ✅ |
| `+0x0fc` | f32 | `ChaffResistRatio` | 24 | 1 | 🟡 |
| `+0x104` | f32 | `ChargingSizeRatio` | 3 | 1 | 🟡 |
| `+0x10c` | f32 | `SphereRadiusBegin` | 9 | 6 | 🟡 |
| `+0x110` | f32 | `SphereRadiusTurn` | 9 | 8 | 🟡 |
| `+0x114` | f32 | `SphereRadiusEnd` | 10 | 8 | ✅ |
| `+0x118` | f32 | `ExplosionTurnTime` | 3 | 2 | 🟡 |
| `+0x11c` | f32 | `ExplosionLifeTime` | 7 | 6 | 🟡 |
| `+0x120` | f32 | `ExplosionDamage_Maximum` | 4 | 4 | 🟡 |
| `+0x124` | f32 | `ExplosionDamage_OuterEdge` | 8 | 6 | 🟡 |
| `+0x128` | f32 | `Explosion_MaxDamageRadius` | 8 | 3 | 🟡 |
| `+0x130` | f32 | `SplitTime_Maximum` | 2 | 2 | 🟡 |
| `+0x134` | deg | `SplitCone` | 2 | 2 | 🟡 |
| `+0x148` | cnt | `NodeCount` | 39 | 4 | ✅ |
| `+0x164` | f32 | `Deceleration` | 9 | 2 | 🟡 |
Unsolved: `BeginGuidanceTime`, `EndGuidanceTimeAdjust`, `Spiral_EndTime`,
`SplitTime_Minimum`, `StartingVelocity`, `Straight1Type`, `st1_df_pitch`,
`pitch0`, and the `sp_qu_*` / `sp_sp_*` / `sp_zg_*` spiral-motion family. Those
are declared by too few records (or by none that the solver could separate) —
they need a state where the shells are actually in flight.
## The answer to the parked Route-B question
Player-weapon fields that are **defaulted on disc**, now read exactly (`·` = the
disc carries the value already):
| wep | LoadingCount | TriggerShotCount | MaximumRange | Power | Heating | Cooling | ReadyInterval | HitRatio | SprayAngle |
|---|---|---|---|---|---|---|---|---|---|
| 01 | · | **1** | · | · | · | · | · | · | · |
| 02 | · | · | · | **100** | · | · | · | **1** | · |
| 03 | · | · | · | · | · | · | · | · | **1** |
| 05 | · | **4** | · | · | · | · | · | · | · |
| 08 | · | · | · | · | · | · | · | **1** | · |
| 09 | · | · | · | · | **0.02** | · | · | · | **1** |
| 11 | **6** | · | · | · | · | · | · | · | · |
| 12 | · | · | · | · | **0** | · | · | **1** | · |
| 13 | · | · | · | **300** | · | · | · | · | · |
| 14 | · | · | · | · | · | · | · | · | **1** |
| 19 | · | · | · | · | · | **0.2** | · | · | **1** |
| 24 | · | **1** | · | · | · | · | · | · | **1** |
| 25 | · | · | **4000** | · | · | · | · | · | · |
| 26 | · | · | · | **500** | · | · | · | · | · |
| 27 | · | **1** | · | · | · | · | · | · | · |
| 28 | **5** | · | · | · | · | · | · | · | · |
| 29 | · | · | · | **100** | · | · | · | · | · |
| 30 | · | **4** | · | · | · | · | · | · | · |
| 36 | **5** | · | · | · | · | · | · | · | · |
| 37 | · | **1** | · | · | · | · | · | · | **1** |
| 38 | · | · | · | · | · | · | · | **1** | · |
| 39 | · | **1** | · | · | · | · | · | · | **1** |
| 40 | · | · | · | · | · | · | **0.1** | · | · |
| 48 | · | · | · | · | · | · | · | · | **1** |
| 50 | · | · | · | · | · | · | · | · | **1** |
| 52 | · | · | · | · | · | · | · | **1** | · |
| 53 | · | **1** | · | · | · | · | · | · | · |
| 54 | · | · | · | · | · | · | · | · | **1** |
| 55 | · | · | · | · | **0** | · | · | **1** | · |
| 56 | · | · | · | · | · | · | · | **1** | · |
| 57 | · | · | · | · | **0** | · | · | **1** | · |
| 58 | · | · | **10000** | · | · | · | · | · | **1** |
| 59 | · | · | · | · | · | · | · | **1** | **1** |
| 60 | · | **4** | · | **1000** | · | · | · | · | · |
| 62 | · | **1** | · | · | **0** | · | · | · | · |
| 66 | · | · | · | · | · | · | · | **1** | · |
| 67 | · | · | · | · | · | · | · | **1** | · |
| 68 | · | · | · | · | · | · | · | **1** | · |
| 69 | · | · | · | · | · | · | · | **1** | · |
| 70 | **0** | · | · | · | · | · | · | **1** | · |
| 71 | · | · | · | **10** | · | · | · | **1** | · |
| 81 | · | · | · | **300** | · | · | · | · | · |
| 82 | · | · | **10000** | · | · | · | · | · | **1** |
| 84 | · | · | · | · | · | · | · | · | **0.1** |
| 85 | · | **1** | · | · | **0** | · | · | · | · |
`HitRatio` is **1.0 for every one of the 24 records that default it** — the
whole `…Ratio` family that blocked Route B is simply "no penalty".
`wep_82`'s defaulted field is `PlayerLaserPower` = **12000** (its `Power`,
12000, is on disc).
## Cross-checks
- **Against the independent screenshot route.** The Arsenal DATA SHEET
([weapon-datasheet-runtime.md](../weapon-datasheet-runtime.md)) established
`Max. Lock Ons == TriggerShotCount`, and read **4** for `wep_05` and `wep_60`
off the panel. The memory read gives **4** for both — two unrelated methods,
same numbers.
- **Against the in-flight HUD.** `NOSE BM 06000` / `MAIN MPM 00300` matches
`wep_01` `LoadingCount = 6000` and `wep_02` `= 300` at `+0x28`.
- **Stability.** All 252 objects are **byte-identical** between the READY ROOM
and the ARSENAL, so these are load-time definition data, not transient state.
- **Self-consistency.** 121 records agree on `LoadingCount`'s offset across 33
distinct values with zero contradictions; `Power`, 101 records / 37 distinct
values. A wrong offset cannot do that.
## Incidental findings
- **A duplicate, conflicting disc record.** Two `.tbl` entries declare
`Shell_TCAF_Ship_AAGun`; one sets `Power = 60.0`, the other omits it. The
runtime object holds **5**, i.e. the *omitting* declaration won. Any
reimplementation loading both will silently pick one — this says which.
- **Not every disc record is instantiated.** 131 disc records produced 126
objects; the 5 with no runtime object are context variants that this save
never loads — `Weapon_TCAF_DeltaSaber_NoseGun_Ttrl` / `_Laser_Ttrl`
(tutorial), `_NoseGun_None`, `Weapon_TCAF_Ship_AAGun_EX5`,
`Weapon_ADAN_Attacker_S_GunTurret`. Running the tutorial should instantiate
the `_Ttrl` pair. No runtime object lacked a disc record.
- Defaulted `Power` values vary per weapon (1, 10, 100, 300, 500, 1000), so they
are **not** one constructor constant. Where they come from — the IDXD's
undecoded binary node/index region, or per-type code defaults — is **not
determined here** (`NEEDS-HUMAN` / follow-up). Either way the runtime values
above are ground truth, and they are now a decoding oracle for that region.
## Reproduce
```bash
export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy
run-canary --audio --apu=sdl --log_mask=13 --logged_profile_slot_0_xuid=E0300000EFBEA3D4 &
tools/re-capture/skip_intro.sh # -> title
# LOAD GAME -> slot 01 -> YES -> READY ROOM (weapon tables load with the save)
cargo run --release -p sylpheed-formats --example idxd_tokens -- \
<disc>/dat/GP_MAIN_GAME_E.pak > /tmp/wep_tokens.txt
python3 tools/re-capture/weapon_runtime.py /tmp/wep_tokens.txt # report
python3 tools/re-capture/weapon_runtime.py /tmp/wep_tokens.txt --csv # full table
```
## What this unlocks
`gmem.py` + "scan for the vtable, solve the layout against disc ground truth" is
**not weapon-specific**. The same three steps apply to any IDXD-backed
definition whose fields the disc defaults — craft/`UNIT` stats, which the Hangar
UI exposes only as a `Gross Weight` class and which
[the menu route could not reach at all](../weapon-datasheet-runtime.md), are the
obvious next target.

View File

@@ -0,0 +1,232 @@
# Weapon DATA SHEET — runtime capture (Route B)
**Status:** 🟡 first dynamic capture, 2026-07-28. The Arsenal's *Gallery Mode* panel is a
direct runtime readout of the IDXD weapon record, which makes it an oracle for the fields
the disc leaves **defaulted**. Two field mappings are ✅ `CONFIRMED`; two defaulted values
are recovered at 🟡 `PROBABLE`. Captured from the retail game under Xenia Canary
(software Vulkan, headless) — see [the container recipe](#how-this-was-captured).
## Problem
`sylpheed-formats::game_data` reads the combat tables out of `dat/GP_MAIN_GAME_E.pak`, but
**a field left at its default value carries no value on disc** — the key is present in the
IDXD string pool with no value token in front of it. Those defaults live in title code, so
a static read can only ever say "not set", never *what* the game uses. That is a real hole
for the reimplementation: e.g. `Weapon_DSaber_P_wep_01_Beam` — the Delta Saber's starting
beam gun — has no `TriggerShotCount` and no `Power` on disc.
Ruled out first: the defaults are **not** hiding in another pak. `hidden/DefTables.pak`
contains no `WEAPON`-schema (`0x6ab4825a`) objects at all, and the other `GP_MAIN_GAME_*`
paks are localized duplicates of the English one.
The two scratch analyses that produced the shopping list live next to the crate:
`crates/sylpheed-formats/examples/defaulted_fields.rs` (per-schema: which keys are declared
but defaulted, and by whom) and `examples/default_owners.rs` (per-key: every owner, valued
or `<DEFAULT>`).
> Caveat on those tools: they classify a token as a *value* only if it is numeric or
> non-identifier-shaped. **Boolean/enum-valued fields therefore read as `<DEFAULT>`
> spuriously** (`Yes`, `Homing`, `Single`, `Burst` are identifier-shaped). Every finding
> below concerns numeric fields, where the classification is sound.
## Finding — the DATA SHEET reads the record
In **ARSENAL → (weapon type) → Y (Gallery Mode)** each entry shows a `DATA SHEET`, and it
is shown for weapons that have **not** been developed yet — only fully hidden (dashed)
entries are withheld. The same panel appears in **HANGAR → (hard point)**, with an extra
`Weapon Type` row.
| DATA SHEET row | IDXD field | Confidence | Evidence |
|---|---|---|---|
| `Ammo Capacity` | `LoadingCount` | ✅ CONFIRMED | every one of the 10 identified entries lands on a `LoadingCount` that exists in its own weapon-type tab — and see the circularity note below |
| `Max. Lock Ons` | `TriggerShotCount` | ✅ CONFIRMED | FALCON 9AM = 12 vs `wep_02`'s 12; BUZZARD 10AM = 22 vs `wep_04`'s 22 (two distinctive values, two independent records) |
| `Hard Point` | mount slot | ✅ CONFIRMED | matches the HANGAR slot the weapon is mountable/equipped on (`STILETTO BG I` → NOSE, `FALCON 9AM` → MAIN WEAPON1) |
| `Range Class` | bucket of `MaximumRange` | 🟡 PROBABLE | monotone in the on-disc metres, see the bracket table below |
| `Damage Class` | bucket of `Power` | 🟡 PROBABLE | monotone in the on-disc power, see below |
| `Weight Class` | bucket of the hangar-table `Weight` | ❔ HYPOTHESIS | only one clean pair so far (`wep_33`, Weight 0.3 → "Light") |
| `Speed Class` | ❔ | ❔ | present only on missiles (MPM = A, ASM = B); no on-disc pairing established |
| `Sight Homing`, `Lock on Overlap` | ❔ (`Available` / ``) | ❔ | the plausible on-disc partners (`Homing`, `OverlapLockon`) are identifier-valued and not yet decoded; **NEEDS-HUMAN** |
### Why the `Ammo Capacity` evidence is not circular
Each UI entry was **identified** by matching its `Ammo Capacity` against the records in
that weapon-type tab, so "the ammo matches" cannot on its own prove the mapping. What does:
1. **The match is forced and unique.** For 10 of the 11 entries, exactly one record in that
tab carries that number (`600` occurs three times overall — `wep_04`, `wep_24`, `wep_55`
— but in three different tabs: MPM, BEAM, B/R). An unrelated quantity would not land on
a valid, tab-unique `LoadingCount` eleven times running.
2. **A second, independent field then agrees.** FALCON 9AM and BUZZARD 10AM were pinned by
ammo alone, and their `Max. Lock Ons` (12, 22) then matched those same records'
`TriggerShotCount` (12, 22) — values that play no part in the identification. A wrong
identification would have to be wrong twice, consistently.
3. **One entry is identified without ammo at all.** STILETTO BG I is described in-game as
"the initially mounted Delta Saber beam gun" and is the weapon the HANGAR shows equipped
on the nose; `wep_01_Beam` is the corresponding record, and its `LoadingCount` 6000 is
what the panel shows.
The weakest identification is LIGHT MACHINE GUN MG I: three guns share `LoadingCount = 3000`
(`wep_09`, `wep_33`, `wep_83`). `wep_33` is picked on `Weight Class = Light` (it has by far
the smallest `Mass`, 1.5 vs 3.6 / 33.0) — 🟡 PROBABLE, not certain.
## Finding — recovered defaults
| Weapon | UI name | Field | On disc | **Runtime** | Conf. |
|---|---|---|---|---|---|
| `Weapon_DSaber_P_wep_05_ASMissile` | TERRIER SMH | `TriggerShotCount` | *(defaulted)* | **4** | 🟡 |
| `Weapon_DSaber_P_wep_60_ASMissile` | HOUND SMH | `TriggerShotCount` | *(defaulted)* | **4** | 🟡 |
Both defaulted weapons read **4**, which is consistent with a single title-code default of
`TriggerShotCount = 4` rather than two per-weapon constants — but two samples cannot tell
those apart. ❔ HYPOTHESIS: *the title-code default for `TriggerShotCount` is 4.* It would
be confirmed by a third weapon that defaults the field and also reads 4 (candidates that
were still locked in this save: `wep_27`, `wep_30`, and the seven Beams), or refuted by one
that reads anything else.
`Power` and `MaximumRange` defaults are **not** exactly recoverable from this panel — it
shows only the letter bucket. They are bracketed instead (below).
## Captured rows
All values are from one session on save slot 01 (Stage 02, "At Standby", 5 % clear, 4101 P);
`✓` marks a value that matches the on-disc record exactly.
| UI name | Record | Rng | Dmg | Spd | Weight | Ammo | Lock | Homing | Overlap | Hard point |
|---|---|---|---|---|---|---|---|---|---|---|
| LIGHT MACHINE GUN MG I | `wep_33_Gun` 🟡 | D | E | | Light | 3000 ✓ | | | | NOSE WEAPON (Nose) |
| BROAD SWORD SG I | *see below* | E | D | | Light | 200 | | | | NOSE WEAPON (Nose) |
| STILETTO BG I | `wep_01_Beam` | E | E | | Light | 6000 ✓ | | | | NOSE WEAPON (Nose) |
| DAGGER BG2 | `wep_37_Beam` | D | E | | Light | 4000 ✓ | | | | NOSE WEAPON (Nose) |
| PILUM BP | `wep_24_Beam` | B | D | | Light | 600 ✓ | | | | MAIN WEAPON1 (Fore) |
| FALCON 9AM | `wep_02_Missile` | D | D | A | Heavy | 300 ✓ | 12 ✓ | Available | | MAIN WEAPON1 (Fore) |
| BUZZARD 10AM | `wep_04_Missile` | C | D | A | Heavy | 600 ✓ | 22 ✓ | | Available | MAIN WEAPON1 (Fore) |
| DART 23 ROCKET | `wep_55_Rocket` | C | D | | Heavy | 600 ✓ | | | | MAIN WEAPON1 (Fore) |
| TERRIER SMH | `wep_05_ASMissile` | D | C | B | Heavy | 45 ✓ | **4** | Available | Available | MAIN WEAPON2 (Rear) |
| HOUND SMH | `wep_60_ASMissile` | B | C | B | Medium | 18 ✓ | **4** | Available | Available | MAIN WEAPON2 (Rear) |
| TOMAHAWK ALPHA RAIL GUN | `wep_03_Cannon` | C | C | | Medium | 75 ✓ | | | | MAIN WEAPON3 (Lower) |
**BROAD SWORD SG I is unidentified — NEEDS-HUMAN.** `Ammo Capacity 200` narrows it to
`wep_38` / `wep_41` / `wep_42_Shotgun` (all `LoadingCount = 200`); "SG" and the GUN tab fit
a shotgun. `wep_42` is excluded by range (2500 m would not share class E with `wep_38`/
`wep_41`'s 3000 m *if* the class is a pure range bucket), leaving `wep_38` vs `wep_41`,
which the panel cannot separate. Its `Damage Class D` also does not fit the `Power` bracket
below (all three shotguns are `Power ≤ 16`, i.e. bucket E), so either the identification or
the "Damage Class = bucket of `Power`" model is wrong for shotguns.
### Letter-class brackets
Sorting the identified rows by their on-disc numbers gives monotone, non-overlapping bands:
```
Range Class E: 3000 (wep_01)
D: 3500 · 4000 · 4000 · 4000 (wep_33, wep_37, wep_02, wep_05)
C: 4500 · 5000 · 5000 (wep_55, wep_04, wep_03)
B: 6500 · 6500 (wep_24, wep_60)
Damage Class E: 10 · 14 · 16 (wep_33, wep_01, wep_37)
D: 70 · 75 · 100 (wep_24, wep_04, wep_55)
C: 200 · 400 (wep_03, wep_05)
```
Two consequences for the reimplementation:
- The **thresholds are not pinned** — only bracketed (e.g. the D/C range boundary lies in
(4000, 4500]). More weapons, or a static read of the title-code table, would pin them.
- They **bracket the defaulted numbers**: `wep_02_Missile`'s defaulted `Power` sits in the
D band (≈ 17…150 by the observed edges) and `wep_60_ASMissile`'s in the C band
(≈ 150…500). 🟡 PROBABLE, and only as good as the bucket model.
## How this was captured
Container recipe (`sylph-container/mission.md`), with two corrections worth keeping:
- **Skip the intro movie with A.** The brief warns it crashes; it does not. Skipping cuts
boot-to-main-menu from ~13 min to **~1 min** under lavapipe. (Thanks: user tip.)
- **The title screen falls back to the attract loop within a few seconds**, so a
screenshot→look→tap cycle always misses it. Poll the framebuffer and tap in the same
process. Both are automated in `scratchpad/skip_intro.sh` (movie detected by frame-to-
frame RMSE; title by the green Ⓐ glyph at pixel 625,618).
- Under lavapipe the game polls input at its own low frame rate: **a 60 ms d-pad tap is
dropped roughly half the time**; 200 ms is reliable and 300 ms starts to auto-repeat.
- The Hangar hard-point weapon carousel is cycled with d-pad **down**, not left/right.
Path: title → A → LOAD GAME → slot 01 → *Load game?* **YES** → READY ROOM → ARSENAL →
type tab (LB/RB) → **Y** for the DATA SHEET → d-pad down through the list.
## Open / next
- Only **9 weapons of ~61** are revealed at 5 % completion, and none of the four whose
`LoadingCount` is defaulted (`wep_11`, `wep_28`, `wep_36`, `wep_70`) is among them.
Progressing the save (or a later save) is what unlocks the rest — the panel itself
already shows undeveloped weapons, so no points need to be spent.
- **Craft (`UNIT`-schema) defaults are not reachable this way.** The Hangar exposes exactly
one craft-level runtime number, `Gross Weight` (a class, "Light"). The defaulted craft
fields (`ShieldRatio`, `BarrelRoll_Count*`, `HoldPosition_*Ratio`, `Slalom_TurnCount_Max`,
`UsingChaffRatio`, …) are AI/flight-model constants with no UI surface; they would need
in-flight behavioural measurement or a guest-memory read, not a menu screenshot.
- The `Sight Homing` / `Lock on Overlap` on-disc partners are still unidentified.
Screenshots for every row above: `/sylph-home/re/caps/` in the container.
Evidence PNGs are committed under [`captures/weapon-datasheet/`](captures/weapon-datasheet/)
(64-colour quantized for size; the numbers stay legible).
---
# Addendum — the in-flight HUD (2026-07-28)
Reached via **TUTORIAL → BASIC CONTROLS / HEADS-UP DISPLAY** from the title menu (no
save needed). Evidence: [`captures/hud-runtime/`](captures/hud-runtime/).
## The HUD corroborates the ammo mapping, independently
The Delta Saber's HUD prints its two equipped weapons as `NOSE <TYPE> <AMMO>` and
`MAIN <TYPE> <AMMO>`. With the loadout known from the HANGAR (nose = STILETTO BG I, main =
FALCON 9AM) the HUD reads **`NOSE BM 06000`** and **`MAIN MPM 00300`** — exactly
`wep_01_Beam`'s `LoadingCount = 6000` and `wep_02_Missile`'s `300`.
This matters because it is **not** the Arsenal panel: the weapons were identified from the
HANGAR loadout, and the numbers come from a different renderer in a different game mode. It
is a genuinely independent confirmation of `Ammo Capacity == LoadingCount`.
The type tags (`BM`, `MPM`) are the same short codes as the Arsenal type tabs.
## HUD element inventory
| HUD element | Backing data | Conf. |
|---|---|---|
| `NOSE` / `MAIN` + type tag + 5-digit ammo | equipped weapon, `LoadingCount` | ✅ |
| `HEAT` / `L.HEAT` bar under each weapon | the `Heating` / `Cooling` pair | 🟡 |
| Speed readout + throttle scale (`0`, `100`, `A/B`) | craft velocity | ✅ |
| `SHIELD` and `ARMOR` bars (two separate pools) | craft `HP` + a shield pool | 🟡 |
| Target reticle: name / type / distance + ring **Armor Gauge** | target unit record | 🟡 |
| `RANGE` gauge with `MAIN` and `NOSE` tick markers | each equipped weapon's `MaximumRange`, plotted against target distance | 🟡 |
| `YOU KILLED: WARSHIPS nnnn` + a second counter | score / `ScorePoint` | ❔ |
The `RANGE` gauge is the interesting one for future work: it draws a **per-weapon marker on
a distance scale**, so a weapon whose `MaximumRange` is defaulted on disc (`wep_25`,
`wep_58`, `wep_82`) would have its value *drawn* rather than bucketed into a letter — if the
scale can be calibrated against two weapons with known ranges, that recovers a real number
where the Arsenal panel only gives a class.
## Craft velocity
Full afterburner (RT) peaked at **1193** against `UN_f001_TCAF_DeltaSaber_T`'s on-disc
`MaximumVelocity = 1200`. 🟡 PROBABLE — one observation, and the readout may have been
still climbing. Cruise sat at 350, which is *not* the record's `CruisingVelocity` (700), so
the number is the current throttle setting, not a named constant; don't read more into it.
## Notes for the next session
- The tutorials need **no save game** and are reachable in ~1 min from a cold boot, which
makes them the cheapest way back into a live flight scene.
- `tools/re-capture/autopilot.py` chases the yellow off-screen waypoint arrow (colour +
shape, `-sample` not `-resize`, ~0.65 s per detection). It **finds the arrow reliably but
oscillates** — the proportional gain is too high for the craft's turn rate. It needs a
damping/derivative term before it can actually fly a waypoint.
- The HEADS-UP DISPLAY tutorial reaches a scripted targeting segment where the ship stops
moving (target distance pinned) and the on-screen controller highlights **A**; tapping and
holding A did not advance it. **NEEDS-HUMAN**: what input that segment wants.
- Canary is unstable here: it died twice mid-session (once loading BEAM `Resource3D`, once
hanging on tutorial teardown) with no crash dump. Re-launch is cheap; just don't assume a
long session survives.

View File

@@ -0,0 +1,35 @@
# Runtime-capture harness (sylph-re container)
Screenshot-driven scripts for reading the running retail game's menus under Xenia
Canary + lavapipe, headless. They assume the container helpers `screenshot`,
`vgamepad`, `pad` are on `$PATH` and `HOME=/sylph-home/re`.
| Script | What it does |
|---|---|
| `skip_intro.sh` | Boot → main menu, unattended. Taps A only while the intro movie is actually playing (frame-to-frame RMSE), then once at the `PRESS Ⓐ BUTTON` title. Static logo screens are left alone, so a stray tap can never land on NEW GAME. |
| `wait_title.sh` | Older variant: wait for the title (green Ⓐ glyph at px 625,618) and tap A. Superseded by `skip_intro.sh`. |
| `step.sh` | One Arsenal navigation step (`down`/`up`/`next`/`prev`/`none`) + a compact capture: weapon list stacked over the `DATA SHEET`. |
| `sweep.sh` | Walk a whole weapon-type list, capturing **only** rows that show a `DATA SHEET` — locked rows (a "Conditions to Develop" panel) are detected by the brightness of the `Range Class` label box and skipped. |
| `type.sh` | Change weapon-type tab N times (RB) and report the header strip. |
| `hp.sh` / `cyc.sh` | Hangar hard-point carousel: `cyc.sh` steps it (d-pad **down**, not left/right) and captures the Name + `DATA SHEET`. |
**Input timing under lavapipe:** the game polls input at its own low frame rate, so a
60 ms d-pad tap is dropped roughly half the time. 200 ms is reliable; 300 ms starts to
auto-repeat (two rows per press).
Findings produced with these: [`docs/re/weapon-datasheet-runtime.md`](../../docs/re/weapon-datasheet-runtime.md).
## Guest-memory tools (no screenshots)
| Script | What it does |
|---|---|
| `gmem.py` | Read the live guest address space out of `/dev/shm/xenia_memory_*` (Xenia's backing file), addressed by guest VA. `find` / `read` / `words`. |
| `weapon_runtime.py` | Solve the `Weapon`/`Shell` struct layouts against the disc records and read the fields the disc defaults. |
| `unit_discover.py` | Find *which* runtime class carries a set of ID strings, assuming no vtable: tallies the word at `pointer_site - k` across distinct IDs. |
| `unit_runtime.py` | Same solver for the `unit\UN_*.tbl` definition objects (vtable `0x820af844`). Unions several snapshots — unit definitions are per-stage. |
| `schema_order.py` | Merge a sub-record's field-declaration order across all tables (topological sort); the layout check that pins fields no table ever values. |
| `order_check.py` | Test `offset = base + 4*index` for one table's sub-record against solver output. |
| `grab_tutorial.sh` | Cold-boot Canary, walk to the Nth TUTORIAL entry, wait for the stage load, snapshot guest RAM. One emulator per capture — backing out of a loaded mission wedges it. |
Snapshot first — `cp --sparse=always /dev/shm/xenia_memory_* snap.bin` (~2 s) — and
point `$GMEM_FILE` at the copy; the running emulator pegs every core under lavapipe.

View File

@@ -0,0 +1,95 @@
#!/usr/bin/env python3
"""Fly the Delta Saber toward the tutorial waypoint by chasing the yellow
off-screen direction arrow.
Detection: no PIL/numpy in the box, so the frame comes from `convert ... txt:-`.
Two things make it fast AND correct:
* `-sample` (point sampling, not `-resize`/`-scale` averaging) — a 1-px-thin
glyph keeps its true colour, so the exact colour predicate still fires while
the dump shrinks ~9x (3.5s -> 0.65s).
* shape, not just colour — the instruction-frame bars and the throttle marker
are the same dim yellow, so the arrow is the largest blob that is roughly as
tall as it is wide, with the fixed throttle marker blacklisted by position.
A ~1.1 s control loop is fast enough to converge; the earlier ~6 s loop was not.
"""
import re, subprocess, sys, time
X0, Y0, W, H = 80, 60, 820, 640 # flight view (the arrow also rides the bottom edge)
K = 3.03 # 1/0.33 sample factor
CX, CY = 640, 405 # crosshair
THROTTLE = (382, 456) # fixed yellow decoy on the speed gauge
PX = re.compile(r"^(\d+),(\d+):.*?srgb\((\d+),(\d+),(\d+)\)")
def pad(*a):
subprocess.run(["/opt/sylph/vgamepad.py", *a], check=False)
def arrow(png="/tmp/ap.png"):
subprocess.run(["/opt/sylph/screenshot.sh", png], check=False,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
out = subprocess.run(["convert", png, "-crop", f"{W}x{H}+{X0}+{Y0}", "+repage",
"-sample", "33%", "txt:-"], capture_output=True, text=True).stdout
pts = set()
for l in out.splitlines()[1:]:
m = PX.match(l)
if not m:
continue
x, y, r, g, b = (int(v) for v in m.groups())
if r >= 60 and g >= 48 and b <= 0.35 * g and (r - g) <= 0.45 * r:
pts.add((x, y))
seen, best = set(), None
for p in pts:
if p in seen:
continue
st, c = [p], []
seen.add(p)
while st:
x, y = st.pop(); c.append((x, y))
for dx in (-1, 0, 1):
for dy in (-1, 0, 1):
q = (x + dx, y + dy)
if q in pts and q not in seen:
seen.add(q); st.append(q)
xs = [q[0] for q in c]; ys = [q[1] for q in c]
cx, cy = X0 + sum(xs) / len(c) * K, Y0 + sum(ys) / len(c) * K
if abs(cx - THROTTLE[0]) < 30 and abs(cy - THROTTLE[1]) < 30:
continue
if len(c) < 8:
continue
if best is None or len(c) > best[2]:
best = (cx, cy, len(c))
return best
def main():
steps = int(sys.argv[1]) if len(sys.argv) > 1 else 25
centred = 0
for i in range(steps):
a = arrow()
if a is None:
print(f"{i:2d}: no arrow -> boost (target ahead)")
pad("trig", "RT", "0.55"); time.sleep(1.2); pad("trig", "RT", "0")
centred += 1
if centred >= 6:
print(" arrival likely — stopping"); return 0
continue
x, y, n = a
dx, dy = x - CX, y - CY
if abs(dx) < 70 and abs(dy) < 70:
centred += 1
print(f"{i:2d}: arrow ({x:.0f},{y:.0f}) centred -> boost")
pad("trig", "RT", "0.55"); time.sleep(1.2); pad("trig", "RT", "0")
continue
centred = 0
lx = max(-1.0, min(1.0, dx / 200))
ly = max(-1.0, min(1.0, dy / 160))
print(f"{i:2d}: arrow ({x:.0f},{y:.0f}) n={n} -> LX {lx:+.2f} LY {ly:+.2f}")
pad("axis", "LX", f"{lx:.2f}"); pad("axis", "LY", f"{ly:.2f}")
time.sleep(0.45)
pad("axis", "LX", "0"); pad("axis", "LY", "0")
print("steps exhausted")
return 1
sys.exit(main())

View File

@@ -0,0 +1,212 @@
#!/usr/bin/env python3
"""Memory-driven autopilot: fly and fight from the game's own world state.
The earlier screen-scraping autopilot oscillated because it only ever saw a
2-D arrow on the HUD and had no rate feedback. This one reads the actual
transform of every entity out of guest RAM, so it can do proper PD control:
the derivative term uses the craft's real body angular velocity, recovered from
two consecutive rotation matrices, not a differenced pixel position.
world state <- /dev/shm/xenia_memory_* (see gworld.py)
control -> the vgamepad FIFO, written directly at loop rate
Offsets come from flight_analyze.py and are passed in / stored in offsets.json;
nothing here hard-codes a value that was not derived from a capture.
"""
import json
import math
import os
import struct
import sys
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gworld # noqa: E402
from flight_probe import Pad # noqa: E402
# --------------------------------------------------------------- 3-D helpers
def dot(a, b):
return a[0] * b[0] + a[1] * b[1] + a[2] * b[2]
def sub(a, b):
return (a[0] - b[0], a[1] - b[1], a[2] - b[2])
def norm(a):
return math.sqrt(dot(a, a))
def clamp(v, lo=-1.0, hi=1.0):
return max(lo, min(hi, v))
def body_rates(R_prev, R, dt):
"""Angular velocity in body axes from two rotation matrices.
R rows are the craft's axes in world space, so R_prev @ R^T is the
incremental rotation; its skew part is the rotation vector.
"""
if dt <= 0:
return (0.0, 0.0, 0.0)
# M = R_prev * R^T (3x3, row-major tuples)
M = [[sum(R_prev[i * 3 + k] * R[j * 3 + k] for k in range(3)) for j in range(3)]
for i in range(3)]
wx = (M[2][1] - M[1][2]) / 2.0
wy = (M[0][2] - M[2][0]) / 2.0
wz = (M[1][0] - M[0][1]) / 2.0
return (wx / dt, wy / dt, wz / dt)
# ------------------------------------------------------------------- reader
class Flight:
def __init__(self, cfg):
self.cfg = cfg
self.w = gworld.World()
self.pos_off = cfg["pos"]
self.rot_off = cfg["rot"]
self.hp_off = cfg.get("hp")
self.player_name = cfg.get("player", "Player")
self.entities = []
self.last_scan = 0.0
def rescan(self):
self.entities = self.w.refresh()
self.last_scan = time.time()
def state(self, off):
buf = self.w.read_off(off, gworld.WINDOW)
if len(buf) < gworld.WINDOW:
return None
pos = struct.unpack_from(">3f", buf, self.pos_off)
rot = struct.unpack_from(">9f", buf, self.rot_off)
if not all(map(math.isfinite, pos + rot)):
return None
hp = struct.unpack_from(">f", buf, self.hp_off)[0] if self.hp_off else None
return {"pos": pos, "rot": rot, "hp": hp}
def player(self):
for va, off, nm in self.entities:
if self.player_name in nm:
s = self.state(off)
if s:
s["name"] = nm
return s
return None
def hostiles(self):
"""ADAN units are the enemy; the disc IDs encode the faction.
`UN_e*` / `UN_be*` = ADAN, `UN_f*` / `UN_bf*` = TCAF (ours).
"""
out = []
for va, off, nm in self.entities:
base = nm[3:]
if not (base.startswith("e") or base.startswith("be")):
continue
s = self.state(off)
if s:
s["name"] = nm
s["off"] = off
out.append(s)
return out
# ---------------------------------------------------------------- controller
class Autopilot:
KP_YAW, KD_YAW = 1.6, 0.35
KP_PITCH, KD_PITCH = 1.6, 0.35
FIRE_CONE = math.radians(6)
FIRE_RANGE = 4000.0
def __init__(self, flight, pad, log=sys.stdout):
self.f = flight
self.pad = pad
self.log = log
self.prev_rot = None
self.prev_t = None
self.target = None
self.firing = False
def pick_target(self, me):
hs = self.f.hostiles()
if not hs:
return None
# nearest, mildly preferring what is already ahead of us
def score(h):
v = sub(h["pos"], me["pos"])
d = norm(v) or 1.0
fwd = me["rot"][6:9]
ahead = dot(v, fwd) / d
return d * (1.0 if ahead > 0 else 2.0)
return min(hs, key=score)
def step(self, dt):
me = self.f.player()
if not me:
return "no-player"
R = me["rot"]
w = body_rates(self.prev_rot, R, dt) if self.prev_rot else (0, 0, 0)
self.prev_rot = R
tgt = self.pick_target(me)
if not tgt:
self.pad.axis("LX", 0.0)
self.pad.axis("LY", 0.0)
self.pad.trig("RT", 0.6)
return "no-target"
v = sub(tgt["pos"], me["pos"])
dist = norm(v) or 1.0
# into body axes: rows are right / up / forward
lx = dot(R[0:3], v)
ly = dot(R[3:6], v)
lz = dot(R[6:9], v)
yaw_err = math.atan2(lx, lz if lz > 1e-3 else 1e-3)
pitch_err = math.atan2(ly, lz if lz > 1e-3 else 1e-3)
if lz < 0: # behind us: turn the short way, hard
yaw_err = math.copysign(math.pi / 2, lx if lx else 1.0)
stick_x = clamp(self.KP_YAW * yaw_err - self.KD_YAW * w[1])
stick_y = clamp(-(self.KP_PITCH * pitch_err - self.KD_PITCH * w[0]))
self.pad.axis("LX", stick_x)
self.pad.axis("LY", stick_y)
self.pad.trig("RT", 1.0 if dist > 1500 else 0.3)
aligned = abs(yaw_err) < self.FIRE_CONE and abs(pitch_err) < self.FIRE_CONE
want_fire = aligned and dist < self.FIRE_RANGE
if want_fire != self.firing:
(self.pad.press if want_fire else self.pad.release)(self.f.cfg["fire_btn"])
self.firing = want_fire
return (f"tgt={tgt['name'][3:20]:<18} d={dist:8.0f} yaw={math.degrees(yaw_err):+6.1f} "
f"pit={math.degrees(pitch_err):+6.1f} stick=({stick_x:+.2f},{stick_y:+.2f}) "
f"fire={int(self.firing)} hp={me['hp']}")
def run(self, seconds, hz=15.0):
self.f.rescan()
t0 = time.time()
last = t0
while time.time() - t0 < seconds:
t = time.time()
if t - self.f.last_scan > 3.0:
self.f.rescan()
msg = self.step(t - last)
last = t
print(f"[{t-t0:6.1f}] {msg}", file=self.log, flush=True)
time.sleep(max(0, 1.0 / hz - (time.time() - t)))
self.pad.reset()
def main():
cfg = json.load(open(sys.argv[1]))
secs = float(sys.argv[2]) if len(sys.argv) > 2 else 60.0
ap = Autopilot(Flight(cfg), Pad())
ap.run(secs)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,207 @@
#!/usr/bin/env python3
"""Fly and fight from guest memory.
World state comes from entities2.py's typing rule: a live entity's position
triple sits at a fixed offset before its definition pointer, so one scan of the
entity heap yields every craft in the scene *with its unit type* — which is what
separates enemies from wingmen, capital ships and the thousands of moving
particles.
Control is PD on the aiming error, with the derivative taken from the craft's
own body angular velocity (recovered from two consecutive orientation matrices)
rather than from the differenced error — that is what the old screen-scraping
autopilot lacked, and why it oscillated.
Usage: autopilot3.py <config.json> [seconds] [--dry]
"""
import json
import math
import os
import struct
import sys
import time
from collections import Counter
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
import gworld # noqa: E402
import entities2 # noqa: E402
from flight_probe import Pad # noqa: E402
HOSTILE = ("e", "be") # UN_e* / UN_be* are ADAN; UN_f*/UN_bf* are ours
def unit_faction(nm):
base = nm[3:]
return "ADAN" if base.startswith("be") or base.startswith("e") else "TCAF"
class World:
def __init__(self, cfg):
self.cfg = cfg
self.w = gworld.World()
self.fd, self.size = self.w.fd, self.w.size
self.defs = entities2.definitions(self.w)
self.delta = cfg["def_delta"]
self.rot_delta = cfg["rot_delta"]
self.rot_stride = cfg.get("rot_stride", 12)
self.fwd_row = cfg["fwd_row"]
self.fwd_sign = cfg["fwd_sign"]
self.ents = []
def rescan(self):
movers = entities2.moving(self.fd, self.size, dt=0.35,
va_range=(self.cfg["va_lo"], self.cfg["va_hi"]))
ents = entities2.typed(self.fd, self.defs, movers, self.delta)
uniq = {}
for off, nm, pos, sp in ents:
uniq.setdefault(off, (off, nm, pos, sp))
self.ents = list(uniq.values())
return self.ents
def pos(self, off):
b = os.pread(self.fd, 12, off)
return np.array(struct.unpack(">3f", b)) if len(b) == 12 else None
def rot(self, off):
n = self.rot_stride * 2 + 12
b = os.pread(self.fd, n, off + self.rot_delta)
if len(b) < n:
return None
M = np.array([struct.unpack_from(">3f", b, self.rot_stride * r) for r in range(3)])
if not np.all(np.isfinite(M)):
return None
if np.max(np.abs(M @ M.T - np.eye(3))) > 5e-3:
return None
return M
def clamp(v, lo=-1.0, hi=1.0):
return max(lo, min(hi, v))
def body_rate(Mprev, M, dt):
if Mprev is None or M is None or dt <= 0:
return np.zeros(3)
D = Mprev @ M.T
w = np.array([D[2, 1] - D[1, 2], D[0, 2] - D[2, 0], D[1, 0] - D[0, 1]]) / 2.0
return w / dt
class Autopilot:
KP, KD = 2.2, 0.45
FIRE_CONE = math.radians(10)
FIRE_RANGE = 6000.0
def __init__(self, world, pad, dry=False):
self.W = world
self.pad = pad
self.dry = dry
self.prevM = None
self.firing = False
def me(self):
for off, nm, pos, sp in self.W.ents:
if "Player" in nm:
return off, nm
return None, None
def step(self, dt):
off, nm = self.me()
if off is None:
return "no-player"
p = self.W.pos(off)
M = self.W.rot(off)
if p is None or M is None:
return "no-state"
fwd = M[self.W.fwd_row] * self.W.fwd_sign
rows = [M[i] for i in range(3)]
right = rows[(self.W.fwd_row + 1) % 3]
up = np.cross(fwd, right)
w = body_rate(self.prevM, M, dt)
self.prevM = M
# nearest hostile, preferring what is already in front
best, bestscore = None, 1e18
for eoff, enm, epos, esp in self.W.ents:
if unit_faction(enm) != "ADAN":
continue
q = self.W.pos(eoff)
if q is None:
continue
v = q - p
d = float(np.linalg.norm(v))
if d < 1e-3:
continue
# Prefer targets we can actually bring the nose onto. Nearest-first
# picks whatever is closest even at 90 deg off the nose, and closing
# on an off-boresight target only raises the bearing rate -- which is
# exactly the lag that kept the first run outside its firing cone.
ahead = float(v @ fwd) / d
ang = math.acos(max(-1.0, min(1.0, ahead)))
score = d * (1.0 + 3.0 * (ang / math.pi) ** 2)
if score < bestscore:
best, bestscore = (eoff, enm, q, v, d), score
if best is None:
if not self.dry:
self.pad.axis("LX", 0.0)
self.pad.axis("LY", 0.0)
return "no-hostiles"
eoff, enm, q, v, d = best
lx = float(v @ right)
ly = float(v @ up)
lz = float(v @ fwd)
yaw = math.atan2(lx, lz if abs(lz) > 1e-3 else 1e-3)
pitch = math.atan2(ly, lz if abs(lz) > 1e-3 else 1e-3)
if lz < 0:
yaw = math.copysign(math.pi / 2, lx if lx else 1.0)
sx = clamp(self.KP * yaw - self.KD * float(w @ up))
sy = clamp(-(self.KP * pitch - self.KD * float(w @ right)))
aligned = abs(yaw) < self.FIRE_CONE and abs(pitch) < self.FIRE_CONE
fire = aligned and d < self.FIRE_RANGE
if not self.dry:
self.pad.axis("LX", sx)
self.pad.axis("LY", sy)
if fire != self.firing:
(self.pad.press if fire else self.pad.release)("RB")
self.firing = fire
self.shots = getattr(self, "shots", 0) + (1 if fire else 0)
return (f"tgt={enm[3:24]:<22} d={d:8.0f} yaw={math.degrees(yaw):+6.1f} "
f"pit={math.degrees(pitch):+6.1f} stick=({sx:+.2f},{sy:+.2f}) fire={int(fire)}")
def run(self, secs, hz=10.0):
t0 = time.time()
last = t0
last_scan = 0.0
while time.time() - t0 < secs:
t = time.time()
if t - last_scan > 2.5:
ents = self.W.rescan()
last_scan = t
c = Counter(unit_faction(e[1]) for e in ents)
print(f"[{t-t0:6.1f}] rescan: {len(ents)} entities {dict(c)}", flush=True)
print(f"[{t-t0:6.1f}] {self.step(t - last)}", flush=True)
last = t
time.sleep(max(0, 1.0 / hz - (time.time() - t)))
if not self.dry:
self.pad.reset()
def main():
cfg = json.load(open(sys.argv[1]))
secs = float(sys.argv[2]) if len(sys.argv) > 2 else 60.0
dry = "--dry" in sys.argv
W = World(cfg)
W.rescan()
ap = Autopilot(W, Pad(), dry=dry)
ap.run(secs)
if __name__ == "__main__":
main()

144
tools/re-capture/binq.py Normal file
View File

@@ -0,0 +1,144 @@
#!/usr/bin/env python3
"""Ask questions of a ctrl_probe.py capture: which words answer to which input?
The capture is a window of the player entity object sampled every tick, tagged
with the pad state that produced it. That makes the interesting question
mechanical: for each 4-byte offset, does its value during phase X differ from
its value during the rest phases either side? A word that only moves while `RT`
is held is that input's state — a throttle setting, an afterburner tank, a heat
gauge — and one that moves in *every* phase is just live physics.
Sub-commands
phases per-phase mean of every offset that moves at all
respond <phase> offsets that move during <phase> and not at rest
near <value> [tol] offsets whose first sample is ~= value (HUD anchor)
trace <off> [off...] full time series of specific offsets (pos-relative hex)
"""
import os
import struct
import sys
import numpy as np
HDR = b"SYLPHCTR"
REC = struct.Struct("<d32sff3f")
def load(path):
with open(path, "rb") as f:
blob = f.read()
assert blob[:8] == HDR, "not a ctrl_probe capture"
n, win, back = struct.unpack_from("<III", blob, 8)
stride = REC.size + win
base = 8 + 12
ts, phase, sp, dodge, pos, wins = [], [], [], [], [], []
for i in range(n):
o = base + i * stride
t, ph, s, dg, x, y, z = REC.unpack_from(blob, o)
ts.append(t)
phase.append(ph.split(b"\0")[0].decode())
sp.append(s)
dodge.append(dg)
pos.append((x, y, z))
wins.append(blob[o + REC.size:o + REC.size + win])
# A run that ended in GAME OVER keeps sampling a dead object, and those
# frames dominate every statistic. $BINQ_TMAX truncates the capture to the
# part that was still flying.
tmax = float(os.environ.get("BINQ_TMAX", "inf"))
if tmax < float("inf"):
keep = [i for i, t in enumerate(ts) if t <= tmax]
n = len(keep)
ts = [ts[i] for i in keep]
phase = [phase[i] for i in keep]
sp = [sp[i] for i in keep]
dodge = [dodge[i] for i in keep]
pos = [pos[i] for i in keep]
wins = [wins[i] for i in keep]
A = np.frombuffer(b"".join(wins), dtype=">f4").reshape(n, win // 4).astype(np.float64)
U = np.frombuffer(b"".join(wins), dtype=">u4").reshape(n, win // 4)
return dict(n=n, win=win, back=back, t=np.array(ts), phase=phase,
speed=np.array(sp), dodge=np.array(dodge),
pos=np.array(pos), F=A, U=U)
def label(d, i):
return f"pos{i * 4 - d['back']:+#07x}"
def finite(d):
F = d["F"]
return np.all(np.isfinite(F), axis=0) & (np.max(np.abs(F), axis=0) < 1e12)
def cmd_phases(d, args):
ok = finite(d)
phases = []
for p in d["phase"]:
if p not in phases:
phases.append(p)
F = d["F"]
mv = np.zeros(F.shape[1])
means = {}
for p in phases:
m = np.array([x == p for x in d["phase"]])
means[p] = F[m].mean(axis=0)
for p in phases:
mv = np.maximum(mv, np.abs(means[p] - means[phases[0]]))
idx = np.flatnonzero(ok & (mv > 1e-3))
order = idx[np.argsort(-mv[idx])][:int(args[0]) if args else 25]
print("offset " + "".join(f"{p[:8]:>10}" for p in phases))
for i in order:
print(f"{label(d, i):<10}" + "".join(f"{means[p][i]:10.2f}" for p in phases))
def cmd_respond(d, args):
want = args[0]
F, ok = d["F"], finite(d)
inp = np.array([p == want for p in d["phase"]])
rest = np.array([p.startswith("rest") or p == "base" for p in d["phase"]])
if not inp.any():
sys.exit(f"no phase {want!r}")
# A word answering to this input must move *while it is held* and be quiet
# at rest; a word that also moves at rest is live physics, not the input.
a = F[inp]
r = F[rest]
d_in = a.max(axis=0) - a.min(axis=0)
d_rest = r.max(axis=0) - r.min(axis=0)
score = d_in - 2.0 * d_rest
idx = np.flatnonzero(ok & (d_in > 1e-3) & (score > 0))
for i in idx[np.argsort(-score[idx])][:20]:
print(f"{label(d, i):<10} in-phase {a[:, i].min():12.3f}..{a[:, i].max():12.3f}"
f" at-rest {r[:, i].min():12.3f}..{r[:, i].max():12.3f}")
if not len(idx):
print("(nothing moves under this input that is quiet at rest)")
def cmd_near(d, args):
v = float(args[0])
tol = float(args[1]) if len(args) > 1 else max(1e-3, abs(v) * 1e-3)
F, U = d["F"], d["U"]
for i in np.flatnonzero(np.abs(F[0] - v) <= tol):
print(f"{label(d, i):<10} f32 {F[0, i]:.4f} -> {F[-1, i]:.4f}")
for i in np.flatnonzero(np.abs(U[0].astype(np.float64) - v) <= tol):
print(f"{label(d, i):<10} u32 {U[0, i]} -> {U[-1, i]}")
def cmd_trace(d, args):
offs = [int(a, 0) for a in args]
idx = [(o + d["back"]) // 4 for o in offs]
print("t phase speed " + " ".join(f"{o:+#07x}" for o in offs))
for k in range(d["n"]):
print(f"{d['t'][k]:6.2f} {d['phase'][k]:<12} {d['speed'][k]:6.0f} "
+ " ".join(f"{d['F'][k, i]:8.2f}" for i in idx))
def main():
d = load(sys.argv[1])
print(f"# {d['n']} ticks, window {d['win']:#x} bytes, back {d['back']:#x}")
cmd = sys.argv[2] if len(sys.argv) > 2 else "phases"
{"phases": cmd_phases, "respond": cmd_respond, "near": cmd_near,
"trace": cmd_trace}[cmd](d, sys.argv[3:])
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,165 @@
#!/usr/bin/env python3
"""Learn which body axis each stick drives, and which button fires.
The first autopilot run nulled yaw but held a steady ~27 deg pitch error, which
is the signature of a wrong pitch axis or sign — guessing `right = row0` and
`up = fwd x right` assumes a handedness the game need not share. So measure it:
hold each stick axis, read the craft's body angular velocity from consecutive
orientation matrices, and see which body axis actually responds and in which
direction.
The fire button is found the same way — by consequence, not assumption: the
nose ammo counter in the player's object must go down.
Usage: calibrate.py <config.json> [out.json]
"""
import json
import math
import os
import struct
import sys
import time
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
import gworld # noqa: E402
import entities2 # noqa: E402
from flight_probe import Pad # noqa: E402
def player_off(w, fd, size, delta):
defs = entities2.definitions(w)
for _ in range(6):
mv = entities2.moving(fd, size, dt=0.5)
ents = entities2.typed(fd, defs, mv, delta)
for off, nm, pos, sp in ents:
if "Player" in nm:
return off
time.sleep(0.5)
return None
def read_rot(fd, off, rot_delta, stride):
n = stride * 2 + 12
b = os.pread(fd, n, off + rot_delta)
if len(b) < n:
return None
M = np.array([struct.unpack_from(">3f", b, stride * r) for r in range(3)])
if not np.all(np.isfinite(M)) or np.max(np.abs(M @ M.T - np.eye(3))) > 5e-3:
return None
return M
def mean_body_rate(fd, off, cfg, secs=2.0, hz=8.0):
"""Mean angular velocity expressed in the craft's own axes."""
prev, t_prev = None, None
acc, n = np.zeros(3), 0
end = time.time() + secs
while time.time() < end:
t = time.time()
M = read_rot(fd, off, cfg["rot_delta"], cfg.get("rot_stride", 12))
if M is not None and prev is not None:
dt = t - t_prev
D = prev @ M.T
w = np.array([D[2, 1] - D[1, 2], D[0, 2] - D[2, 0], D[1, 0] - D[0, 1]]) / 2.0
if dt > 0 and np.linalg.norm(w) < 0.5:
# into body axes
acc += M @ (w / dt)
n += 1
if M is not None:
prev, t_prev = M, t
time.sleep(max(0, 1.0 / hz - (time.time() - t)))
return acc / max(n, 1)
def main():
cfg = json.load(open(sys.argv[1]))
out = sys.argv[2] if len(sys.argv) > 2 else sys.argv[1]
w = gworld.World()
fd, size = w.fd, w.size
off = player_off(w, fd, size, cfg["def_delta"])
if off is None:
sys.exit("player not found")
print(f"# player pos va {gmem.primary_va(off):#010x}")
pad = Pad()
res = {}
for axis, name in (("LX", "yaw"), ("LY", "pitch")):
pad.reset()
time.sleep(1.0)
base = mean_body_rate(fd, off, cfg, 1.2)
pad.axis(axis, 0.9)
time.sleep(0.4)
wpos = mean_body_rate(fd, off, cfg, 2.0)
pad.axis(axis, 0.0)
time.sleep(1.2)
d = wpos - base
k = int(np.argmax(np.abs(d)))
res[name] = {"axis": k, "sign": 1 if d[k] > 0 else -1,
"mag": float(abs(d[k]))}
print(f"# {axis} (+0.9) -> body rate {d.round(3)} => {name} axis {k} "
f"sign {res[name]['sign']:+d}")
pad.reset()
# ---- fire button: the nose ammo counter must fall -----------------------
blob = os.pread(fd, 0x1000, max(0, off - 0x800))
ammo_offs = []
for i in range(0, len(blob) - 3, 4):
(u,) = struct.unpack_from(">I", blob, i)
(f,) = struct.unpack_from(">f", blob, i)
if u == 6000 or (math.isfinite(f) and abs(f - 6000.0) < 0.5):
ammo_offs.append((max(0, off - 0x800) + i) - off)
print(f"# ammo-like words (==6000) near the player: "
f"{[hex(o) for o in ammo_offs] or 'none'}")
def ammo():
vals = []
for d in ammo_offs:
b = os.pread(fd, 4, off + d)
if len(b) == 4:
vals.append(struct.unpack(">I", b)[0])
return vals
fire_btn = None
if ammo_offs:
for btn in ("RB", "LB", "A", "B", "X", "Y"):
before = ammo()
pad.press(btn)
time.sleep(1.2)
pad.release(btn)
time.sleep(0.5)
after = ammo()
drop = [a - b for a, b in zip(before, after)]
print(f"# {btn}: ammo delta {drop}")
if any(x > 0 for x in drop):
fire_btn = btn
break
for trig in ("RT", "LT"):
if fire_btn:
break
before = ammo()
pad.trig(trig, 1.0)
time.sleep(1.2)
pad.trig(trig, 0.0)
time.sleep(0.5)
after = ammo()
drop = [a - b for a, b in zip(before, after)]
print(f"# {trig}: ammo delta {drop}")
if any(x > 0 for x in drop):
fire_btn = trig
pad.reset()
cfg["yaw_axis"] = res["yaw"]["axis"]
cfg["yaw_sign"] = res["yaw"]["sign"]
cfg["pitch_axis"] = res["pitch"]["axis"]
cfg["pitch_sign"] = res["pitch"]["sign"]
cfg["fire"] = fire_btn
cfg["ammo_offs"] = ammo_offs
json.dump(cfg, open(out, "w"), indent=1)
print("\n" + json.dumps(cfg, indent=1))
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,272 @@
#!/usr/bin/env python3
"""Which control is the throttle, and where does the craft keep its own state?
Two questions, one flight. Both are answered by *consequence* rather than by
reading a field we hope is the right one:
* **Throttle.** The craft's speed is measured from its own position — a finite
difference on the position triple in guest RAM — so no speed field has to be
found first. The probe then holds each candidate input in turn and asks which
one changes that measured speed. (`findspeed.py` failed the other way round:
it assumed `RT` was the throttle and went looking for a field that rose.)
* **Own state.** Every tick also copies a window of the player entity object.
Afterwards, offsets whose float value tracks the measured speed are candidate
speed/throttle fields, and offsets that only ever *fall* are candidate
hull/shield/ammo — the numbers survival needs.
Flying straight into a firefight for 90 s is how earlier runs died, so the loop
keeps the collision avoidance from navigator.py armed the whole time and marks
any sample where it had to intervene: a phase that had to dodge is not a clean
speed measurement, and says so rather than being quietly averaged in.
Usage: ctrl_probe.py <config.json> <out-prefix> [hold_s] [settle_s]
"""
import json
import math
import os
import struct
import sys
import time
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
import navigator # noqa: E402
from flight_probe import Pad # noqa: E402
# 0x800 each way: the shield lives at pos+0x430 (own_state.py), which a
# 0x400 window silently cropped out of the first capture.
WIN_BACK = 0x800 # bytes of the player object kept before the position
WIN_FWD = 0x800 # ...and after
WIN = WIN_BACK + WIN_FWD
HZ = 20.0
# (label, [pad commands]) — everything the pad can do that might be a throttle.
# RB is left out: it is the fire button (autopilot-memory-driven.md) and firing
# during a speed measurement only invites return fire.
CANDIDATES = [
("RT", [("trig", "RT", 1.0)]),
("LT", [("trig", "LT", 1.0)]),
("RT+LT", [("trig", "RT", 1.0), ("trig", "LT", 1.0)]),
("A", [("press", "A")]),
("B", [("press", "B")]),
("X", [("press", "X")]),
("Y", [("press", "Y")]),
("LB", [("press", "LB")]),
("LS", [("press", "LS")]),
("RS", [("press", "RS")]),
("RY_up", [("axis", "RY", -1.0)]),
("RY_down", [("axis", "RY", 1.0)]),
("RX_right", [("axis", "RX", 1.0)]),
("dpad_up", [("dpad", "up")]),
("dpad_down", [("dpad", "down")]),
("dpad_left", [("dpad", "left")]),
("dpad_right", [("dpad", "right")]),
]
def apply(pad, cmds):
for c in cmds:
if c[0] == "trig":
pad.trig(c[1], c[2])
elif c[0] == "axis":
pad.axis(c[1], c[2])
elif c[0] == "press":
pad.press(c[1])
elif c[0] == "dpad":
pad.f.write(f"dpad {c[1]}\n")
def clear(pad):
pad.reset()
pad.f.write("dpad center\n")
class Run:
def __init__(self, cfg, prefix, hold, settle):
self.W = navigator.World(cfg)
self.nav = navigator.Navigator(self.W, None, dry=True)
self.prefix = prefix
self.hold = hold
self.settle = settle
self.pad = Pad()
self.rows = [] # (t, phase, pos, speed, dodged)
self.win = [] # raw window bytes per tick
ents = self.W.scan()
me = [(off, va) for off, va in ents if "Player" in self.W.defs[va]]
if not me:
sys.exit("player entity not in the scan — not in flight?")
self.me_off = me[0][0]
self.me_name = self.W.defs[me[0][1]]
print(f"# player {self.me_name} pos off {self.me_off:#x} "
f"va {gmem.primary_va(self.me_off):#x} | {len(ents)} entities",
flush=True)
# ------------------------------------------------------------- helpers
def pos(self):
return self.W.pos(self.me_off)
def sticks_for(self, vec):
"""Stick deflections that point the nose along `vec` (escape steering)."""
M = self.W.rot(self.me_off)
if M is None:
return 0.0, 0.0
fwd = M[self.W.fwd_row] * self.W.fwd_sign
right = M[(self.W.fwd_row + 1) % 3]
up = np.cross(fwd, right)
ez = float(np.dot(vec, fwd))
yaw = math.atan2(float(np.dot(vec, right)), ez if abs(ez) > 1e-3 else 1e-3)
pitch = math.atan2(float(np.dot(vec, up)), ez if abs(ez) > 1e-3 else 1e-3)
return (max(-1.0, min(1.0, 2.0 * yaw)), max(-1.0, min(1.0, -2.0 * pitch)))
# ---------------------------------------------------------------- phase
def phase(self, label, cmds, secs):
clear(self.pad)
apply(self.pad, cmds)
t_end = time.time() + secs
dodged = False
while time.time() < t_end:
t = time.time()
p = self.pos()
if p is None:
break
self.rows.append([t, label, p, 0.0, 0])
self.win.append(os.pread(self.W.fd, WIN, self.me_off - WIN_BACK))
# avoidance runs at 5 Hz; a real threat overrides the phase and the
# samples from here on are flagged
if len(self.rows) % 4 == 0:
ents = self.W.sample(t)
me = [e for e in ents if e[0] == self.me_off]
if me:
_, _, mp, mv, mr = me[0]
push, worst = self.nav.avoidance(mp, mv, mr, ents, self.me_off)
if float(np.linalg.norm(push)) > 0.6:
sx, sy = self.sticks_for(navigator.norm(push))
self.pad.axis("LX", sx)
self.pad.axis("LY", sy)
dodged = True
self.rows[-1][4] = 1
elif dodged:
self.pad.axis("LX", 0.0)
self.pad.axis("LY", 0.0)
time.sleep(max(0.0, 1.0 / HZ - (time.time() - t)))
return dodged
def run(self):
t0 = time.time()
self.phase("base", [], self.settle * 2)
for label, cmds in CANDIDATES:
d = self.phase(label, cmds, self.hold)
self.phase(f"rest_{label}", [], self.settle)
sp = self.phase_speed(label)
print(f"[{time.time()-t0:6.1f}] {label:<10} "
f"v0={sp[0]:7.1f} v1={sp[1]:7.1f} d={sp[1]-sp[0]:+7.1f}"
f"{' (dodged)' if d else ''}", flush=True)
clear(self.pad)
self.speeds()
self.dump()
# ------------------------------------------------------------ analysis
def speeds(self):
"""Fill in per-tick speed by central difference on position."""
for i, r in enumerate(self.rows):
j, k = max(0, i - 2), min(len(self.rows) - 1, i + 2)
dt = self.rows[k][0] - self.rows[j][0]
if dt > 1e-3:
r[3] = float(np.linalg.norm(self.rows[k][2] - self.rows[j][2])) / dt
def phase_speed(self, label):
"""(speed early, speed late) within a phase — needs speeds() first."""
self.speeds()
v = [r[3] for r in self.rows if r[1] == label]
if len(v) < 6:
return (0.0, 0.0)
n = max(2, len(v) // 4)
return (float(np.mean(v[:n])), float(np.mean(v[-n:])))
def dump(self):
with open(self.prefix + ".csv", "w") as f:
f.write("t,phase,x,y,z,speed,dodged\n")
t0 = self.rows[0][0]
for t, ph, p, sp, dg in self.rows:
f.write(f"{t-t0:.3f},{ph},{p[0]:.3f},{p[1]:.3f},{p[2]:.3f},"
f"{sp:.3f},{dg}\n")
with open(self.prefix + ".bin", "wb") as f:
f.write(b"SYLPHCTR")
f.write(struct.pack("<III", len(self.rows), WIN, WIN_BACK))
t0 = self.rows[0][0]
for (t, ph, p, sp, dg), w in zip(self.rows, self.win):
f.write(struct.pack("<d32sff3f", t - t0, ph.encode()[:32], sp,
float(dg), *[float(c) for c in p]))
f.write(w.ljust(WIN, b"\0"))
print(f"# wrote {self.prefix}.csv and {self.prefix}.bin "
f"({len(self.rows)} ticks)", flush=True)
# ---- per-phase summary
print("\n# phase n v_early v_late delta dodged")
order, seen = [], set()
for r in self.rows:
if r[1] not in seen:
seen.add(r[1])
order.append(r[1])
for ph in order:
v = [r[3] for r in self.rows if r[1] == ph]
dg = sum(r[4] for r in self.rows if r[1] == ph)
if len(v) < 6:
continue
n = max(2, len(v) // 4)
a, b = float(np.mean(v[:n])), float(np.mean(v[-n:]))
print(f" {ph:<12} {len(v):4d} {a:9.1f} {b:9.1f} {b-a:+9.1f} {dg:5d}")
# ---- which words in the object track speed, and which only fall
W_ = np.frombuffer(b"".join(x.ljust(WIN, b"\0") for x in self.win),
dtype=">f4").reshape(len(self.win), WIN // 4)
sp = np.array([r[3] for r in self.rows], dtype=np.float64)
with np.errstate(invalid="ignore", over="ignore"):
A = W_.astype(np.float64)
ok = np.all(np.isfinite(A), axis=0) & (np.max(np.abs(A), axis=0) < 1e9)
var = np.std(A, axis=0)
cand = np.flatnonzero(ok & (var > 1e-6))
cor = []
for i in cand:
c = np.corrcoef(A[:, i], sp)[0, 1]
if np.isfinite(c):
cor.append((abs(c), c, i))
cor.sort(reverse=True)
print("\n# object words correlating with measured speed "
"(offset relative to the position triple)")
for ac, c, i in cor[:12]:
off = i * 4 - WIN_BACK
print(f" pos{off:+#07x} r={c:+.3f} "
f"range {A[:, i].min():.3f} .. {A[:, i].max():.3f}")
print("\n# words that never rise (candidate hull / shield / ammo)")
shown = 0
for i in cand:
col = A[:, i]
if col[-1] >= col[0] - 1e-6:
continue
if np.max(np.diff(col)) > 1e-6:
continue
off = i * 4 - WIN_BACK
print(f" pos{off:+#07x} {col[0]:.3f} -> {col[-1]:.3f}")
shown += 1
if shown >= 12:
break
if not shown:
print(" (none — nothing in the window decreased monotonically)")
def main():
cfg = json.load(open(sys.argv[1]))
prefix = sys.argv[2]
hold = float(sys.argv[3]) if len(sys.argv) > 3 else 4.0
settle = float(sys.argv[4]) if len(sys.argv) > 4 else 2.5
Run(cfg, prefix, hold, settle).run()
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# One background task: bind the player's transform, then run the control probe.
#
# Everything that must not be interrupted lives in a single task here, because
# the display and the emulator both die on their own every few minutes in this
# container and nothing may depend on surviving between tool calls.
#
# Assumes the game is ALREADY in flight (launch_mission.sh). Pass --boot to
# have it get there itself.
set -u
export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy DISPLAY=:98
export PYTHONPATH=/sylph-home/.local/lib/python3.12/site-packages
SD="$(cd "$(dirname "$0")" && pwd)"
CFG=/tmp/nav-live.json
PRE=/tmp/ctrl
HOLD=3.0
SETTLE=2.0
if [ "${1:-}" = "--boot" ]; then
shift
"$SD/launch_mission.sh" fly || { echo "BOOT FAILED"; exit 1; }
fi
[ $# -ge 1 ] && CFG="$1"
[ $# -ge 2 ] && PRE="$2"
[ $# -ge 3 ] && HOLD="$3"
[ $# -ge 4 ] && SETTLE="$4"
python3 "$SD/entities2.py" self 0x130 "$CFG" || { echo "TRANSFORM BIND FAILED"; exit 1; }
echo "--- config: $(cat "$CFG")"
python3 "$SD/ctrl_probe.py" "$CFG" "$PRE" "$HOLD" "$SETTLE"
echo "PROBE DONE"

9
tools/re-capture/cyc.sh Executable file
View File

@@ -0,0 +1,9 @@
#!/usr/bin/env bash
# Step the Hangar weapons-container carousel right and capture the Name+DATA SHEET.
set -u
export HOME=/sylph-home/re
OUT=/sylph-home/re/caps; mkdir -p "$OUT"
vgamepad dpad right; sleep 0.30; vgamepad dpad center; sleep 3.0
screenshot /tmp/h.png >/dev/null 2>&1
convert /tmp/h.png -crop 530x480+700+95 +repage "$OUT/$1.png"
echo "$OUT/$1.png"

View File

@@ -0,0 +1,219 @@
#!/usr/bin/env python3
"""Type every live entity, and locate the player, via the definition pointer.
A live entity object keeps a pointer to its parsed `.tbl` definition (vtable
0x820af844, addresses known) at a **fixed offset from its position triple**.
Finding that offset once types every moving object in the scene at a stroke:
enemies, friendlies and us, separated from the thousands of moving particles.
The offset is *derived*, not assumed: for every moving triple, every nearby word
is checked against the set of definition addresses, and the winning delta is the
one that repeats across many independent entities.
Sub-commands:
delta find the (position -> definition pointer) offset
list [delta] type every live entity and print it
self [delta] the player's entity, its object window, and its orientation
"""
import math
import os
import struct
import sys
import time
from collections import Counter
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
import gworld # noqa: E402
def definitions(w):
out = {}
for off in w.scan_vtable(gworld.DEF_VTABLE):
nm = w.name_of(off)
if nm and nm.startswith("UN_"):
va = gmem.primary_va(off)
if va is not None:
out[struct.pack(">I", va)] = nm
return out
# Entity objects live in one heap region; scanning only it turns a 250 MB
# double-read into a few MB. That matters for more than speed: the full scan
# competes with the emulator for every core under lavapipe, and a game that
# does not advance a frame between the two samples has nothing "moving" in it.
ENT_VA_LO, ENT_VA_HI = 0xBD000000, 0xBE000000
def moving(fd, size, dt=0.5, lo=1.0, hi=4000.0, va_range=(ENT_VA_LO, ENT_VA_HI)):
def arrays():
if va_range:
f0, f1 = gmem.va_to_off(va_range[0]), gmem.va_to_off(va_range[1])
else:
f0, f1 = 0, size
for a, b in gmem.extents(fd, size):
a, b = max(a, f0), min(b, f1)
n = (b - a) // 4 * 4
if n >= 64:
yield a, np.frombuffer(os.pread(fd, n, a), dtype=">f4")
s0 = dict(arrays())
t0 = time.time()
time.sleep(dt)
s1 = dict(arrays())
t1 = time.time()
out = []
# Match extents by their file offset. Zipping the two lists positionally is
# wrong: the game allocates between the samples, the sparse extent layout
# shifts, and every subsequent pair misaligns -- which silently reports
# almost nothing as moving.
for o, a in s0.items():
b = s1.get(o)
if b is None or len(a) != len(b):
continue
with np.errstate(invalid="ignore"):
af = np.nan_to_num(a.astype(np.float64), nan=0, posinf=0, neginf=0)
bf = np.nan_to_num(b.astype(np.float64), nan=0, posinf=0, neginf=0)
d = bf - af
idx = np.flatnonzero(np.abs(d) > 1e-4)
cand = idx[np.isin(idx + 1, idx) & np.isin(idx + 2, idx)]
for i in cand:
v = np.array([d[i], d[i + 1], d[i + 2]])
sp = float(np.linalg.norm(v)) / (t1 - t0)
if lo < sp < hi:
out.append((o + int(i) * 4, tuple(af[i:i + 3]), sp))
return out
def find_delta(fd, defs, movers, radius=0x400):
votes = Counter()
for off, pos, sp in movers[:4000]:
lo = max(0, off - radius)
blob = os.pread(fd, radius * 2, lo)
for k in range(0, len(blob) - 3, 4):
if blob[k:k + 4] in defs:
votes[(lo + k) - off] += 1
return votes
def typed(fd, defs, movers, delta):
out = []
for off, pos, sp in movers:
try:
b = os.pread(fd, 4, off + delta)
except OSError:
continue
nm = defs.get(b)
if nm:
out.append((off, nm, pos, sp))
return out
def main():
cmd = sys.argv[1] if len(sys.argv) > 1 else "delta"
w = gworld.World()
fd, size = w.fd, w.size
defs = definitions(w)
print(f"# {len(defs)} unit definitions", flush=True)
movers = moving(fd, size)
print(f"# {len(movers)} moving triples", flush=True)
if cmd == "delta":
votes = find_delta(fd, defs, movers)
print("# candidate (position -> definition pointer) deltas:")
for d, n in votes.most_common(8):
print(f" {d:+#08x} seen {n}")
return
delta = int(sys.argv[2], 0) if len(sys.argv) > 2 else 0x130
ents = typed(fd, defs, movers, delta)
uniq = {}
for off, nm, pos, sp in ents:
uniq.setdefault((nm, tuple(round(c, 1) for c in pos)), (off, nm, pos, sp))
ents = list(uniq.values())
print(f"# {len(ents)} typed live entities (delta {delta:+#x})")
if cmd == "list":
c = Counter(nm for _, nm, _, _ in ents)
for nm, k in c.most_common():
print(f" {k:4d} {nm}")
for off, nm, pos, sp in sorted(ents, key=lambda e: e[1])[:60]:
print(f" {gmem.primary_va(off):#010x} {nm:<40} "
f"({pos[0]:+9.1f},{pos[1]:+9.1f},{pos[2]:+9.1f}) {sp:7.1f}/s")
return
if cmd == "self":
me = [e for e in ents if "Player" in e[1]]
if not me:
sys.exit("player entity not found")
off, nm, pos, sp = me[0]
print(f"# player entity: pos va {gmem.primary_va(off):#010x} {nm}")
print(f"# pos ({pos[0]:+.1f},{pos[1]:+.1f},{pos[2]:+.1f}) speed {sp:.1f}/s")
# orientation inside the same object
# The rotation is stored with a 16-byte row stride (a 4x4 transform
# whose translation row is the position we already have), so a test for
# nine *contiguous* floats structurally cannot find it. Test both.
lo = max(0, off - 0x800)
blob = os.pread(fd, 0x1000, lo)
found = []
for k in range(0, len(blob) - 48, 4):
for stride, tag in ((12, "3x3"), (16, "4x4")):
try:
rows = [np.array(struct.unpack_from(">3f", blob, k + stride * r))
for r in range(3)]
except struct.error:
continue
M = np.array(rows)
if not np.all(np.isfinite(M)) or np.max(np.abs(M)) > 1.001:
continue
if np.max(np.abs(M @ M.T - np.eye(3))) > 3e-3:
continue
if abs(np.linalg.det(M) - 1.0) > 1e-2:
continue
found.append(((lo + k) - off, M, stride))
break
print(f"# orthonormal 3x3 blocks inside the player object: {len(found)}")
for d, M, st in found[:6]:
print(f" pos{d:+#07x} stride {st}: " + " ".join(
"(" + ",".join(f"{v:+.3f}" for v in row) + ")" for row in M))
if len(sys.argv) > 3 and found:
import json
# Which of the orthonormal blocks is the CRAFT's attitude? The one
# with a row along the direction of travel. Taking found[0] is what
# produced a config with rot_delta -0x764 and a nonsense forward
# axis: several blocks inside the object are orthonormal (bone or
# camera frames), and only the craft's own has a row that tracks
# where the craft is going.
p0 = np.array(pos)
time.sleep(0.35)
p1 = np.array(struct.unpack(">3f", os.pread(fd, 12, off)))
step = p1 - p0
if np.linalg.norm(step) < 1e-3:
sys.exit("craft is not moving — cannot bind the forward axis")
vdir = step / np.linalg.norm(step)
best = None
for d, M, st in found:
cs = [float(M[r] @ vdir) for r in range(3)]
r = int(np.argmax([abs(c) for c in cs]))
if best is None or abs(cs[r]) > abs(best[3]):
best = (d, M, st, cs[r], r)
rot_delta, M, rot_stride, cos, row = best
sign = 1 if cos > 0 else -1
print(f"# attitude block pos{rot_delta:+#07x} stride {rot_stride}: "
f"forward = row {row} (sign {sign:+d}), cos={cos:+.3f}")
if abs(cos) < 0.9:
print("# WARNING: no block tracks the flight path (|cos| < 0.9)"
" — the craft may be drifting hard; re-run while flying straight")
cfg = {"def_delta": delta, "rot_delta": rot_delta,
"rot_stride": rot_stride,
"fwd_row": row, "fwd_sign": sign, "fwd_cos": round(cos, 4),
"va_lo": ENT_VA_LO, "va_hi": ENT_VA_HI}
json.dump(cfg, open(sys.argv[3], "w"), indent=1)
print("# wrote " + sys.argv[3] + ": " + json.dumps(cfg))
return
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,44 @@
#!/usr/bin/env bash
# One background task: boot -> Stage 02 -> fly the pilot while a second reader
# records EVERY entity's hull, so the escort question ("who is losing, and how
# fast") is answered from memory instead of from the HUD.
#
# Same one-task rule as fly_session.sh: the display and the emulator both die on
# their own in this container, so nothing may depend on surviving between tool
# calls.
set -u
export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy DISPLAY=:98
export PYTHONPATH=/sylph-home/.local/lib/python3.12/site-packages
SD="$(cd "$(dirname "$0")" && pwd)"
SECS="${1:-300}"
TAG="${2:-escort}"
SHOTS=/sylph-home/re/shots
CFG=/tmp/nav-live.json
"$SD/launch_mission.sh" fly || { echo "BOOT FAILED"; exit 1; }
python3 "$SD/entities2.py" self 0x130 "$CFG" || { echo "BIND FAILED"; exit 1; }
echo "--- config: $(cat "$CFG")"
echo "=== initial entity table ==="
python3 "$SD/mission_state.py" scan "$CFG"
# The HUD's REMAINING OB counter has no known address yet, so the correlation
# material is a screenshot every 20 s stamped against the same clock as the
# memory samples.
( for i in $(seq 1 20); do
printf '%s SHOT %02d\n' "$(date +%s.%N)" "$i" >> "/tmp/$TAG-shots.log"
screenshot "$SHOTS/$TAG-$i.png" >/dev/null 2>&1
sleep 20
done ) &
SHOTTER=$!
date +%s.%N > "/tmp/$TAG-t0"
python3 "$SD/mission_state.py" watch "$CFG" "$SECS" 1 "/tmp/$TAG-mission.jsonl" \
> "/tmp/$TAG-mission.log" 2>&1 &
WATCHER=$!
python3 "$SD/pilot.py" "$CFG" "$SECS" > "/tmp/$TAG-pilot.log" 2>&1
wait $WATCHER 2>/dev/null
kill $SHOTTER 2>/dev/null
screenshot "$SHOTS/$TAG-end.png" >/dev/null 2>&1
echo "SESSION DONE"

View File

@@ -0,0 +1,120 @@
#!/usr/bin/env python3
"""Find the player craft's live position in guest RAM, from motion alone.
The spawn records at vtable 0x820af030 turned out to hold no live state (every
word is constant across a 29 s capture), so the flying entity is some other
object. Rather than guess which class, this finds it by what it must *do*:
a position triple moves in a straight line at constant speed while coasting.
Method: sample all of RAM K times ~dt apart, keep word indices whose float
value changes every time, then require three consecutive such words to satisfy
* equal step length each interval (constant speed), and
* a constant direction (straight flight),
* with speed in a sane range for a craft.
Only the wholly-mechanical properties of motion are used; no offset, class or
encoding is assumed.
Usage: findplayer.py [samples] [interval_s]
"""
import math
import os
import struct
import sys
import time
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
def snapshot(fd, size):
"""[(start_off, np.ndarray big-endian f32)] over allocated extents."""
out = []
for a, b in gmem.extents(fd, size):
n = (b - a) // 4 * 4
if n < 64:
continue
buf = os.pread(fd, n, a)
out.append((a, np.frombuffer(buf, dtype=">f4")))
return out
def main():
K = int(sys.argv[1]) if len(sys.argv) > 1 else 5
dt = float(sys.argv[2]) if len(sys.argv) > 2 else 0.35
path = gmem.mem_path()
fd = os.open(path, os.O_RDONLY)
size = os.path.getsize(path)
print(f"# sampling {K}x every {dt}s", flush=True)
snaps, times = [], []
for k in range(K):
t = time.time()
snaps.append(snapshot(fd, size))
times.append(t)
print(f"# sample {k} ({len(snaps[-1])} extents, "
f"{sum(len(a) for _, a in snaps[-1])*4/1e6:.0f} MB) "
f"in {time.time()-t:.2f}s", flush=True)
time.sleep(max(0, dt - (time.time() - t)))
# extents must line up across samples for a word-wise comparison
shapes = [tuple((o, len(a)) for o, a in s) for s in snaps]
if len(set(shapes)) != 1:
common = set(shapes[0])
for sh in shapes[1:]:
common &= set(sh)
print(f"# extents shifted between samples; using {len(common)} common ones")
keep = lambda s: [(o, a) for o, a in s if (o, len(a)) in common]
snaps = [keep(s) for s in snaps]
hits = []
for e in range(len(snaps[0])):
base = snaps[0][e][0]
arrs = [np.nan_to_num(s[e][1].astype(np.float64), nan=0, posinf=0, neginf=0)
for s in snaps]
# per-interval deltas
d = [arrs[k + 1] - arrs[k] for k in range(K - 1)]
moving = np.ones(len(arrs[0]), dtype=bool)
for dk in d:
moving &= np.abs(dk) > 1e-3
idx = np.flatnonzero(moving)
if len(idx) == 0:
continue
# candidate triples: i, i+1, i+2 all moving
cand = idx[np.isin(idx + 1, idx) & np.isin(idx + 2, idx)]
for i in cand:
steps = []
dirs = []
ok = True
for k in range(K - 1):
v = np.array([d[k][i], d[k][i + 1], d[k][i + 2]])
n = float(np.linalg.norm(v))
ddt = times[k + 1] - times[k]
sp = n / ddt
if not (20.0 < sp < 3000.0):
ok = False
break
steps.append(sp)
dirs.append(v / n)
if not ok or len(steps) < 3:
continue
if max(steps) - min(steps) > 0.25 * (sum(steps) / len(steps)):
continue
if min(float(np.dot(dirs[0], dd)) for dd in dirs) < 0.985:
continue
va = gmem.primary_va(base + int(i) * 4)
pos = tuple(float(arrs[0][i + j]) for j in range(3))
hits.append((va, base + int(i) * 4, sum(steps) / len(steps), pos))
print(f"\n# {len(hits)} position-like triples (straight, constant speed)")
hits.sort(key=lambda h: -h[2])
for va, off, sp, pos in hits[:40]:
print(f" va {va:#010x} speed {sp:8.1f}/s pos "
f"({pos[0]:+11.1f},{pos[1]:+11.1f},{pos[2]:+11.1f})")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,147 @@
#!/usr/bin/env python3
"""Scan ALL of guest RAM for 3x3 orthonormal float blocks (rotation matrices).
Vectorised with numpy: nine shifted views of each extent give every candidate
9-float window at once, so the whole 250 MB working set tests in seconds.
A rotation matrix is a very strong signature — unit rows, mutually orthogonal —
so the hits are almost entirely real orientations. Two passes taken a moment
apart, with a known stick input in between, then say which of them is *ours*.
Usage: findrot_global.py [--track seconds]
"""
import os
import struct
import sys
import time
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
FIFO = "/tmp/sylph-vgamepad.fifo"
TOL = 2e-3
def pad(line):
with open(FIFO, "w") as f:
f.write(line + "\n")
def scan(fd, size):
"""[(file_offset, 9 floats)] for every orthonormal 3x3 block."""
hits = []
for a, b in gmem.extents(fd, size):
n = (b - a) // 4 * 4
if n < 64:
continue
arr = np.frombuffer(os.pread(fd, n, a), dtype=">f4").astype(np.float32)
if arr.size < 16:
continue
m = arr.size - 8
cols = [arr[i:i + m] for i in range(9)]
with np.errstate(invalid="ignore", over="ignore"):
finite = np.ones(m, dtype=bool)
for c in cols:
finite &= np.isfinite(c) & (np.abs(c) <= 1.001)
# row norms
n0 = cols[0] ** 2 + cols[1] ** 2 + cols[2] ** 2
n1 = cols[3] ** 2 + cols[4] ** 2 + cols[5] ** 2
n2 = cols[6] ** 2 + cols[7] ** 2 + cols[8] ** 2
ok = finite
ok &= np.abs(n0 - 1) < TOL
ok &= np.abs(n1 - 1) < TOL
ok &= np.abs(n2 - 1) < TOL
d01 = cols[0] * cols[3] + cols[1] * cols[4] + cols[2] * cols[5]
d02 = cols[0] * cols[6] + cols[1] * cols[7] + cols[2] * cols[8]
d12 = cols[3] * cols[6] + cols[4] * cols[7] + cols[5] * cols[8]
ok &= np.abs(d01) < TOL
ok &= np.abs(d02) < TOL
ok &= np.abs(d12) < TOL
for i in np.flatnonzero(ok):
hits.append(a + int(i) * 4)
return hits
def read_mat(fd, off):
b = os.pread(fd, 36, off)
if len(b) < 36:
return None
return np.array(struct.unpack(">9f", b))
def main():
fd = os.open(gmem.mem_path(), os.O_RDONLY)
size = os.path.getsize(gmem.mem_path())
pad("reset")
time.sleep(0.6)
t0 = time.time()
hits = scan(fd, size)
print(f"# {len(hits)} orthonormal 3x3 blocks in RAM ({time.time()-t0:.1f}s)", flush=True)
if not hits:
return
# which of them rotate when WE yaw? measure change under left vs right
def deltas(lx, secs=2.5, hz=6.0):
"""Mean per-step rotation vector while the stick is held.
Sampled incrementally: at a few degrees per step the skew part of
A·Bᵀ is the rotation vector, which it is not over a 2 s turn. Any block
that stops being orthonormal mid-phase is memory that got reused, not an
orientation, and is dropped.
"""
pad(f"axis LX {lx}")
time.sleep(0.5)
seq = []
for _ in range(int(secs * hz)):
t = time.time()
seq.append({o: read_mat(fd, o) for o in hits})
time.sleep(max(0, 1.0 / hz - (time.time() - t)))
pad("axis LX 0")
time.sleep(1.0)
def ortho(m):
if m is None or not np.all(np.isfinite(m)):
return False
M = m.reshape(3, 3)
return np.max(np.abs(M @ M.T - np.eye(3))) < 5e-3
out = {}
for o in hits:
ms = [s[o] for s in seq]
if not all(ortho(m) for m in ms):
continue
ws = []
for a, b in zip(ms, ms[1:]):
M = a.reshape(3, 3) @ b.reshape(3, 3).T
w = np.array([M[2, 1] - M[1, 2], M[0, 2] - M[2, 0], M[1, 0] - M[0, 1]]) / 2
if np.linalg.norm(w) < 0.5: # small-angle regime only
ws.append(w)
if len(ws) >= 4:
out[o] = np.mean(ws, axis=0)
return out
dl = deltas(-0.9)
dr = deltas(+0.9)
rows = []
for o in hits:
if o not in dl or o not in dr:
continue
a, b = dl[o], dr[o]
na, nb = np.linalg.norm(a), np.linalg.norm(b)
if na < 0.01 or nb < 0.01:
continue
cos = float(a @ b / (na * nb))
rows.append((cos, o, na, nb))
rows.sort()
print("\n# orientation blocks that rotate OPPOSITE ways for left vs right stick")
for cos, o, na, nb in rows[:12]:
va = gmem.primary_va(o)
print(f" va {va:#010x} cos={cos:+.3f} |wL|={na:.3f} |wR|={nb:.3f}")
if not rows:
print(" (none)")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,146 @@
#!/usr/bin/env python3
"""Identify which moving object in RAM is the *player* craft, by input correlation.
Position triples are easy to find (findplayer.py); saying which one is us is the
hard part, and no static property answers it. This does: hold hard-left yaw for
a few seconds, then hard-right, and look for the object whose turn axis reverses
in phase with the stick. Every other craft is AI-flown and uncorrelated with our
input, so the discriminator is causal rather than circumstantial.
Phase 1 finds candidate triples from two whole-RAM samples; after that only the
candidates' 12 bytes are re-read, so the sampling loop is cheap.
Usage: findself.py [phase_seconds] [hz]
"""
import math
import os
import struct
import sys
import time
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
FIFO = "/tmp/sylph-vgamepad.fifo"
def pad(line):
with open(FIFO, "w") as f:
f.write(line + "\n")
def snapshot(fd, size):
return [(a, np.frombuffer(os.pread(fd, (b - a) // 4 * 4, a), dtype=">f4"))
for a, b in gmem.extents(fd, size) if (b - a) >= 64]
def candidates(fd, size, dt=0.35):
s0 = snapshot(fd, size)
t0 = time.time()
time.sleep(dt)
s1 = snapshot(fd, size)
t1 = time.time()
out = []
for (o, a), (o2, b) in zip(s0, s1):
if o != o2 or len(a) != len(b):
continue
with np.errstate(invalid="ignore"):
af = np.nan_to_num(a.astype(np.float64), nan=0, posinf=0, neginf=0)
bf = np.nan_to_num(b.astype(np.float64), nan=0, posinf=0, neginf=0)
d = bf - af
idx = np.flatnonzero(np.abs(d) > 1e-3)
cand = idx[np.isin(idx + 1, idx) & np.isin(idx + 2, idx)]
for i in cand:
v = np.array([d[i], d[i + 1], d[i + 2]])
sp = float(np.linalg.norm(v)) / (t1 - t0)
if 80.0 < sp < 2000.0:
out.append(o + int(i) * 4)
return out
def sample(fd, offs):
t = time.time()
pos = np.empty((len(offs), 3))
for k, o in enumerate(offs):
b = os.pread(fd, 12, o)
if len(b) < 12:
pos[k] = np.nan
continue
pos[k] = struct.unpack(">3f", b)
return t, pos
def turn_axis(series):
"""Mean normalised cross(v_k, v_k+1) over a phase, per candidate."""
ts = [t for t, _ in series]
ps = [p for _, p in series]
vs = []
for k in range(len(ps) - 1):
dt = ts[k + 1] - ts[k]
vs.append((ps[k + 1] - ps[k]) / max(dt, 1e-3))
ax = np.zeros((ps[0].shape[0], 3))
n = 0
for k in range(len(vs) - 1):
c = np.cross(vs[k], vs[k + 1])
nrm = np.linalg.norm(c, axis=1, keepdims=True)
with np.errstate(invalid="ignore", divide="ignore"):
ax += np.where(nrm > 1e-6, c / nrm, 0.0)
n += 1
return ax / max(n, 1)
def main():
secs = float(sys.argv[1]) if len(sys.argv) > 1 else 3.0
hz = float(sys.argv[2]) if len(sys.argv) > 2 else 6.0
path = gmem.mem_path()
fd = os.open(path, os.O_RDONLY)
size = os.path.getsize(path)
pad("reset")
time.sleep(0.5)
offs = candidates(fd, size)
print(f"# {len(offs)} moving-triple candidates", flush=True)
if not offs:
sys.exit("nothing is moving — in flight?")
phases = {}
for name, lx in (("left", -0.9), ("right", 0.9)):
pad(f"axis LX {lx}")
time.sleep(0.6) # let the turn establish
series = []
n = int(secs * hz)
for _ in range(n):
t0 = time.time()
series.append(sample(fd, offs))
time.sleep(max(0, 1.0 / hz - (time.time() - t0)))
phases[name] = turn_axis(series)
pad("axis LX 0")
time.sleep(1.2)
print(f"# phase {name} done", flush=True)
pad("reset")
aL, aR = phases["left"], phases["right"]
nL = np.linalg.norm(aL, axis=1)
nR = np.linalg.norm(aR, axis=1)
with np.errstate(invalid="ignore", divide="ignore"):
cos = np.sum(aL * aR, axis=1) / (nL * nR)
good = np.isfinite(cos) & (nL > 0.5) & (nR > 0.5)
order = np.argsort(np.where(good, cos, 9e9))
print("\n# objects whose turn axis REVERSES with the stick (most negative first)")
shown = 0
for k in order:
if not good[k]:
break
print(f" va {gmem.primary_va(offs[k]):#010x} cos(axisL,axisR) = {cos[k]:+.3f}"
f" |L|={nL[k]:.2f} |R|={nR[k]:.2f}")
shown += 1
if shown >= 12:
break
if not shown:
print(" (none — try longer phases)")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,115 @@
#!/usr/bin/env python3
"""Locate the player's flight object via its speed scalar, then its position.
The HUD prints the craft's speed, so it is a known quantity we can *change on
demand* — the classic two-state value scan, which is far more reliable than
hunting for a structure by shape:
1. coast -> RAM holds the cruise speed somewhere; collect every float ≈ it;
2. boost -> the real one rises; everything coincidental is filtered out;
3. coast -> it must come back down.
Whatever survives all three is the player's speed. Its object then contains the
position, which is confirmed independently: a position triple near that scalar
must move at exactly the speed the scalar reports.
Usage: findspeed.py <cruise> [boost_wait]
"""
import os
import struct
import sys
import time
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
FIFO = "/tmp/sylph-vgamepad.fifo"
def pad(line):
with open(FIFO, "w") as f:
f.write(line + "\n")
def extents_arrays(fd, size):
for a, b in gmem.extents(fd, size):
n = (b - a) // 4 * 4
if n >= 64:
yield a, np.frombuffer(os.pread(fd, n, a), dtype=">f4")
def scan_equal(fd, size, val, tol):
hits = []
for a, arr in extents_arrays(fd, size):
with np.errstate(invalid="ignore"):
ok = np.isfinite(arr) & (np.abs(arr.astype(np.float64) - val) <= tol)
for i in np.flatnonzero(ok):
hits.append(a + int(i) * 4)
return hits
def read_f(fd, off):
b = os.pread(fd, 4, off)
return struct.unpack(">f", b)[0] if len(b) == 4 else float("nan")
def main():
cruise = float(sys.argv[1])
wait = float(sys.argv[2]) if len(sys.argv) > 2 else 4.0
fd = os.open(gmem.mem_path(), os.O_RDONLY)
size = os.path.getsize(gmem.mem_path())
pad("reset")
time.sleep(2.0)
c0 = scan_equal(fd, size, cruise, 2.0)
print(f"# pass 1 (coast {cruise}): {len(c0)} candidates", flush=True)
pad("trig RT 1.0")
time.sleep(wait)
c1 = [o for o in c0 if read_f(fd, o) > cruise + 40]
print(f"# pass 2 (boost): {len(c1)} rose above {cruise+40:.0f}", flush=True)
vals = {o: read_f(fd, o) for o in c1}
pad("reset")
time.sleep(wait + 2.0)
c2 = [o for o in c1 if abs(read_f(fd, o) - cruise) <= 6.0]
print(f"# pass 3 (coast again): {len(c2)} returned to ≈{cruise}", flush=True)
for o in c2[:20]:
print(f" va {gmem.primary_va(o):#010x} boost value was {vals[o]:.1f}")
if not c2:
print("# no survivors — is the craft actually accelerating?")
return
# position triple in the same object: must move at exactly that speed
print("\n# looking for a position triple near each survivor", flush=True)
RAD = 0x400
for o in c2[:8]:
lo = max(0, o - RAD)
n = RAD * 2
t0 = time.time()
a0 = np.frombuffer(os.pread(fd, n, lo), dtype=">f4").astype(np.float64)
time.sleep(0.4)
t1 = time.time()
a1 = np.frombuffer(os.pread(fd, n, lo), dtype=">f4").astype(np.float64)
sp_now = read_f(fd, o)
dt = t1 - t0
best = []
for i in range(len(a0) - 2):
d = a1[i:i + 3] - a0[i:i + 3]
if not np.all(np.isfinite(d)):
continue
v = float(np.linalg.norm(d)) / dt
if abs(v - sp_now) <= max(8.0, 0.06 * sp_now):
best.append((lo + i * 4, v, tuple(a0[i:i + 3])))
print(f" survivor va {gmem.primary_va(o):#010x} (speed {sp_now:.1f}): "
f"{len(best)} matching triples")
for off, v, p in best[:4]:
print(f" pos va {gmem.primary_va(off):#010x} delta-off "
f"{off - o:+#07x} |v|={v:7.1f} ({p[0]:+9.1f},{p[1]:+9.1f},{p[2]:+9.1f})")
if __name__ == "__main__":
main()

42
tools/re-capture/fire_probe.sh Executable file
View File

@@ -0,0 +1,42 @@
#!/usr/bin/env bash
# Does the pad actually DISCHARGE a weapon? Hold each candidate input in turn
# and photograph the HUD's ammo counters.
#
# Why this exists: the autopilot's HUD kill counters read WARSHIPS 0000 /
# WARPLANES 0000 at the end of every run, in every targeting configuration, so
# "fraction of frames with the guns commanded on" was never measuring anything.
# Before tuning aim any further, settle the prior question — whether the fire
# command reaches the gun at all. The HUD carries a live ammo count (`MAIN MPM
# 00300`, matched to `LoadingCount` by the weapon RE), so the counter falling
# during a hold is direct evidence of a discharge, and the counter sitting still
# through every button is direct evidence that we have never fired a shot.
#
# `RB fires` came from an earlier session; this re-tests it rather than assuming
# it, and sweeps the other buttons so a wrong mapping cannot hide.
set -u
export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy DISPLAY=:98
export PYTHONPATH=/sylph-home/.local/lib/python3.12/site-packages
SD="$(cd "$(dirname "$0")" && pwd)"
SHOTS=/sylph-home/re/shots
HOLD="${1:-4}"
shot(){ screenshot "$SHOTS/fire-$1.png" >/dev/null 2>&1; echo " shot $1"; }
"$SD/launch_mission.sh" fly || { echo "BOOT FAILED"; exit 1; }
sleep 3
echo "=== probing (hold ${HOLD}s each) ==="
shot "00-idle"
# RB first: it is the incumbent claim. Then the rest, so a wrong mapping cannot
# hide behind it. A/B/X/Y may also switch weapon or open something — that is
# fine for a probe, and the final frame records wherever it ended up.
for b in RB LB Y X B A LS RS; do
vgamepad press "$b"; sleep "$HOLD"; shot "hold-$b"; vgamepad release "$b"; sleep 1.5
done
# triggers are analogue, not buttons
vgamepad trig RT 1.0; sleep "$HOLD"; shot "hold-RT"; vgamepad trig RT 0.0; sleep 1.5
vgamepad trig LT 1.0; sleep "$HOLD"; shot "hold-LT"; vgamepad trig LT 0.0; sleep 1.5
vgamepad reset
shot "99-final"
echo "PROBE DONE"

View File

@@ -0,0 +1,157 @@
#!/usr/bin/env python3
"""Derive the player craft's live transform offsets from a flight_probe capture.
Nothing here is guessed from plausible-looking numbers. Two independent
structural facts do the work:
1. a rotation matrix is 9 floats that are orthonormal with det +1 — a
property essentially no other data has;
2. a position triple's frame-to-frame delta must point along the craft's
own forward axis when it is flying straight, and its magnitude must be the
same for every frame at constant speed.
Test 2 validates the position candidate *and* the rotation candidate against
each other, so a coincidence has to satisfy both at once.
Usage: flight_analyze.py <probe.bin> [name-substring]
"""
import math
import os
import struct
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gworld # noqa: E402
def load(path):
f = open(path, "rb")
assert f.read(8) == b"SYLPHPRB"
n_inst, window, n_in = struct.unpack("<III", f.read(12))
insts = []
for _ in range(n_inst):
va, off = struct.unpack("<IQ", f.read(12))
nm = f.read(64).split(b"\0")[0].decode()
insts.append((va, off, nm))
frames = []
rec = 8 + 4 * n_in + 32 + n_inst * window
while True:
b = f.read(rec)
if len(b) < rec:
break
t = struct.unpack_from("<d", b, 0)[0]
inp = struct.unpack_from("<%df" % n_in, b, 8)
label = struct.unpack_from("<32s", b, 8 + 4 * n_in)[0].split(b"\0")[0].decode()
base = 8 + 4 * n_in + 32
bufs = [b[base + i * window: base + (i + 1) * window] for i in range(n_inst)]
frames.append((t, inp, label, bufs))
return insts, window, frames
def f32(buf, o):
return struct.unpack_from(">f", buf, o)[0]
def vec(buf, o):
return (f32(buf, o), f32(buf, o + 4), f32(buf, o + 8))
def sub(a, b):
return (a[0] - b[0], a[1] - b[1], a[2] - b[2])
def norm(a):
return math.sqrt(sum(c * c for c in a))
def dot(a, b):
return sum(x * y for x, y in zip(a, b))
def analyse(insts, window, frames, want):
idx = [i for i, (_, _, nm) in enumerate(insts) if want in nm]
if not idx:
sys.exit(f"no instance matching {want!r}; have: "
+ ", ".join(sorted({nm for _, _, nm in insts})))
i = idx[0]
va, off, nm = insts[i]
print(f"# player instance {va:#010x} {nm} ({len(frames)} frames)")
# ---- rotation blocks that hold up across every frame ------------------
per_frame = []
for t, inp, lab, bufs in frames:
per_frame.append({o for o, kind, m in gworld.find_rotations(bufs[i]) if kind == "3x3"})
stable = set.intersection(*per_frame) if per_frame else set()
print(f"# 3x3 orthonormal blocks valid in ALL frames: "
+ (", ".join(f"{o:#05x}" for o in sorted(stable)) or "none"))
# ---- position candidates ---------------------------------------------
# constant-speed straight flight: |delta| equal every frame, direction fixed
idle = [k for k, (t, inp, lab, b) in enumerate(frames) if lab.startswith("idle")]
if len(idle) < 6:
idle = list(range(min(20, len(frames))))
seg = idle[:20]
cands = []
for o in range(0, window - 12, 4):
ds, ok = [], True
for a, b in zip(seg, seg[1:]):
if b != a + 1:
continue
dt = frames[b][0] - frames[a][0]
p0, p1 = vec(frames[a][3][i], o), vec(frames[b][3][i], o)
if not all(map(math.isfinite, p0 + p1)):
ok = False
break
d = sub(p1, p0)
if dt <= 0:
ok = False
break
ds.append((norm(d) / dt, d))
if not ok or len(ds) < 4:
continue
sp = [s for s, _ in ds]
mean = sum(sp) / len(sp)
if mean < 1.0 or mean > 5000.0: # a craft, not a counter
continue
spread = max(sp) - min(sp)
if spread > 0.15 * mean: # constant speed while coasting
continue
# direction must be steady too
dirs = [(d[0] / (norm(d) or 1), d[1] / (norm(d) or 1), d[2] / (norm(d) or 1))
for _, d in ds]
if min(dot(dirs[0], d) for d in dirs) < 0.99:
continue
cands.append((o, mean, dirs[0]))
print(f"# position candidates (steady speed + steady heading while coasting): "
f"{len(cands)}")
for o, sp, d in cands[:12]:
print(f" +{o:#05x} speed {sp:8.2f}/s dir ({d[0]:+.3f},{d[1]:+.3f},{d[2]:+.3f})")
# ---- cross-validate: velocity must lie along a rotation row -----------
print("\n# cross-check: does the motion direction match a rotation-matrix axis?")
best = []
for o, sp, d in cands:
for ro in sorted(stable):
m = struct.unpack_from(">9f", frames[seg[0]][3][i], ro)
for r, label in ((m[0:3], "row0/right"), (m[3:6], "row1/up"), (m[6:9], "row2/fwd")):
c = abs(dot(d, r))
if c > 0.98:
best.append((c, o, ro, label, sp))
best.sort(reverse=True)
if not best:
print(" none — position and orientation candidates do not corroborate")
for c, o, ro, label, sp in best[:10]:
print(f" pos +{o:#05x} ∥ rot +{ro:#05x} {label} |cos|={c:.5f} speed {sp:.1f}")
return stable, cands
def main():
path = sys.argv[1]
want = sys.argv[2] if len(sys.argv) > 2 else "Player"
insts, window, frames = load(path)
analyse(insts, window, frames, want)
if __name__ == "__main__":
main()

107
tools/re-capture/flight_probe.py Executable file
View File

@@ -0,0 +1,107 @@
#!/usr/bin/env python3
"""Drive a scripted input sequence in flight while sampling guest RAM, so the
player craft's live transform can be *derived* rather than guessed.
The point is correlation: each sample is tagged with the stick/trigger state
that produced it, so afterwards we can ask "which floats move only while the
yaw axis is deflected?" and "which triple integrates to the velocity?".
Writes a .npz-ish flat binary: a header of (va, offset, name) per instance,
then per frame a timestamp, the input vector, and every instance's raw window.
Usage: flight_probe.py <out.bin> [seconds]
"""
import os
import struct
import sys
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gworld # noqa: E402
FIFO = "/tmp/sylph-vgamepad.fifo"
class Pad:
"""Talk to the vgamepad server directly over its FIFO.
The `vgamepad` CLI spawns a process per command (~20 ms); a control loop
cannot afford that, and `tap`/`hold` additionally sleep *inside* the server.
Writing lines to the FIFO ourselves keeps a tick under a millisecond.
"""
def __init__(self):
self.f = open(FIFO, "w", buffering=1)
self.state = {"LX": 0.0, "LY": 0.0, "RX": 0.0, "RY": 0.0, "LT": 0.0, "RT": 0.0}
def axis(self, name, v):
self.state[name] = v
self.f.write(f"axis {name} {v:.3f}\n")
def trig(self, name, v):
self.state[name] = v
self.f.write(f"trig {name} {v:.3f}\n")
def press(self, b):
self.f.write(f"press {b}\n")
def release(self, b):
self.f.write(f"release {b}\n")
def reset(self):
self.f.write("reset\n")
for k in self.state:
self.state[k] = 0.0
def vector(self):
return [self.state[k] for k in ("LX", "LY", "RX", "RY", "LT", "RT")]
# (duration_s, description, action)
SCRIPT = [
(3.0, "idle", lambda p: p.reset()),
(4.0, "pitch-up", lambda p: p.axis("LY", -0.9)),
(2.0, "idle2", lambda p: p.reset()),
(4.0, "yaw-right", lambda p: p.axis("LX", 0.9)),
(2.0, "idle3", lambda p: p.reset()),
(4.0, "yaw-left", lambda p: p.axis("LX", -0.9)),
(2.0, "idle4", lambda p: p.reset()),
(5.0, "boost", lambda p: p.trig("RT", 1.0)),
(3.0, "idle5", lambda p: p.reset()),
]
def main():
out = sys.argv[1]
w = gworld.World()
inst = w.refresh()
print(f"# {len(inst)} instances", flush=True)
if not inst:
sys.exit("no instances — not in a mission?")
pad = Pad()
hz = 10.0
with open(out, "wb") as f:
f.write(b"SYLPHPRB")
f.write(struct.pack("<III", len(inst), gworld.WINDOW, 6))
for va, off, nm in inst:
f.write(struct.pack("<IQ", va, off) + nm.encode()[:63].ljust(64, b"\0"))
t_start = time.time()
for dur, label, act in SCRIPT:
act(pad)
print(f"# {label} ({dur}s)", flush=True)
n = int(dur * hz)
for _ in range(n):
t = time.time()
f.write(struct.pack("<d", t - t_start))
f.write(struct.pack("<6f", *pad.vector()))
f.write(struct.pack("<32s", label.encode()[:32]))
for va, off, nm in inst:
f.write(w.read_off(off, gworld.WINDOW).ljust(gworld.WINDOW, b"\0"))
time.sleep(max(0, 1.0 / hz - (time.time() - t)))
pad.reset()
print(f"# wrote {out} ({os.path.getsize(out)/1e6:.1f} MB)", flush=True)
if __name__ == "__main__":
main()

27
tools/re-capture/fly_session.sh Executable file
View File

@@ -0,0 +1,27 @@
#!/usr/bin/env bash
# One background task: boot -> mission -> bind the transform -> fly the pilot,
# with periodic screenshots so the outcome has visual evidence and not just a log.
#
# It is one task on purpose: the display and the emulator both die on their own
# every few minutes in this container, so nothing may depend on surviving
# between tool calls.
set -u
export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy DISPLAY=:98
export PYTHONPATH=/sylph-home/.local/lib/python3.12/site-packages
SD="$(cd "$(dirname "$0")" && pwd)"
SECS="${1:-240}"
TAG="${2:-pilot}"
SHOTS=/sylph-home/re/shots
CFG=/tmp/nav-live.json
"$SD/launch_mission.sh" fly || { echo "BOOT FAILED"; exit 1; }
python3 "$SD/entities2.py" self 0x130 "$CFG" || { echo "BIND FAILED"; exit 1; }
echo "--- config: $(cat "$CFG")"
python3 "$SD/own_state.py" "$CFG" 3 "/tmp/$TAG-own.json"
( for i in $(seq 1 12); do sleep 25; screenshot "$SHOTS/$TAG-$i.png" >/dev/null 2>&1; done ) &
SHOTTER=$!
python3 "$SD/pilot.py" "$CFG" "$SECS"
kill $SHOTTER 2>/dev/null
screenshot "$SHOTS/$TAG-end.png" >/dev/null 2>&1
echo "SESSION DONE"

168
tools/re-capture/gmem.py Normal file
View File

@@ -0,0 +1,168 @@
#!/usr/bin/env python3
"""Read the *live* guest memory of a running Xenia Canary.
Canary backs the whole guest address space with a single shared-memory file,
`/dev/shm/xenia_memory_<pid-ish>`, so the guest's RAM is readable from the host
with no debugger, no emulator patch and no pause: open the file, seek, read.
The file is one flat image of Xenia's *physical* backing store; guest virtual
addresses map into it through Xenia's fixed table (memory.cc `map_info`).
`va_to_off()` implements that table, so callers work in guest VAs.
Sub-commands
find <pattern> search every allocated extent, print guest VAs
read <va> [len] hexdump guest memory
words <va> [n] dump n big-endian u32 / f32 pairs
`<pattern>` is a python literal-ish string: plain text, or `hex:0011aabb`.
Numbers accept 0x form. The scan uses SEEK_DATA so the ~4.6 GB of sparse holes
cost nothing.
"""
import os
import re
import sys
import struct
# (guest_va_lo, guest_va_hi_inclusive, file_offset_of_lo) — Xenia memory.cc.
MAP = [
(0x00000000, 0x3FFFFFFF, 0x00000000),
(0x40000000, 0x7EFFFFFF, 0x40000000),
(0x7F000000, 0x7F0FFFFF, 0x00000000),
(0x7F100000, 0x7FFFFFFF, 0x00100000),
(0x80000000, 0x8FFFFFFF, 0x80000000),
(0x90000000, 0x9FFFFFFF, 0x80000000),
(0xA0000000, 0xBFFFFFFF, 0x100000000),
(0xC0000000, 0xDFFFFFFF, 0x100000000),
(0xE0000000, 0xFFFFFFFF, 0x100000000),
]
def va_to_off(va):
for lo, hi, base in MAP:
if lo <= va <= hi:
return base + (va - lo)
raise ValueError(f"va {va:#x} outside the guest map")
def off_to_vas(off):
"""All guest VAs that alias this file offset (the map is many-to-one)."""
out = []
for lo, hi, base in MAP:
if base <= off <= base + (hi - lo):
out.append(lo + (off - base))
return out
def primary_va(off):
"""The most useful VA for an offset: physical 0xA0000000+ / xex 0x80000000+."""
vas = off_to_vas(off)
return vas[0] if vas else None
def mem_path():
# A snapshot (`cp --sparse=always /dev/shm/xenia_memory_* snap.bin`, ~2 s)
# reads identically and does not contend with the running emulator, which
# pegs every core under lavapipe. Point $GMEM_FILE at one to work offline.
env = os.environ.get("GMEM_FILE")
if env:
return env
if len(sys.argv) > 1 and sys.argv[1].startswith("/dev/shm/"):
return sys.argv.pop(1)
cands = [f"/dev/shm/{n}" for n in os.listdir("/dev/shm") if n.startswith("xenia_memory_")]
if not cands:
sys.exit("no /dev/shm/xenia_memory_* — is Canary running?")
if len(cands) > 1:
sys.exit(f"several memory files, pass one explicitly: {cands}")
return cands[0]
def extents(fd, size):
"""Yield (start, end) of the file's allocated (non-hole) ranges."""
pos = 0
while pos < size:
try:
data = os.lseek(fd, pos, os.SEEK_DATA)
except OSError:
return
try:
hole = os.lseek(fd, data, os.SEEK_HOLE)
except OSError:
hole = size
yield (data, hole)
pos = hole
def parse_pattern(s):
if s.startswith("hex:"):
return bytes.fromhex(s[4:])
return s.encode("latin-1")
def cmd_find(f, size, args):
pat = parse_pattern(args[0])
limit = int(args[1]) if len(args) > 1 else 64
hits = 0
CHUNK = 1 << 24
for start, end in extents(f.fileno(), size):
pos = start
carry = b""
carry_at = start
while pos < end:
f.seek(pos)
buf = f.read(min(CHUNK, end - pos))
if not buf:
break
blob = carry + buf
base = carry_at
for m in re.finditer(re.escape(pat), blob):
off = base + m.start()
va = primary_va(off)
print(f"{off:#013x} va {va:#010x}" if va is not None else f"{off:#013x} va ?")
hits += 1
if hits >= limit:
return
keep = len(pat) - 1
carry = blob[-keep:] if keep else b""
carry_at = base + len(blob) - len(carry)
pos += len(buf)
if hits == 0:
print("(no hits)", file=sys.stderr)
def cmd_read(f, size, args):
va = int(args[0], 0)
n = int(args[1], 0) if len(args) > 1 else 256
f.seek(va_to_off(va))
data = f.read(n)
for i in range(0, len(data), 16):
row = data[i : i + 16]
hexs = " ".join(f"{b:02x}" for b in row)
txt = "".join(chr(b) if 0x20 <= b < 0x7F else "." for b in row)
print(f"{va + i:08x} {hexs:<47} |{txt}|")
def cmd_words(f, size, args):
va = int(args[0], 0)
n = int(args[1], 0) if len(args) > 1 else 32
f.seek(va_to_off(va))
data = f.read(n * 4)
for i in range(0, len(data) - 3, 4):
(u,) = struct.unpack_from(">I", data, i)
(fl,) = struct.unpack_from(">f", data, i)
fs = f"{fl:.6g}" if -1e30 < fl < 1e30 else ""
print(f"{va + i:08x} +{i:04x} {u:#010x} {u:>12} {fs}")
def main():
path = mem_path()
if len(sys.argv) < 2:
sys.exit(__doc__)
cmd, args = sys.argv[1], sys.argv[2:]
size = os.path.getsize(path)
with open(path, "rb") as f:
{"find": cmd_find, "read": cmd_read, "words": cmd_words}[cmd](f, size, args)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,70 @@
#!/usr/bin/env bash
# Capture one tutorial's unit definitions: cold-boot Canary, walk
# title -> TUTORIAL -> the Nth entry, wait for the stage to load, snapshot RAM.
#
# One emulator per tutorial on purpose: backing out of a loaded mission via
# PAUSE -> BACK TO MENU wedged the emulator (log stops, CPU spins), while a
# cold boot here is ~15 s. The unit definitions are all parsed at stage load,
# so a single snapshot right after the load is everything this needs.
set -u
N="${1:?usage: grab_tutorial.sh <index 0..5> <outfile>}"
OUT="${2:?}"
export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy DISPLAY=:98
SD="$(cd "$(dirname "$0")" && pwd)"
RC="/home/fabi/RE - Project Sylpheed/sylpheed-reborn/tools/re-capture"
# The container entrypoint's Xvfb/openbox do NOT restart on their own, and a
# dead Xvfb leaves a <defunct> entry that still matches `pgrep` -- so test the
# display with xdpyinfo, never by process name.
ensure_display(){
if ! xdpyinfo -display "$DISPLAY" >/dev/null 2>&1; then
echo "[grab] $DISPLAY is down -> restarting Xvfb + openbox"
setsid nohup Xvfb "$DISPLAY" -screen 0 1280x720x24 -ac -nolisten tcp \
+extension GLX +extension RANDR </dev/null >/tmp/xvfb98.log 2>&1 &
for _ in $(seq 1 50); do xdpyinfo -display "$DISPLAY" >/dev/null 2>&1 && break; sleep 0.2; done
setsid nohup env DISPLAY="$DISPLAY" HOME=/sylph-home openbox </dev/null >/tmp/openbox98.log 2>&1 &
sleep 1
fi
xdpyinfo -display "$DISPLAY" >/dev/null 2>&1 || { echo "DISPLAY UNAVAILABLE"; exit 1; }
}
step(){ vgamepad dpad "$1"; sleep 0.2; vgamepad dpad center; sleep 0.6; }
shot(){ screenshot "$SD/$1" >/dev/null 2>&1; }
# Container PID 1 is `sleep infinity`, which never reaps, so every killed
# emulator stays as a <defunct> entry that `pgrep` still matches. Ask ps for the
# state and ignore anything zombie, or this always thinks one is running.
alive(){ ps -o pid=,stat= -C xenia_canary 2>/dev/null | awk '$2 !~ /^Z/ {print $1}'; }
pkill -x xenia_canary 2>/dev/null; sleep 2
[ -n "$(alive)" ] && { kill -9 $(alive) 2>/dev/null; sleep 2; }
rm -f /dev/shm/xenia_memory_* /dev/shm/xenia_code_cache_* 2>/dev/null
ensure_display
cd /sylph-home/re
setsid nohup run-canary --audio --apu=sdl --log_mask=13 \
--logged_profile_slot_0_xuid=E0300000EFBEA3D4 </dev/null >/dev/null 2>&1 &
sleep 5
"$RC/skip_intro.sh" 600 || { echo "BOOT FAILED"; exit 1; }
# The main menu is not input-ready for ~10 s after the title tap under lavapipe;
# d-pad presses before that are silently dropped, and the A then lands on NEW
# GAME instead of TUTORIAL. This wait is the whole fix.
sleep 14
shot "nav-$N-menu.png"
step down; step down # NEW GAME -> TUTORIAL
vgamepad tap A 250; sleep 8
shot "nav-$N-list.png"
i=0; while [ "$i" -lt "$N" ]; do step down; i=$((i+1)); done
shot "nav-$N-pick.png"
vgamepad tap A 250
# the stage load takes ~25-45 s under lavapipe; wait for the screen to settle
sleep 50
shot "nav-$N-loaded.png"
MEM=$(ls /dev/shm/xenia_memory_* 2>/dev/null | head -1)
[ -n "$MEM" ] || { echo "NO SHM"; exit 1; }
cp --sparse=always "$MEM" "$SD/$OUT" || exit 1
echo "SNAPSHOT $OUT ($(du -h "$SD/$OUT" | cut -f1))"

203
tools/re-capture/gworld.py Executable file
View File

@@ -0,0 +1,203 @@
#!/usr/bin/env python3
"""Live world-state reader: the running game's entities, straight out of guest RAM.
Builds on gmem.py (Canary backs the guest address space with
/dev/shm/xenia_memory_*, so it is an ordinary file). The difference here is that
this is meant to run *in a loop* while the game plays, not against a snapshot:
the memory file is opened once and re-read with pread, and the expensive
whole-RAM vtable scan is done rarely and cached.
Two classes matter (see docs/re/structures/unit-struct-runtime.md):
0x820af844 the parsed `.tbl` definition — one per unit type
0x820af030 the spawned entity instance — one per thing in the scene
This module finds instances and reads their live transform. The transform
offsets are NOT guessed: `find_transform` scans an instance for a 3x3 block of
floats that is orthonormal to 1e-3 with det = +1, which a rotation matrix is and
essentially nothing else is.
Sub-commands:
entities list live instances (name, address)
probe <seconds> [hz] [out] record every instance's raw bytes over time
orient report orthonormal 3x3 blocks per instance
"""
import os
import re
import struct
import sys
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
INST_VTABLE = 0x820AF030
DEF_VTABLE = 0x820AF844
WINDOW = 0x600 # bytes of an instance we read; its real size is unknown
NAME_STR_OFF = 0x10
class World:
def __init__(self, path=None):
self.path = path or gmem.mem_path()
self.fd = os.open(self.path, os.O_RDONLY)
self.size = os.path.getsize(self.path)
self.instances = [] # [(va, file_off, name)]
# ---------------------------------------------------------------- io
def read(self, va, n):
return os.pread(self.fd, n, gmem.va_to_off(va))
def read_off(self, off, n):
return os.pread(self.fd, n, off)
def u32(self, va):
return struct.unpack(">I", self.read(va, 4))[0]
def cstr(self, va, n=96):
b = os.pread(self.fd, n, gmem.va_to_off(va)).split(b"\0")[0]
try:
return b.decode("ascii")
except UnicodeDecodeError:
return None
# ------------------------------------------------------------- scan
def scan_vtable(self, vt):
pat = struct.pack(">I", vt)
hits = []
for a, b in gmem.extents(self.fd, self.size):
blob = os.pread(self.fd, b - a, a)
start = 0
while True:
i = blob.find(pat, start)
if i < 0:
break
off = a + i
if off % 4 == 0:
hits.append(off)
start = i + 4
return sorted(set(hits))
def name_of(self, off):
"""The unit ID an object at `off` names, via its name-record pointer."""
raw = self.read_off(off + 4, 4)
if len(raw) < 4:
return None
(p,) = struct.unpack(">I", raw)
try:
return self.cstr(p + NAME_STR_OFF)
except ValueError:
return None
def refresh(self, vt=INST_VTABLE):
"""Re-scan for live instances. Costs ~1 s; call it rarely."""
out = []
for off in self.scan_vtable(vt):
nm = self.name_of(off)
if nm and nm.startswith("UN_"):
out.append((gmem.primary_va(off), off, nm))
self.instances = out
return out
# --------------------------------------------------------------- geometry
def floats(buf, off, n):
return struct.unpack_from(">" + "f" * n, buf, off)
def is_rotation(m, tol=2e-3):
"""m = 9 floats, row-major. Orthonormal rows + det +1?"""
r = [m[0:3], m[3:6], m[6:9]]
for row in r:
n2 = sum(c * c for c in row)
if abs(n2 - 1.0) > tol:
return False
for i in range(3):
for j in range(i + 1, 3):
if abs(sum(r[i][k] * r[j][k] for k in range(3))) > tol:
return False
det = (r[0][0] * (r[1][1] * r[2][2] - r[1][2] * r[2][1])
- r[0][1] * (r[1][0] * r[2][2] - r[1][2] * r[2][0])
+ r[0][2] * (r[1][0] * r[2][1] - r[1][1] * r[2][0]))
return abs(det - 1.0) < 1e-2
def find_rotations(buf, stride=4):
"""Offsets where 9 consecutive floats form a rotation matrix.
Also tries the 4-float-per-row (3x4 / 4x4 matrix) stride, which is how a
transform with a translation column is normally stored.
"""
out = []
for off in range(0, len(buf) - 36, stride):
m = floats(buf, off, 9)
if all(abs(c) <= 1.001 for c in m) and is_rotation(m):
out.append((off, "3x3", m))
for off in range(0, len(buf) - 48, stride):
rows = [floats(buf, off + 16 * i, 3) for i in range(3)]
m = rows[0] + rows[1] + rows[2]
if all(abs(c) <= 1.001 for c in m) and is_rotation(m):
out.append((off, "4x4", m))
return out
# ------------------------------------------------------------------ cmds
def cmd_entities(w, args):
t0 = time.time()
inst = w.refresh()
print(f"# {len(inst)} live instances (scan {time.time()-t0:.2f}s)")
from collections import Counter
c = Counter(n for _, _, n in inst)
for nm, k in c.most_common():
print(f" {k:4d} {nm}")
def cmd_orient(w, args):
inst = w.refresh()
want = args[0] if args else None
for va, off, nm in inst:
if want and want not in nm:
continue
buf = w.read_off(off, WINDOW)
rots = find_rotations(buf)
print(f"\n{va:#010x} {nm} -> {len(rots)} rotation-like block(s)")
for o, kind, m in rots[:6]:
print(f" +{o:#05x} {kind} "
+ " ".join(f"{v:+.3f}" for v in m))
def cmd_probe(w, args):
secs = float(args[0]) if args else 6.0
hz = float(args[1]) if len(args) > 1 else 5.0
out = args[2] if len(args) > 2 else "probe.bin"
inst = w.refresh()
print(f"# probing {len(inst)} instances for {secs}s at {hz}Hz -> {out}")
with open(out, "wb") as f:
f.write(struct.pack("<II", len(inst), WINDOW))
for va, off, nm in inst:
nb = nm.encode()[:63].ljust(64, b"\0")
f.write(struct.pack("<II", va, off) + nb)
n = int(secs * hz)
for i in range(n):
t = time.time()
f.write(struct.pack("<d", t))
for va, off, nm in inst:
f.write(w.read_off(off, WINDOW).ljust(WINDOW, b"\0"))
time.sleep(max(0, 1.0 / hz - (time.time() - t)))
print(f"# wrote {n} frames")
def main():
if len(sys.argv) < 2:
sys.exit(__doc__)
w = World()
cmd, args = sys.argv[1], sys.argv[2:]
{"entities": cmd_entities, "orient": cmd_orient, "probe": cmd_probe}[cmd](w, args)
if __name__ == "__main__":
main()

11
tools/re-capture/hp.sh Executable file
View File

@@ -0,0 +1,11 @@
#!/usr/bin/env bash
# Cycle the Hangar hard-point weapon carousel one step right and capture the
# Name + DATA SHEET panel (the mountable-weapon list for that hard point).
set -u
export HOME=/sylph-home/re
OUT=/sylph-home/re/caps; mkdir -p "$OUT"
[ "${1:-}" = "none" ] || { vgamepad dpad right; sleep 0.20; vgamepad dpad center; }
sleep 2.5
screenshot /tmp/h.png >/dev/null 2>&1
convert /tmp/h.png -crop 495x460+705+95 +repage "$OUT/$2.png"
echo "$OUT/$2.png"

View File

@@ -0,0 +1,75 @@
#!/usr/bin/env bash
# Boot Canary and drive it into the Stage 02 mission from save slot 01, then
# LEAVE IT RUNNING (unlike grab_tutorial.sh, which snapshots and exits) so a
# live control loop can attach to /dev/shm.
#
# Nav is the verified route: title -> LOAD GAME -> slot 01 -> YES -> READY ROOM
# -> TAKE OFF. With `--hangar` it stops in the HANGAR instead so a loadout can
# be picked first.
set -u
MODE="${1:-fly}"
export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy DISPLAY=:98
SD="$(cd "$(dirname "$0")" && pwd)"
SHOTS=/sylph-home/re/shots
alive(){ ps -o pid=,stat= -C xenia_canary 2>/dev/null | awk '$2 !~ /^Z/ {print $1}'; }
# DO NOT `setsid` THE DISPLAY OR THE EMULATOR. They used to be detached into
# their own sessions so they would outlive the shell that started them. What
# that actually bought was the opposite: a process nothing owns is a process
# nothing keeps alive, and both were being reaped a couple of minutes in — the
# long-standing "Xvfb and the emulator die on their own every few minutes" note.
# Run this whole script as ONE tracked background task and leave Xvfb, openbox
# and xenia as its children: they then live exactly as long as the session does.
# `nohup` still shields them from a stray HUP; the exit-status wrapper means a
# death is reported with the server's own account instead of being inferred.
ensure_display(){
if ! xdpyinfo -display "$DISPLAY" >/dev/null 2>&1; then
rm -f "/tmp/.X${DISPLAY#:}-lock" 2>/dev/null || true
nohup bash -c 'Xvfb "$0" -screen 0 1280x720x24 -ac -nolisten tcp \
+extension GLX +extension RANDR >/tmp/xvfb98.log 2>&1
echo "$(date +%T) XVFB EXIT $? (128+N means signal N)" >>/tmp/xvfb-exit.log' \
"$DISPLAY" </dev/null >/dev/null 2>&1 &
for _ in $(seq 1 50); do xdpyinfo -display "$DISPLAY" >/dev/null 2>&1 && break; sleep 0.2; done
nohup env DISPLAY="$DISPLAY" HOME=/sylph-home openbox </dev/null >/tmp/openbox98.log 2>&1 &
sleep 1
fi
xdpyinfo -display "$DISPLAY" >/dev/null 2>&1 || { echo "DISPLAY UNAVAILABLE"; exit 1; }
}
step(){ vgamepad dpad "$1"; sleep 0.25; vgamepad dpad center; sleep 0.7; }
shot(){ screenshot "$SHOTS/$1" >/dev/null 2>&1; }
pkill -x xenia_canary 2>/dev/null; sleep 2
[ -n "$(alive)" ] && { kill -9 $(alive) 2>/dev/null; sleep 2; }
rm -f /dev/shm/xenia_memory_* /dev/shm/xenia_code_cache_* 2>/dev/null
ensure_display
cd /sylph-home/re
nohup run-canary --audio --apu=sdl --log_mask=13 \
--logged_profile_slot_0_xuid=E0300000EFBEA3D4 </dev/null >/dev/null 2>&1 &
sleep 5
"$SD/skip_intro.sh" 600 || { echo "BOOT FAILED (skip_intro exit $?)"; exit 1; }
sleep 14 # main menu is not input-ready before this
step down # NEW GAME -> LOAD GAME
vgamepad tap A 250; sleep 8 # save list, slot 01 preselected
vgamepad tap A 250; sleep 4 # "Load game?" -- cursor starts on NO
step up
vgamepad tap A 250
sleep 28 # -> READY ROOM
shot "lm-readyroom.png"
if [ "$MODE" = "--hangar" ]; then
step down; step down # BRIEFINGS -> HANGAR
vgamepad tap A 250; sleep 12
shot "lm-hangar.png"
echo "STOPPED IN HANGAR"; exit 0
fi
step up # BRIEFINGS -> TAKE OFF
vgamepad tap A 250
# The launch cinematic + stage load + objective card is NOT a fixed 75 s under
# lavapipe; wait for the flight HUD itself.
"$SD/wait_flight.sh" 300 || { echo "NEVER REACHED FLIGHT"; exit 1; }
shot "lm-flight.png"
echo "IN FLIGHT (emulator left running)"

View File

@@ -0,0 +1,105 @@
#!/usr/bin/env python3
"""Type the live entities: link each moving position back to a unit definition.
findplayer.py locates positions by motion alone, but a bare (x,y,z) says nothing
about *what* is moving. Every live entity should reference its parsed `.tbl`
definition (vtable 0x820af844, one object per unit type, addresses known), so
scanning the neighbourhood of each position for a word equal to a definition
address both types the entity and reveals the live object's own layout — the
delta from that pointer to the position triple is the position offset.
Usage: liveents.py [radius_hex]
"""
import os
import struct
import sys
from collections import Counter
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
import gworld # noqa: E402
def definitions(w):
"""{definition_va: unit_id}"""
out = {}
for off in w.scan_vtable(gworld.DEF_VTABLE):
nm = w.name_of(off)
if nm and nm.startswith("UN_"):
va = gmem.primary_va(off)
if va is not None:
out[va] = nm
return out
def main():
radius = int(sys.argv[1], 16) if len(sys.argv) > 1 else 0x600
w = gworld.World()
defs = definitions(w)
print(f"# {len(defs)} unit definitions")
by_word = {struct.pack(">I", va): nm for va, nm in defs.items()}
# sample twice to locate movers, exactly as findplayer.py does
import time
fd, size = w.fd, w.size
def snap():
return [(a, np.frombuffer(os.pread(fd, (b - a) // 4 * 4, a), dtype=">f4"))
for a, b in gmem.extents(fd, size) if (b - a) >= 64]
s0 = snap(); t0 = time.time()
time.sleep(0.4)
s1 = snap(); t1 = time.time()
shapes0 = {(o, len(a)) for o, a in s0}
pairs = [(o, a, b) for (o, a), (o2, b) in zip(s0, s1)
if o == o2 and len(a) == len(b) and (o, len(a)) in shapes0]
movers = []
for base, a, b in pairs:
af = np.nan_to_num(a.astype(np.float64), nan=0, posinf=0, neginf=0)
bf = np.nan_to_num(b.astype(np.float64), nan=0, posinf=0, neginf=0)
d = bf - af
m = np.abs(d) > 1e-3
idx = np.flatnonzero(m)
cand = idx[np.isin(idx + 1, idx) & np.isin(idx + 2, idx)]
for i in cand:
v = np.array([d[i], d[i + 1], d[i + 2]])
sp = float(np.linalg.norm(v)) / (t1 - t0)
if 20.0 < sp < 3000.0:
movers.append((base + int(i) * 4, tuple(float(af[i + j]) for j in range(3)), sp))
print(f"# {len(movers)} moving triples; typing them...")
typed, untyped = [], 0
seen = set()
for off, pos, sp in movers:
lo = max(0, off - radius)
blob = os.pread(fd, radius * 2, lo)
hit = None
for k in range(0, len(blob) - 3, 4):
nm = by_word.get(blob[k:k + 4])
if nm:
hit = (lo + k, nm)
break
if hit:
ptr_off, nm = hit
key = (ptr_off, nm)
if key in seen:
continue
seen.add(key)
typed.append((ptr_off, off - ptr_off, nm, pos, sp))
else:
untyped += 1
print(f"# typed {len(typed)}, untyped {untyped}")
deltas = Counter(d for _, d, _, _, _ in typed)
print(f"# most common (def-pointer -> position) deltas: {deltas.most_common(6)}")
for ptr_off, d, nm, pos, sp in sorted(typed, key=lambda x: x[2])[:40]:
va = gmem.primary_va(ptr_off)
print(f" obj~{va:#010x} defptr+{d:#06x} {nm:<40} "
f"({pos[0]:+10.1f},{pos[1]:+10.1f},{pos[2]:+10.1f}) {sp:7.1f}/s")
if __name__ == "__main__":
main()

34
tools/re-capture/lock_probe.sh Executable file
View File

@@ -0,0 +1,34 @@
#!/usr/bin/env bash
# Which input SELECTS a target, and does a lock then build?
#
# 98 guided missiles produced 2 kills. A guided missile with nothing to guide to
# flies straight, so the suspicion is that the loop has never selected a target
# at all: the HUD carries a `TARGET` marker and a lock reticle, and no button in
# pilot.py has ever touched them. fire_probe.sh already showed LB/X/B/A/LS/RS do
# not discharge a weapon — but "does not fire" says nothing about "does not
# select", so sweep them again watching the RETICLE instead of the ammo.
#
# Captures the centre of the screen (reticle + lock brackets) and the right-hand
# target panel, so a selection or a building lock is visible either way.
set -u
export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy DISPLAY=:98
SD="$(cd "$(dirname "$0")" && pwd)"
SHOTS=/sylph-home/re/shots
HOLD="${1:-3}"
shot(){ screenshot "$SHOTS/lock-$1.png" >/dev/null 2>&1; echo " shot $1"; }
"$SD/launch_mission.sh" fly || { echo "BOOT FAILED"; exit 1; }
sleep 3
shot "00-idle"
# tap, not hold: a select is an edge, and holding a cycle button would just spin
# through every contact. Two taps each, so a cycle that lands on nothing the
# first time still shows on the second.
for b in RS LS LB B X A Y; do
vgamepad tap "$b" 200; sleep 0.4; vgamepad tap "$b" 200; sleep "$HOLD"
shot "tap-$b"
done
vgamepad reset
shot "99-final"
echo "PROBE DONE"

167
tools/re-capture/mission_state.py Executable file
View File

@@ -0,0 +1,167 @@
#!/usr/bin/env python3
"""Mission state from guest RAM: every entity's hull, not just the player's.
`own_state.py` found the player's hull by anchoring on a solved definition
field: an undamaged craft carries its definition's own `HP` (+0x054), so the
live counter is the copy of that number that *falls*. The result was
`hull = position + 0x154`.
The escort question needs the same number for **someone else's** ship. Stage 02
is lost when the ACROPOLIS sinks, not when the player dies, and the 240 s run in
autopilot-memory-driven.md hit GAME OVER with our own hull untouched. So the
claim to test here is that `+0x154` is a property of the *entity class*, not of
the player object: every entity, hostile or friendly, fighter or capital ship,
should hold its own definition's `HP` there at spawn and lose it when hit.
That is falsifiable in one run: read `pos+0x154` and the definition `HP` for
every entity in the scene and compare. If the anchor is class-wide, the ratio
is 1.0 for everything undamaged and nothing else lines up by accident; if it is
player-specific, most entities hold something unrelated.
Sub-commands:
scan [cfg] one table of every entity: hull vs definition HP
watch <cfg> <secs> [hz] [out] sample over time; report what took damage
"""
import json
import math
import os
import struct
import sys
import time
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
import navigator # noqa: E402
HULL_OFF = 0x154 # own_state.py, player craft
DEF_HP = 0x054 # unit-struct-runtime.md, ✅ CONFIRMED
DEF_SIZE_R = 0x50
def f32(fd, off):
b = os.pread(fd, 4, off)
if len(b) < 4:
return float("nan")
(v,) = struct.unpack(">f", b)
return v
class Mission:
def __init__(self, cfg):
self.W = navigator.World(cfg)
self.def_hp = {}
for va in self.W.defs:
self.def_hp[va] = f32(self.W.fd, gmem.va_to_off(va) + DEF_HP)
def snapshot(self):
"""[(off, name, faction, radius, pos, hull, hp_max)] for every entity."""
out = []
for off, va in self.W.ents:
p = self.W.pos(off)
if p is None:
continue
hull = f32(self.W.fd, off + HULL_OFF)
nm = self.W.defs[va]
out.append((off, nm, navigator.faction(nm), self.W.radius[va],
p, hull, self.def_hp.get(va, float("nan"))))
return out
def fmt(e):
off, nm, fac, r, p, hull, hp0 = e
frac = hull / hp0 if hp0 and math.isfinite(hp0) and hp0 > 0 else float("nan")
return (f"{gmem.primary_va(off):#010x} {nm[:34]:<34} {fac:<4} r={r:6.0f} "
f"({p[0]:+8.0f},{p[1]:+8.0f},{p[2]:+8.0f}) hull={hull:9.1f} "
f"HP={hp0:9.1f} frac={frac:6.3f}")
def cmd_scan(cfg):
m = Mission(cfg)
m.W.scan()
ents = m.snapshot()
print(f"# {len(ents)} entities")
ok = sum(1 for e in ents
if math.isfinite(e[6]) and e[6] > 0 and abs(e[5] / e[6] - 1.0) < 1e-3)
fin = sum(1 for e in ents if math.isfinite(e[6]) and e[6] > 0)
print(f"# hull(+{HULL_OFF:#x}) == definition HP for {ok}/{fin} entities "
f"whose definition has an HP")
for e in sorted(ents, key=lambda e: -e[3]):
print(" " + fmt(e))
def cmd_watch(cfg, secs, hz, out):
m = Mission(cfg)
m.W.scan()
base = {e[0]: e for e in m.snapshot()}
print(f"# watching {len(base)} entities for {secs:g}s at {hz:g}Hz", flush=True)
fh = open(out, "w") if out else None
t0 = time.time()
last_scan = t0
last_seen = {}
while time.time() - t0 < secs:
t = time.time()
if t - last_scan > 15.0: # new waves spawn; re-enumerate rarely
m.W.scan()
last_scan = t
for e in m.snapshot():
base.setdefault(e[0], e)
ents = m.snapshot()
rec = {"t": round(t - t0, 2),
"n": len(ents),
"adan": sum(1 for e in ents if e[2] == "ADAN"),
"tcaf": sum(1 for e in ents if e[2] == "TCAF"),
"ents": []}
for e in ents:
off, nm, fac, r, p, hull, hp0 = e
b = base.get(off)
drop = (b[5] - hull) if b and math.isfinite(b[5]) else 0.0
big = r >= navigator.Navigator.BIG_RADIUS
if big or drop > 0.5:
rec["ents"].append({"va": gmem.primary_va(off), "nm": nm,
"fac": fac, "r": round(r, 1),
"hull": round(hull, 1), "hp0": round(hp0, 1),
"drop": round(drop, 1),
"pos": [round(float(c), 1) for c in p]})
if drop > 0.5 and abs(last_seen.get(off, 0.0) - hull) > 0.5:
last_seen[off] = hull
print(f"[{t-t0:6.1f}] HIT {nm[:30]:<30} {fac} "
f"hull {hull:9.1f}/{hp0:9.1f} (-{drop:.0f})", flush=True)
if fh:
fh.write(json.dumps(rec) + "\n")
fh.flush()
time.sleep(max(0.0, 1.0 / hz - (time.time() - t)))
if fh:
fh.close()
print("\n# damage summary")
ents = {e[0]: e for e in m.snapshot()}
for off, b in sorted(base.items(), key=lambda kv: -kv[1][3]):
cur = ents.get(off)
gone = cur is None
hull = cur[5] if cur else float("nan")
if not gone and abs(hull - b[5]) < 0.5 and b[3] < navigator.Navigator.BIG_RADIUS:
continue
print(f" {b[1][:34]:<34} {b[2]:<4} r={b[3]:6.0f} "
f"{b[5]:9.1f} -> {hull:9.1f}" + (" GONE" if gone else ""))
def main():
if len(sys.argv) < 3:
sys.exit(__doc__)
cmd = sys.argv[1]
cfg = json.load(open(sys.argv[2]))
if cmd == "scan":
cmd_scan(cfg)
elif cmd == "watch":
cmd_watch(cfg,
float(sys.argv[3]) if len(sys.argv) > 3 else 120.0,
float(sys.argv[4]) if len(sys.argv) > 4 else 1.0,
sys.argv[5] if len(sys.argv) > 5 else None)
else:
sys.exit(__doc__)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,397 @@
#!/usr/bin/env python3
"""Drift-aware navigation with collision avoidance, driven from guest memory.
Three things the pursuit loop in autopilot3.py did not do:
* **See everything.** It enumerated entities by looking for things that *move*,
so stations, hulls and parked structures were invisible — exactly the objects
you crash into. This scans the entity heap for words that equal a known unit
definition address and takes `position = hit - 0x130`, which finds every
entity whether it is moving or not.
* **Know how big they are.** Each entity's definition carries `Size_Radius`
(+0x50) and `Size_X/Y/Z` (+0x30/34/38) — fields already solved in
docs/re/structures/unit-struct-runtime.md — so the avoidance radius is the
game's own number, not a guess.
* **Account for drift.** The craft does not turn where it points: velocity lags
the nose like an aircraft with sideslip. Steering the *nose* at a target
therefore steers the *flight path* somewhere else, wide and late. This
measures the lag online (the angle between nose and velocity, and how fast
the velocity vector is actually swinging) and commands the nose *ahead* of
where the flight path should go, by that lag.
Avoidance is closest-point-of-approach, not distance: what matters is whether
the two paths will intersect within a horizon, which is why a fast crossing
target is dangerous at 2 km and a station drifting away is not at 300 m.
Usage: navigator.py <config.json> [seconds] [--dry]
"""
import json
import math
import os
import struct
import sys
import time
from collections import Counter
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
import gworld # noqa: E402
import entities2 # noqa: E402
from flight_probe import Pad # noqa: E402
# confirmed fields of the parsed unit definition (unit-struct-runtime.md)
DEF_SIZE_X, DEF_SIZE_Y, DEF_SIZE_Z, DEF_SIZE_R = 0x30, 0x34, 0x38, 0x50
def norm(v):
n = float(np.linalg.norm(v))
return v / n if n > 1e-9 else v * 0.0
def ang(a, b):
return math.acos(max(-1.0, min(1.0, float(np.dot(norm(a), norm(b))))))
class World:
def __init__(self, cfg):
self.cfg = cfg
self.w = gworld.World()
self.fd, self.size = self.w.fd, self.w.size
self.delta = cfg["def_delta"]
self.rot_delta = cfg["rot_delta"]
self.rot_stride = cfg.get("rot_stride", 12)
self.fwd_row = cfg["fwd_row"]
self.fwd_sign = cfg["fwd_sign"]
self.defs = {} # def_va -> name
self.def_word = {} # 4-byte BE -> def_va
self.radius = {} # def_va -> collision radius
self.prev = {} # pos_off -> (t, pos) for velocity
self.vel = {} # pos_off -> smoothed velocity
self.ents = []
self._load_defs()
def _load_defs(self):
for off in self.w.scan_vtable(gworld.DEF_VTABLE):
nm = self.w.name_of(off)
if not (nm and nm.startswith("UN_")):
continue
va = gmem.primary_va(off)
if va is None:
continue
self.defs[va] = nm
self.def_word[struct.pack(">I", va)] = va
b = os.pread(self.fd, 0x60, off)
try:
sx, sy, sz = (struct.unpack_from(">f", b, o)[0]
for o in (DEF_SIZE_X, DEF_SIZE_Y, DEF_SIZE_Z))
sr = struct.unpack_from(">f", b, DEF_SIZE_R)[0]
except struct.error:
sx = sy = sz = sr = 0.0
vals = [v for v in (sr, sx, sy, sz) if math.isfinite(v) and 0 < v < 1e5]
self.radius[va] = max(vals) if vals else 50.0
# ------------------------------------------------------------- entities
def scan(self):
"""Every entity in the heap — moving or not — by its definition pointer."""
lo = gmem.va_to_off(self.cfg["va_lo"])
hi = gmem.va_to_off(self.cfg["va_hi"])
found = []
# numpy, not a per-word python loop: the entity heap is 16 MB, so
# stepping it 4 bytes at a time costs seconds per scan and the control
# loop spends its whole budget scanning instead of flying.
want = np.array(sorted(self.defs.keys()), dtype=np.uint32)
for a, b in gmem.extents(self.fd, self.size):
a, b = max(a, lo), min(b, hi)
n = (b - a) // 4 * 4
if n < 64:
continue
arr = np.frombuffer(os.pread(self.fd, n, a), dtype=">u4").astype(np.uint32)
for k4 in np.flatnonzero(np.isin(arr, want)):
k = int(k4) * 4
va = int(arr[k4])
poff = a + k - self.delta
if poff < 0:
continue
pb = os.pread(self.fd, 12, poff)
if len(pb) < 12:
continue
p = np.array(struct.unpack(">3f", pb))
if not np.all(np.isfinite(p)) or np.max(np.abs(p)) > 1e7:
continue
found.append((poff, va))
# De-duplicate by POSITION: one entity is mirrored at several
# addresses, so keying on the address keeps every copy and the scene
# looks several times more crowded than it is.
seen, uniq = {}, {}
for poff, va in found:
p = self.pos(poff)
if p is None:
continue
key = (va, tuple(np.round(p, 0)))
if key in seen:
continue
seen[key] = poff
uniq[poff] = va
self.ents = list(uniq.items())
return self.ents
def pos(self, off):
b = os.pread(self.fd, 12, off)
if len(b) < 12:
return None
p = np.array(struct.unpack(">3f", b))
return p if np.all(np.isfinite(p)) else None
def rot(self, off):
n = self.rot_stride * 2 + 12
b = os.pread(self.fd, n, off + self.rot_delta)
if len(b) < n:
return None
M = np.array([struct.unpack_from(">3f", b, self.rot_stride * r) for r in range(3)])
if not np.all(np.isfinite(M)) or np.max(np.abs(M @ M.T - np.eye(3))) > 5e-3:
return None
return M
def sample(self, t):
"""[(off, name, pos, vel, radius)] with velocity by finite difference."""
out = []
for off, va in self.ents:
p = self.pos(off)
if p is None:
continue
# A frame the emulator did not advance gives an identical position
# and a bogus zero velocity, which then reads as "stopped" and
# wrecks both the drift estimate and every closing-rate. Keep the
# last good velocity instead, and smooth it.
prev = self.prev.get(off)
v = self.vel.get(off, np.zeros(3))
if prev is not None:
dt = t - prev[0]
moved = float(np.linalg.norm(p - prev[1]))
if dt > 0.02 and moved > 1e-4:
inst = (p - prev[1]) / dt
if float(np.linalg.norm(inst)) < 5000.0:
v = 0.5 * v + 0.5 * inst if np.any(v) else inst
self.prev[off] = (t, p)
else:
self.prev[off] = (t, p)
self.vel[off] = v
out.append((off, self.defs[va], p, v, self.radius[va]))
return out
def faction(nm):
b = nm[3:]
return "ADAN" if b.startswith("be") or b.startswith("e") else "TCAF"
class Navigator:
KP, KD = 2.2, 0.45
FIRE_CONE = math.radians(9)
FIRE_RANGE = 6000.0
HORIZON = 6.0 # s of look-ahead for collision checks
MARGIN_BIG = 220.0 # clearance around hulls and structures
MARGIN_SMALL = 45.0 # ...around fighters, which manoeuvre themselves
BIG_RADIUS = 120.0 # above this an entity counts as a structure
SELF_MIRROR = 25.0 # the same object is mirrored at several addresses
def __init__(self, W, pad, dry=False, log=sys.stdout):
self.W = W
self.pad = pad
self.dry = dry
self.log = log
self.prevM = None
self.firing = False
self.tau = 0.8 # velocity-lag time constant, refined online
# ------------------------------------------------------------ drift
def update_tau(self, fwd, vel, prev_vhat, dt):
"""How long the flight path takes to catch the nose.
The velocity vector swings toward the nose; the angle between them
divided by the rate the velocity is actually swinging is that lag, and
it is what the nose has to be commanded ahead by.
"""
if prev_vhat is None or dt <= 1e-3:
return
vh = norm(vel)
if np.linalg.norm(vh) < 1e-6:
return
swing = ang(prev_vhat, vh) / dt # rad/s the path is turning
lag = ang(fwd, vh) # rad the path is behind
if swing > 0.02 and lag > 0.02:
tau = lag / swing
if 0.05 < tau < 5.0:
self.tau = 0.9 * self.tau + 0.1 * tau
# ------------------------------------------------- collision avoidance
def avoidance(self, me_p, me_v, me_r, ents, me_off):
"""Sum of escape directions, weighted by how soon and how close."""
push = np.zeros(3)
worst = None
for off, nm, p, v, r in ents:
if off == me_off:
continue
rel_p = p - me_p
rel_v = v - me_v
d = float(np.linalg.norm(rel_p))
# The same entity exists at several mirrored addresses, so our own
# copy shows up as an obstacle at zero distance and pins the sticks
# at full deflection forever. Anything this close is us.
if d < self.SELF_MIRROR:
continue
# A wingman flying formation is metres away by design and steers
# itself; giving it a hull-sized margin makes the loop thrash.
big = r >= self.BIG_RADIUS
margin = self.MARGIN_BIG if big else self.MARGIN_SMALL
if not big and faction(nm) == "TCAF":
margin *= 0.5
safe = me_r + r + margin
if d > 1e4:
continue
vv = float(np.dot(rel_v, rel_v))
t_cpa = 0.0 if vv < 1e-6 else -float(np.dot(rel_p, rel_v)) / vv
t_cpa = max(0.0, min(self.HORIZON, t_cpa))
cpa = rel_p + rel_v * t_cpa
miss = float(np.linalg.norm(cpa))
if miss >= safe:
continue
urgency = (1.0 - miss / safe) * (1.0 - t_cpa / self.HORIZON)
if urgency <= 0:
continue
esc = -norm(cpa) if miss > 1e-3 else norm(np.cross(rel_p, me_v))
if np.linalg.norm(esc) < 1e-6:
esc = norm(np.cross(rel_p, np.array([0.0, 1.0, 0.0])))
push += esc * urgency
if worst is None or urgency > worst[0]:
worst = (urgency, nm, d, miss, t_cpa)
return push, worst
# -------------------------------------------------------------- target
def pick(self, me_p, me_v, fwd, ents, me_off):
speed = max(float(np.linalg.norm(me_v)), 1.0)
best, bestscore = None, 1e18
for off, nm, p, v, r in ents:
if off == me_off or faction(nm) != "ADAN":
continue
rel = p - me_p
d = float(np.linalg.norm(rel))
if d < 1e-3:
continue
# lead: where it will be when a shot gets there
lead = p + v * (d / max(speed, 200.0))
rel_l = lead - me_p
theta = ang(rel_l, fwd)
score = d * (1.0 + 3.0 * (theta / math.pi) ** 2)
if score < bestscore:
best, bestscore = (off, nm, lead, rel_l, d), score
return best
# ---------------------------------------------------------------- step
def step(self, t, dt, prev_vhat):
ents = self.W.sample(t)
me = None
for e in ents:
if "Player" in e[1]:
me = e
break
if me is None:
return "no-player", prev_vhat
me_off, me_nm, me_p, me_v, me_r = me
M = self.W.rot(me_off)
if M is None:
return "no-orientation", prev_vhat
fwd = M[self.W.fwd_row] * self.W.fwd_sign
right = M[(self.W.fwd_row + 1) % 3]
up = np.cross(fwd, right)
speed = float(np.linalg.norm(me_v))
vhat = norm(me_v) if speed > 1.0 else fwd
self.update_tau(fwd, me_v, prev_vhat, dt)
# body angular velocity for the damping term
w = np.zeros(3)
if self.prevM is not None and dt > 1e-3:
D = self.prevM @ M.T
w = np.array([D[2, 1] - D[1, 2], D[0, 2] - D[2, 0], D[1, 0] - D[0, 1]]) / (2 * dt)
self.prevM = M
tgt = self.pick(me_p, me_v, fwd, ents, me_off)
goal = norm(tgt[3]) if tgt else fwd
push, worst = self.avoidance(me_p, me_v, me_r, ents, me_off)
pn = float(np.linalg.norm(push))
# avoidance outranks the target when it is urgent
want = norm(goal + push * (3.0 if pn > 0.6 else 1.5)) if pn > 1e-6 else goal
# Command the NOSE ahead of where the flight path must go, by the
# measured lag -- steering the nose straight at the target makes the
# path arrive wide and late.
nose_cmd = norm(want + (want - vhat) * min(self.tau * 1.6, 2.5))
ex = float(np.dot(nose_cmd, right))
ey = float(np.dot(nose_cmd, up))
ez = float(np.dot(nose_cmd, fwd))
yaw = math.atan2(ex, ez if abs(ez) > 1e-3 else 1e-3)
pitch = math.atan2(ey, ez if abs(ez) > 1e-3 else 1e-3)
if ez < 0:
yaw = math.copysign(math.pi / 2, ex if ex else 1.0)
sx = max(-1.0, min(1.0, self.KP * yaw - self.KD * float(np.dot(w, up))))
sy = max(-1.0, min(1.0, -(self.KP * pitch - self.KD * float(np.dot(w, right)))))
# fire only when the *flight path* is clear and the nose is on target
aim_ok = tgt and abs(yaw) < self.FIRE_CONE and abs(pitch) < self.FIRE_CONE
fire = bool(aim_ok and tgt[4] < self.FIRE_RANGE and pn < 1.2)
if not self.dry:
self.pad.axis("LX", sx)
self.pad.axis("LY", sy)
if fire != self.firing:
(self.pad.press if fire else self.pad.release)("RB")
self.firing = fire
drift = math.degrees(ang(fwd, vhat))
msg = (f"spd={speed:6.0f} drift={drift:5.1f}d tau={self.tau:4.2f} "
f"yaw={math.degrees(yaw):+6.1f} pit={math.degrees(pitch):+6.1f} "
f"stick=({sx:+.2f},{sy:+.2f}) fire={int(fire)}")
if tgt:
msg += f" tgt={tgt[1][3:22]:<20} d={tgt[4]:7.0f}"
if worst:
msg += (f" | AVOID {worst[1][3:20]} miss={worst[3]:6.0f} "
f"t={worst[4]:4.1f}s u={worst[0]:.2f}")
return msg, vhat
def run(self, secs, hz=10.0):
self.W.scan()
t0 = time.time()
last, last_scan, prev_vhat = t0, 0.0, None
while time.time() - t0 < secs:
t = time.time()
if t - last_scan > 4.0:
ents = self.W.scan()
last_scan = t
c = Counter(faction(self.W.defs[va]) for _, va in ents)
print(f"[{t-t0:6.1f}] scan: {len(ents)} entities {dict(c)}",
file=self.log, flush=True)
msg, prev_vhat = self.step(t, t - last, prev_vhat)
last = t
print(f"[{t-t0:6.1f}] {msg}", file=self.log, flush=True)
time.sleep(max(0, 1.0 / hz - (time.time() - t)))
if not self.dry:
self.pad.reset()
def main():
cfg = json.load(open(sys.argv[1]))
secs = float(sys.argv[2]) if len(sys.argv) > 2 else 90.0
W = World(cfg)
Navigator(W, Pad(), dry="--dry" in sys.argv).run(secs)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,95 @@
#!/usr/bin/env python3
"""Test the hypothesis that a unit sub-record's runtime fields are laid out in
*schema declaration order*, one 4-byte slot each.
The IDXD string pool lists a sub-record's field names in declaration order,
including the ones this .tbl leaves at their default (they appear as a bare key
with no preceding value). If the runtime struct is `base + 4*index` over that
list, then the offsets the solver found independently must all satisfy it.
That is a much stronger check than any single field's agreement count: one base
offset has to explain dozens of independently-derived anchors at once.
Usage: order_check.py <raw-token-dump> <tbl-hash> <SubRecord> <solved.txt>
"""
import re
import sys
NUMERIC = re.compile(r"^[-+]?[0-9]*\.?[0-9]+[fF]?$|^0x[0-9a-fA-F]+$")
VALUE_WORDS = {"Yes", "No", "YES", "NO", "On", "Off", "ON", "OFF"}
def section(dump, tbl, sub, subs):
"""Ordered token list of one sub-record of one .tbl."""
toks, on = [], False
for line in open(dump):
if line.startswith("RAW "):
on = line.split()[1] == tbl
continue
if on and line.startswith("T "):
toks.append(line[2:].rstrip("\n"))
starts = [i for i, t in enumerate(toks) if t in subs or t.lstrip("(") in subs]
for n, i in enumerate(starts):
name = toks[i].lstrip("(")
if name != sub:
continue
end = starts[n + 1] if n + 1 < len(starts) else len(toks)
return toks[i + 1 : end]
return []
def keys_in_order(toks):
"""Field names, in declaration order: every token that is not a value.
In the numeric sub-records every value is a number literal, so a token that
is not numeric and not a Yes/No word is a key.
"""
out = []
for t in toks:
if NUMERIC.match(t) or t in VALUE_WORDS:
continue
if t not in out:
out.append(t)
return out
def solved_offsets(path):
out = {}
for line in open(path):
m = re.match(r"\s*(0x[0-9a-f]+)\s+(\w+)\s+(\w+)\s+(\d+)\s+(\d+)\s+(\d+)\s*$", line)
if m:
off, enc, field, agree, dist, bad = m.groups()
out[field] = (int(off, 16), enc, int(agree), int(dist), int(bad))
return out
def main():
dump, tbl, sub, solvedf = sys.argv[1:5]
subs = {"Generic", "Maneuver", "Shield", "Explosion", "Mass", "Effect", "SE"}
keys = keys_in_order(section(dump, tbl, sub, subs))
solved = solved_offsets(solvedf)
anchors = [(i, k, solved[k]) for i, k in enumerate(keys) if k in solved]
if not anchors:
sys.exit(f"no solved field lands in {sub}")
# base that the most anchors agree on
votes = {}
for i, k, (off, *_) in anchors:
votes.setdefault(off - 4 * i, []).append(k)
base, winners = max(votes.items(), key=lambda kv: len(kv[1]))
print(f"# {sub}: {len(keys)} declared fields, {len(anchors)} solved anchors")
print(f"# best base = {base:#x} -> {len(winners)}/{len(anchors)} anchors fit "
f"offset = base + 4*index\n")
for i, k in enumerate(keys):
off = base + 4 * i
if k in solved:
so, enc, agree, dist, bad = solved[k]
mark = "fits" if so == off else f"CONFLICT solver={so:#x}"
print(f" +{off:#05x} [{i:3d}] {k:<34} {enc} agree={agree:<3} "
f"dist={dist:<3} {mark}")
else:
print(f" +{off:#05x} [{i:3d}] {k:<34} — (never valued on disc)")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,117 @@
#!/usr/bin/env python3
"""Where does the craft keep its hull and shield? Anchor on the definition.
The parsed unit definition already has solved fields (unit-struct-runtime.md):
`HP` at `+0x054`, shield `MaxValue` at `+0x238`, shield `ChargeSpeed` at
`+0x244`. A craft that has taken no damage is at full hull and full shield, so
its live entity object must *contain those very numbers*. That turns "find the
HP field" from a value-scan over 4 GB into: read two floats from the definition,
then look for them inside the entity object.
Then watch the candidates while the craft is under fire. The live field is the
one that falls; a copy of the definition value that never moves is not it.
Usage: own_state.py <config.json> [watch_seconds] [out.json]
"""
import json
import os
import struct
import sys
import time
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
import navigator # noqa: E402
DEF_HP, DEF_SHIELD_MAX, DEF_SHIELD_CHG = 0x054, 0x238, 0x244
BACK, FWD = 0x800, 0x800
def near(a, v):
return np.abs(a - v) <= max(1e-3, abs(v) * 1e-4)
def main():
cfg = json.load(open(sys.argv[1]))
secs = float(sys.argv[2]) if len(sys.argv) > 2 else 40.0
out = sys.argv[3] if len(sys.argv) > 3 else None
W = navigator.World(cfg)
ents = W.scan()
me = [(off, va) for off, va in ents if "Player" in W.defs[va]]
if not me:
sys.exit("player entity not found — not in flight?")
me_off, def_va = me[0]
print(f"# player {W.defs[def_va]} pos off {me_off:#x} def {def_va:#010x}")
d = os.pread(W.fd, 0x300, gmem.va_to_off(def_va))
want = {}
for nm, o in (("HP", DEF_HP), ("Shield_MaxValue", DEF_SHIELD_MAX),
("Shield_ChargeSpeed", DEF_SHIELD_CHG)):
(v,) = struct.unpack_from(">f", d, o)
want[nm] = v
print(f"# definition {nm:<18} = {v:g}")
blob = os.pread(W.fd, BACK + FWD, me_off - BACK)
arr = np.frombuffer(blob, dtype=">f4").astype(np.float64)
cands = [] # (label, delta_from_position)
for nm, v in want.items():
if not np.isfinite(v) or v == 0.0:
continue
for i in np.flatnonzero(near(arr, v)):
cands.append((nm, int(i) * 4 - BACK))
print(f"# {len(cands)} candidate offsets inside the entity object:")
for nm, dlt in cands:
print(f" pos{dlt:+#07x} == definition {nm}")
if not cands:
print("# none — the object does not carry the definition's own numbers"
" at this offset window; widen BACK/FWD or the craft is damaged")
# ---- watch them; the live field is the one that moves
hist = {c: [] for c in cands}
t0 = time.time()
last_print = 0.0
while time.time() - t0 < secs:
p = W.pos(me_off)
if p is None:
print("# player object gone (death / stage change?)")
break
w = os.pread(W.fd, BACK + FWD, me_off - BACK)
a = np.frombuffer(w, dtype=">f4").astype(np.float64)
for c in cands:
i = (c[1] + BACK) // 4
hist[c].append(float(a[i]))
t = time.time() - t0
if t - last_print > 4.0:
last_print = t
cur = " ".join(f"{c[0][:4]}{c[1]:+#x}={hist[c][-1]:.1f}" for c in cands[:6])
print(f"[{t:6.1f}] {cur}", flush=True)
time.sleep(0.2)
print("\n# offset anchor first last min moved")
moving = []
for c in cands:
h = np.array(hist[c])
if not len(h):
continue
mv = float(h.max() - h.min())
print(f" pos{c[1]:+#07x} {c[0]:<18} {h[0]:8.1f} {h[-1]:8.1f} "
f"{h.min():8.1f} {mv:7.3f}")
if mv > 1e-3:
moving.append({"anchor": c[0], "delta": c[1],
"first": h[0], "last": float(h[-1]),
"min": float(h.min())})
if not moving:
print("# nothing moved — the craft took no damage during the window")
if out:
json.dump({"player": W.defs[def_va], "def_va": def_va,
"definition": want,
"candidates": [{"anchor": a, "delta": b} for a, b in cands],
"moved": moving}, open(out, "w"), indent=1)
print(f"# wrote {out}")
if __name__ == "__main__":
main()

Some files were not shown because too many files have changed in this diff Show More