47 Commits

Author SHA1 Message Date
claude-re
8ee3bfbcef re-capture: stop detaching the display and the emulator
ensure_display() and the run-canary launch used 'setsid nohup', to make them
outlive the shell that started them. That is what has been killing every long
run: a setsid'd process belongs to no supervised tree, and both Xvfb and xenia
were reaped a couple of minutes in — the 'they die on their own every few
minutes' note in the project memory. Measured: a bare Xvfb with no emulator
running exited 0 (a clean shutdown, not a crash and not the OOM killer) at the
exact moment a turn ended, and a session whose display was setsid'd from inside
a tracked task died the same way 12 s after launch.

Run launch_mission.sh as one tracked background task and keep Xvfb, openbox and
xenia as its children. Adds an exit-status wrapper so a future death reports the
server's own exit code (128+N for signal N) instead of being inferred, and
clears a stale X lock before starting.
2026-07-30 17:12:00 +00:00
claude-re
91fb022caf re-capture: a dead display must not look like a playing movie
skip_intro.sh compared two screenshots to detect the intro movie. When the X
server died 3 min into a run, `screenshot` failed silently and left both PNGs
at their previous contents — two stale files, whose RMSE is a constant non-zero
number, i.e. exactly the signature of a changing screen. The loop then reported
"movie -> skip A" every 5 s for the whole 600 s timeout with no emulator and no
display alive, and the session wasted 10 minutes before saying BOOT FAILED.

Both waiters now verify, every iteration, that the screenshot was actually
written, that the display answers xdpyinfo, and that a non-zombie xenia_canary
exists — with distinct exit codes (3 display, 4 emulator, 5 capture) so the
session log names the cause instead of timing out.

Also adds mission_state.py + escort_session.sh: read pos+0x154 against each
definition's HP for EVERY entity, to test whether the hull anchor is a property
of the entity class rather than of the player object (the escort question).
2026-07-30 17:06:08 +00:00
claude-re
93d6534efe docs/re: open a backlog, first item = capital ships assemble wrong in the viewer
The runtime-capture write-up declares static ship assembly exact, but its test
covers exactly one ship (e106). Records what is suspect (engine-cluster rig,
the X-reflect twin heuristic, cross-id turrets), and that the F10 capture is
already the oracle to settle it on a second class.
2026-07-30 17:01:32 +00:00
41c34dc3ca re: surviving is not winning -- Stage 02 is an escort
The second 240 s flight ended on GAME OVER with our hull untouched at
1500/1500. Nothing shot us down; the ACROPOLIS was sunk while the
pilot pursued an attacker two kilometres away. "Nearest hostile
fighter" is the wrong objective function for this stage.

The fix is available with what is already solved: the protected ship's
hull is readable with the same anchor as ours (position+0x154, maximum
= its own definition HP), so target priority can be "closing on the
asset" and the escort can be scored live.

Also records that a frozen log tail is what mission-end looks like
from outside the emulator, not a wedge.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 05:43:42 +00:00
2e9903d0fc re: the autopilot now survives, and kills
Three measurements, then a pilot built on them.

* Hull is position+0x154. Found by anchoring on a field the definition
  already had solved (HP = 1500) rather than scanning for a value that
  falls: an undamaged craft must contain its own definition's number.
  Confirmed by the trace across a death -- 30/60/90 per hit, negative
  at 0, GAME OVER on screen.
* RT accelerates, LT brakes, and the throttle is a persistent setting
  (488 -> 1510 -> 174 units/s, measured as displacement per second of
  the craft's own position, so no speed field was needed). This
  overturns the earlier "RT is not the throttle", which came from
  assuming the control and hunting for a field.
* Shield is probably position+0x430 (== definition MaxValue 400), not
  yet confirmed live -- nothing had damaged it.

pilot.py is a state machine on damage (ENGAGE / EVADE / RETIRE) that
treats turrets as keep-out zones instead of targets. It flew Stage 02
for 300 s with the hull untouched at 1500/1500 and took the first
confirmed kill (WARPLANES 0001); every run the day before was dead
inside 35 s.

Two bugs the live run exposed and this fixes: gating the guns on the
commanded direction keeps them cold whenever avoidance is steering
(gate on the target instead), and an orbit-plus-brake rule made it
circle one attacker for 40 s outside its own firing cone.

Also: boot to in-flight is now ~100 s unattended, because
wait_flight.sh waits for the HUD's own shield bar instead of a fixed
75 s sleep that lavapipe does not honour; entities2.py picks the
attitude block by matching the measured flight path (taking the first
orthonormal block gave a bone/camera frame); and the entity-heap scan
is numpy instead of a per-word Python loop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 05:38:00 +00:00
4623387f6c re: the memory-driven autopilot now flies, tracks and shoots
Three findings turned the previous dead end into a working loop, each checked
against something independent rather than assumed:

  * a live entity's definition pointer sits at position + 0x130, so one heap
    scan types every craft in the scene -- which is what separates ~20 real
    combatants from ~30000 moving particles. The result is coherent: wingmen,
    enemy turrets and attackers, friendly capital ships, one Player.
  * orientation is a 3x3 at position - 0x70 stored with a 16-BYTE ROW STRIDE
    (a 4x4 whose translation row is the position). The earlier search for nine
    contiguous floats could not find this by construction, which is why the
    first pass wrongly concluded there was no transform. Confirmed by its row 2
    matching measured direction of travel at cos = +1.000.
  * RB is the fire button, established by consequence: of RB/LB/A/B/X/Y/RT/LT
    it is the only one that makes the nose-ammo counter in RAM fall.

Control is PD on the aiming error with the derivative from body angular
velocity, and target selection weighted by off-boresight angle -- pure
nearest-first kept picking targets 90 deg off the nose, whose bearing rate then
outran the turn rate and held the craft outside its firing cone at a steady 27
deg pitch error.

Observed: distance to target closing monotonically, yaw error driven from -8 deg
to ~0, fire=1 once inside the cone, ammo counter falling.

Not solved: survival. There is no evasion, no shield/armour awareness and no
throttle control, so it flies a straight pursuit into defended space and is
shot down; every long run has ended in GAME OVER. Mission completion needs
those, plus objective-aware target priority.
2026-07-29 19:32:37 +00:00
72365b217a re: memory-driven autopilot -- infrastructure, and an honest account of what is not solved
Reads the live world out of guest RAM and drives the pad from it. Working:
loop-rate memory reads, whole-RAM float scanning with numpy (1270 orthonormal
3x3 blocks in 6.2 s), entity enumeration by unit type (116 live instances in
Stage 02), pad control written straight into the vgamepad FIFO (the CLI spawns
a process per command and its tap/hold sleep inside the server, so neither is
usable in a control loop), unattended mission entry, and the Hangar loadout --
the "Recommended" control is AUTO SELECT, which at 5 % progress is a no-op
because only two weapons are developed and both are already mounted.

Not working, and the reason the craft is not yet flown: the class 0x820af030
is NOT the live entity. It has one object per spawned thing and carries the
unit-ID string, which is why it looked like the entity list, but every one of
its 384 words is constant across a 29 s in-flight capture. No transform lives
in it or one pointer hop from it. Input correlation (hard left yaw vs hard
right, looking for a turn axis that reverses) does find self-like objects at
cos = -0.99, but they cluster in what looks like a camera volume rather than
the craft, and with no definition pointer near them the trick of learning one
entity's layout and applying it to the rest has nothing to anchor on -- so the
33418 moving triples in a firefight cannot be split into enemies, friendlies
and bullets, and there is nothing to aim at.

Two dead ends are recorded so they are not repeated: RT is not the throttle
(the two-state speed scan therefore found nothing), and comparing orientation
matrices 2 s apart is outside the small-angle regime, which is what produced
"angular velocities" of 30000.

Also corrects the claim in unit-struct-runtime.md that 0x820af030 holds live
state. The definition class 0x820af844 and every value derived from it are
unaffected.

autopilot2.py (a PD controller using body angular velocity from consecutive
rotation matrices) is committed but has never had a valid config to run
against, and is marked as untested.
2026-07-29 19:04:33 +00:00
22d2518847 re: don't let zombie emulators fool the liveness check
Container PID 1 is `sleep infinity` and never reaps, so every emulator this
script kills stays as a <defunct> entry that `pgrep -x` still matches -- after
a few captures the script always believed one was running. Same trap as the
Xvfb check. Ask ps for the process state and skip anything in Z.
2026-07-29 18:24:18 +00:00
1ee2d5e406 re: widen unit coverage to all six tutorials, and prove the fields are run-invariant
Captures the five remaining tutorials (grab_tutorial.sh, one cold boot each)
and re-solves over seven snapshots from seven separate emulator runs. Coverage
18 -> 21 units, confirmed fields 22 -> 27 (Size_Y, MassScore, MinimumVelocity,
AV_PitchPlus_Min, AV_PitchMinus_Min join the zero-contradiction set).

The multi-run union exposed something the single-run check could not: the same
unit's object is NOT byte-identical between runs. unit_runtime.py --crosscheck
pins down why -- exactly 15 words differ, 13 of them holding guest heap
pointers, and none of them is a solved or interpolated field offset. So every
value reported is run-invariant, which is a stronger statement than the
within-run identity check that came before it.

The two non-pointer stragglers are a real caveat, now documented rather than
smoothed over: +0x2c8 reads 8000.0 for UN_f001_TCAF_DeltaSaber_T_Ttrl in two
tutorials and 10000.0 in a third, with a 0/1 flag at +0x2d0. A couple of words
in the object are set per stage, so it is mostly but not entirely the parsed
table. Unidentified, marked NEEDS-HUMAN.

Coverage honesty: all six tutorials together add only 3 units the missions do
not already have -- they reuse one training box, one drone and the player
craft. Further coverage needs story progress, not more tutorials.

Two navigation facts encoded in grab_tutorial.sh, both learned by breaking
them: the main menu is not input-ready for ~10 s after the title tap and early
d-pad presses are dropped (which sends the A to NEW GAME); and NEW GAME is not
a shortcut to Stage 01 -- it gates on DIFFICULTY then plays the prologue. A
NEW GAME excursion to the READY ROOM leaves game01/savedata byte-identical,
verified by diff against a backup.
2026-07-29 18:22:47 +00:00
2196ac112d re: unit definitions are parsed at stage load, not on wave spawn
Counted the definition objects at three points in Stage 02 -- just after
take-off, ~12 min into combat, and after GAME OVER: 14 objects, same 14 IDs
each time. Capturing a stage therefore costs one load and one snapshot; the
mission does not have to be played or survived.
2026-07-29 17:31:45 +00:00
2fa4c33d1a re: solve the runtime Unit (craft/vessel) struct from live guest memory
Craft stats were the last parked Route-B target: the Hangar exposes only a
weight class, and in menus the flight-model object is not instantiated at all.
It is instantiated in-mission, so this reads it there.

The definition class is discovered rather than assumed (unit_discover.py):
scanning for pointers to the disc ID strings and tallying the word behind each
pointer site picks out vtable 0x820af844 -- one object per unit ID, name-record
pointer at +0x04, string at +0x10, exactly the Weapon shape. The neighbouring
class 0x820af030 is the spawned entity, not the definition; the two are told
apart by one-object-per-ID and by 14/14 objects being byte-identical across two
snapshots 12 minutes and one firefight apart.

22 fields bind with zero contradictions on >=3 distinct values. Beyond that,
the Maneuver sub-record turns out to be laid out in schema declaration order,
4 bytes per field, base 0x9c with a two-slot gap after AA_Roll_Min -- 29
independently solved anchors fit two exact bases with no conflicts. That rule
pins five fields NO unit table ever values (PitchDragFactor, RollDragFactor,
DragFactorThreshold, ArterBurner_Acc, DecPitchFactor); PitchDrag == RollDrag ==
the disc's YawDrag on 18/18 units corroborates the interpolation.

Angles are float32 radians at runtime and degrees on disc.

Coverage is 18 of 110 units: unlike weapons, unit definitions are instantiated
per stage, so it grows by visiting missions. The AI-behaviour tail of Maneuver
is explicitly NOT resolved and marked tentative in the CSV.

Captured from Xenia Canary (BASIC CONTROLS tutorial + Stage 02 from save 01).
2026-07-29 17:27:53 +00:00
6a41525c41 re: generalize the two capture tools beyond weapons
idxd_tokens took the sub-record type list as an argument already but still
hard-filtered entries to those declaring a `Weapon_*` id, so it could not be
pointed at another schema. Filter on the requested type names instead -- it now
splits `EnumUnit` (Generic/Maneuver/Shield/Explosion/Frame/Turret/...) too.

gmem: honour $GMEM_FILE. `cp --sparse=always /dev/shm/xenia_memory_* snap.bin`
takes ~2 s and reads identically, which matters because a running Canary pegs
every core under lavapipe and makes repeated live reads stall unpredictably.

Groundwork for the craft (UNIT) stats. Not a finding yet: in menus only the
player craft's *name* is resident -- the flight-model object is not instantiated
until a mission loads, so that needs an in-mission snapshot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 06:05:13 +00:00
9daac6f1f0 re: read the defaulted weapon stats out of live guest memory
Canary backs the whole guest address space with one shared-memory file, so the
game's RAM is readable from the host while it runs -- no debugger, no emulator
patch. That turns the parked Route-B question (fields the IDXD omits because
they sit at their default) from "unreachable" into a table lookup.

gmem.py maps guest VAs into /dev/shm/xenia_memory_* via Xenia's fixed table and
searches only the allocated extents, so a full-RAM scan is ~0.2 s.

weapon_runtime.py finds the parsed objects by scanning for their vtable, then
*solves* the struct layout instead of guessing it: every (field, offset,
encoding) triple is scored against the disc records, and a binding is accepted
only with zero contradictions -- and marked confirmed only when >=10 records
agree on >=3 distinct values, because a field whose samples are all one number
matches any offset holding that constant.

Result: Weapon (0xc0) and Shell (0x200) mapped, 4393 values the disc does not
carry, for all 126 weapons at 5 % save progress. Cross-checks hold -- the
Arsenal DATA SHEET read 4 for wep_05/wep_60 Max. Lock Ons and the memory read
agrees; the in-flight HUD's 06000/00300 match LoadingCount; all 252 objects are
byte-identical across a screen change, so this is definition data, not state.

Two findings fell out: `IsCharging = Yes` switches LoadingCount and
TriggerShotCount to float32 (it partitions the 8 float-encoded records exactly),
and two .tbl entries declare Shell_TCAF_Ship_AAGun with conflicting Power -- the
runtime keeps the one that omits it.

Where the defaulted values *come from* (the IDXD's undecoded binary node region
vs code defaults) is not settled here; they vary per weapon, so they are not one
constructor constant.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 05:29:34 +00:00
e3f3d55d29 re: ui layout generalizes — the title main menu rebuilds too
Second screen, independent of the pause work: GP_TITLE.pak's ptbtn01..05.rat give
X=542 for all five items and Y=162/242/322/402/482, an 80px pitch where the pause
menu used 70. Measured against the main-menu screenshot the sprite tops sit at a
constant +46px for all five, which is exactly the 45px of Xenia window chrome plus
one — so the record's Y is the sprite's top edge in the guest framebuffer, to the
pixel, on a screen the format was not derived from.

GP_TITLE also splits by sub-screen the way the pause pak splits by context:
ptbtn00 (PRESS A BUTTON), ptbtn01..05 (main menu), ptbtn11..13 (EXTRAS submenu),
pgloading_* (loading screen).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 19:44:29 +00:00
9a86a09f8d re: ui — the RATC bundle header is the screen's element declaration table
u32 count at 0x14, then fixed 60-byte entries of [name | 4 flag words | pivotX |
pivotY | 0]. The table lists both .t32 sprites and .rat records, so it is the
screen's element list.

Verified: for all 7 .t32 entries in the tutorial pause bundle the declared pivot
is exactly half the decoded texture's dimensions, 7/7 with no mismatch. That also
settles what the pair at 0x50/0x54 of a .rat record is — a .rat simply opens with
one of these declarations.

The language split is visible inside one bundle: the English build's .t32
declarations carry correct English pivots while its .rat declarations carry
Japanese-derived ones, so the sprite table is regenerated per language and the
layout layer is inherited from the Japanese master.

Adds tools/re-capture/ratc_decls.py (parse the table, check pivots against the
decoded textures).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 19:40:57 +00:00
aaa4ea60b3 re: ui .rat — pivot field, language-independence, PRMD, and a tightened claim
Follow-up decoding on the layout record:

- 0x50/0x54 is the sprite PIVOT: exactly half the texture size on 4 of 5 plain
  sprites. It is a rotation/scale centre, not a draw offset — X/Y remains the
  top-left, which is what actually reproduces the screenshot.
- The records are byte-identical across the English and Japanese bundles: layout
  is authored once and only the sprites are swapped. That explains pgpbtn01's
  pivot implying a 226px texture that neither shipped language has.
- loop1.rat is NOT the screen composition — it is a looping sprite animation
  (3-name table + ~30 keyframes at one position). The eff*/deli*/msg draw list is
  therefore still unfound, and that gap is now stated plainly.
- The 380-byte top-level entries are PRMD primitives: a colour plus four explicit
  corner coordinates for a full-screen quad — the pause dim overlay. The format
  is tag-driven ('opt ', 'PRMD', 'end '), not a fixed struct.

Also demotes one piece of evidence. The screenshot is of the TUTORIAL pause menu,
so only pgp_ttrl_* can be checked against it; the in-mission records mapping onto
the same screenshot under a constant offset works only because both builds share
the 70px pitch and proves nothing. In-mission coordinates are now marked
unverified.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 19:38:04 +00:00
621b9dd6e9 re: crack the UI layout record — the retail UI rebuilds from the disc
Each UI sprite <name>.t32 ships with a <name>.rat companion, and that companion
is the layout record: big-endian u32, a 1280x720 design space at 0x18, and a
placement block of scaleX/scaleY/tint/X/Y. Animated elements repeat the block
with a trailing time field (keyframes); an 'opt ' tag carries a length-prefixed
link to the focused-state record.

Verified in that order, records first: across the four pause buttons exactly one
field varies, stepping 268/338/408/478 at a constant 70px pitch with X fixed at
226; the three items visible in a screenshot of the running game measure
346.0/415.5/485.5, mapping onto those numbers under one constant offset to
within 0.5px. The tutorial build's records are outright framebuffer coordinates,
and compositing its sprites there reproduces the screenshot pixel-accurately.

Also records two mistakes the A/B caught, since a static-only reading would have
shipped both: texture width does not identify a language (Japanese fits English
widths), and the in-mission and tutorial pause menus are separate sprite sets
rather than one set re-packed into slots.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 19:05:21 +00:00
07eff85819 re: in-flight HUD — independent confirmation of the ammo mapping
The tutorials need no save and are ~1 min from a cold boot, so they are the
cheapest route into a live flight scene. The HUD prints the equipped weapons as
NOSE BM 06000 / MAIN MPM 00300 — matching wep_01's LoadingCount 6000 and wep_02's
300, with the weapons identified from the HANGAR loadout rather than from the
ammo number. That makes it a genuinely independent confirmation of
Ammo Capacity == LoadingCount, from a different renderer in a different mode.

Also inventories the HUD elements (heat gauges, shield/armor pools, target armor
gauge, per-weapon RANGE markers) and notes that the RANGE gauge draws each
weapon's MaximumRange on a scale — a route to a real number where the Arsenal
panel only gives a letter class.

Adds autopilot.py (waypoint-arrow chaser). It detects reliably but oscillates;
recorded as such rather than as a working tool.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:50:46 +00:00
c3b5f3f401 re: state the real (non-circular) evidence for Ammo Capacity == LoadingCount
The 'N/N exact matches' framing was circular: each UI entry was identified BY its
ammo value. Replaced with what actually carries the claim — tab-uniqueness of the
forced match, the independent Max. Lock Ons agreement on two of them, and one
entry (STILETTO BG I) identified from the HANGAR loadout rather than from ammo.
Also downgrades LIGHT MACHINE GUN MG I to PROBABLE: three guns share ammo 3000.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:09:03 +00:00
ba33c533da re: runtime DATA SHEET reads the live weapon record (Route B, first capture)
Drove the retail game headless (Canary + lavapipe, vgamepad) to the Ready Room
Arsenal and read the Gallery-Mode DATA SHEET for every weapon the 5%-progress
save reveals.

Two UI->field mappings are CONFIRMED against weapons whose values ARE on disc:
  Ammo Capacity  == LoadingCount      (9/9 exact matches)
  Max. Lock Ons  == TriggerShotCount  (FALCON 9AM 12 == wep_02; BUZZARD 22 == wep_04)

That makes the panel an oracle for fields the disc leaves defaulted, and it is
shown for undeveloped weapons too, so no points need spending. Recovered
TriggerShotCount for wep_05 and wep_60 (both 4, on-disc absent), and bracketed
the defaulted Power / MaximumRange via the Range/Damage letter classes.

Ruled out first: the defaults are not in hidden/DefTables.pak (no WEAPON-schema
objects there) nor in the localized GP_MAIN_GAME_* duplicates.

Also adds the two analysis examples that produce the shopping list of defaulted
fields, the screenshot harness used to drive the menus, and the evidence PNGs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:01:25 +00:00
MechaCat02
c067761e56 docs: handoff — static ship placement exact, capture-verified (2026-07-26)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 19:54:55 +02:00
MechaCat02
ea32e77e45 ship: engine placement CONFIRMED by 43-instance capture; exhaust markers added
capture_verify example: clusters every part's ship-relative transform across
the new multi-snapshot, multi-instance F10 captures (5 snapshots, 43 e106
instances, all angles). Verdict: the dominant clusters match the static
assembly to ~1 unit on EVERY part — including BOTH engine nacelles at
(+-131, -133, -131) — so the "engines inside the hull" appearance is the
game's own placement: the engine geometry sits recessed in the aft hull, and
the visible "thrusters" in-game are exhaust FX drawn at the GN_Jet/GN_SJet
frames (Z ~ -570, past the stern).

To close that perception gap the viewer now draws simple exhaust cones at the
game's own jet frames (ship::exhaust_frames; part of the external-parts
toggle). The jet frames flip Z, so the cone apex is authored at +Z and lands
trailing aft — verified in the offline render.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 19:44:12 +02:00
MechaCat02
31f2637e6c viewer,ship: external parts ON by default + Ships catalog = composite ships only
Fixes the two things the user saw in the Ships browser:
- "Can't find the thrusters": show_external defaulted OFF, hiding the engine
  rig / bridge / turrets entirely. Externals are EXACT since the node-TRS fix,
  so the toggle now defaults ON (relabelled; off = bare hull bodies).
- "Some parts very far away": families without a composite scene graph (fighter
  morph sets like f002 whose detached parts are authored far from the origin,
  shared turret/prop part families) fell into the stack-at-origin fallback and
  rendered as piles with stray pieces. ShipModel now carries has_composite and
  the catalog lists only real assembled ships.

Offline audit across ALL stages (ship_audit example): the only far-away
outliers were composite-less f002 morph parts (now filtered); every capital
ship assembles coherently (e108 places its 7 turret instances, f105 its
mirrored shield generators). All primary composites are single-key tracks —
multikey exists only in animation composites (e901 attacks, missile-open),
which assemble_ship does not select.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 19:15:34 +02:00
MechaCat02
737b61242e ship: STATIC assembly is now EXACT — node-TRS off-by-one cracked via the capture
The runtime capture served its true purpose: as the answer key that exposed the
static encoding. The XBG7 node joint table (node+0x44) holds NINE keyframe
channels [TX TY TZ RY RX RZ SX SY SZ] behind EIGHT pointer slots shifted one
track forward: channel k+1 = f64@(ptr[k]+8), and channel 0 -- TX, which no
pointer names -- sits at ptr[0]-0x48 (tracks are 0x50-byte records, value at
+8). The old 8-slot read took TY as X, RY as Y (usually 0 -- why both hulls
stacked on the centreline) and never saw TX at all (the +-264 hull offsets).
Euler order is Ry(ch3)*Rx(ch4)*Rz(ch5) with angles applied directly, pinned by
the captured engine nacelle Rx(-15)*Rz(30) decomposition. mesh.rs gains
read_trs9/node_rotation; scene_world_nodes and node_transforms both fixed
(saber fin overrides unaffected).

assemble_ship rewritten fully static and exact:
- every composite-node instance placed (alias duplicates collapsed, real
  instances kept) including cross-id turret mounts (rou_e303_wep_01_root x2)
- the e_rou_<id>_eng cluster rig mounts at GN_Engine_01 (two mirrored nacelles
  +-131 with -+30-degree cant + centre engine) -- world = frame o rig_local
- brg/sld at their exact GN frames (frames were always exact)
- shared-geometry twin pairs at +-TX: X-reflect the instance whose offset
  opposes the geometry's dominant side (viewer reverses winding on det<0)
- squeezed node names resolve (wep02 -> wep_02_01)

Ground-truth gate: ship::tests::static_assembly_matches_runtime_capture asserts
static == the baked e106 capture for all 8 parts (T<1.0, R<0.02) plus both
nacelles, both turrets, and the starboard-hull mirror. Viewer now uses pure
static assembly; ship_capture + the baked table remain as the verification
oracle. Renders show a coherent, bilaterally-symmetric destroyer.

81 formats-lib + all disc tests green; viewer builds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 19:00:49 +02:00
MechaCat02
015e49ac8a ship: diagnostics — baked-table renderer + evidence the placement IS static
ship_render example: assemble a ship from the baked capture table (or --static)
and write orthographic top/side PPM renders + per-part world bounds — the
offline eye for placement bugs.

Findings recorded (docs to follow with the fix):
- e106 baked eng_01 is CROSS-INSTANCE contamination: the F10 capture de-dups by
  vertex-buffer address alone, so for a part drawn by several fleet ships only
  the FIRST instance's transform survives; eng_01's 30-degree rotation and
  below-hull position belong to a different destroyer. Fleet formation made it
  reproduce across captures, defeating the cross-validation.
- The static composite DOES carry the exact placement: BE-f64 +-264.0 (the
  lateral hull pair offset the static assembler loses) and -164.044 (the bridge
  Z we captured as -164.037) sit in e_rou_e106's node joint tables. The static
  decode is incomplete, not the data — full static assembly is achievable, with
  captures demoted to a verification oracle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 18:39:34 +02:00
MechaCat02
d9442a2106 ship_capture: bake e106 COMPLETE from the 2026-07-26 F10 capture — all 8 parts
The second F10 capture (ship at _m LOD distance) closes e106 entirely: all 8
parts placed including the bridge both earlier captures missed, cross-validating
the doc's 7 known translations to 0.1 units and adding brg_01 at the exact
centreline (0, 153.3, -164.0).

Matching hardened by what the real capture taught us:
- LOD-aware: each part tries every variant vcount (base/_m/_l/_d) — a distant
  ship draws its LOD copies, same local frame.
- Position-validated, SET-based, against the UNION of a part's variants: buffer
  order != decode order, and one draw's vcount equalled the _m count while its
  buffer held the FULL 1633-vert geometry. A coincidental vcount (foreign
  51-vert mesh vs the bridge) is rejected by geometry.
- Runtime mirror handled: twins share one file geometry; the engine uploads the
  starboard copy X-reflected. Mirror-validated draws bake diag(-1,1,1) and the
  viewer reverses triangle winding for det<0 so front faces stay outward.
- Near-axis rotation entries snapped to exact 0/+-1 for a clean table; eng_01
  keeps its genuine 30-degree nacelle rotation.

data/ship_placements.txt now ships e106 (viewer renders via the captured table);
embedded_e106_is_complete locks the baked data. part_pos + capture_match helper
examples added. 10 ship_capture tests; 80 formats-lib tests green; viewer builds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 18:26:23 +02:00
MechaCat02
4efc0278b1 ship_capture: ingest the draw-logger format + quantify the static gaps
The correlator now accepts BOTH capture formats: the F10 ship-capture snapshot
and the draw-logger format (mission_draws.log / xenia_re_draws.log) via
parse_drawlog — group the ship shader's draws by vertex-buffer base, vcount =
size_words/stride_words, keep the first c0..c2 WVP. So a capital ship seen in a
normal instrumented run bakes without a dedicated F10 pass. correlate_capture
auto-detects the format. Shared normalize_wvp between both parsers.

Also add examples/ship_dump.rs (offline static-placement inspector) and record
the measured static-assembler gaps for e106 in the doc: bdy_01/bdy_02 overlap
(runtime port/starboard mirror at X=+-264), eng_02 and wep_02_01 never placed.
These confirm exact placement is runtime-only — nothing more to squeeze
statically; the capture-driven table is the only path. The draw logs present on
this box are the player fighter, so no capital-ship table can be baked offline.

7 ship_capture tests (adds draw-log parse + correlate); 77 formats-lib green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:57:08 +02:00
MechaCat02
c6ca5ce9e7 ship: bake pipeline — correlator as a tested module + viewer prefers captured table
Promote the F10 ship-capture correlator from an example into a reusable, unit-
tested library module (ship_capture): parse_capture (log -> per-draw rigid
WorldView from the c0..c2 WVP constants), correlate (match parts to draws by
vertex count, express each in the reference part's frame), and a checked-in
placement-table format (serialize_table/parse_table, embedded via
embedded_placement from data/ship_placements.txt).

build_ship_model now prefers a ship's captured placement over the static
assemble_ship when a table entry exists (empty table -> unchanged fallback).
correlate_capture example refactored onto the module + gains --emit to print a
table block. Doc updated: bake plumbing done; remaining is running real F10
captures to populate the table (needs the emulator).

5 new ship_capture tests (parse/normalize/correlate/table round-trip); 76
formats-lib tests + viewer build green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:48:00 +02:00
MechaCat02
a68405216f re: runtime-capture pipeline for exact capital-ship part placement
Static external-part placement (ship::assemble_ship) is only approximate — a GN_*
frame is the mount pivot, and the real offset lives in a runtime detail rig. So
capture ground truth from Canary instead.

Finding: the guest vertex buffer holds LOCAL coords (byte-identical to the .xpr),
so capital-ship parts are placed entirely in the vertex shader. The per-part
transform is in the VS float constants — c0..c2 are the WorldViewProjection rows;
normalising each by its norm (= projection x/y/z scales, sy/sx = 16:9) yields the
rigid WorldView (rows orthonormal, det +1). Expressing every part relative to a
reference part cancels the camera: rel_p = WV_ref⁻¹·WV_p, a pure ship-space rigid
transform.

- examples/correlate_capture.rs: parse xenia_ship_capture.log, decode the ship's
  parts, match by vertex count, recover each part's ship-relative placement.
- docs/re/ship-placement-runtime-capture.md: full method + the capture protocol
  (Canary branch capture-ship-placement, F10 hotkey) + validated e106 result.

Validated on e106 (ADAN Destroyer, Stage_S01): all 7 drawn parts matched by
vertex count; bdy_01/bdy_02 recovered as a PORT/STARBOARD pair (X = ∓264) that
static had overlapping; engines correctly aft; extent 872×670×2181. Only brg_01
(bridge) missing — culled at that camera angle, needs a bridge-facing capture.

HANDOFF: next = re-capture bridge, bake a per-ship placement table the viewer
prefers over assemble_ship, then capture the cruiser / ACROPOLIS / TCAF ships.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 10:50:26 +02:00
MechaCat02
79c78cc7e1 ship: apply scale + gate approximate external parts behind a toggle
Two changes after user feedback that externals sit near-but-wrong (shield inset
into the hull, thruster floating close):

- Scale: the joint table's slots 5–7 are per-axis scale (a GN_ShieldG frame ships
  at 0.5, GN_Jet FX at 2.4–6.5) and were being ignored, rendering scaled parts at
  the wrong size. ScenePart now carries `s` and applies R·(S·v)+T.
- External placement is fundamentally approximate: a GN_* frame is the mount
  PIVOT (f105's two GN_ShieldG frames are both on the centreline), while the
  part's outboard/rotational offset lives in a detail sub-rig / runtime code this
  static pass can't recover. So assemble_ship gains `include_external`: the hull
  tier (rou_ nodes) is exact and always drawn; the external tier (bridge / shield
  / engine at GN_ frames) is opt-in via a "Show external parts" checkbox in the
  Ships browser, off by default so the clean, correct hull is the default view.

The module doc is corrected to the scene-graph mechanism (the old "draw parts
untransformed" claim was wrong).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 22:11:47 +02:00
MechaCat02
8c8e2dbeb4 ship: use only the primary composite — fixes parts floating mid-ship
The sub-composites (`e_rou_<id>_eng`, `_wep_*`) are LOCAL detail rigs: their
nodes are posed in their own frame, not ship-space, so folding them in placed
the engines at ~mid-ship (Z≈99) instead of aft — the "floating in air" parts.

The PRIMARY composite (`e_rou_<id>`, the one with the most nodes) already carries
the full ship-space layout: every hull body AND every `GN_*` hardpoint frame
(GN_Bridge at the stern, GN_Engine aft, GN_ShieldG amidships…). Assemble from
that one composite only; engines/bridge/shield now resolve to their real aft/mid
frames. Robust to the odd single-composite case (e108's `e_rou_e108_Missile_open`
is still picked as its own primary).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 21:52:20 +02:00
MechaCat02
a8b5691d1c ship: assemble capital ships from the composite scene graph (real placement)
The first cut drew every part at the origin, overlapping into a mess — the parts'
vertex buffers are authored around the origin, NOT pre-placed. The real world
placement lives in the composite scene-graph resource `e_rou_<id>`, exactly like
node_transforms poses the DeltaSaber's fins within one resource — but here the
nodes reference SEPARATE geometry resources.

Two tiers, matching how the game splits a ship (as the user described):
- Hull — `rou_<id>_*` scene nodes name the hull body resources and carry their
  world transform (bodies that together form the hull, not individually
  destructible). Composed parent∘child down the first-child/next-sibling tree.
- External hardpoints — `GN_*` frame nodes are the Vessel mounts (`GN_Bridge_01`,
  `GN_ShieldG_01`, `GN_Engine_01`, matching the IDXD recipe Frame names). Each
  bridge / shield-generator / engine part is placed at its frame by category+idx —
  the individually-destructible parts.

- mesh: `scene_world_nodes` — walk a composite, emit every `rou_`/`GN_` node with
  its composed world transform (same record layout + TRS convention as
  node_transforms, cross-resource). ScenePart{resource,m,t}+apply.
- ship: `assemble_ship(bytes, id)` — resolve hull nodes to the ship's own parts
  (shared cross-id turrets excluded — they need the per-mount recipe), match
  external parts to GN frames, fall back to a raw draw when a prop has no
  composite. is_base_part now also drops `_dNN`/`_lNN`/`_mNN` LOD copies.
- viewer: build_ship_model runs assemble_ship, decodes only referenced resources,
  bakes each part's world pose into its vertices (one posed copy per placement),
  then assembles via the shared prepare path. RequestShipRender carries the id.

Verified headless: parts now spread into coherent single-ship extents
(e108 185×174×683, f106 703×479×2079, f101 carrier 560×539×1712) instead of
piling at the origin. Turret placement (shared eNNN gun models on many GN_*Gun*
frames) still pending the per-mount recipe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 21:44:25 +02:00
MechaCat02
720a5fec27 viewer: Ships browser (View ▸ Ships) — assemble & view whole capital ships
Adds a Ships window that scans the stage containers, reconstructs each capital
ship from its XBG7 part family (via sylpheed_formats::ship), and renders the
whole assembled model in the 3D view on click.

- iso_loader: ShipBrowser resource + ShipRow (id, faction, name, parts, stage,
  and linked Vessel stats: HP/size/turrets/bridges/shield gens), RequestShipCatalog
  + RequestShipRender events, ShipCatalogLoaded message. build_ship_catalog scans
  Stage_S01..S30, groups resources into ships, joins Vessel recipes by id, and
  sorts capital ships (Vessel-matched) first. handle_ship_render_request decodes
  only the ship's parts (models_named) and assembles them off-thread, reusing the
  existing XprPrepared → apply_prepared_xpr render + camera-focus path.
- prepare_models gains an `assemble` mode (prepare_ship): parts share ONE recentre
  so each keeps its ship-local offset (bridge fore, engines aft) instead of the
  per-model recentre / thumbnail grid used for a stage browse.
- ui: draw_ships_ui (own system, keeps draw_viewer_ui under the 16-param limit) —
  faction filter chips, name/id search, collapsible per-ship stat cards with an
  "Assemble & view in 3D" button; colour-coded by faction. View ▸ Ships menu item.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 21:16:15 +02:00
MechaCat02
cacb576ecd formats: ship module — reconstruct capital ships from XBG7 part families
Project Sylpheed never stores a capital ship as a single mesh. Each ship is
modelled, then split into destructible parts (hull segments, bridge, engines,
integral turrets), each shipped as its own XBG7 resource inside a
hidden/resource3d/Stage_SNN.xpr container. The split lets the game hide/replace
a part when the player destroys it. Crucially each part keeps its ship-local
position, so reconstructing the whole ship is: draw every base part of one id
together, untransformed.

New `ship` module: Faction (from the e/f/n id prefix — ADAN/TCAF/Neutral),
ShipModel, ship_id_of, is_base_part (skips _l/_m/_d LODs, _bNN/_dead/_break
destruction geometry, e_rou_ scene nodes), and ships_in_container to group a
stage's resources into whole ships.

Supporting:
- mesh::xbg7_resource_names — list a container's XBG7 resource names (directory
  walk only, no geometry decode).
- Xbg7Model::models_named — decode only the named resources (assemble a ship
  from its ~8 parts instead of the whole ~300-resource stage).
- game_data::Vessel gains `model` (the rou_<id> hull stem) to link stats to the
  3D part family.

Tests: id/part classification + a disc-gated reconstruction of the ADAN e108
frigate from Stage_S02.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 21:16:04 +02:00
MechaCat02
df724b6bcb viewer: Game Data browser (View ▸ Game Data)
Wire the decoded game_data + localization tables into the viewer as a
browsable window. View ▸ Game Data opens a floating panel with category
tabs — Weapons, Craft, Capital Ships, Characters, Missions, Arsenal,
Flights — decoded off-thread from GP_MAIN_GAME_E + GP_HANGAR_ARSENAL.

- Weapons/Craft/Ships: sortable stat grids (power/velocity/range;
  HP/cruise/accel/radar/turrets; HP/length/turrets/bridges/shield-gen),
  with a live name filter.
- Characters: name (localized) + faction (colour-coded) + portrait count.
- Missions: collapsible per stage — location, resolved objectives, fail
  conditions, and the enemy roster with HP.
- Arsenal: the player's nose/arm weapon options in four columns.
- Flights: distinct wingman line-ups.

Its own egui system (keeps draw_viewer_ui under Bevy's 16-param limit);
a RequestGameData event from the menu opens it and lazily decodes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 20:44:40 +02:00
MechaCat02
d41b065813 game_data: Arsenal loader — player weapon options per hardpoint
Decode the player weapon arsenal from the UNITS tables
(GP_HANGAR_ARSENAL.pak): the selectable weapons per Delta Saber
hardpoint. Each `STANDARD_<hp>` header (Nose / Arm1-3) lists its options
up to the next header — extracted by shape, unioned + deduped across
configs.

The arsenal reads with the game's own names: NOSE guns (Stiletto,
Rapier, Broad Sword, Machine Cannon), ARM missiles/bombs (Falcon, Eagle,
White Shark, Piranha, Tomahawk Rail Gun, Maelstrom Bomb). Example
`arsenal` prints them. +1 test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 20:30:13 +02:00
MechaCat02
f2991b61ab game_data: load_weapons catches variant-schema weapons
The player's Delta Saber missile / special weapons (Weapon_DSaber_P_wep_*)
live in variant schemas (439b1f29, 28295901), not the main WEAPON schema,
so the schema-only filter missed 10 of them. Match all `Weapon`-shaped
records by table name instead (excluding the EnumWeapon name lists),
deduped by id: 121 → 131 weapons.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 20:26:35 +02:00
MechaCat02
f5182f9649 game_data: PilotRoster loader — player squadron flight assignments
Decode the UNITS/ZUNITS player-unit tables (GP_HANGAR_ARSENAL.pak) into
flight assignments: which pilot flies each callsign (Rhino1→Katana,
Bird1→Sandra, …), plus the player craft. Assignments are `Callsign<N>-
Pilot` tokens, extracted by shape.

138 configs; the distinct line-ups trace the campaign's story — Raymond
leads Rhino flight early (Rhino1=Raymond, Rhino2=Katana), then Katana
takes the lead (Rhino1=Katana, Rhino2=Ellen, Rhino3=Gene, Rhino4=Yoji).
Example `flights` prints the line-ups. +1 test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 20:14:22 +02:00
MechaCat02
551291f869 game_data: UnitRoster loader — per-mission battle compositions
Decode the EnumUnit tables (schema 35b8dc67) into mission rosters: the
distinct combatant `UN_*` ids for one battle, stage props (asteroids,
collision meshes) dropped. The tables aren't tagged with their stage on
disc, so `stage` is inferred from any `UN_S<NN>_…` prop id in the roster
and left None otherwise (honest — ~8 of 31 rosters self-identify).

Joined against load_units / load_vessels this gives each battle's
combatants with stats, e.g. S01 = ADAN Turret (100 HP), Attacker
(500 HP), Destroyer (10000 HP). Example `battle` prints them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 20:11:32 +02:00
MechaCat02
6e704291be game_data: DemoMessage loader — the dialogue backbone
Decode the message tables (schema b412e6d8) into dialogue lines: each
`Message_NNN` sub-record → a DemoMessage with the speaker, portrait
(face), voice-clip token, and caption page text-keys. Fields are found
by prefix (Character*/Face*/VOICE_*/id-prefixed pages), so the
positional / first-record-defines-schema layout doesn't matter.

10,263 lines parse; ~815 name an explicit speaker (the rest are system /
continuation lines — left unattributed rather than guessed). Paired with
localization::TextIndex the attributed ones read straight out — e.g.
KATANA [VOICE_RHIN_000] "Ellen, get back into formation!". The
voice-clip token ties each line to its audio bank, closing the loop with
the cutscene-voice work. +1 test; example `dialogue` prints
speaker + clip + resolved text.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 19:59:14 +02:00
MechaCat02
2930770060 game_data: Squadron loader — mission flight groups
Decode the per-stage Enumerate_Squadrons tables (UnitGroup_S<NN>.tbl)
into flight groups: formation shape, AI behaviour, side, and the member
craft. Definitions are delimited by the FormationID key; members are the
first `count` UN_ craft after the header (anything past them belongs to
the next squadron or the stage roster — the naive "all UN_ in span"
over-collected). e.g. the player's Rhino flight = a 2-craft TCAF
squadron of Delta Sabers flying Formation_2_Rhino. Schema hash varies
per stage so we match on the table name. Squadron id is positional and
left None when the id list and definitions don't line up (honest, not
guessed). +1 test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 19:54:48 +02:00
MechaCat02
d73601fe45 localization: resolve the game's UTF-16 text keys to display strings
Project Sylpheed stores its UI/story text in IXUD entries as UTF-16LE,
interleaved `text, key` (a prose string immediately followed by the
identifier that names it). TextIndex::build(pak) scans a language pak's
IXUD entries and indexes every pair — 8457 English keys from
GP_MAIN_GAME_E: objectives, hints, lose conditions, character names,
and cutscene dialogue.

This makes the whole game read in English: the 16-mission campaign
(Repel the surprise attack → Shoot down Margras → the Prometheus Driver
finale), each mission's briefing, and the named cast (Katana, Raymond,
Ellen, Crichton, …) all resolve. API: get / objectives / lose_conditions
/ hints / character_name. 3 tests.

Examples: campaign / dossier print the readable campaign + cast.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 19:49:42 +02:00
MechaCat02
99f2ef8871 game_data: typed loaders for the combat / character / mission tables
Decode Project Sylpheed's IDXD data model into plain, cloneable structs
(from GP_MAIN_GAME_E.pak; the D/F/I/J/S paks are localized dups):

- Weapon (121), CraftUnit (89, with the AI flight model in .fields),
  Vessel (23, structural component counts), PlayerConfig (24) — the
  full combat balance.
- Character (68 — faction + portrait set), Stage (29 missions:
  location, phases, and the per-stage resource tables it wires up).
- Generic Record / load_records(schema) reads any of the 105 IDXD
  schemas without a bespoke struct.

Each blob = one entity; token[0] names the table, fields are
value-before-key. Named Option<> fields for the common stats + a full
`fields` map; defaulted fields stay None (they're code-resident, not on
disc — honest blanks, never guessed). 8 tests against the real disc.

Examples: iso_map / deep_map census the ISO's formats and IDXD schemas;
dm_extract / dm_rows / dm_roster / mission_map dump the decoded data.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 19:49:21 +02:00
MechaCat02
161359e151 Fix hokyu voice: resolve by subtitle demo-id, not ship category
The 13 manifest-unbound hokyu (resupply) cutscenes reuse 5 recordings
(VOICE_D_450..454). The category guess (ship LS/DS × source carrier/
tanker) was wrong for some. The real selector is the cutscene's subtitle
demo-id: 600→450, 601→451, 602→452, 603→453, 604→454 — derived from the
5 BOUND hokyu (each carries both a demo-id and a VOICETRACK), so it
self-validates. Fixes hokyu_LS_s11A/s15A (→451, were wrongly 450) and
hokyu_LS_s24A/s27A (→silent, no voice cue). User-confirmed all correct.

hokyu_voice_token() builds the demo→token map from the bound hokyu and
looks up the target movie's demo-id; replaces the category fallback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 19:48:56 +02:00
MechaCat02
053aa81953 Cutscene voice: resolve movies via continuous-stream cue regions
Cracked how the game maps a movie to its voice track and reimplemented it,
fixing wrong/short voices for RT and hokyu cutscenes.

## Mechanism (RE'd from a Canary file-I/O trace, user-confirmed by ear)

The movie voices are ONE continuous XMA stream, physically chunked into
`<lang>\Movie\VOICE_*.slb` (and `\etc\VOICE_D_*.slb`) sound.pak TOC entries
whose boundaries do NOT align with the cutscene cues. A single cue routinely
spans two `.slb` chunks, so a `.slb` does not necessarily hold the track its
name claims (they are off-by-one vs the cues). Each cue ends at an inline
trailer descriptor `(sound_id:u32be, 0x11, …)` whose id repeats at +0x800.

Resolution chain (all derivable on-disc, no guest-code RE):
  1. movie -> cue token         (ADVERTISE_MOVIE manifest VOICETRACK)
  2. cue token -> sound-id       (master sound registry, tables.pak
                                  a2c8c185=eng / b04238e0=jpn, schema 0x13cb84ba)
  3. sound-id -> [start,end)      scan the stream for trailer(id) and its
                                  predecessor; cue audio = the bytes between,
                                  read continuously across chunk boundaries.

Validated: decoded voice length matches each movie within ~0.4s, including
cross-boundary cues (ADV/RT01A/RT01B/RT01C/S00A/S01A + the 5 bound hokyu).

## Changes

- sylpheed-formats/src/movie_voice.rs (new): registry_voice_ids(),
  find_descriptor(), find_descriptor_before() (gap-tolerant predecessor).
  Unit-tested.
- viewer iso_loader.rs: resolve_movie_voice_region() + decode_voice_region()
  (continuous read via existing read_segment_range + to_xma_riffs). Voice/
  audio requests now region-first; a `\Movie\` token that fails region stays
  SILENT (never plays the wrong off-by-one chunk). Removed the old
  movie_is_demo_based suppress hack.
- Anchor tries subdirs Movie/etc/Voice so hokyu `\etc\VOICE_D_*` resolve too.
- Examples resolve_all/validate_cues/hokyu_final document + validate the pipeline.

## Status

- RT / S / ADV cutscene voices: CORRECT (user-confirmed).
- 5 manifest-bound hokyu (VOICE_D_450-454): CORRECT (user-confirmed).

## OPEN (handoff)

13 of 18 hokyu movies have NO manifest VOICETRACK; the game reuses the 5
recordings across stages via a code rule. `hokyu_fallback_token()` currently
guesses by category (ship LS/DS x source carrier-A/tanker-H: LS/A->450,
LS/H->453, DS/A->452, DS/H->454). USER REPORTS THIS IS SOMETIMES WRONG.
- LS/carrier has two takes (450 from s02A, 451 from s09A); the early-vs-late
  split is unknown — unbound LS/A currently all default to 450.
- Definitive fix = Canary `--phase_a_fileio_only` file.read trace of an unbound
  hokyu (e.g. hokyu_LS_s03A) to see which VOICE_D the game actually streams,
  then encode the real rule. Same method that cracked the RT mapping.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 06:53:55 +02:00
MechaCat02
ed5c2f24c3 feat(formats): decode the movie manifest's full mission/phase structure
The movie manifest (tables.pak, schema 0x067025b9) is the game's authoritative mission -> phase -> cutscene table. Previously we only scraped VOICE_/SUBTITLE_/.wmv strings by prefix; now we decode the real two-array (slot-keys, then value-records) structure.

MovieEntry gains slot, kind (System/Intro/Phase/PhaseEnd/Supply), mission, phase, subtitle, and telop (the previously-ignored .prt on-screen text overlay). Valueless slots (stages with no resupply video) are recovered without decoding the binary node region, via the fully-derivable MS<NN><X> -> S<NN><X> anchors.

Backward compatible: movie/voice_token and resolve_voice_entry/voice_token/is_manifest are unchanged. Validated end-to-end on the real disc: 101 entries (104 slots - 3 gaps), typed System=6/Intro=27/Phase=32/PhaseEnd=18/Supply=18. examples/manifest_map.rs dumps the full map.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:49:19 +02:00
MechaCat02
eee1607e1b feat(formats,viewer): standalone voice player covers all line categories
list_voice_clips now enumerates every spoken-line category so the standalone audio player covers them all — bound movie voices (\Movie\), in-mission radio (\Voice\, \etc\), and mission briefings (\Briefing\, whose BR<NN>_<MM> names lack "VOICE"); music (BGM_*) stays excluded.

Viewer voice-library fixes: invalidate the library when a new game source loads, and don't latch an empty result as "loaded" (sounds.tbl always has thousands of clips, so empty = a failed read) so re-opening the menu retries.

Voice decode reverted to the first sub-wave (base behaviour): a naive multi-sub-wave concat produced the wrong track for RT movies. The per-sub-wave splitter (to_xma_riffs) is kept + documented for the eventual real fix, which needs ground-truth playback instrumentation to resolve segments-vs-takes-vs-timecode placement.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:49:09 +02:00
121 changed files with 24089 additions and 117 deletions

1
.gitignore vendored
View File

@@ -20,3 +20,4 @@ Thumbs.db
# Trunk build output
dist/
__pycache__/

View File

@@ -0,0 +1,21 @@
# Capital-ship part placements — runtime-captured ground truth (see
# docs/re/ship-placement-runtime-capture.md and src/ship_capture.rs).
#
# Populate with:
# SYLPHEED_ISO=... cargo run --release --example correlate_capture -- \
# <capture.log> <Stage_SNN> <ship_id> [ref_part_substr] --emit
# then paste the emitted `ship …` block(s) below. The viewer prefers a ship's
# entry here over the static assembler when present.
#
# Per placement line: <part> <R00 R01 R02 R10 R11 R12 R20 R21 R22> <T0 T1 T2>
#
ship e106 ref=e106_bdy_04
e106_bdy_01 1 0 0 0 1 0 0 0 1 -263.9948 -150.43242 1164.8667
e106_bdy_02 -1 0 0 0 1 0 0 0 1 264.0088 -150.41339 1164.8651
e106_bdy_03 1 0 0 0 1 0 0 0 1 0.0029247368 -34.517372 1075.878
e106_bdy_04 1 0 0 0 1 0 0 0 1 0 0 0
e106_brg_01 1 0 0 0 1 0 0 0 1 -0.005471501 153.31795 -164.03748
e106_eng_01 0.8659965 -0.50002277 0 0.4829627 0.83651507 0.25885975 -0.12941553 -0.22417325 0.96592265 131.2386 -133.06625 -132.06075
e106_eng_02 1 0 0 0 1 0 0 0 1 0.0013684053 -49.088192 -139.47827
e106_wep_02_01 1 0 0 0 1 0 0 0 1 0.000040663195 49.126797 1009.06744

View File

@@ -0,0 +1,9 @@
use sylpheed_formats::{game_data as gd, PakArchive};
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let pak=PakArchive::open(format!("{disc}/dat/GP_HANGAR_ARSENAL.pak")).unwrap();
let a=gd::load_arsenal(&pak);
for (hp,list) in [("NOSE",&a.nose),("ARM1",&a.arm1),("ARM2",&a.arm2),("ARM3",&a.arm3)]{
println!("{hp} ({}): {:?}", list.len(), list.iter().take(8).collect::<Vec<_>>());
}
}

View File

@@ -0,0 +1,19 @@
use sylpheed_formats::{game_data as gd, PakArchive};
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let pak=PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")).unwrap();
let units=gd::load_units(&pak); let vessels=gd::load_vessels(&pak);
let hp=|id:&str|->String{
units.iter().find(|u|u.id.as_deref()==Some(id)).and_then(|u|u.hp).map(|h|format!("{h:.0}hp"))
.or_else(||vessels.iter().find(|v|v.id.as_deref()==Some(id)).and_then(|v|v.hp).map(|h|format!("{h:.0}HP")))
.unwrap_or("·".into())
};
let rosters=gd::load_unit_rosters(&pak);
for r in rosters.iter().filter(|r|r.stage.is_some()).take(4){
println!("\n{}{} combatants:", r.stage.as_deref().unwrap(), r.units.len());
for u in r.units.iter().filter(|u|u.contains("ADAN")).take(6){
let short=u.trim_start_matches("UN_").split('_').skip(1).collect::<Vec<_>>().join("_");
println!(" {short:32} {}", hp(u));
}
}
}

View File

@@ -0,0 +1,24 @@
use sylpheed_formats::{game_data, localization::TextIndex, PakArchive};
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let pak=PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")).unwrap();
let text=TextIndex::build(&pak);
let mut stages=game_data::load_stages(&pak);
stages.retain(|s|s.id.starts_with('S') && s.id.len()==3 && s.id[1..].parse::<u32>().map(|n|n<=16).unwrap_or(false));
stages.sort_by(|a,b|a.id.cmp(&b.id));
println!("═══ CAMPAIGN (S01S16) ═══");
for s in &stages{
let obj=text.objectives(&s.id,1);
println!("\n{} · {}", s.id, s.location.as_deref().unwrap_or("?"));
for o in obj.iter().take(2){ println!("{o}"); }
}
// roster with real names
let mut chars=game_data::load_characters(&pak);
chars.retain(|c|c.faction.as_deref()==Some("TCAF") && c.unique==Some(true) && c.faces.len()>=4);
println!("\n═══ PRINCIPAL CAST (TCAF, named) ═══");
for c in &chars{
let id=c.id.as_deref().unwrap_or("");
let name=text.character_name(id.trim_start_matches("Character")).or_else(||c.name_key.as_deref().and_then(|k|text.get(k))).unwrap_or("?");
println!(" {name:12} ({} portraits) [{}]", c.faces.len(), id.trim_start_matches("Character"));
}
}

View File

@@ -0,0 +1,43 @@
//! Identify which stage + ship a capture log came from: match the capture's
//! draw vertex counts against every resource (incl. LODs) of every Stage_SNN.xpr.
//! cargo run --release --example capture_match -- <capture.log> <resource3d dir>
use sylpheed_formats::mesh::{xbg7_resource_names, Xbg7Model};
use sylpheed_formats::ship_capture::{parse_capture, parse_drawlog};
use std::collections::{BTreeMap, HashSet};
fn main() {
let a: Vec<String> = std::env::args().collect();
let text = std::fs::read_to_string(&a[1]).unwrap();
let mut draws = parse_capture(&text);
if draws.is_empty() { draws = parse_drawlog(&text); }
let caps: HashSet<u32> = draws.iter().map(|d| d.vcount).filter(|v| *v >= 100).collect();
eprintln!("{} draws, {} distinct vcounts>=100", draws.len(), caps.len());
let mut entries: Vec<_> = std::fs::read_dir(&a[2]).unwrap().filter_map(|e| e.ok())
.filter(|e| { let n = e.file_name().to_string_lossy().to_string(); n.starts_with("Stage_") && n.ends_with(".xpr") })
.map(|e| e.path()).collect();
entries.sort();
for path in entries {
let bytes = std::fs::read(&path).unwrap();
let names = xbg7_resource_names(&bytes);
let want: HashSet<String> = names.iter().cloned().collect();
let models = Xbg7Model::models_named(&bytes, &want, &|| false);
// resource -> vcount; group hits by ship-id prefix
let mut hits: BTreeMap<String, Vec<(String, u32)>> = BTreeMap::new();
for m in &models {
let vc: usize = m.meshes.iter().map(|s| s.positions.len()).sum();
if vc >= 100 && caps.contains(&(vc as u32)) {
let id = m.name.get(..4).unwrap_or("?").to_string();
hits.entry(id).or_default().push((m.name.clone(), vc as u32));
}
}
let total: usize = hits.values().map(|v| v.len()).sum();
if total >= 3 {
println!("== {} : {} matching resources ==", path.file_name().unwrap().to_string_lossy(), total);
for (id, v) in &hits {
let s: Vec<String> = v.iter().map(|(n, c)| format!("{n}({c})")).collect();
println!(" {id}: {}", s.join(" "));
}
}
}
}

View File

@@ -0,0 +1,192 @@
//! Verify static ship assembly against multi-snapshot, multi-instance F10
//! captures: cluster every part's ship-relative transform across ALL ships in
//! ALL snapshots, and compare the dominant clusters with `assemble_ship`.
//!
//! cargo run --release --example capture_verify -- <stage.xpr> <ship_id> <log> [log…]
//!
//! Every draw is position-validated against the ship's parts (any LOD, set
//! based). Each `bdy_04`(-LOD) draw acts as a ship reference; every validated
//! part draw within ship range contributes `rel = WV_ref⁻¹ ∘ WV_part`. The true
//! per-part placements appear as tight clusters repeated once PER SHIP per
//! snapshot; cross-ship pairings scatter and stay below the cluster threshold.
use std::collections::HashSet;
use sylpheed_formats::mesh::{xbg7_resource_names, Xbg7Model};
use sylpheed_formats::ship::{assemble_ship, is_base_part, ship_id_of};
use sylpheed_formats::ship_capture::{parse_capture, CapturedDraw};
type M3 = [[f64; 3]; 3];
fn transpose(m: &M3) -> M3 {
[
[m[0][0], m[1][0], m[2][0]],
[m[0][1], m[1][1], m[2][1]],
[m[0][2], m[1][2], m[2][2]],
]
}
fn mmul(a: &M3, b: &M3) -> M3 {
let mut o = [[0.0; 3]; 3];
for i in 0..3 {
for j in 0..3 {
o[i][j] = (0..3).map(|k| a[i][k] * b[k][j]).sum();
}
}
o
}
fn mv(m: &M3, v: [f64; 3]) -> [f64; 3] {
[
m[0][0] * v[0] + m[0][1] * v[1] + m[0][2] * v[2],
m[1][0] * v[0] + m[1][1] * v[1] + m[1][2] * v[2],
m[2][0] * v[0] + m[2][1] * v[1] + m[2][2] * v[2],
]
}
fn main() {
let a: Vec<String> = std::env::args().collect();
let bytes = std::fs::read(&a[1]).unwrap();
let id = &a[2];
// Part position sets (base + LODs share the local frame) + per-variant vcounts.
let names = xbg7_resource_names(&bytes);
let base_parts: Vec<String> = names
.iter()
.filter(|n| is_base_part(n) && ship_id_of(n) == Some(id.as_str()))
.cloned()
.collect();
let mut want: HashSet<String> = base_parts.iter().cloned().collect();
for p in &base_parts {
for suf in ["_m", "_l", "_d"] {
let c = format!("{p}{suf}");
if names.contains(&c) {
want.insert(c);
}
}
}
let models = Xbg7Model::models_named(&bytes, &want, &|| false);
struct Part {
base: String,
vcounts: Vec<u32>,
pos: Vec<[f32; 3]>, // union of variants
}
let mut parts: Vec<Part> = Vec::new();
for p in &base_parts {
let mut vcounts = Vec::new();
let mut pos = Vec::new();
for cand in [p.clone(), format!("{p}_m"), format!("{p}_l"), format!("{p}_d")] {
if let Some(m) = models.iter().find(|m| m.name == cand) {
let mp: Vec<[f32; 3]> =
m.meshes.iter().flat_map(|s| s.positions.iter().copied()).collect();
vcounts.push(mp.len() as u32);
pos.extend(mp);
}
}
parts.push(Part { base: p.clone(), vcounts, pos });
}
// Validate a draw against a part (direct or X-mirrored), like ship_capture.
let validate = |d: &CapturedDraw, p: &Part| -> Option<bool> {
if !p.vcounts.contains(&d.vcount) || d.pos.is_empty() {
return None;
}
let near = |a: &[f32; 3], b: &[f32; 3]| {
(a[0] - b[0]).abs() <= 1e-2 && (a[1] - b[1]).abs() <= 1e-2 && (a[2] - b[2]).abs() <= 1e-2
};
let (mut direct, mut mirror) = (0usize, 0usize);
for q in &d.pos {
if p.pos.iter().any(|v| near(q, v)) {
direct += 1;
}
let qm = [-q[0], q[1], q[2]];
if p.pos.iter().any(|v| near(&qm, v)) {
mirror += 1;
}
}
let need = d.pos.len().div_ceil(2);
if direct >= need && direct >= mirror {
Some(false)
} else if mirror >= need {
Some(true)
} else {
None
}
};
// Collect rel samples per part across all logs.
let mut samples: std::collections::BTreeMap<String, Vec<[f64; 3]>> = Default::default();
for log in &a[3..] {
let text = std::fs::read_to_string(log).unwrap();
let draws = parse_capture(&text);
// label validated draws
let mut labeled: Vec<(usize, usize, bool)> = Vec::new(); // (draw, part, mirrored)
for (di, d) in draws.iter().enumerate() {
for (pi, p) in parts.iter().enumerate() {
if let Some(mir) = validate(d, p) {
labeled.push((di, pi, mir));
break;
}
}
}
let refs: Vec<usize> = labeled
.iter()
.filter(|(_, pi, _)| parts[*pi].base.ends_with("bdy_04"))
.map(|(di, ..)| *di)
.collect();
for &ri in &refs {
let rf = &draws[ri];
let rt = transpose(&rf.r);
for &(di, pi, _mir) in &labeled {
let d = &draws[di];
let dt = [d.t[0] - rf.t[0], d.t[1] - rf.t[1], d.t[2] - rf.t[2]];
let rel = mv(&rt, dt);
if rel.iter().map(|v| v * v).sum::<f64>().sqrt() < 2600.0 {
// also require the rel rotation be finite-sane
let _rm = mmul(&rt, &d.r);
samples.entry(parts[pi].base.clone()).or_default().push(rel);
}
}
}
eprintln!("{log}: {} validated draws, {} bdy_04 refs", labeled.len(), refs.len());
}
// Cluster per part (greedy, 8-unit radius), print clusters with ≥3 samples.
println!("\n== dominant ship-relative placements (clusters ≥3 samples) ==");
for (part, mut v) in samples {
v.sort_by(|x, y| x[0].partial_cmp(&y[0]).unwrap());
let mut clusters: Vec<([f64; 3], usize)> = Vec::new();
for s in &v {
match clusters.iter_mut().find(|(c, _)| {
(c[0] - s[0]).abs() < 8.0 && (c[1] - s[1]).abs() < 8.0 && (c[2] - s[2]).abs() < 8.0
}) {
Some((c, n)) => {
for k in 0..3 {
c[k] = (c[k] * *n as f64 + s[k]) / (*n as f64 + 1.0);
}
*n += 1;
}
None => clusters.push((*s, 1)),
}
}
clusters.sort_by(|x, y| y.1.cmp(&x.1));
let tops: Vec<String> = clusters
.iter()
.filter(|(_, n)| *n >= 3)
.take(6)
.map(|(c, n)| format!("[{:7.1} {:7.1} {:7.1}]×{n}", c[0], c[1], c[2]))
.collect();
println!(" {part:20} {}", tops.join(" "));
}
// Static assembly rel bdy_04 for comparison.
println!("\n== static assemble_ship rel bdy_04 ==");
let placed = assemble_ship(&bytes, id, true);
if let Some(r4) = placed.iter().find(|p| p.resource == format!("{id}_bdy_04")) {
for p in &placed {
println!(
" {:20} [{:7.1} {:7.1} {:7.1}]",
p.resource,
p.t[0] - r4.t[0],
p.t[1] - r4.t[1],
p.t[2] - r4.t[2]
);
}
}
}

View File

@@ -0,0 +1,136 @@
//! Recover exact capital-ship part placement from a Canary capture log
//! and (optionally) emit a checked-in placement-table block.
//!
//! The correlation math lives in [`sylpheed_formats::ship_capture`]; this example
//! is the CLI wrapper: it reads the log, decodes the ship's parts from the disc to
//! get their vertex counts + leading positions (the match keys), correlates, and
//! prints the result. With `--emit` it prints the `ship …` block to paste into
//! `crates/sylpheed-formats/data/ship_placements.txt`.
//!
//! Matching is **LOD-aware**: a ship on screen at distance is drawn with its
//! `_m`/`_l` LOD copies, which live in the same local frame as the base part — so
//! each base part tries its own vcount first, then its LOD variants', and the
//! recovered transform is recorded under the base name. Same-vcount twins
//! (mirrored port/starboard hulls) are routed by the draw's position dump.
//!
//! Usage:
//! SYLPHEED_ISO=... cargo run --release --example correlate_capture -- \
//! <capture.log> <Stage_SNN> <ship_id> [ref_part_substr] [--emit]
//! e.g. SYLPHEED_ISO="/home/fabi/RE Project Sylpheed/Project Sylpheed - Arc of
//! Deception (USA, Europe) (En,Ja).iso" \
//! cargo run --release --example correlate_capture -- \
//! xenia_ship_capture.log Stage_S01 e106 bdy_04 --emit
use sylpheed_formats::mesh::{xbg7_resource_names, Xbg7Model};
use sylpheed_formats::ship::{is_base_part, ship_id_of};
use sylpheed_formats::ship_capture::{
correlate, parse_capture, parse_drawlog, serialize_table, PartKey,
};
use sylpheed_formats::xiso::open_iso;
use std::collections::HashSet;
use std::path::Path;
fn main() {
let args: Vec<String> = std::env::args().collect();
let positional: Vec<&String> = args[1..].iter().filter(|a| !a.starts_with("--")).collect();
let emit = args.iter().any(|a| a == "--emit");
if positional.len() < 3 {
eprintln!(
"usage: correlate_capture <capture.log> <Stage_SNN> <ship_id> [ref_part_substr] [--emit]"
);
std::process::exit(2);
}
let (log, stage, id) = (positional[0], positional[1], positional[2]);
let ref_sub = positional.get(3).map(|s| s.as_str()).unwrap_or("bdy_04");
let iso = std::env::var("SYLPHEED_ISO").expect("SYLPHEED_ISO");
// Accept either the F10 ship-capture format or the draw-logger format;
// auto-detect by trying the F10 parser first.
let text = std::fs::read_to_string(log).expect("read log");
let mut draws = parse_capture(&text);
if draws.is_empty() {
draws = parse_drawlog(&text);
eprintln!("parsed {} draws (draw-logger format)", draws.len());
} else {
eprintln!("parsed {} draws (F10 capture format)", draws.len());
}
// Decode the ship's base parts AND their LOD copies (vcount + leading
// positions are the match keys; LODs share the base part's local frame).
let bytes = {
let rt = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap();
rt.block_on(async {
let mut r = open_iso(Path::new(&iso)).await.unwrap();
r.read_file(&format!("hidden/resource3d/{stage}.xpr")).await.unwrap()
})
};
let names = xbg7_resource_names(&bytes);
let base_parts: Vec<String> = names
.iter()
.filter(|n| is_base_part(n) && ship_id_of(n) == Some(id.as_str()))
.cloned()
.collect();
// Candidate resources per base part: itself + `_m`/`_l`/`_d` LOD copies.
let mut want: HashSet<String> = base_parts.iter().cloned().collect();
for p in &base_parts {
for suf in ["_m", "_l", "_d"] {
let cand = format!("{p}{suf}");
if names.contains(&cand) {
want.insert(cand);
}
}
}
let models = Xbg7Model::models_named(&bytes, &want, &|| false);
let positions_of = |name: &str| -> Option<Vec<[f32; 3]>> {
let m = models.iter().find(|m| m.name == name)?;
Some(m.meshes.iter().flat_map(|s| s.positions.iter().copied()).collect())
};
// One PartKey per (part, variant-vcount present in the capture) — correlate
// takes the first that validates. Validation uses the UNION of the part's
// variant position sets: a draw's `vcount` can match one LOD while its
// buffer holds another variant's geometry (seen on the real e106: the
// 558-vcount draw carried the FULL 1633-vert bdy_04 buffer), and every
// variant is the same part in the same local frame.
let mut keys: Vec<PartKey> = Vec::new();
for part in &base_parts {
let variants =
[part.clone(), format!("{part}_m"), format!("{part}_l"), format!("{part}_d")];
let union: Vec<[f32; 3]> =
variants.iter().filter_map(|v| positions_of(v)).flatten().collect();
let mut any = false;
for cand in &variants {
if let Some(pos) = positions_of(cand) {
let vcount = pos.len() as u32;
if draws.iter().any(|d| d.vcount == vcount) {
let lod = if cand == part { "full" } else { cand.rsplit('_').next().unwrap_or("?") };
eprintln!(" {part:20} try vcount={vcount:6} [{lod}]");
keys.push(PartKey { part: part.clone(), vcount, ref_pos: union.clone() });
any = true;
}
}
}
if !any {
eprintln!(" {part:20} — no draw matches any LOD (culled/off-screen?)");
}
}
let Some(ship) = correlate(id, &draws, &keys, ref_sub) else {
eprintln!("no parts matched a captured draw");
return;
};
eprintln!("\nreference = {} → ship-relative placement:", ship.reference);
for p in &ship.parts {
eprintln!(" {:18} T=[{:9.1}{:9.1}{:9.1}]", p.part, p.t[0], p.t[1], p.t[2]);
}
for part in &base_parts {
if !ship.parts.iter().any(|p| &p.part == part) {
eprintln!(" {part:18} — NOT placed (culled, or its only vcount hit failed position validation)");
}
}
if emit {
print!("{}", serialize_table(std::slice::from_ref(&ship)));
}
}

View File

@@ -0,0 +1,59 @@
use sylpheed_formats::{ratc, idxd::IdxdObject, PakArchive};
use std::collections::BTreeMap;
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
// 1. RATC child-type census across the big RATC paks
let mut childtypes:BTreeMap<String,u64>=BTreeMap::new();
let mut ratc_unparsed=0u64;
for pk in ["GP_READY_ROOM","GP_MOVIE_THEATER","GP_DIALOG","GP_DEBRIEFING_PILOTLOG","GP_TITLE","GP_BUNK"]{
let Ok(arc)=PakArchive::open(format!("{disc}/dat/{pk}.pak")) else{continue};
for e in arc.entries(){
let Ok(b)=arc.read(e) else{continue};
if !ratc::is_ratc(&b){continue;}
match ratc::parse(&b){
Some(kids)=> for k in kids{
let ext=k.name.rsplit('.').next().unwrap_or("?").to_lowercase();
*childtypes.entry(ext).or_default()+=1;
},
None=>ratc_unparsed+=1,
}
}
}
println!("=== RATC child-type census (big RATC paks) ===");
let mut cv:Vec<_>=childtypes.into_iter().collect(); cv.sort_by_key(|x|std::cmp::Reverse(x.1));
for (e,c) in &cv{ println!(" .{e:8} ×{c}"); }
println!(" (RATC bundles that failed to parse: {ratc_unparsed})");
// 2. The 00000002 mystery format (GP_MAIN_GAME_E)
let arc=PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")).unwrap();
for e in arc.entries(){
let Ok(b)=arc.read(e) else{continue};
if b.len()>=4 && b[0..4]==[0,0,0,2]{
let hx:String=b[..48.min(b.len())].iter().map(|x|format!("{x:02x}")).collect::<Vec<_>>().join(" ");
let asc:String=b[..64.min(b.len())].iter().map(|&x|if(0x20..0x7f).contains(&x){x as char}else{'.'}).collect();
println!("\n=== 00000002 format sample ({}B) ===\n{hx}\n{asc}",b.len());
break;
}
}
// 3. Sample the top undecoded IDXD schemas: first tokens (guess semantics)
println!("\n=== top undecoded IDXD schemas — sample tokens (semantic hints) ===");
let targets:[u32;6]=[0xb412e6d8,0x026379ab,0x43faa517,0x3c5b0549,0x6ab4825a,0x0426e81d];
for want in targets{
let mut shown=false;
for e in arc.entries(){
if shown{break;}
let Ok(b)=arc.read(e) else{continue};
if b.len()<12 || &b[0..4]!=b"IDXD"{continue;}
let s=u32::from_be_bytes([b[8],b[9],b[10],b[11]]);
if s!=want{continue;}
if let Ok(o)=IdxdObject::parse(&b){
let t=o.tokens();
let sample:Vec<String>=t.iter().take(14).map(|s|{let s=s.chars().take(18).collect::<String>();s}).collect();
println!(" {want:08x}: {sample:?}");
shown=true;
}
}
if !shown{ println!(" {want:08x}: (parse failed / not found)"); }
}
}

View File

@@ -0,0 +1,50 @@
//! Scratch analysis: for one IDXD key, list every object that DECLARES it and
//! whether it carries a value on disc or is left at the title-code default.
//!
//! Run: cargo run -p sylpheed-formats --example default_owners -- <KEY> [KEY...]
use sylpheed_formats::idxd::IdxdObject;
use sylpheed_formats::pak::PakArchive;
fn is_number(s: &str) -> bool {
let s = s.strip_suffix(['f', 'F']).unwrap_or(s);
let t = s.strip_prefix(['-', '+']).unwrap_or(s);
!t.is_empty() && t.chars().all(|c| c.is_ascii_digit() || c == '.')
}
fn is_key(s: &str) -> bool {
!s.is_empty()
&& s.chars().all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
&& s.chars().next().is_some_and(|c| c.is_ascii_alphabetic() || c == '_')
&& !is_number(s)
}
fn is_value(s: &str) -> bool {
is_number(s) || !is_key(s)
}
fn main() {
let root = std::env::var("SYLPHEED_DISC")
.unwrap_or_else(|_| "/home/fabi/RE - Project Sylpheed/sylph_extract".into());
let wanted: Vec<String> = std::env::args().skip(1).collect();
let arc = PakArchive::open(std::path::Path::new(&root).join("dat/GP_MAIN_GAME_E.pak")).unwrap();
for e in arc.entries() {
let Ok(bytes) = arc.read(e) else { continue };
let Ok(obj) = IdxdObject::parse(&bytes) else { continue };
let toks = obj.tokens().to_vec();
let mut hits: Vec<String> = vec![];
for (i, t) in toks.iter().enumerate() {
if !wanted.iter().any(|w| w == t) {
continue;
}
let valued = i > 0 && is_value(&toks[i - 1]);
hits.push(if valued {
format!("{t}={}", toks[i - 1])
} else {
format!("{t}=<DEFAULT>")
});
}
if !hits.is_empty() {
println!("0x{:08x} {:<44} {}", obj.schema_hash, obj.identity(), hits.join(" "));
}
}
}

View File

@@ -0,0 +1,135 @@
//! Scratch analysis: which IDXD fields are DECLARED but left at their default on
//! disc, per schema. Those defaults live in title code, so they can only be read
//! from the running game — this prints the shopping list for that dynamic capture.
//!
//! Run: cargo run -p sylpheed-formats --example defaulted_fields -- <disc-root>
use std::collections::{BTreeMap, BTreeSet};
use sylpheed_formats::idxd::IdxdObject;
use sylpheed_formats::pak::PakArchive;
fn is_number(s: &str) -> bool {
let s = s.strip_suffix(['f', 'F']).unwrap_or(s);
let t = s.strip_prefix(['-', '+']).unwrap_or(s);
!t.is_empty() && t.chars().all(|c| c.is_ascii_digit() || c == '.')
}
/// Same shape-test the parser uses: an identifier-looking token that is not a
/// value literal is a field-name key.
fn is_key(s: &str) -> bool {
!s.is_empty()
&& s.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
&& s.chars().next().is_some_and(|c| c.is_ascii_alphabetic() || c == '_')
&& !is_number(s)
}
fn is_value(s: &str) -> bool {
is_number(s) || !is_key(s)
}
fn main() {
let root = std::env::args()
.nth(1)
.unwrap_or_else(|| "/home/fabi/RE - Project Sylpheed/sylph_extract".into());
let paks: Vec<String> = std::env::args().skip(2).collect();
let paks = if paks.is_empty() {
vec![
"dat/GP_MAIN_GAME_E.pak".to_string(),
"dat/GP_HANGAR_ARSENAL.pak".to_string(),
]
} else {
paks
};
for pak_rel in &paks {
let path = std::path::Path::new(&root).join(pak_rel);
let Ok(arc) = PakArchive::open(&path) else {
eprintln!("-- skip {pak_rel} (open failed)");
continue;
};
println!("\n================ {pak_rel} ================");
// schema -> key -> (n_set, n_defaulted, distinct values, example owners)
type Stat = (usize, usize, BTreeSet<String>, Vec<String>);
let mut per_schema: BTreeMap<u32, (usize, BTreeMap<String, Stat>)> = BTreeMap::new();
for e in arc.entries() {
let Ok(bytes) = arc.read(e) else { continue };
let Ok(obj) = IdxdObject::parse(&bytes) else {
continue;
};
let ident = obj.identity();
let toks = obj.tokens().to_vec();
let entry = per_schema.entry(obj.schema_hash).or_default();
entry.0 += 1;
// Track which keys this object declares, and whether each is valued.
let mut seen_here: BTreeMap<String, Option<String>> = BTreeMap::new();
for (i, t) in toks.iter().enumerate() {
if !is_key(t) {
continue;
}
let prev = if i == 0 { None } else { Some(&toks[i - 1]) };
let valued = prev.map(|p| is_value(p)).unwrap_or(false);
let v = if valued { Some(toks[i - 1].clone()) } else { None };
seen_here.entry(t.clone()).or_insert(v);
}
for (k, v) in seen_here {
let s = entry.1.entry(k).or_default();
match v {
Some(val) => {
s.0 += 1;
if s.2.len() < 12 {
s.2.insert(val);
}
}
None => {
s.1 += 1;
if s.3.len() < 6 {
s.3.push(ident.clone());
}
}
}
}
}
for (schema, (n_obj, keys)) in per_schema {
if n_obj < 2 {
continue;
}
let name = match schema {
0x0426_e81d => "PLAYER",
0x6ab4_825a => "WEAPON",
0x43fa_a517 => "UNIT",
0x3c5b_0549 => "VESSEL",
0xbd86_d41c => "CHARACTER",
0x3c9a_e32e => "STAGE",
0xb412_e6d8 => "MESSAGE",
_ => "?",
};
let defaulted: Vec<_> = keys
.iter()
.filter(|(_, s)| s.1 > 0)
.collect();
if defaulted.is_empty() {
continue;
}
println!("\n--- schema 0x{schema:08x} {name} ({n_obj} objects) ---");
println!(
"{:<30} {:>5} {:>5} {}",
"KEY", "set", "dflt", "values seen (≤12) | owners defaulting"
);
for (k, (n_set, n_def, vals, owners)) in defaulted {
let vv: Vec<&str> = vals.iter().map(|s| s.as_str()).collect();
println!(
"{:<30} {:>5} {:>5} {} | {}",
k,
n_set,
n_def,
vv.join(","),
owners.join(",")
);
}
}
}
}

View File

@@ -0,0 +1,13 @@
use sylpheed_formats::{game_data, localization::TextIndex, PakArchive};
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let pak=PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")).unwrap();
let text=TextIndex::build(&pak);
let msgs=game_data::load_demo_messages(&pak);
println!("{} dialogue lines total\n", msgs.len());
for m in msgs.iter().filter(|m|m.character.is_some()&&!m.page_keys.is_empty()).take(8){
let who=m.character.as_deref().unwrap_or("?").trim_start_matches("Character");
let line:String=m.page_keys.iter().filter_map(|k|text.get(k)).collect::<Vec<_>>().join(" ");
println!(" {who:10} [{}] “{}", m.voice_clip.as_deref().unwrap_or("-"), line.chars().take(64).collect::<String>());
}
}

View File

@@ -0,0 +1,29 @@
use sylpheed_formats::{idxd::IdxdObject, PakArchive};
use std::collections::BTreeMap;
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let arc=PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")).unwrap();
let tables:[(u32,&str);4]=[(0x0426e81d,"Player / physics+scoring"),(0x6ab4825a,"Weapon"),(0x43faa517,"Unit / craft"),(0x3c5b0549,"Vessel / capital ship")];
for (want,label) in tables{
// collect all records of this schema
let mut recs:Vec<IdxdObject>=vec![];
for e in arc.entries(){ let Ok(b)=arc.read(e) else{continue}; let Ok(o)=IdxdObject::parse(&b) else{continue}; if o.schema_hash==want{recs.push(o);} }
// field-union (explicit-valued keys), with occurrence count
let mut cols:BTreeMap<String,u32>=BTreeMap::new();
for o in &recs{ for (k,_) in o.resolved_fields(){ *cols.entry(k.into()).or_default()+=1; } }
println!("\n╔══ {label} [{want:08x}] {} records ══", recs.len());
let mut cv:Vec<_>=cols.into_iter().collect(); cv.sort_by_key(|x|std::cmp::Reverse(x.1));
let colstr:String=cv.iter().take(28).map(|(k,c)|format!("{k}({c})")).collect::<Vec<_>>().join(" ");
println!("║ numeric/enum fields: {colstr}");
// dump 2 sample records: identity + explicit fields
for o in recs.iter().take(2){
let id=o.get_raw("ID").unwrap_or("?");
let name=o.get_raw("Name").unwrap_or("");
print!("║ • {id}");
if !name.is_empty(){print!(" «{name}»");}
println!();
let fields:Vec<String>=o.resolved_fields().iter().map(|(k,v)|format!("{k}={v}")).collect();
for chunk in fields.chunks(5){ println!("{}", chunk.join(" ")); }
}
}
}

View File

@@ -0,0 +1,11 @@
use sylpheed_formats::{game_data, PakArchive};
use std::collections::BTreeMap;
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let pak=PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")).unwrap();
let chars=game_data::load_characters(&pak);
let mut byfac:BTreeMap<String,Vec<String>>=Default::default();
for c in &chars{ byfac.entry(c.faction.clone().unwrap_or("·".into())).or_default().push(format!("{}({})",c.id.clone().unwrap_or_default().trim_start_matches("Character"),c.faces.len())); }
println!("{} characters across {} factions:",chars.len(),byfac.len());
for (f,mut v) in byfac{ v.sort(); println!(" [{f}] {}", v.join(" ")); }
}

View File

@@ -0,0 +1,14 @@
use sylpheed_formats::{idxd::IdxdObject, PakArchive};
fn short(s:&str)->String{ s.trim_start_matches("Weapon_").trim_start_matches("UN_").trim_start_matches("UnitName_UN_").into() }
fn g<'a>(o:&'a IdxdObject,k:&str)->String{ o.get_f32(k).map(|v|{if v==v.trunc(){format!("{}",v as i64)}else{format!("{v}")}}).or_else(||o.get_raw(k).filter(|x|x.chars().next().map(|c|c.is_ascii_digit()).unwrap_or(false)).map(|s|s.to_string())).unwrap_or("·".into()) }
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let arc=PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")).unwrap();
let recs=|want:u32|->Vec<IdxdObject>{ arc.entries().iter().filter_map(|e|arc.read(e).ok()).filter_map(|b|IdxdObject::parse(&b).ok()).filter(|o|o.schema_hash==want).collect() };
println!("### WEAPONS (name | target | load | int | power | vel | range | trig)");
for o in recs(0x6ab4825a).iter().take(16){ println!("{:<34}\t{}\t{}\t{}\t{}\t{}\t{}\t{}", short(o.get_raw("ID").unwrap_or("?")), o.get_raw("TargetType").unwrap_or("·"), g(o,"LoadingCount"),g(o,"Interval"),g(o,"Power"),g(o,"Velocity"),g(o,"MaximumRange"),g(o,"TriggerShotCount")); }
println!("\n### UNITS/CRAFT (name | HP | cruise | accel | radar | turrets | score)");
for o in recs(0x43faa517).iter().take(16){ println!("{:<38}\t{}\t{}\t{}\t{}\t{}\t{}", short(o.get_raw("ID").unwrap_or("?")), g(o,"HP"),g(o,"CruisingVelocity"),g(o,"Acceleration"),g(o,"RadarRange"),g(o,"TurretCount"),g(o,"ScorePoint")); }
println!("\n### VESSELS/CAPITAL SHIPS (name | HP | Sz_X | Sz_Z | radar | turrets | bridges | hatches | shieldgen | thrusters)");
for o in recs(0x3c5b0549).iter(){ println!("{:<30}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}", short(o.get_raw("ID").unwrap_or("?")), g(o,"HP"),g(o,"Size_X"),g(o,"Size_Z"),g(o,"RadarRange"),g(o,"TurretCount"),g(o,"BridgeCount"),g(o,"HatchCount"),g(o,"ShieldGeneratorCount"),g(o,"ThrusterCount")); }
}

View File

@@ -0,0 +1,15 @@
use sylpheed_formats::{game_data, localization::TextIndex, PakArchive};
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let pak=PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")).unwrap();
let text=TextIndex::build(&pak);
for n in 1..=16u32{
let sid=format!("S{n:02}");
// primary objective across phases (first that resolves)
let obj:Vec<String>=(1..=3).flat_map(|p|text.objectives(&sid,p)).map(|s|s.to_string()).collect();
let lose:Vec<String>=(1..=3).flat_map(|p|text.lose_conditions(&sid,p)).map(|s|s.to_string()).collect();
let full_obj=obj.join(" ");
let full_lose=lose.into_iter().take(2).collect::<Vec<_>>().join(" ");
println!("{sid}\t{full_obj}\t{full_lose}");
}
}

View File

@@ -0,0 +1,17 @@
use sylpheed_formats::{game_data as gd, PakArchive};
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let pak=PakArchive::open(format!("{disc}/dat/GP_HANGAR_ARSENAL.pak")).unwrap();
let rosters=gd::load_pilot_rosters(&pak);
// distinct rosters by their pilot set
let mut seen=std::collections::BTreeSet::new(); let mut shown=0;
println!("{} pilot-roster configs; distinct line-ups:", rosters.len());
for r in &rosters{
let key:String=r.pilots.iter().map(|(c,p)|format!("{c}:{p}")).collect::<Vec<_>>().join(",");
if seen.insert(key) && shown<8 {
shown+=1;
let flt:Vec<String>=r.pilots.iter().map(|(c,p)|format!("{c}={p}")).collect();
println!(" {}", flt.join(" "));
}
}
}

View File

@@ -0,0 +1,43 @@
use sylpheed_formats::{hash::name_hash, movie_manifest, movie_subtitle as ms, movie_voice, slb, PakArchive};
use std::fs;use std::io::{Read,Seek,SeekFrom};use std::process::Command;
fn rg(disc:&str,g:u64,n:usize)->Vec<u8>{let mut segs=vec![];let mut cum=0u64;for i in 0..5{let p=format!("{disc}/dat/sound.p{i:02}");if let Ok(m)=fs::metadata(&p){segs.push((cum,m.len(),p));cum+=m.len();}}let mut out=vec![];let(mut need,mut pos)=(n,g);for(base,len,path)in &segs{if need==0||pos>=base+len||pos<*base{continue;}let local=pos-base;let take=need.min((len-local)as usize);let mut f=fs::File::open(path).unwrap();f.seek(SeekFrom::Start(local)).unwrap();let mut b=vec![0u8;take];f.read_exact(&mut b).unwrap();out.extend_from_slice(&b);need-=take;pos+=take as u64;}out}
fn ct(h:&[u8],n:&[u8])->bool{h.windows(n.len()).any(|w|w==n)}
fn dur(w:&str)->String{let o=Command::new("ffprobe").args(["-v","error","-show_entries","format=duration","-of","default=nw=1:nk=1",w]).output().unwrap();String::from_utf8_lossy(&o.stdout).trim().to_string()}
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let tpak=PakArchive::open(format!("{disc}/dat/tables.pak")).unwrap();
let man=tpak.entries().iter().find_map(|e|tpak.read(e).ok().filter(|b|movie_manifest::is_manifest(b))).unwrap();
let lpak=PakArchive::open(format!("{disc}/dat/movie/eng.pak")).unwrap();
let reg=tpak.entries().iter().find_map(|e|tpak.read(e).ok().filter(|b|ct(b,b"eng\\Movie\\VOICE_ADV.slb"))).unwrap();
let ids=movie_voice::registry_voice_ids(&reg);
let stoc=fs::read(format!("{disc}/dat/sound.pak")).unwrap();
let ents=PakArchive::parse_toc(&stoc).unwrap();
// demo->token from bound hokyu
let hok:Vec<_>=movie_manifest::parse(&man).into_iter().filter(|m|m.movie.starts_with("hokyu_")).collect();
let resolve=|movie:&str|->Option<String>{
movie_manifest::voice_token(&man,movie).or_else(||{
let want=ms::track_voice_cues(&lpak,movie).first().map(|&(d,_)|d)?;
hok.iter().find_map(|e|{let t=e.voice_token.clone().filter(|_|e.movie.starts_with("hokyu_"))?; ms::track_voice_cues(&lpak,&e.movie).iter().any(|&(d,_)|d==want).then_some(t)})
})
};
for e in &hok{
let mv=&e.movie;
let bound=e.voice_token.is_some();
let tok=resolve(mv);
let demo=ms::track_voice_cues(&lpak,mv).first().map(|&(d,_)|d);
let mut d="".to_string();
if let Some(tok)=&tok{
if let Some(&id)=ids.get(tok){
if let Some(anchor)=["Movie","etc","Voice"].iter().find_map(|dir|{let h=name_hash(&format!("eng\\{dir}\\{tok}.slb"));ents.binary_search_by_key(&h,|x|x.name_hash).ok().map(|i|ents[i].offset as u64)}){
let ws=(anchor.saturating_sub(2*1024*1024))&!3; let win=rg(&disc,ws,8*1024*1024);
if let Some(el)=movie_voice::find_descriptor(&win,id){let end=ws+el as u64;
let start=movie_voice::find_descriptor(&win,id.wrapping_sub(1)).or_else(||movie_voice::find_descriptor_before(&win,el)).map(|o|ws+o as u64).filter(|&s|s<end&&end-s<1_500_000).unwrap_or(anchor);
let region=rg(&disc,start,(end-start)as usize); let mut rf=slb::to_xma_riffs(&region); if rf.is_empty(){rf=slb::to_xma_riff_best(&region).into_iter().collect();}
if let Some(r)=rf.first(){let xp=format!("/tmp/hd_{mv}.xma.wav");let wp=format!("/tmp/hd_{mv}.wav");fs::write(&xp,r).unwrap();let _=Command::new("ffmpeg").args(["-hide_banner","-v","error","-y","-i",&xp,&wp]).status();d=dur(&wp);}
}
}
}
}
println!("{mv:16} {:8} demo={:?} -> {:11} voice={d}s", if bound{"BOUND"}else{"unbound"}, demo, tok.unwrap_or("(silent)".into()));
}
}

View File

@@ -0,0 +1,36 @@
use sylpheed_formats::{hash::name_hash, movie_manifest, movie_voice, slb, PakArchive};
use std::fs;use std::io::{Read,Seek,SeekFrom};use std::process::Command;
fn rg(disc:&str,g:u64,n:usize)->Vec<u8>{let mut segs=vec![];let mut cum=0u64;for i in 0..5{let p=format!("{disc}/dat/sound.p{i:02}");if let Ok(m)=fs::metadata(&p){segs.push((cum,m.len(),p));cum+=m.len();}}let mut out=vec![];let(mut need,mut pos)=(n,g);for(base,len,path)in &segs{if need==0||pos>=base+len||pos<*base{continue;}let local=pos-base;let take=need.min((len-local)as usize);let mut f=fs::File::open(path).unwrap();f.seek(SeekFrom::Start(local)).unwrap();let mut b=vec![0u8;take];f.read_exact(&mut b).unwrap();out.extend_from_slice(&b);need-=take;pos+=take as u64;}out}
fn contains(h:&[u8],n:&[u8])->bool{h.windows(n.len()).any(|w|w==n)}
fn dur(w:&str)->String{let o=Command::new("ffprobe").args(["-v","error","-show_entries","format=duration","-of","default=nw=1:nk=1",w]).output().unwrap();String::from_utf8_lossy(&o.stdout).trim().to_string()}
fn hokyu_fallback(m:&str)->Option<String>{if !m.starts_with("hokyu_"){return None}let ls=m.contains("_LS_");let ds=m.contains("_DS_");let a=m.ends_with('A');let h=m.ends_with('H');Some(match(ls,ds,a,h){(true,_,true,_)=>"VOICE_D_450",(true,_,_,true)=>"VOICE_D_453",(_,true,true,_)=>"VOICE_D_452",(_,true,_,true)=>"VOICE_D_454",_=>return None}.into())}
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let tpak=PakArchive::open(format!("{disc}/dat/tables.pak")).unwrap();
let manifest=tpak.entries().iter().find_map(|e|tpak.read(e).ok().filter(|b|movie_manifest::is_manifest(b))).unwrap();
let code="eng";
let registry=tpak.entries().iter().find_map(|e|tpak.read(e).ok().filter(|b|contains(b,format!("{code}\\Movie\\VOICE_ADV.slb").as_bytes()))).unwrap();
let ids=movie_voice::registry_voice_ids(&registry);
let stoc=fs::read(format!("{disc}/dat/sound.pak")).unwrap();
let entries=PakArchive::parse_toc(&stoc).unwrap();
let hokyu:Vec<String>=movie_manifest::parse(&manifest).into_iter().filter(|m|m.movie.starts_with("hokyu_")).map(|m|m.movie).collect();
for movie in &hokyu{
let bound=movie_manifest::voice_token(&manifest,movie);
let token=bound.clone().or_else(||hokyu_fallback(movie));
let Some(token)=token else{println!("{movie:16} NO TOKEN"); continue};
let src = if bound.is_some(){"bound"}else{"fallback"};
let Some(&id)=ids.get(&token) else{println!("{movie:16} {token} not in registry");continue};
let Some(anchor)=["Movie","etc","Voice"].iter().find_map(|d|{let h=name_hash(&format!("{code}\\{d}\\{token}.slb"));entries.binary_search_by_key(&h,|e|e.name_hash).ok().map(|i|entries[i].offset as u64)}) else{println!("{movie:16} no anchor");continue};
let win_start=(anchor.saturating_sub(2*1024*1024))&!3;
let window=rg(&disc,win_start,8*1024*1024);
let Some(el)=movie_voice::find_descriptor(&window,id) else{println!("{movie:16} desc not found");continue};
let end=win_start+el as u64;
let start=movie_voice::find_descriptor(&window,id.wrapping_sub(1)).or_else(||movie_voice::find_descriptor_before(&window,el)).map(|o|win_start+o as u64).filter(|&s|s<end&&end-s<1_500_000).unwrap_or(anchor);
let region=rg(&disc,start,(end-start)as usize);
let mut riffs=slb::to_xma_riffs(&region); if riffs.is_empty(){riffs=slb::to_xma_riff_best(&region).into_iter().collect();}
let mut d="?".into();
if let Some(r)=riffs.first(){let xp=format!("/tmp/hf_{movie}.xma.wav");let wp=format!("/tmp/hf_{movie}.wav");fs::write(&xp,r).unwrap();let _=Command::new("ffmpeg").args(["-hide_banner","-v","error","-y","-i",&xp,&wp]).status();d=dur(&wp);}
let mv=dur(&format!("{disc}/dat/movie/{movie}.wmv"));
println!("{movie:16} {src:8} {token:12} id={id} voice={d}s movie={mv}s");
}
}

View File

@@ -0,0 +1,108 @@
//! RE diagnostic: emit every `weapon\Weapon_*.tbl` in a pak as machine-readable
//! records, split at the sub-object boundaries the schema declares.
//!
//! An IDXD `.tbl` for a weapon holds `count` sub-records — a `Weapon` and its
//! `Shell` — flattened into one string pool. `sylpheed-cli pak dump` shows them
//! merged, which is ambiguous (both declare `ID`/`Name`). The title's schema
//! name pool gives the split point: the pool contains the literal type names
//! (`Weapon`, `Shell`), and each sub-record's fields follow its type name.
//!
//! Here we split on a type-name token appearing as a *key* position, so each
//! record is emitted with its own ID and field set:
//!
//! ```text
//! REC <tbl-hash> <index> <TypeName> <ID>
//! F <key> <value>
//! ```
//!
//! Run: cargo run --release -p sylpheed-formats --example idxd_tokens -- <pak> [types...]
use sylpheed_formats::idxd::IdxdObject;
use sylpheed_formats::pak::PakArchive;
fn is_number(s: &str) -> bool {
let s = s.strip_suffix(['f', 'F']).unwrap_or(s);
let t = s.strip_prefix(['-', '+']).unwrap_or(s);
!t.is_empty() && t.chars().all(|c| c.is_ascii_digit() || c == '.')
}
/// `Yes`/`No`-style scalar literals are values, not field names, even though
/// they are identifier-shaped.
fn is_enum_value(s: &str) -> bool {
matches!(s, "Yes" | "No" | "YES" | "NO" | "On" | "Off" | "ON" | "OFF")
}
fn is_key_like(s: &str) -> bool {
!s.is_empty()
&& s.chars().all(|c| c.is_ascii_alphanumeric() || c == '_')
&& s.chars().next().is_some_and(|c| c.is_ascii_alphabetic() || c == '_')
&& !is_number(s)
&& !is_enum_value(s)
}
fn main() {
let mut args = std::env::args().skip(1);
let pak_path = args.next().expect("usage: idxd_tokens <pak> [TypeName...]");
let types: Vec<String> = {
let v: Vec<String> = args.collect();
if v.is_empty() {
vec!["Weapon".into(), "Shell".into()]
} else {
v
}
};
let pak = PakArchive::open(&pak_path).expect("open pak");
for entry in pak.entries() {
let Ok(bytes) = pak.read(entry) else { continue };
let Ok(obj) = IdxdObject::parse(&bytes) else { continue };
let toks = obj.tokens();
// Only entries that actually declare one of the requested sub-record
// types (the pool-start heuristic can prefix one stray byte, so match a
// short suffix rather than equality -- same rule as the splitter below).
if !toks
.iter()
.any(|t| types.iter().any(|ty| t == ty || (t.ends_with(ty.as_str()) && t.len() <= ty.len() + 2)))
{
continue;
}
if std::env::var("SYLPH_RAW").is_ok() {
println!("RAW {:08x}", entry.name_hash);
for t in toks {
println!("T {t}");
}
}
// Sub-record boundaries: a bare type-name token. The first one is the
// schema's own declaration (`EnumWeapon`-style header lives in title
// code, not here), so we take every occurrence in order.
let mut bounds: Vec<(usize, &str)> = Vec::new();
for (i, t) in toks.iter().enumerate() {
// The pool-start heuristic can glue one stray binary byte onto the
// first token ("YWeapon"), so match a short suffix, not equality.
if let Some(ty) = types
.iter()
.find(|ty| t == *ty || (t.ends_with(ty.as_str()) && t.len() <= ty.len() + 2))
{
bounds.push((i, ty.as_str()));
}
}
for (n, &(start, ty)) in bounds.iter().enumerate() {
let end = bounds.get(n + 1).map(|b| b.0).unwrap_or(toks.len());
let slice = &toks[start..end];
// The record's own ID value is the token straight after its type
// name (`Shell` -> `Shell_DSaber_P_wep_01_Beam`). The `ID`/`Name`
// *keys* are interned once in the pool, so only the first record
// shows them -- position-of-key lookup would miss the rest.
let id = slice.get(1).map(|s| s.as_str()).unwrap_or("?");
println!("REC {:08x} {n} {ty} {id}", entry.name_hash);
for i in 1..slice.len() {
let (k, v) = (slice[i].as_str(), slice[i - 1].as_str());
if is_key_like(k) && (!is_key_like(v) || is_number(v)) {
println!("F {k} {v}");
}
}
}
}
}

View File

@@ -0,0 +1,50 @@
use sylpheed_formats::PakArchive;
use std::collections::BTreeMap;
fn be32(b:&[u8],o:usize)->u32{ if o+4<=b.len(){u32::from_be_bytes([b[o],b[o+1],b[o+2],b[o+3]])}else{0} }
fn magic(b:&[u8])->String{
if b.len()<4 {return "(<4B)".into();}
let m=&b[0..4];
if m.iter().all(|&c|(0x20..0x7f).contains(&c)){ String::from_utf8_lossy(m).into() }
else if m==b"\x89PNG"{"PNG".into()} else if m==[0,1,0,0]{"ttf".into()}
else { format!("{:02x}{:02x}{:02x}{:02x}",m[0],m[1],m[2],m[3]) }
}
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
// Known/understood IDXD schemas (semantic parsers we have)
let known_schema:BTreeMap<u32,&str>=[(0x067025b9,"ADVERTISE_MOVIE (movie_manifest)"),(0x13cb84ba,"sound registry / sounds.tbl")].into();
let mut fmt_count:BTreeMap<String,(u64,u64)>=BTreeMap::new(); // fmt -> (count, bytes)
let mut schema_census:BTreeMap<u32,(u64,u64,String)>=BTreeMap::new(); // schema -> (count,bytes,sample pak)
let mut unknown_magics:BTreeMap<String,(u64,Vec<String>)>=BTreeMap::new();
let paks:Vec<String>={let mut v=vec![]; for e in std::fs::read_dir(format!("{disc}/dat")).unwrap(){let p=e.unwrap().path(); if p.extension().map(|x|x=="pak").unwrap_or(false){v.push(p.file_stem().unwrap().to_string_lossy().into());}} v.sort(); v};
println!("pak entries decompressed formats");
for pk in &paks{
let Ok(arc)=PakArchive::open(format!("{disc}/dat/{pk}.pak")) else{continue};
let mut per:BTreeMap<String,u64>=BTreeMap::new();
let mut tot=0u64; let n=arc.entries().len();
for e in arc.entries(){
let Ok(b)=arc.read(e) else{continue};
tot+=b.len() as u64;
let mut f=magic(&b);
if f=="IDXD"{ let s=be32(&b,8); schema_census.entry(s).or_insert((0,0,pk.clone())).0+=1; schema_census.get_mut(&s).unwrap().1+=b.len() as u64; f=format!("IDXD:{s:08x}"); }
else if !["XPR2","IXUD","LSTA","RATC","RIFF","XBG7","OTTO","PNG","ttf","DDS ","T8AD"].contains(&f.as_str()){
let ent=unknown_magics.entry(f.clone()).or_insert((0,vec![])); ent.0+=1; if ent.1.len()<3 && !ent.1.contains(pk){ent.1.push(pk.clone());}
}
*per.entry(if f.starts_with("IDXD:"){"IDXD".into()}else{f.clone()}).or_default()+=1;
let g=fmt_count.entry(if f.starts_with("IDXD:"){"IDXD".into()}else{f}).or_insert((0,0)); g.0+=1; g.1+=b.len() as u64;
}
let mut fs:Vec<_>=per.into_iter().collect(); fs.sort_by_key(|x|std::cmp::Reverse(x.1));
let top:String=fs.iter().take(4).map(|(k,v)|format!("{k}×{v}")).collect::<Vec<_>>().join(" ");
println!("{pk:26} {n:>7} {:>10}KB {top}", tot/1024);
}
println!("\n=== FORMAT TOTALS (across all paks) ===");
let mut fv:Vec<_>=fmt_count.into_iter().collect(); fv.sort_by_key(|x|std::cmp::Reverse(x.1.1));
for (f,(c,b)) in &fv{ println!(" {f:12} {c:>6} entries {:>8}KB", b/1024); }
println!("\n=== IDXD SCHEMA CENSUS ({} distinct schemas) ===", schema_census.len());
let mut sv:Vec<_>=schema_census.into_iter().collect(); sv.sort_by_key(|x|std::cmp::Reverse(x.1.1));
for (s,(c,b,pk)) in &sv{
let tag=known_schema.get(s).copied().unwrap_or("??? UNDECODED");
println!(" {s:08x} {c:>5} ent {:>7}KB e.g. {pk:22} {tag}", b/1024);
}
println!("\n=== UNKNOWN / UNCLASSIFIED MAGICS ===");
for (m,(c,pks)) in &unknown_magics{ println!(" {m:12} ×{c:<5} in {pks:?}"); }
}

View File

@@ -0,0 +1,46 @@
//! Validation: dump the parsed mission/phase cutscene map from the real manifest.
use sylpheed_formats::movie_manifest::{self, MovieKind};
use sylpheed_formats::pak::PakArchive;
fn main() {
let disc = std::env::var("SYLPHEED_DISC").expect("set SYLPHEED_DISC");
let arc = PakArchive::open(format!("{disc}/dat/tables.pak")).unwrap();
let man = arc
.entries()
.iter()
.find_map(|e| arc.read(e).ok().filter(|b| movie_manifest::is_manifest(b)))
.expect("manifest");
let entries = movie_manifest::parse(&man);
let mut counts = [0usize; 5];
for e in &entries {
counts[match e.kind {
MovieKind::System => 0,
MovieKind::Intro => 1,
MovieKind::Phase => 2,
MovieKind::PhaseEnd => 3,
MovieKind::Supply => 4,
}] += 1;
let m = e.mission.map(|x| x.to_string()).unwrap_or_default();
let p = e.phase.map(|x| x.to_string()).unwrap_or_default();
println!(
"{:<22} {:<9} m={:<2} p={:<1} {:<16} {:<14} tel={:<14} sub={}",
e.slot,
format!("{:?}", e.kind),
m,
p,
e.movie,
e.voice_token.as_deref().unwrap_or("-"),
e.telop.as_deref().unwrap_or("-"),
e.subtitle.as_deref().unwrap_or("-"),
);
}
eprintln!(
"TOTAL {} entries — System={} Intro={} Phase={} PhaseEnd={} Supply={}",
entries.len(),
counts[0],
counts[1],
counts[2],
counts[3],
counts[4]
);
}

View File

@@ -0,0 +1,47 @@
use sylpheed_formats::{idxd::IdxdObject, PakArchive};
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let arc=PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")).unwrap();
// 1. All stages: BackGroundID + phases + stage tag (from EnumUnit_SNN)
println!("=== STAGES (StageResource 3c9ae32e) ===");
let mut rows=vec![];
for e in arc.entries(){
let Ok(b)=arc.read(e) else{continue}; let Ok(o)=IdxdObject::parse(&b) else{continue};
if o.schema_hash!=0x3c9ae32e {continue;}
let t=o.tokens();
let bg=o.get_raw("BackGroundID").unwrap_or("?").to_string();
let stage=t.iter().find_map(|s|s.strip_prefix("EnumUnit_").map(|x|x.trim_end_matches(".tbl").to_string())).unwrap_or("?".into());
let phases=t.iter().filter(|s|s.starts_with("Phase_")).count();
let unitgrp=t.iter().any(|s|s.starts_with("UnitGroup_"));
let route=t.iter().any(|s|s.starts_with("Route_"));
let formation=t.iter().any(|s|s.starts_with("FormationSet_"));
rows.push((stage,bg,phases,unitgrp,route,formation));
}
rows.sort();
for (s,bg,p,ug,rt,fm) in &rows{ println!(" {s:8} location={bg:14} phases={p} [squadrons:{} route:{} formations:{}]", if *ug{"Y"}else{"·"},if *rt{"Y"}else{"·"},if *fm{"Y"}else{"·"}); }
// 2. Parse objectives table into (stage,phase) -> key counts
println!("\n=== OBJECTIVES (033b5b7e) — per stage/phase key groups ===");
for e in arc.entries(){
let Ok(b)=arc.read(e) else{continue}; let Ok(o)=IdxdObject::parse(&b) else{continue};
if o.schema_hash!=0x033b5b7e {continue;}
let t=o.tokens();
let mut groups:std::collections::BTreeMap<String,u32>=Default::default();
for tok in t{ if let Some(p)=tok.find("_Objective_").or_else(||tok.find("_Lose_")).or_else(||tok.find("_Hint_")){ let key=&tok[..p]; *groups.entry(key.trim_start_matches(|c:char|!c.is_ascii_alphabetic()).into()).or_default()+=1; } }
let stages:std::collections::BTreeSet<String>=groups.keys().filter_map(|k|k.get(..3).map(|s|s.to_string())).collect();
println!(" covers {} stages: {:?}", stages.len(), stages);
println!(" sample S01_P1 group counts: {:?}", groups.iter().filter(|(k,_)|k.starts_with("S01_P1")).collect::<Vec<_>>());
break;
}
// 3. Resolve objective TEXT: search all pak entries for the key "S01_P1_Objective_00" as an ID with English text
println!("\n=== OBJECTIVE TEXT resolution probe ===");
for e in arc.entries(){
let Ok(b)=arc.read(e) else{continue}; let Ok(o)=IdxdObject::parse(&b) else{continue};
let t=o.tokens();
if let Some(i)=t.iter().position(|s|s=="S01_P1_Objective_00"){
let lo=i.saturating_sub(1); let hi=(i+3).min(t.len());
println!(" schema {:08x}: ...{:?}...", o.schema_hash, &t[lo..hi]);
}
}
}

View File

@@ -0,0 +1,89 @@
//! RE diagnostic: raw-dump a composite scene graph's node records — name, table
//! pointers, and the FULL joint-table slot values (not just the 8 the decoder
//! currently reads) — to derive the true TRS slot layout against captured
//! ground truth.
//! cargo run --release --example node_dump -- <Stage.xpr path> <composite name> [slots]
use sylpheed_formats::mesh::xbg7_descriptor_range;
fn be32(d: &[u8], o: usize) -> u32 {
if o + 4 > d.len() {
return 0;
}
u32::from_be_bytes([d[o], d[o + 1], d[o + 2], d[o + 3]])
}
fn be_f64(d: &[u8], o: usize) -> f64 {
if o + 8 > d.len() {
return f64::NAN;
}
f64::from_be_bytes(d[o..o + 8].try_into().unwrap())
}
fn main() {
let a: Vec<String> = std::env::args().collect();
let bytes = std::fs::read(&a[1]).unwrap();
let comp = &a[2];
let nslots: usize = a.get(3).and_then(|s| s.parse().ok()).unwrap_or(12);
let (desc, desc_end) = xbg7_descriptor_range(&bytes, comp).expect("composite found");
let d = &bytes[desc..desc_end];
println!("descriptor [{desc:#x}..{desc_end:#x}) len {:#x}", d.len());
let mut i = 0usize;
while i + 3 <= d.len() {
let starts = (i + 4 <= d.len() && &d[i..i + 4] == b"rou_")
|| (i + 3 <= d.len() && &d[i..i + 3] == b"GN_");
if !starts {
i += 1;
continue;
}
let mut j = i;
while j < d.len() && d[j] != 0 && d[j].is_ascii_graphic() {
j += 1;
}
let name = std::str::from_utf8(&d[i..j]).unwrap_or("?").to_string();
if name.ends_with("_col") || name.ends_with("_spc") || name.ends_with("_gls") || name.ends_with("_lum") {
i = j.max(i + 1);
continue;
}
// find the 0xFFFFFFFF end marker
let mut ff = i;
let scan_end = (i + 0x90).min(d.len().saturating_sub(4));
while ff < scan_end && be32(d, ff) != 0xFFFF_FFFF {
ff += 4;
}
if ff >= scan_end || ff < 0x10 {
i = j.max(i + 1);
continue;
}
let t44 = be32(d, ff - 0x10) as usize;
let t48 = be32(d, ff - 0x0C) as usize;
let child = be32(d, ff - 0x08) as usize;
let sib = be32(d, ff - 0x04) as usize;
// Dump the record's u32s between name end and marker for structure context.
let rec_u32: Vec<String> = ((j + 1)..ff)
.step_by(4)
.map(|o| format!("{:08X}", be32(d, o)))
.collect();
println!("\nNODE {name} @{i:#x} ff@{ff:#x} t44={t44:#x} t48={t48:#x} child={child:#x} sib={sib:#x}");
println!(" rec: {}", rec_u32.join(" "));
if t44 != 0 && t44 < d.len() {
// Joint table: nslots pointer slots. Each slot record appears to be a
// keyframe track: a value ALSO sits 0x48 BEFORE the pointed address
// (the rest-pose key A), while the decoder currently reads ptr+8 (B).
for k in 0..nslots {
let p = be32(d, t44 + k * 4) as usize;
if p == 0 || p + 24 > d.len() {
continue;
}
let a = if p >= 0x48 { be_f64(d, p - 0x48) } else { f64::NAN };
let b = be_f64(d, p + 8);
let head = if p >= 0x50 { be32(d, p - 0x50) } else { 0 };
let t_a = if p >= 0x50 { be_f64(d, p - 0x50) } else { f64::NAN };
let t_b = be_f64(d, p);
println!(
" slot{k:2} @{p:#x} head={head:08X}: A={a:12.5} (t={t_a:8.3}) B={b:12.5} (t={t_b:8.3})"
);
}
}
i = j.max(i + 1);
}
}

View File

@@ -0,0 +1,20 @@
//! Print a resource's first N decoded positions (buffer order), for matching a
//! capture draw's `pos:` dump to a part:
//! cargo run --release --example part_pos -- <xpr path> <resource> [n]
use std::collections::HashSet;
use sylpheed_formats::mesh::Xbg7Model;
fn main() {
let a: Vec<String> = std::env::args().collect();
let bytes = std::fs::read(&a[1]).unwrap();
let n: usize = a.get(3).and_then(|s| s.parse().ok()).unwrap_or(4);
let want: HashSet<String> = [a[2].clone()].into();
for m in Xbg7Model::models_named(&bytes, &want, &|| false) {
let vc: usize = m.meshes.iter().map(|s| s.positions.len()).sum();
print!("{} vtx={}:", m.name, vc);
for p in m.meshes.iter().flat_map(|s| &s.positions).take(n) {
print!(" ({:.4},{:.4},{:.4})", p[0], p[1], p[2]);
}
println!();
}
}

View File

@@ -0,0 +1,36 @@
use sylpheed_formats::{hash::name_hash, movie_manifest, movie_voice, slb, PakArchive};
use std::fs;use std::io::{Read,Seek,SeekFrom};use std::process::Command;
fn rg(disc:&str,g:u64,n:usize)->Vec<u8>{let mut segs=vec![];let mut cum=0u64;for i in 0..5{let p=format!("{disc}/dat/sound.p{i:02}");if let Ok(m)=fs::metadata(&p){segs.push((cum,m.len(),p));cum+=m.len();}}let mut out=vec![];let(mut need,mut pos)=(n,g);for(base,len,path)in &segs{if need==0||pos>=base+len||pos<*base{continue;}let local=pos-base;let take=need.min((len-local)as usize);let mut f=fs::File::open(path).unwrap();f.seek(SeekFrom::Start(local)).unwrap();let mut b=vec![0u8;take];f.read_exact(&mut b).unwrap();out.extend_from_slice(&b);need-=take;pos+=take as u64;}out}
fn contains(h:&[u8],n:&[u8])->bool{h.windows(n.len()).any(|w|w==n)}
fn dur(w:&str)->String{let o=Command::new("ffprobe").args(["-v","error","-show_entries","format=duration","-of","default=nw=1:nk=1",w]).output().unwrap();String::from_utf8_lossy(&o.stdout).trim().to_string()}
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let tpak=PakArchive::open(format!("{disc}/dat/tables.pak")).unwrap();
let manifest=tpak.entries().iter().find_map(|e|tpak.read(e).ok().filter(|b|movie_manifest::is_manifest(b))).unwrap();
let code="eng";
let registry=tpak.entries().iter().find_map(|e|tpak.read(e).ok().filter(|b|contains(b,format!("{code}\\Movie\\VOICE_ADV.slb").as_bytes()))).unwrap();
let ids=movie_voice::registry_voice_ids(&registry);
let stoc=fs::read(format!("{disc}/dat/sound.pak")).unwrap();
let entries=PakArchive::parse_toc(&stoc).unwrap();
for movie in ["ADV","RT01A","RT01B","RT01C_1","S00A","S01A","hokyu_LS_s02A","hokyu_LS_s09A","hokyu_LS_s02H","hokyu_DS_s07H"]{
let Some(token)=movie_manifest::voice_token(&manifest,movie) else { println!("{movie:16} no token"); continue };
let Some(&id)=ids.get(&token) else { println!("{movie:16} token {token} not in registry"); continue };
let Some(anchor)=["Movie","etc","Voice"].iter().find_map(|d|{let h=name_hash(&format!("{code}\\{d}\\{token}.slb"));entries.binary_search_by_key(&h,|e|e.name_hash).ok().map(|i|entries[i].offset as u64)}) else { println!("{movie:16} no anchor for {token}"); continue };
let win_start=(anchor.saturating_sub(2*1024*1024))&!3;
let window=rg(&disc,win_start,8*1024*1024);
let Some(end_local)=movie_voice::find_descriptor(&window,id) else { println!("{movie:16} id {id}: desc NOT FOUND"); continue };
let end=win_start+end_local as u64;
let start=movie_voice::find_descriptor(&window,id.wrapping_sub(1))
.or_else(||movie_voice::find_descriptor_before(&window,end_local))
.map(|o|win_start+o as u64)
.filter(|&s|s<end && end-s<1_500_000)
.unwrap_or(anchor);
let region=rg(&disc,start,(end-start) as usize);
let mut riffs=slb::to_xma_riffs(&region);
if riffs.is_empty(){ riffs=slb::to_xma_riff_best(&region).into_iter().collect(); }
let mut d="?".into();
if let Some(r)=riffs.first(){ let xp=format!("/tmp/ra2_{movie}.xma.wav");let wp=format!("/tmp/ra2_{movie}.wav");fs::write(&xp,r).unwrap();let _=Command::new("ffmpeg").args(["-hide_banner","-v","error","-y","-i",&xp,&wp]).status();d=dur(&wp);}
let mv=dur(&format!("{disc}/dat/movie/{movie}.wmv"));
println!("{movie:16} id={id:5} region={}KB riffs={} voice={d}s movie={mv}s", (end-start)/1024, riffs.len());
}
}

View File

@@ -0,0 +1,141 @@
//! Audit static ship assembly across ALL stages: flag parts placed far outside
//! their ship's cluster (placement bugs) and joint tracks with more than one
//! keyframe (which the single-key TRS read does not model).
//! cargo run --release --example ship_audit -- <resource3d dir>
use std::collections::HashSet;
use sylpheed_formats::mesh::{xbg7_descriptor_range, xbg7_resource_names, Xbg7Model};
use sylpheed_formats::ship::{assemble_ship, ships_in_container};
fn be32(d: &[u8], o: usize) -> u32 {
if o + 4 > d.len() {
return 0;
}
u32::from_be_bytes([d[o], d[o + 1], d[o + 2], d[o + 3]])
}
/// Count joint-track records whose key count != 1 in a composite's descriptor.
fn multikey_tracks(bytes: &[u8], comp: &str) -> usize {
let Some((desc, desc_end)) = xbg7_descriptor_range(bytes, comp) else { return 0 };
let d = &bytes[desc..desc_end];
let mut n = 0usize;
let mut i = 0usize;
while i + 4 <= d.len() {
let starts = &d[i..i.min(d.len() - 4) + 4] == b"rou_"
|| (i + 3 <= d.len() && &d[i..i + 3] == b"GN_");
if !starts {
i += 1;
continue;
}
let mut j = i;
while j < d.len() && d[j] != 0 && d[j].is_ascii_graphic() {
j += 1;
}
let mut ff = i;
let scan_end = (i + 0x90).min(d.len().saturating_sub(4));
while ff < scan_end && be32(d, ff) != 0xFFFF_FFFF {
ff += 4;
}
if ff < scan_end && ff >= 0x10 {
let t44 = be32(d, ff - 0x10) as usize;
if t44 != 0 && t44 + 32 <= d.len() {
for k in 0..8 {
let p = be32(d, t44 + k * 4) as usize;
if p >= 0x50 && p + 16 <= d.len() {
let head = be32(d, p - 0x50);
if head != 1 && head != 0 {
n += 1;
}
}
}
}
}
i = j.max(i + 1);
}
n
}
fn main() {
let dir = std::env::args().nth(1).unwrap();
let mut entries: Vec<_> = std::fs::read_dir(&dir)
.unwrap()
.filter_map(|e| e.ok())
.map(|e| e.path())
.filter(|p| {
let n = p.file_name().unwrap_or_default().to_string_lossy().to_string();
n.starts_with("Stage_") && n.ends_with(".xpr")
})
.collect();
entries.sort();
for path in entries {
let stage = path.file_name().unwrap().to_string_lossy().to_string();
let bytes = std::fs::read(&path).unwrap();
let names = xbg7_resource_names(&bytes);
// Multi-key animation tracks per composite (breaks the single-key read).
for n in names.iter().filter(|n| n.contains("rou_") && !n.contains("break")) {
let mk = multikey_tracks(&bytes, n);
if mk > 0 {
println!("MULTIKEY {stage} {n}: {mk} tracks");
}
}
for ship in ships_in_container(&bytes) {
if ship.parts.len() < 2 {
continue;
}
let placed = assemble_ship(&bytes, &ship.id, true);
if placed.len() < 2 {
continue;
}
let want: HashSet<String> = placed.iter().map(|p| p.resource.clone()).collect();
let models = Xbg7Model::models_named(&bytes, &want, &|| false);
// Per-placement world centroid.
let mut cents: Vec<(String, [f32; 3])> = Vec::new();
for p in &placed {
let Some(m) = models.iter().find(|m| m.name == p.resource) else { continue };
let (mut c, mut n) = ([0.0f32; 3], 0f32);
for sub in &m.meshes {
for v in &sub.positions {
let w = p.apply(*v);
c[0] += w[0];
c[1] += w[1];
c[2] += w[2];
n += 1.0;
}
}
if n > 0.0 {
cents.push((p.resource.clone(), [c[0] / n, c[1] / n, c[2] / n]));
}
}
if cents.len() < 2 {
continue;
}
// Ship cluster = median centroid; flag parts > 3× the median spread.
let med = |k: usize| -> f32 {
let mut v: Vec<f32> = cents.iter().map(|(_, c)| c[k]).collect();
v.sort_by(|a, b| a.partial_cmp(b).unwrap());
v[v.len() / 2]
};
let m = [med(0), med(1), med(2)];
let dists: Vec<f32> = cents
.iter()
.map(|(_, c)| {
((c[0] - m[0]).powi(2) + (c[1] - m[1]).powi(2) + (c[2] - m[2]).powi(2)).sqrt()
})
.collect();
let mut sd: Vec<f32> = dists.clone();
sd.sort_by(|a, b| a.partial_cmp(b).unwrap());
let spread = sd[sd.len() / 2].max(50.0);
for ((res, c), dist) in cents.iter().zip(&dists) {
if *dist > 6.0 * spread && *dist > 800.0 {
println!(
"OUTLIER {stage} {}: {res} centroid=[{:.0} {:.0} {:.0}] dist={:.0} (spread {:.0})",
ship.id, c[0], c[1], c[2], dist, spread
);
}
}
}
}
println!("audit done");
}

View File

@@ -0,0 +1,50 @@
//! Dump a ship's static placement + scene-graph nodes for a stage container FILE
//! (works offline from an extracted disc, no ISO needed).
//! cargo run --release --example ship_dump -- <Stage_SNN.xpr path> <ship_id>
use sylpheed_formats::mesh::{scene_world_nodes, xbg7_resource_names, Xbg7Model};
use sylpheed_formats::ship::{assemble_ship, is_base_part, ship_id_of};
use std::collections::HashSet;
fn main() {
let a: Vec<String> = std::env::args().collect();
let (path, id) = (&a[1], &a[2]);
let bytes = std::fs::read(path).unwrap();
let names = xbg7_resource_names(&bytes);
// Base parts + their vertex counts.
let want: HashSet<String> = names.iter().filter(|n| is_base_part(n) && ship_id_of(n)==Some(id.as_str())).cloned().collect();
let models = Xbg7Model::models_named(&bytes, &want, &|| false);
println!("== base parts ({}) ==", want.len());
for m in &models {
let vc: usize = m.meshes.iter().map(|s| s.positions.len()).sum();
println!(" {:20} vtx={}", m.name, vc);
}
// Composites present for this id.
println!("== composites ==");
for n in &names {
if n.contains(&format!("rou_{id}")) && ship_id_of(n).is_none() {
let nodes = scene_world_nodes(&bytes, n);
println!(" {:24} {} nodes", n, nodes.len());
}
}
// assemble_ship result: centroids + extent.
for ext in [false, true] {
let placed = assemble_ship(&bytes, id, ext);
println!("== assemble_ship(external={ext}) -> {} parts ==", placed.len());
let mut lo=[f32::MAX;3]; let mut hi=[f32::MIN;3];
for p in &placed {
let src = models.iter().find(|m| m.name==p.resource);
let (mut c, mut n) = ([0.0f32;3], 0f32);
if let Some(src)=src { for sub in &src.meshes { for v in &sub.positions {
let w=p.apply(*v); for k in 0..3 { c[k]+=w[k]; lo[k]=lo[k].min(w[k]); hi[k]=hi[k].max(w[k]); } n+=1.0; }}}
let n=n.max(1.0);
println!(" {:20} T=[{:8.1}{:8.1}{:8.1}] centroid=[{:8.1}{:8.1}{:8.1}]",
p.resource, p.t[0],p.t[1],p.t[2], c[0]/n,c[1]/n,c[2]/n);
}
if placed.iter().any(|p| models.iter().any(|m| m.name==p.resource)) {
println!(" EXTENT=[{:.0} {:.0} {:.0}]", hi[0]-lo[0], hi[1]-lo[1], hi[2]-lo[2]);
}
}
}

View File

@@ -0,0 +1,141 @@
//! Diagnostic: assemble a ship from the BAKED capture table (or the static
//! scene graph with --static) and render orthographic PNGs + print per-part
//! world bounds, to see exactly what the viewer shows.
//! cargo run --release --example ship_render -- <Stage_SNN.xpr path> <ship_id> <out_prefix> [--static]
use std::collections::HashSet;
use sylpheed_formats::mesh::{ScenePart, Xbg7Model};
use sylpheed_formats::ship::assemble_ship;
use sylpheed_formats::ship_capture::{embedded_placement, to_scene_parts};
fn main() {
let a: Vec<String> = std::env::args().collect();
let use_static = a.iter().any(|s| s == "--static");
let (path, id, out) = (&a[1], &a[2], &a[3]);
let bytes = std::fs::read(path).unwrap();
let placed: Vec<ScenePart> = if use_static {
assemble_ship(&bytes, id, true)
} else {
let cap = embedded_placement(id).expect("ship not in baked table");
to_scene_parts(&cap)
};
let want: HashSet<String> = placed.iter().map(|p| p.resource.clone()).collect();
let models = Xbg7Model::models_named(&bytes, &want, &|| false);
// World triangles + per-part bounds.
let mut tris: Vec<[[f32; 3]; 3]> = Vec::new();
for p in &placed {
let Some(m) = models.iter().find(|m| m.name == p.resource) else { continue };
let mut lo = [f32::MAX; 3];
let mut hi = [f32::MIN; 3];
for sub in &m.meshes {
let w: Vec<[f32; 3]> = sub.positions.iter().map(|v| p.apply(*v)).collect();
for v in &w {
for k in 0..3 {
lo[k] = lo[k].min(v[k]);
hi[k] = hi[k].max(v[k]);
}
}
for t in sub.indices.chunks_exact(3) {
tris.push([w[t[0] as usize], w[t[1] as usize], w[t[2] as usize]]);
}
}
println!(
"{:20} X[{:8.1},{:8.1}] Y[{:8.1},{:8.1}] Z[{:8.1},{:8.1}]",
p.resource, lo[0], hi[0], lo[1], hi[1], lo[2], hi[2]
);
}
// Exhaust cones at the GN_Jet/GN_SJet frames (the game's visible thrusters).
for f in sylpheed_formats::ship::exhaust_frames(&bytes, id) {
const SEG: usize = 12;
let (r, len) = (22.0f32, 140.0f32);
let ring: Vec<[f32; 3]> = (0..SEG)
.map(|s| {
let a = s as f32 / SEG as f32 * std::f32::consts::TAU;
f.apply([a.cos() * r, a.sin() * r, 0.0])
})
.collect();
let apex = f.apply([0.0, 0.0, len]);
for s in 0..SEG {
tris.push([apex, ring[(s + 1) % SEG], ring[s]]);
}
println!(" exhaust frame {:16} T=[{:8.1}{:8.1}{:8.1}]", f.resource, f.t[0], f.t[1], f.t[2]);
}
println!("total {} tris from {} placements", tris.len(), placed.len());
// Orthographic z-buffered flat renders: top (X/Z, look down Y) and side (Z/Y, look down X).
for (name, ax, ay, az) in [("top", 0usize, 2usize, 1usize), ("side", 2usize, 1usize, 0usize)] {
let size = 900usize;
let (mut lo, mut hi) = ([f32::MAX; 3], [f32::MIN; 3]);
for t in &tris {
for v in t {
for k in 0..3 {
lo[k] = lo[k].min(v[k]);
hi[k] = hi[k].max(v[k]);
}
}
}
let cx = (lo[ax] + hi[ax]) * 0.5;
let cy = (lo[ay] + hi[ay]) * 0.5;
let ext = (hi[ax] - lo[ax]).max(hi[ay] - lo[ay]) * 0.55;
let scale = (size as f32 * 0.5) / ext;
let mut img = vec![0u8; size * size * 3];
let mut zbuf = vec![f32::MIN; size * size];
for t in &tris {
// screen coords
let p: Vec<[f32; 3]> = t
.iter()
.map(|v| {
[
(v[ax] - cx) * scale + size as f32 * 0.5,
(cy - v[ay]) * scale + size as f32 * 0.5,
v[az],
]
})
.collect();
// flat shade by triangle normal Z-ish
let e1 = [t[1][0] - t[0][0], t[1][1] - t[0][1], t[1][2] - t[0][2]];
let e2 = [t[2][0] - t[0][0], t[2][1] - t[0][1], t[2][2] - t[0][2]];
let n = [
e1[1] * e2[2] - e1[2] * e2[1],
e1[2] * e2[0] - e1[0] * e2[2],
e1[0] * e2[1] - e1[1] * e2[0],
];
let nl = (n[0] * n[0] + n[1] * n[1] + n[2] * n[2]).sqrt().max(1e-6);
let lum = (n[az].abs() / nl * 200.0 + 40.0) as u8;
// bbox raster
let minx = p.iter().map(|v| v[0]).fold(f32::MAX, f32::min).max(0.0) as usize;
let maxx = (p.iter().map(|v| v[0]).fold(f32::MIN, f32::max).min(size as f32 - 1.0)) as usize;
let miny = p.iter().map(|v| v[1]).fold(f32::MAX, f32::min).max(0.0) as usize;
let maxy = (p.iter().map(|v| v[1]).fold(f32::MIN, f32::max).min(size as f32 - 1.0)) as usize;
let det = (p[1][0] - p[0][0]) * (p[2][1] - p[0][1]) - (p[2][0] - p[0][0]) * (p[1][1] - p[0][1]);
if det.abs() < 1e-6 {
continue;
}
for y in miny..=maxy {
for x in minx..=maxx {
let (fx, fy) = (x as f32 + 0.5, y as f32 + 0.5);
let w0 = ((p[1][0] - fx) * (p[2][1] - fy) - (p[2][0] - fx) * (p[1][1] - fy)) / det;
let w1 = ((p[2][0] - fx) * (p[0][1] - fy) - (p[0][0] - fx) * (p[2][1] - fy)) / det;
let w2 = 1.0 - w0 - w1;
if w0 < 0.0 || w1 < 0.0 || w2 < 0.0 {
continue;
}
let z = w0 * p[0][2] + w1 * p[1][2] + w2 * p[2][2];
let idx = y * size + x;
if z > zbuf[idx] {
zbuf[idx] = z;
img[idx * 3] = lum;
img[idx * 3 + 1] = lum;
img[idx * 3 + 2] = lum;
}
}
}
}
let file = format!("{out}_{name}.ppm");
let mut ppm = format!("P6\n{size} {size}\n255\n").into_bytes();
ppm.extend_from_slice(&img);
std::fs::write(&file, ppm).unwrap();
println!("wrote {file}");
}
}

View File

@@ -0,0 +1,33 @@
use sylpheed_formats::{idxd::IdxdObject, PakArchive};
use std::collections::BTreeMap;
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let arc=PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")).unwrap();
// schemas whose type0/content is squadron/formation/route/group/position
let mut hit:BTreeMap<u32,(u32,String,String)>=BTreeMap::new();
for e in arc.entries(){
let Ok(b)=arc.read(e) else{continue}; let Ok(o)=IdxdObject::parse(&b) else{continue};
let t=o.tokens(); let t0=t.first().cloned().unwrap_or_default();
let joined:String=t.iter().take(20).map(|s|s.as_str()).collect::<Vec<_>>().join(" ");
if ["Squadron","Formation","Route","UnitGroup","Position","Spawn","Wave","Frame","NullFrame"].iter().any(|k|t0.contains(k)||joined.contains(k)){
let ent=hit.entry(o.schema_hash).or_insert((0,t0.clone(),joined.chars().take(90).collect()));
ent.0+=1;
}
}
let mut v:Vec<_>=hit.into_iter().collect(); v.sort_by_key(|x|std::cmp::Reverse(x.1.0));
for (s,(c,t0,sample)) in &v{ println!("[{s:08x}] ×{c:<4} type0={t0:<18.18} :: {sample}"); }
// dump a Squadron/UnitGroup blob fully to see if it has positions (binary floats) or refs
println!("\n─── sample squadron/formation blob (first matching) ───");
for e in arc.entries(){
let Ok(b)=arc.read(e) else{continue}; let Ok(o)=IdxdObject::parse(&b) else{continue};
let t=o.tokens(); let t0=t.first().cloned().unwrap_or_default();
if t0.contains("Squadron")||t0.contains("Formation")||t0.contains("UnitGroup"){
println!("schema {:08x} type0={t0} count={} tokens={}, blob {}B", o.schema_hash,o.count,t.len(),b.len());
for (i,tk) in t.iter().take(30).enumerate(){ print!("{i:>2}:{tk:<20.20}"); if i%4==3{println!();} } println!();
// scan blob for float-like values (reasonable coords) in the binary region
let nfloats=(0..b.len().saturating_sub(4)).step_by(4).filter(|&i|{let f=f32::from_be_bytes([b[i],b[i+1],b[i+2],b[i+3]]); f.is_finite()&&f.abs()>1.0&&f.abs()<1e6}).count();
println!(" ~{nfloats} plausible BE-float words in blob (positions live in binary region if high)");
break;
}
}
}

View File

@@ -0,0 +1,11 @@
use sylpheed_formats::{game_data, PakArchive};
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
let pak=PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")).unwrap();
let sq=game_data::load_squadrons(&pak);
println!("{} squadron definitions", sq.len());
for s in sq.iter().filter(|s|s.side.as_deref()==Some("TCAF")).take(6){
let mem:Vec<String>=s.members.iter().map(|m|m.trim_start_matches("UN_").chars().take(18).collect()).collect();
println!(" {:8} {:22} {:24} x{} {:?}", s.id.clone().unwrap_or("?".into()), s.formation_id.clone().unwrap_or_default(), s.ai_id.clone().unwrap_or_default(), s.count.unwrap_or(0), mem);
}
}

View File

@@ -0,0 +1,55 @@
//! Scratch analysis: inventory one UI screen's pak — every entry, and for RATC
//! entries the children they bundle. Optionally write an entry's decompressed
//! bytes out for hex inspection.
//!
//! Run: cargo run -p sylpheed-formats --example ui_screen -- <PAK> [--dump 0xHASH out.bin]
use sylpheed_formats::pak::{self, PakArchive};
use sylpheed_formats::ratc;
fn main() {
let mut args = std::env::args().skip(1);
let pak = args.next().expect("usage: ui_screen <pak> [--dump 0xHASH out.bin]");
let arc = PakArchive::open(&pak).expect("open pak");
let rest: Vec<String> = args.collect();
if rest.first().map(|s| s == "--dump").unwrap_or(false) {
let h = u32::from_str_radix(rest[1].trim_start_matches("0x"), 16).unwrap();
let bytes = arc.read_by_hash(h).expect("entry present").expect("decompress");
std::fs::write(&rest[2], &bytes).unwrap();
println!("wrote {} bytes to {}", bytes.len(), rest[2]);
return;
}
if rest.first().map(|s| s == "--child").unwrap_or(false) {
// --child 0xHASH <child-name> <out>: carve one RATC child out by its
// listed offset/size, so a 165-byte layout record can be hexdumped alone.
let h = u32::from_str_radix(rest[1].trim_start_matches("0x"), 16).unwrap();
let bytes = arc.read_by_hash(h).expect("entry present").expect("decompress");
let kids = ratc::parse(&bytes).expect("ratc");
let k = kids.iter().find(|k| k.name == rest[2]).expect("child not found");
std::fs::write(&rest[3], &bytes[k.offset..k.offset + k.size]).unwrap();
println!("wrote {} B ({} @ {:#x}) to {}", k.size, k.name, k.offset, rest[3]);
return;
}
println!("{pak}: {} entries", arc.len());
for e in arc.entries() {
let Ok(bytes) = arc.read(e) else {
println!(" {:08x} <decompress failed>", e.name_hash);
continue;
};
let label = pak::inner_format_label(&bytes);
println!(" {:08x} {:>9} B {}", e.name_hash, bytes.len(), label);
if ratc::is_ratc(&bytes) {
if let Some(kids) = ratc::parse(&bytes) {
for k in &kids {
println!(" - {:<40} {:>9} B {}", k.name, k.size, k.kind);
}
if kids.is_empty() {
println!(" (no children found)");
}
}
}
}
}

View File

@@ -0,0 +1,32 @@
use sylpheed_formats::slb;
use std::fs;use std::io::{Read,Seek,SeekFrom};use std::process::Command;
fn rg(disc:&str,g:u64,n:usize)->Vec<u8>{let mut segs=vec![];let mut cum=0u64;for i in 0..5{let p=format!("{disc}/dat/sound.p{i:02}");if let Ok(m)=fs::metadata(&p){segs.push((cum,m.len(),p));cum+=m.len();}}let mut out=vec![];let(mut need,mut pos)=(n,g);for(base,len,path)in &segs{if need==0||pos>=base+len||pos<*base{continue;}let local=pos-base;let take=need.min((len-local)as usize);let mut f=fs::File::open(path).unwrap();f.seek(SeekFrom::Start(local)).unwrap();let mut b=vec![0u8;take];f.read_exact(&mut b).unwrap();out.extend_from_slice(&b);need-=take;pos+=take as u64;}out}
fn dur(w:&str)->String{let o=Command::new("ffprobe").args(["-v","error","-show_entries","format=duration:stream=channels","-of","default=nw=1:nk=1",w]).output().unwrap();String::from_utf8_lossy(&o.stdout).replace('\n'," ")}
fn movdur(disc:&str,m:&str)->String{ dur(&format!("{disc}/dat/movie/{m}")) }
fn main(){
let disc=std::env::var("SYLPHEED_DISC").unwrap();
// descriptor trailer global offsets (from scan) => cue N data = [desc(N-1)..desc(N)]
// (id, end_off, movie)
let cues=[(1600u32,437044592u64,"ADV.wmv"),(1601,437345648,"RT01A.wmv"),(1602,437712240,"RT01B.wmv"),
(1603,438080880,"RT01C_1.wmv"),(1604,438451568,"RT01C_2.wmv"),(1605,438789488,"RT02A.wmv")];
for k in 1..cues.len(){
let (id,end,mov)=cues[k];
let start=cues[k-1].1;
let region=rg(&disc,start,(end-start) as usize + 12000); // +tail to include full data before next trailer
let riffs=slb::to_xma_riffs(&region);
let mut total=0.0f32; let mut parts=vec![];
for (j,r) in riffs.iter().enumerate(){
let xp=format!("/tmp/cue{id}_{j}.xma.wav"); let wp=format!("/tmp/cue{id}_{j}.wav");
fs::write(&xp,r).unwrap();
let _=Command::new("ffmpeg").args(["-hide_banner","-v","error","-y","-i",&xp,&wp]).status();
let d=dur(&wp); parts.push(d.clone());
total+=d.split_whitespace().next().unwrap_or("0").parse::<f32>().unwrap_or(0.0);
}
// spanning?
let span_adv_end=437547264u64;
let spans = start < span_adv_end && end > span_adv_end || (start/1_000 != end/1_000);
println!("cue {id} ({mov}): region[{start}..{end}] {} bytes, {} riff(s), Σ={:.1}s | movie={} parts={:?}",
end-start, riffs.len(), total, movdur(&disc,mov), parts);
}
println!("\nWAVs at /tmp/cue16XX_*.wav (listen: RT01B=1602, RT01C_1=1603, RT01C_2=1604)");
}

View File

@@ -0,0 +1,980 @@
//! Typed loaders for Project Sylpheed's combat data tables.
//!
//! The game keeps its balance data in reflective [`crate::idxd`] tables — one
//! blob per entity, `token[0]` naming the table, fields laid out value-before-key.
//! This module maps the four combat schemas to plain, cloneable structs: the
//! common stats as named `Option<…>` fields, and **every** explicitly-set field in
//! a [`fields`](Weapon::fields) map so nothing is lost.
//!
//! Reverse-engineered 2026-07-23 from `GP_MAIN_GAME_E.pak` (English; the D/F/I/J/S
//! paks are localized duplicates). Fields left at their default value omit their
//! value on disc — those live in title code, not here — so a `None` means "not set
//! on disc", never a guess.
//!
//! ```no_run
//! use sylpheed_formats::{game_data, PakArchive};
//! let pak = PakArchive::open("dat/GP_MAIN_GAME_E.pak").unwrap();
//! for w in game_data::load_weapons(&pak) {
//! println!("{} — power {:?}, range {:?}", w.id.unwrap_or_default(), w.power, w.max_range);
//! }
//! ```
use crate::idxd::IdxdObject;
use crate::pak::PakArchive;
use std::collections::BTreeMap;
/// IDXD schema ids of the combat tables (the `schema_hash` field of each blob).
pub mod schema {
/// Player craft config: physics, cameras, scoring, difficulty, render pipeline.
pub const PLAYER: u32 = 0x0426_e81d;
/// Weapon definitions (guns, missiles, beams, bombs).
pub const WEAPON: u32 = 0x6ab4_825a;
/// Craft / units — fighters, bombers, turrets, with their AI flight model.
pub const UNIT: u32 = 0x43fa_a517;
/// Vessels — capital ships, with structural component counts.
pub const VESSEL: u32 = 0x3c5b_0549;
/// Characters — pilots / crew / comms, with faction and portrait set.
pub const CHARACTER: u32 = 0xbd86_d41c;
/// Stage resource manifest — per mission: location, phases, resource tables.
pub const STAGE: u32 = 0x3c9a_e32e;
/// Message / demo dialogue — per line: speaker, portrait, voice clip, pages.
pub const MESSAGE: u32 = 0xb412_e6d8;
}
/// Parse a numeric field value, tolerating a trailing `f`/`F` (e.g. `"3.0f"`).
fn as_f32(v: Option<&String>) -> Option<f32> {
let v = v?;
let v = v.strip_suffix(['f', 'F']).unwrap_or(v);
v.parse().ok()
}
fn as_i64(v: Option<&String>) -> Option<i64> {
v?.parse().ok()
}
/// Collect every explicitly-set field of an IDXD blob into a `key → value` map,
/// plus the identity strings. Identifier-valued fields (`Model`, `ShotType`, …)
/// that carry a bare name rather than a value are not in the map — read those from
/// [`IdxdObject::get_raw`] if needed.
fn field_map(o: &IdxdObject) -> BTreeMap<String, String> {
o.resolved_fields()
.into_iter()
.map(|(k, v)| (k.to_string(), v.to_string()))
.collect()
}
/// Read every record of `schema` from a pak, mapping each with `build`.
fn load_table<T>(pak: &PakArchive, schema: u32, build: impl Fn(&IdxdObject) -> Option<T>) -> Vec<T> {
pak.entries()
.iter()
.filter_map(|e| pak.read(e).ok())
.filter_map(|b| IdxdObject::parse(&b).ok())
.filter(|o| o.schema_hash == schema)
.filter_map(|o| build(&o))
.collect()
}
// ── Weapon ────────────────────────────────────────────────────────────────────
/// A weapon definition (schema [`schema::WEAPON`]).
#[derive(Debug, Clone)]
pub struct Weapon {
pub id: Option<String>,
pub name: Option<String>,
/// Target mask, e.g. `"Vessel,Craft,Structure"`.
pub target_type: Option<String>,
pub power: Option<f32>,
pub velocity: Option<f32>,
pub min_velocity: Option<f32>,
pub max_velocity: Option<f32>,
pub min_range: Option<f32>,
pub max_range: Option<f32>,
/// Ammo / reload budget (`LoadingCount`).
pub loading_count: Option<i64>,
/// Seconds between shots.
pub interval: Option<f32>,
/// Shots per trigger pull (volley / lock count).
pub trigger_shot_count: Option<i64>,
pub mass: Option<f32>,
pub heating: Option<f32>,
pub cooling: Option<f32>,
pub life_time: Option<f32>,
/// All explicitly-set fields (a superset of the named ones above).
pub fields: BTreeMap<String, String>,
}
impl Weapon {
fn from_idxd(o: &IdxdObject) -> Option<Self> {
let f = field_map(o);
Some(Weapon {
id: o.get_raw("ID").map(str::to_string),
name: o.get_raw("Name").map(str::to_string),
target_type: o.get_raw("TargetType").map(str::to_string),
power: as_f32(f.get("Power")),
velocity: as_f32(f.get("Velocity")),
min_velocity: as_f32(f.get("MinimumVelocity")),
max_velocity: as_f32(f.get("MaximumVelocity")),
min_range: as_f32(f.get("MinimumRange")),
max_range: as_f32(f.get("MaximumRange")),
loading_count: as_i64(f.get("LoadingCount")),
interval: as_f32(f.get("Interval")),
trigger_shot_count: as_i64(f.get("TriggerShotCount")),
mass: as_f32(f.get("Mass")),
heating: as_f32(f.get("Heating")),
cooling: as_f32(f.get("Cooling")),
life_time: as_f32(f.get("LifeTime")),
fields: f,
})
}
/// Any explicit field parsed as `f32` (for the long-tail stats not named above).
pub fn field_f32(&self, key: &str) -> Option<f32> {
as_f32(self.fields.get(key))
}
}
/// Load every weapon from a pak. Catches all `Weapon`-shaped records, not just the
/// main [`schema::WEAPON`] — player special/missile weapons live in variant schemas
/// (e.g. `Weapon_DSaber_P_wep_*` in `GP_HANGAR_ARSENAL.pak`). Deduplicated by id.
pub fn load_weapons(pak: &PakArchive) -> Vec<Weapon> {
let mut seen = std::collections::BTreeSet::new();
let mut out = Vec::new();
for e in pak.entries() {
let Ok(b) = pak.read(e) else { continue };
let Ok(o) = IdxdObject::parse(&b) else { continue };
// `Weapon`/`QWeapon`/… but not the `EnumWeapon` name lists.
let t0 = o.tokens().first().map(String::as_str).unwrap_or("");
if !t0.ends_with("Weapon") || t0.contains("Enum") {
continue;
}
if let Some(w) = Weapon::from_idxd(&o) {
if w.id.as_deref().is_some_and(|id| seen.insert(id.to_string())) {
out.push(w);
}
}
}
out
}
// ── Craft / unit ────────────────────────────────────────────────────────────────
/// A craft / unit — fighter, bomber, turret (schema [`schema::UNIT`]). Carries the
/// AI flight model in [`fields`](CraftUnit::fields) (`AV_Pitch*`, `BarrelRoll_*`,
/// `TurnAttack_*`, …).
#[derive(Debug, Clone)]
pub struct CraftUnit {
pub id: Option<String>,
pub name: Option<String>,
pub hp: Option<f32>,
pub is_destructible: Option<bool>,
pub size_x: Option<f32>,
pub size_y: Option<f32>,
pub size_z: Option<f32>,
pub size_radius: Option<f32>,
pub radar_range: Option<f32>,
pub fcs_range: Option<f32>,
pub cruising_velocity: Option<f32>,
pub maximum_velocity: Option<f32>,
pub acceleration: Option<f32>,
pub deceleration: Option<f32>,
pub shield_ratio: Option<f32>,
pub turret_count: Option<i64>,
pub score_point: Option<i64>,
pub fields: BTreeMap<String, String>,
}
impl CraftUnit {
fn from_idxd(o: &IdxdObject) -> Option<Self> {
let f = field_map(o);
Some(CraftUnit {
id: o.get_raw("ID").map(str::to_string),
name: o.get_raw("Name").map(str::to_string),
hp: as_f32(f.get("HP")),
is_destructible: o.get_bool("IsDestructible"),
size_x: as_f32(f.get("Size_X")),
size_y: as_f32(f.get("Size_Y")),
size_z: as_f32(f.get("Size_Z")),
size_radius: as_f32(f.get("Size_Radius")),
radar_range: as_f32(f.get("RadarRange")),
fcs_range: as_f32(f.get("FCSRange")),
cruising_velocity: as_f32(f.get("CruisingVelocity")),
maximum_velocity: as_f32(f.get("MaximumVelocity")),
acceleration: as_f32(f.get("Acceleration")),
deceleration: as_f32(f.get("Deceleration")),
shield_ratio: as_f32(f.get("ShieldRatio")),
turret_count: as_i64(f.get("TurretCount")),
score_point: as_i64(f.get("ScorePoint")),
fields: f,
})
}
pub fn field_f32(&self, key: &str) -> Option<f32> {
as_f32(self.fields.get(key))
}
}
/// Load every craft / unit from a pak.
pub fn load_units(pak: &PakArchive) -> Vec<CraftUnit> {
load_table(pak, schema::UNIT, CraftUnit::from_idxd)
}
// ── Vessel / capital ship ───────────────────────────────────────────────────────
/// A capital ship (schema [`schema::VESSEL`]). Component counts drive the
/// destructible hardpoints.
#[derive(Debug, Clone)]
pub struct Vessel {
pub id: Option<String>,
pub name: Option<String>,
/// The 3D hull resource stem, e.g. `rou_e105` — the `eNNN`/`fNNN` id links to
/// the XBG7 part family (`e105_bdy_*`, `e105_brg_*`, …) inside a `Stage_SNN.xpr`.
pub model: Option<String>,
pub hp: Option<f32>,
pub size_x: Option<f32>,
pub size_y: Option<f32>,
pub size_z: Option<f32>,
pub radar_range: Option<f32>,
pub fcs_range: Option<f32>,
pub maximum_velocity: Option<f32>,
pub shield_ratio: Option<f32>,
pub score_point: Option<i64>,
pub turret_count: Option<i64>,
pub bridge_count: Option<i64>,
pub hatch_count: Option<i64>,
pub shield_generator_count: Option<i64>,
pub thruster_count: Option<i64>,
pub fields: BTreeMap<String, String>,
}
impl Vessel {
fn from_idxd(o: &IdxdObject) -> Option<Self> {
let f = field_map(o);
Some(Vessel {
id: o.get_raw("ID").map(str::to_string),
name: o.get_raw("Name").map(str::to_string),
model: o.get_raw("Model").map(str::to_string),
hp: as_f32(f.get("HP")),
size_x: as_f32(f.get("Size_X")),
size_y: as_f32(f.get("Size_Y")),
size_z: as_f32(f.get("Size_Z")),
radar_range: as_f32(f.get("RadarRange")),
fcs_range: as_f32(f.get("FCSRange")),
maximum_velocity: as_f32(f.get("MaximumVelocity")),
shield_ratio: as_f32(f.get("ShieldRatio")),
score_point: as_i64(f.get("ScorePoint")),
turret_count: as_i64(f.get("TurretCount")),
bridge_count: as_i64(f.get("BridgeCount")),
hatch_count: as_i64(f.get("HatchCount")),
shield_generator_count: as_i64(f.get("ShieldGeneratorCount")),
thruster_count: as_i64(f.get("ThrusterCount")),
fields: f,
})
}
pub fn field_f32(&self, key: &str) -> Option<f32> {
as_f32(self.fields.get(key))
}
}
/// Load every capital ship from a pak.
pub fn load_vessels(pak: &PakArchive) -> Vec<Vessel> {
load_table(pak, schema::VESSEL, Vessel::from_idxd)
}
// ── Player config ───────────────────────────────────────────────────────────────
/// Player craft config (schema [`schema::PLAYER`]) — one per mission. Physics,
/// projectile caps and scoring here; cameras, difficulty and the render pipeline
/// live in [`fields`](PlayerConfig::fields).
#[derive(Debug, Clone)]
pub struct PlayerConfig {
pub air_drag_factor: Option<f32>,
pub gravity_factor: Option<f32>,
pub bullet_limit: Option<i64>,
pub laser_limit: Option<i64>,
pub homing_limit: Option<i64>,
pub space_size: Option<f32>,
pub supply_range: Option<f32>,
pub main_mission_bonus: Option<i64>,
pub rank_score_s: Option<i64>,
pub rank_score_a: Option<i64>,
pub rank_score_b: Option<i64>,
pub rank_score_c: Option<i64>,
pub rank_score_d: Option<i64>,
pub fields: BTreeMap<String, String>,
}
impl PlayerConfig {
fn from_idxd(o: &IdxdObject) -> Option<Self> {
let f = field_map(o);
Some(PlayerConfig {
air_drag_factor: as_f32(f.get("AirDragFactor")),
gravity_factor: as_f32(f.get("GravityFactor")),
bullet_limit: as_i64(f.get("BulletLimit")),
laser_limit: as_i64(f.get("LaserLimit")),
homing_limit: as_i64(f.get("HomingLimit")),
space_size: as_f32(f.get("SpaceSize")),
supply_range: as_f32(f.get("SupplyRange")),
main_mission_bonus: as_i64(f.get("MainMissionBonus")),
rank_score_s: as_i64(f.get("RankScore_S")),
rank_score_a: as_i64(f.get("RankScore_A")),
rank_score_b: as_i64(f.get("RankScore_B")),
rank_score_c: as_i64(f.get("RankScore_C")),
rank_score_d: as_i64(f.get("RankScore_D")),
fields: f,
})
}
pub fn field_f32(&self, key: &str) -> Option<f32> {
as_f32(self.fields.get(key))
}
}
/// Load every player config (one per mission) from a pak.
pub fn load_player_configs(pak: &PakArchive) -> Vec<PlayerConfig> {
load_table(pak, schema::PLAYER, PlayerConfig::from_idxd)
}
// ── Character ───────────────────────────────────────────────────────────────────
/// A pilot / crew portrait: an emotion id (`FaceRAYMOND_07`) and its texture
/// (`pjf003_C02.t32`, a T8aD tile inside a RATC bundle).
#[derive(Debug, Clone)]
pub struct Face {
pub id: String,
pub texture: String,
}
/// A character (schema [`CHARACTER`](schema::CHARACTER)) — pilots, crew, comms.
#[derive(Debug, Clone)]
pub struct Character {
pub id: Option<String>,
/// Localized display-name key (resolve against the game's string tables).
pub name_key: Option<String>,
/// Faction / side, e.g. `TCAF` (player) or `ADAN` (enemy).
pub faction: Option<String>,
pub unique: Option<bool>,
/// Portrait set: each emotion face and the texture that draws it.
pub faces: Vec<Face>,
pub fields: BTreeMap<String, String>,
}
impl Character {
fn from_idxd(o: &IdxdObject) -> Option<Self> {
let t = o.tokens();
// Faces are `(texture, FaceID)` pairs after the `Faces` marker (value-
// before-key: the FaceID names the emotion, the `.t32` before it is its art).
let mut faces = Vec::new();
if let Some(start) = t.iter().position(|s| s == "Faces") {
for i in (start + 1)..t.len() {
if t[i].starts_with("Face") && i > 0 {
faces.push(Face {
id: t[i].clone(),
texture: t[i - 1].clone(),
});
}
}
}
Some(Character {
id: o.get_raw("ID").map(str::to_string),
name_key: o.get_raw("Name").map(str::to_string),
faction: o.get_raw("SideID").map(str::to_string),
unique: o.get_bool("Unique"),
faces,
fields: field_map(o),
})
}
}
/// Load every character from a pak.
pub fn load_characters(pak: &PakArchive) -> Vec<Character> {
load_table(pak, schema::CHARACTER, Character::from_idxd)
}
// ── Stage / mission ─────────────────────────────────────────────────────────────
/// The raw token immediately before `key` in `tokens` (value-before-key), for
/// identifier-valued fields the generic resolver skips (filenames, locations).
fn raw_before<'a>(tokens: &'a [String], key: &str) -> Option<&'a str> {
let i = tokens.iter().position(|t| t == key)?;
(i > 0).then(|| tokens[i - 1].as_str())
}
/// A mission / stage (schema [`STAGE`](schema::STAGE)). The manifest that wires a
/// mission together: its location, phase count, and the per-stage tables that
/// populate it (unit roster, squadrons, formations, flight routes, AI, objectives).
/// Resolve the referenced `*_S<NN>.tbl` tables with [`load_records`] for the
/// spawn/formation detail.
#[derive(Debug, Clone)]
pub struct Stage {
/// Stage tag derived from the unit-table name, e.g. `"S01"`.
pub id: String,
/// In-world location (`BackGroundID`), e.g. `Lebendorf`, `Acheron`, `Earth`.
pub location: Option<String>,
/// Number of `Phase_N` sections.
pub phases: u32,
pub unit_table: Option<String>,
pub character_table: Option<String>,
pub squadron_table: Option<String>,
pub formation_table: Option<String>,
pub route_table: Option<String>,
pub ai_params_table: Option<String>,
pub subobjective_table: Option<String>,
pub message_table: Option<String>,
/// Every `resource-kind → table` reference in the manifest.
pub resources: BTreeMap<String, String>,
}
impl Stage {
fn from_idxd(o: &IdxdObject) -> Option<Self> {
let t = o.tokens();
// Each resource is `table-name → Enumerate<Kind>` (value-before-key).
let mut resources = BTreeMap::new();
for i in 1..t.len() {
let key = t[i].as_str();
if key.starts_with("Enumerate") || key == "MessageSet" || key == "NamePlate" {
resources.insert(key.to_string(), t[i - 1].clone());
}
}
let unit_table = raw_before(t, "EnumerateUnit").map(str::to_string);
let id = unit_table
.as_deref()
.and_then(|s| s.strip_prefix("EnumUnit_"))
.map(|s| s.trim_end_matches(".tbl").to_string())
.unwrap_or_default();
Some(Stage {
id,
location: o.get_raw("BackGroundID").map(str::to_string),
phases: t.iter().filter(|s| s.starts_with("Phase_")).count() as u32,
unit_table,
character_table: raw_before(t, "EnumerateCharacter").map(str::to_string),
squadron_table: raw_before(t, "EnumerateSquadron").map(str::to_string),
formation_table: raw_before(t, "EnumerateFormation").map(str::to_string),
route_table: raw_before(t, "EnumerateNullFrame").map(str::to_string),
ai_params_table: raw_before(t, "EnumerateAIParams").map(str::to_string),
subobjective_table: raw_before(t, "EnumerateSubobjective").map(str::to_string),
message_table: raw_before(t, "MessageSet").map(str::to_string),
resources,
})
}
}
/// Load every stage / mission manifest from a pak.
pub fn load_stages(pak: &PakArchive) -> Vec<Stage> {
load_table(pak, schema::STAGE, Stage::from_idxd)
}
// ── Squadron / flight group ─────────────────────────────────────────────────────
/// A squadron — a flight group that spawns together: its formation shape, AI
/// behaviour, side, and member craft (with their pilots). Parsed from the
/// per-stage `Enumerate_Squadrons` tables (`UnitGroup_S<NN>.tbl`). The player's
/// Rhino flight, for instance, is a 2-craft TCAF squadron flying `Formation_2_Rhino`.
#[derive(Debug, Clone)]
pub struct Squadron {
/// Squadron tag (`TCN001`, `ADT101`, …), matched positionally to its
/// definition; `None` if the id list and definitions don't line up.
pub id: Option<String>,
pub formation_id: Option<String>,
pub ai_id: Option<String>,
/// Faction — `TCAF` (allied) or `ADAN` (enemy).
pub side: Option<String>,
pub count: Option<i64>,
/// Member craft (`UN_*` unit ids).
pub members: Vec<String>,
}
/// Load every squadron from a pak. Scans all `Enumerate_Squadrons` tables (their
/// schema hash varies per stage, but the table name is stable).
pub fn load_squadrons(pak: &PakArchive) -> Vec<Squadron> {
let mut out = Vec::new();
for e in pak.entries() {
let Ok(b) = pak.read(e) else { continue };
let Ok(o) = IdxdObject::parse(&b) else { continue };
let t = o.tokens();
if !t.first().is_some_and(|s| s.contains("Enumerate_Squadron")) {
continue;
}
// Definitions are delimited by the `FormationID` key (its value, the
// formation, sits just before it). The id list precedes the first one.
let fpos: Vec<usize> = t
.iter()
.enumerate()
.filter(|(_, s)| *s == "FormationID")
.map(|(i, _)| i)
.collect();
let Some(&first) = fpos.first() else { continue };
// The squadron id list: tokens after the header up to the first formation.
let ids: Vec<&String> = t[1..first.saturating_sub(1)].iter().collect();
for (k, &fi) in fpos.iter().enumerate() {
let start = fi.saturating_sub(1);
let end = fpos.get(k + 1).map(|&n| n.saturating_sub(1)).unwrap_or(t.len());
let span = &t[start..end];
let before = |key: &str| -> Option<String> {
span.iter().position(|s| s == key).and_then(|i| (i > 0).then(|| span[i - 1].clone()))
};
let count = before("Count").and_then(|v| v.parse::<i64>().ok());
// Members are the first `count` `UN_` craft after the header block
// (which ends at DisableInterval, else Count); anything past them
// belongs to the next squadron or the stage roster.
let member_start = span
.iter()
.position(|s| s == "DisableInterval")
.or_else(|| span.iter().position(|s| s == "Count"))
.map(|i| i + 1)
.unwrap_or(0);
let take = count.unwrap_or(0).max(0) as usize;
out.push(Squadron {
id: (ids.len() == fpos.len()).then(|| ids[k].clone()),
formation_id: Some(t[fi - 1].clone()),
ai_id: before("AIID"),
side: before("SideID"),
count,
members: span[member_start..]
.iter()
.filter(|s| s.starts_with("UN_"))
.take(take)
.cloned()
.collect(),
});
}
}
out
}
// ── Demo message / dialogue line ────────────────────────────────────────────────
/// One dialogue line (schema [`MESSAGE`](schema::MESSAGE)): who speaks, with which
/// portrait and voice clip, and the text-key(s) of its caption pages. Resolve the
/// page keys against a [`crate::localization::TextIndex`] for the words, and the
/// voice clip against the movie-voice banks for the audio.
#[derive(Debug, Clone)]
pub struct DemoMessage {
/// Line id, e.g. `MSG_VOICE_A_007`.
pub id: String,
/// Speaker (`CharacterRAYMOND`) when the line names one; many system /
/// continuation lines are unattributed (`None`).
pub character: Option<String>,
/// Portrait / emotion (`FaceRAYMOND_07`).
pub face: Option<String>,
/// Voice clip token (`VOICE_A_007`).
pub voice_clip: Option<String>,
/// Caption page text keys (`MSG_VOICE_A_007_000_00`, …) — resolve for the text.
pub page_keys: Vec<String>,
}
/// Load every dialogue line from a pak. Each `Message_NNN` sub-record within the
/// message blobs becomes one [`DemoMessage`]; fields are found by prefix, so the
/// positional/defaulted layout doesn't matter.
pub fn load_demo_messages(pak: &PakArchive) -> Vec<DemoMessage> {
let is_marker = |s: &str| {
s.strip_prefix("Message_").is_some_and(|n| !n.is_empty() && n.bytes().all(|b| b.is_ascii_digit()))
};
let mut out = Vec::new();
for e in pak.entries() {
let Ok(b) = pak.read(e) else { continue };
let Ok(o) = IdxdObject::parse(&b) else { continue };
if o.schema_hash != schema::MESSAGE {
continue;
}
let t = o.tokens();
let marks: Vec<usize> = t.iter().enumerate().filter(|(_, s)| is_marker(s)).map(|(i, _)| i).collect();
for (k, &m) in marks.iter().enumerate() {
let end = marks.get(k + 1).copied().unwrap_or(t.len());
let slice = &t[m + 1..end];
let Some(id) = slice.first().cloned() else { continue };
out.push(DemoMessage {
character: slice.iter().find(|s| s.starts_with("Character")).cloned(),
face: slice.iter().find(|s| s.starts_with("Face")).cloned(),
voice_clip: slice.iter().find(|s| s.starts_with("VOICE_")).cloned(),
page_keys: slice
.iter()
.filter(|s| s.len() > id.len() && s.starts_with(&id) && s[id.len()..].starts_with('_'))
.cloned()
.collect(),
id,
});
}
}
out
}
// ── Unit roster (per-mission combatants) ────────────────────────────────────────
/// Schema of the per-stage `EnumUnit` roster tables.
const ENUM_UNIT_SCHEMA: u32 = 0x35b8_dc67;
/// A mission's unit roster — the craft, vessels and props that can appear in one
/// battle (an `EnumUnit_S<NN>` table). Join the ids against [`load_units`] /
/// [`load_vessels`] for stats. The table isn't tagged with its stage on disc, so
/// [`stage`](UnitRoster::stage) is inferred from any `S<NN>_`-prefixed prop id in
/// the roster (asteroid collision meshes etc.) and is `None` when none is present.
#[derive(Debug, Clone)]
pub struct UnitRoster {
pub stage: Option<String>,
/// Distinct combatant unit ids (`UN_*`), excluding stage props (asteroids,
/// collision meshes).
pub units: Vec<String>,
}
/// Load every mission unit roster from a pak.
pub fn load_unit_rosters(pak: &PakArchive) -> Vec<UnitRoster> {
let mut out = Vec::new();
for e in pak.entries() {
let Ok(b) = pak.read(e) else { continue };
let Ok(o) = IdxdObject::parse(&b) else { continue };
if o.schema_hash != ENUM_UNIT_SCHEMA {
continue;
}
let ids: Vec<&str> = o.tokens().iter().filter(|s| s.starts_with("UN_")).map(String::as_str).collect();
// Stage tag from an `UN_S<NN>_…` prop id (e.g. UN_S01_Asteroid_cmesh_…).
let stage = ids.iter().find_map(|s| {
let r = s.trim_start_matches("UN_");
let bytes = r.as_bytes();
(r.len() >= 3 && bytes[0] == b'S' && bytes[1].is_ascii_digit() && bytes[2].is_ascii_digit())
.then(|| r[..3].to_string())
});
// Combatants: drop the stage props, dedup preserving order.
let mut units = Vec::new();
for id in ids {
let prop = id.contains("Asteroid") || id.contains("cmesh") || id.contains("_Box");
if !prop && !units.iter().any(|u| u == id) {
units.push(id.to_string());
}
}
if !units.is_empty() {
out.push(UnitRoster { stage, units });
}
}
out
}
// ── Pilot roster (player squadron assignments) ──────────────────────────────────
/// The player squadron's flight assignments for a mission — which pilot flies each
/// callsign (`Rhino1` → `Katana`, `Bird1` → `Sandra`, …). Parsed from the
/// `UNITS`/`ZUNITS` player-unit tables (in `GP_HANGAR_ARSENAL.pak`); assignments
/// vary per mission (different missions field different wingmen).
#[derive(Debug, Clone)]
pub struct PilotRoster {
/// `(callsign, pilot)` pairs in flight order, e.g. `("Rhino1", "Katana")`.
pub pilots: Vec<(String, String)>,
/// The player craft unit id (`UN_f002_TCAF_DeltaSaber…`).
pub player_unit: Option<String>,
}
/// True for a `Callsign<N>-Pilot` token (a single hyphen, digit-suffixed callsign,
/// alphabetic pilot) — the flight-assignment shape.
fn is_assignment(s: &str) -> bool {
let Some((cs, pilot)) = s.split_once('-') else { return false };
!pilot.is_empty()
&& pilot.chars().all(|c| c.is_ascii_alphabetic())
&& cs.len() >= 2
&& cs.as_bytes()[cs.len() - 1].is_ascii_digit()
&& cs.chars().all(|c| c.is_ascii_alphanumeric())
}
/// Load every player pilot roster from a pak (scan `GP_HANGAR_ARSENAL.pak`). One
/// per player-unit config; rosters with no assignment tokens are skipped.
pub fn load_pilot_rosters(pak: &PakArchive) -> Vec<PilotRoster> {
let mut out = Vec::new();
for e in pak.entries() {
let Ok(b) = pak.read(e) else { continue };
let Ok(o) = IdxdObject::parse(&b) else { continue };
let t = o.tokens();
let pilots: Vec<(String, String)> = t
.iter()
.filter(|s| is_assignment(s))
.filter_map(|s| s.split_once('-').map(|(c, p)| (c.to_string(), p.to_string())))
.collect();
if pilots.len() >= 2 {
out.push(PilotRoster {
pilots,
player_unit: t.iter().find(|s| s.starts_with("UN_")).cloned(),
});
}
}
out
}
// ── Arsenal (player weapon options per hardpoint) ───────────────────────────────
/// The player's selectable weapons, by hardpoint (from the `UNITS` player-unit
/// tables in `GP_HANGAR_ARSENAL.pak`). These are the arsenal display ids — the
/// Delta Saber's nose guns (`Stiletto_BG1`, `Broad_Sword_SG1`, …) and arm missiles
/// (`Falcon_9AM`, `White_Shark_T53R`, …) the player equips in the hangar.
#[derive(Debug, Clone, Default)]
pub struct Arsenal {
/// Nose slot — the fixed forward gun options.
pub nose: Vec<String>,
/// The three arm hardpoints — missile / bomb / special options.
pub arm1: Vec<String>,
pub arm2: Vec<String>,
pub arm3: Vec<String>,
}
/// The weapon ids listed under a `STANDARD_<hp>` header (skipping its `Type` key),
/// up to the next `STANDARD_`/`PlayerSET_` header.
fn hardpoint_options(tokens: &[String], header: &str) -> Vec<String> {
let Some(start) = tokens.iter().position(|s| s == header) else { return Vec::new() };
tokens[start + 1..]
.iter()
.skip_while(|s| *s == "Type")
.take_while(|s| !s.starts_with("STANDARD_") && !s.starts_with("PlayerSET"))
.filter(|s| *s != "Type")
.cloned()
.collect()
}
/// Load the player arsenal from a pak (`GP_HANGAR_ARSENAL.pak`). Unions the weapon
/// options across every player-unit config, deduped in first-seen order.
pub fn load_arsenal(pak: &PakArchive) -> Arsenal {
let mut a = Arsenal::default();
let mut seen = [(); 4].map(|_| std::collections::BTreeSet::new());
let slots: [(&str, usize); 4] = [
("STANDARD_NOSE", 0),
("STANDARD_ARM1", 1),
("STANDARD_ARM2", 2),
("STANDARD_ARM3", 3),
];
for e in pak.entries() {
let Ok(b) = pak.read(e) else { continue };
let Ok(o) = IdxdObject::parse(&b) else { continue };
if !o.tokens().first().is_some_and(|s| s.ends_with("UNITS")) {
continue;
}
for (header, idx) in slots {
let dst = match idx {
0 => &mut a.nose,
1 => &mut a.arm1,
2 => &mut a.arm2,
_ => &mut a.arm3,
};
for w in hardpoint_options(o.tokens(), header) {
if seen[idx].insert(w.clone()) {
dst.push(w);
}
}
}
}
a
}
// ── Generic record access (any of the ~105 schemas) ─────────────────────────────
/// A generically-decoded IDXD record: its table name, schema id, identity, and
/// every explicitly-set field. Use this to read the ~99 schemas without a bespoke
/// struct (missions, UI layouts, effects, enums, …).
#[derive(Debug, Clone)]
pub struct Record {
/// The table name — `token[0]` (e.g. `Weapon`, `StageResource`, `Sperkers`).
/// May carry a stray leading byte from the string-pool boundary on some
/// records; group by [`schema`](Self::schema), not this, for reliability.
pub table: String,
pub schema: u32,
pub id: Option<String>,
pub name: Option<String>,
/// Every explicitly-set field (`key → value`).
pub fields: BTreeMap<String, String>,
}
impl Record {
fn from_idxd(o: &IdxdObject) -> Option<Self> {
Some(Record {
table: o.tokens().first().cloned().unwrap_or_default(),
schema: o.schema_hash,
id: o.get_raw("ID").map(str::to_string),
name: o.get_raw("Name").map(str::to_string),
fields: field_map(o),
})
}
pub fn f32(&self, key: &str) -> Option<f32> {
as_f32(self.fields.get(key))
}
pub fn i64(&self, key: &str) -> Option<i64> {
as_i64(self.fields.get(key))
}
}
/// Load every record of an arbitrary `schema` from a pak, generically. Pair with
/// the [`schema`] constants or a hash from the IDXD census.
pub fn load_records(pak: &PakArchive, schema: u32) -> Vec<Record> {
load_table(pak, schema, Record::from_idxd)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn as_f32_tolerates_trailing_f() {
assert_eq!(as_f32(Some(&"3.0f".to_string())), Some(3.0));
assert_eq!(as_f32(Some(&"-0.5".to_string())), Some(-0.5));
assert_eq!(as_f32(Some(&"Vessel,Craft".to_string())), None);
}
// Integration tests against a real disc — skipped unless SYLPHEED_DISC is set.
fn pak() -> Option<PakArchive> {
let disc = std::env::var("SYLPHEED_DISC").ok()?;
PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")).ok()
}
#[test]
fn loads_weapons() {
let Some(pak) = pak() else { return };
let ws = load_weapons(&pak);
assert!(ws.len() >= 100, "expected ≥100 weapons, got {}", ws.len());
let rocket = ws
.iter()
.find(|w| w.id.as_deref().is_some_and(|s| s.contains("DeltaSaber_Rocket")))
.expect("Delta Saber rocket present");
assert_eq!(rocket.velocity, Some(3000.0));
assert_eq!(rocket.max_range, Some(6000.0));
assert_eq!(rocket.loading_count, Some(1000));
assert_eq!(rocket.target_type.as_deref(), Some("Vessel,Craft,Structure"));
}
#[test]
fn loads_units_and_vessels() {
let Some(pak) = pak() else { return };
let units = load_units(&pak);
assert!(units.len() >= 80);
// Player craft: HP 1000, radar 500000.
let player = units
.iter()
.find(|u| u.id.as_deref().is_some_and(|s| s.contains("DeltaSaber_T")))
.expect("player craft present");
assert_eq!(player.hp, Some(1000.0));
assert_eq!(player.radar_range, Some(500000.0));
let vessels = load_vessels(&pak);
let flagship = vessels
.iter()
.find(|v| v.id.as_deref().is_some_and(|s| s.contains("SDBattleship")))
.expect("SD-Battleship present");
assert_eq!(flagship.hp, Some(100000.0));
assert_eq!(flagship.turret_count, Some(25));
}
#[test]
fn loads_player_configs() {
let Some(pak) = pak() else { return };
let cfgs = load_player_configs(&pak);
assert!(!cfgs.is_empty());
let c = &cfgs[0];
assert_eq!(c.bullet_limit, Some(512));
assert_eq!(c.laser_limit, Some(32));
assert_eq!(c.rank_score_s, Some(10000));
}
#[test]
fn loads_characters() {
let Some(pak) = pak() else { return };
let chars = load_characters(&pak);
assert!(chars.len() >= 20, "expected many characters, got {}", chars.len());
let raymond = chars
.iter()
.find(|c| c.id.as_deref() == Some("CharacterRAYMOND"))
.expect("Raymond present");
assert_eq!(raymond.faction.as_deref(), Some("TCAF"));
assert!(!raymond.faces.is_empty());
assert!(raymond.faces.iter().all(|f| f.texture.ends_with(".t32")));
}
#[test]
fn loads_stages() {
let Some(pak) = pak() else { return };
let stages = load_stages(&pak);
let s01 = stages.iter().find(|s| s.id == "S01").expect("stage S01 present");
assert_eq!(s01.location.as_deref(), Some("Lebendorf"));
assert_eq!(s01.phases, 3);
assert_eq!(s01.unit_table.as_deref(), Some("EnumUnit_S01.tbl"));
assert_eq!(s01.squadron_table.as_deref(), Some("UnitGroup_S01.tbl"));
// 16-mission main campaign is present.
for n in 1..=16 {
assert!(stages.iter().any(|s| s.id == format!("S{n:02}")), "missing S{n:02}");
}
}
#[test]
fn loads_squadrons() {
let Some(pak) = pak() else { return };
let sq = load_squadrons(&pak);
assert!(sq.len() >= 20, "expected many squadrons, got {}", sq.len());
// A TCAF Rhino flight of Delta Sabers, membership matching its count.
let rhino = sq
.iter()
.find(|s| {
s.side.as_deref() == Some("TCAF")
&& s.formation_id.as_deref().is_some_and(|f| f.contains("Rhino"))
&& s.members.iter().any(|m| m.contains("DeltaSaber"))
})
.expect("a TCAF Rhino Delta Saber squadron");
assert_eq!(rhino.members.len() as i64, rhino.count.unwrap_or(-1));
assert!(rhino.members.iter().all(|m| m.contains("DeltaSaber")));
}
#[test]
fn loads_demo_messages() {
let Some(pak) = pak() else { return };
let msgs = load_demo_messages(&pak);
assert!(msgs.len() > 200, "expected many dialogue lines, got {}", msgs.len());
// Most lines are `MSG_*` ids with a speaker; page keys extend the id.
let msg_ids = msgs.iter().filter(|m| m.id.starts_with("MSG_")).count();
assert!(msg_ids > msgs.len() * 9 / 10, "{msg_ids}/{}", msgs.len());
// Attributed dialogue (an explicit speaker) is a minority — most lines are
// unattributed system/continuation messages — but there are hundreds.
let attributed = msgs.iter().filter(|m| m.character.is_some()).count();
assert!(attributed > 500, "attributed lines: {attributed}");
let m = msgs.iter().find(|m| m.character.is_some() && !m.page_keys.is_empty()).unwrap();
assert!(m.page_keys.iter().all(|k| k.starts_with(&m.id)));
}
#[test]
fn loads_unit_rosters() {
let Some(pak) = pak() else { return };
let rosters = load_unit_rosters(&pak);
assert!(rosters.len() > 20, "expected many rosters, got {}", rosters.len());
// At least a few tag their stage; all rosters carry combatants, no props.
assert!(rosters.iter().filter(|r| r.stage.is_some()).count() >= 4);
assert!(rosters.iter().all(|r| {
!r.units.is_empty() && r.units.iter().all(|u| !u.contains("Asteroid"))
}));
// A roster tagged S01 exists and names the player's Delta Saber.
let s01 = rosters.iter().find(|r| r.stage.as_deref() == Some("S01"));
assert!(s01.is_some_and(|r| r.units.iter().any(|u| u.contains("DeltaSaber"))));
}
#[test]
fn loads_pilot_rosters() {
let Ok(disc) = std::env::var("SYLPHEED_DISC") else { return };
let Ok(pak) = PakArchive::open(format!("{disc}/dat/GP_HANGAR_ARSENAL.pak")) else { return };
let rosters = load_pilot_rosters(&pak);
assert!(!rosters.is_empty(), "expected player pilot rosters");
// Some config assigns Katana to a Rhino callsign (she leads Rhino flight).
assert!(rosters.iter().any(|r| {
r.pilots.iter().any(|(cs, p)| cs.starts_with("Rhino") && p == "Katana")
}));
// Bird flight exists too.
assert!(rosters.iter().any(|r| r.pilots.iter().any(|(cs, _)| cs.starts_with("Bird"))));
}
#[test]
fn loads_arsenal() {
let Ok(disc) = std::env::var("SYLPHEED_DISC") else { return };
let Ok(pak) = PakArchive::open(format!("{disc}/dat/GP_HANGAR_ARSENAL.pak")) else { return };
let a = load_arsenal(&pak);
assert!(a.nose.len() >= 5 && a.arm1.len() >= 5, "nose {} arm1 {}", a.nose.len(), a.arm1.len());
assert!(a.nose.iter().any(|w| w.starts_with("Stiletto")));
assert!(a.arm1.iter().any(|w| w.starts_with("Falcon")));
// Headers never leak into the option lists.
assert!([&a.nose, &a.arm1, &a.arm2, &a.arm3]
.iter()
.all(|v| v.iter().all(|w| !w.starts_with("STANDARD_") && *w != "Type")));
}
#[test]
fn generic_records_read_any_schema() {
let Some(pak) = pak() else { return };
// Stage-resource table via the generic reader (no bespoke struct).
let stages = load_records(&pak, 0x3c9a_e32e);
assert!(!stages.is_empty());
assert!(stages.iter().all(|r| r.table.ends_with("StageResource")));
}
}

View File

@@ -59,6 +59,18 @@ pub mod slb;
// The movie manifest: authoritative movie → subtitle → voice index.
pub mod movie_manifest;
pub mod movie_voice;
pub mod game_data;
pub mod localization;
// Whole-ship assembly from XBG7 part families (capital ships as split parts).
pub mod ship;
// Exact capital-ship placement from a runtime F10 ship-capture (ground truth).
pub mod ship_capture;
/// Re-export the most commonly used types at the crate root.
pub use font::FontInfo;
pub use idxd::{IdxdError, IdxdObject};

View File

@@ -0,0 +1,179 @@
//! Localization — resolve the game's text keys to display strings.
//!
//! Project Sylpheed stores its UI/story text in `IXUD` entries as **UTF-16LE**,
//! interleaved `text, key, text, key, …`: a prose string immediately followed by
//! the identifier that names it. Objectives (`S01_P1_Objective_00`), hints, lose
//! conditions, character names (`CharacterName_CharacterRAYMOND`) and cutscene
//! dialogue (`MSG_*`) all resolve this way. One pak per language — pass
//! `GP_MAIN_GAME_E.pak` for English, `GP_MAIN_GAME_J.pak` for Japanese, etc.
//!
//! ```no_run
//! use sylpheed_formats::{localization::TextIndex, PakArchive};
//! let pak = PakArchive::open("dat/GP_MAIN_GAME_E.pak").unwrap();
//! let text = TextIndex::build(&pak);
//! assert_eq!(text.character_name("CharacterRAYMOND"), Some("Raymond"));
//! for line in text.objectives("S01", 1) { println!("• {line}"); }
//! ```
use crate::pak::PakArchive;
use std::collections::BTreeMap;
/// A UTF-16 unit that belongs to a printable text run (ASCII + Latin-1 + Latin
/// Extended-A). Control codes and the CJK planes split tokens.
fn is_text_unit(u: u16) -> bool {
matches!(u, 0x20..=0x7e | 0xa0..=0xff | 0x100..=0x17f)
}
/// Split a UTF-16LE blob into printable tokens (mirrors the subtitle tokenizer:
/// step 2 inside a run, 1 on a break to resync odd alignment).
fn utf16_tokens(bytes: &[u8]) -> Vec<String> {
let mut out = Vec::new();
let mut cur = String::new();
let mut i = 0;
while i + 1 < bytes.len() {
let u = u16::from_le_bytes([bytes[i], bytes[i + 1]]);
if is_text_unit(u) {
if let Some(c) = char::from_u32(u as u32) {
cur.push(c);
}
i += 2;
} else {
if cur.chars().count() >= 2 {
out.push(std::mem::take(&mut cur));
} else {
cur.clear();
}
i += 1;
}
}
if cur.chars().count() >= 2 {
out.push(cur);
}
out
}
/// An identifier token — the key half of a `text, key` pair.
fn is_key(s: &str) -> bool {
s.len() >= 5
&& s.contains('_')
&& s.chars().all(|c| c.is_ascii_alphanumeric() || c == '_')
&& s.chars().any(|c| c.is_ascii_uppercase())
}
/// A display-text token — anything with a lowercase letter that isn't itself a key.
fn is_prose(s: &str) -> bool {
!is_key(s) && s.chars().any(|c| c.is_ascii_lowercase())
}
/// Text-key → display-string index for one language, built from a `GP_MAIN_GAME_*`
/// pak.
#[derive(Debug, Clone, Default)]
pub struct TextIndex {
entries: BTreeMap<String, String>,
}
impl TextIndex {
/// Scan a language pak's `IXUD` entries and index every `text → key` pair.
pub fn build(pak: &PakArchive) -> Self {
let mut entries = BTreeMap::new();
for e in pak.entries() {
let Ok(b) = pak.read(e) else { continue };
if b.len() < 4 || &b[0..4] != b"IXUD" {
continue;
}
let toks = utf16_tokens(&b);
for w in toks.windows(2) {
if is_key(&w[1]) && is_prose(&w[0]) {
entries.entry(w[1].clone()).or_insert_with(|| w[0].clone());
}
}
}
TextIndex { entries }
}
/// The display string for a text key, if present.
pub fn get(&self, key: &str) -> Option<&str> {
self.entries.get(key).map(String::as_str)
}
/// Number of indexed keys.
pub fn len(&self) -> usize {
self.entries.len()
}
pub fn is_empty(&self) -> bool {
self.entries.is_empty()
}
/// The objective lines for a stage/phase, e.g. `("S01", 1)`. Empty if none
/// resolve (unused objective slots are simply absent on disc).
pub fn objectives(&self, stage: &str, phase: u32) -> Vec<&str> {
self.numbered(&format!("{stage}_P{phase}_Objective"))
}
/// The lose-condition lines for a stage/phase.
pub fn lose_conditions(&self, stage: &str, phase: u32) -> Vec<&str> {
self.numbered(&format!("{stage}_P{phase}_Lose"))
}
/// The hint lines for a stage/phase.
pub fn hints(&self, stage: &str, phase: u32) -> Vec<&str> {
self.numbered(&format!("{stage}_P{phase}_Hint"))
}
/// A character's display name from its id (`CharacterRAYMOND` → `Raymond`).
pub fn character_name(&self, character_id: &str) -> Option<&str> {
self.get(&format!("CharacterName_{character_id}"))
}
/// Collect a `<base>_NN` run of lines while they resolve.
fn numbered(&self, base: &str) -> Vec<&str> {
(0..16)
.filter_map(|i| self.get(&format!("{base}_{i:02}")))
.collect()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn tokenizes_mixed_runs() {
// "Hi" NUL "K_1" as UTF-16LE → two tokens.
let mut b = Vec::new();
for s in ["Hi", "\0", "K_1x"] {
for c in s.chars() {
b.extend_from_slice(&(c as u16).to_le_bytes());
}
}
let t = utf16_tokens(&b);
assert!(t.contains(&"Hi".to_string()));
assert!(t.contains(&"K_1x".to_string()));
}
#[test]
fn key_and_prose_classify() {
assert!(is_key("S01_P1_Objective_00"));
assert!(is_key("CharacterName_CharacterRAYMOND"));
assert!(!is_key("Repel the attack."));
assert!(is_prose("Repel the attack."));
assert!(is_prose("Raymond"));
assert!(!is_prose("S01_P1_Objective_00"));
}
// Disc-backed — skipped unless SYLPHEED_DISC is set.
#[test]
fn resolves_real_text() {
let Ok(disc) = std::env::var("SYLPHEED_DISC") else { return };
let Ok(pak) = PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")) else { return };
let t = TextIndex::build(&pak);
assert!(t.len() > 5000, "expected a large index, got {}", t.len());
assert_eq!(t.character_name("CharacterRAYMOND"), Some("Raymond"));
assert_eq!(
t.get("S01_P1_Objective_00"),
Some("Repel the enemy's surprise attack.")
);
assert!(!t.objectives("S01", 1).is_empty());
}
}

View File

@@ -151,6 +151,37 @@ pub fn count_xbg7(bytes: &[u8]) -> usize {
n
}
/// List every XBG7 resource name in an XPR2 container, in directory order.
///
/// Cheap: walks only the resource directory (no geometry decode). Used to
/// discover which container holds a named hull / part model when assembling a
/// multi-part ship from a [`crate::game_data::Vessel`] recipe.
pub fn xbg7_resource_names(bytes: &[u8]) -> Vec<String> {
let mut out = Vec::new();
if bytes.len() < 16 || &bytes[..4] != b"XPR2" {
return out;
}
let mut cur = Cursor::new(bytes);
let header = match Xpr2Header::read(&mut cur) {
Ok(h) => h,
Err(_) => return out,
};
const DIR_BASE: usize = 0x10;
for _ in 0..header.num_resources {
let e = match Xpr2ResourceEntry::read(&mut cur) {
Ok(e) => e,
Err(_) => break,
};
if &e.type_tag != b"XBG7" {
continue;
}
if let Some(n) = read_cstr(bytes, e.name_offset as usize + DIR_BASE) {
out.push(n);
}
}
out
}
impl Xbg7Model {
/// Total vertex / triangle counts across all sub-meshes.
pub fn totals(&self) -> (usize, usize) {
@@ -402,6 +433,18 @@ impl Xbg7Model {
Self::anchor_models_cancellable(bytes, min_consistency, &|| false)
}
/// Decode only the named XBG7 resources from a container (content-anchored,
/// abortable). Used to assemble a whole ship from just its part family
/// (`e106_bdy_*`, `e106_brg_*`, …) instead of decoding the whole ~300-resource
/// stage. Order follows the container's directory. See [`crate::ship`].
pub fn models_named(
bytes: &[u8],
wanted: &std::collections::HashSet<String>,
should_cancel: &(dyn Fn() -> bool + Sync),
) -> Vec<Xbg7Model> {
Self::anchor_models_filtered(bytes, 0.0, should_cancel, Some(wanted))
}
/// [`Xbg7Model::anchor_models`] with a cancellation poll checked between
/// resources (a large stage container holds hundreds). See
/// [`Xbg7Model::stage_models_cancellable`].
@@ -409,6 +452,15 @@ impl Xbg7Model {
bytes: &[u8],
min_consistency: f32,
should_cancel: &(dyn Fn() -> bool + Sync),
) -> Vec<Xbg7Model> {
Self::anchor_models_filtered(bytes, min_consistency, should_cancel, None)
}
fn anchor_models_filtered(
bytes: &[u8],
min_consistency: f32,
should_cancel: &(dyn Fn() -> bool + Sync),
wanted: Option<&std::collections::HashSet<String>>,
) -> Vec<Xbg7Model> {
let mut out = Vec::new();
if bytes.len() < 16 || &bytes[..4] != b"XPR2" {
@@ -464,6 +516,11 @@ impl Xbg7Model {
}
let name = read_cstr(bytes, e.name_offset as usize + DIR_BASE)
.unwrap_or_else(|| "XBG7".to_string());
if let Some(w) = wanted {
if !w.contains(&name) {
continue;
}
}
resources.push(Res {
name,
markers,
@@ -1346,6 +1403,13 @@ pub fn submesh_albedos(bytes: &[u8], resource_name: &str) -> Vec<(usize, usize,
}
/// Locate the descriptor byte range of the named XBG7 resource in an XPR2 file.
/// Diagnostic access to a resource's raw XBG7 descriptor byte range — used by
/// the node-graph reverse-engineering examples. Not part of the decode API.
#[doc(hidden)]
pub fn xbg7_descriptor_range(bytes: &[u8], resource_name: &str) -> Option<(usize, usize)> {
xbg7_descriptor(bytes, resource_name)
}
fn xbg7_descriptor(bytes: &[u8], resource_name: &str) -> Option<(usize, usize)> {
if bytes.len() < 16 || &bytes[..4] != b"XPR2" {
return None;
@@ -1545,12 +1609,58 @@ fn rot_x(a: f32) -> M3 {
let (s, c) = a.sin_cos();
[[1.0, 0.0, 0.0], [0.0, c, -s], [0.0, s, c]]
}
/// Rotation about Y (vertical) — mixes X and Z (yaw).
fn rot_y(a: f32) -> M3 {
let (s, c) = a.sin_cos();
[[c, 0.0, s], [0.0, 1.0, 0.0], [-s, 0.0, c]]
}
/// Rotation about Z (fore/aft) — mixes X and Y (V-tail cant / roll).
fn rot_z(a: f32) -> M3 {
let (s, c) = a.sin_cos();
[[c, -s, 0.0], [s, c, 0.0], [0.0, 0.0, 1.0]]
}
/// Read a node's **9-channel TRS** `[TX TY TZ RY RX RZ SX SY SZ]` from its joint
/// table at `t44`.
///
/// The table holds **8 pointer slots**, but the node has **nine** keyframe
/// tracks (3 translation, 3 rotation, 3 scale), each a 0x50-byte record with its
/// value at `+8`. The pointers are shifted one record forward: `ptr[k]` points at
/// track `k+1`'s record, so channel `k+1`'s value is `f64 @ ptr[k]+8` and channel
/// 0 (TX!) — which no pointer names — sits one record *before* the first, at
/// `ptr[0] 0x48`.
///
/// This off-by-one is what hid every lateral offset: the old 8-slot read took
/// TY as "X", RY as "Y" (usually 0 — why hulls stacked on the centreline) and
/// never saw TX at all (the ±264 hull pair offset). Derived 2026-07-26 from the
/// e106 F10 runtime capture and verified against the captured transforms of all
/// 8 destroyer parts (see docs/re/ship-placement-runtime-capture.md).
fn read_trs9(d: &[u8], t44: usize) -> [f64; 9] {
let mut v = [0.0f64; 9];
let p0 = be32(d, t44) as usize;
if p0 >= 0x48 && p0 + 16 <= d.len() {
v[0] = be_f64(d, p0 - 0x48);
}
for k in 0..8 {
let p = be32(d, t44 + k * 4) as usize;
if p != 0 && p + 16 <= d.len() {
v[k + 1] = be_f64(d, p + 8);
}
}
v
}
/// Compose a node's local rotation from its three Euler channels
/// `(ch3, ch4, ch5) = (RY, RX, RZ)` — **`Ry·Rx·Rz`**, angles applied directly.
/// Convention pinned by the e106 engine-rig runtime capture: the nacelle node
/// stores `(RX, RZ) = (15°, +30°)` and the captured WorldView rotation equals
/// `Rx(15°)·Rz(30°)` exactly (decomposed element-for-element). The older saber
/// analysis script recovered the transpose of this convention; the fin override
/// [`saber_measured`] keeps those parts pinned either way.
fn node_rotation(ry: f64, rx: f64, rz: f64) -> M3 {
m3_mul(m3_mul(rot_y(ry as f32), rot_x(rx as f32)), rot_z(rz as f32))
}
/// The DeltaSaber fin-assembly part names — the structural signature that marks a
/// DeltaSaber-family model (`_T`/`_W`/`_A` = f001/f002/f004; same layout, slightly
/// different vertex counts) so the measured placements below apply to all three.
@@ -1628,18 +1738,9 @@ pub fn node_transforms(bytes: &[u8], resource_name: &str) -> Vec<NodePlacement>
let head_end = d.len().min(0x1684);
let prefix = format!("rou_{resource_name}_");
// 8-value TRS from a node's `+0x44` joint table (8 pointers, each f64 @ +8);
// returns identity-safe zeros when the table is absent.
let read_trs = |t44: usize| -> [f64; 8] {
let mut v = [0.0f64; 8];
for (k, slot) in v.iter_mut().enumerate() {
let p = be32(d, t44 + k * 4) as usize;
if p != 0 && p + 16 <= d.len() {
*slot = be_f64(d, p + 8);
}
}
v
};
// 9-channel TRS `[TX TY TZ RY RX RZ SX SY SZ]` from the node's `+0x44` joint
// table (see [`read_trs9`] — the 8 pointers are shifted one track forward).
let read_trs = |t44: usize| read_trs9(d, t44);
// Phase 1: collect node records in document order, each with its local
// affine and the offset where its subtree ends (its next sibling).
@@ -1695,18 +1796,18 @@ pub fn node_transforms(bytes: &[u8], resource_name: &str) -> Vec<NodePlacement>
let trs = if t44 != 0 && t44 + 32 <= d.len() {
read_trs(t44)
} else {
[0.0; 8]
[0.0; 9]
};
if std::env::var("XNODEDUMP").is_ok() {
eprintln!(
"NODE {name} vtx={vtx} t44={t44:#x} t48={t48:#x} ff@{ff:#x} sib@{sib_ptr:#x} TRS=[{:.4} {:.4} {:.4} | {:.4} {:.4} | {:.4} {:.4} {:.4}]",
trs[0], trs[1], trs[2], trs[3], trs[4], trs[5], trs[6], trs[7]
"NODE {name} vtx={vtx} t44={t44:#x} t48={t48:#x} ff@{ff:#x} sib@{sib_ptr:#x} TRS=[{:.4} {:.4} {:.4} | {:.4} {:.4} {:.4} | {:.4} {:.4} {:.4}]",
trs[0], trs[1], trs[2], trs[3], trs[4], trs[5], trs[6], trs[7], trs[8]
);
}
// Local transform: translation (t0=X, t2=up→Y, t1=fore/aft→Z), and a
// rotation Rz(r1)·Rx(r0) (r1 = V-tail cant about fore/aft, r0 = pitch).
let local_t = [trs[0] as f32, trs[2] as f32, trs[1] as f32];
let local_m = m3_mul(rot_z(trs[4] as f32), rot_x(trs[3] as f32));
// Local transform: translation (TX, TY, TZ) + Euler rotation (see
// [`node_rotation`]).
let local_t = [trs[0] as f32, trs[1] as f32, trs[2] as f32];
let local_m = node_rotation(trs[3], trs[4], trs[5]);
// Next sibling: its name starts 4 bytes before the pointer, and there a
// real node record begins with "rou_". Everything between here and there
// is this node's subtree.
@@ -1783,6 +1884,136 @@ pub fn node_transforms(bytes: &[u8], resource_name: &str) -> Vec<NodePlacement>
out
}
/// A geometry resource placed by a composite scene graph, with its world affine.
#[derive(Debug, Clone)]
pub struct ScenePart {
/// The geometry resource to draw (e.g. `e106_bdy_01`).
pub resource: String,
/// World rotation (row-major 3×3).
pub m: [[f32; 3]; 3],
/// World translation, `(X, up→Y, fore/aft→Z)`.
pub t: [f32; 3],
/// Per-axis local scale (a shield generator ships at 0.5, a jet FX at 2.4).
pub s: [f32; 3],
}
impl ScenePart {
/// Apply the world transform to a local vertex: `R·(S·v) + T` (scale is a
/// leaf-local factor; the composite's structural nodes are all unit-scale).
pub fn apply(&self, v: [f32; 3]) -> [f32; 3] {
let sv = [v[0] * self.s[0], v[1] * self.s[1], v[2] * self.s[2]];
[
self.m[0][0] * sv[0] + self.m[0][1] * sv[1] + self.m[0][2] * sv[2] + self.t[0],
self.m[1][0] * sv[0] + self.m[1][1] * sv[1] + self.m[1][2] * sv[2] + self.t[1],
self.m[2][0] * sv[0] + self.m[2][1] * sv[1] + self.m[2][2] * sv[2] + self.t[2],
]
}
}
/// Walk a composite scene graph (`e_rou_eNNN`) and return **every** node with its
/// composed world transform (parent∘child down the first-child/next-sibling tree).
///
/// A capital ship's parts are authored around the origin in their own resources;
/// the composite's node tree carries the world placement — `rou_<id>_*` nodes name
/// the hull geometry resources, and `GN_*` nodes are the Vessel hardpoint frames
/// (bridge / engine / shield-generator / turret mounts). The [`crate::ship`]
/// assembler resolves those names to resources and places the parts. Same node
/// record layout + TRS convention as [`node_transforms`], but cross-resource.
pub fn scene_world_nodes(bytes: &[u8], composite_name: &str) -> Vec<ScenePart> {
let Some((desc, desc_end)) = xbg7_descriptor(bytes, composite_name) else {
return Vec::new();
};
let d = &bytes[desc..desc_end];
let head_end = d.len();
let read_trs = |t44: usize| read_trs9(d, t44);
// A node record begins with a name starting `rou_` or `GN_`.
let node_name_at = |i: usize| -> Option<(String, usize)> {
let starts = i + 4 <= d.len() && &d[i..i + 4] == b"rou_"
|| i + 3 <= d.len() && &d[i..i + 3] == b"GN_";
if !starts {
return None;
}
let mut j = i;
while j < d.len() && d[j] != 0 && d[j].is_ascii_graphic() {
j += 1;
}
let name = std::str::from_utf8(&d[i..j]).ok()?.to_string();
if name.ends_with("_col")
|| name.ends_with("_spc")
|| name.ends_with("_gls")
|| name.ends_with("_lum")
{
return None;
}
Some((name, j))
};
struct Rec {
name_start: usize,
name: String,
local_m: M3,
local_t: [f32; 3],
local_s: [f32; 3],
sib_end: Option<usize>,
}
let mut recs: Vec<Rec> = Vec::new();
let mut i = 0usize;
while i + 3 <= head_end {
let Some((name, j)) = node_name_at(i) else {
i += 1;
continue;
};
let mut ff = i;
let scan_end = (i + 0x90).min(d.len().saturating_sub(4));
while ff < scan_end && be32(d, ff) != 0xFFFF_FFFF {
ff += 4;
}
if ff >= scan_end || ff < 0x10 {
i = j.max(i + 1);
continue;
}
let t44 = be32(d, ff - 0x10) as usize;
let sib_ptr = be32(d, ff - 0x04) as usize;
let trs = if t44 != 0 && t44 + 32 <= d.len() {
read_trs(t44)
} else {
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0]
};
let local_t = [trs[0] as f32, trs[1] as f32, trs[2] as f32];
let local_m = node_rotation(trs[3], trs[4], trs[5]);
// Channels 68 are per-axis scale (a hardpoint part ships at e.g. 0.5).
let sc = |v: f64| if v.abs() < 1e-6 { 1.0 } else { v as f32 };
let local_s = [sc(trs[6]), sc(trs[7]), sc(trs[8])];
// Next sibling begins 4 bytes before its pointer, at a `rou_`/`GN_` name.
let sib_end = sib_ptr.checked_sub(4).filter(|&s| {
s > i && s + 4 <= head_end && (&d[s..s + 4] == b"rou_" || &d[s..s + 3] == b"GN_")
});
recs.push(Rec { name_start: i, name, local_m, local_t, local_s, sib_end });
i = j.max(i + 1);
}
// Compose transforms down the first-child / next-sibling tree.
let mut out = Vec::new();
let mut stack: Vec<(usize, M3, [f32; 3])> = Vec::new();
for rec in &recs {
while stack.last().is_some_and(|s| s.0 <= rec.name_start) {
stack.pop();
}
let (pm, pt) = stack.last().map(|s| (s.1, s.2)).unwrap_or((M3_ID, [0.0; 3]));
let wm = m3_mul(pm, rec.local_m);
let r = m3_vec(pm, rec.local_t);
let wt = [r[0] + pt[0], r[1] + pt[1], r[2] + pt[2]];
out.push(ScenePart { resource: rec.name.clone(), m: wm, t: wt, s: rec.local_s });
let end = rec
.sib_end
.unwrap_or_else(|| stack.last().map(|s| s.0).unwrap_or(head_end));
stack.push((end, wm, wt));
}
out
}
// ── Tests ───────────────────────────────────────────────────────────────────
#[cfg(test)]

View File

@@ -1,36 +1,81 @@
//! The movie manifest: the game's authoritative movie → subtitle → **voice**
//! index (statically reversed).
//! The movie manifest: the game's authoritative **mission → phase → cutscene**
//! table (statically reversed). Beyond the movie→voice binding, it defines the
//! whole cutscene *structure*: which video (+ subtitle, + on-screen text overlay,
//! + voice) plays at each mission phase, and of what kind (story intro, in-mission
//! phase cutscene, resupply, …).
//!
//! `dat/tables.pak` holds one `IDXD` table (a `Z1`+zlib block) whose string pool
//! lists every cutscene in play order. Each movie contributes a run of ASCII
//! strings, always led by `<movie>.wmv` and optionally followed by:
//! - `<pak>+…​.prt` — an overlay/print-art reference,
//! - `<pak>+SUBTITLE_<movie>.tbl` — its caption timing track,
//! - `VOICE_<token>` — the **voice track basename**.
//! ## On-disc structure (`dat/tables.pak`, entry hash `0x5b983a08`)
//!
//! The voice token is the payoff: it is the real bank name to look up in
//! `sounds.tbl`, and it is **not** always `VOICE_<movie>`. Most story/radio
//! movies use `VOICE_<movie>` (in `<lang>\Movie\`), but several `hokyu_*`
//! (resupply) movies bind to in-mission radio clips like `VOICE_D_450` (in
//! `<lang>\etc\`), and many `hokyu_*` movies + the boot logos have **no** voice
//! token at all — i.e. the game plays no voice-over for them. Guessing
//! `VOICE_<movie>` therefore both misses the `VOICE_D_*` cases and invents a
//! non-existent track for the silent ones; this manifest is the ground truth.
//! One `IDXD` table (`Z1`+zlib block), schema `0x067025b9`. Unlike a flat IDXD
//! object, this one is an **array of records**, and its string pool is laid out
//! as **two consecutive arrays**:
//! 1. the **slot KEYS** — `LOGO1`, `ADVERTISE_MOVIE`, `STAFF_ROLL`, then one
//! per cutscene: `MS<NN><X>`, `STAGE<NN>_PHASE0<N>`, `STAGE<NN>_PHASE_END…`,
//! `S<NN>_SUPPLY_<ACROPOLIS|TANKER>`, in play order;
//! 2. the **value RECORDS**, in the same order — each a run led by `<movie>.wmv`
//! and (optionally) a `<pak>+…​.prt` overlay (field `TELOP`), a
//! `<pak>+SUBTITLE_<movie>.tbl` (field `SUBTITLE`), and a `VOICE_<token>`
//! (field `VOICETRACK`). A few slot keys are **valueless** (e.g. stages 5 and
//! 12 have no resupply video) — those slots have no record and are skipped.
//!
//! We read the manifest by grouping the string pool on `.wmv` (the pool is
//! emitted in the same order as the records), which is robust without decoding
//! the `IDXD` record structure.
//! The **slot key is the semantic role** — this is how the game separates a
//! cutscene voice from in-mission radio chatter (structurally, by which slot/bank,
//! not one boolean): `MS*` = story intro (`VOICE_S*`), `STAGE*_PHASE*` = radio-box
//! cutscene (`VOICE_RT*`), `S*_SUPPLY_*` = resupply (`VOICE_D_45x` or none).
//! Resupply videos are **reused across stages** (`S04_SUPPLY_ACROPOLIS` →
//! `hokyu_DS_s02A`), so the mapping is genuinely a table, not derivable from names.
//!
//! The voice token is likewise **not** always `VOICE_<movie>`: most movies use
//! `VOICE_<movie>` (in `<lang>\Movie\`), but the resupply cutscenes bind to
//! in-mission radio clips like `VOICE_D_450` (in `<lang>\etc\`), and some have no
//! voice at all. This manifest is the ground truth.
//!
//! ## Alignment
//!
//! Slot keys and value records would be a 1:1 positional zip but for the valueless
//! slots. We recover the gaps without decoding the binary node/index region by
//! using the fully-derivable **`MS<NN><X>` → `S<NN><X>` anchors**: a non-`MS` slot
//! whose record would be an `MS` target actually belongs to the upcoming `MS`
//! slot, so the non-`MS` slot is a gap. (Decoding the node region would make this
//! exact and also unlock the defaulted numeric fields in other IDXD tables.)
use crate::slb::VoiceLang;
/// One movie's manifest row.
/// The role a cutscene slot plays in the mission flow, from its slot key.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MovieKind {
/// `LOGO*`, `ADVERTISE_MOVIE`, `STAFF_ROLL` — boot/system movies.
System,
/// `MS<NN><X>` — story intro movie (camera on the speaker; stereo `VOICE_S*`).
Intro,
/// `STAGE<NN>_PHASE0<N>` — in-mission phase cutscene (radio box; `VOICE_RT*`).
Phase,
/// `STAGE<NN>_PHASE_END[_0N]` — phase-end cutscene.
PhaseEnd,
/// `S<NN>_SUPPLY_<ACROPOLIS|TANKER>` — resupply (hokyu) cutscene.
Supply,
}
/// One cutscene slot's manifest row: its semantic slot plus the bound assets.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct MovieEntry {
/// Movie basename without extension, e.g. `S13A`, `hokyu_DS_s13A`.
/// The slot key, e.g. `STAGE01_PHASE01`, `MS00A`, `S02_SUPPLY_ACROPOLIS`.
/// Empty when parsed from a blob without the slot-key array (fallback).
pub slot: String,
/// The slot's role in the mission flow (intro vs. phase cutscene vs. supply …).
pub kind: MovieKind,
/// Mission/stage number (`STAGE<NN>` / `MS<NN>` / `S<NN>_SUPPLY`), if any.
pub mission: Option<u8>,
/// Phase number for [`MovieKind::Phase`]/[`MovieKind::PhaseEnd`], if any.
pub phase: Option<u8>,
/// Movie basename without extension, e.g. `S13A`, `RT01A`, `hokyu_DS_s13A`.
pub movie: String,
/// Voice bank basename (e.g. `VOICE_S13A`, `VOICE_D_450`), or `None` when the
/// game assigns no voice-over to this movie.
/// Voice bank basename (`VOICE_S13A`, `VOICE_D_450`), or `None` for no voice.
pub voice_token: Option<String>,
/// Subtitle table basename (`SUBTITLE_S13A.tbl`), pak prefix stripped, if any.
pub subtitle: Option<String>,
/// On-screen text-overlay (`TELOP`) basename (`pwrt01.prt`), if any.
pub telop: Option<String>,
}
/// Does this blob look like the movie manifest? (`IDXD` whose string pool
@@ -44,25 +89,154 @@ pub fn is_manifest(bytes: &[u8]) -> bool {
&& contains(bytes, b"SUBTITLE_")
}
/// Parse the manifest's string pool into per-movie rows, in play order.
/// The field-type keys that appear once in the value region as a schema template.
const FIELD_KEYS: [&str; 4] = ["MOVIE", "VOICETRACK", "SUBTITLE", "TELOP"];
/// Parse the manifest into per-slot cutscene rows, in play order.
///
/// Each returned [`MovieEntry`] carries a movie; valueless slots (e.g. stage 5's
/// absent resupply) are omitted. When the two-array slot/record layout is present
/// (the real manifest) every row is tagged with its authoritative slot/kind/
/// mission/phase; otherwise (a blob without the key array) the kind is inferred
/// from the movie name and `slot` is empty.
pub fn parse(bytes: &[u8]) -> Vec<MovieEntry> {
let mut entries: Vec<MovieEntry> = Vec::new();
for run in ascii_runs(bytes, 3) {
if let Some(name) = run.strip_suffix(".wmv") {
entries.push(MovieEntry {
let toks = ascii_runs(bytes, 3);
// Locate the slot-key array (`LOGO1` … first `<movie>.wmv`) and the value
// region (from the first `.wmv` onward). Fall back to a keyless scrape.
let first_wmv = toks.iter().position(|t| t.ends_with(".wmv"));
let key_start = toks.iter().position(|t| t == "LOGO1");
let (slot_keys, value_toks): (Vec<&str>, &[String]) = match (key_start, first_wmv) {
(Some(k), Some(w)) if k < w => (
toks[k..w].iter().map(String::as_str).collect(),
&toks[w..],
),
_ => (Vec::new(), toks.as_slice()),
};
// Build the value records (drop the schema-template field keys).
let mut records: Vec<MovieEntry> = Vec::new();
for t in value_toks.iter().filter(|t| !FIELD_KEYS.contains(&t.as_str())) {
if let Some(name) = t.strip_suffix(".wmv") {
records.push(MovieEntry {
slot: String::new(),
kind: kind_from_movie(name),
mission: None,
phase: None,
movie: name.to_string(),
voice_token: None,
subtitle: None,
telop: None,
});
} else if run.starts_with("VOICE_") {
// Belongs to the movie record currently being built.
if let Some(last) = entries.last_mut() {
if last.voice_token.is_none() {
last.voice_token = Some(run.clone());
} else if let Some(rec) = records.last_mut() {
if let Some(v) = t.strip_prefix("VOICE_") {
if rec.voice_token.is_none() {
rec.voice_token = Some(format!("VOICE_{v}"));
}
} else if t.ends_with(".tbl") {
rec.subtitle.get_or_insert_with(|| after_plus(t));
} else if t.ends_with(".prt") {
rec.telop.get_or_insert_with(|| after_plus(t));
}
}
}
entries
// Attach slot keys to records, recovering valueless-slot gaps via MS anchors.
if !slot_keys.is_empty() {
let ms_targets: std::collections::HashSet<String> = slot_keys
.iter()
.filter_map(|k| ms_target(k))
.collect();
let mut j = 0usize;
for slot in &slot_keys {
if j >= records.len() {
break;
}
// A gap: this slot's "record" is actually the next MS slot's movie.
let would_be = &records[j].movie;
let is_gap = match ms_target(slot) {
// MS slot: only binds if its record is the expected `S<NN><X>`.
Some(expected) => *would_be != expected,
// Non-MS slot: a gap iff the record belongs to an upcoming MS slot.
None => ms_targets.contains(would_be),
};
if is_gap {
continue;
}
let (kind, mission, phase) = classify_slot(slot);
let rec = &mut records[j];
rec.slot = (*slot).to_string();
rec.kind = kind;
rec.mission = mission;
rec.phase = phase;
j += 1;
}
}
records
}
/// If `slot` is an `MS<NN><X>` intro slot, the movie it must bind to (`S<NN><X>`).
fn ms_target(slot: &str) -> Option<String> {
let rest = slot.strip_prefix("MS")?;
// `<NN><X>`: two digits then a letter suffix.
if rest.len() >= 3 && rest[..2].bytes().all(|b| b.is_ascii_digit()) {
Some(format!("S{rest}"))
} else {
None
}
}
/// Semantic role + mission/phase from a slot key.
fn classify_slot(slot: &str) -> (MovieKind, Option<u8>, Option<u8>) {
if let Some(rest) = slot.strip_prefix("MS") {
if rest.len() >= 2 {
return (MovieKind::Intro, rest[..2].parse().ok(), None);
}
}
if let Some(rest) = slot.strip_prefix("STAGE") {
let nn: String = rest.chars().take_while(|c| c.is_ascii_digit()).collect();
if let Ok(m) = nn.parse::<u8>() {
if slot.contains("PHASE_END") {
// trailing `_NN` if present (`STAGE12_PHASE_END_02`)
let phase = slot.rsplit('_').next().and_then(|x| x.parse().ok());
return (MovieKind::PhaseEnd, Some(m), phase);
}
// `STAGE<NN>_PHASE0<N>`
let phase = rest[nn.len()..]
.strip_prefix("_PHASE")
.and_then(|p| p.parse().ok());
return (MovieKind::Phase, Some(m), phase);
}
}
if slot.contains("_SUPPLY") {
if let Some(rest) = slot.strip_prefix('S') {
let nn: String = rest.chars().take_while(|c| c.is_ascii_digit()).collect();
return (MovieKind::Supply, nn.parse().ok(), None);
}
}
(MovieKind::System, None, None)
}
/// Fallback kind from a movie basename (used when no slot-key array is present).
fn kind_from_movie(movie: &str) -> MovieKind {
if movie.starts_with("hokyu_") {
MovieKind::Supply
} else if movie.starts_with("RT") {
MovieKind::Phase
} else if movie.len() >= 3
&& movie.starts_with('S')
&& movie[1..3].bytes().all(|b| b.is_ascii_digit())
{
MovieKind::Intro
} else {
MovieKind::System
}
}
/// The part of a `<pak>+<name>` reference after the `+` (or the whole string).
fn after_plus(s: &str) -> String {
s.rsplit('+').next().unwrap_or(s).to_string()
}
/// The voice bank basename bound to `movie`, if the manifest lists one.
@@ -151,12 +325,82 @@ mod tests {
#[test]
fn groups_movies_and_binds_voice() {
// Keyless fallback path: movie + voice + subtitle + inferred kind.
let m = parse(&synth());
assert_eq!(m.len(), 3);
assert_eq!(m[0].movie, "S13A");
assert_eq!(m[0].voice_token.as_deref(), Some("VOICE_S13A"));
assert_eq!(m[0].kind, MovieKind::Intro);
assert_eq!(m[0].subtitle.as_deref(), Some("SUBTITLE_S13A.tbl"));
assert_eq!(m[1].voice_token.as_deref(), Some("VOICE_D_450"));
assert_eq!(m[1].kind, MovieKind::Supply);
assert_eq!(m[2].voice_token, None, "hokyu_DS_s13A has no voice-over");
assert_eq!(m[2].kind, MovieKind::Supply);
}
/// Build a two-array (keys-then-records) manifest like the real one.
fn synth_two_array(keys: &[&str], records: &[&[&str]]) -> Vec<u8> {
let mut b = b"IDXD".to_vec();
b.extend_from_slice(&[0, 0, 0, 0x05, 0, 0]);
for k in keys {
b.extend_from_slice(k.as_bytes());
b.push(0);
}
for rec in records {
for s in *rec {
b.extend_from_slice(s.as_bytes());
b.push(0);
}
}
b
}
#[test]
fn two_array_slots_kind_mission_phase() {
let blob = synth_two_array(
&["LOGO1", "MS01A", "STAGE01_PHASE01", "STAGE01_PHASE_END_02"],
&[
&["logo1.wmv", "MOVIE"],
&["S01A.wmv", "eng.pak+SUBTITLE_S01A.tbl", "VOICE_S01A"],
&["RT01A.wmv", "eng.pak+pwrt01.prt", "VOICE_RT01A", "VOICETRACK"],
&["RT01C_2.wmv", "VOICE_RT01C_2"],
],
);
let m = parse(&blob);
assert_eq!(m.len(), 4);
assert_eq!((m[0].slot.as_str(), m[0].kind), ("LOGO1", MovieKind::System));
assert_eq!(
(m[1].slot.as_str(), m[1].kind, m[1].mission),
("MS01A", MovieKind::Intro, Some(1))
);
assert_eq!(
(m[2].slot.as_str(), m[2].kind, m[2].mission, m[2].phase),
("STAGE01_PHASE01", MovieKind::Phase, Some(1), Some(1))
);
assert_eq!(m[2].telop.as_deref(), Some("pwrt01.prt"));
assert_eq!(
(m[3].slot.as_str(), m[3].kind, m[3].phase),
("STAGE01_PHASE_END_02", MovieKind::PhaseEnd, Some(2))
);
}
#[test]
fn valueless_supply_slot_is_skipped_via_ms_anchor() {
// S05_SUPPLY has no record; its "record" (S06A) belongs to the next MS slot.
let blob = synth_two_array(
&["LOGO1", "MS05A", "S05_SUPPLY_ACROPOLIS", "MS06A"],
&[
&["logo1.wmv"],
&["S05A.wmv", "VOICE_S05A"],
&["S06A.wmv", "VOICE_S06A"],
],
);
let m = parse(&blob);
assert_eq!(m.len(), 3, "the valueless supply slot yields no entry");
assert_eq!((m[1].slot.as_str(), m[1].movie.as_str()), ("MS05A", "S05A"));
// S06A binds to MS06A, NOT to the intervening valueless supply slot.
assert_eq!((m[2].slot.as_str(), m[2].movie.as_str()), ("MS06A", "S06A"));
assert!(m.iter().all(|e| e.slot != "S05_SUPPLY_ACROPOLIS"));
}
#[test]

View File

@@ -162,6 +162,43 @@ pub fn track_demo_ids(lang_pak: &PakArchive, movie_basename: &str) -> Vec<u32> {
.collect()
}
/// Per-line `(demo id, start seconds)` for a movie's timing track — the radio-box
/// voice schedule. Each `MSG_DEMO_<id>` reference takes the start time of the
/// timing that follows it (a caption may list two demo lines under one timing;
/// both then share that start). Empty for inline-text / absent tracks (those are
/// story movies whose voice is the single manifest `VOICETRACK`, not per-line).
pub fn track_voice_cues(lang_pak: &PakArchive, movie_basename: &str) -> Vec<(u32, f32)> {
let Some(Ok(track)) = lang_pak.read_by_hash(track_key(movie_basename)) else {
return Vec::new();
};
if !is_ixud(&track) {
return Vec::new();
}
let toks = utf16le_tokens(&track);
let start = toks
.iter()
.position(|t| t.ends_with("SUBTITLE"))
.map(|i| i + 1)
.unwrap_or(0);
let mut out = Vec::new();
let mut pending: Vec<u32> = Vec::new();
for tok in &toks[start..] {
match parse_timing(tok) {
Some((s, _)) => {
for d in pending.drain(..) {
out.push((d, s));
}
}
None => {
if let Some(d) = demo_ref(tok) {
pending.push(d);
}
}
}
}
out
}
/// Build `demo id → ordered caption lines` from a `GP_MAIN_GAME_<L>` pack.
///
/// Scans every IXUD block, pairing each text token with the immediately

View File

@@ -0,0 +1,142 @@
//! Cutscene-voice resolution: movie → sound-id → continuous-stream byte region.
//!
//! Reverse-engineered 2026-07-22 (Canary file-I/O trace, user-confirmed by ear).
//! The movie voices form **one continuous XMA stream**, physically chunked into
//! `<lang>\Movie\VOICE_*.slb` `sound.pak` TOC entries whose boundaries do **not**
//! align with the cutscene cues — a single cue routinely spans two `.slb` chunks,
//! so a `.slb` does not necessarily hold the track its name claims. Each cue's
//! audio ends at an inline **trailer descriptor** `(sound_id: u32be, 0x11, …)`
//! whose id repeats at +0x800. Cue N's audio = the stream bytes
//! `[descriptor(N-1) .. descriptor(N)]`, read continuously across chunk
//! boundaries. Verified: decoded voice length matches each movie within ~0.3 s,
//! including cross-boundary cues (RT01B/RT01C span `VOICE_ADV`→`VOICE_RT01A`→…).
//!
//! Resolution chain:
//! 1. movie → cue name (`ADVERTISE_MOVIE` manifest `VOICETRACK`, e.g. `VOICE_RT01A`)
//! 2. cue name → sound-id (master sound registry, [`registry_voice_ids`], → 1601)
//! 3. sound-id → `[start, end]` global offsets via [`find_descriptor`] over the
//! `sound.pNN` stream (the caller supplies the bytes + the physical anchor).
use std::collections::HashMap;
/// Parse the master sound registry — the large per-language `tables.pak` IDXD
/// entry (schema `0x13cb84ba`) that carries the `<lang>\Movie\VOICE_*.slb` paths —
/// into `cue-name → sound-id` from its `NAME\0 ID\0` string pool
/// (e.g. `VOICE_ADV`→1600, `VOICE_RT01A`→1601).
pub fn registry_voice_ids(registry: &[u8]) -> HashMap<String, u32> {
let mut toks: Vec<&[u8]> = Vec::new();
let mut start = 0usize;
for i in 0..registry.len() {
if registry[i] == 0 {
if i > start {
toks.push(&registry[start..i]);
}
start = i + 1;
}
}
let mut out = HashMap::new();
for w in toks.windows(2) {
let (name, num) = (w[0], w[1]);
if name.starts_with(b"VOICE_")
&& !num.is_empty()
&& num.iter().all(u8::is_ascii_digit)
{
if let (Ok(n), Ok(id)) = (
std::str::from_utf8(name),
std::str::from_utf8(num).unwrap().parse::<u32>(),
) {
out.entry(n.to_string()).or_insert(id);
}
}
}
out
}
/// The constant marker following a cue's sound-id in its inline trailer.
const DESC_MARK: u32 = 0x11;
/// The trailer repeats its id this many bytes later; used to reject a coincidental
/// `(id, 0x11)` pair inside XMA audio (probability of a false match is ~2⁻⁶⁴).
const DESC_REPEAT: usize = 0x800;
/// Byte offset (within `buf`, a slice of the continuous voice stream) of cue
/// `id`'s trailer descriptor: the first big-endian `(id, 0x11)` pair whose id
/// repeats at `+0x800`. `None` if absent in the slice.
pub fn find_descriptor(buf: &[u8], id: u32) -> Option<usize> {
if buf.len() < DESC_REPEAT + 4 {
return None;
}
let be = |o: usize| u32::from_be_bytes([buf[o], buf[o + 1], buf[o + 2], buf[o + 3]]);
let end = buf.len() - (DESC_REPEAT + 4);
let mut o = 0;
while o <= end {
if be(o) == id && be(o + 4) == DESC_MARK && be(o + DESC_REPEAT) == id {
return Some(o);
}
o += 4;
}
None
}
/// Plausible sound-id range for a trailer (all real cue ids are well under this).
const ID_MAX: u32 = 0x1_0000;
/// Offset of the nearest cue trailer strictly **before** `before` — any plausible
/// id. Used to bound a cue's START when its `id-1` predecessor doesn't exist
/// (the sound-id sequence has gaps, e.g. `VOICE_D_453`=7142 → `VOICE_D_454`=7188).
/// Scans downward and returns the first (largest-offset) valid `(id, 0x11)` with
/// the `+0x800` id-repeat. `None` if there is no trailer below `before`.
pub fn find_descriptor_before(buf: &[u8], before: usize) -> Option<usize> {
if before < 4 {
return None;
}
let cap = buf.len().checked_sub(DESC_REPEAT + 4)?;
let be = |o: usize| u32::from_be_bytes([buf[o], buf[o + 1], buf[o + 2], buf[o + 3]]);
// Start strictly below `before` (4-aligned) so the target's own trailer is
// never returned as its predecessor.
let mut o = (before - 1).min(cap) & !3;
loop {
let id = be(o);
if id >= 1 && id < ID_MAX && be(o + 4) == DESC_MARK && be(o + DESC_REPEAT) == id {
return Some(o);
}
if o < 4 {
return None;
}
o -= 4;
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn registry_parses_name_id_pairs() {
let mut b = Vec::new();
for tok in [
b"VOICE_ADV".as_slice(),
b"1600",
b"VOICE_RT01A",
b"1601",
b"NOT_A_VOICE",
b"9",
] {
b.extend_from_slice(tok);
b.push(0);
}
let ids = registry_voice_ids(&b);
assert_eq!(ids.get("VOICE_ADV"), Some(&1600));
assert_eq!(ids.get("VOICE_RT01A"), Some(&1601));
assert_eq!(ids.get("NOT_A_VOICE"), None);
}
#[test]
fn find_descriptor_matches_repeat() {
let mut b = vec![0u8; 0x900];
b[0..4].copy_from_slice(&1601u32.to_be_bytes());
b[4..8].copy_from_slice(&0x11u32.to_be_bytes());
b[DESC_REPEAT..DESC_REPEAT + 4].copy_from_slice(&1601u32.to_be_bytes());
assert_eq!(find_descriptor(&b, 1601), Some(0));
assert_eq!(find_descriptor(&b, 1600), None);
}
}

View File

@@ -0,0 +1,649 @@
//! Whole-ship assembly from XBG7 part families.
//!
//! Project Sylpheed's capital ships are never stored as a single mesh. A ship is
//! modelled, then **split into destructible parts** — hull segments, bridge,
//! engines, integral turrets — each shipped as its own XBG7 resource inside a
//! `hidden/resource3d/Stage_SNN.xpr` container. The split lets the game hide a
//! part when the player destroys it (bridge, shield generator, thruster) and swap
//! in a `_break` variant.
//!
//! Each part's vertices are authored around the origin; the world placement lives
//! in the **primary composite scene graph** `e_rou_<id>` (see [`assemble_ship`]).
//! Its node hierarchy (`rou_<id>_root`→`…_front`/`…_rear`→`rou_<id>_bdy_NN`) names
//! the separate hull resources and carries their world TRS — the cross-resource
//! analogue of how [`mesh::node_transforms`] poses one model's sub-parts. The
//! `GN_*` nodes in the same composite are the Vessel hardpoint frames (bridge /
//! shield / engine / turret mounts, matching the IDXD recipe `Frame` names).
//!
//! Resource naming (learned from the retail stage containers):
//! - `<id>` is `[a-z]NNN` (`e106`, `f105`, `n050`). The leading letter is the
//! faction: `e` = ADAN (enemy), `f` = TCAF (player side), `n` = neutral /
//! structures / debris.
//! - A base part is `<id>_<name>` (`e106_bdy_01`, `e106_brg_01`, `f105_sld_02`).
//! - `_l` / `_m` / `_d`(`NN`) suffixes are lower level-of-detail copies; `_bNN` /
//! `_dead` / `_break` are destruction geometry; `_joint` / `_open` / `_Near` are
//! logic/animation nodes. All are skipped.
//! - `e_rou_<id>` is the primary composite; `e_rou_<id>_eng` / `_wep_*` are LOCAL
//! detail rigs (their nodes are in their own frame, not ship-space) and are NOT
//! used for placement.
use crate::mesh::{scene_world_nodes, xbg7_resource_names, ScenePart};
use std::collections::HashSet;
/// Which side a ship belongs to, from the first letter of its resource id.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Faction {
/// `e…` — ADAN (the enemy).
Adan,
/// `f…` — TCAF (the player's coalition).
Tcaf,
/// `n…` — neutral: stations, structures, asteroids, debris.
Neutral,
/// Anything else (`j…`, `s…`, …).
Other,
}
impl Faction {
fn from_id(id: &str) -> Faction {
match id.as_bytes().first() {
Some(b'e') => Faction::Adan,
Some(b'f') => Faction::Tcaf,
Some(b'n') => Faction::Neutral,
_ => Faction::Other,
}
}
/// Short human label.
pub fn label(self) -> &'static str {
match self {
Faction::Adan => "ADAN",
Faction::Tcaf => "TCAF",
Faction::Neutral => "Neutral",
Faction::Other => "Other",
}
}
}
/// A ship reconstructed from one XBG7 part family in a stage container.
#[derive(Debug, Clone)]
pub struct ShipModel {
/// The family id, e.g. `e106` — matches a [`crate::game_data::Vessel`] whose
/// `model` is `rou_e106`.
pub id: String,
/// Faction inferred from the id's leading letter.
pub faction: Faction,
/// Base geometry resource names to draw together, in directory order.
pub parts: Vec<String>,
/// Whether a composite scene graph (`e_rou_<id>`) exists — i.e. this is a
/// real assembled ship. Families without one (fighter morph sets, shared
/// turret/prop parts) have no placement data and would stack at the origin.
pub has_composite: bool,
}
/// The `[a-z]NNN` id prefix of a resource name, if it has one (`e106_bdy_01`
/// → `e106`). Returns `None` for scene/logic resources (`e_rou_e106`) and
/// anything not starting with a letter + three digits.
pub fn ship_id_of(resource: &str) -> Option<&str> {
let b = resource.as_bytes();
if b.len() >= 4
&& b[0].is_ascii_lowercase()
&& b[1].is_ascii_digit()
&& b[2].is_ascii_digit()
&& b[3].is_ascii_digit()
{
// Reject a longer alnum run (`e1234…`): the id is exactly letter+3 digits,
// followed by end-of-string or a non-alphanumeric (`_`).
if b.get(4).is_none_or(|c| !c.is_ascii_alphanumeric()) {
return Some(&resource[..4]);
}
}
None
}
/// Whether a resource is a base geometry part (not a LOD copy, destruction
/// variant, or scene/logic node) that should be drawn in a whole-ship render.
pub fn is_base_part(resource: &str) -> bool {
if ship_id_of(resource).is_none() {
return false;
}
// Level-of-detail copies of a base part: a trailing `_l`/`_m`/`_d` token,
// optionally with a number (`_d01`, `_l02`).
if let Some(last) = resource.rsplit('_').next() {
let mut cs = last.chars();
if matches!(cs.next(), Some('l' | 'm' | 'd')) && cs.all(|c| c.is_ascii_digit()) {
return false;
}
}
// Destruction / logic geometry.
if resource.contains("break")
|| resource.contains("dead")
|| resource.contains("_joint")
|| resource.contains("_open")
|| resource.contains("_Near")
{
return false;
}
// Break-piece geometry carries a bare `b` / `bNN` token (`e106_brg_01_b_02`,
// `e108_eng_01_b01`). `bdy` etc. are safe — only an exact `b`-then-digits token
// counts.
if resource.split('_').skip(1).any(|t| {
t == "b" || (t.starts_with('b') && t.len() >= 2 && t[1..].bytes().all(|c| c.is_ascii_digit()))
}) {
return false;
}
true
}
/// Group every base part in an XPR2 container into whole ships, keyed by id.
///
/// Ships are returned sorted by id. A stage container also holds shared turret /
/// prop models as their own ids — every group with at least one base part is
/// returned; the caller can filter by [`ShipModel::parts`] length or faction.
pub fn ships_in_container(bytes: &[u8]) -> Vec<ShipModel> {
let names = xbg7_resource_names(bytes);
let mut ids: Vec<String> = Vec::new();
let mut ships: std::collections::HashMap<String, ShipModel> = std::collections::HashMap::new();
for n in &names {
if !is_base_part(n) {
continue;
}
let id = ship_id_of(n).unwrap().to_string();
let entry = ships.entry(id.clone()).or_insert_with(|| {
ids.push(id.clone());
let has_composite = !composites_for(&names, &id).is_empty();
ShipModel {
id: id.clone(),
faction: Faction::from_id(&id),
parts: Vec::new(),
has_composite,
}
});
entry.parts.push(n.clone());
}
ids.sort();
ids.into_iter().map(|id| ships.remove(&id).unwrap()).collect()
}
/// Map a composite scene-graph node name to the geometry resource it draws.
/// Hull nodes are `rou_<id>_<part>[_root|_mov|_NN]`; the geometry resource is the
/// `<id>_<part>` stem. Structural nodes (`rou_e106_front`, `…_root`) resolve to
/// nothing and only pose their children.
fn resolve_hull_resource(node: &str, resources: &HashSet<String>) -> Option<String> {
let stem = node.rsplit_once("rou_").map(|(_, s)| s).unwrap_or(node);
let try_stem = |s: &str| -> Option<String> {
if resources.contains(s) {
return Some(s.to_string());
}
// Squeezed digit run (`wep02` names the `wep_02_01` resource): insert an
// underscore before the digits and try the `_01` first-part too.
if let Some(pos) = s.rfind(|c: char| c.is_ascii_digit()) {
let mut start = pos;
while start > 0 && s.as_bytes()[start - 1].is_ascii_digit() {
start -= 1;
}
if start > 0 && s.as_bytes()[start - 1] != b'_' {
let split = format!("{}_{}", &s[..start], &s[start..]);
if resources.contains(&split) {
return Some(split);
}
let first = format!("{split}_01");
if resources.contains(&first) {
return Some(first);
}
}
}
None
};
if let Some(r) = try_stem(stem) {
return Some(r);
}
for suf in ["_root", "_mov"] {
if let Some(s) = stem.strip_suffix(suf) {
if let Some(r) = try_stem(s) {
return Some(r);
}
}
}
// Trailing `_NN` instance index (`eng_01_1` → `eng_01`).
if let Some(pos) = stem.rfind('_') {
if !stem[pos + 1..].is_empty() && stem[pos + 1..].bytes().all(|c| c.is_ascii_digit()) {
if let Some(r) = try_stem(&stem[..pos]) {
return Some(r);
}
}
}
None
}
/// The trailing digit run of a name (`GN_Bridge_01` → `01`, `sld_02` → `02`).
fn trailing_index(name: &str) -> &str {
let bytes = name.as_bytes();
let mut start = bytes.len();
while start > 0 && bytes[start - 1].is_ascii_digit() {
start -= 1;
}
&name[start..]
}
/// Which stage-container resources are the composite scene graphs for ship `id`
/// (`e_rou_e106`, `e_rou_e106_eng`, …). Excludes destruction (`_break`) scenes.
fn composites_for<'a>(names: &'a [String], id: &str) -> Vec<&'a String> {
let needle = format!("rou_{id}");
names
.iter()
.filter(|n| {
n.contains(&needle)
&& ship_id_of(n).is_none()
&& !n.contains("break")
&& !n.contains("dead")
})
.collect()
}
/// Reconstruct a whole ship as a list of geometry-resource placements in a shared
/// ship-local frame — the placement the game builds at runtime.
///
/// Fully static and EXACT (validated part-for-part against an e106 runtime
/// capture — see `docs/re/ship-placement-runtime-capture.md`):
/// 1. **Composite nodes** — every `rou_*` node in the primary composite
/// (`e_rou_<id>`) places a geometry resource, INCLUDING repeated instances
/// and cross-id turret mounts (`rou_e303_wep_01_root` ×2 on the e106 hull).
/// 2. **Detail rigs** — `e_rou_<id>_eng` is the engine cluster rig; its nodes
/// (e.g. two `eng_01` nacelle instances + `eng_02`) mount at the primary
/// composite's `GN_Engine_01` frame: `world = frame ∘ rig_local`.
/// 3. **GN hardpoints** — remaining unplaced `brg`/`sld` parts sit exactly at
/// their `GN_Bridge_NN` / `GN_ShieldG_NN` frame (frames carry full TRS).
/// 4. **Mirrored twins** — a port/starboard pair (`bdy_01`/`bdy_02`) shares one
/// geometry; the instance whose lateral offset points the geometry's dominant
/// side inboard is drawn X-reflected (capture-verified; det < 0 → the caller
/// reverses winding).
///
/// `include_external` keeps tiers 24 (off = bare composite-node hull).
pub fn assemble_ship(bytes: &[u8], id: &str, include_external: bool) -> Vec<ScenePart> {
let names = xbg7_resource_names(bytes);
let resources: HashSet<String> = names.iter().cloned().collect();
// The PRIMARY composite carries the ship-space layout (hull bodies, turret
// mounts + every `GN_*` hardpoint frame): the one with the most nodes.
let scenes: Vec<(String, Vec<ScenePart>)> = composites_for(&names, id)
.into_iter()
.filter(|c| !c.ends_with("_joint"))
.map(|c| (c.clone(), scene_world_nodes(bytes, c)))
.collect();
let Some((_, nodes)) = scenes.iter().max_by_key(|(_, n)| n.len()) else {
// No composite at all → raw fallback below.
let mut placed = Vec::new();
const ID: [[f32; 3]; 3] = [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]];
for part in names.iter().filter(|n| is_base_part(n) && ship_id_of(n) == Some(id)) {
placed.push(ScenePart { resource: part.clone(), m: ID, t: [0.0; 3], s: [1.0; 3] });
}
return placed;
};
let mut placed: Vec<ScenePart> = Vec::new();
let mut placed_res: HashSet<String> = HashSet::new();
// GN hardpoint frames, world-space, from the primary composite.
let mut frames: Vec<ScenePart> = Vec::new();
// Tier 1: every composite node placement, instances included. Cross-id
// resources (shared e303/e4NN turret models the composite mounts) count too.
for node in nodes {
if node.resource.starts_with("GN_") {
frames.push(node.clone());
} else if let Some(res) = resolve_hull_resource(&node.resource, &resources) {
if ship_id_of(&res) != Some(id) && !include_external {
continue; // hull-only view: own parts only
}
// Distinct INSTANCES keep their own placement; a nested alias of the
// same resource at the same transform (`bdy_01_mov` + its identity
// `bdy_01` child) collapses to one draw.
let dup = placed.iter().any(|p| {
p.resource == res
&& (p.t[0] - node.t[0]).abs() < 0.01
&& (p.t[1] - node.t[1]).abs() < 0.01
&& (p.t[2] - node.t[2]).abs() < 0.01
});
if !dup {
placed_res.insert(res.clone());
placed.push(ScenePart { resource: res, m: node.m, t: node.t, s: node.s });
}
}
}
// Tier 2: the engine cluster rig, mounted at GN_Engine_01. The rig's nodes
// are in the FRAME's local space (two mirrored `eng_01` nacelles + centre
// `eng_02` for e106) — capture-verified to 0.1 units.
if include_external {
let rig_name = format!("e_rou_{id}_eng");
if names.contains(&rig_name) {
if let Some(frame) = frames.iter().find(|f| f.resource.starts_with("GN_Engine")) {
for rn in scene_world_nodes(bytes, &rig_name) {
let Some(res) = resolve_hull_resource(&rn.resource, &resources) else {
continue;
};
// world = frame ∘ rig_local
let m = m3_mul_pub(frame.m, rn.m);
let rt = m3_vec_pub(frame.m, rn.t);
let t = [rt[0] + frame.t[0], rt[1] + frame.t[1], rt[2] + frame.t[2]];
placed_res.insert(res.clone());
placed.push(ScenePart { resource: res, m, t, s: rn.s });
}
}
}
}
// Tier 3: remaining external parts at their exact GN frame (full TRS).
const CATS: [(&str, &str); 3] = [("brg", "Bridge"), ("sld", "ShieldG"), ("eng", "Engine")];
for part in names
.iter()
.filter(|_| include_external)
.filter(|n| is_base_part(n) && ship_id_of(n) == Some(id))
{
if placed_res.contains(part) {
continue;
}
let rest = &part[(id.len() + 1).min(part.len())..]; // "brg_01"
let mut toks = rest.split('_');
let cat = toks.next().unwrap_or("");
let idx = toks.next().unwrap_or("");
let Some((_, gncat)) = CATS.iter().find(|(c, _)| *c == cat) else {
continue;
};
if let Some(frame) =
frames.iter().find(|f| f.resource.contains(gncat) && trailing_index(&f.resource) == idx)
{
placed.push(ScenePart { resource: part.clone(), m: frame.m, t: frame.t, s: frame.s });
placed_res.insert(part.clone());
}
}
// Tier 4: mirror one of a shared-geometry twin pair. The engine uploads the
// second of a port/starboard pair X-reflected (capture-verified on
// bdy_01/bdy_02): for two same-geometry resources placed at ±TX, the
// instance whose TX sign matches the geometry's dominant-X side draws
// mirrored (so the pair ends up symmetric instead of twice the same side).
apply_twin_mirrors(bytes, &mut placed);
// Fallback for a ship with no composite scene graph (a simple prop): draw its
// base parts untransformed rather than nothing.
if placed.is_empty() {
const ID: [[f32; 3]; 3] = [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]];
for part in names.iter().filter(|n| is_base_part(n) && ship_id_of(n) == Some(id)) {
placed.push(ScenePart { resource: part.clone(), m: ID, t: [0.0; 3], s: [1.0; 3] });
}
}
placed
}
/// The ship's exhaust mount frames (`GN_Jet_*` / `GN_SJet_*`), world-space, from
/// the primary composite. These are where the game renders the engine exhaust
/// FX — the visible "thrusters". The engine GEOMETRY itself sits recessed in
/// the hull (capture-verified across 43 e106 instances: the game draws
/// `eng_01`/`eng_02` exactly where [`assemble_ship`] puts them); without the
/// FX the assembled ship reads engine-less, so a viewer can draw exhaust
/// markers at these frames.
pub fn exhaust_frames(bytes: &[u8], id: &str) -> Vec<ScenePart> {
let names = xbg7_resource_names(bytes);
let scenes: Vec<Vec<ScenePart>> = composites_for(&names, id)
.into_iter()
.filter(|c| !c.ends_with("_joint"))
.map(|c| scene_world_nodes(bytes, c))
.collect();
let Some(nodes) = scenes.into_iter().max_by_key(|n| n.len()) else {
return Vec::new();
};
nodes
.into_iter()
.filter(|n| n.resource.starts_with("GN_Jet") || n.resource.starts_with("GN_SJet"))
.collect()
}
/// Detect shared-geometry port/starboard twin RESOURCES placed at ±TX and mark
/// the dominant-side instance as an X-reflection (negate the matrix X column).
/// Twins are `<stem>_01`/`<stem>_02` pairs with equal vertex data; the reflected
/// one is the instance whose TX sign equals the geometry's mean-X sign (the
/// un-reflected drawing already covers that side's opposite).
fn apply_twin_mirrors(bytes: &[u8], placed: &mut [ScenePart]) {
use crate::mesh::Xbg7Model;
// Candidate pairs: resources differing only in a trailing _01/_02, both
// placed exactly once, at opposite-sign TX of equal magnitude.
let mut pairs: Vec<(usize, usize)> = Vec::new();
for i in 0..placed.len() {
let a = &placed[i];
let Some(stem) = a.resource.strip_suffix("_01") else { continue };
let twin = format!("{stem}_02");
let Some(j) = placed.iter().position(|p| p.resource == twin) else { continue };
let b = &placed[j];
if (a.t[0] + b.t[0]).abs() < 0.5 && a.t[0].abs() > 1.0 {
pairs.push((i, j));
}
}
if pairs.is_empty() {
return;
}
let want: HashSet<String> = pairs
.iter()
.flat_map(|&(i, j)| [placed[i].resource.clone(), placed[j].resource.clone()])
.collect();
let models = Xbg7Model::models_named(bytes, &want, &|| false);
for (i, j) in pairs {
let get = |r: &str| models.iter().find(|m| m.name == r);
let (Some(ma), Some(mb)) = (get(&placed[i].resource), get(&placed[j].resource)) else {
continue;
};
let first = |m: &Xbg7Model| -> Vec<[f32; 3]> {
m.meshes.iter().flat_map(|s| s.positions.iter().copied()).take(16).collect()
};
let (fa, fb) = (first(ma), first(mb));
if fa.len() != fb.len()
|| !fa.iter().zip(&fb).all(|(p, q)| {
(p[0] - q[0]).abs() < 1e-3 && (p[1] - q[1]).abs() < 1e-3 && (p[2] - q[2]).abs() < 1e-3
})
{
continue; // distinct (already-mirrored) geometries — nothing to do
}
let mean_x: f32 = ma
.meshes
.iter()
.flat_map(|s| s.positions.iter())
.map(|p| p[0])
.sum::<f32>()
/ ma.meshes.iter().map(|s| s.positions.len()).sum::<usize>().max(1) as f32;
if mean_x.abs() < 1e-3 {
continue; // symmetric geometry — mirroring is a no-op
}
// Reflect the instance whose lateral offset OPPOSES the geometry's
// dominant side (drawn plain it would fold onto the centreline; the
// capture shows the engine reflects exactly this one — for e106 the
// 264 port instance draws the file data plain, the +264 one mirrored).
let k = if (placed[i].t[0] > 0.0) == (mean_x > 0.0) { j } else { i };
for row in &mut placed[k].m {
row[0] = -row[0];
}
}
}
// Small helpers mirroring mesh.rs's private affine ops.
fn m3_mul_pub(a: [[f32; 3]; 3], b: [[f32; 3]; 3]) -> [[f32; 3]; 3] {
let mut o = [[0.0f32; 3]; 3];
for r in 0..3 {
for c in 0..3 {
o[r][c] = a[r][0] * b[0][c] + a[r][1] * b[1][c] + a[r][2] * b[2][c];
}
}
o
}
fn m3_vec_pub(a: [[f32; 3]; 3], v: [f32; 3]) -> [f32; 3] {
[
a[0][0] * v[0] + a[0][1] * v[1] + a[0][2] * v[2],
a[1][0] * v[0] + a[1][1] * v[1] + a[1][2] * v[2],
a[2][0] * v[0] + a[2][1] * v[1] + a[2][2] * v[2],
]
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn id_extraction() {
assert_eq!(ship_id_of("e106_bdy_01"), Some("e106"));
assert_eq!(ship_id_of("f105_sld_02_l"), Some("f105"));
assert_eq!(ship_id_of("e_rou_e106"), None);
assert_eq!(ship_id_of("_rou_e106_break"), None);
assert_eq!(ship_id_of("Base"), None);
}
#[test]
fn base_part_filter() {
assert!(is_base_part("e106_bdy_01"));
assert!(is_base_part("e106_brg_01"));
assert!(is_base_part("f105_wep_02_01"));
assert!(!is_base_part("e106_bdy_01_l")); // LOD
assert!(!is_base_part("e106_bdy_01_m")); // LOD
assert!(!is_base_part("e106_brg_01_b_02")); // break piece
assert!(!is_base_part("_rou_e106_break")); // logic
assert!(!is_base_part("e_rou_e106")); // scene node
}
#[test]
fn faction_from_id() {
assert_eq!(Faction::from_id("e106"), Faction::Adan);
assert_eq!(Faction::from_id("f105"), Faction::Tcaf);
assert_eq!(Faction::from_id("n050"), Faction::Neutral);
assert_eq!(Faction::from_id("j004"), Faction::Other);
}
// Disc-gated: reconstruct the ADAN cruiser family from a real stage container.
#[test]
fn ships_from_real_stage() {
let Ok(iso) = std::env::var("SYLPHEED_ISO") else {
eprintln!("SYLPHEED_ISO unset — skipping");
return;
};
let bytes = {
use crate::xiso::open_iso;
let rt = tokio::runtime::Builder::new_current_thread().build().unwrap();
rt.block_on(async {
let mut r = open_iso(std::path::Path::new(&iso)).await.unwrap();
r.read_file("hidden/resource3d/Stage_S02.xpr").await.unwrap()
})
};
let ships = ships_in_container(&bytes);
assert!(!ships.is_empty(), "stage should hold ships");
// The ADAN frigate e108 appears in S02 with hull + engine + weapon parts.
let e108 = ships.iter().find(|s| s.id == "e108").expect("e108 present");
assert_eq!(e108.faction, Faction::Adan);
assert!(e108.parts.iter().any(|p| p.contains("_bdy_")));
assert!(e108.parts.iter().all(|p| is_base_part(p)));
}
// Disc-gated GROUND-TRUTH test: the fully-static assembly must reproduce the
// runtime F10 capture (the baked e106 table) part-for-part — translations
// AND the engine-rig rotation — plus the multi-instance parts the capture's
// vbase-dedup could not see (two nacelles, two turrets).
#[test]
fn static_assembly_matches_runtime_capture() {
let Ok(iso) = std::env::var("SYLPHEED_ISO") else {
eprintln!("SYLPHEED_ISO unset — skipping");
return;
};
let bytes = {
use crate::xiso::open_iso;
let rt = tokio::runtime::Builder::new_current_thread().build().unwrap();
rt.block_on(async {
let mut r = open_iso(std::path::Path::new(&iso)).await.unwrap();
r.read_file("hidden/resource3d/Stage_S01.xpr").await.unwrap()
})
};
let placed = assemble_ship(&bytes, "e106", true);
let cap = crate::ship_capture::embedded_placement("e106").expect("e106 baked");
// Express static placements relative to bdy_04 (the capture's reference).
let r4 = placed
.iter()
.find(|p| p.resource == "e106_bdy_04")
.expect("bdy_04 placed")
.clone();
let rel = |p: &crate::mesh::ScenePart| -> [f32; 3] {
[p.t[0] - r4.t[0], p.t[1] - r4.t[1], p.t[2] - r4.t[2]]
};
for want in &cap.parts {
let best = placed
.iter()
.filter(|p| p.resource == want.part)
.map(|p| {
let t = rel(p);
let d = (t[0] - want.t[0]).abs()
+ (t[1] - want.t[1]).abs()
+ (t[2] - want.t[2]).abs();
(d, p)
})
.min_by(|a, b| a.0.partial_cmp(&b.0).unwrap())
.unwrap_or_else(|| panic!("{} not placed statically", want.part));
assert!(
best.0 < 1.0,
"{}: static rel-T {:?} != captured {:?} (Δ={:.2})",
want.part,
rel(best.1),
want.t,
best.0
);
// Rotations must match too (the engine rig is the interesting case).
let m = &best.1.m;
for r in 0..3 {
for c in 0..3 {
assert!(
(m[r][c] - want.m[r][c]).abs() < 0.02,
"{}: static M row{r} {:?} != captured {:?}",
want.part,
m[r],
want.m[r]
);
}
}
}
// Multi-instance coverage the capture couldn't see (vbase dedup).
let count = |res: &str| placed.iter().filter(|p| p.resource == res).count();
assert_eq!(count("e106_eng_01"), 2, "both engine nacelles placed");
assert_eq!(count("e303_wep_01"), 2, "both shared turrets placed");
// The mirrored starboard hull reflects (det < 0), the port one doesn't.
let det = |m: &[[f32; 3]; 3]| {
m[0][0] * (m[1][1] * m[2][2] - m[1][2] * m[2][1])
- m[0][1] * (m[1][0] * m[2][2] - m[1][2] * m[2][0])
+ m[0][2] * (m[1][0] * m[2][1] - m[1][1] * m[2][0])
};
let one = |res: &str| placed.iter().find(|p| p.resource == res).unwrap();
assert!(det(&one("e106_bdy_02").m) < 0.0, "starboard hull mirrored");
assert!(det(&one("e106_bdy_01").m) > 0.0, "port hull plain");
}
// Disc-gated: the scene graph must SPREAD a ship's parts, not stack them at
// the origin. Reconstruct the ADAN frigate e106 and assert its bridge sits
// clearly aft of its forward hull body (a real ship layout, not a pile).
#[test]
fn assemble_spreads_parts() {
let Ok(iso) = std::env::var("SYLPHEED_ISO") else {
eprintln!("SYLPHEED_ISO unset — skipping");
return;
};
let bytes = {
use crate::xiso::open_iso;
let rt = tokio::runtime::Builder::new_current_thread().build().unwrap();
rt.block_on(async {
let mut r = open_iso(std::path::Path::new(&iso)).await.unwrap();
r.read_file("hidden/resource3d/Stage_S02.xpr").await.unwrap()
})
};
let placed = assemble_ship(&bytes, "e106", true);
assert!(placed.len() >= 5, "e106 should place its hull + external parts");
// The forward hull body and the bridge must land at distinct fore/aft Z.
let z = |res: &str| placed.iter().find(|p| p.resource == res).map(|p| p.t[2]);
let bdy = z("e106_bdy_01").expect("bdy_01 placed");
let brg = z("e106_brg_01").expect("brg_01 placed at its GN_Bridge frame");
assert!(
(bdy - brg).abs() > 200.0,
"bridge ({brg}) and forward body ({bdy}) must be far apart, not stacked"
);
}
}

View File

@@ -0,0 +1,694 @@
//! Exact capital-ship part placement from a Canary **F10 ship-capture** log —
//! the runtime ground truth that static [`crate::ship::assemble_ship`] only
//! approximates for external parts.
//!
//! ## Why a capture is needed
//!
//! A captured part's vertex BUFFER holds **local** coordinates (byte-identical to
//! the `.xpr`), so capital-ship parts are placed **entirely in the vertex shader**
//! — the buffer carries no placement. The per-part transform lives in the ship
//! shader's **vertex float constants**: `c0..c2` are the **WorldViewProjection**
//! matrix rows. Each row's norm is `(sx, sy, 1)` (the projection x/y scales, with
//! `sy/sx ≈ 1.78 = 16:9`); dividing a row by its norm yields the rigid
//! **WorldView** row (verified orthonormal, `det = +1`) and `row[3]/norm` is that
//! axis' view-space translation.
//!
//! The camera View cancels when every part is expressed relative to a **reference
//! part**: `rel_p = WV_ref⁻¹ · WV_p = (Rᵀ_ref·R_p , Rᵀ_ref·(T_p T_ref))` — a pure
//! ship-space rigid transform. That is what [`correlate`] emits and what the
//! checked-in [placement table](parse_table) stores, so the viewer can assemble a
//! ship exactly without re-capturing.
//!
//! ## Pipeline
//!
//! 1. Canary F10 → `xenia_ship_capture.log` (per-draw `vbase`/`vcount` + the first
//! 48 VS float4 constants). [`parse_capture`] → [`CapturedDraw`]s.
//! 2. [`correlate`] matches each ship base part to a draw **by vertex count**
//! (unique per part) and expresses it in the reference part's frame →
//! [`ShipPlacement`].
//! 3. [`serialize_table`]/[`parse_table`] persist it as a checked-in data file
//! (`data/ship_placements.txt`, embedded via [`embedded_placement`]); the viewer
//! prefers it over the static assembler when present.
use crate::mesh::ScenePart;
type M3 = [[f64; 3]; 3];
/// One captured draw's rigid **WorldView**: rotation rows `r` + view-space
/// translation `t`, recovered from the `c0..c2` WVP constants.
#[derive(Debug, Clone, PartialEq)]
pub struct CapturedDraw {
/// Guest vertex-buffer base address (the draw's identity for de-duping).
pub vbase: u32,
/// Vertex count — the key that matches a draw to a decoded part.
pub vcount: u32,
/// WorldView rotation rows (orthonormal).
pub r: M3,
/// WorldView view-space translation.
pub t: [f64; 3],
/// First few LOCAL vertex positions dumped with the draw (buffer order).
/// Used to disambiguate same-vcount twins (mirrored port/starboard parts).
pub pos: Vec<[f32; 3]>,
}
/// A ship part to match against the capture. `part` is the **base** part name
/// (what goes in the placement table); `vcount`/`ref_pos` come from whichever
/// resource variant is being tried (base or an `_m`/`_l` LOD copy — a LOD is the
/// same part in the same local frame, so its captured transform is the part's).
#[derive(Debug, Clone, PartialEq)]
pub struct PartKey {
/// Base part name for the table, e.g. `e106_bdy_01`.
pub part: String,
/// The tried resource's vertex count (the draw match key).
pub vcount: u32,
/// The tried resource's decoded positions (any order — validation is
/// set-based), used to validate a vcount hit and to route mirrored twins.
/// Empty = match by vcount alone.
pub ref_pos: Vec<[f32; 3]>,
}
/// A part's ship-relative rigid placement (in the reference part's frame).
#[derive(Debug, Clone, PartialEq)]
pub struct PartPlacement {
/// Geometry resource name, e.g. `e106_eng_01`.
pub part: String,
/// Rotation rows.
pub m: [[f32; 3]; 3],
/// Ship-relative translation.
pub t: [f32; 3],
}
/// A whole ship's captured placement: every part in a shared ship-local frame.
#[derive(Debug, Clone, PartialEq)]
pub struct ShipPlacement {
/// Ship family id, e.g. `e106`.
pub id: String,
/// The reference part whose frame the placements are expressed in.
pub reference: String,
/// Ship-relative placement of each matched part.
pub parts: Vec<PartPlacement>,
}
/// Parse a Canary ship-capture log into per-draw rigid WorldView transforms.
///
/// Only draws that carry the `c0..c2` constants are returned (a culled/occluded
/// part produces no draw and is simply absent). Robust to the exact spacing of
/// the `DRAW …` / `vsconst …` lines.
pub fn parse_capture(text: &str) -> Vec<CapturedDraw> {
let mut out = Vec::new();
let mut vbase = 0u32;
let mut vcount = 0u32;
let mut pos: Vec<[f32; 3]> = Vec::new();
let mut consts: Vec<(usize, [f64; 4])> = Vec::new();
let flush = |vbase: u32,
vcount: u32,
pos: &mut Vec<[f32; 3]>,
consts: &[(usize, [f64; 4])],
out: &mut Vec<CapturedDraw>| {
let pos = std::mem::take(pos);
if vbase == 0 {
return;
}
let get = |i: usize| consts.iter().find(|(k, _)| *k == i).map(|(_, v)| *v);
let (Some(c0), Some(c1), Some(c2)) = (get(0), get(1), get(2)) else {
return; // no WorldView for this draw — skip it
};
if let Some((r, t)) = normalize_wvp([c0, c1, c2]) {
out.push(CapturedDraw { vbase, vcount, r, t, pos });
}
};
for line in text.lines() {
let l = line.trim();
if let Some(rest) = l.strip_prefix("DRAW ") {
flush(vbase, vcount, &mut pos, &consts, &mut out);
consts.clear();
let f = |k: &str| rest.split_whitespace().find_map(|t| t.strip_prefix(k));
vbase = f("vbase=0x").and_then(|s| u32::from_str_radix(s, 16).ok()).unwrap_or(0);
vcount = f("vcount=").and_then(|s| s.parse().ok()).unwrap_or(0);
} else if l.starts_with("pos:") || l.starts_with("positions:") {
pos = parse_pos_line(l, 8);
} else if l.starts_with("vsconst") {
for cap in l.split('c').skip(1) {
let Some((idx, rest)) = cap.split_once('=') else { continue };
let Ok(i) = idx.trim().parse::<usize>() else { continue };
let nums: Vec<f64> = rest
.trim_start_matches('(')
.split(')')
.next()
.unwrap_or("")
.split(',')
.filter_map(|x| x.trim().parse().ok())
.collect();
if nums.len() == 4 {
consts.push((i, [nums[0], nums[1], nums[2], nums[3]]));
}
}
}
}
flush(vbase, vcount, &mut pos, &consts, &mut out);
out
}
/// Parse a `pos: (x,y,z) (x,y,z) …` dump line into up to `max` positions.
fn parse_pos_line(l: &str, max: usize) -> Vec<[f32; 3]> {
let mut out = Vec::new();
for group in l.split('(').skip(1) {
let Some(inner) = group.split(')').next() else { continue };
let nums: Vec<f32> = inner.split(',').filter_map(|x| x.trim().parse().ok()).collect();
if nums.len() == 3 {
out.push([nums[0], nums[1], nums[2]]);
if out.len() >= max {
break;
}
}
}
out
}
/// The vertex shader capital ships (and the player fighter) are drawn with — the
/// `c0..c2` WVP-row layout [`parse_capture`]/[`parse_drawlog`] rely on.
pub const SHIP_VS_HASH: &str = "0xC7F781F4C1D58054";
/// Parse the **draw-logger** format (`xenia_re_draws.log` / `mission_draws.log`,
/// the `--log_draws` cvar) into per-part rigid transforms — the alternative to the
/// F10 [`parse_capture`] snapshot. That log is what a normal instrumented run
/// already produces, so a capital ship seen in-mission can be baked without a
/// dedicated F10 capture.
///
/// A capital ship's parts each own a **distinct vertex buffer**, so we group by
/// the `stream … base=…` address, take its vertex count as `size_words /
/// stride_words`, and keep the first `c0..c2` WVP seen for that buffer. Only draws
/// with `vs=`[`SHIP_VS_HASH`] are kept (the ship shader), so HUD/skybox draws are
/// ignored. (The player fighter shares ONE buffer across its fin draws and so
/// collapses to a single entry here — fine, capital ships are the target.)
pub fn parse_drawlog(text: &str) -> Vec<CapturedDraw> {
let mut out = Vec::new();
let mut seen: std::collections::HashSet<u32> = std::collections::HashSet::new();
let mut is_ship = false;
let mut base = 0u32;
let mut stride = 0u32;
let mut size = 0u32;
let mut pos: Vec<[f32; 3]> = Vec::new();
let mut consts: Vec<(usize, [f64; 4])> = Vec::new();
let mut flush = |base: u32,
size: u32,
stride: u32,
pos: &mut Vec<[f32; 3]>,
consts: &[(usize, [f64; 4])],
seen: &mut std::collections::HashSet<u32>,
out: &mut Vec<CapturedDraw>| {
let pos = std::mem::take(pos);
if base == 0 || stride == 0 || !seen.insert(base) {
return;
}
let get = |i: usize| consts.iter().find(|(k, _)| *k == i).map(|(_, v)| *v);
let (Some(c0), Some(c1), Some(c2)) = (get(0), get(1), get(2)) else { return };
if let Some((r, t)) = normalize_wvp([c0, c1, c2]) {
out.push(CapturedDraw { vbase: base, vcount: size / stride, r, t, pos });
}
};
for line in text.lines() {
let l = line.trim();
if let Some(rest) = l.strip_prefix("DRAW ") {
flush(base, size, stride, &mut pos, &consts, &mut seen, &mut out);
consts.clear();
base = 0;
stride = 0;
size = 0;
is_ship = rest.contains(&format!("vs={SHIP_VS_HASH}"));
} else if is_ship && l.starts_with("positions:") {
pos = parse_pos_line(l, 8);
} else if is_ship && l.starts_with("stream ") {
let f = |k: &str| l.split_whitespace().find_map(|t| t.strip_prefix(k));
if let Some(b) = f("base=0x").and_then(|s| u32::from_str_radix(s, 16).ok()) {
base = b;
}
stride = f("stride_words=").and_then(|s| s.parse().ok()).unwrap_or(stride);
size = f("size_words=").and_then(|s| s.parse().ok()).unwrap_or(size);
} else if is_ship && l.starts_with('c') {
// `c<idx> x y z w` — the vsconst rows (space-separated).
let mut it = l.splitn(2, char::is_whitespace);
let Some(tag) = it.next() else { continue };
let Ok(i) = tag[1..].parse::<usize>() else { continue };
let nums: Vec<f64> = it.next().unwrap_or("").split_whitespace().filter_map(|x| x.parse().ok()).collect();
if nums.len() >= 4 {
consts.push((i, [nums[0], nums[1], nums[2], nums[3]]));
}
}
}
flush(base, size, stride, &mut pos, &consts, &mut seen, &mut out);
out
}
/// Normalize the three `c0..c2` WVP rows to a rigid WorldView `(R rows, T)` by
/// dividing each row by its (projection-scale) norm. `None` if any row is
/// degenerate.
fn normalize_wvp(rows: [[f64; 4]; 3]) -> Option<(M3, [f64; 3])> {
let mut r = [[0.0; 3]; 3];
let mut t = [0.0; 3];
for (i, row) in rows.iter().enumerate() {
let n = (row[0] * row[0] + row[1] * row[1] + row[2] * row[2]).sqrt();
if n < 1e-6 {
return None;
}
r[i] = [row[0] / n, row[1] / n, row[2] / n];
t[i] = row[3] / n;
}
Some((r, t))
}
/// Validate a vcount hit by the draw's dumped positions against the part's
/// decoded position SET (order-independent — decode order can differ from buffer
/// order). Returns `Some((hits, mirrored))` when at least half the dumped
/// positions are found among the part's vertices, either directly or **all
/// X-negated** — the engine uploads the second of a mirrored port/starboard pair
/// as an X-reflection of the shared file geometry, so the guest buffer disagrees
/// in X sign with every decoded copy. `None` = the dump belongs to a different
/// model (a coincidental vcount).
fn pos_validate(draw: &CapturedDraw, ref_pos: &[[f32; 3]]) -> Option<(usize, bool)> {
if draw.pos.is_empty() || ref_pos.is_empty() {
return Some((0, false)); // no data to validate with — accept neutrally
}
let near = |a: &[f32; 3], b: &[f32; 3]| {
(a[0] - b[0]).abs() <= 1e-2 && (a[1] - b[1]).abs() <= 1e-2 && (a[2] - b[2]).abs() <= 1e-2
};
let mut direct = 0usize;
let mut mirror = 0usize;
for p in &draw.pos {
if ref_pos.iter().any(|v| near(p, v)) {
direct += 1;
}
let pm = [-p[0], p[1], p[2]];
if ref_pos.iter().any(|v| near(&pm, v)) {
mirror += 1;
}
}
let need = draw.pos.len().div_ceil(2);
if direct >= need && direct >= mirror {
Some((direct, false))
} else if mirror >= need {
Some((mirror, true))
} else {
None
}
}
/// Correlate captured draws to a ship's parts and express each in the reference
/// part's frame.
///
/// Each [`PartKey`]'s `vcount` is the match key; when several draws share the
/// vcount (mirrored port/starboard twins) the draw whose dumped positions match
/// the part's `ref_pos` validates best is chosen. `ref_sub` selects the reference part by
/// substring (e.g. `bdy_04`); the first matched part is used if none contains it.
/// Parts with no matching captured draw (culled at that camera angle) are
/// omitted. Returns `None` if nothing matched.
pub fn correlate(
id: &str,
draws: &[CapturedDraw],
parts: &[PartKey],
ref_sub: &str,
) -> Option<ShipPlacement> {
let mut matched: Vec<(String, M3, [f64; 3], bool)> = Vec::new();
let mut used: std::collections::HashSet<u32> = std::collections::HashSet::new();
for key in parts {
// Several PartKeys may carry the same part (one per LOD-variant vcount);
// the first that validates wins, the rest are skipped.
if matched.iter().any(|(p, ..)| p == &key.part) {
continue;
}
// A vcount hit alone can be a coincidence (small LODs share counts across
// unrelated models — a 51-vert draw once matched the bridge but was a
// different mesh). Candidates failing position validation are REJECTED;
// among validated candidates the best hit count wins (routes twins), and
// a mirror-validated match records the X-reflection.
let mut best: Option<(&CapturedDraw, usize, bool)> = None;
for d in draws.iter().filter(|d| d.vcount == key.vcount && !used.contains(&d.vbase)) {
let Some((score, mirrored)) = pos_validate(d, &key.ref_pos) else {
continue; // positions disagree — not this part
};
if best.map_or(true, |(_, s, _)| score > s) {
best = Some((d, score, mirrored));
}
}
if let Some((d, _, mirrored)) = best {
used.insert(d.vbase);
matched.push((key.part.clone(), d.r, d.t, mirrored));
}
}
let ref_idx = matched.iter().position(|(p, ..)| p.contains(ref_sub)).unwrap_or(0);
let (ref_part, ref_r, ref_t, _) = matched.get(ref_idx)?.clone();
let rt_ref = transpose(&ref_r);
let parts_out = matched
.iter()
.map(|(part, r, t, mirrored)| {
let mut rel_r = mmul(&rt_ref, r);
let dt = [t[0] - ref_t[0], t[1] - ref_t[1], t[2] - ref_t[2]];
let rel_t = mat_vec(&rt_ref, dt);
// The captured WorldView transforms the *uploaded* buffer; for the
// mirrored twin that buffer is the X-reflection of the file geometry,
// so the file-local placement is R·diag(1,1,1) — negate column 0.
if *mirrored {
for row in &mut rel_r {
row[0] = -row[0];
}
}
PartPlacement {
part: part.clone(),
m: snap_m3(&rel_r),
t: [rel_t[0] as f32, rel_t[1] as f32, rel_t[2] as f32],
}
})
.collect();
Some(ShipPlacement { id: id.to_string(), reference: ref_part, parts: parts_out })
}
/// Snap near-axis rotation entries (float noise from the WV products) to exact
/// 0/±1 so the checked-in table is clean; real rotations are untouched.
fn snap_m3(m: &M3) -> [[f32; 3]; 3] {
let snap = |v: f64| -> f32 {
if v.abs() < 5e-4 {
0.0
} else if (v - 1.0).abs() < 5e-4 {
1.0
} else if (v + 1.0).abs() < 5e-4 {
-1.0
} else {
v as f32
}
};
[
[snap(m[0][0]), snap(m[0][1]), snap(m[0][2])],
[snap(m[1][0]), snap(m[1][1]), snap(m[1][2])],
[snap(m[2][0]), snap(m[2][1]), snap(m[2][2])],
]
}
/// Convert a captured placement into viewer [`ScenePart`]s (rigid, unit scale).
pub fn to_scene_parts(ship: &ShipPlacement) -> Vec<ScenePart> {
ship.parts
.iter()
.map(|p| ScenePart { resource: p.part.clone(), m: p.m, t: p.t, s: [1.0, 1.0, 1.0] })
.collect()
}
/// The checked-in placement table, embedded at build time. Empty until captures
/// are baked in with `correlate_capture --emit`.
const EMBEDDED_TABLE: &str = include_str!("../data/ship_placements.txt");
/// The captured placement for ship `id` from the embedded table, if baked in.
pub fn embedded_placement(id: &str) -> Option<ShipPlacement> {
parse_table(EMBEDDED_TABLE).into_iter().find(|s| s.id == id)
}
/// Serialize a placement table to the checked-in text format (see [`parse_table`]).
pub fn serialize_table(ships: &[ShipPlacement]) -> String {
let mut s = String::new();
s.push_str("# Capital-ship part placements — runtime-captured ground truth.\n");
s.push_str("# Generated by: cargo run --release --example correlate_capture -- \\\n");
s.push_str("# <capture.log> <Stage_SNN> <ship_id> [ref_part_substr] --emit\n");
s.push_str("# Per line: <part> <R00 R01 R02 R10 R11 R12 R20 R21 R22> <T0 T1 T2>\n");
for ship in ships {
s.push_str(&format!("\nship {} ref={}\n", ship.id, ship.reference));
for p in &ship.parts {
s.push_str(&format!(
" {} {} {} {} {} {} {} {} {} {} {} {} {}\n",
p.part,
p.m[0][0], p.m[0][1], p.m[0][2],
p.m[1][0], p.m[1][1], p.m[1][2],
p.m[2][0], p.m[2][1], p.m[2][2],
p.t[0], p.t[1], p.t[2],
));
}
}
s
}
/// Parse the checked-in placement table. `#` comments and blank lines are ignored;
/// a `ship <id> ref=<part>` line starts a block, and each following
/// `<part> <9 rotation floats> <3 translation floats>` line is one placement.
pub fn parse_table(text: &str) -> Vec<ShipPlacement> {
let mut ships: Vec<ShipPlacement> = Vec::new();
for line in text.lines() {
let l = line.trim();
if l.is_empty() || l.starts_with('#') {
continue;
}
if let Some(rest) = l.strip_prefix("ship ") {
let mut it = rest.split_whitespace();
let id = it.next().unwrap_or("").to_string();
let reference = it
.next()
.and_then(|s| s.strip_prefix("ref="))
.unwrap_or("")
.to_string();
ships.push(ShipPlacement { id, reference, parts: Vec::new() });
} else if let Some(ship) = ships.last_mut() {
let mut it = l.split_whitespace();
let part = it.next().unwrap_or("").to_string();
let nums: Vec<f32> = it.filter_map(|x| x.parse().ok()).collect();
if part.is_empty() || nums.len() != 12 {
continue;
}
ship.parts.push(PartPlacement {
part,
m: [
[nums[0], nums[1], nums[2]],
[nums[3], nums[4], nums[5]],
[nums[6], nums[7], nums[8]],
],
t: [nums[9], nums[10], nums[11]],
});
}
}
ships
}
fn transpose(m: &M3) -> M3 {
[
[m[0][0], m[1][0], m[2][0]],
[m[0][1], m[1][1], m[2][1]],
[m[0][2], m[1][2], m[2][2]],
]
}
fn mmul(a: &M3, b: &M3) -> M3 {
let mut o = [[0.0; 3]; 3];
for i in 0..3 {
for j in 0..3 {
o[i][j] = (0..3).map(|k| a[i][k] * b[k][j]).sum();
}
}
o
}
fn mat_vec(m: &M3, v: [f64; 3]) -> [f64; 3] {
[
m[0][0] * v[0] + m[0][1] * v[1] + m[0][2] * v[2],
m[1][0] * v[0] + m[1][1] * v[1] + m[1][2] * v[2],
m[2][0] * v[0] + m[2][1] * v[1] + m[2][2] * v[2],
]
}
#[cfg(test)]
mod tests {
use super::*;
/// A `vsconst` line for an already-unit WorldView (norm 1) so R = rows and
/// T = row[3]: identity rotation, translation `t`.
fn draw_line(vbase: u32, vcount: u32, t: [f64; 3]) -> String {
format!(
"DRAW vbase=0x{vbase:X} stride=32 vcount={vcount} indices=0 prim=tri vs=0x1\n \
vsconst base=0: c0=(1,0,0,{}) c1=(0,1,0,{}) c2=(0,0,1,{})\n",
t[0], t[1], t[2]
)
}
fn key(part: &str, vcount: u32) -> PartKey {
PartKey { part: part.to_string(), vcount, ref_pos: Vec::new() }
}
#[test]
fn parse_recovers_worldview() {
let log = draw_line(0x1000, 3, [10.0, 0.0, 0.0]);
let draws = parse_capture(&log);
assert_eq!(draws.len(), 1);
assert_eq!(draws[0].vcount, 3);
assert_eq!(draws[0].t, [10.0, 0.0, 0.0]);
assert_eq!(draws[0].r, [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]]);
}
#[test]
fn parse_normalizes_projection_scale() {
// A row scaled by the projection sx=2 must normalize back to unit, and its
// translation divides by the same norm.
let log = "DRAW vbase=0x2000 vcount=4\n \
vsconst base=0: c0=(2,0,0,20) c1=(0,2,0,0) c2=(0,0,1,0)\n";
let d = &parse_capture(log)[0];
assert!((d.r[0][0] - 1.0).abs() < 1e-9);
assert!((d.t[0] - 10.0).abs() < 1e-9, "20/2 = 10");
}
#[test]
fn correlate_expresses_parts_in_reference_frame() {
// Two parts, identity rotation: ref at view (10,0,0), other at (10,0,50).
// Relative to ref, the other must sit at (0,0,50).
let log = format!(
"{}{}",
draw_line(0x1000, 3, [10.0, 0.0, 0.0]),
draw_line(0x2000, 4, [10.0, 0.0, 50.0])
);
let draws = parse_capture(&log);
let parts = vec![key("e106_bdy_04", 3), key("e106_eng_01", 4)];
let ship = correlate("e106", &draws, &parts, "bdy_04").unwrap();
assert_eq!(ship.reference, "e106_bdy_04");
let refp = ship.parts.iter().find(|p| p.part == "e106_bdy_04").unwrap();
assert_eq!(refp.t, [0.0, 0.0, 0.0]);
let eng = ship.parts.iter().find(|p| p.part == "e106_eng_01").unwrap();
assert_eq!(eng.t, [0.0, 0.0, 50.0]);
}
#[test]
fn drawlog_format_parses_capital_ship_parts() {
// Two capital-ship parts, each its own vertex buffer (distinct base),
// stride 6 → vcount = size_words/6. Identity WVP with known translations.
let log = format!(
"DRAW prim=4 indices=6 src=0 index[...] vs={SHIP_VS_HASH}\n \
stream fc=0 base=0x12A60228 stride_words=6 size_words=9798 endian=2 type=3\n \
vsconst:\n c0 1.0 0.0 0.0 10.0\n c1 0.0 1.0 0.0 0.0\n c2 0.0 0.0 1.0 0.0\n\
DRAW prim=4 indices=6 src=0 index[...] vs={SHIP_VS_HASH}\n \
stream fc=0 base=0x12ABF2CC stride_words=6 size_words=4890 endian=2 type=3\n \
vsconst:\n c0 1.0 0.0 0.0 10.0\n c1 0.0 1.0 0.0 0.0\n c2 0.0 0.0 1.0 50.0\n\
DRAW prim=4 indices=3 src=0 index[...] vs=0xDEADBEEF00000000\n \
stream fc=0 base=0x99990000 stride_words=6 size_words=18 endian=2 type=3\n \
vsconst:\n c0 1.0 0.0 0.0 0.0\n c1 0.0 1.0 0.0 0.0\n c2 0.0 0.0 1.0 0.0\n"
);
let draws = parse_drawlog(&log);
// Two ship-shader buffers; the non-ship shader draw is ignored.
assert_eq!(draws.len(), 2);
assert_eq!(draws[0].vcount, 1633); // 9798/6
assert_eq!(draws[1].vcount, 815); // 4890/6
// Correlate: part B sits 50 along Z from reference part A.
let parts = vec![key("e106_bdy_04", 1633), key("e106_bdy_03", 815)];
let ship = correlate("e106", &draws, &parts, "bdy_04").unwrap();
let b = ship.parts.iter().find(|p| p.part == "e106_bdy_03").unwrap();
assert_eq!(b.t, [0.0, 0.0, 50.0]);
}
/// The real e106 bdy_01/bdy_02 case: both twin resources decode to
/// IDENTICAL file geometry; the engine uploads the second instance as an
/// X-reflection, so one captured buffer disagrees in X sign with the file.
/// Both draws must be placed, and the mirrored one must bake the X-flip
/// (negated first matrix column) so file-local geometry lands port-side.
#[test]
fn runtime_mirrored_twin_placed_with_reflection() {
let log = "DRAW vbase=0x1000 stride=24 vcount=426 indices=21 prim=4 vs=0x1\n \
pos: (134.4215,133.8319,-118.1757) (178.8384,85.3847,238.0463)\n \
vsconst base=0: c0=(1,0,0,264) c1=(0,1,0,0) c2=(0,0,1,0)\n\
DRAW vbase=0x2000 stride=24 vcount=426 indices=21 prim=4 vs=0x1\n \
pos: (-134.4215,133.8319,-118.1757) (-178.8384,85.3847,238.0463)\n \
vsconst base=0: c0=(1,0,0,-264) c1=(0,1,0,0) c2=(0,0,1,0)\n\
DRAW vbase=0x3000 stride=24 vcount=558 indices=9 prim=4 vs=0x1\n \
vsconst base=0: c0=(1,0,0,0) c1=(0,1,0,0) c2=(0,0,1,0)\n";
let draws = parse_capture(log);
assert_eq!(draws.len(), 3);
assert_eq!(draws[0].pos.len(), 2);
// Both twins carry the SAME (file) positions — +X side geometry.
let file_pos = vec![[134.4215, 133.8319, -118.1757], [178.8384, 85.3847, 238.0463]];
let parts = vec![
PartKey { part: "e106_bdy_01".to_string(), vcount: 426, ref_pos: file_pos.clone() },
PartKey { part: "e106_bdy_02".to_string(), vcount: 426, ref_pos: file_pos },
key("e106_bdy_04", 558),
];
let ship = correlate("e106", &draws, &parts, "bdy_04").unwrap();
assert_eq!(ship.parts.len(), 3, "both twins + reference placed");
let get = |p: &str| ship.parts.iter().find(|x| x.part == p).unwrap().clone();
// bdy_01 validated directly → the +264 draw, identity rotation.
let a = get("e106_bdy_01");
assert_eq!(a.t, [264.0, 0.0, 0.0]);
assert_eq!(a.m[0][0], 1.0);
// bdy_02 validated as the MIRROR → the 264 draw, X-flip baked in.
let b = get("e106_bdy_02");
assert_eq!(b.t, [-264.0, 0.0, 0.0]);
assert_eq!(b.m[0][0], -1.0, "mirrored twin must negate the X column");
assert_eq!(b.m[1][1], 1.0);
}
/// A draw whose vcount matches but whose position dump disagrees must be
/// REJECTED, not placed — the real false-positive: a foreign 51-vert model
/// matched `e106_brg_01_l` by count alone and put the bridge 2 km off-hull.
#[test]
fn vcount_coincidence_rejected_by_positions() {
let log = "DRAW vbase=0x1000 stride=24 vcount=51 indices=36 prim=4 vs=0x1\n \
pos: (-0.0000,46.2359,-12.9454) (-0.0000,-6.1936,264.8687)\n \
vsconst base=0: c0=(1,0,0,1975) c1=(0,1,0,0) c2=(0,0,1,0)\n\
DRAW vbase=0x3000 stride=24 vcount=558 indices=9 prim=4 vs=0x1\n \
vsconst base=0: c0=(1,0,0,0) c1=(0,1,0,0) c2=(0,0,1,0)\n";
let draws = parse_capture(log);
let parts = vec![
PartKey {
part: "e106_brg_01".to_string(),
vcount: 51,
// The REAL bridge LOD's vertices — disagree with the dump.
ref_pos: vec![[35.2480, 26.0376, 49.2504], [6.0, 55.9632, 41.1370]],
},
key("e106_bdy_04", 558),
];
let ship = correlate("e106", &draws, &parts, "bdy_04").unwrap();
assert!(
!ship.parts.iter().any(|p| p.part == "e106_brg_01"),
"coincidental vcount match must not place the bridge"
);
assert_eq!(ship.parts.len(), 1);
}
#[test]
fn table_round_trips() {
let ship = ShipPlacement {
id: "e106".to_string(),
reference: "e106_bdy_04".to_string(),
parts: vec![
PartPlacement {
part: "e106_bdy_04".to_string(),
m: [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]],
t: [0.0, 0.0, 0.0],
},
PartPlacement {
part: "e106_eng_01".to_string(),
m: [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]],
t: [131.0, -133.0, -132.0],
},
],
};
let text = serialize_table(std::slice::from_ref(&ship));
let back = parse_table(&text);
assert_eq!(back, vec![ship]);
}
#[test]
fn embedded_table_parses() {
// The checked-in data file must always parse (even if empty).
let _ = parse_table(EMBEDDED_TABLE);
}
/// The baked e106 capture (2026-07-26 F10, Stage_S01): all 8 parts placed,
/// port/starboard pair symmetric with the mirror on bdy_02, bridge on the
/// centreline. Guards the checked-in data against accidental edits.
#[test]
fn embedded_e106_is_complete() {
let e106 = embedded_placement("e106").expect("e106 baked in");
assert_eq!(e106.reference, "e106_bdy_04");
assert_eq!(e106.parts.len(), 8, "all 8 e106 parts placed");
let get = |p: &str| e106.parts.iter().find(|x| x.part == p).unwrap();
// Port/starboard hull pair: X = ∓264, the starboard copy mirrored.
assert!((get("e106_bdy_01").t[0] + 264.0).abs() < 0.1);
assert!((get("e106_bdy_02").t[0] - 264.0).abs() < 0.1);
assert_eq!(get("e106_bdy_02").m[0][0], -1.0);
assert_eq!(get("e106_bdy_01").m[0][0], 1.0);
// Bridge: centreline, above and aft of the hull reference.
let brg = get("e106_brg_01");
assert!(brg.t[0].abs() < 0.1 && brg.t[1] > 150.0 && brg.t[2] < -160.0);
}
}

View File

@@ -79,7 +79,12 @@ pub fn list_voice_clips(sounds_tbl: &[u8], lang: VoiceLang) -> Vec<VoiceClip> {
}
if i - start >= 6 {
if let Ok(s) = std::str::from_utf8(&sounds_tbl[start..i]) {
if s.starts_with(&prefix) && s.ends_with(".slb") && s.contains("VOICE") {
// Every spoken-line category, so the standalone player covers them
// all: in-mission radio (`\Voice\`, `\etc\`) and bound movie voices
// (`\Movie\`) all carry `VOICE_`; mission-briefing lines live in
// `\Briefing\` as `BR<NN>_<MM>.slb` (no `VOICE` in the name).
let is_voice = s.contains("VOICE") || s.contains("\\Briefing\\");
if s.starts_with(&prefix) && s.ends_with(".slb") && is_voice {
if seen.insert(s.to_string()) {
out.push(parse_voice_clip(s));
}
@@ -113,8 +118,58 @@ fn parse_voice_clip(name: &str) -> VoiceClip {
}
}
/// Build one standalone XMA1 `RIFF/WAVE` **per sub-wave** in a `.slb` bank, in
/// on-disc order. A `.slb` is an XACT bank of one or more sub-waves; the sub-waves
/// are EITHER alternate takes (the first is the whole track — e.g. `VOICE_S01A`,
/// `VOICE_ADV`: sub0 ≈ the movie length) OR sequential segments that must be
/// concatenated (e.g. `VOICE_RT07A`: 24s + 14s + 11s ≈ the 50s movie). The caller
/// decodes each to PCM, concatenates in order, and clamps to the movie length —
/// that yields the full track for segment banks while the clamp drops the
/// duplicate takes for alternate-take banks. (Dynamic RE via Canary file-I/O
/// tracing confirmed the movie→voice binding; this fixes the *decode* of `RT*`.)
pub fn to_xma_riffs(slb: &[u8]) -> Vec<Vec<u8>> {
let mut out = Vec::new();
if find(slb, b"RIFF", 0).is_none() {
// Headerless single-stream bank.
if let Some(data) = slb.get(HEADERLESS_DATA_OFFSET..) {
if !data.is_empty() {
out.push(build_riff(&synth_xma1_fmt(2, 2, 48000), data));
}
}
return out;
}
let mut pos = 0usize;
while let Some(ri) = find(slb, b"RIFF", pos) {
// Parse this sub-wave's fmt + data (declared size is honest per sub-wave).
let Some(fi) = find(slb, b"fmt ", ri) else { break };
let Some(fsz) = le32(slb, fi + 4) else { break };
let Some(fmt_end) = fi.checked_add(8).and_then(|v| v.checked_add(fsz as usize)) else {
break;
};
if fmt_end > slb.len() {
break;
}
let Some(di) = find(slb, b"data", fi) else { break };
let Some(dsz) = le32(slb, di + 4) else { break };
let Some(ds) = di.checked_add(8) else { break };
let de = ds
.checked_add(dsz as usize)
.unwrap_or(slb.len())
.min(slb.len());
if let Some(data) = slb.get(ds..de) {
if !data.is_empty() {
out.push(build_riff(&slb[fi..fmt_end], data));
}
}
// Advance past this sub-wave's data to find the next RIFF.
pos = de.max(ri + 4);
}
out
}
/// Build a standalone XMA1 `RIFF/WAVE` from a `.slb` bank, decodable by FFmpeg's
/// `xma1`. Returns `None` if the bank is too small / malformed.
/// `xma1`. Returns `None` if the bank is too small / malformed. This is the
/// FIRST sub-wave only; prefer [`to_xma_riffs`] for correct multi-segment banks.
pub fn to_xma_riff(slb: &[u8]) -> Option<Vec<u8>> {
if let Some(ri) = find(slb, b"RIFF", 0) {
// RIFF layout: a `.slb` is an XACT bank of one or more sub-waves, each
@@ -145,6 +200,38 @@ pub fn to_xma_riff(slb: &[u8]) -> Option<Vec<u8>> {
}
}
/// Rebuild a standalone XMA1 `RIFF/WAVE` from a single-stream `.slb`, robust to
/// the layout variants seen in `<lang>\etc\` radio clips. Unlike [`to_xma_riff`]
/// (which assumes `RIFF → fmt → data` in order), this picks the **largest `data`
/// chunk anywhere** in the bank — some radio banks store the audio *before* the
/// trailing `RIFF`/`fmt` metadata (an empty post-`RIFF` `data` chunk), which the
/// ordered scan misses. Pairs it with the first `fmt ` chunk; falls back to the
/// headerless layout. Returns `None` only when no usable audio can be found.
pub fn to_xma_riff_best(slb: &[u8]) -> Option<Vec<u8>> {
// Largest usable `data` chunk (bounded by its declared size and the buffer).
let mut best: Option<(usize, usize)> = None; // (data offset, usable payload len)
let mut i = 0;
while let Some(di) = find(slb, b"data", i) {
let declared = le32(slb, di + 4).unwrap_or(0) as usize;
let usable = declared.min(slb.len().saturating_sub(di + 8));
if best.map_or(true, |(_, b)| usable > b) {
best = Some((di, usable));
}
i = di + 4;
}
if let (Some(fi), Some((di, sz))) = (find(slb, b"fmt ", 0), best) {
if sz > 512 {
let fsz = le32(slb, fi + 4)? as usize;
let fmt_end = (fi + 8 + fsz).min(slb.len());
let data = slb.get(di + 8..di + 8 + sz)?;
return Some(build_riff(slb.get(fi..fmt_end)?, data));
}
}
// Headerless fallback: fixed data offset, synthesized XMA1 stereo/48k fmt.
let data = slb.get(HEADERLESS_DATA_OFFSET..)?;
(!data.is_empty()).then(|| build_riff(&synth_xma1_fmt(2, 2, 48000), data))
}
/// A minimal `fmt ` chunk carrying an XMA1 `XMAWAVEFORMAT` (one stream).
fn synth_xma1_fmt(channels: u8, channel_mask: u16, rate: u32) -> Vec<u8> {
let mut fmt = Vec::with_capacity(40);
@@ -235,6 +322,40 @@ mod tests {
assert_eq!(&riff[dpos + 8..], &[1, 2, 3, 4]);
}
#[test]
fn list_voice_clips_covers_movie_radio_and_briefing() {
// The standalone player must enumerate every spoken-line category: bound
// movie voices, in-mission radio (\etc\ + \Voice\), and briefing (\Briefing\,
// whose BR<NN>_<MM> names lack "VOICE"). Music (BGM_*) must stay excluded.
let mut tbl = Vec::new();
for s in [
"eng\\Movie\\VOICE_S13A.slb",
"eng\\etc\\VOICE_D_450.slb",
"eng\\Voice\\VOICE_ADAN_010.slb",
"eng\\Briefing\\BR01_01.slb",
"eng\\bgm\\BGM_001.slb",
] {
tbl.extend_from_slice(s.as_bytes());
tbl.push(0);
}
let names: Vec<String> = list_voice_clips(&tbl, VoiceLang::English)
.into_iter()
.map(|c| c.name)
.collect();
for want in [
"eng\\Movie\\VOICE_S13A.slb",
"eng\\etc\\VOICE_D_450.slb",
"eng\\Voice\\VOICE_ADAN_010.slb",
"eng\\Briefing\\BR01_01.slb",
] {
assert!(names.iter().any(|n| n == want), "missing {want}");
}
assert!(
!names.iter().any(|n| n.contains("BGM_")),
"music must not be listed as a voice clip"
);
}
#[test]
fn rebuilds_riff_from_headerless() {
let mut slb = vec![0u8; HEADERLESS_DATA_OFFSET];

File diff suppressed because it is too large Load Diff

View File

@@ -10,10 +10,11 @@ use bevy::prelude::*;
use bevy_egui::{egui, EguiContexts};
use crate::iso_loader::{
AudioPreview, FileInfo, FileSelected, ImageRgba, IsoState, ModelPreview, MovieSubtitles,
MovieVoice, PakContent, PakView, RequestAudio, RequestOpenDir, RequestOpenIso, RequestSubtitles,
RequestVoiceLibrary, SkyboxPreview, TextPreview, TexturePreview, VideoPreview, VoiceLibrary,
IsoLoaderSystemSet,
AudioPreview, FileInfo, FileSelected, GameCategory, GameData, ImageRgba, IsoState, ModelPreview,
MovieSubtitles, MovieVoice, PakContent, PakView, RequestAudio, RequestGameData, RequestOpenDir,
RequestOpenIso, RequestShipCatalog, RequestShipRender, RequestSubtitles, RequestVoiceLibrary,
ShipBrowser, SkyboxPreview, TextPreview, TexturePreview,
VideoPreview, VoiceLibrary, IsoLoaderSystemSet,
};
use crate::ViewerState;
use sylpheed_formats::SubLang;
@@ -25,6 +26,8 @@ impl Plugin for ViewerUiPlugin {
app.insert_resource(FileBrowserState::default());
// Run after the iso_loader chain so we see the frame's final state.
app.add_systems(Update, draw_viewer_ui.after(IsoLoaderSystemSet));
app.add_systems(Update, draw_game_data_ui.after(IsoLoaderSystemSet));
app.add_systems(Update, draw_ships_ui.after(IsoLoaderSystemSet));
}
}
@@ -136,6 +139,8 @@ struct UiEvents<'w> {
subtitles: EventWriter<'w, RequestSubtitles>,
audio: EventWriter<'w, RequestAudio>,
voice_lib: EventWriter<'w, RequestVoiceLibrary>,
game_data: EventWriter<'w, RequestGameData>,
ships: EventWriter<'w, RequestShipCatalog>,
}
fn draw_viewer_ui(
@@ -193,6 +198,18 @@ fn draw_viewer_ui(
}
ui.close_menu();
}
if ui.button("🗃 Game Data…").clicked() {
// The Game Data window (its own system) reads this event to
// open + lazily decode the tables.
events.game_data.send_default();
ui.close_menu();
}
if ui.button("🚀 Ships…").clicked() {
// The Ships window (its own system) reads this event to open +
// lazily scan the stage containers for assemblable ships.
events.ships.send_default();
ui.close_menu();
}
});
ui.menu_button("Help", |ui| {
@@ -227,37 +244,73 @@ fn draw_viewer_ui(
ui.label("Filter:");
ui.text_edit_singleline(&mut voice_lib.filter);
});
let f = voice_lib.filter.to_lowercase();
// Group the thousands of entries as directory → speaker so the
// list is navigable (e.g. browse `Voice` by character to find a
// cutscene's radio line). `name` is `<lang>\<dir>\<file>.slb`.
use std::collections::BTreeMap;
let mut groups: BTreeMap<&str, BTreeMap<&str, Vec<&sylpheed_formats::slb::VoiceClip>>> =
BTreeMap::new();
for c in &voice_lib.clips {
if !f.is_empty() && !c.name.to_lowercase().contains(&f) {
continue;
}
let dir = c.name.rsplit('\\').nth(1).unwrap_or("?");
groups
.entry(dir)
.or_default()
.entry(c.speaker.as_str())
.or_default()
.push(c);
}
let shown: usize = groups.values().flat_map(|s| s.values()).map(Vec::len).sum();
ui.label(
egui::RichText::new(format!("{} clips", voice_lib.clips.len()))
egui::RichText::new(format!("{shown} / {} clips", voice_lib.clips.len()))
.weak()
.small(),
);
ui.separator();
let f = voice_lib.filter.to_lowercase();
let clips: Vec<_> = voice_lib
.clips
.iter()
.filter(|c| f.is_empty() || c.name.to_lowercase().contains(&f))
.take(2000)
.cloned()
.collect();
let filtering = !f.is_empty();
egui::ScrollArea::vertical().show(ui, |ui| {
for c in &clips {
ui.horizontal(|ui| {
if ui.button("").on_hover_text(&c.name).clicked() {
audio.generation = audio.generation.wrapping_add(1);
audio.loading = true;
audio.active = true;
audio.name = c.display.clone();
events.audio.send(RequestAudio {
clip: c.name.clone(),
display: c.display.clone(),
movie: None,
generation: audio.generation,
});
}
ui.label(&c.display);
});
for (dir, speakers) in &groups {
let dtotal: usize = speakers.values().map(Vec::len).sum();
egui::CollapsingHeader::new(format!("📁 {dir} ({dtotal})"))
.id_salt(("vdir", *dir))
.default_open(filtering)
.show(ui, |ui| {
for (speaker, clips) in speakers {
egui::CollapsingHeader::new(format!(
"{speaker} ({})",
clips.len()
))
.id_salt(("vspk", *dir, *speaker))
.default_open(filtering || clips.len() <= 6)
.show(ui, |ui| {
for c in clips {
ui.horizontal(|ui| {
if ui
.button("")
.on_hover_text(&c.name)
.clicked()
{
audio.generation =
audio.generation.wrapping_add(1);
audio.loading = true;
audio.active = true;
audio.name = c.display.clone();
events.audio.send(RequestAudio {
clip: c.name.clone(),
display: c.display.clone(),
movie: None,
generation: audio.generation,
});
}
ui.label(&c.display);
});
}
});
}
});
}
});
}
@@ -1244,3 +1297,397 @@ fn draw_skybox_grid(ui: &mut egui::Ui, skybox: &SkyboxPreview) {
});
});
}
// ── Game Data browser ──────────────────────────────────────────────────────────
fn fnum(v: Option<f32>) -> String {
match v {
Some(x) if x == x.trunc() => format!("{}", x as i64),
Some(x) => format!("{x}"),
None => "·".into(),
}
}
fn fint(v: Option<i64>) -> String {
v.map(|x| x.to_string()).unwrap_or_else(|| "·".into())
}
/// The Game Data browser — a floating window over the decoded IDXD tables
/// (weapons, craft, ships, cast, missions, arsenal, flights). Its own system so
/// `draw_viewer_ui` stays under Bevy's system-parameter limit; a `RequestGameData`
/// event (from the View menu) opens it and lazily kicks off decoding.
#[cfg(not(target_arch = "wasm32"))]
fn draw_game_data_ui(
mut contexts: EguiContexts,
mut game_data: ResMut<GameData>,
mut requests: EventReader<RequestGameData>,
) {
if requests.read().next().is_some() {
game_data.open = true;
if !game_data.loaded {
game_data.loading = true;
}
}
if !game_data.open {
return;
}
let ctx = contexts.ctx_mut().clone();
let mut open = true;
egui::Window::new("🗃 Game Data")
.default_width(700.0)
.default_height(560.0)
.open(&mut open)
.show(&ctx, |ui| {
if game_data.loading {
ui.horizontal(|ui| {
ui.spinner();
ui.label("Decoding game tables…");
});
ctx.request_repaint();
return;
}
if !game_data.loaded {
ui.label("Open a game source first (File ▸ Open…).");
return;
}
let GameData { category, filter, snapshot, .. } = &mut *game_data;
ui.horizontal_wrapped(|ui| {
for cat in GameCategory::ALL {
ui.selectable_value(category, cat, cat.label());
}
});
let filterable = !matches!(category, GameCategory::Arsenal | GameCategory::Flights);
if filterable {
ui.horizontal(|ui| {
ui.label("🔍");
ui.text_edit_singleline(filter);
if !filter.is_empty() && ui.small_button("").clicked() {
filter.clear();
}
});
}
ui.separator();
let f = filter.to_lowercase();
let hit = |s: &str| f.is_empty() || s.to_lowercase().contains(&f);
egui::ScrollArea::vertical().auto_shrink([false, false]).show(ui, |ui| match *category {
GameCategory::Weapons => {
egui::Grid::new("g_weap").striped(true).num_columns(6).show(ui, |ui| {
for h in ["Weapon", "Targets", "Power", "Velocity", "Range", "Reload"] {
ui.strong(h);
}
ui.end_row();
for w in &snapshot.weapons {
let name = w.id.as_deref().unwrap_or("?").trim_start_matches("Weapon_");
if !hit(name) {
continue;
}
ui.label(name);
ui.label(w.target_type.as_deref().unwrap_or("·"));
ui.label(fnum(w.power));
ui.label(fnum(w.velocity));
ui.label(fnum(w.max_range));
ui.label(fint(w.loading_count));
ui.end_row();
}
});
}
GameCategory::Craft => {
egui::Grid::new("g_craft").striped(true).num_columns(6).show(ui, |ui| {
for h in ["Craft", "HP", "Cruise", "Accel", "Radar", "Turrets"] {
ui.strong(h);
}
ui.end_row();
for u in &snapshot.craft {
let name = u.id.as_deref().unwrap_or("?").trim_start_matches("UN_");
if !hit(name) {
continue;
}
ui.label(name);
ui.label(fnum(u.hp));
ui.label(fnum(u.cruising_velocity));
ui.label(fnum(u.acceleration));
ui.label(fnum(u.radar_range));
ui.label(fint(u.turret_count));
ui.end_row();
}
});
}
GameCategory::Vessels => {
egui::Grid::new("g_ves").striped(true).num_columns(6).show(ui, |ui| {
for h in ["Vessel", "HP", "Length", "Turrets", "Bridges", "Shield gen"] {
ui.strong(h);
}
ui.end_row();
for v in &snapshot.vessels {
let name = v.id.as_deref().unwrap_or("?").trim_start_matches("UN_");
if !hit(name) {
continue;
}
ui.label(name);
ui.label(fnum(v.hp));
ui.label(fnum(v.size_z));
ui.label(fint(v.turret_count));
ui.label(fint(v.bridge_count));
ui.label(fint(v.shield_generator_count));
ui.end_row();
}
});
}
GameCategory::Characters => {
egui::Grid::new("g_char").striped(true).num_columns(3).show(ui, |ui| {
for h in ["Name", "Faction", "Portraits"] {
ui.strong(h);
}
ui.end_row();
for c in &snapshot.characters {
if !hit(&c.name) && !hit(&c.faction) {
continue;
}
ui.label(&c.name);
let col = if c.faction == "TCAF" {
egui::Color32::from_rgb(90, 160, 232)
} else if c.faction == "ADAN" {
egui::Color32::from_rgb(224, 86, 122)
} else {
egui::Color32::GRAY
};
ui.colored_label(col, if c.faction.is_empty() { "" } else { &c.faction });
ui.label(c.faces.to_string());
ui.end_row();
}
});
}
GameCategory::Missions => {
for m in &snapshot.missions {
if !hit(&m.id) && !hit(&m.location) && !m.objectives.iter().any(|o| hit(o)) {
continue;
}
let head = format!("{} · {} · {} phases", m.id, m.location, m.phases);
egui::CollapsingHeader::new(head).id_salt(&m.id).show(ui, |ui| {
if !m.objectives.is_empty() {
ui.strong("Objectives");
for o in &m.objectives {
ui.label(format!("{o}"));
}
}
if !m.lose.is_empty() {
ui.add_space(4.0);
ui.strong("Fail conditions");
for l in &m.lose {
ui.colored_label(egui::Color32::from_rgb(224, 86, 122), l);
}
}
if !m.enemies.is_empty() {
ui.add_space(4.0);
ui.strong("Enemy roster");
for e in &m.enemies {
ui.label(format!("{e}"));
}
}
});
}
}
GameCategory::Arsenal => {
ui.columns(4, |cols| {
for (i, (title, list)) in [
("Nose", &snapshot.arsenal.nose),
("Arm 1", &snapshot.arsenal.arm1),
("Arm 2", &snapshot.arsenal.arm2),
("Arm 3", &snapshot.arsenal.arm3),
]
.into_iter()
.enumerate()
{
cols[i].strong(format!("{title} ({})", list.len()));
for w in list {
cols[i].label(w.replace('_', " "));
}
}
});
}
GameCategory::Flights => {
ui.label(
egui::RichText::new("Distinct wingman line-ups (story order)").weak().small(),
);
ui.add_space(4.0);
for (n, lineup) in snapshot.flights.iter().enumerate() {
egui::CollapsingHeader::new(format!("Line-up {}", n + 1))
.id_salt(n)
.default_open(n == 0)
.show(ui, |ui| {
egui::Grid::new(("g_flight", n)).striped(true).num_columns(2).show(ui, |ui| {
for (cs, pilot) in lineup {
ui.label(cs);
ui.strong(pilot);
ui.end_row();
}
});
});
}
}
});
});
game_data.open &= open;
}
// ── Ships browser ───────────────────────────────────────────────────────────────
/// The Ships browser — a floating window over the capital ships reconstructed
/// from XBG7 part families. Picking a ship assembles its parts and renders the
/// whole model in the 3D view. Its own system (keeps `draw_viewer_ui` under the
/// parameter limit); a `RequestShipCatalog` event (View menu) opens it and lazily
/// kicks off the stage scan, while clicking a row fires `RequestShipRender`.
#[cfg(not(target_arch = "wasm32"))]
fn draw_ships_ui(
mut contexts: EguiContexts,
mut ships: ResMut<ShipBrowser>,
mut requests: EventReader<RequestShipCatalog>,
mut render: EventWriter<RequestShipRender>,
) {
if requests.read().next().is_some() {
ships.open = true;
}
if !ships.open {
return;
}
let ctx = contexts.ctx_mut().clone();
let mut open = true;
egui::Window::new("🚀 Ships")
.default_width(560.0)
.default_height(560.0)
.open(&mut open)
.show(&ctx, |ui| {
if ships.loading {
ui.horizontal(|ui| {
ui.spinner();
ui.label("Scanning stage models for ships…");
});
ctx.request_repaint();
return;
}
if !ships.loaded {
ui.label("Open a game source first (File ▸ Open…).");
return;
}
// Faction filter chips.
ui.horizontal(|ui| {
for (label, val) in
[("All", ""), ("ADAN", "ADAN"), ("TCAF", "TCAF"), ("Neutral", "Neutral")]
{
let sel = ships.faction == val;
if ui.selectable_label(sel, label).clicked() {
ships.faction = val.to_string();
}
}
});
ui.horizontal(|ui| {
ui.label("🔍");
ui.text_edit_singleline(&mut ships.filter);
if !ships.filter.is_empty() && ui.small_button("").clicked() {
ships.filter.clear();
}
});
ui.horizontal(|ui| {
ui.checkbox(&mut ships.show_external, "External parts (bridge/engines/turrets)");
ui.label(
egui::RichText::new("(bridge / shield gens / engines — approximate placement)")
.weak()
.small(),
);
});
ui.separator();
let ShipBrowser { rows, filter, faction, selected, show_external, .. } = &mut *ships;
let show_external = *show_external;
let needle = filter.to_lowercase();
let mut to_render: Option<(String, String, String)> = None;
egui::ScrollArea::vertical().show(ui, |ui| {
let mut last_section = String::new();
for row in rows.iter() {
if !faction.is_empty() && &row.faction != faction {
continue;
}
if !needle.is_empty()
&& !row.name.to_lowercase().contains(&needle)
&& !row.id.contains(&needle)
{
continue;
}
// Section header: capital ships first, then other assemblies.
let section = if row.has_vessel { "Capital ships" } else { "Other assemblies" };
if section != last_section {
ui.add_space(4.0);
ui.label(egui::RichText::new(section).strong().weak());
last_section = section.to_string();
}
let is_sel = selected.as_deref() == Some(row.id.as_str());
let color = match row.faction.as_str() {
"TCAF" => egui::Color32::from_rgb(120, 170, 235),
"ADAN" => egui::Color32::from_rgb(235, 120, 120),
_ => egui::Color32::from_rgb(180, 180, 180),
};
egui::CollapsingHeader::new(
egui::RichText::new(format!("{} · {}", row.name, row.id)).color(color),
)
.id_salt(&row.id)
.default_open(false)
.show(ui, |ui| {
egui::Grid::new(("shipstat", &row.id)).num_columns(2).show(ui, |ui| {
ui.label("Faction");
ui.strong(&row.faction);
ui.end_row();
if let Some(hp) = row.hp {
ui.label("Hull HP");
ui.strong(format!("{hp:.0}"));
ui.end_row();
}
if let Some((x, y, z)) = row.size {
ui.label("Size (m)");
ui.strong(format!("{x:.0} × {y:.0} × {z:.0}"));
ui.end_row();
}
if row.turrets.is_some() || row.shield_gens.is_some() {
ui.label("Hardpoints");
ui.strong(format!(
"{} turrets · {} bridges · {} shield gens",
fint(row.turrets),
fint(row.bridges),
fint(row.shield_gens),
));
ui.end_row();
}
ui.label("Model parts");
ui.strong(format!("{}", row.parts.len()));
ui.end_row();
ui.label("Appears in");
ui.strong(row.stages.join(", "));
ui.end_row();
});
let btn = egui::Button::new(if is_sel {
"● Showing in 3D view"
} else {
"▶ Assemble & view in 3D"
});
if ui.add(btn).clicked() {
to_render = Some((
row.id.clone(),
row.stage_file.clone(),
format!("{} ({})", row.name, row.id),
));
}
});
}
});
if let Some((id, file, label)) = to_render {
let id2 = id.clone();
*selected = Some(id);
render.send(RequestShipRender { file, id: id2, external: show_external, label });
}
});
ships.open &= open;
}

View File

@@ -0,0 +1,71 @@
# Handoff — capital-ship placement: static assembly EXACT (2026-07-26)
Branch `feature/ipfb-idxd-parser`, commits `c6ca5ce..ea32e77`. Canary fork
branch `capture-ship-placement`, tip `877163792`.
## Outcome
**Capital ships assemble exactly from the ISO alone — no captures needed.**
Runtime captures served as the oracle that cracked the static encoding and now
remain as verification tooling only.
## The two format discoveries (docs/re/ship-placement-runtime-capture.md)
1. **Node-TRS off-by-one** (`mesh.rs::read_trs9`): an XBG7 scene-graph node's
joint table holds NINE keyframe channels `[TX TY TZ RY RX RZ SX SY SZ]`
behind EIGHT pointer slots shifted one track forward — channel k+1 =
`f64@(ptr[k]+8)`, channel 0 (TX) = `f64@(ptr[0]0x48)`. The old read never
saw TX (why hulls stacked on the centreline) and misassigned TY/RY.
2. **Euler order** (`mesh.rs::node_rotation`): `Ry(ch3)·Rx(ch4)·Rz(ch5)`,
angles direct — pinned by decomposing the captured engine-nacelle rotation
`Rx(15°)·Rz(30°)` element-for-element.
## assemble_ship (ship.rs) — fully static, capture-validated
- Every composite-node instance (alias duplicates collapsed), including
cross-id turret mounts (`rou_e303_wep_01_root` ×2 on e106; e108 places 7).
- Engine cluster rig `e_rou_<id>_eng` mounts at `GN_Engine_01`
(two mirrored nacelles + centre engine).
- `brg`/`sld` at their exact GN frames (frames were always exact).
- Shared-geometry ±TX twins: the instance whose offset opposes the geometry's
dominant side draws X-reflected (viewer reverses winding on det<0).
- `exhaust_frames()`: the `GN_Jet`/`GN_SJet` frames — the game renders its
engine-exhaust FX there; the viewer draws marker cones at them (the engine
GEOMETRY is recessed in the hull by design; without the FX ships read
engine-less — that was the "engines inside the hull" report).
## Validation chain (all green)
- `ship::tests::static_assembly_matches_runtime_capture` (SYLPHEED_ISO): static
== the baked e106 capture, all 8 parts, T<1.0 R<0.02, nacelles ×2, turrets
×2, starboard mirror.
- `examples/capture_verify.rs` over the user's 5 multi-angle snapshots
(43 e106 instances): dominant clusters match static to ~1 unit on every part
incl. both nacelles. Run:
`cargo run --release --example capture_verify -- <Stage.xpr> e106 <logs…>`
- Offline audit (`examples/ship_audit.rs`) across all stages: no outliers among
composite ships; composite-less families (fighter morph sets, turret/prop
parts) are filtered from the viewer's Ships catalog.
- 81 formats-lib + all disc tests; viewer builds.
## Canary instrumentation (branch `capture-ship-placement`)
F10 → `xenia_ship_capture_NN.log` per press (multi-angle in one run), de-duped
by (vertex-buffer, c0..c2 WVP hash) so every instance of every part is
recorded. Launch:
```
cd xenia-canary/build/bin/Linux/Release
./xenia_canary "<…/Project Sylpheed - Arc of Deception (USA, Europe) (En,Ja).iso>"
```
## Open / next
- Verify remaining capital ships against the existing 5 snapshots (they contain
every on-screen ship): e105 (its engine block assembles dorsally — the one
placement to confirm), f105/f106/f101, e102. Same capture_verify invocation
with the other ship id.
- AA-gun models at `GN_AAGun*` frames need the Vessel recipe binding.
- SJet (maneuvering-vernier) marker orientation is approximate; main aft plumes
verified.
- `_mov` node animation keys (hull-opening sequences) — rest pose rendered now.
- Viewer GUI verification by the user (exhaust cones, catalog cleanup).

46
docs/re/BACKLOG.md Normal file
View File

@@ -0,0 +1,46 @@
# RE backlog
Open items that are *not* being worked right now. Each entry says what is wrong or
unknown, what evidence exists, and what the first step would be. Move an item into
`INDEX.md` (with a `structures/…md` or a parser + test) once it is actually settled.
---
## Capital ships assemble wrong in the viewer
**Reported:** 2026-07-30, by the user. **Status:** ❔ open, not investigated.
The reborn viewer builds capital ships from the split XBG7 parts via
`sylpheed-formats::ship::assemble_ship`, and they come out **wrong** — parts in the
wrong place / wrong orientation.
**Why this is a real finding and not a known limitation:** the RE write-up
[`ship-placement-runtime-capture.md`](ship-placement-runtime-capture.md) declares
static assembly ✅ **exact** as of 2026-07-26 — 9-channel joint tables
`[TX TY TZ RY RX RZ SX SY SZ]`, Euler `Ry·Rx·Rz`, with
`ship::tests::static_assembly_matches_runtime_capture` asserting static == runtime
capture (T < 1.0, R < 0.02). So either the viewer is not using that path, or the
claim generalises worse than the test suggests.
**The likely gap:** that test is **one ship** — the `e106` destroyer, 8 parts plus
two nacelles, two turrets and the hull mirror. Nothing pins the other classes.
Rules that were derived from `e106` and could easily be `e106`-specific:
- the engine cluster rig mounted at `GN_Engine_01` (two mirrored nacelles + centre);
- "X-reflect the shared-geometry twin whose lateral offset opposes the geometry's
dominant side" — a heuristic, not a decoded flag;
- cross-id turret instancing (×2).
**First step (the oracle already exists):** re-run the runtime capture on a *different*
capital ship and diff static vs captured, exactly as `e106` was done — F10 in the
`capture-ship-placement` build of `xenia-canary-native` dumps the ship shader's
`c0..c2` WorldViewProjection rows per part; `WV_ref⁻¹ · WV_p` is the ship-space rigid
transform, which is ground truth. Pick a class whose rig differs from `e106`
(different engine count, a ship with no `sld`, a carrier). Then extend
`static_assembly_matches_runtime_capture` into a per-ship table so a regression in one
class cannot hide behind `e106` passing.
**Also worth ruling out first, cheaply:** that the viewer's own transform stack (scale,
handedness, node-instance recursion) is not re-breaking a correct assembly — compare
the viewer's placement against `assemble_ship`'s output directly before blaming the
format layer.

View File

@@ -20,6 +20,19 @@ Promote to a prose `structures/…md` file when a format needs behavioural notes
| IXUD subtitle | 🟡 | `sylpheed-formats/src/ixud.rs` | timed cues; **movie↔track link unknown** (dynamic item) |
| Fonts (ttf/otf/ttc) | ✅ | `sylpheed-formats/src/font.rs` | standard OpenType, parsed via ttf-parser |
| XBG7 mesh | 🟡/❔ | `sylpheed-formats/src/mesh.rs` + `tests/mesh_disc.rs` ([xbg7](structures/xbg7-mesh.md)) | weapons/props: declaration-driven variable stride (36 models), GPU-confirmed. **Stage containers: 5662 sub-models across 22 stages** via content-anchored grouped pools (`stage_models`). Quantized hero bodies (DeltaSaber `f004`) still declined |
| Capital-ship part placement | 🟡 | `sylpheed-formats/src/ship.rs` (static) + [runtime capture](ship-placement-runtime-capture.md) | hull placement static-exact; external parts approximate statically. **Runtime capture** (Canary F10 → VS-constant WorldView) gives ground truth — validated on `e106` destroyer; not yet baked into the viewer |
| Weapon fields defaulted on disc | ✅ | [runtime struct](structures/weapon-struct-runtime.md) · [DATA SHEET route](weapon-datasheet-runtime.md) | **Solved.** Canary maps guest RAM into `/dev/shm`, so the parsed `Weapon`/`Shell` objects are readable live; their layout is solved against disc ground truth (zero contradictions over 100+ records). All 126 weapons, exact numbers, no story progress needed — [4 393 values](captures/weapon-runtime-fields.csv) the disc does not carry. Supersedes the letter-bucket limit of the DATA SHEET route, which now serves as the independent cross-check |
| Unit (craft/vessel) fields defaulted on disc | ✅/🟡 | [runtime struct](structures/unit-struct-runtime.md) | The parsed `unit\UN_*.tbl` definition object, vtable `0x820af844`, ≥`0x380` bytes, one per unit — **discovered, not assumed** (`unit_discover.py`), and distinguished from the spawned-entity class `0x820af030` by being one-per-ID and byte-constant within a run. Across runs only pointer words move — `--crosscheck` proves **no reported field offset is run-dependent** (two words, `+0x2c8`/`+0x2d0`, are stage-dependent and remain unidentified). 27 fields ✅ (21 units, 7 runs); the `Maneuver` block is **schema declaration order, 4 bytes/field, base `0x9c` with a two-slot gap after `AA_Roll_Min`** (29 anchors, 0 conflicts), which also pins 5 fields *no* disc record ever values. Angles are **radians at runtime, degrees on disc**. Unlike weapons, unit definitions are instantiated **per stage**, so coverage (21/110) grows by visiting missions — [values](captures/unit-runtime-fields.csv) |
| UI screen layout (`.rat`) | ✅/🟡 | [ui-rat-layout](structures/ui-rat-layout.md) | One pak per UI screen; each RATC = one (context × language) build; every `<name>.t32` sprite has a `<name>.rat` **layout record** (BE u32; 1280×720 design space; scale/tint/X/Y, keyframes for animated elements, `opt ` link to the focused state). **The tutorial PAUSE menu and the title main menu both rebuild pixel-accurately from the disc.** `loop1.rat` (screen-level draw order) not yet decoded |
## Runtime / dynamic-capture technique
| Technique | Conf. | Spec | Notes |
|-----------|-------|------|-------|
| Live guest-memory read | ✅ | [`tools/re-capture/gmem.py`](../../tools/re-capture/gmem.py) | Canary backs the guest address space with `/dev/shm/xenia_memory_*`; guest VAs map in through Xenia's fixed table. Full-RAM search ~0.2 s (sparse, `SEEK_DATA`). No debugger, no emulator patch, game keeps running |
| IDXD object layout solver | ✅ | [`tools/re-capture/weapon_runtime.py`](../../tools/re-capture/weapon_runtime.py) | Scan RAM for a class's vtable → enumerate its objects → brute-force `(field, offset, encoding)` against the disc records. Accepts a binding only on **zero** contradictions. Generalizes to any IDXD-backed definition |
| Live entity state, anchored on the definition | ✅ | [`tools/re-capture/own_state.py`](../../tools/re-capture/own_state.py) · [autopilot](autopilot-memory-driven.md) | An undamaged craft holds its definition's own numbers, so a *solved definition field* locates the matching live field without a value scan: definition `HP` (1500) → **hull at `position+0x154`**, confirmed by a trace across a death (30/60/90 per hit, negative at 0). Reusable for any live counter whose maximum the definition carries |
| Input → dynamics calibration | ✅ | [`tools/re-capture/ctrl_probe.py`](../../tools/re-capture/ctrl_probe.py) · [`binq.py`](../../tools/re-capture/binq.py) | Hold each pad input in turn and measure the craft's speed as displacement/s of its own position triple — no speed field needed first. Settled the throttle: **`RT` accelerates, `LT` brakes, and the setting persists** (488 → 1510 → 174 units/s), overturning an earlier field-scan conclusion |
## Functions / code paths

View File

@@ -0,0 +1,299 @@
# Memory-driven autopilot — build log and current state
**Status: 🟢 IT FLIES, KILLS AND SURVIVES — but it loses the mission anyway.**
Updated 2026-07-30. `pilot.py` flew Stage 02 for **300 s with the hull untouched
at 1500/1500** and scored the first confirmed autopilot kill (`YOU KILLED
WARPLANES 0001` on the HUD, screenshots `shots/pilot1-*.png`); the scene's
hostile count fell from 134 to 111 over the run. The day before, every run was
dead inside 35 s. What is still missing is the *end* of a mission: the objective
counter (`REMAINING OB`) rises as new waves spawn, and nothing yet tracks which
targets actually close it out — and the second run proved the point the hard
way: `GAME OVER` with the hull at 1500/1500, because Stage 02 is an **escort**
and the ACROPOLIS was sunk while the pilot chased fighters two kilometres away.
## 2026-07-30 — the numbers survival needs
Three things the loop was missing were measured this session, each by
consequence rather than by reading a field and hoping.
### Hull is `position + 0x154` ✅ CONFIRMED
The unit definition already had `HP` solved at `+0x054`
([unit-struct-runtime](structures/unit-struct-runtime.md)); the Delta Saber's is
**1500**. A craft that has taken no damage must therefore *contain that number*,
which turns "find the HP field" into a two-float lookup rather than a value scan
(`own_state.py`). It appears once in the entity object, at `pos+0x154`, and the
trace across a death settles it (`ctrl_probe.py` capture, `binq.py trace`):
```
t phase hull
0.00 base 1500.00 <- == definition HP
23.25 rest_A 1380.00 <- first hit, -120
26.68 … 27.18 B 1320 … 930 <- seven hits in 0.5 s
31.93 X 150.00
35.21 Y -30.00 <- goes negative
35.26 Y -180.00 -> GAME OVER on screen
```
Damage arrives in 30/60/90-point steps and the field goes *negative* at death,
so it is the raw hull counter, not a clamped display value. **1500 hull lost in
12 s** of sitting in a turret's line of fire is the whole reason every earlier
run died.
### Shield is `position + 0x430` 🟡 PROBABLE — not yet confirmed live
Same anchor trick: the definition's shield `MaxValue` is **400** and
`ChargeSpeed` **25**, and the entity object holds `400.0` at `+0x430`, `+0x434`
and `+0x438`, with `25.0` at `+0x448`. Which of the three is the *current* value
is unproven — the capture that spanned the death used a ±0x400 window and
cropped them out. `ctrl_probe.py` now samples ±0x800.
### `RT` accelerates, `LT` brakes, and the throttle is a *setting* ✅ CONFIRMED
`ctrl_probe.py` holds each input in turn and measures the craft's own speed as
displacement per second from the position triple, so no speed field is needed.
Distance flown / phase duration, one 3 s hold each, sticks neutral:
| phase | speed (units/s) | | phase | speed (units/s) |
|---|---|---|---|---|
| base (no input) | 488 | | A | 287 |
| **RT** | **1510** | | B | 139 |
| rest after RT | 1056 | | X | 125 |
| **LT** | **174** | | Y | (dying) |
| rest after LT | 428 | | LB / LS / RS / RY / RX / dpad | no effect |
RT triples the speed, LT cuts it to a third, and **the braked state persists**:
after the LT phase the craft sat at 125140 units/s with the sticks and triggers
neutral for the remaining 40 s, and nothing but RT brought it back. So these are
a throttle setting, not a momentary boost — which also means a control loop must
send only the *changes*.
This **corrects** the earlier note in this file ("`RT` is *not* the throttle,
and no button tested is"). That conclusion came from `findspeed.py`, which
assumed the control and went looking for a *field* that rose; measuring the
speed directly reverses it.
### Two method corrections
* **Pick the attitude block by the flight path, not by address order.** The
player object contains **20** orthonormal 3×3 blocks (identity frames, bone
or camera frames), and `pos-0x70` and `pos-0x30` hold the *same* matrix.
Taking `found[0]` wrote a config with `rot_delta = -0x764` and a nonsense
forward axis; `entities2.py self` now scores every block against the measured
direction of travel and picks the best (`cos = +1.000`, row 2, sign +1).
* **The entity-heap scan has to be numpy.** A per-word Python loop over the
16 MB entity region costs seconds per scan, which is the whole budget of a
10 Hz control loop; `np.isin` over a `>u4` view is milliseconds.
### Stage 02 as an autopilot testbed (from the in-flight HUD)
`OBJECTIVE: shoot down all invading enemy fighters while watching out for
attacks on the ACROPOLIS` · `DEFEAT: your fighter is shot down, or the ACROPOLIS
is sunk` · `HINT: you can resupply at the ACROPOLIS`. The HUD shows
**`REMAINING OB 004`** — only four objective targets — so this mission is
winnable by an autopilot that survives. It also shows separate **SHIELD** and
**ARMOR** bars (matching a 400-point shield over 1500 hull), `A/B 7,635`
afterburner, and `NOSE BM 06000` / `MAIN MPM 00300` ammo.
## What the loop did before that, observed
```
[ 82.5] tgt=e007_ADAN_Turret d=3384 yaw= -7.3 pit=+14.6 stick=(-0.13,-0.34) fire=0
[ 85.7] tgt=e007_ADAN_Turret d=2797 yaw= +1.9 pit=+27.2 stick=(+0.20,-0.59) fire=0
[117.3] tgt=e007_ADAN_Turret d=4211 yaw= +5.7 pit= +9.0 stick=(+0.25,-0.40) fire=1
```
Distance closes monotonically, yaw error is driven from 8° to ~0, and once both
errors are inside the firing cone it holds RB and the ammo counter falls. A
rescan reports the scene as e.g. `136 entities {'TCAF': 16, 'ADAN': 120}`.
**The chain that made it work** — each link checked, not assumed:
1. **Entity typing.** A live entity's definition pointer sits at
**position + 0x130**. One heap scan then yields every craft *with its unit
type*, which is what separates 20-odd real combatants from ~30 000 moving
particles. Verified by the result being coherent: wingmen, enemy turrets and
attackers, friendly capital ships, and exactly one `…_Player`.
2. **Orientation.** A 3×3 rotation at **position 0x70**, stored with a
**16-byte row stride** (a 4×4 transform whose translation row *is* the
position). An earlier search for nine *contiguous* floats structurally could
not find this, which is why the first pass concluded "no transform". The
binding is confirmed independently: its row 2 matches the craft's measured
direction of travel with **cos = +1.000**.
3. **The fire button is RB** — established by consequence, not by guessing:
of RB/LB/A/B/X/Y/RT/LT, pressing RB is the only one that makes the nose-ammo
counter in RAM fall (5958 → 5940). (This entry also claimed `RT` is *not* the
throttle — **wrong**, see the 2026-07-30 measurement above.)
4. **Control.** PD on the aiming error with the derivative taken from the
craft's own body angular velocity (from two consecutive rotation matrices),
and target selection weighted by off-boresight angle
(`score = d·(1 + 3·(θ/π)²)`) rather than pure nearest — closing on a target
90° off the nose only raises the bearing rate, which is what held the first
run outside its firing cone at a steady ~27° pitch error.
## Goal
Fly and fight a mission by reading the game's own world state out of guest RAM
and driving the pad from it — the RE payoff being an oracle for the
reimplementation's flight model and AI, and a way to reach missions the save
cannot otherwise reach (unit coverage for
[unit-struct-runtime](structures/unit-struct-runtime.md) is capped at 21/110
because definitions load **per stage**).
## What works (verified)
| Piece | Tool | Evidence |
|---|---|---|
| Live guest-RAM reads at loop rate | `gworld.py` | `/dev/shm` file opened once, `pread` per tick; a whole-RAM scan is ~6 s, a targeted read is microseconds |
| Whole-RAM float scanning | numpy over `SEEK_DATA` extents | 1 270 orthonormal 3×3 blocks located in 6.2 s |
| Entity enumeration by unit type | `gworld.py entities` | 116 live instances in Stage 02, typed by unit ID, incl. exactly one `…_Player` |
| Pad control at loop rate | `flight_probe.Pad` | writes command lines straight into the vgamepad FIFO; the `vgamepad` CLI spawns a process per command and its `tap`/`hold` sleep *inside* the server, so neither is usable in a control loop |
| Unattended mission entry | `launch_mission.sh` | title → LOAD GAME → slot 01 → READY ROOM → TAKE OFF → in flight, repeatable |
| Hangar loadout | `launch_mission.sh --hangar` | the "Recommended" control is **AUTO SELECT — "Mount most suitable weapons"**, already the default cursor position. At 5 % progress it is a no-op: only two weapons are developed, and they are already mounted |
| Finding moving objects | `findplayer.py` | position triples recovered from motion alone — straight-line, constant-speed filter over K whole-RAM samples |
## What is NOT solved
**Survival, and therefore mission completion.** The loop has no evasion, no
shield/armour awareness and no throttle control, so it flies a straight pursuit
into defended space and is eventually shot down — every long run so far has
ended in GAME OVER. Completing a mission needs, at least: reading own
shield/armour, breaking off when hit, and prioritising the mission's actual
objective targets over the nearest turret.
### Superseded (kept because the reasoning still matters)
The notes below were written before the chain above worked. They remain true as
statements about `0x820af030`, which is *not* the live entity —
1. **The `0x820af030` class is not the live entity.** It has one object per
spawned thing and carries the unit-ID string, so it looked like the entity
list — but over a 29 s in-flight capture **all 384 words of it are
constant** (`whatchanges.py`). It is a static spawn record. The earlier
claim in `unit-struct-runtime.md` that this class is "the spawned entity
instance — live state" is **wrong in the second half**: it is per-spawn, but
it is not live state. The parts of that document that depend on the
*definition* class `0x820af844` are unaffected.
2. **No transform in or one hop from that object**: no orthonormal 3×3, and no
unit quaternion, within `0x2000` of it or behind any of its 121 pointers.
3. **Input correlation finds *a* self-object, but not obviously the craft.**
Holding hard-left then hard-right yaw and looking for a position whose turn
axis reverses (`findself.py`, `selfstate.py`) gives clean hits
(`cos ≈ 0.99`), but they cluster at `0x40009xxx` in what looks like an
8-corner box with ±45 000 coordinates — a camera/skybox volume that follows
the player, not the craft. Its speed (≈359/s) is suspiciously close to the
HUD's 350, which supports "follows the player" but is not proof of identity.
4. **Entity typing is unavailable**: no definition pointer within ±0x800 of the
self-position, so the trick of learning one object's layout and applying it
to all the others has nothing to anchor on. Without typing, the 33 418
moving triples in a firefight cannot be separated into enemies, friendlies
and bullets, so there is nothing to aim at.
## Dead ends, recorded so they are not re-run
* **Speed-scan for the player object** (`findspeed.py`, the classic two-state
value scan: coast → boost → coast). Sound method, but **`RT` is not the
throttle** — 5 864 floats matched the cruise speed and none rose. The control
actually bound to acceleration was never established, and the run that would
have established it ended in GAME OVER.
* **Comparing orientation matrices 2 s apart.** At a real turn rate that is far
outside the small-angle regime, so the skew part of `A·Bᵀ` is not the rotation
vector and the "angular velocity" comes out as ~30 000. Sample incrementally
(6 Hz) and re-check orthonormality on every read — blocks found by a scan get
overwritten between the scan and the read.
## The second run lost the mission **without being hit** (2026-07-30)
A second 240 s flight, with the two fixes above, ended on the `GAME OVER`
screen — while the hull read **1500/1500 on the last live tick**. Nothing shot
us down. The other defeat condition fired: *the ACROPOLIS is sunk*. The HUD had
been showing a red `WARNING` banner for a while, and the pilot spent the whole
run pursuing an `e010_ADAN_Attacker_S` two kilometres away.
So surviving is necessary and not sufficient, and "nearest hostile fighter" is
the wrong objective function for this stage. **The mission is an escort.** What
follows:
* **Prioritise hostiles by their distance to the protected asset, not to us.**
The attackers worth killing are the ones closing on the ACROPOLIS.
* **The protected asset's health is readable with the same anchor as ours** —
hull at `position + 0x154`, its maximum being its own definition's `HP`. That
gives a live "are we winning" signal for the escort, and it should drive the
target choice directly.
* A frozen tail in the log (identical position, speed and target for the last
five seconds) is what mission-end looks like from the outside, **not** an
emulator wedge. Worth knowing before diagnosing the wrong thing.
* Practical: do **not** pipe a long run's log through `tail` — that discards
everything but the end, and the interesting part of this run is gone.
## After survival, the blocker is lethality (2026-07-30)
The 300 s run took **no damage at all** and killed **one** warplane, spending
~800 rounds of nose ammo (`06000``05193`) to do it, while `REMAINING OB` rose
from `004` to `011` as fresh waves spawned. So attrition at this rate never
finishes the mission, and the ranking of open problems has changed:
1. **Hit rate.** It opens fire at 25 km with a 9° cone and a crude lead
(`p + v·d/speed`, no projectile speed). The `Shell` records in
[weapon-struct-runtime](structures/weapon-struct-runtime.md) carry the real
projectile speed and `MaximumRange` per weapon — the lead and the firing
range should come from *those*, not from constants.
2. **Which targets count.** `REMAINING OB` is the mission's own objective
counter and it is on screen, so it is in RAM; finding it turns "shoot
whatever is nearest" into "shoot what closes the mission". Objective-marked
entities also draw an `OB` badge in the HUD, so the flag is likely a word in
the entity object.
3. **Confirming the shield word** — needs a run that actually takes damage; the
pilot is now good enough at avoiding that to make it awkward, so drive
straight at a turret on purpose with `--dry` steering disabled.
4. **Does the ACROPOLIS repair?** RETIRE mode has never triggered (the hull
never fell), so the resupply hint is still untested.
## The next step that unblocks the most (superseded — kept for the reasoning)
**Update 2026-07-30: this is no longer the blocker.** Entity typing via the
definition pointer already solved target selection, so the game's own target
pointer is now a convenience rather than a prerequisite. It would still be the
cheapest route to problem 2 above (objective targets), because whatever the HUD
locks on to is what the game itself considers a target.
**Find the game's own target pointer instead of typing entities ourselves.**
The HUD has a lock-on system (a `TARGET` marker and a target-cycle button), so
a global almost certainly holds a pointer to the currently-targeted entity.
Reading that gives an enemy's live object address directly — which yields both
target selection *and* the entity layout (position offset within it), i.e. it
collapses problems 3 and 4 into one. It is also cheap to find: cycle the target
with the pad and watch which pointer-shaped global changes in step.
## Operational notes
* An unattended craft **dies** — the ship flies straight into a firefight, and
two long scans were invalidated by a GAME OVER mid-run. Any scan longer than
~30 s needs either a survivable holding pattern or a fresh mission.
* `Xvfb` and the emulator die on their own every few minutes here, cleanly
(exit 0), cause unidentified. Everything that must not be interrupted is run
as **one background task** that starts the display, the emulator, the
navigation and the measurement together, so nothing has to survive between
tool calls.
* `pgrep` cannot be used for liveness in this container: PID 1 is
`sleep infinity` and never reaps, so dead processes linger as `<defunct>` and
still match by name. Use `ps -o stat=` and skip `Z`, or `xdpyinfo` for X.
* numpy is not installed system-wide; `pip install --break-system-packages
numpy` puts it in `/sylph-home/.local`, which is only on `sys.path` when
`HOME=/sylph-home`. Scripts run with `HOME=/sylph-home/re` need
`PYTHONPATH=/sylph-home/.local/lib/python3.12/site-packages`.
## Files
`pilot.py` (**the survival loop**) · `ctrl_probe.py` (input → speed calibration,
plus a per-tick window of the player object) · `binq.py` (query that capture) ·
`own_state.py` (definition-anchored hull/shield lookup) · `fly_session.sh`
(boot → mission → bind → fly, one task) · `wait_flight.sh` (wait for the real
HUD instead of a fixed sleep) · `navigator.py` (drift-aware steering + CPA
avoidance, reused by the pilot) ·
`gworld.py` (live reader + entity list) · `flight_probe.py` (scripted inputs +
sampling, and the `Pad` FIFO client) · `flight_analyze.py` · `whatchanges.py`
(encoding-agnostic "which words are live") · `findplayer.py` · `findself.py` ·
`findrot_global.py` · `findspeed.py` · `liveents.py` · `selfstate.py` (the
whole chain → JSON) · `autopilot2.py` (PD controller; **untested — it has never
had a valid config to run against**) · `launch_mission.sh`.

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.7 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

File diff suppressed because it is too large Load Diff

Binary file not shown.

After

Width:  |  Height:  |  Size: 195 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 233 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 218 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 188 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 167 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 220 KiB

File diff suppressed because it is too large Load Diff

Binary file not shown.

After

Width:  |  Height:  |  Size: 98 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 104 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 117 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 112 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 53 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 53 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 45 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 102 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 159 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 53 KiB

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,180 @@
# Capital-ship part placement — runtime capture (ground truth)
**Status:** ✅✅ **STATIC ASSEMBLY IS EXACT — no captures needed anymore
(2026-07-26).** The capture became the oracle that cracked the static encoding:
the node joint tables are **9 keyframe channels `[TX TY TZ RY RX RZ SX SY SZ]`
with the 8 table pointers shifted one track forward** (channel 0 = TX sits at
`ptr[0]0x48`; the old 8-slot read took TY as X, never saw TX — why hulls
stacked on the centreline). Euler order `Ry·Rx·Rz`, angles direct
(`mesh.rs::read_trs9` / `node_rotation`). `assemble_ship` now places every
composite-node instance (cross-id turrets ×2), mounts the `e_rou_<id>_eng`
cluster rig at `GN_Engine_01` (two mirrored nacelles + centre engine), puts
`brg`/`sld` at their exact GN frames, and X-reflects the shared-geometry twin
whose lateral offset opposes the geometry's dominant side.
`ship::tests::static_assembly_matches_runtime_capture` asserts static ==
captured for all 8 e106 parts (T < 1.0, R < 0.02) + both nacelles + both
turrets + the hull mirror. The viewer uses pure static assembly; the baked
table + correlator remain as the verification oracle only.
## Problem
Capital ships ship as **split XBG7 parts** (hull bodies, bridge, engines, turrets)
in `hidden/resource3d/Stage_SNN.xpr`. Reconstructing the whole ship needs each
part's placement. Static reverse-engineering of the composite scene graph
(`e_rou_<id>`, `GN_*` frames) got the **hull right but external parts only
approximately** — a `GN_*` frame is the mount *pivot*, and the real outboard /
rotational offset lives in a detail sub-rig or in runtime code not recoverable
statically. See `sylpheed-formats::ship::assemble_ship` (the static best-effort).
## Finding
The **guest vertex buffer holds LOCAL coordinates** — byte-identical to the
`.xpr` (verified: `e106_bdy_04`'s first vertex `(-20.0000,153.1539,45.9997)`
matches the decode exactly). So parts are placed **entirely in the vertex
shader**; the buffer carries no placement. (Contrast: static *stage* geometry is
pre-transformed into world space and byte-matches the `.xpr` at file offsets —
that's a different case.)
The per-part transform is in the **vertex-shader float constants**. For the ship
shader (`vs=0xC7F781F4C1D58054`), constants `c0..c2` are the **WorldViewProjection**
matrix rows:
- Their norms are `(sx, sy, 1)` = the projection x/y scales; `sy/sx ≈ 1.78 = 16:9`
aspect. Dividing each row by its norm gives the rigid **WorldView** (verified:
normalized rows orthonormal to 1e-4, `det = +1`).
- The camera View cancels when expressing every part relative to a reference part:
`rel_p = WV_ref⁻¹ · WV_p` = `(Rᵀ_ref·R_p , Rᵀ_ref·(T_p T_ref))` — a pure
ship-space rigid transform. Applying `rel_p` to part `p`'s local geometry
assembles the ship exactly, in the reference part's frame.
(`c4..c14` also vary per part — likely a normal matrix / a second WVP variant /
WorldView split; not needed, `c0..c2` suffice.)
## Capture instrumentation
Branch **`capture-ship-placement`** in the `xenia-canary-native` worktree (off the
`instrument-current`-descended native branch, which already has GPU `log_draws`).
- **F10** in the emulator window → `xe::gpu::RequestShipCaptureFrame()`.
- Dumps the next ≤8000 draws (de-duped by vertex-buffer address) to
**`xenia_ship_capture.log`** in the binary's dir
(`build/bin/Linux/Release/`). Per draw:
```
DRAW vbase=0x… stride=… vcount=… indices=… prim=… vs=0x…
pos: (x,y,z) … up to 64 LOCAL vertex positions (== .xpr)
vsconst base=N: c0=(x,y,z,w) c1=… … first 48 VS float4 constants
```
- Files: `src/xenia/gpu/command_processor.{cc,h}` (`CaptureShipDrawForRE`,
`RequestShipCaptureFrame`), `src/xenia/app/emulator_window.cc` (F10 key).
- Run: `run-canary-native.sh` (HW Vulkan, interactive). Play into the mission,
frame the ship side-on, press F10.
## Correlator (now a library module)
The correlation math lives in **`sylpheed-formats::ship_capture`** (unit-tested with
synthetic captures): `parse_capture` → `CapturedDraw`s, `correlate(id, draws,
parts, ref_sub)` → a `ShipPlacement` (each part in the reference frame), and
`serialize_table`/`parse_table` for the checked-in text table.
Two capture formats are accepted (auto-detected): the F10 ship-capture snapshot
([`parse_capture`]) and the **draw-logger** format `mission_draws.log`/
`xenia_re_draws.log` (`parse_drawlog` — groups the ship shader `SHIP_VS_HASH`'s
draws by vertex-buffer `base`, `vcount = size_words/stride_words`). So a capital
ship seen in a normal instrumented run can be baked without a dedicated F10 pass.
### Matching rules (learned from the real 2026-07-26 capture)
- **LOD-aware**: a distant ship draws its `_m`/`_l` copies — same part, same
local frame, different vcount. Each part tries every variant vcount.
- **Position-validated**: a vcount hit alone can be a coincidence (a foreign
51-vert mesh nearly hijacked the bridge slot). Every candidate draw's dumped
positions must be found in the part's decoded position set or it is rejected.
Validation is **set-based** (buffer order ≠ decode order) against the **union**
of the part's variants — the real capture had a draw whose `vcount` equalled
the `_m` count while its buffer held the FULL-detail geometry.
- **Runtime mirror**: a port/starboard pair shares one file geometry; the engine
uploads the second instance **X-reflected**. A draw that validates only under
X-negation is accepted as the mirror, and the placement bakes `diag(1,1,1)`
(the viewer reverses triangle winding for det < 0).
- The mission ship shader can be a different hash per lighting variant
(`0x3A0829CC71516789` in this capture) — the F10 path doesn't filter by hash,
it validates by geometry instead.
`examples/correlate_capture.rs` is the CLI wrapper:
```
SYLPHEED_ISO=… cargo run --release --example correlate_capture -- \
<capture.log> <Stage_SNN> <ship_id> [ref_part_substr] [--emit]
```
Decodes the ship's base parts (for their vertex counts = the match key), correlates,
prints each part's ship-relative transform, and with `--emit` prints the `ship …`
block to paste into the placement table.
## Baked table + viewer preference
`crates/sylpheed-formats/data/ship_placements.txt` is the checked-in placement
table (embedded via `ship_capture::embedded_placement`). The viewer's
`build_ship_model` uses a ship's captured entry when present, else falls back to
the static `assemble_ship` (`external` toggle). **The table is currently empty** —
a real F10 capture log is needed to bake in `e106` (and others); paste the
`--emit` block into that file and rebuild.
## Validated result — `e106` ADAN Destroyer (Mission 1 / Stage_S01, side view)
Vertex-count → part map (all matched exactly): bdy_01/02 = 1261, bdy_03 = 815,
bdy_04 = 1633, eng_01 = 583, eng_02 = 491, wep_02_01 = 1002, brg_01 = 202.
Ship-relative placement (reference = `bdy_04`, aft hull):
| part | T (ship-relative) | note |
|---|---|---|
| bdy_04 | (0, 0, 0) | reference, aft hull |
| eng_01 | (131, 133, 132) | **aft** (Z), starboard |
| eng_02 | (0, 49, 140) | aft, centre |
| bdy_01 | (**264**, 150, 1165) | **port** hull, forward |
| bdy_02 | (**+264**, 150, 1165) | **starboard** hull, forward |
| bdy_03 | (0, 35, 1076) | forward keel/spine |
| wep_02_01 | (0, 49, 1009) | forward weapon |
Assembled extent **872 × 670 × 2181** (a coherent ~2181-long destroyer).
**Key wins vs static:** `bdy_01`/`bdy_02` are a **port/starboard pair** (X = ∓264),
which static had overlapping at one point; engines land correctly aft; the ship is
not the 1894-tall tower the static Y put out.
**Missing:** `brg_01` (bridge, vcount 202) was **culled/occluded** at that camera
angle (0 draws) — needs a second capture framing the superstructure.
## What the static assembler gets wrong (measured, `e106` in Stage_S01)
Decoding the real container (`examples/ship_dump.rs`) shows the static
`assemble_ship` is **incomplete** — the capture is required to fix all of:
- **`bdy_01` / `bdy_02` overlap**: both get the *same* composite transform
(T = (116, 0, 857)); the real ship is a **port/starboard pair** at X = ∓264
(the mirror is applied at runtime, like the DeltaSaber fins in `saber_measured`).
- **`eng_02` and `wep_02_01` are never placed**: they aren't `rou_` hull nodes and
aren't in the external category list, so they drop out entirely.
- Only `brg_01` + `eng_01` get the approximate `GN_*`-frame treatment.
So even the *hull* tier is not fully correct, and no static signal supplies the
missing offsets (verified: the part vertex buffers are pure local coords). The
capture-driven table is the only path to correctness — there is nothing more to
squeeze statically.
> Offline status: the draw logs on this box are the **player fighter**, not a
> capital ship, so no capital-ship table can be baked here yet. Capture `e106`
> (Stage_S01, side-on) on the emulator box — F10 **or** just save
> `mission_draws.log` with the ship on screen — then `correlate_capture … --emit`.
## Next steps
1. ~~**Bake:** promote the correlator to a module + checked-in table + viewer
preference.~~ **DONE** (`ship_capture` module, `data/ship_placements.txt`,
`build_ship_model` prefers it). Remaining: run the captures below and paste the
`--emit` blocks in — needs the emulator (F10), can't be done offline.
2. **e106 + bridge:** re-capture `e106` (Stage_S01) framing the superstructure so
`brg_01` (culled last time) is included, then `--emit` → bake.
3. **Other ships:** one side-on F10 capture each for the cruiser (`e105`,
Stage_S02), ACROPOLIS (`f101`), TCAF cruiser/destroyer (`f105`/`f106`).
4. **Turrets:** the shared `e3NN`/`e4NN` gun models appear as their own draws in the
capture too — correlate them to place turrets (static never resolved these).

View File

@@ -0,0 +1,182 @@
# `.rat` — the UI element layout / animation record
**Status:**`CONFIRMED` for placement (2026-07-28). The retail UI can be
**reassembled from the disc**: the tutorial PAUSE menu rebuilds pixel-accurately from its
sprites placed at the coordinates in their `.rat` records — no fitting, no manual nudging.
![real vs rebuilt](captures/ui-layout/pause-tutorial-real-vs-rebuilt.png)
*Left: the running game (Canary screenshot). Right: rebuilt from `GP_PAUSE_MENU.pak` alone.
The remaining differences are the animated frame/glow sprites (`*eff*`) that were not
placed, and the live 3D background.*
## Screen composition
The UI is **one pak per screen**`GP_TITLE`, `GP_PAUSE_MENU`, `GP_READY_ROOM`,
`GP_SAVE_LOAD`, `GP_MISSION_SELECT`, `GP_OPTIONS`, `GP_SYSTEM`, `GP_TUTORIAL`,
`GP_STAGE_CLEAR`, `GP_MOVIE_THEATER`, `GP_DEBRIEFING_PILOTLOG`, `GP_LEADERBOARD`,
`GP_MISSION_LOG`, `GP_BUNK`, `GP_DIALOG`, `GP_GAMEOVER`, `GP_CHALLENGE`,
`GP_HANGAR_ARSENAL`.
Inside a screen pak, each top-level [RATC](../INDEX.md) bundle is **one (context × language)
build of that screen**. Its own header is the screen's **element declaration table**, and
the elements themselves follow as children:
| child | what it is |
|---|---|
| `<name>.t32` | the sprite ([T8aD](texture-color-k8888.md)) |
| `<name>.rat` | that sprite's **layout record** (this document) |
| `<screen>loop1.rat` | a looping sprite animation (see below) |
### The bundle header — element declaration table
```
0x14 u32 entry count
0x20 entry[count], 60 bytes each:
+0 char[28] element name, NUL-padded ("pgp_ttrl_eff10.t32", "pgp_ttrl_btn10.rat")
+28 u32 ×4 flags (0xffffffff / 0xffffffff / 0 / 0xffffffff on every entry seen)
+48 u32 pivot X
+52 u32 pivot Y
+56 u32 0
```
The table lists **both** sprites and `.rat` records — it is the screen's element list.
`pgp_ttrl` declares 11: six `eff*`, `msg`, and four `.rat`s (`title`, `btn10..12`).
**Verified:** for all 7 `.t32` entries the declared pivot is *exactly* half the decoded
texture's dimensions — `eff10` 408×120 → 204,60; `eff21` 428×360 → 214,180; `msg` 381×38 →
190,19; and so on, 7/7 with no mismatch (`tools/re-capture/ratc_decls.py`).
`GP_PAUSE_MENU.pak`'s six bundles are `{in-mission, tutorial} × {English, Japanese}`, with
the two in-mission builds each present twice at identical size. The purpose of that
duplicate is **NEEDS-HUMAN** (resolution or aspect variant?), and where the other four
shipped languages live is likewise unresolved — this pak holds only EN and JP.
Naming is transparent: `pgp` = pause screen, `pgp_ttrl_` = its tutorial context, `btnNN` =
menu item, `btnNNf` = that item's **focused** sprite, `eff*` = frame/glow decoration,
`deli*` = the item divider, `title`, `msg`.
## Record layout
Big-endian u32 throughout (Xbox 360), and **tag-driven**: 4-char ASCII tags (`opt `,
`PRMD`, `end `) mark sections, so a record is a stream of blocks rather than a fixed struct.
A minimal record (a static button) is 165 bytes:
```
0x00 "RATC" magic — a RATC bundle reused as a data record
0x08 u32 payload size
0x14 u32 entry count (loop1.rat: 3, matching its 3 sprite names)
0x18 u32 design width = 1280
0x1c u32 design height = 720
0x20 char[16] the sprite this record places, e.g. "pgpbtn00.t32"
0x50 u32 pivot X = texture width / 2
0x54 u32 pivot Y = texture height / 2
...
── placement block ──
u32 scale X = 100 (percent)
u32 scale Y = 100
u32 tint = 0xffffffff (RGBA, white = untinted)
u32 X ← top-left position
u32 Y ←
u32 time (keyframe records only)
...
"opt " u32 len char[len] link to another record, e.g. "pgpbtn00f.rat"
```
- **X/Y is the sprite's top-left**, not its centre: compositing at these coordinates
reproduces the screenshot, which drawing centred on them would not.
- **The pivot at 0x50/0x54 is half the texture size** — 4 of the 5 plain sprites match
exactly (`pgpbtn00` 86×42 → 43,21; `pgpbtn05` 221×42 → 110,21; `pgptitle` 202×73 →
101,36; `pgpbtn04` 172×43 → 86,22). It is a rotation/scale centre, not a draw offset.
- **Animated elements are a keyframe list.** `pgptitle.rat` (752 B) is the same placement
block repeated with a varying trailing `time` field — the PAUSE title's fly-in.
- **`opt `** carries a length-prefixed record name. On `pgpbtn00.rat` it points at
`pgpbtn00f.rat`, i.e. *normal state → focused state*. Focused records place their sprite
42 px left and 8 px up of the base, because the focused art includes the selection ring
that hangs off the left edge; their pivot is a constant (21,25) rather than half-size.
- `<screen>loop1.rat` is **not** a screen composition — it is a looping sprite animation:
a 3-name table (`pgpeff34/35/36.t32`) plus ~30 keyframes all at one position.
- The small (380 B) top-level entries are **`PRMD` primitives**, not sprites: a colour and
four explicit corner coordinates `(0,0) (1280,0) (0,720) (1280,720)` — the full-screen
quad that dims the scene behind the pause menu — terminated by `end `.
### The records are language-independent
`pgpbtn00.rat` is **byte-identical** in the English and Japanese bundles. The layout is
authored once and only the `.t32` sprites are swapped, which has two consequences:
- The baked pivot belongs to *whichever build the record was authored from*, not to the
sprite actually shipped beside it. That is why `pgpbtn01`'s pivot (113 → a 226 px wide
texture) matches neither the English sprite (207) nor the Japanese one (148).
- The split is visible inside a single bundle: in the **English** tutorial build, every
`.t32` declaration carries the correct English pivot (7/7), while the `.rat` declarations
carry Japanese-derived ones (`btn10` → 43,21 = 86/2, the *Japanese* sprite). So the
`.t32` table is regenerated per language and the `.rat` layer is inherited from the
Japanese master.
- **Do not infer anything about a language from a texture size** — see the traps below.
## Evidence
Positions were read out of the records and checked against a screenshot of the running
game, twice, in that order — the records were never fitted to the picture.
1. **Differential.** Across `pgpbtn00/01/04/05.rat`, exactly one field varies and it steps
`268 → 338 → 408 → 478` — a constant 70 px pitch — while the field before it is 226 in
all four. A vertical menu: constant X, evenly spaced Y.
2. **Absolute placement — the decisive test.** The *tutorial* build's records give
546/288, 546/358, 546/428 and 540/119. Compositing its sprites at exactly those numbers,
with no offset and no fitting, reproduces the screenshot (image above).
3. **Pivot.** 4 of 5 plain sprites carry exactly half their texture's dimensions at
0x50/0x54 (above).
> A caution on step 2, because the first pass here got it subtly wrong: the screenshot is
> of the **tutorial** pause menu, so only the `pgp_ttrl_*` records can be checked against
> it. The in-mission records (X = 226) also map onto the same screenshot under a single
> constant offset — but that only works because both builds share the 70 px pitch, and it
> proves nothing. The in-mission coordinates remain **unverified**: confirming them needs a
> screenshot of a pause during an actual mission.
## It generalizes — the title screen
The same method run against `GP_TITLE.pak` reproduces the **main menu**, which is a
different screen with a different item count and a different pitch:
![main menu real vs rebuilt](captures/ui-layout/title-mainmenu-real-vs-rebuilt.png)
`ptbtn01..05.rat` give X = 542 for all five and Y = 162 / 242 / 322 / 402 / 482 — an
**80 px** pitch, where the pause menu used 70. Measured against the screenshot, the sprite
tops land at a constant **+46 px** for all five (one reads 45, a 1-px edge-detection
wobble), and 46 is exactly the 45 px of Xenia window chrome plus one. So the record's Y is
the sprite's top edge in the guest framebuffer, to the pixel, on a second screen.
`GP_TITLE.pak` also splits by sub-screen the way the pause pak splits by context:
`ptbtn00` alone (the `PRESS Ⓐ BUTTON` prompt), `ptbtn01..05` (main menu), `ptbtn11..13`
(the EXTRAS submenu), plus `pgloading_*` for the loading screen.
## Two traps this caught
Both were mistakes made during this analysis, caught by comparing against the real game —
recording them because a static-only reading would have shipped them:
- **Texture width does not identify a language.** English `RESUME` (166 px) and Japanese
`再開` (86 px) differ hugely, but Japanese `通信ログ` (148 px) is within a few px of an
English label. The first language assignment made here was wrong; rendering the sprites
is the only reliable check. (The records being language-independent makes this worse:
a record's baked pivot implies a texture width that matches *no* shipped sprite.)
- **The in-mission and tutorial pause menus are different sprite sets, not one set
re-packed.** In-mission is `RESUME / RADIO LOG / OPTIONS / BACK TO TITLE` (4 items,
`pgpbtnNN`); the tutorial is `RESUME / OPTIONS / BACK TO MENU` (3 items,
`pgp_ttrl_btn1N`). Matching the 3-item screenshot against the 4-item set suggests a
runtime slot-packing rule that does not exist.
## Next
- **The `eff*` / `deli*` / `msg` placements are still missing** — those sprites have no
`.rat` of their own, and `loop1.rat` turned out to be an animation, not a composition.
So the screen's draw list lives somewhere not yet found (the parent RATC's own header
region, or title code). That is the gap between the rebuild above and a complete screen.
- The same method should now unroll the other screens directly; `GP_HANGAR_ARSENAL.pak`
(789 T8aD + 510 RATC) is the big one, and the ARSENAL `DATA SHEET` panel documented in
[weapon-datasheet-runtime.md](../weapon-datasheet-runtime.md) is a ready-made oracle for it.
- Tooling: `crates/sylpheed-formats/examples/ui_screen.rs` (inventory a screen pak, carve a
named RATC child), `sylpheed-cli pak textures` (decode every sprite).

View File

@@ -0,0 +1,274 @@
# Runtime `Unit` struct (craft / vessel definitions) — read from live guest memory
**Confidence: ✅ CONFIRMED** for the 27 fields marked ✅ below (each binding is
reproduced by 1021 independent disc records, on ≥3 distinct values, with
**zero** contradictions), plus the Maneuver block's declaration-order layout
(29 anchors, two bases, no conflicts). 🟡 PROBABLE for the fields interpolated
between confirmed anchors. 🟡/❔ for the thin single-value bindings, which are
listed but must not be trusted yet.
Captured 2026-07-29 from Xenia Canary running the retail disc in the sylph-re
container: **all six tutorials** and **Stage 02 "Declaration of War"** loaded
from save slot 01. 21 of the disc's 110 units, over 7 snapshots from 7 separate
emulator runs.
## Why this exists
Craft stats were the one Route-B target the previous two passes could not
reach. The Hangar exposes only `Gross Weight` (a class, not a number), and
[the menu route](../weapon-datasheet-runtime.md) has no surface at all for the
flight model. From the memory side, menus were equally useless: **only the
player craft's name string is resident there — the definition objects do not
exist until a mission loads.**
They do exist in-mission. This documents their layout and the values the disc
leaves defaulted.
## Finding the class (discovered, not assumed)
[`tools/re-capture/unit_discover.py`](../../../tools/re-capture/unit_discover.py)
takes no vtable as input. It locates every disc unit-ID string in a memory
snapshot, finds every aligned word pointing at `string_va - d` for a range of
`d`, and tallies the word at `pointer_site - k` across **distinct** unit IDs.
One `(d, k, word)` combination wins by a wide margin:
| δ (name-record → string) | ID pointer at | word | distinct units |
|---|---|---|---|
| `0x10` | object `+0x04` | `0x820af030` | 4 |
| `0x10` | object `+0x04` | `0x820af844` | 4 |
— i.e. exactly the `Weapon` shape: the object holds a name-record pointer at
`+0x04`, and the ID string sits at `name_record + 0x10`.
The two vtables are **two different things**, and telling them apart matters:
| vtable | what it is | evidence |
|---|---|---|
| `0x820af030` | **spawned entity record** — one per spawned thing, but **not live state** (see below) | 12 objects for 4 IDs; the same ID appears many times (one per box in the scene); irregular spacing |
| `0x820af844` | **parsed definition** — the `.tbl` | exactly one object per distinct unit ID; minimum spacing `0x380` |
**Correction (2026-07-29, from the autopilot work):** `0x820af030` was
described here as holding live state. It does not — across a 29 s in-flight
capture **all 384 words of it are constant**. It is one record per spawned
thing, but the flying entity's transform is somewhere else entirely. See
[autopilot-memory-driven](../autopilot-memory-driven.md). Nothing below depends
on it; the definition class `0x820af844` is unaffected.
Only `0x820af844` is used. It is the runtime image of the `.tbl`:
* **Within one run** it is byte-identical across two snapshots taken ~12 minutes
apart with combat in between (14/14 objects, 0 differing bytes) — definition
data, not live state.
* **Across runs** the same unit is *not* byte-identical, and that had to be
explained rather than waved away. `unit_runtime.py --crosscheck` compares
every unit that appears in more than one snapshot (5 of them, over 7 runs):
exactly **15 words differ**, and 13 of them hold guest pointers
(`0x8xxxxxxx`/`0xbxxxxxxx` — heap addresses, which move per process).
**No solved or interpolated field offset is among the 15** — every value
reported here is run-invariant.
* The two non-pointer stragglers, `+0x2c8` and `+0x2d0`, are **stage-dependent**:
for one and the same unit (`UN_f001_TCAF_DeltaSaber_T_Ttrl`) `+0x2c8` reads
`8000.0` in two tutorials and `10000.0` in a third, with `+0x2d0` a 0/1 flag
beside it. So the object is *mostly* but not *entirely* the parsed table —
a couple of words are set per stage. Unidentified; **NEEDS-HUMAN**.
One `.tbl`**one** object. A unit table is several sub-records (`Generic`,
`Maneuver`, `Shield`, `Explosion`, `Mass`, `Effect`, `SE`, `Turret_00N`), and
they are all flattened into that single ≥`0x380`-byte object — unlike weapons,
where `Weapon` and `Shell` are separate arrays.
## Solving the layout
Same discipline as
[`weapon_runtime.py`](../../../tools/re-capture/weapon_runtime.py): score every
`(field, byte-offset, encoding)` triple against the disc records and accept a
binding only on **zero contradictions**, requiring ≥3 distinct values so a
field whose samples are all one number cannot match any offset holding that
constant.
```bash
python3 tools/re-capture/unit_runtime.py unit_tokens.txt snap_a.bin snap_b.bin --csv
```
Snapshots are **unioned** — each mission instantiates only the units in its own
stage, so coverage grows by visiting stages. `cp --sparse=always` a copy of
`/dev/shm/xenia_memory_*` first (~2 s); the running emulator pegs every core
under lavapipe and makes repeated live reads flaky.
### Encoding note — angles are radians at runtime
Every `AV_*` / `AA_*` / `*Bank*` / `Turn_AngularVelocity` field is stored as
**float32 radians**, while the disc writes **degrees**. The solver needed a
`rad` encoding (`degrees(f32)`) to bind them at all; 18 units agree on
`AV_PitchPlus_Max` alone. A reimplementation reading the `.tbl` must convert.
## Confirmed layout
✅ = ≥10 disc records agree on ≥3 distinct values, zero contradict.
| offset | enc | field | agree | distinct |
|---|---|---|---:|---:|
| `+0x030` | f32 | `Size_X` | 21 | 13 |
| `+0x034` | f32 | `Size_Y` | 12 | 7 |
| `+0x038` | f32 | `Size_Z` | 19 | 13 |
| `+0x040` | f32 | `Color_R` | 21 | 5 |
| `+0x044` | f32 | `Color_G` | 19 | 6 |
| `+0x048` | f32 | `Color_B` | 17 | 5 |
| `+0x050` | f32 | `Size_Radius` | 12 | 10 |
| `+0x054` | f32 | `HP` | 19 | 10 |
| `+0x074` | f32 | `ResistanceToOptics` | 11 | 3 |
| `+0x08c` | f32 | `ScorePoint` | 21 | 9 |
| `+0x094` | f32 | `MassScore` | 10 | 7 |
| `+0x09c` | f32 | `MinimumVelocity` | 11 | 3 |
| `+0x0a0` | f32 | `MaximumVelocity` | 19 | 8 |
| `+0x0a4` | f32 | `CruisingVelocity` | 17 | 7 |
| `+0x0a8` | f32 | `Acceleration` | 18 | 5 |
| `+0x0ac` | f32 | `Deceleration` | 17 | 5 |
| `+0x0b0` | rad | `AV_PitchPlus_Max` | 18 | 7 |
| `+0x0b4` | rad | `AV_PitchPlus_Min` | 10 | 6 |
| `+0x0c4` | rad | `AV_PitchMinus_Min` | 10 | 5 |
| `+0x0f8` | f32 | `SideThrustVelocity_Max` | 14 | 3 |
| `+0x238` | f32 | `MaxValue` (Shield) | 13 | 6 |
| `+0x244` | f32 | `ChargeSpeed` (Shield) | 13 | 6 |
| `+0x270` | f32 | `DestroyMotionTime` | 19 | 7 |
| `+0x2a0` | f32 | `RadarRange` | 18 | 9 |
| `+0x2a4` | f32 | `FCSRange` | 12 | 8 |
| `+0x2b4` | f32 | `AttackVesselPoint` | 14 | 8 |
| `+0x2bc` | f32 | `DefencePoint` | 12 | 7 |
`HQRatio +0x058`, `ShieldRatio +0x05c`,
`ThrusterRatio +0x060`, `ResistanceToShell +0x078`,
`ResistanceToExplosion +0x07c`, `ResistanceToPlayer +0x080`,
`ResistanceParalyze +0x084`, `BridgeCount +0x070` (i32),
`DryMass +0x274`, `GrossMass +0x278`,
`LowerHPThresholdRatio +0x298`, `AttackCraftPoint +0x2b8` bind with zero
contradictions on fewer records or fewer distinct values — 🟡 PROBABLE. The
full solver output is in
[`captures/unit-runtime-fields.csv`](../captures/unit-runtime-fields.csv)
(1 827 values, `conf` column).
## The Maneuver block is laid out in schema declaration order
This is the strongest structural result and it is independent of any single
field's agreement count.
[`schema_order.py`](../../../tools/re-capture/schema_order.py) merges the
`Maneuver` field-name order from **all 113 unit tables** by topological sort
over their pairwise "k[i] precedes k[i+1]" constraints. The merge is acyclic and
**every one of the 113 tables is a subsequence of the merged 102-field order**
so that order is the schema's.
Against it, the solved offsets fall into two exact runs:
| declaration indices | offset rule | anchors that fit |
|---|---|---|
| 0 … 20 (`MinimumVelocity``AA_Roll_Min`) | `0x09c + 4·i` | 21 / 21 |
| 21 … 32 (`SideThrustVelocity_Max``AccPitchFactor`) | `0x0a4 + 4·i` | 8 / 8 |
One 4-byte slot per field, with a **two-slot gap after `AA_Roll_Min`**
(`0x0f0``0x0f4`, purpose unknown). 29 independently-derived anchors, zero
conflicts, across a 0x9c0x125 span.
### Fields NO disc record ever values — 🟡 PROBABLE
Five `Maneuver` fields are declared by the schema but left at their default by
*every* one of the 110 unit tables, so no amount of disc analysis can ever
reach them. The declaration-order rule pins them between confirmed anchors
(`YawDragFactor +0x104``ArterBurner_Vc +0x114`, and
`ReverseThrust_Vc +0x118``ReverseThrust_Acc +0x120`):
| offset | field | craft (`f00*`, `e0*`) | capital ships (`*1**`, `e2*`) | inert (`SchlosBase`, `Box`) |
|---|---|---:|---:|---:|
| `+0x108` | `PitchDragFactor` | 3 | 2 | 0 |
| `+0x10c` | `RollDragFactor` | 3 | 2 | 0 |
| `+0x110` | `DragFactorThreshold` | 0.5 | 0.5 | 0 |
| `+0x11c` | `ArterBurner_Acc` | 2 | 2 | 0 |
| `+0x128` | `DecPitchFactor` | 30 | 30 | 0 |
Corroboration beyond the interpolation, checked over all 18 units:
* `PitchDragFactor == RollDragFactor == YawDragFactor` holds **18/18** — and
`YawDragFactor` is *disc-supplied* (3.0 for craft, 2.0 for warships), so two
interpolated offsets reproduce a known number, per unit, every time.
* `DecPitchFactor == AccPitchFactor` holds **17/18**. The exception is
`UN_e015_ADAN_Puppy` (`AccPitchFactor` 1, `DecPitchFactor` 0.5) — which
defaults *both* on disc, so it is two independent fields that happen to be
set equal elsewhere, not a broken binding.
`UN_e015_ADAN_Puppy` also breaks the craft/warship bucketing above for
`DecPitchFactor` (0.5, not 30); the per-unit values are in the CSV.
The tail of the `Maneuver` block (the AI-behaviour fields — `SideRoll_*`,
`BarrelRoll_*`, `TurnAttack_*`, `HoldPosition_*`, `Slalom_*`, `Through_*`,
`SolidCutoff_*`, and the `AxisMode` / `AB_*` sub-block) is **NOT resolved**.
Those fields are declared by only a handful of tables and almost always with a
single distinct value, so the solver's bindings there are coincidences: it
placed `Slalom_CutoffRatio` at `+0x00c` and `TurnAttack_DoubleTimeMin` at
`+0x110`, both of which the declaration-order rule contradicts. They are marked
`tentative` in the CSV. **NEEDS-HUMAN / needs more coverage** — more stages
would give those fields distinct values and settle it.
## The player craft, `UN_f001_TCAF_DeltaSaber_T_Player`
30 of its fields are defaulted on disc. Notable recovered values:
| field | value | note |
|---|---|---|
| `Size_Radius` | 10 | ✅ |
| `FCSRange` | 500000 | ✅ — same as `RadarRange` |
| `ChargeSpeed` (shield) | 25 | ✅ |
| `ResistanceToOptics` | 1 | ✅ |
| `HQRatio` / `ShieldRatio` / `ThrusterRatio` | 1 / 1 / 1 | 🟡 |
| `ResistanceToShell` / `ResistanceToExplosion` | 1 / 1 | 🟡 |
| `DryMass` | 100 | 🟡 (`GrossMass` 250 is on disc) |
| `LowerHPThresholdRatio` | 0.3 | 🟡 |
| `ChargeDelay_Break` | 10 | 🟡 |
| `MassScore` | 0 | 🟡 |
Every AI-behaviour field the solver bound reads **0** for the player craft,
which is the expected shape (the player is not AI-driven) — but see the caveat
above: those offsets are not settled, so treat the zeros as consistent, not
proven.
The `…Ratio` family that the Route-B target list parked is **1.0 for almost
every unit**, with real exceptions that only the runtime shows:
`UN_e105_ADAN_Cruiser` `HQRatio` = 0.2, `UN_bf001_TCAF_SchlosBase` and
`UN_e106_ADAN_Destroyer` `ThrusterRatio` = 0.2,
`UN_n001_TTRL_Box` `ShieldRatio` = 0.3.
## Coverage and how to extend it
21 of 110 units. Unlike weapons — where one snapshot held all 126 — **unit
definitions are instantiated per stage**, so coverage is bounded by the stages
reachable from the save (slot 01 is at 5 %, Stage 02). `unit_runtime.py` unions
any number of snapshots and re-solves, and more units directly promote the 🟡
bindings to ✅ by adding distinct values — the six tutorials took the confirmed
set from 22 fields to 27.
The tutorials are nearly exhausted as a source: all six together contribute only
3 units the missions do not already have (`UN_f001_TCAF_DeltaSaber_T_Ttrl`,
`UN_e015_ADAN_Puppy_2`, `UN_f001_TCAF_DeltaSaber_T_Player_Ttrl2`) — they reuse
one training box, one target drone and the player craft. **Real coverage now
needs real missions**, i.e. story progress on the save.
`tools/re-capture/grab_tutorial.sh` captures one tutorial per invocation
(cold boot → menu → Nth entry → snapshot, ~2.5 min). It cold-boots for each
because backing out of a loaded mission via PAUSE → BACK TO MENU wedges the
emulator. Two timing facts it encodes, both learned the hard way: the main menu
is **not input-ready for ~10 s** after the title tap, and d-pad presses before
that are silently dropped — which sends the A to `NEW GAME` instead of
`TUTORIAL`. And `NEW GAME` is not a cheap way to reach Stage 01: it gates on a
DIFFICULTY menu and then plays the prologue movie.
A NEW GAME excursion as far as the READY ROOM leaves
`535107D4/00000001/game01/savedata` byte-identical — only the profile `.gpd`
achievement files change — so it does not endanger the 5 % save. Verified by
diff against a backup, not assumed.
**A stage's whole unit set is parsed at load, not as waves spawn** — checked by
counting the definition objects at three points in Stage 02: immediately after
take-off, ~12 minutes in mid-combat, and after GAME OVER. 14 objects, the same
14 IDs, every time. So capturing a stage costs one load and one snapshot; there
is no need to play it, and no need to survive it.
Stages captured so far: `Ttrl` (BASIC CONTROLS), Stage 02.

View File

@@ -0,0 +1,294 @@
# Runtime `Weapon` / `Shell` structs — read from live guest memory
**Confidence: ✅ CONFIRMED** for the fields marked ✅ below (each binding is
reproduced by 10125 independent disc records with **zero** contradictions);
🟡 for the thin ones. Captured 2026-07-29 from Xenia Canary running the retail
disc, save slot 01 (Stage 02, 5 % progress), READY ROOM and ARSENAL.
## Why this exists
`weapon\Weapon_*.tbl` is an [IDXD](../INDEX.md) record whose string pool **omits
every field left at its default**. That parked a long list of stats as
unreadable — the `…Ratio` / `…Count` family, `Power`, `MaximumRange`. The
[Arsenal DATA SHEET route](../weapon-datasheet-runtime.md) recovered a few of
them but only as letter buckets (Range/Damage as `A``E`), and only for the 9
weapons unlocked at 5 % progress.
This reads the values **directly out of the running game's memory** instead.
All 126 weapons, all fields, exact numbers, in one pass — and it needs no story
progress, because the definitions are parsed at load time whether or not the
player has unlocked the weapon.
## The lever: Canary maps guest RAM into `/dev/shm`
Xenia Canary backs the entire guest address space with one shared-memory file,
`/dev/shm/xenia_memory_<id>`. It is a plain file: **the guest's RAM is readable
from the host with `open`/`seek`/`read`, live, no debugger and no emulator
patch.** Guest VAs map into it through Xenia's fixed table (`memory.cc`), which
[`tools/re-capture/gmem.py`](../../../tools/re-capture/gmem.py) implements:
```bash
python3 tools/re-capture/gmem.py find "Weapon_DSaber_P_wep_01" # search all of RAM
python3 tools/re-capture/gmem.py words 0xbccce500 48 # dump as BE u32/f32
```
The file is sparse (~212 MB resident of 4.5 GB), and the scan uses
`SEEK_DATA`/`SEEK_HOLE`, so a full-RAM search costs ~0.2 s.
## Finding the objects
1. The title's **schema field-name pool** is in the XEX at `0x82086a30`, in
declaration order: `EnumWeapon`, type `Weapon` (`ID`, `Name`, `TargetType`,
`CartridgeModelName`), then type `Shell` (`ID`, `Name`, `MovementType`, …
`Explosion_MaxDamageRadius`). This is exactly the on-disc field order. No
pointer to these strings exists anywhere in RAM — PPC builds the addresses
with `lis`/`ori` immediate pairs — so the schema descriptor cannot be found
by pointer-chasing; the layout has to be solved instead (below).
2. Each parsed record becomes **two C++ objects**, a `Weapon` and its `Shell`,
each in its own contiguous array, each identified by its **vtable pointer**:
| class | vtable VA | stride | count |
|-------|-----------|--------|-------|
| `Weapon` | `0x820af548` | `0xc0` | 126 |
| `Shell` | `0x820af58c` | `0x200` | 126 |
The vtable VAs are static (XEX `.data`); the array base addresses are heap
and are **not** assumed — the tool locates every object by scanning RAM for
the vtable word.
3. Object `+0x04` points at a 0x40-byte **name record**; the ID string sits at
`+0x10` inside it. That is what keys each object back to its disc record.
`Weapon``Shell` pairing is by ID (`Weapon_X``Shell_X`) — the two arrays
are in **different orders**, so index-pairing would be wrong.
## Solving the layout (the part that makes this evidence, not guesswork)
[`tools/re-capture/weapon_runtime.py`](../../../tools/re-capture/weapon_runtime.py)
brute-forces every `(field, byte offset, encoding)` triple and scores it against
the disc: how many records does this offset *reproduce*, and how many does it
*contradict*? A binding is accepted only with **zero contradictions**, and is
marked ✅ only when ≥10 records agree on ≥3 distinct values.
That threshold matters: a field whose disc samples are all the same number
matches any offset holding that constant, so agreement count alone is not
evidence — the number of **distinct** values pinned down is. Two fields landing
on one offset is impossible in a real struct, so collisions are resolved to the
better-evidenced field and the loser is reported as unsolved. Bindings that
contradict more than 20 % of their samples are discarded outright rather than
reported on their agreeing subset.
Encodings found: `f32`, `u32`, `deg` (**angles are stored in radians**;
`SprayAngle`, `AngularVelocity` and `SplitCone` are degrees on disc), and `cnt`
— see the next section.
### `IsCharging` switches the counter encoding ✅
`LoadingCount` and `TriggerShotCount` at `+0x28` / `+0x44` are **int32 on 118
records and float32 on 8**. The 8 are exactly the records with
`IsCharging = Yes` — an exact partition, found by testing every disc field/value
pair against the float-encoded set. A charging weapon drains its magazine
continuously, so its counter needs a fraction.
Any reimplementation must branch on `IsCharging` when reading these two fields;
reading them as int unconditionally yields `1125515264` instead of `150`.
## Struct maps
See [`docs/re/captures/weapon-runtime-fields.csv`](../captures/weapon-runtime-fields.csv)
for the full machine-readable table — 7 182 rows, every confirmed field × every
one of the 126 records, tagged `disc` or `defaulted-on-disc`. **4 393 of those
values were not readable from the disc at all.**
### `Weapon` (`0xc0` bytes)
| offset | enc | field | agree | distinct | conf |
|--------|-----|-------|------:|---------:|------|
| `+0x010` | u32 | `ReticleType` | 62 | 13 | ✅ |
| `+0x01c` | u32 | `SpecialWeaponType` | 73 | 6 | ✅ |
| `+0x028` | cnt | `LoadingCount` | 121 | 33 | ✅ |
| `+0x02c` | f32 | `Interval` | 125 | 28 | ✅ |
| `+0x030` | f32 | `ReadyInterval` | 120 | 10 | ✅ |
| `+0x034` | f32 | `Heating` | 114 | 30 | ✅ |
| `+0x038` | f32 | `Cooling` | 106 | 21 | ✅ |
| `+0x03c` | f32 | `Mass` | 122 | 42 | ✅ |
| `+0x040` | f32 | `HitRatio` | 102 | 5 | ✅ |
| `+0x044` | cnt | `TriggerShotCount` | 114 | 17 | ✅ |
| `+0x048` | f32 | `TriggerShotInterval` | 99 | 11 | ✅ |
| `+0x050` | deg | `SprayAngle` | 96 | 17 | ✅ |
| `+0x06c` | f32 | `LockIntervalSingle` | 61 | 9 | ✅ |
| `+0x070` | f32 | `LockIntervalMulti` | 28 | 9 | ✅ |
| `+0x09c` | f32 | `MaximumCharging` | 3 | 3 | 🟡 |
Also identified structurally, not by the solver: `+0x00` vtable, `+0x04` name
record, `+0x0c` `TargetType` as a bitmask (`Vessel|Craft|Structure` = 7),
`+0x14`/`+0x18` name hashes, `+0x7c`/`+0x80` muzzle-flash FX name record + hash.
Unsolved (no consistent offset): `Cracker_ShotInterval`, `Cracker_SubShotCount`,
`MinimumCharging`, `MultiTargetCount`.
### `Shell` (`0x200` bytes)
| offset | enc | field | agree | distinct | conf |
|--------|-----|-------|------:|---------:|------|
| `+0x00c` | cnt | `DeleteFadeSpeed` | 39 | 2 | 🟡 |
| `+0x010` | cnt | `GuidanceType` | 9 | 4 | 🟡 |
| `+0x014` | cnt | `SpiralType` | 3 | 1 | 🟡 |
| `+0x020` | f32 | `Length` | 107 | 3 | ✅ |
| `+0x024` | f32 | `Volume` | 19 | 3 | ✅ |
| `+0x028` | f32 | `ShellMass` | 110 | 34 | ✅ |
| `+0x03c` | f32 | `HP` | 28 | 2 | 🟡 |
| `+0x040` | f32 | `LifeTime` | 94 | 43 | ✅ |
| `+0x044` | f32 | `FadeInTime` | 13 | 2 | 🟡 |
| `+0x048` | f32 | `FadeOutTime` | 7 | 1 | 🟡 |
| `+0x04c` | f32 | `Velocity` | 106 | 15 | ✅ |
| `+0x050` | f32 | `MinimumVelocity` | 11 | 2 | 🟡 |
| `+0x054` | f32 | `MaximumVelocity` | 29 | 7 | ✅ |
| `+0x058` | deg | `AngularVelocity` | 49 | 19 | ✅ |
| `+0x05c` | f32 | `Acceleration` | 9 | 4 | 🟡 |
| `+0x064` | f32 | `BeginGuidanceTimeAdjust` | 24 | 2 | 🟡 |
| `+0x068` | f32 | `EndGuidanceTime` | 19 | 4 | ✅ |
| `+0x070` | f32 | `Spiral_BeginTime` | 2 | 2 | 🟡 |
| `+0x074` | f32 | `Spiral_BeginTimeAdjust` | 25 | 2 | 🟡 |
| `+0x08c` | f32 | `MinimumRange` | 43 | 7 | ✅ |
| `+0x090` | f32 | `MaximumRange` | 117 | 21 | ✅ |
| `+0x0a0` | f32 | `Color_R` | 18 | 4 | ✅ |
| `+0x0a4` | f32 | `Color_G` | 121 | 3 | ✅ |
| `+0x0a8` | f32 | `Color_B` | 100 | 2 | 🟡 |
| `+0x0b4` | f32 | `Radius` | 89 | 10 | ✅ |
| `+0x0b8` | f32 | `Power` | 101 | 37 | ✅ |
| `+0x0bc` | f32 | `FailedDamageRatio` | 2 | 2 | 🟡 |
| `+0x0c0` | f32 | `PlayerLaserPower` | 11 | 6 | ✅ |
| `+0x0fc` | f32 | `ChaffResistRatio` | 24 | 1 | 🟡 |
| `+0x104` | f32 | `ChargingSizeRatio` | 3 | 1 | 🟡 |
| `+0x10c` | f32 | `SphereRadiusBegin` | 9 | 6 | 🟡 |
| `+0x110` | f32 | `SphereRadiusTurn` | 9 | 8 | 🟡 |
| `+0x114` | f32 | `SphereRadiusEnd` | 10 | 8 | ✅ |
| `+0x118` | f32 | `ExplosionTurnTime` | 3 | 2 | 🟡 |
| `+0x11c` | f32 | `ExplosionLifeTime` | 7 | 6 | 🟡 |
| `+0x120` | f32 | `ExplosionDamage_Maximum` | 4 | 4 | 🟡 |
| `+0x124` | f32 | `ExplosionDamage_OuterEdge` | 8 | 6 | 🟡 |
| `+0x128` | f32 | `Explosion_MaxDamageRadius` | 8 | 3 | 🟡 |
| `+0x130` | f32 | `SplitTime_Maximum` | 2 | 2 | 🟡 |
| `+0x134` | deg | `SplitCone` | 2 | 2 | 🟡 |
| `+0x148` | cnt | `NodeCount` | 39 | 4 | ✅ |
| `+0x164` | f32 | `Deceleration` | 9 | 2 | 🟡 |
Unsolved: `BeginGuidanceTime`, `EndGuidanceTimeAdjust`, `Spiral_EndTime`,
`SplitTime_Minimum`, `StartingVelocity`, `Straight1Type`, `st1_df_pitch`,
`pitch0`, and the `sp_qu_*` / `sp_sp_*` / `sp_zg_*` spiral-motion family. Those
are declared by too few records (or by none that the solver could separate) —
they need a state where the shells are actually in flight.
## The answer to the parked Route-B question
Player-weapon fields that are **defaulted on disc**, now read exactly (`·` = the
disc carries the value already):
| wep | LoadingCount | TriggerShotCount | MaximumRange | Power | Heating | Cooling | ReadyInterval | HitRatio | SprayAngle |
|---|---|---|---|---|---|---|---|---|---|
| 01 | · | **1** | · | · | · | · | · | · | · |
| 02 | · | · | · | **100** | · | · | · | **1** | · |
| 03 | · | · | · | · | · | · | · | · | **1** |
| 05 | · | **4** | · | · | · | · | · | · | · |
| 08 | · | · | · | · | · | · | · | **1** | · |
| 09 | · | · | · | · | **0.02** | · | · | · | **1** |
| 11 | **6** | · | · | · | · | · | · | · | · |
| 12 | · | · | · | · | **0** | · | · | **1** | · |
| 13 | · | · | · | **300** | · | · | · | · | · |
| 14 | · | · | · | · | · | · | · | · | **1** |
| 19 | · | · | · | · | · | **0.2** | · | · | **1** |
| 24 | · | **1** | · | · | · | · | · | · | **1** |
| 25 | · | · | **4000** | · | · | · | · | · | · |
| 26 | · | · | · | **500** | · | · | · | · | · |
| 27 | · | **1** | · | · | · | · | · | · | · |
| 28 | **5** | · | · | · | · | · | · | · | · |
| 29 | · | · | · | **100** | · | · | · | · | · |
| 30 | · | **4** | · | · | · | · | · | · | · |
| 36 | **5** | · | · | · | · | · | · | · | · |
| 37 | · | **1** | · | · | · | · | · | · | **1** |
| 38 | · | · | · | · | · | · | · | **1** | · |
| 39 | · | **1** | · | · | · | · | · | · | **1** |
| 40 | · | · | · | · | · | · | **0.1** | · | · |
| 48 | · | · | · | · | · | · | · | · | **1** |
| 50 | · | · | · | · | · | · | · | · | **1** |
| 52 | · | · | · | · | · | · | · | **1** | · |
| 53 | · | **1** | · | · | · | · | · | · | · |
| 54 | · | · | · | · | · | · | · | · | **1** |
| 55 | · | · | · | · | **0** | · | · | **1** | · |
| 56 | · | · | · | · | · | · | · | **1** | · |
| 57 | · | · | · | · | **0** | · | · | **1** | · |
| 58 | · | · | **10000** | · | · | · | · | · | **1** |
| 59 | · | · | · | · | · | · | · | **1** | **1** |
| 60 | · | **4** | · | **1000** | · | · | · | · | · |
| 62 | · | **1** | · | · | **0** | · | · | · | · |
| 66 | · | · | · | · | · | · | · | **1** | · |
| 67 | · | · | · | · | · | · | · | **1** | · |
| 68 | · | · | · | · | · | · | · | **1** | · |
| 69 | · | · | · | · | · | · | · | **1** | · |
| 70 | **0** | · | · | · | · | · | · | **1** | · |
| 71 | · | · | · | **10** | · | · | · | **1** | · |
| 81 | · | · | · | **300** | · | · | · | · | · |
| 82 | · | · | **10000** | · | · | · | · | · | **1** |
| 84 | · | · | · | · | · | · | · | · | **0.1** |
| 85 | · | **1** | · | · | **0** | · | · | · | · |
`HitRatio` is **1.0 for every one of the 24 records that default it** — the
whole `…Ratio` family that blocked Route B is simply "no penalty".
`wep_82`'s defaulted field is `PlayerLaserPower` = **12000** (its `Power`,
12000, is on disc).
## Cross-checks
- **Against the independent screenshot route.** The Arsenal DATA SHEET
([weapon-datasheet-runtime.md](../weapon-datasheet-runtime.md)) established
`Max. Lock Ons == TriggerShotCount`, and read **4** for `wep_05` and `wep_60`
off the panel. The memory read gives **4** for both — two unrelated methods,
same numbers.
- **Against the in-flight HUD.** `NOSE BM 06000` / `MAIN MPM 00300` matches
`wep_01` `LoadingCount = 6000` and `wep_02` `= 300` at `+0x28`.
- **Stability.** All 252 objects are **byte-identical** between the READY ROOM
and the ARSENAL, so these are load-time definition data, not transient state.
- **Self-consistency.** 121 records agree on `LoadingCount`'s offset across 33
distinct values with zero contradictions; `Power`, 101 records / 37 distinct
values. A wrong offset cannot do that.
## Incidental findings
- **A duplicate, conflicting disc record.** Two `.tbl` entries declare
`Shell_TCAF_Ship_AAGun`; one sets `Power = 60.0`, the other omits it. The
runtime object holds **5**, i.e. the *omitting* declaration won. Any
reimplementation loading both will silently pick one — this says which.
- **Not every disc record is instantiated.** 131 disc records produced 126
objects; the 5 with no runtime object are context variants that this save
never loads — `Weapon_TCAF_DeltaSaber_NoseGun_Ttrl` / `_Laser_Ttrl`
(tutorial), `_NoseGun_None`, `Weapon_TCAF_Ship_AAGun_EX5`,
`Weapon_ADAN_Attacker_S_GunTurret`. Running the tutorial should instantiate
the `_Ttrl` pair. No runtime object lacked a disc record.
- Defaulted `Power` values vary per weapon (1, 10, 100, 300, 500, 1000), so they
are **not** one constructor constant. Where they come from — the IDXD's
undecoded binary node/index region, or per-type code defaults — is **not
determined here** (`NEEDS-HUMAN` / follow-up). Either way the runtime values
above are ground truth, and they are now a decoding oracle for that region.
## Reproduce
```bash
export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy
run-canary --audio --apu=sdl --log_mask=13 --logged_profile_slot_0_xuid=E0300000EFBEA3D4 &
tools/re-capture/skip_intro.sh # -> title
# LOAD GAME -> slot 01 -> YES -> READY ROOM (weapon tables load with the save)
cargo run --release -p sylpheed-formats --example idxd_tokens -- \
<disc>/dat/GP_MAIN_GAME_E.pak > /tmp/wep_tokens.txt
python3 tools/re-capture/weapon_runtime.py /tmp/wep_tokens.txt # report
python3 tools/re-capture/weapon_runtime.py /tmp/wep_tokens.txt --csv # full table
```
## What this unlocks
`gmem.py` + "scan for the vtable, solve the layout against disc ground truth" is
**not weapon-specific**. The same three steps apply to any IDXD-backed
definition whose fields the disc defaults — craft/`UNIT` stats, which the Hangar
UI exposes only as a `Gross Weight` class and which
[the menu route could not reach at all](../weapon-datasheet-runtime.md), are the
obvious next target.

View File

@@ -0,0 +1,232 @@
# Weapon DATA SHEET — runtime capture (Route B)
**Status:** 🟡 first dynamic capture, 2026-07-28. The Arsenal's *Gallery Mode* panel is a
direct runtime readout of the IDXD weapon record, which makes it an oracle for the fields
the disc leaves **defaulted**. Two field mappings are ✅ `CONFIRMED`; two defaulted values
are recovered at 🟡 `PROBABLE`. Captured from the retail game under Xenia Canary
(software Vulkan, headless) — see [the container recipe](#how-this-was-captured).
## Problem
`sylpheed-formats::game_data` reads the combat tables out of `dat/GP_MAIN_GAME_E.pak`, but
**a field left at its default value carries no value on disc** — the key is present in the
IDXD string pool with no value token in front of it. Those defaults live in title code, so
a static read can only ever say "not set", never *what* the game uses. That is a real hole
for the reimplementation: e.g. `Weapon_DSaber_P_wep_01_Beam` — the Delta Saber's starting
beam gun — has no `TriggerShotCount` and no `Power` on disc.
Ruled out first: the defaults are **not** hiding in another pak. `hidden/DefTables.pak`
contains no `WEAPON`-schema (`0x6ab4825a`) objects at all, and the other `GP_MAIN_GAME_*`
paks are localized duplicates of the English one.
The two scratch analyses that produced the shopping list live next to the crate:
`crates/sylpheed-formats/examples/defaulted_fields.rs` (per-schema: which keys are declared
but defaulted, and by whom) and `examples/default_owners.rs` (per-key: every owner, valued
or `<DEFAULT>`).
> Caveat on those tools: they classify a token as a *value* only if it is numeric or
> non-identifier-shaped. **Boolean/enum-valued fields therefore read as `<DEFAULT>`
> spuriously** (`Yes`, `Homing`, `Single`, `Burst` are identifier-shaped). Every finding
> below concerns numeric fields, where the classification is sound.
## Finding — the DATA SHEET reads the record
In **ARSENAL → (weapon type) → Y (Gallery Mode)** each entry shows a `DATA SHEET`, and it
is shown for weapons that have **not** been developed yet — only fully hidden (dashed)
entries are withheld. The same panel appears in **HANGAR → (hard point)**, with an extra
`Weapon Type` row.
| DATA SHEET row | IDXD field | Confidence | Evidence |
|---|---|---|---|
| `Ammo Capacity` | `LoadingCount` | ✅ CONFIRMED | every one of the 10 identified entries lands on a `LoadingCount` that exists in its own weapon-type tab — and see the circularity note below |
| `Max. Lock Ons` | `TriggerShotCount` | ✅ CONFIRMED | FALCON 9AM = 12 vs `wep_02`'s 12; BUZZARD 10AM = 22 vs `wep_04`'s 22 (two distinctive values, two independent records) |
| `Hard Point` | mount slot | ✅ CONFIRMED | matches the HANGAR slot the weapon is mountable/equipped on (`STILETTO BG I` → NOSE, `FALCON 9AM` → MAIN WEAPON1) |
| `Range Class` | bucket of `MaximumRange` | 🟡 PROBABLE | monotone in the on-disc metres, see the bracket table below |
| `Damage Class` | bucket of `Power` | 🟡 PROBABLE | monotone in the on-disc power, see below |
| `Weight Class` | bucket of the hangar-table `Weight` | ❔ HYPOTHESIS | only one clean pair so far (`wep_33`, Weight 0.3 → "Light") |
| `Speed Class` | ❔ | ❔ | present only on missiles (MPM = A, ASM = B); no on-disc pairing established |
| `Sight Homing`, `Lock on Overlap` | ❔ (`Available` / ``) | ❔ | the plausible on-disc partners (`Homing`, `OverlapLockon`) are identifier-valued and not yet decoded; **NEEDS-HUMAN** |
### Why the `Ammo Capacity` evidence is not circular
Each UI entry was **identified** by matching its `Ammo Capacity` against the records in
that weapon-type tab, so "the ammo matches" cannot on its own prove the mapping. What does:
1. **The match is forced and unique.** For 10 of the 11 entries, exactly one record in that
tab carries that number (`600` occurs three times overall — `wep_04`, `wep_24`, `wep_55`
— but in three different tabs: MPM, BEAM, B/R). An unrelated quantity would not land on
a valid, tab-unique `LoadingCount` eleven times running.
2. **A second, independent field then agrees.** FALCON 9AM and BUZZARD 10AM were pinned by
ammo alone, and their `Max. Lock Ons` (12, 22) then matched those same records'
`TriggerShotCount` (12, 22) — values that play no part in the identification. A wrong
identification would have to be wrong twice, consistently.
3. **One entry is identified without ammo at all.** STILETTO BG I is described in-game as
"the initially mounted Delta Saber beam gun" and is the weapon the HANGAR shows equipped
on the nose; `wep_01_Beam` is the corresponding record, and its `LoadingCount` 6000 is
what the panel shows.
The weakest identification is LIGHT MACHINE GUN MG I: three guns share `LoadingCount = 3000`
(`wep_09`, `wep_33`, `wep_83`). `wep_33` is picked on `Weight Class = Light` (it has by far
the smallest `Mass`, 1.5 vs 3.6 / 33.0) — 🟡 PROBABLE, not certain.
## Finding — recovered defaults
| Weapon | UI name | Field | On disc | **Runtime** | Conf. |
|---|---|---|---|---|---|
| `Weapon_DSaber_P_wep_05_ASMissile` | TERRIER SMH | `TriggerShotCount` | *(defaulted)* | **4** | 🟡 |
| `Weapon_DSaber_P_wep_60_ASMissile` | HOUND SMH | `TriggerShotCount` | *(defaulted)* | **4** | 🟡 |
Both defaulted weapons read **4**, which is consistent with a single title-code default of
`TriggerShotCount = 4` rather than two per-weapon constants — but two samples cannot tell
those apart. ❔ HYPOTHESIS: *the title-code default for `TriggerShotCount` is 4.* It would
be confirmed by a third weapon that defaults the field and also reads 4 (candidates that
were still locked in this save: `wep_27`, `wep_30`, and the seven Beams), or refuted by one
that reads anything else.
`Power` and `MaximumRange` defaults are **not** exactly recoverable from this panel — it
shows only the letter bucket. They are bracketed instead (below).
## Captured rows
All values are from one session on save slot 01 (Stage 02, "At Standby", 5 % clear, 4101 P);
`✓` marks a value that matches the on-disc record exactly.
| UI name | Record | Rng | Dmg | Spd | Weight | Ammo | Lock | Homing | Overlap | Hard point |
|---|---|---|---|---|---|---|---|---|---|---|
| LIGHT MACHINE GUN MG I | `wep_33_Gun` 🟡 | D | E | | Light | 3000 ✓ | | | | NOSE WEAPON (Nose) |
| BROAD SWORD SG I | *see below* | E | D | | Light | 200 | | | | NOSE WEAPON (Nose) |
| STILETTO BG I | `wep_01_Beam` | E | E | | Light | 6000 ✓ | | | | NOSE WEAPON (Nose) |
| DAGGER BG2 | `wep_37_Beam` | D | E | | Light | 4000 ✓ | | | | NOSE WEAPON (Nose) |
| PILUM BP | `wep_24_Beam` | B | D | | Light | 600 ✓ | | | | MAIN WEAPON1 (Fore) |
| FALCON 9AM | `wep_02_Missile` | D | D | A | Heavy | 300 ✓ | 12 ✓ | Available | | MAIN WEAPON1 (Fore) |
| BUZZARD 10AM | `wep_04_Missile` | C | D | A | Heavy | 600 ✓ | 22 ✓ | | Available | MAIN WEAPON1 (Fore) |
| DART 23 ROCKET | `wep_55_Rocket` | C | D | | Heavy | 600 ✓ | | | | MAIN WEAPON1 (Fore) |
| TERRIER SMH | `wep_05_ASMissile` | D | C | B | Heavy | 45 ✓ | **4** | Available | Available | MAIN WEAPON2 (Rear) |
| HOUND SMH | `wep_60_ASMissile` | B | C | B | Medium | 18 ✓ | **4** | Available | Available | MAIN WEAPON2 (Rear) |
| TOMAHAWK ALPHA RAIL GUN | `wep_03_Cannon` | C | C | | Medium | 75 ✓ | | | | MAIN WEAPON3 (Lower) |
**BROAD SWORD SG I is unidentified — NEEDS-HUMAN.** `Ammo Capacity 200` narrows it to
`wep_38` / `wep_41` / `wep_42_Shotgun` (all `LoadingCount = 200`); "SG" and the GUN tab fit
a shotgun. `wep_42` is excluded by range (2500 m would not share class E with `wep_38`/
`wep_41`'s 3000 m *if* the class is a pure range bucket), leaving `wep_38` vs `wep_41`,
which the panel cannot separate. Its `Damage Class D` also does not fit the `Power` bracket
below (all three shotguns are `Power ≤ 16`, i.e. bucket E), so either the identification or
the "Damage Class = bucket of `Power`" model is wrong for shotguns.
### Letter-class brackets
Sorting the identified rows by their on-disc numbers gives monotone, non-overlapping bands:
```
Range Class E: 3000 (wep_01)
D: 3500 · 4000 · 4000 · 4000 (wep_33, wep_37, wep_02, wep_05)
C: 4500 · 5000 · 5000 (wep_55, wep_04, wep_03)
B: 6500 · 6500 (wep_24, wep_60)
Damage Class E: 10 · 14 · 16 (wep_33, wep_01, wep_37)
D: 70 · 75 · 100 (wep_24, wep_04, wep_55)
C: 200 · 400 (wep_03, wep_05)
```
Two consequences for the reimplementation:
- The **thresholds are not pinned** — only bracketed (e.g. the D/C range boundary lies in
(4000, 4500]). More weapons, or a static read of the title-code table, would pin them.
- They **bracket the defaulted numbers**: `wep_02_Missile`'s defaulted `Power` sits in the
D band (≈ 17…150 by the observed edges) and `wep_60_ASMissile`'s in the C band
(≈ 150…500). 🟡 PROBABLE, and only as good as the bucket model.
## How this was captured
Container recipe (`sylph-container/mission.md`), with two corrections worth keeping:
- **Skip the intro movie with A.** The brief warns it crashes; it does not. Skipping cuts
boot-to-main-menu from ~13 min to **~1 min** under lavapipe. (Thanks: user tip.)
- **The title screen falls back to the attract loop within a few seconds**, so a
screenshot→look→tap cycle always misses it. Poll the framebuffer and tap in the same
process. Both are automated in `scratchpad/skip_intro.sh` (movie detected by frame-to-
frame RMSE; title by the green Ⓐ glyph at pixel 625,618).
- Under lavapipe the game polls input at its own low frame rate: **a 60 ms d-pad tap is
dropped roughly half the time**; 200 ms is reliable and 300 ms starts to auto-repeat.
- The Hangar hard-point weapon carousel is cycled with d-pad **down**, not left/right.
Path: title → A → LOAD GAME → slot 01 → *Load game?* **YES** → READY ROOM → ARSENAL →
type tab (LB/RB) → **Y** for the DATA SHEET → d-pad down through the list.
## Open / next
- Only **9 weapons of ~61** are revealed at 5 % completion, and none of the four whose
`LoadingCount` is defaulted (`wep_11`, `wep_28`, `wep_36`, `wep_70`) is among them.
Progressing the save (or a later save) is what unlocks the rest — the panel itself
already shows undeveloped weapons, so no points need to be spent.
- **Craft (`UNIT`-schema) defaults are not reachable this way.** The Hangar exposes exactly
one craft-level runtime number, `Gross Weight` (a class, "Light"). The defaulted craft
fields (`ShieldRatio`, `BarrelRoll_Count*`, `HoldPosition_*Ratio`, `Slalom_TurnCount_Max`,
`UsingChaffRatio`, …) are AI/flight-model constants with no UI surface; they would need
in-flight behavioural measurement or a guest-memory read, not a menu screenshot.
- The `Sight Homing` / `Lock on Overlap` on-disc partners are still unidentified.
Screenshots for every row above: `/sylph-home/re/caps/` in the container.
Evidence PNGs are committed under [`captures/weapon-datasheet/`](captures/weapon-datasheet/)
(64-colour quantized for size; the numbers stay legible).
---
# Addendum — the in-flight HUD (2026-07-28)
Reached via **TUTORIAL → BASIC CONTROLS / HEADS-UP DISPLAY** from the title menu (no
save needed). Evidence: [`captures/hud-runtime/`](captures/hud-runtime/).
## The HUD corroborates the ammo mapping, independently
The Delta Saber's HUD prints its two equipped weapons as `NOSE <TYPE> <AMMO>` and
`MAIN <TYPE> <AMMO>`. With the loadout known from the HANGAR (nose = STILETTO BG I, main =
FALCON 9AM) the HUD reads **`NOSE BM 06000`** and **`MAIN MPM 00300`** — exactly
`wep_01_Beam`'s `LoadingCount = 6000` and `wep_02_Missile`'s `300`.
This matters because it is **not** the Arsenal panel: the weapons were identified from the
HANGAR loadout, and the numbers come from a different renderer in a different game mode. It
is a genuinely independent confirmation of `Ammo Capacity == LoadingCount`.
The type tags (`BM`, `MPM`) are the same short codes as the Arsenal type tabs.
## HUD element inventory
| HUD element | Backing data | Conf. |
|---|---|---|
| `NOSE` / `MAIN` + type tag + 5-digit ammo | equipped weapon, `LoadingCount` | ✅ |
| `HEAT` / `L.HEAT` bar under each weapon | the `Heating` / `Cooling` pair | 🟡 |
| Speed readout + throttle scale (`0`, `100`, `A/B`) | craft velocity | ✅ |
| `SHIELD` and `ARMOR` bars (two separate pools) | craft `HP` + a shield pool | 🟡 |
| Target reticle: name / type / distance + ring **Armor Gauge** | target unit record | 🟡 |
| `RANGE` gauge with `MAIN` and `NOSE` tick markers | each equipped weapon's `MaximumRange`, plotted against target distance | 🟡 |
| `YOU KILLED: WARSHIPS nnnn` + a second counter | score / `ScorePoint` | ❔ |
The `RANGE` gauge is the interesting one for future work: it draws a **per-weapon marker on
a distance scale**, so a weapon whose `MaximumRange` is defaulted on disc (`wep_25`,
`wep_58`, `wep_82`) would have its value *drawn* rather than bucketed into a letter — if the
scale can be calibrated against two weapons with known ranges, that recovers a real number
where the Arsenal panel only gives a class.
## Craft velocity
Full afterburner (RT) peaked at **1193** against `UN_f001_TCAF_DeltaSaber_T`'s on-disc
`MaximumVelocity = 1200`. 🟡 PROBABLE — one observation, and the readout may have been
still climbing. Cruise sat at 350, which is *not* the record's `CruisingVelocity` (700), so
the number is the current throttle setting, not a named constant; don't read more into it.
## Notes for the next session
- The tutorials need **no save game** and are reachable in ~1 min from a cold boot, which
makes them the cheapest way back into a live flight scene.
- `tools/re-capture/autopilot.py` chases the yellow off-screen waypoint arrow (colour +
shape, `-sample` not `-resize`, ~0.65 s per detection). It **finds the arrow reliably but
oscillates** — the proportional gain is too high for the craft's turn rate. It needs a
damping/derivative term before it can actually fly a waypoint.
- The HEADS-UP DISPLAY tutorial reaches a scripted targeting segment where the ship stops
moving (target distance pinned) and the on-screen controller highlights **A**; tapping and
holding A did not advance it. **NEEDS-HUMAN**: what input that segment wants.
- Canary is unstable here: it died twice mid-session (once loading BEAM `Resource3D`, once
hanging on tutorial teardown) with no crash dump. Re-launch is cheap; just don't assume a
long session survives.

View File

@@ -0,0 +1,35 @@
# Runtime-capture harness (sylph-re container)
Screenshot-driven scripts for reading the running retail game's menus under Xenia
Canary + lavapipe, headless. They assume the container helpers `screenshot`,
`vgamepad`, `pad` are on `$PATH` and `HOME=/sylph-home/re`.
| Script | What it does |
|---|---|
| `skip_intro.sh` | Boot → main menu, unattended. Taps A only while the intro movie is actually playing (frame-to-frame RMSE), then once at the `PRESS Ⓐ BUTTON` title. Static logo screens are left alone, so a stray tap can never land on NEW GAME. |
| `wait_title.sh` | Older variant: wait for the title (green Ⓐ glyph at px 625,618) and tap A. Superseded by `skip_intro.sh`. |
| `step.sh` | One Arsenal navigation step (`down`/`up`/`next`/`prev`/`none`) + a compact capture: weapon list stacked over the `DATA SHEET`. |
| `sweep.sh` | Walk a whole weapon-type list, capturing **only** rows that show a `DATA SHEET` — locked rows (a "Conditions to Develop" panel) are detected by the brightness of the `Range Class` label box and skipped. |
| `type.sh` | Change weapon-type tab N times (RB) and report the header strip. |
| `hp.sh` / `cyc.sh` | Hangar hard-point carousel: `cyc.sh` steps it (d-pad **down**, not left/right) and captures the Name + `DATA SHEET`. |
**Input timing under lavapipe:** the game polls input at its own low frame rate, so a
60 ms d-pad tap is dropped roughly half the time. 200 ms is reliable; 300 ms starts to
auto-repeat (two rows per press).
Findings produced with these: [`docs/re/weapon-datasheet-runtime.md`](../../docs/re/weapon-datasheet-runtime.md).
## Guest-memory tools (no screenshots)
| Script | What it does |
|---|---|
| `gmem.py` | Read the live guest address space out of `/dev/shm/xenia_memory_*` (Xenia's backing file), addressed by guest VA. `find` / `read` / `words`. |
| `weapon_runtime.py` | Solve the `Weapon`/`Shell` struct layouts against the disc records and read the fields the disc defaults. |
| `unit_discover.py` | Find *which* runtime class carries a set of ID strings, assuming no vtable: tallies the word at `pointer_site - k` across distinct IDs. |
| `unit_runtime.py` | Same solver for the `unit\UN_*.tbl` definition objects (vtable `0x820af844`). Unions several snapshots — unit definitions are per-stage. |
| `schema_order.py` | Merge a sub-record's field-declaration order across all tables (topological sort); the layout check that pins fields no table ever values. |
| `order_check.py` | Test `offset = base + 4*index` for one table's sub-record against solver output. |
| `grab_tutorial.sh` | Cold-boot Canary, walk to the Nth TUTORIAL entry, wait for the stage load, snapshot guest RAM. One emulator per capture — backing out of a loaded mission wedges it. |
Snapshot first — `cp --sparse=always /dev/shm/xenia_memory_* snap.bin` (~2 s) — and
point `$GMEM_FILE` at the copy; the running emulator pegs every core under lavapipe.

View File

@@ -0,0 +1,95 @@
#!/usr/bin/env python3
"""Fly the Delta Saber toward the tutorial waypoint by chasing the yellow
off-screen direction arrow.
Detection: no PIL/numpy in the box, so the frame comes from `convert ... txt:-`.
Two things make it fast AND correct:
* `-sample` (point sampling, not `-resize`/`-scale` averaging) — a 1-px-thin
glyph keeps its true colour, so the exact colour predicate still fires while
the dump shrinks ~9x (3.5s -> 0.65s).
* shape, not just colour — the instruction-frame bars and the throttle marker
are the same dim yellow, so the arrow is the largest blob that is roughly as
tall as it is wide, with the fixed throttle marker blacklisted by position.
A ~1.1 s control loop is fast enough to converge; the earlier ~6 s loop was not.
"""
import re, subprocess, sys, time
X0, Y0, W, H = 80, 60, 820, 640 # flight view (the arrow also rides the bottom edge)
K = 3.03 # 1/0.33 sample factor
CX, CY = 640, 405 # crosshair
THROTTLE = (382, 456) # fixed yellow decoy on the speed gauge
PX = re.compile(r"^(\d+),(\d+):.*?srgb\((\d+),(\d+),(\d+)\)")
def pad(*a):
subprocess.run(["/opt/sylph/vgamepad.py", *a], check=False)
def arrow(png="/tmp/ap.png"):
subprocess.run(["/opt/sylph/screenshot.sh", png], check=False,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
out = subprocess.run(["convert", png, "-crop", f"{W}x{H}+{X0}+{Y0}", "+repage",
"-sample", "33%", "txt:-"], capture_output=True, text=True).stdout
pts = set()
for l in out.splitlines()[1:]:
m = PX.match(l)
if not m:
continue
x, y, r, g, b = (int(v) for v in m.groups())
if r >= 60 and g >= 48 and b <= 0.35 * g and (r - g) <= 0.45 * r:
pts.add((x, y))
seen, best = set(), None
for p in pts:
if p in seen:
continue
st, c = [p], []
seen.add(p)
while st:
x, y = st.pop(); c.append((x, y))
for dx in (-1, 0, 1):
for dy in (-1, 0, 1):
q = (x + dx, y + dy)
if q in pts and q not in seen:
seen.add(q); st.append(q)
xs = [q[0] for q in c]; ys = [q[1] for q in c]
cx, cy = X0 + sum(xs) / len(c) * K, Y0 + sum(ys) / len(c) * K
if abs(cx - THROTTLE[0]) < 30 and abs(cy - THROTTLE[1]) < 30:
continue
if len(c) < 8:
continue
if best is None or len(c) > best[2]:
best = (cx, cy, len(c))
return best
def main():
steps = int(sys.argv[1]) if len(sys.argv) > 1 else 25
centred = 0
for i in range(steps):
a = arrow()
if a is None:
print(f"{i:2d}: no arrow -> boost (target ahead)")
pad("trig", "RT", "0.55"); time.sleep(1.2); pad("trig", "RT", "0")
centred += 1
if centred >= 6:
print(" arrival likely — stopping"); return 0
continue
x, y, n = a
dx, dy = x - CX, y - CY
if abs(dx) < 70 and abs(dy) < 70:
centred += 1
print(f"{i:2d}: arrow ({x:.0f},{y:.0f}) centred -> boost")
pad("trig", "RT", "0.55"); time.sleep(1.2); pad("trig", "RT", "0")
continue
centred = 0
lx = max(-1.0, min(1.0, dx / 200))
ly = max(-1.0, min(1.0, dy / 160))
print(f"{i:2d}: arrow ({x:.0f},{y:.0f}) n={n} -> LX {lx:+.2f} LY {ly:+.2f}")
pad("axis", "LX", f"{lx:.2f}"); pad("axis", "LY", f"{ly:.2f}")
time.sleep(0.45)
pad("axis", "LX", "0"); pad("axis", "LY", "0")
print("steps exhausted")
return 1
sys.exit(main())

View File

@@ -0,0 +1,212 @@
#!/usr/bin/env python3
"""Memory-driven autopilot: fly and fight from the game's own world state.
The earlier screen-scraping autopilot oscillated because it only ever saw a
2-D arrow on the HUD and had no rate feedback. This one reads the actual
transform of every entity out of guest RAM, so it can do proper PD control:
the derivative term uses the craft's real body angular velocity, recovered from
two consecutive rotation matrices, not a differenced pixel position.
world state <- /dev/shm/xenia_memory_* (see gworld.py)
control -> the vgamepad FIFO, written directly at loop rate
Offsets come from flight_analyze.py and are passed in / stored in offsets.json;
nothing here hard-codes a value that was not derived from a capture.
"""
import json
import math
import os
import struct
import sys
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gworld # noqa: E402
from flight_probe import Pad # noqa: E402
# --------------------------------------------------------------- 3-D helpers
def dot(a, b):
return a[0] * b[0] + a[1] * b[1] + a[2] * b[2]
def sub(a, b):
return (a[0] - b[0], a[1] - b[1], a[2] - b[2])
def norm(a):
return math.sqrt(dot(a, a))
def clamp(v, lo=-1.0, hi=1.0):
return max(lo, min(hi, v))
def body_rates(R_prev, R, dt):
"""Angular velocity in body axes from two rotation matrices.
R rows are the craft's axes in world space, so R_prev @ R^T is the
incremental rotation; its skew part is the rotation vector.
"""
if dt <= 0:
return (0.0, 0.0, 0.0)
# M = R_prev * R^T (3x3, row-major tuples)
M = [[sum(R_prev[i * 3 + k] * R[j * 3 + k] for k in range(3)) for j in range(3)]
for i in range(3)]
wx = (M[2][1] - M[1][2]) / 2.0
wy = (M[0][2] - M[2][0]) / 2.0
wz = (M[1][0] - M[0][1]) / 2.0
return (wx / dt, wy / dt, wz / dt)
# ------------------------------------------------------------------- reader
class Flight:
def __init__(self, cfg):
self.cfg = cfg
self.w = gworld.World()
self.pos_off = cfg["pos"]
self.rot_off = cfg["rot"]
self.hp_off = cfg.get("hp")
self.player_name = cfg.get("player", "Player")
self.entities = []
self.last_scan = 0.0
def rescan(self):
self.entities = self.w.refresh()
self.last_scan = time.time()
def state(self, off):
buf = self.w.read_off(off, gworld.WINDOW)
if len(buf) < gworld.WINDOW:
return None
pos = struct.unpack_from(">3f", buf, self.pos_off)
rot = struct.unpack_from(">9f", buf, self.rot_off)
if not all(map(math.isfinite, pos + rot)):
return None
hp = struct.unpack_from(">f", buf, self.hp_off)[0] if self.hp_off else None
return {"pos": pos, "rot": rot, "hp": hp}
def player(self):
for va, off, nm in self.entities:
if self.player_name in nm:
s = self.state(off)
if s:
s["name"] = nm
return s
return None
def hostiles(self):
"""ADAN units are the enemy; the disc IDs encode the faction.
`UN_e*` / `UN_be*` = ADAN, `UN_f*` / `UN_bf*` = TCAF (ours).
"""
out = []
for va, off, nm in self.entities:
base = nm[3:]
if not (base.startswith("e") or base.startswith("be")):
continue
s = self.state(off)
if s:
s["name"] = nm
s["off"] = off
out.append(s)
return out
# ---------------------------------------------------------------- controller
class Autopilot:
KP_YAW, KD_YAW = 1.6, 0.35
KP_PITCH, KD_PITCH = 1.6, 0.35
FIRE_CONE = math.radians(6)
FIRE_RANGE = 4000.0
def __init__(self, flight, pad, log=sys.stdout):
self.f = flight
self.pad = pad
self.log = log
self.prev_rot = None
self.prev_t = None
self.target = None
self.firing = False
def pick_target(self, me):
hs = self.f.hostiles()
if not hs:
return None
# nearest, mildly preferring what is already ahead of us
def score(h):
v = sub(h["pos"], me["pos"])
d = norm(v) or 1.0
fwd = me["rot"][6:9]
ahead = dot(v, fwd) / d
return d * (1.0 if ahead > 0 else 2.0)
return min(hs, key=score)
def step(self, dt):
me = self.f.player()
if not me:
return "no-player"
R = me["rot"]
w = body_rates(self.prev_rot, R, dt) if self.prev_rot else (0, 0, 0)
self.prev_rot = R
tgt = self.pick_target(me)
if not tgt:
self.pad.axis("LX", 0.0)
self.pad.axis("LY", 0.0)
self.pad.trig("RT", 0.6)
return "no-target"
v = sub(tgt["pos"], me["pos"])
dist = norm(v) or 1.0
# into body axes: rows are right / up / forward
lx = dot(R[0:3], v)
ly = dot(R[3:6], v)
lz = dot(R[6:9], v)
yaw_err = math.atan2(lx, lz if lz > 1e-3 else 1e-3)
pitch_err = math.atan2(ly, lz if lz > 1e-3 else 1e-3)
if lz < 0: # behind us: turn the short way, hard
yaw_err = math.copysign(math.pi / 2, lx if lx else 1.0)
stick_x = clamp(self.KP_YAW * yaw_err - self.KD_YAW * w[1])
stick_y = clamp(-(self.KP_PITCH * pitch_err - self.KD_PITCH * w[0]))
self.pad.axis("LX", stick_x)
self.pad.axis("LY", stick_y)
self.pad.trig("RT", 1.0 if dist > 1500 else 0.3)
aligned = abs(yaw_err) < self.FIRE_CONE and abs(pitch_err) < self.FIRE_CONE
want_fire = aligned and dist < self.FIRE_RANGE
if want_fire != self.firing:
(self.pad.press if want_fire else self.pad.release)(self.f.cfg["fire_btn"])
self.firing = want_fire
return (f"tgt={tgt['name'][3:20]:<18} d={dist:8.0f} yaw={math.degrees(yaw_err):+6.1f} "
f"pit={math.degrees(pitch_err):+6.1f} stick=({stick_x:+.2f},{stick_y:+.2f}) "
f"fire={int(self.firing)} hp={me['hp']}")
def run(self, seconds, hz=15.0):
self.f.rescan()
t0 = time.time()
last = t0
while time.time() - t0 < seconds:
t = time.time()
if t - self.f.last_scan > 3.0:
self.f.rescan()
msg = self.step(t - last)
last = t
print(f"[{t-t0:6.1f}] {msg}", file=self.log, flush=True)
time.sleep(max(0, 1.0 / hz - (time.time() - t)))
self.pad.reset()
def main():
cfg = json.load(open(sys.argv[1]))
secs = float(sys.argv[2]) if len(sys.argv) > 2 else 60.0
ap = Autopilot(Flight(cfg), Pad())
ap.run(secs)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,207 @@
#!/usr/bin/env python3
"""Fly and fight from guest memory.
World state comes from entities2.py's typing rule: a live entity's position
triple sits at a fixed offset before its definition pointer, so one scan of the
entity heap yields every craft in the scene *with its unit type* — which is what
separates enemies from wingmen, capital ships and the thousands of moving
particles.
Control is PD on the aiming error, with the derivative taken from the craft's
own body angular velocity (recovered from two consecutive orientation matrices)
rather than from the differenced error — that is what the old screen-scraping
autopilot lacked, and why it oscillated.
Usage: autopilot3.py <config.json> [seconds] [--dry]
"""
import json
import math
import os
import struct
import sys
import time
from collections import Counter
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
import gworld # noqa: E402
import entities2 # noqa: E402
from flight_probe import Pad # noqa: E402
HOSTILE = ("e", "be") # UN_e* / UN_be* are ADAN; UN_f*/UN_bf* are ours
def unit_faction(nm):
base = nm[3:]
return "ADAN" if base.startswith("be") or base.startswith("e") else "TCAF"
class World:
def __init__(self, cfg):
self.cfg = cfg
self.w = gworld.World()
self.fd, self.size = self.w.fd, self.w.size
self.defs = entities2.definitions(self.w)
self.delta = cfg["def_delta"]
self.rot_delta = cfg["rot_delta"]
self.rot_stride = cfg.get("rot_stride", 12)
self.fwd_row = cfg["fwd_row"]
self.fwd_sign = cfg["fwd_sign"]
self.ents = []
def rescan(self):
movers = entities2.moving(self.fd, self.size, dt=0.35,
va_range=(self.cfg["va_lo"], self.cfg["va_hi"]))
ents = entities2.typed(self.fd, self.defs, movers, self.delta)
uniq = {}
for off, nm, pos, sp in ents:
uniq.setdefault(off, (off, nm, pos, sp))
self.ents = list(uniq.values())
return self.ents
def pos(self, off):
b = os.pread(self.fd, 12, off)
return np.array(struct.unpack(">3f", b)) if len(b) == 12 else None
def rot(self, off):
n = self.rot_stride * 2 + 12
b = os.pread(self.fd, n, off + self.rot_delta)
if len(b) < n:
return None
M = np.array([struct.unpack_from(">3f", b, self.rot_stride * r) for r in range(3)])
if not np.all(np.isfinite(M)):
return None
if np.max(np.abs(M @ M.T - np.eye(3))) > 5e-3:
return None
return M
def clamp(v, lo=-1.0, hi=1.0):
return max(lo, min(hi, v))
def body_rate(Mprev, M, dt):
if Mprev is None or M is None or dt <= 0:
return np.zeros(3)
D = Mprev @ M.T
w = np.array([D[2, 1] - D[1, 2], D[0, 2] - D[2, 0], D[1, 0] - D[0, 1]]) / 2.0
return w / dt
class Autopilot:
KP, KD = 2.2, 0.45
FIRE_CONE = math.radians(10)
FIRE_RANGE = 6000.0
def __init__(self, world, pad, dry=False):
self.W = world
self.pad = pad
self.dry = dry
self.prevM = None
self.firing = False
def me(self):
for off, nm, pos, sp in self.W.ents:
if "Player" in nm:
return off, nm
return None, None
def step(self, dt):
off, nm = self.me()
if off is None:
return "no-player"
p = self.W.pos(off)
M = self.W.rot(off)
if p is None or M is None:
return "no-state"
fwd = M[self.W.fwd_row] * self.W.fwd_sign
rows = [M[i] for i in range(3)]
right = rows[(self.W.fwd_row + 1) % 3]
up = np.cross(fwd, right)
w = body_rate(self.prevM, M, dt)
self.prevM = M
# nearest hostile, preferring what is already in front
best, bestscore = None, 1e18
for eoff, enm, epos, esp in self.W.ents:
if unit_faction(enm) != "ADAN":
continue
q = self.W.pos(eoff)
if q is None:
continue
v = q - p
d = float(np.linalg.norm(v))
if d < 1e-3:
continue
# Prefer targets we can actually bring the nose onto. Nearest-first
# picks whatever is closest even at 90 deg off the nose, and closing
# on an off-boresight target only raises the bearing rate -- which is
# exactly the lag that kept the first run outside its firing cone.
ahead = float(v @ fwd) / d
ang = math.acos(max(-1.0, min(1.0, ahead)))
score = d * (1.0 + 3.0 * (ang / math.pi) ** 2)
if score < bestscore:
best, bestscore = (eoff, enm, q, v, d), score
if best is None:
if not self.dry:
self.pad.axis("LX", 0.0)
self.pad.axis("LY", 0.0)
return "no-hostiles"
eoff, enm, q, v, d = best
lx = float(v @ right)
ly = float(v @ up)
lz = float(v @ fwd)
yaw = math.atan2(lx, lz if abs(lz) > 1e-3 else 1e-3)
pitch = math.atan2(ly, lz if abs(lz) > 1e-3 else 1e-3)
if lz < 0:
yaw = math.copysign(math.pi / 2, lx if lx else 1.0)
sx = clamp(self.KP * yaw - self.KD * float(w @ up))
sy = clamp(-(self.KP * pitch - self.KD * float(w @ right)))
aligned = abs(yaw) < self.FIRE_CONE and abs(pitch) < self.FIRE_CONE
fire = aligned and d < self.FIRE_RANGE
if not self.dry:
self.pad.axis("LX", sx)
self.pad.axis("LY", sy)
if fire != self.firing:
(self.pad.press if fire else self.pad.release)("RB")
self.firing = fire
self.shots = getattr(self, "shots", 0) + (1 if fire else 0)
return (f"tgt={enm[3:24]:<22} d={d:8.0f} yaw={math.degrees(yaw):+6.1f} "
f"pit={math.degrees(pitch):+6.1f} stick=({sx:+.2f},{sy:+.2f}) fire={int(fire)}")
def run(self, secs, hz=10.0):
t0 = time.time()
last = t0
last_scan = 0.0
while time.time() - t0 < secs:
t = time.time()
if t - last_scan > 2.5:
ents = self.W.rescan()
last_scan = t
c = Counter(unit_faction(e[1]) for e in ents)
print(f"[{t-t0:6.1f}] rescan: {len(ents)} entities {dict(c)}", flush=True)
print(f"[{t-t0:6.1f}] {self.step(t - last)}", flush=True)
last = t
time.sleep(max(0, 1.0 / hz - (time.time() - t)))
if not self.dry:
self.pad.reset()
def main():
cfg = json.load(open(sys.argv[1]))
secs = float(sys.argv[2]) if len(sys.argv) > 2 else 60.0
dry = "--dry" in sys.argv
W = World(cfg)
W.rescan()
ap = Autopilot(W, Pad(), dry=dry)
ap.run(secs)
if __name__ == "__main__":
main()

144
tools/re-capture/binq.py Normal file
View File

@@ -0,0 +1,144 @@
#!/usr/bin/env python3
"""Ask questions of a ctrl_probe.py capture: which words answer to which input?
The capture is a window of the player entity object sampled every tick, tagged
with the pad state that produced it. That makes the interesting question
mechanical: for each 4-byte offset, does its value during phase X differ from
its value during the rest phases either side? A word that only moves while `RT`
is held is that input's state — a throttle setting, an afterburner tank, a heat
gauge — and one that moves in *every* phase is just live physics.
Sub-commands
phases per-phase mean of every offset that moves at all
respond <phase> offsets that move during <phase> and not at rest
near <value> [tol] offsets whose first sample is ~= value (HUD anchor)
trace <off> [off...] full time series of specific offsets (pos-relative hex)
"""
import os
import struct
import sys
import numpy as np
HDR = b"SYLPHCTR"
REC = struct.Struct("<d32sff3f")
def load(path):
with open(path, "rb") as f:
blob = f.read()
assert blob[:8] == HDR, "not a ctrl_probe capture"
n, win, back = struct.unpack_from("<III", blob, 8)
stride = REC.size + win
base = 8 + 12
ts, phase, sp, dodge, pos, wins = [], [], [], [], [], []
for i in range(n):
o = base + i * stride
t, ph, s, dg, x, y, z = REC.unpack_from(blob, o)
ts.append(t)
phase.append(ph.split(b"\0")[0].decode())
sp.append(s)
dodge.append(dg)
pos.append((x, y, z))
wins.append(blob[o + REC.size:o + REC.size + win])
# A run that ended in GAME OVER keeps sampling a dead object, and those
# frames dominate every statistic. $BINQ_TMAX truncates the capture to the
# part that was still flying.
tmax = float(os.environ.get("BINQ_TMAX", "inf"))
if tmax < float("inf"):
keep = [i for i, t in enumerate(ts) if t <= tmax]
n = len(keep)
ts = [ts[i] for i in keep]
phase = [phase[i] for i in keep]
sp = [sp[i] for i in keep]
dodge = [dodge[i] for i in keep]
pos = [pos[i] for i in keep]
wins = [wins[i] for i in keep]
A = np.frombuffer(b"".join(wins), dtype=">f4").reshape(n, win // 4).astype(np.float64)
U = np.frombuffer(b"".join(wins), dtype=">u4").reshape(n, win // 4)
return dict(n=n, win=win, back=back, t=np.array(ts), phase=phase,
speed=np.array(sp), dodge=np.array(dodge),
pos=np.array(pos), F=A, U=U)
def label(d, i):
return f"pos{i * 4 - d['back']:+#07x}"
def finite(d):
F = d["F"]
return np.all(np.isfinite(F), axis=0) & (np.max(np.abs(F), axis=0) < 1e12)
def cmd_phases(d, args):
ok = finite(d)
phases = []
for p in d["phase"]:
if p not in phases:
phases.append(p)
F = d["F"]
mv = np.zeros(F.shape[1])
means = {}
for p in phases:
m = np.array([x == p for x in d["phase"]])
means[p] = F[m].mean(axis=0)
for p in phases:
mv = np.maximum(mv, np.abs(means[p] - means[phases[0]]))
idx = np.flatnonzero(ok & (mv > 1e-3))
order = idx[np.argsort(-mv[idx])][:int(args[0]) if args else 25]
print("offset " + "".join(f"{p[:8]:>10}" for p in phases))
for i in order:
print(f"{label(d, i):<10}" + "".join(f"{means[p][i]:10.2f}" for p in phases))
def cmd_respond(d, args):
want = args[0]
F, ok = d["F"], finite(d)
inp = np.array([p == want for p in d["phase"]])
rest = np.array([p.startswith("rest") or p == "base" for p in d["phase"]])
if not inp.any():
sys.exit(f"no phase {want!r}")
# A word answering to this input must move *while it is held* and be quiet
# at rest; a word that also moves at rest is live physics, not the input.
a = F[inp]
r = F[rest]
d_in = a.max(axis=0) - a.min(axis=0)
d_rest = r.max(axis=0) - r.min(axis=0)
score = d_in - 2.0 * d_rest
idx = np.flatnonzero(ok & (d_in > 1e-3) & (score > 0))
for i in idx[np.argsort(-score[idx])][:20]:
print(f"{label(d, i):<10} in-phase {a[:, i].min():12.3f}..{a[:, i].max():12.3f}"
f" at-rest {r[:, i].min():12.3f}..{r[:, i].max():12.3f}")
if not len(idx):
print("(nothing moves under this input that is quiet at rest)")
def cmd_near(d, args):
v = float(args[0])
tol = float(args[1]) if len(args) > 1 else max(1e-3, abs(v) * 1e-3)
F, U = d["F"], d["U"]
for i in np.flatnonzero(np.abs(F[0] - v) <= tol):
print(f"{label(d, i):<10} f32 {F[0, i]:.4f} -> {F[-1, i]:.4f}")
for i in np.flatnonzero(np.abs(U[0].astype(np.float64) - v) <= tol):
print(f"{label(d, i):<10} u32 {U[0, i]} -> {U[-1, i]}")
def cmd_trace(d, args):
offs = [int(a, 0) for a in args]
idx = [(o + d["back"]) // 4 for o in offs]
print("t phase speed " + " ".join(f"{o:+#07x}" for o in offs))
for k in range(d["n"]):
print(f"{d['t'][k]:6.2f} {d['phase'][k]:<12} {d['speed'][k]:6.0f} "
+ " ".join(f"{d['F'][k, i]:8.2f}" for i in idx))
def main():
d = load(sys.argv[1])
print(f"# {d['n']} ticks, window {d['win']:#x} bytes, back {d['back']:#x}")
cmd = sys.argv[2] if len(sys.argv) > 2 else "phases"
{"phases": cmd_phases, "respond": cmd_respond, "near": cmd_near,
"trace": cmd_trace}[cmd](d, sys.argv[3:])
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,165 @@
#!/usr/bin/env python3
"""Learn which body axis each stick drives, and which button fires.
The first autopilot run nulled yaw but held a steady ~27 deg pitch error, which
is the signature of a wrong pitch axis or sign — guessing `right = row0` and
`up = fwd x right` assumes a handedness the game need not share. So measure it:
hold each stick axis, read the craft's body angular velocity from consecutive
orientation matrices, and see which body axis actually responds and in which
direction.
The fire button is found the same way — by consequence, not assumption: the
nose ammo counter in the player's object must go down.
Usage: calibrate.py <config.json> [out.json]
"""
import json
import math
import os
import struct
import sys
import time
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
import gworld # noqa: E402
import entities2 # noqa: E402
from flight_probe import Pad # noqa: E402
def player_off(w, fd, size, delta):
defs = entities2.definitions(w)
for _ in range(6):
mv = entities2.moving(fd, size, dt=0.5)
ents = entities2.typed(fd, defs, mv, delta)
for off, nm, pos, sp in ents:
if "Player" in nm:
return off
time.sleep(0.5)
return None
def read_rot(fd, off, rot_delta, stride):
n = stride * 2 + 12
b = os.pread(fd, n, off + rot_delta)
if len(b) < n:
return None
M = np.array([struct.unpack_from(">3f", b, stride * r) for r in range(3)])
if not np.all(np.isfinite(M)) or np.max(np.abs(M @ M.T - np.eye(3))) > 5e-3:
return None
return M
def mean_body_rate(fd, off, cfg, secs=2.0, hz=8.0):
"""Mean angular velocity expressed in the craft's own axes."""
prev, t_prev = None, None
acc, n = np.zeros(3), 0
end = time.time() + secs
while time.time() < end:
t = time.time()
M = read_rot(fd, off, cfg["rot_delta"], cfg.get("rot_stride", 12))
if M is not None and prev is not None:
dt = t - t_prev
D = prev @ M.T
w = np.array([D[2, 1] - D[1, 2], D[0, 2] - D[2, 0], D[1, 0] - D[0, 1]]) / 2.0
if dt > 0 and np.linalg.norm(w) < 0.5:
# into body axes
acc += M @ (w / dt)
n += 1
if M is not None:
prev, t_prev = M, t
time.sleep(max(0, 1.0 / hz - (time.time() - t)))
return acc / max(n, 1)
def main():
cfg = json.load(open(sys.argv[1]))
out = sys.argv[2] if len(sys.argv) > 2 else sys.argv[1]
w = gworld.World()
fd, size = w.fd, w.size
off = player_off(w, fd, size, cfg["def_delta"])
if off is None:
sys.exit("player not found")
print(f"# player pos va {gmem.primary_va(off):#010x}")
pad = Pad()
res = {}
for axis, name in (("LX", "yaw"), ("LY", "pitch")):
pad.reset()
time.sleep(1.0)
base = mean_body_rate(fd, off, cfg, 1.2)
pad.axis(axis, 0.9)
time.sleep(0.4)
wpos = mean_body_rate(fd, off, cfg, 2.0)
pad.axis(axis, 0.0)
time.sleep(1.2)
d = wpos - base
k = int(np.argmax(np.abs(d)))
res[name] = {"axis": k, "sign": 1 if d[k] > 0 else -1,
"mag": float(abs(d[k]))}
print(f"# {axis} (+0.9) -> body rate {d.round(3)} => {name} axis {k} "
f"sign {res[name]['sign']:+d}")
pad.reset()
# ---- fire button: the nose ammo counter must fall -----------------------
blob = os.pread(fd, 0x1000, max(0, off - 0x800))
ammo_offs = []
for i in range(0, len(blob) - 3, 4):
(u,) = struct.unpack_from(">I", blob, i)
(f,) = struct.unpack_from(">f", blob, i)
if u == 6000 or (math.isfinite(f) and abs(f - 6000.0) < 0.5):
ammo_offs.append((max(0, off - 0x800) + i) - off)
print(f"# ammo-like words (==6000) near the player: "
f"{[hex(o) for o in ammo_offs] or 'none'}")
def ammo():
vals = []
for d in ammo_offs:
b = os.pread(fd, 4, off + d)
if len(b) == 4:
vals.append(struct.unpack(">I", b)[0])
return vals
fire_btn = None
if ammo_offs:
for btn in ("RB", "LB", "A", "B", "X", "Y"):
before = ammo()
pad.press(btn)
time.sleep(1.2)
pad.release(btn)
time.sleep(0.5)
after = ammo()
drop = [a - b for a, b in zip(before, after)]
print(f"# {btn}: ammo delta {drop}")
if any(x > 0 for x in drop):
fire_btn = btn
break
for trig in ("RT", "LT"):
if fire_btn:
break
before = ammo()
pad.trig(trig, 1.0)
time.sleep(1.2)
pad.trig(trig, 0.0)
time.sleep(0.5)
after = ammo()
drop = [a - b for a, b in zip(before, after)]
print(f"# {trig}: ammo delta {drop}")
if any(x > 0 for x in drop):
fire_btn = trig
pad.reset()
cfg["yaw_axis"] = res["yaw"]["axis"]
cfg["yaw_sign"] = res["yaw"]["sign"]
cfg["pitch_axis"] = res["pitch"]["axis"]
cfg["pitch_sign"] = res["pitch"]["sign"]
cfg["fire"] = fire_btn
cfg["ammo_offs"] = ammo_offs
json.dump(cfg, open(out, "w"), indent=1)
print("\n" + json.dumps(cfg, indent=1))
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,272 @@
#!/usr/bin/env python3
"""Which control is the throttle, and where does the craft keep its own state?
Two questions, one flight. Both are answered by *consequence* rather than by
reading a field we hope is the right one:
* **Throttle.** The craft's speed is measured from its own position — a finite
difference on the position triple in guest RAM — so no speed field has to be
found first. The probe then holds each candidate input in turn and asks which
one changes that measured speed. (`findspeed.py` failed the other way round:
it assumed `RT` was the throttle and went looking for a field that rose.)
* **Own state.** Every tick also copies a window of the player entity object.
Afterwards, offsets whose float value tracks the measured speed are candidate
speed/throttle fields, and offsets that only ever *fall* are candidate
hull/shield/ammo — the numbers survival needs.
Flying straight into a firefight for 90 s is how earlier runs died, so the loop
keeps the collision avoidance from navigator.py armed the whole time and marks
any sample where it had to intervene: a phase that had to dodge is not a clean
speed measurement, and says so rather than being quietly averaged in.
Usage: ctrl_probe.py <config.json> <out-prefix> [hold_s] [settle_s]
"""
import json
import math
import os
import struct
import sys
import time
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
import navigator # noqa: E402
from flight_probe import Pad # noqa: E402
# 0x800 each way: the shield lives at pos+0x430 (own_state.py), which a
# 0x400 window silently cropped out of the first capture.
WIN_BACK = 0x800 # bytes of the player object kept before the position
WIN_FWD = 0x800 # ...and after
WIN = WIN_BACK + WIN_FWD
HZ = 20.0
# (label, [pad commands]) — everything the pad can do that might be a throttle.
# RB is left out: it is the fire button (autopilot-memory-driven.md) and firing
# during a speed measurement only invites return fire.
CANDIDATES = [
("RT", [("trig", "RT", 1.0)]),
("LT", [("trig", "LT", 1.0)]),
("RT+LT", [("trig", "RT", 1.0), ("trig", "LT", 1.0)]),
("A", [("press", "A")]),
("B", [("press", "B")]),
("X", [("press", "X")]),
("Y", [("press", "Y")]),
("LB", [("press", "LB")]),
("LS", [("press", "LS")]),
("RS", [("press", "RS")]),
("RY_up", [("axis", "RY", -1.0)]),
("RY_down", [("axis", "RY", 1.0)]),
("RX_right", [("axis", "RX", 1.0)]),
("dpad_up", [("dpad", "up")]),
("dpad_down", [("dpad", "down")]),
("dpad_left", [("dpad", "left")]),
("dpad_right", [("dpad", "right")]),
]
def apply(pad, cmds):
for c in cmds:
if c[0] == "trig":
pad.trig(c[1], c[2])
elif c[0] == "axis":
pad.axis(c[1], c[2])
elif c[0] == "press":
pad.press(c[1])
elif c[0] == "dpad":
pad.f.write(f"dpad {c[1]}\n")
def clear(pad):
pad.reset()
pad.f.write("dpad center\n")
class Run:
def __init__(self, cfg, prefix, hold, settle):
self.W = navigator.World(cfg)
self.nav = navigator.Navigator(self.W, None, dry=True)
self.prefix = prefix
self.hold = hold
self.settle = settle
self.pad = Pad()
self.rows = [] # (t, phase, pos, speed, dodged)
self.win = [] # raw window bytes per tick
ents = self.W.scan()
me = [(off, va) for off, va in ents if "Player" in self.W.defs[va]]
if not me:
sys.exit("player entity not in the scan — not in flight?")
self.me_off = me[0][0]
self.me_name = self.W.defs[me[0][1]]
print(f"# player {self.me_name} pos off {self.me_off:#x} "
f"va {gmem.primary_va(self.me_off):#x} | {len(ents)} entities",
flush=True)
# ------------------------------------------------------------- helpers
def pos(self):
return self.W.pos(self.me_off)
def sticks_for(self, vec):
"""Stick deflections that point the nose along `vec` (escape steering)."""
M = self.W.rot(self.me_off)
if M is None:
return 0.0, 0.0
fwd = M[self.W.fwd_row] * self.W.fwd_sign
right = M[(self.W.fwd_row + 1) % 3]
up = np.cross(fwd, right)
ez = float(np.dot(vec, fwd))
yaw = math.atan2(float(np.dot(vec, right)), ez if abs(ez) > 1e-3 else 1e-3)
pitch = math.atan2(float(np.dot(vec, up)), ez if abs(ez) > 1e-3 else 1e-3)
return (max(-1.0, min(1.0, 2.0 * yaw)), max(-1.0, min(1.0, -2.0 * pitch)))
# ---------------------------------------------------------------- phase
def phase(self, label, cmds, secs):
clear(self.pad)
apply(self.pad, cmds)
t_end = time.time() + secs
dodged = False
while time.time() < t_end:
t = time.time()
p = self.pos()
if p is None:
break
self.rows.append([t, label, p, 0.0, 0])
self.win.append(os.pread(self.W.fd, WIN, self.me_off - WIN_BACK))
# avoidance runs at 5 Hz; a real threat overrides the phase and the
# samples from here on are flagged
if len(self.rows) % 4 == 0:
ents = self.W.sample(t)
me = [e for e in ents if e[0] == self.me_off]
if me:
_, _, mp, mv, mr = me[0]
push, worst = self.nav.avoidance(mp, mv, mr, ents, self.me_off)
if float(np.linalg.norm(push)) > 0.6:
sx, sy = self.sticks_for(navigator.norm(push))
self.pad.axis("LX", sx)
self.pad.axis("LY", sy)
dodged = True
self.rows[-1][4] = 1
elif dodged:
self.pad.axis("LX", 0.0)
self.pad.axis("LY", 0.0)
time.sleep(max(0.0, 1.0 / HZ - (time.time() - t)))
return dodged
def run(self):
t0 = time.time()
self.phase("base", [], self.settle * 2)
for label, cmds in CANDIDATES:
d = self.phase(label, cmds, self.hold)
self.phase(f"rest_{label}", [], self.settle)
sp = self.phase_speed(label)
print(f"[{time.time()-t0:6.1f}] {label:<10} "
f"v0={sp[0]:7.1f} v1={sp[1]:7.1f} d={sp[1]-sp[0]:+7.1f}"
f"{' (dodged)' if d else ''}", flush=True)
clear(self.pad)
self.speeds()
self.dump()
# ------------------------------------------------------------ analysis
def speeds(self):
"""Fill in per-tick speed by central difference on position."""
for i, r in enumerate(self.rows):
j, k = max(0, i - 2), min(len(self.rows) - 1, i + 2)
dt = self.rows[k][0] - self.rows[j][0]
if dt > 1e-3:
r[3] = float(np.linalg.norm(self.rows[k][2] - self.rows[j][2])) / dt
def phase_speed(self, label):
"""(speed early, speed late) within a phase — needs speeds() first."""
self.speeds()
v = [r[3] for r in self.rows if r[1] == label]
if len(v) < 6:
return (0.0, 0.0)
n = max(2, len(v) // 4)
return (float(np.mean(v[:n])), float(np.mean(v[-n:])))
def dump(self):
with open(self.prefix + ".csv", "w") as f:
f.write("t,phase,x,y,z,speed,dodged\n")
t0 = self.rows[0][0]
for t, ph, p, sp, dg in self.rows:
f.write(f"{t-t0:.3f},{ph},{p[0]:.3f},{p[1]:.3f},{p[2]:.3f},"
f"{sp:.3f},{dg}\n")
with open(self.prefix + ".bin", "wb") as f:
f.write(b"SYLPHCTR")
f.write(struct.pack("<III", len(self.rows), WIN, WIN_BACK))
t0 = self.rows[0][0]
for (t, ph, p, sp, dg), w in zip(self.rows, self.win):
f.write(struct.pack("<d32sff3f", t - t0, ph.encode()[:32], sp,
float(dg), *[float(c) for c in p]))
f.write(w.ljust(WIN, b"\0"))
print(f"# wrote {self.prefix}.csv and {self.prefix}.bin "
f"({len(self.rows)} ticks)", flush=True)
# ---- per-phase summary
print("\n# phase n v_early v_late delta dodged")
order, seen = [], set()
for r in self.rows:
if r[1] not in seen:
seen.add(r[1])
order.append(r[1])
for ph in order:
v = [r[3] for r in self.rows if r[1] == ph]
dg = sum(r[4] for r in self.rows if r[1] == ph)
if len(v) < 6:
continue
n = max(2, len(v) // 4)
a, b = float(np.mean(v[:n])), float(np.mean(v[-n:]))
print(f" {ph:<12} {len(v):4d} {a:9.1f} {b:9.1f} {b-a:+9.1f} {dg:5d}")
# ---- which words in the object track speed, and which only fall
W_ = np.frombuffer(b"".join(x.ljust(WIN, b"\0") for x in self.win),
dtype=">f4").reshape(len(self.win), WIN // 4)
sp = np.array([r[3] for r in self.rows], dtype=np.float64)
with np.errstate(invalid="ignore", over="ignore"):
A = W_.astype(np.float64)
ok = np.all(np.isfinite(A), axis=0) & (np.max(np.abs(A), axis=0) < 1e9)
var = np.std(A, axis=0)
cand = np.flatnonzero(ok & (var > 1e-6))
cor = []
for i in cand:
c = np.corrcoef(A[:, i], sp)[0, 1]
if np.isfinite(c):
cor.append((abs(c), c, i))
cor.sort(reverse=True)
print("\n# object words correlating with measured speed "
"(offset relative to the position triple)")
for ac, c, i in cor[:12]:
off = i * 4 - WIN_BACK
print(f" pos{off:+#07x} r={c:+.3f} "
f"range {A[:, i].min():.3f} .. {A[:, i].max():.3f}")
print("\n# words that never rise (candidate hull / shield / ammo)")
shown = 0
for i in cand:
col = A[:, i]
if col[-1] >= col[0] - 1e-6:
continue
if np.max(np.diff(col)) > 1e-6:
continue
off = i * 4 - WIN_BACK
print(f" pos{off:+#07x} {col[0]:.3f} -> {col[-1]:.3f}")
shown += 1
if shown >= 12:
break
if not shown:
print(" (none — nothing in the window decreased monotonically)")
def main():
cfg = json.load(open(sys.argv[1]))
prefix = sys.argv[2]
hold = float(sys.argv[3]) if len(sys.argv) > 3 else 4.0
settle = float(sys.argv[4]) if len(sys.argv) > 4 else 2.5
Run(cfg, prefix, hold, settle).run()
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# One background task: bind the player's transform, then run the control probe.
#
# Everything that must not be interrupted lives in a single task here, because
# the display and the emulator both die on their own every few minutes in this
# container and nothing may depend on surviving between tool calls.
#
# Assumes the game is ALREADY in flight (launch_mission.sh). Pass --boot to
# have it get there itself.
set -u
export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy DISPLAY=:98
export PYTHONPATH=/sylph-home/.local/lib/python3.12/site-packages
SD="$(cd "$(dirname "$0")" && pwd)"
CFG=/tmp/nav-live.json
PRE=/tmp/ctrl
HOLD=3.0
SETTLE=2.0
if [ "${1:-}" = "--boot" ]; then
shift
"$SD/launch_mission.sh" fly || { echo "BOOT FAILED"; exit 1; }
fi
[ $# -ge 1 ] && CFG="$1"
[ $# -ge 2 ] && PRE="$2"
[ $# -ge 3 ] && HOLD="$3"
[ $# -ge 4 ] && SETTLE="$4"
python3 "$SD/entities2.py" self 0x130 "$CFG" || { echo "TRANSFORM BIND FAILED"; exit 1; }
echo "--- config: $(cat "$CFG")"
python3 "$SD/ctrl_probe.py" "$CFG" "$PRE" "$HOLD" "$SETTLE"
echo "PROBE DONE"

9
tools/re-capture/cyc.sh Executable file
View File

@@ -0,0 +1,9 @@
#!/usr/bin/env bash
# Step the Hangar weapons-container carousel right and capture the Name+DATA SHEET.
set -u
export HOME=/sylph-home/re
OUT=/sylph-home/re/caps; mkdir -p "$OUT"
vgamepad dpad right; sleep 0.30; vgamepad dpad center; sleep 3.0
screenshot /tmp/h.png >/dev/null 2>&1
convert /tmp/h.png -crop 530x480+700+95 +repage "$OUT/$1.png"
echo "$OUT/$1.png"

View File

@@ -0,0 +1,219 @@
#!/usr/bin/env python3
"""Type every live entity, and locate the player, via the definition pointer.
A live entity object keeps a pointer to its parsed `.tbl` definition (vtable
0x820af844, addresses known) at a **fixed offset from its position triple**.
Finding that offset once types every moving object in the scene at a stroke:
enemies, friendlies and us, separated from the thousands of moving particles.
The offset is *derived*, not assumed: for every moving triple, every nearby word
is checked against the set of definition addresses, and the winning delta is the
one that repeats across many independent entities.
Sub-commands:
delta find the (position -> definition pointer) offset
list [delta] type every live entity and print it
self [delta] the player's entity, its object window, and its orientation
"""
import math
import os
import struct
import sys
import time
from collections import Counter
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
import gworld # noqa: E402
def definitions(w):
out = {}
for off in w.scan_vtable(gworld.DEF_VTABLE):
nm = w.name_of(off)
if nm and nm.startswith("UN_"):
va = gmem.primary_va(off)
if va is not None:
out[struct.pack(">I", va)] = nm
return out
# Entity objects live in one heap region; scanning only it turns a 250 MB
# double-read into a few MB. That matters for more than speed: the full scan
# competes with the emulator for every core under lavapipe, and a game that
# does not advance a frame between the two samples has nothing "moving" in it.
ENT_VA_LO, ENT_VA_HI = 0xBD000000, 0xBE000000
def moving(fd, size, dt=0.5, lo=1.0, hi=4000.0, va_range=(ENT_VA_LO, ENT_VA_HI)):
def arrays():
if va_range:
f0, f1 = gmem.va_to_off(va_range[0]), gmem.va_to_off(va_range[1])
else:
f0, f1 = 0, size
for a, b in gmem.extents(fd, size):
a, b = max(a, f0), min(b, f1)
n = (b - a) // 4 * 4
if n >= 64:
yield a, np.frombuffer(os.pread(fd, n, a), dtype=">f4")
s0 = dict(arrays())
t0 = time.time()
time.sleep(dt)
s1 = dict(arrays())
t1 = time.time()
out = []
# Match extents by their file offset. Zipping the two lists positionally is
# wrong: the game allocates between the samples, the sparse extent layout
# shifts, and every subsequent pair misaligns -- which silently reports
# almost nothing as moving.
for o, a in s0.items():
b = s1.get(o)
if b is None or len(a) != len(b):
continue
with np.errstate(invalid="ignore"):
af = np.nan_to_num(a.astype(np.float64), nan=0, posinf=0, neginf=0)
bf = np.nan_to_num(b.astype(np.float64), nan=0, posinf=0, neginf=0)
d = bf - af
idx = np.flatnonzero(np.abs(d) > 1e-4)
cand = idx[np.isin(idx + 1, idx) & np.isin(idx + 2, idx)]
for i in cand:
v = np.array([d[i], d[i + 1], d[i + 2]])
sp = float(np.linalg.norm(v)) / (t1 - t0)
if lo < sp < hi:
out.append((o + int(i) * 4, tuple(af[i:i + 3]), sp))
return out
def find_delta(fd, defs, movers, radius=0x400):
votes = Counter()
for off, pos, sp in movers[:4000]:
lo = max(0, off - radius)
blob = os.pread(fd, radius * 2, lo)
for k in range(0, len(blob) - 3, 4):
if blob[k:k + 4] in defs:
votes[(lo + k) - off] += 1
return votes
def typed(fd, defs, movers, delta):
out = []
for off, pos, sp in movers:
try:
b = os.pread(fd, 4, off + delta)
except OSError:
continue
nm = defs.get(b)
if nm:
out.append((off, nm, pos, sp))
return out
def main():
cmd = sys.argv[1] if len(sys.argv) > 1 else "delta"
w = gworld.World()
fd, size = w.fd, w.size
defs = definitions(w)
print(f"# {len(defs)} unit definitions", flush=True)
movers = moving(fd, size)
print(f"# {len(movers)} moving triples", flush=True)
if cmd == "delta":
votes = find_delta(fd, defs, movers)
print("# candidate (position -> definition pointer) deltas:")
for d, n in votes.most_common(8):
print(f" {d:+#08x} seen {n}")
return
delta = int(sys.argv[2], 0) if len(sys.argv) > 2 else 0x130
ents = typed(fd, defs, movers, delta)
uniq = {}
for off, nm, pos, sp in ents:
uniq.setdefault((nm, tuple(round(c, 1) for c in pos)), (off, nm, pos, sp))
ents = list(uniq.values())
print(f"# {len(ents)} typed live entities (delta {delta:+#x})")
if cmd == "list":
c = Counter(nm for _, nm, _, _ in ents)
for nm, k in c.most_common():
print(f" {k:4d} {nm}")
for off, nm, pos, sp in sorted(ents, key=lambda e: e[1])[:60]:
print(f" {gmem.primary_va(off):#010x} {nm:<40} "
f"({pos[0]:+9.1f},{pos[1]:+9.1f},{pos[2]:+9.1f}) {sp:7.1f}/s")
return
if cmd == "self":
me = [e for e in ents if "Player" in e[1]]
if not me:
sys.exit("player entity not found")
off, nm, pos, sp = me[0]
print(f"# player entity: pos va {gmem.primary_va(off):#010x} {nm}")
print(f"# pos ({pos[0]:+.1f},{pos[1]:+.1f},{pos[2]:+.1f}) speed {sp:.1f}/s")
# orientation inside the same object
# The rotation is stored with a 16-byte row stride (a 4x4 transform
# whose translation row is the position we already have), so a test for
# nine *contiguous* floats structurally cannot find it. Test both.
lo = max(0, off - 0x800)
blob = os.pread(fd, 0x1000, lo)
found = []
for k in range(0, len(blob) - 48, 4):
for stride, tag in ((12, "3x3"), (16, "4x4")):
try:
rows = [np.array(struct.unpack_from(">3f", blob, k + stride * r))
for r in range(3)]
except struct.error:
continue
M = np.array(rows)
if not np.all(np.isfinite(M)) or np.max(np.abs(M)) > 1.001:
continue
if np.max(np.abs(M @ M.T - np.eye(3))) > 3e-3:
continue
if abs(np.linalg.det(M) - 1.0) > 1e-2:
continue
found.append(((lo + k) - off, M, stride))
break
print(f"# orthonormal 3x3 blocks inside the player object: {len(found)}")
for d, M, st in found[:6]:
print(f" pos{d:+#07x} stride {st}: " + " ".join(
"(" + ",".join(f"{v:+.3f}" for v in row) + ")" for row in M))
if len(sys.argv) > 3 and found:
import json
# Which of the orthonormal blocks is the CRAFT's attitude? The one
# with a row along the direction of travel. Taking found[0] is what
# produced a config with rot_delta -0x764 and a nonsense forward
# axis: several blocks inside the object are orthonormal (bone or
# camera frames), and only the craft's own has a row that tracks
# where the craft is going.
p0 = np.array(pos)
time.sleep(0.35)
p1 = np.array(struct.unpack(">3f", os.pread(fd, 12, off)))
step = p1 - p0
if np.linalg.norm(step) < 1e-3:
sys.exit("craft is not moving — cannot bind the forward axis")
vdir = step / np.linalg.norm(step)
best = None
for d, M, st in found:
cs = [float(M[r] @ vdir) for r in range(3)]
r = int(np.argmax([abs(c) for c in cs]))
if best is None or abs(cs[r]) > abs(best[3]):
best = (d, M, st, cs[r], r)
rot_delta, M, rot_stride, cos, row = best
sign = 1 if cos > 0 else -1
print(f"# attitude block pos{rot_delta:+#07x} stride {rot_stride}: "
f"forward = row {row} (sign {sign:+d}), cos={cos:+.3f}")
if abs(cos) < 0.9:
print("# WARNING: no block tracks the flight path (|cos| < 0.9)"
" — the craft may be drifting hard; re-run while flying straight")
cfg = {"def_delta": delta, "rot_delta": rot_delta,
"rot_stride": rot_stride,
"fwd_row": row, "fwd_sign": sign, "fwd_cos": round(cos, 4),
"va_lo": ENT_VA_LO, "va_hi": ENT_VA_HI}
json.dump(cfg, open(sys.argv[3], "w"), indent=1)
print("# wrote " + sys.argv[3] + ": " + json.dumps(cfg))
return
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,44 @@
#!/usr/bin/env bash
# One background task: boot -> Stage 02 -> fly the pilot while a second reader
# records EVERY entity's hull, so the escort question ("who is losing, and how
# fast") is answered from memory instead of from the HUD.
#
# Same one-task rule as fly_session.sh: the display and the emulator both die on
# their own in this container, so nothing may depend on surviving between tool
# calls.
set -u
export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy DISPLAY=:98
export PYTHONPATH=/sylph-home/.local/lib/python3.12/site-packages
SD="$(cd "$(dirname "$0")" && pwd)"
SECS="${1:-300}"
TAG="${2:-escort}"
SHOTS=/sylph-home/re/shots
CFG=/tmp/nav-live.json
"$SD/launch_mission.sh" fly || { echo "BOOT FAILED"; exit 1; }
python3 "$SD/entities2.py" self 0x130 "$CFG" || { echo "BIND FAILED"; exit 1; }
echo "--- config: $(cat "$CFG")"
echo "=== initial entity table ==="
python3 "$SD/mission_state.py" scan "$CFG"
# The HUD's REMAINING OB counter has no known address yet, so the correlation
# material is a screenshot every 20 s stamped against the same clock as the
# memory samples.
( for i in $(seq 1 20); do
printf '%s SHOT %02d\n' "$(date +%s.%N)" "$i" >> "/tmp/$TAG-shots.log"
screenshot "$SHOTS/$TAG-$i.png" >/dev/null 2>&1
sleep 20
done ) &
SHOTTER=$!
date +%s.%N > "/tmp/$TAG-t0"
python3 "$SD/mission_state.py" watch "$CFG" "$SECS" 1 "/tmp/$TAG-mission.jsonl" \
> "/tmp/$TAG-mission.log" 2>&1 &
WATCHER=$!
python3 "$SD/pilot.py" "$CFG" "$SECS" > "/tmp/$TAG-pilot.log" 2>&1
wait $WATCHER 2>/dev/null
kill $SHOTTER 2>/dev/null
screenshot "$SHOTS/$TAG-end.png" >/dev/null 2>&1
echo "SESSION DONE"

View File

@@ -0,0 +1,120 @@
#!/usr/bin/env python3
"""Find the player craft's live position in guest RAM, from motion alone.
The spawn records at vtable 0x820af030 turned out to hold no live state (every
word is constant across a 29 s capture), so the flying entity is some other
object. Rather than guess which class, this finds it by what it must *do*:
a position triple moves in a straight line at constant speed while coasting.
Method: sample all of RAM K times ~dt apart, keep word indices whose float
value changes every time, then require three consecutive such words to satisfy
* equal step length each interval (constant speed), and
* a constant direction (straight flight),
* with speed in a sane range for a craft.
Only the wholly-mechanical properties of motion are used; no offset, class or
encoding is assumed.
Usage: findplayer.py [samples] [interval_s]
"""
import math
import os
import struct
import sys
import time
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
def snapshot(fd, size):
"""[(start_off, np.ndarray big-endian f32)] over allocated extents."""
out = []
for a, b in gmem.extents(fd, size):
n = (b - a) // 4 * 4
if n < 64:
continue
buf = os.pread(fd, n, a)
out.append((a, np.frombuffer(buf, dtype=">f4")))
return out
def main():
K = int(sys.argv[1]) if len(sys.argv) > 1 else 5
dt = float(sys.argv[2]) if len(sys.argv) > 2 else 0.35
path = gmem.mem_path()
fd = os.open(path, os.O_RDONLY)
size = os.path.getsize(path)
print(f"# sampling {K}x every {dt}s", flush=True)
snaps, times = [], []
for k in range(K):
t = time.time()
snaps.append(snapshot(fd, size))
times.append(t)
print(f"# sample {k} ({len(snaps[-1])} extents, "
f"{sum(len(a) for _, a in snaps[-1])*4/1e6:.0f} MB) "
f"in {time.time()-t:.2f}s", flush=True)
time.sleep(max(0, dt - (time.time() - t)))
# extents must line up across samples for a word-wise comparison
shapes = [tuple((o, len(a)) for o, a in s) for s in snaps]
if len(set(shapes)) != 1:
common = set(shapes[0])
for sh in shapes[1:]:
common &= set(sh)
print(f"# extents shifted between samples; using {len(common)} common ones")
keep = lambda s: [(o, a) for o, a in s if (o, len(a)) in common]
snaps = [keep(s) for s in snaps]
hits = []
for e in range(len(snaps[0])):
base = snaps[0][e][0]
arrs = [np.nan_to_num(s[e][1].astype(np.float64), nan=0, posinf=0, neginf=0)
for s in snaps]
# per-interval deltas
d = [arrs[k + 1] - arrs[k] for k in range(K - 1)]
moving = np.ones(len(arrs[0]), dtype=bool)
for dk in d:
moving &= np.abs(dk) > 1e-3
idx = np.flatnonzero(moving)
if len(idx) == 0:
continue
# candidate triples: i, i+1, i+2 all moving
cand = idx[np.isin(idx + 1, idx) & np.isin(idx + 2, idx)]
for i in cand:
steps = []
dirs = []
ok = True
for k in range(K - 1):
v = np.array([d[k][i], d[k][i + 1], d[k][i + 2]])
n = float(np.linalg.norm(v))
ddt = times[k + 1] - times[k]
sp = n / ddt
if not (20.0 < sp < 3000.0):
ok = False
break
steps.append(sp)
dirs.append(v / n)
if not ok or len(steps) < 3:
continue
if max(steps) - min(steps) > 0.25 * (sum(steps) / len(steps)):
continue
if min(float(np.dot(dirs[0], dd)) for dd in dirs) < 0.985:
continue
va = gmem.primary_va(base + int(i) * 4)
pos = tuple(float(arrs[0][i + j]) for j in range(3))
hits.append((va, base + int(i) * 4, sum(steps) / len(steps), pos))
print(f"\n# {len(hits)} position-like triples (straight, constant speed)")
hits.sort(key=lambda h: -h[2])
for va, off, sp, pos in hits[:40]:
print(f" va {va:#010x} speed {sp:8.1f}/s pos "
f"({pos[0]:+11.1f},{pos[1]:+11.1f},{pos[2]:+11.1f})")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,147 @@
#!/usr/bin/env python3
"""Scan ALL of guest RAM for 3x3 orthonormal float blocks (rotation matrices).
Vectorised with numpy: nine shifted views of each extent give every candidate
9-float window at once, so the whole 250 MB working set tests in seconds.
A rotation matrix is a very strong signature — unit rows, mutually orthogonal —
so the hits are almost entirely real orientations. Two passes taken a moment
apart, with a known stick input in between, then say which of them is *ours*.
Usage: findrot_global.py [--track seconds]
"""
import os
import struct
import sys
import time
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
FIFO = "/tmp/sylph-vgamepad.fifo"
TOL = 2e-3
def pad(line):
with open(FIFO, "w") as f:
f.write(line + "\n")
def scan(fd, size):
"""[(file_offset, 9 floats)] for every orthonormal 3x3 block."""
hits = []
for a, b in gmem.extents(fd, size):
n = (b - a) // 4 * 4
if n < 64:
continue
arr = np.frombuffer(os.pread(fd, n, a), dtype=">f4").astype(np.float32)
if arr.size < 16:
continue
m = arr.size - 8
cols = [arr[i:i + m] for i in range(9)]
with np.errstate(invalid="ignore", over="ignore"):
finite = np.ones(m, dtype=bool)
for c in cols:
finite &= np.isfinite(c) & (np.abs(c) <= 1.001)
# row norms
n0 = cols[0] ** 2 + cols[1] ** 2 + cols[2] ** 2
n1 = cols[3] ** 2 + cols[4] ** 2 + cols[5] ** 2
n2 = cols[6] ** 2 + cols[7] ** 2 + cols[8] ** 2
ok = finite
ok &= np.abs(n0 - 1) < TOL
ok &= np.abs(n1 - 1) < TOL
ok &= np.abs(n2 - 1) < TOL
d01 = cols[0] * cols[3] + cols[1] * cols[4] + cols[2] * cols[5]
d02 = cols[0] * cols[6] + cols[1] * cols[7] + cols[2] * cols[8]
d12 = cols[3] * cols[6] + cols[4] * cols[7] + cols[5] * cols[8]
ok &= np.abs(d01) < TOL
ok &= np.abs(d02) < TOL
ok &= np.abs(d12) < TOL
for i in np.flatnonzero(ok):
hits.append(a + int(i) * 4)
return hits
def read_mat(fd, off):
b = os.pread(fd, 36, off)
if len(b) < 36:
return None
return np.array(struct.unpack(">9f", b))
def main():
fd = os.open(gmem.mem_path(), os.O_RDONLY)
size = os.path.getsize(gmem.mem_path())
pad("reset")
time.sleep(0.6)
t0 = time.time()
hits = scan(fd, size)
print(f"# {len(hits)} orthonormal 3x3 blocks in RAM ({time.time()-t0:.1f}s)", flush=True)
if not hits:
return
# which of them rotate when WE yaw? measure change under left vs right
def deltas(lx, secs=2.5, hz=6.0):
"""Mean per-step rotation vector while the stick is held.
Sampled incrementally: at a few degrees per step the skew part of
A·Bᵀ is the rotation vector, which it is not over a 2 s turn. Any block
that stops being orthonormal mid-phase is memory that got reused, not an
orientation, and is dropped.
"""
pad(f"axis LX {lx}")
time.sleep(0.5)
seq = []
for _ in range(int(secs * hz)):
t = time.time()
seq.append({o: read_mat(fd, o) for o in hits})
time.sleep(max(0, 1.0 / hz - (time.time() - t)))
pad("axis LX 0")
time.sleep(1.0)
def ortho(m):
if m is None or not np.all(np.isfinite(m)):
return False
M = m.reshape(3, 3)
return np.max(np.abs(M @ M.T - np.eye(3))) < 5e-3
out = {}
for o in hits:
ms = [s[o] for s in seq]
if not all(ortho(m) for m in ms):
continue
ws = []
for a, b in zip(ms, ms[1:]):
M = a.reshape(3, 3) @ b.reshape(3, 3).T
w = np.array([M[2, 1] - M[1, 2], M[0, 2] - M[2, 0], M[1, 0] - M[0, 1]]) / 2
if np.linalg.norm(w) < 0.5: # small-angle regime only
ws.append(w)
if len(ws) >= 4:
out[o] = np.mean(ws, axis=0)
return out
dl = deltas(-0.9)
dr = deltas(+0.9)
rows = []
for o in hits:
if o not in dl or o not in dr:
continue
a, b = dl[o], dr[o]
na, nb = np.linalg.norm(a), np.linalg.norm(b)
if na < 0.01 or nb < 0.01:
continue
cos = float(a @ b / (na * nb))
rows.append((cos, o, na, nb))
rows.sort()
print("\n# orientation blocks that rotate OPPOSITE ways for left vs right stick")
for cos, o, na, nb in rows[:12]:
va = gmem.primary_va(o)
print(f" va {va:#010x} cos={cos:+.3f} |wL|={na:.3f} |wR|={nb:.3f}")
if not rows:
print(" (none)")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,146 @@
#!/usr/bin/env python3
"""Identify which moving object in RAM is the *player* craft, by input correlation.
Position triples are easy to find (findplayer.py); saying which one is us is the
hard part, and no static property answers it. This does: hold hard-left yaw for
a few seconds, then hard-right, and look for the object whose turn axis reverses
in phase with the stick. Every other craft is AI-flown and uncorrelated with our
input, so the discriminator is causal rather than circumstantial.
Phase 1 finds candidate triples from two whole-RAM samples; after that only the
candidates' 12 bytes are re-read, so the sampling loop is cheap.
Usage: findself.py [phase_seconds] [hz]
"""
import math
import os
import struct
import sys
import time
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
FIFO = "/tmp/sylph-vgamepad.fifo"
def pad(line):
with open(FIFO, "w") as f:
f.write(line + "\n")
def snapshot(fd, size):
return [(a, np.frombuffer(os.pread(fd, (b - a) // 4 * 4, a), dtype=">f4"))
for a, b in gmem.extents(fd, size) if (b - a) >= 64]
def candidates(fd, size, dt=0.35):
s0 = snapshot(fd, size)
t0 = time.time()
time.sleep(dt)
s1 = snapshot(fd, size)
t1 = time.time()
out = []
for (o, a), (o2, b) in zip(s0, s1):
if o != o2 or len(a) != len(b):
continue
with np.errstate(invalid="ignore"):
af = np.nan_to_num(a.astype(np.float64), nan=0, posinf=0, neginf=0)
bf = np.nan_to_num(b.astype(np.float64), nan=0, posinf=0, neginf=0)
d = bf - af
idx = np.flatnonzero(np.abs(d) > 1e-3)
cand = idx[np.isin(idx + 1, idx) & np.isin(idx + 2, idx)]
for i in cand:
v = np.array([d[i], d[i + 1], d[i + 2]])
sp = float(np.linalg.norm(v)) / (t1 - t0)
if 80.0 < sp < 2000.0:
out.append(o + int(i) * 4)
return out
def sample(fd, offs):
t = time.time()
pos = np.empty((len(offs), 3))
for k, o in enumerate(offs):
b = os.pread(fd, 12, o)
if len(b) < 12:
pos[k] = np.nan
continue
pos[k] = struct.unpack(">3f", b)
return t, pos
def turn_axis(series):
"""Mean normalised cross(v_k, v_k+1) over a phase, per candidate."""
ts = [t for t, _ in series]
ps = [p for _, p in series]
vs = []
for k in range(len(ps) - 1):
dt = ts[k + 1] - ts[k]
vs.append((ps[k + 1] - ps[k]) / max(dt, 1e-3))
ax = np.zeros((ps[0].shape[0], 3))
n = 0
for k in range(len(vs) - 1):
c = np.cross(vs[k], vs[k + 1])
nrm = np.linalg.norm(c, axis=1, keepdims=True)
with np.errstate(invalid="ignore", divide="ignore"):
ax += np.where(nrm > 1e-6, c / nrm, 0.0)
n += 1
return ax / max(n, 1)
def main():
secs = float(sys.argv[1]) if len(sys.argv) > 1 else 3.0
hz = float(sys.argv[2]) if len(sys.argv) > 2 else 6.0
path = gmem.mem_path()
fd = os.open(path, os.O_RDONLY)
size = os.path.getsize(path)
pad("reset")
time.sleep(0.5)
offs = candidates(fd, size)
print(f"# {len(offs)} moving-triple candidates", flush=True)
if not offs:
sys.exit("nothing is moving — in flight?")
phases = {}
for name, lx in (("left", -0.9), ("right", 0.9)):
pad(f"axis LX {lx}")
time.sleep(0.6) # let the turn establish
series = []
n = int(secs * hz)
for _ in range(n):
t0 = time.time()
series.append(sample(fd, offs))
time.sleep(max(0, 1.0 / hz - (time.time() - t0)))
phases[name] = turn_axis(series)
pad("axis LX 0")
time.sleep(1.2)
print(f"# phase {name} done", flush=True)
pad("reset")
aL, aR = phases["left"], phases["right"]
nL = np.linalg.norm(aL, axis=1)
nR = np.linalg.norm(aR, axis=1)
with np.errstate(invalid="ignore", divide="ignore"):
cos = np.sum(aL * aR, axis=1) / (nL * nR)
good = np.isfinite(cos) & (nL > 0.5) & (nR > 0.5)
order = np.argsort(np.where(good, cos, 9e9))
print("\n# objects whose turn axis REVERSES with the stick (most negative first)")
shown = 0
for k in order:
if not good[k]:
break
print(f" va {gmem.primary_va(offs[k]):#010x} cos(axisL,axisR) = {cos[k]:+.3f}"
f" |L|={nL[k]:.2f} |R|={nR[k]:.2f}")
shown += 1
if shown >= 12:
break
if not shown:
print(" (none — try longer phases)")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,115 @@
#!/usr/bin/env python3
"""Locate the player's flight object via its speed scalar, then its position.
The HUD prints the craft's speed, so it is a known quantity we can *change on
demand* — the classic two-state value scan, which is far more reliable than
hunting for a structure by shape:
1. coast -> RAM holds the cruise speed somewhere; collect every float ≈ it;
2. boost -> the real one rises; everything coincidental is filtered out;
3. coast -> it must come back down.
Whatever survives all three is the player's speed. Its object then contains the
position, which is confirmed independently: a position triple near that scalar
must move at exactly the speed the scalar reports.
Usage: findspeed.py <cruise> [boost_wait]
"""
import os
import struct
import sys
import time
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
FIFO = "/tmp/sylph-vgamepad.fifo"
def pad(line):
with open(FIFO, "w") as f:
f.write(line + "\n")
def extents_arrays(fd, size):
for a, b in gmem.extents(fd, size):
n = (b - a) // 4 * 4
if n >= 64:
yield a, np.frombuffer(os.pread(fd, n, a), dtype=">f4")
def scan_equal(fd, size, val, tol):
hits = []
for a, arr in extents_arrays(fd, size):
with np.errstate(invalid="ignore"):
ok = np.isfinite(arr) & (np.abs(arr.astype(np.float64) - val) <= tol)
for i in np.flatnonzero(ok):
hits.append(a + int(i) * 4)
return hits
def read_f(fd, off):
b = os.pread(fd, 4, off)
return struct.unpack(">f", b)[0] if len(b) == 4 else float("nan")
def main():
cruise = float(sys.argv[1])
wait = float(sys.argv[2]) if len(sys.argv) > 2 else 4.0
fd = os.open(gmem.mem_path(), os.O_RDONLY)
size = os.path.getsize(gmem.mem_path())
pad("reset")
time.sleep(2.0)
c0 = scan_equal(fd, size, cruise, 2.0)
print(f"# pass 1 (coast {cruise}): {len(c0)} candidates", flush=True)
pad("trig RT 1.0")
time.sleep(wait)
c1 = [o for o in c0 if read_f(fd, o) > cruise + 40]
print(f"# pass 2 (boost): {len(c1)} rose above {cruise+40:.0f}", flush=True)
vals = {o: read_f(fd, o) for o in c1}
pad("reset")
time.sleep(wait + 2.0)
c2 = [o for o in c1 if abs(read_f(fd, o) - cruise) <= 6.0]
print(f"# pass 3 (coast again): {len(c2)} returned to ≈{cruise}", flush=True)
for o in c2[:20]:
print(f" va {gmem.primary_va(o):#010x} boost value was {vals[o]:.1f}")
if not c2:
print("# no survivors — is the craft actually accelerating?")
return
# position triple in the same object: must move at exactly that speed
print("\n# looking for a position triple near each survivor", flush=True)
RAD = 0x400
for o in c2[:8]:
lo = max(0, o - RAD)
n = RAD * 2
t0 = time.time()
a0 = np.frombuffer(os.pread(fd, n, lo), dtype=">f4").astype(np.float64)
time.sleep(0.4)
t1 = time.time()
a1 = np.frombuffer(os.pread(fd, n, lo), dtype=">f4").astype(np.float64)
sp_now = read_f(fd, o)
dt = t1 - t0
best = []
for i in range(len(a0) - 2):
d = a1[i:i + 3] - a0[i:i + 3]
if not np.all(np.isfinite(d)):
continue
v = float(np.linalg.norm(d)) / dt
if abs(v - sp_now) <= max(8.0, 0.06 * sp_now):
best.append((lo + i * 4, v, tuple(a0[i:i + 3])))
print(f" survivor va {gmem.primary_va(o):#010x} (speed {sp_now:.1f}): "
f"{len(best)} matching triples")
for off, v, p in best[:4]:
print(f" pos va {gmem.primary_va(off):#010x} delta-off "
f"{off - o:+#07x} |v|={v:7.1f} ({p[0]:+9.1f},{p[1]:+9.1f},{p[2]:+9.1f})")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,157 @@
#!/usr/bin/env python3
"""Derive the player craft's live transform offsets from a flight_probe capture.
Nothing here is guessed from plausible-looking numbers. Two independent
structural facts do the work:
1. a rotation matrix is 9 floats that are orthonormal with det +1 — a
property essentially no other data has;
2. a position triple's frame-to-frame delta must point along the craft's
own forward axis when it is flying straight, and its magnitude must be the
same for every frame at constant speed.
Test 2 validates the position candidate *and* the rotation candidate against
each other, so a coincidence has to satisfy both at once.
Usage: flight_analyze.py <probe.bin> [name-substring]
"""
import math
import os
import struct
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gworld # noqa: E402
def load(path):
f = open(path, "rb")
assert f.read(8) == b"SYLPHPRB"
n_inst, window, n_in = struct.unpack("<III", f.read(12))
insts = []
for _ in range(n_inst):
va, off = struct.unpack("<IQ", f.read(12))
nm = f.read(64).split(b"\0")[0].decode()
insts.append((va, off, nm))
frames = []
rec = 8 + 4 * n_in + 32 + n_inst * window
while True:
b = f.read(rec)
if len(b) < rec:
break
t = struct.unpack_from("<d", b, 0)[0]
inp = struct.unpack_from("<%df" % n_in, b, 8)
label = struct.unpack_from("<32s", b, 8 + 4 * n_in)[0].split(b"\0")[0].decode()
base = 8 + 4 * n_in + 32
bufs = [b[base + i * window: base + (i + 1) * window] for i in range(n_inst)]
frames.append((t, inp, label, bufs))
return insts, window, frames
def f32(buf, o):
return struct.unpack_from(">f", buf, o)[0]
def vec(buf, o):
return (f32(buf, o), f32(buf, o + 4), f32(buf, o + 8))
def sub(a, b):
return (a[0] - b[0], a[1] - b[1], a[2] - b[2])
def norm(a):
return math.sqrt(sum(c * c for c in a))
def dot(a, b):
return sum(x * y for x, y in zip(a, b))
def analyse(insts, window, frames, want):
idx = [i for i, (_, _, nm) in enumerate(insts) if want in nm]
if not idx:
sys.exit(f"no instance matching {want!r}; have: "
+ ", ".join(sorted({nm for _, _, nm in insts})))
i = idx[0]
va, off, nm = insts[i]
print(f"# player instance {va:#010x} {nm} ({len(frames)} frames)")
# ---- rotation blocks that hold up across every frame ------------------
per_frame = []
for t, inp, lab, bufs in frames:
per_frame.append({o for o, kind, m in gworld.find_rotations(bufs[i]) if kind == "3x3"})
stable = set.intersection(*per_frame) if per_frame else set()
print(f"# 3x3 orthonormal blocks valid in ALL frames: "
+ (", ".join(f"{o:#05x}" for o in sorted(stable)) or "none"))
# ---- position candidates ---------------------------------------------
# constant-speed straight flight: |delta| equal every frame, direction fixed
idle = [k for k, (t, inp, lab, b) in enumerate(frames) if lab.startswith("idle")]
if len(idle) < 6:
idle = list(range(min(20, len(frames))))
seg = idle[:20]
cands = []
for o in range(0, window - 12, 4):
ds, ok = [], True
for a, b in zip(seg, seg[1:]):
if b != a + 1:
continue
dt = frames[b][0] - frames[a][0]
p0, p1 = vec(frames[a][3][i], o), vec(frames[b][3][i], o)
if not all(map(math.isfinite, p0 + p1)):
ok = False
break
d = sub(p1, p0)
if dt <= 0:
ok = False
break
ds.append((norm(d) / dt, d))
if not ok or len(ds) < 4:
continue
sp = [s for s, _ in ds]
mean = sum(sp) / len(sp)
if mean < 1.0 or mean > 5000.0: # a craft, not a counter
continue
spread = max(sp) - min(sp)
if spread > 0.15 * mean: # constant speed while coasting
continue
# direction must be steady too
dirs = [(d[0] / (norm(d) or 1), d[1] / (norm(d) or 1), d[2] / (norm(d) or 1))
for _, d in ds]
if min(dot(dirs[0], d) for d in dirs) < 0.99:
continue
cands.append((o, mean, dirs[0]))
print(f"# position candidates (steady speed + steady heading while coasting): "
f"{len(cands)}")
for o, sp, d in cands[:12]:
print(f" +{o:#05x} speed {sp:8.2f}/s dir ({d[0]:+.3f},{d[1]:+.3f},{d[2]:+.3f})")
# ---- cross-validate: velocity must lie along a rotation row -----------
print("\n# cross-check: does the motion direction match a rotation-matrix axis?")
best = []
for o, sp, d in cands:
for ro in sorted(stable):
m = struct.unpack_from(">9f", frames[seg[0]][3][i], ro)
for r, label in ((m[0:3], "row0/right"), (m[3:6], "row1/up"), (m[6:9], "row2/fwd")):
c = abs(dot(d, r))
if c > 0.98:
best.append((c, o, ro, label, sp))
best.sort(reverse=True)
if not best:
print(" none — position and orientation candidates do not corroborate")
for c, o, ro, label, sp in best[:10]:
print(f" pos +{o:#05x} ∥ rot +{ro:#05x} {label} |cos|={c:.5f} speed {sp:.1f}")
return stable, cands
def main():
path = sys.argv[1]
want = sys.argv[2] if len(sys.argv) > 2 else "Player"
insts, window, frames = load(path)
analyse(insts, window, frames, want)
if __name__ == "__main__":
main()

107
tools/re-capture/flight_probe.py Executable file
View File

@@ -0,0 +1,107 @@
#!/usr/bin/env python3
"""Drive a scripted input sequence in flight while sampling guest RAM, so the
player craft's live transform can be *derived* rather than guessed.
The point is correlation: each sample is tagged with the stick/trigger state
that produced it, so afterwards we can ask "which floats move only while the
yaw axis is deflected?" and "which triple integrates to the velocity?".
Writes a .npz-ish flat binary: a header of (va, offset, name) per instance,
then per frame a timestamp, the input vector, and every instance's raw window.
Usage: flight_probe.py <out.bin> [seconds]
"""
import os
import struct
import sys
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gworld # noqa: E402
FIFO = "/tmp/sylph-vgamepad.fifo"
class Pad:
"""Talk to the vgamepad server directly over its FIFO.
The `vgamepad` CLI spawns a process per command (~20 ms); a control loop
cannot afford that, and `tap`/`hold` additionally sleep *inside* the server.
Writing lines to the FIFO ourselves keeps a tick under a millisecond.
"""
def __init__(self):
self.f = open(FIFO, "w", buffering=1)
self.state = {"LX": 0.0, "LY": 0.0, "RX": 0.0, "RY": 0.0, "LT": 0.0, "RT": 0.0}
def axis(self, name, v):
self.state[name] = v
self.f.write(f"axis {name} {v:.3f}\n")
def trig(self, name, v):
self.state[name] = v
self.f.write(f"trig {name} {v:.3f}\n")
def press(self, b):
self.f.write(f"press {b}\n")
def release(self, b):
self.f.write(f"release {b}\n")
def reset(self):
self.f.write("reset\n")
for k in self.state:
self.state[k] = 0.0
def vector(self):
return [self.state[k] for k in ("LX", "LY", "RX", "RY", "LT", "RT")]
# (duration_s, description, action)
SCRIPT = [
(3.0, "idle", lambda p: p.reset()),
(4.0, "pitch-up", lambda p: p.axis("LY", -0.9)),
(2.0, "idle2", lambda p: p.reset()),
(4.0, "yaw-right", lambda p: p.axis("LX", 0.9)),
(2.0, "idle3", lambda p: p.reset()),
(4.0, "yaw-left", lambda p: p.axis("LX", -0.9)),
(2.0, "idle4", lambda p: p.reset()),
(5.0, "boost", lambda p: p.trig("RT", 1.0)),
(3.0, "idle5", lambda p: p.reset()),
]
def main():
out = sys.argv[1]
w = gworld.World()
inst = w.refresh()
print(f"# {len(inst)} instances", flush=True)
if not inst:
sys.exit("no instances — not in a mission?")
pad = Pad()
hz = 10.0
with open(out, "wb") as f:
f.write(b"SYLPHPRB")
f.write(struct.pack("<III", len(inst), gworld.WINDOW, 6))
for va, off, nm in inst:
f.write(struct.pack("<IQ", va, off) + nm.encode()[:63].ljust(64, b"\0"))
t_start = time.time()
for dur, label, act in SCRIPT:
act(pad)
print(f"# {label} ({dur}s)", flush=True)
n = int(dur * hz)
for _ in range(n):
t = time.time()
f.write(struct.pack("<d", t - t_start))
f.write(struct.pack("<6f", *pad.vector()))
f.write(struct.pack("<32s", label.encode()[:32]))
for va, off, nm in inst:
f.write(w.read_off(off, gworld.WINDOW).ljust(gworld.WINDOW, b"\0"))
time.sleep(max(0, 1.0 / hz - (time.time() - t)))
pad.reset()
print(f"# wrote {out} ({os.path.getsize(out)/1e6:.1f} MB)", flush=True)
if __name__ == "__main__":
main()

27
tools/re-capture/fly_session.sh Executable file
View File

@@ -0,0 +1,27 @@
#!/usr/bin/env bash
# One background task: boot -> mission -> bind the transform -> fly the pilot,
# with periodic screenshots so the outcome has visual evidence and not just a log.
#
# It is one task on purpose: the display and the emulator both die on their own
# every few minutes in this container, so nothing may depend on surviving
# between tool calls.
set -u
export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy DISPLAY=:98
export PYTHONPATH=/sylph-home/.local/lib/python3.12/site-packages
SD="$(cd "$(dirname "$0")" && pwd)"
SECS="${1:-240}"
TAG="${2:-pilot}"
SHOTS=/sylph-home/re/shots
CFG=/tmp/nav-live.json
"$SD/launch_mission.sh" fly || { echo "BOOT FAILED"; exit 1; }
python3 "$SD/entities2.py" self 0x130 "$CFG" || { echo "BIND FAILED"; exit 1; }
echo "--- config: $(cat "$CFG")"
python3 "$SD/own_state.py" "$CFG" 3 "/tmp/$TAG-own.json"
( for i in $(seq 1 12); do sleep 25; screenshot "$SHOTS/$TAG-$i.png" >/dev/null 2>&1; done ) &
SHOTTER=$!
python3 "$SD/pilot.py" "$CFG" "$SECS"
kill $SHOTTER 2>/dev/null
screenshot "$SHOTS/$TAG-end.png" >/dev/null 2>&1
echo "SESSION DONE"

168
tools/re-capture/gmem.py Normal file
View File

@@ -0,0 +1,168 @@
#!/usr/bin/env python3
"""Read the *live* guest memory of a running Xenia Canary.
Canary backs the whole guest address space with a single shared-memory file,
`/dev/shm/xenia_memory_<pid-ish>`, so the guest's RAM is readable from the host
with no debugger, no emulator patch and no pause: open the file, seek, read.
The file is one flat image of Xenia's *physical* backing store; guest virtual
addresses map into it through Xenia's fixed table (memory.cc `map_info`).
`va_to_off()` implements that table, so callers work in guest VAs.
Sub-commands
find <pattern> search every allocated extent, print guest VAs
read <va> [len] hexdump guest memory
words <va> [n] dump n big-endian u32 / f32 pairs
`<pattern>` is a python literal-ish string: plain text, or `hex:0011aabb`.
Numbers accept 0x form. The scan uses SEEK_DATA so the ~4.6 GB of sparse holes
cost nothing.
"""
import os
import re
import sys
import struct
# (guest_va_lo, guest_va_hi_inclusive, file_offset_of_lo) — Xenia memory.cc.
MAP = [
(0x00000000, 0x3FFFFFFF, 0x00000000),
(0x40000000, 0x7EFFFFFF, 0x40000000),
(0x7F000000, 0x7F0FFFFF, 0x00000000),
(0x7F100000, 0x7FFFFFFF, 0x00100000),
(0x80000000, 0x8FFFFFFF, 0x80000000),
(0x90000000, 0x9FFFFFFF, 0x80000000),
(0xA0000000, 0xBFFFFFFF, 0x100000000),
(0xC0000000, 0xDFFFFFFF, 0x100000000),
(0xE0000000, 0xFFFFFFFF, 0x100000000),
]
def va_to_off(va):
for lo, hi, base in MAP:
if lo <= va <= hi:
return base + (va - lo)
raise ValueError(f"va {va:#x} outside the guest map")
def off_to_vas(off):
"""All guest VAs that alias this file offset (the map is many-to-one)."""
out = []
for lo, hi, base in MAP:
if base <= off <= base + (hi - lo):
out.append(lo + (off - base))
return out
def primary_va(off):
"""The most useful VA for an offset: physical 0xA0000000+ / xex 0x80000000+."""
vas = off_to_vas(off)
return vas[0] if vas else None
def mem_path():
# A snapshot (`cp --sparse=always /dev/shm/xenia_memory_* snap.bin`, ~2 s)
# reads identically and does not contend with the running emulator, which
# pegs every core under lavapipe. Point $GMEM_FILE at one to work offline.
env = os.environ.get("GMEM_FILE")
if env:
return env
if len(sys.argv) > 1 and sys.argv[1].startswith("/dev/shm/"):
return sys.argv.pop(1)
cands = [f"/dev/shm/{n}" for n in os.listdir("/dev/shm") if n.startswith("xenia_memory_")]
if not cands:
sys.exit("no /dev/shm/xenia_memory_* — is Canary running?")
if len(cands) > 1:
sys.exit(f"several memory files, pass one explicitly: {cands}")
return cands[0]
def extents(fd, size):
"""Yield (start, end) of the file's allocated (non-hole) ranges."""
pos = 0
while pos < size:
try:
data = os.lseek(fd, pos, os.SEEK_DATA)
except OSError:
return
try:
hole = os.lseek(fd, data, os.SEEK_HOLE)
except OSError:
hole = size
yield (data, hole)
pos = hole
def parse_pattern(s):
if s.startswith("hex:"):
return bytes.fromhex(s[4:])
return s.encode("latin-1")
def cmd_find(f, size, args):
pat = parse_pattern(args[0])
limit = int(args[1]) if len(args) > 1 else 64
hits = 0
CHUNK = 1 << 24
for start, end in extents(f.fileno(), size):
pos = start
carry = b""
carry_at = start
while pos < end:
f.seek(pos)
buf = f.read(min(CHUNK, end - pos))
if not buf:
break
blob = carry + buf
base = carry_at
for m in re.finditer(re.escape(pat), blob):
off = base + m.start()
va = primary_va(off)
print(f"{off:#013x} va {va:#010x}" if va is not None else f"{off:#013x} va ?")
hits += 1
if hits >= limit:
return
keep = len(pat) - 1
carry = blob[-keep:] if keep else b""
carry_at = base + len(blob) - len(carry)
pos += len(buf)
if hits == 0:
print("(no hits)", file=sys.stderr)
def cmd_read(f, size, args):
va = int(args[0], 0)
n = int(args[1], 0) if len(args) > 1 else 256
f.seek(va_to_off(va))
data = f.read(n)
for i in range(0, len(data), 16):
row = data[i : i + 16]
hexs = " ".join(f"{b:02x}" for b in row)
txt = "".join(chr(b) if 0x20 <= b < 0x7F else "." for b in row)
print(f"{va + i:08x} {hexs:<47} |{txt}|")
def cmd_words(f, size, args):
va = int(args[0], 0)
n = int(args[1], 0) if len(args) > 1 else 32
f.seek(va_to_off(va))
data = f.read(n * 4)
for i in range(0, len(data) - 3, 4):
(u,) = struct.unpack_from(">I", data, i)
(fl,) = struct.unpack_from(">f", data, i)
fs = f"{fl:.6g}" if -1e30 < fl < 1e30 else ""
print(f"{va + i:08x} +{i:04x} {u:#010x} {u:>12} {fs}")
def main():
path = mem_path()
if len(sys.argv) < 2:
sys.exit(__doc__)
cmd, args = sys.argv[1], sys.argv[2:]
size = os.path.getsize(path)
with open(path, "rb") as f:
{"find": cmd_find, "read": cmd_read, "words": cmd_words}[cmd](f, size, args)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,70 @@
#!/usr/bin/env bash
# Capture one tutorial's unit definitions: cold-boot Canary, walk
# title -> TUTORIAL -> the Nth entry, wait for the stage to load, snapshot RAM.
#
# One emulator per tutorial on purpose: backing out of a loaded mission via
# PAUSE -> BACK TO MENU wedged the emulator (log stops, CPU spins), while a
# cold boot here is ~15 s. The unit definitions are all parsed at stage load,
# so a single snapshot right after the load is everything this needs.
set -u
N="${1:?usage: grab_tutorial.sh <index 0..5> <outfile>}"
OUT="${2:?}"
export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy DISPLAY=:98
SD="$(cd "$(dirname "$0")" && pwd)"
RC="/home/fabi/RE - Project Sylpheed/sylpheed-reborn/tools/re-capture"
# The container entrypoint's Xvfb/openbox do NOT restart on their own, and a
# dead Xvfb leaves a <defunct> entry that still matches `pgrep` -- so test the
# display with xdpyinfo, never by process name.
ensure_display(){
if ! xdpyinfo -display "$DISPLAY" >/dev/null 2>&1; then
echo "[grab] $DISPLAY is down -> restarting Xvfb + openbox"
setsid nohup Xvfb "$DISPLAY" -screen 0 1280x720x24 -ac -nolisten tcp \
+extension GLX +extension RANDR </dev/null >/tmp/xvfb98.log 2>&1 &
for _ in $(seq 1 50); do xdpyinfo -display "$DISPLAY" >/dev/null 2>&1 && break; sleep 0.2; done
setsid nohup env DISPLAY="$DISPLAY" HOME=/sylph-home openbox </dev/null >/tmp/openbox98.log 2>&1 &
sleep 1
fi
xdpyinfo -display "$DISPLAY" >/dev/null 2>&1 || { echo "DISPLAY UNAVAILABLE"; exit 1; }
}
step(){ vgamepad dpad "$1"; sleep 0.2; vgamepad dpad center; sleep 0.6; }
shot(){ screenshot "$SD/$1" >/dev/null 2>&1; }
# Container PID 1 is `sleep infinity`, which never reaps, so every killed
# emulator stays as a <defunct> entry that `pgrep` still matches. Ask ps for the
# state and ignore anything zombie, or this always thinks one is running.
alive(){ ps -o pid=,stat= -C xenia_canary 2>/dev/null | awk '$2 !~ /^Z/ {print $1}'; }
pkill -x xenia_canary 2>/dev/null; sleep 2
[ -n "$(alive)" ] && { kill -9 $(alive) 2>/dev/null; sleep 2; }
rm -f /dev/shm/xenia_memory_* /dev/shm/xenia_code_cache_* 2>/dev/null
ensure_display
cd /sylph-home/re
setsid nohup run-canary --audio --apu=sdl --log_mask=13 \
--logged_profile_slot_0_xuid=E0300000EFBEA3D4 </dev/null >/dev/null 2>&1 &
sleep 5
"$RC/skip_intro.sh" 600 || { echo "BOOT FAILED"; exit 1; }
# The main menu is not input-ready for ~10 s after the title tap under lavapipe;
# d-pad presses before that are silently dropped, and the A then lands on NEW
# GAME instead of TUTORIAL. This wait is the whole fix.
sleep 14
shot "nav-$N-menu.png"
step down; step down # NEW GAME -> TUTORIAL
vgamepad tap A 250; sleep 8
shot "nav-$N-list.png"
i=0; while [ "$i" -lt "$N" ]; do step down; i=$((i+1)); done
shot "nav-$N-pick.png"
vgamepad tap A 250
# the stage load takes ~25-45 s under lavapipe; wait for the screen to settle
sleep 50
shot "nav-$N-loaded.png"
MEM=$(ls /dev/shm/xenia_memory_* 2>/dev/null | head -1)
[ -n "$MEM" ] || { echo "NO SHM"; exit 1; }
cp --sparse=always "$MEM" "$SD/$OUT" || exit 1
echo "SNAPSHOT $OUT ($(du -h "$SD/$OUT" | cut -f1))"

203
tools/re-capture/gworld.py Executable file
View File

@@ -0,0 +1,203 @@
#!/usr/bin/env python3
"""Live world-state reader: the running game's entities, straight out of guest RAM.
Builds on gmem.py (Canary backs the guest address space with
/dev/shm/xenia_memory_*, so it is an ordinary file). The difference here is that
this is meant to run *in a loop* while the game plays, not against a snapshot:
the memory file is opened once and re-read with pread, and the expensive
whole-RAM vtable scan is done rarely and cached.
Two classes matter (see docs/re/structures/unit-struct-runtime.md):
0x820af844 the parsed `.tbl` definition — one per unit type
0x820af030 the spawned entity instance — one per thing in the scene
This module finds instances and reads their live transform. The transform
offsets are NOT guessed: `find_transform` scans an instance for a 3x3 block of
floats that is orthonormal to 1e-3 with det = +1, which a rotation matrix is and
essentially nothing else is.
Sub-commands:
entities list live instances (name, address)
probe <seconds> [hz] [out] record every instance's raw bytes over time
orient report orthonormal 3x3 blocks per instance
"""
import os
import re
import struct
import sys
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import gmem # noqa: E402
INST_VTABLE = 0x820AF030
DEF_VTABLE = 0x820AF844
WINDOW = 0x600 # bytes of an instance we read; its real size is unknown
NAME_STR_OFF = 0x10
class World:
def __init__(self, path=None):
self.path = path or gmem.mem_path()
self.fd = os.open(self.path, os.O_RDONLY)
self.size = os.path.getsize(self.path)
self.instances = [] # [(va, file_off, name)]
# ---------------------------------------------------------------- io
def read(self, va, n):
return os.pread(self.fd, n, gmem.va_to_off(va))
def read_off(self, off, n):
return os.pread(self.fd, n, off)
def u32(self, va):
return struct.unpack(">I", self.read(va, 4))[0]
def cstr(self, va, n=96):
b = os.pread(self.fd, n, gmem.va_to_off(va)).split(b"\0")[0]
try:
return b.decode("ascii")
except UnicodeDecodeError:
return None
# ------------------------------------------------------------- scan
def scan_vtable(self, vt):
pat = struct.pack(">I", vt)
hits = []
for a, b in gmem.extents(self.fd, self.size):
blob = os.pread(self.fd, b - a, a)
start = 0
while True:
i = blob.find(pat, start)
if i < 0:
break
off = a + i
if off % 4 == 0:
hits.append(off)
start = i + 4
return sorted(set(hits))
def name_of(self, off):
"""The unit ID an object at `off` names, via its name-record pointer."""
raw = self.read_off(off + 4, 4)
if len(raw) < 4:
return None
(p,) = struct.unpack(">I", raw)
try:
return self.cstr(p + NAME_STR_OFF)
except ValueError:
return None
def refresh(self, vt=INST_VTABLE):
"""Re-scan for live instances. Costs ~1 s; call it rarely."""
out = []
for off in self.scan_vtable(vt):
nm = self.name_of(off)
if nm and nm.startswith("UN_"):
out.append((gmem.primary_va(off), off, nm))
self.instances = out
return out
# --------------------------------------------------------------- geometry
def floats(buf, off, n):
return struct.unpack_from(">" + "f" * n, buf, off)
def is_rotation(m, tol=2e-3):
"""m = 9 floats, row-major. Orthonormal rows + det +1?"""
r = [m[0:3], m[3:6], m[6:9]]
for row in r:
n2 = sum(c * c for c in row)
if abs(n2 - 1.0) > tol:
return False
for i in range(3):
for j in range(i + 1, 3):
if abs(sum(r[i][k] * r[j][k] for k in range(3))) > tol:
return False
det = (r[0][0] * (r[1][1] * r[2][2] - r[1][2] * r[2][1])
- r[0][1] * (r[1][0] * r[2][2] - r[1][2] * r[2][0])
+ r[0][2] * (r[1][0] * r[2][1] - r[1][1] * r[2][0]))
return abs(det - 1.0) < 1e-2
def find_rotations(buf, stride=4):
"""Offsets where 9 consecutive floats form a rotation matrix.
Also tries the 4-float-per-row (3x4 / 4x4 matrix) stride, which is how a
transform with a translation column is normally stored.
"""
out = []
for off in range(0, len(buf) - 36, stride):
m = floats(buf, off, 9)
if all(abs(c) <= 1.001 for c in m) and is_rotation(m):
out.append((off, "3x3", m))
for off in range(0, len(buf) - 48, stride):
rows = [floats(buf, off + 16 * i, 3) for i in range(3)]
m = rows[0] + rows[1] + rows[2]
if all(abs(c) <= 1.001 for c in m) and is_rotation(m):
out.append((off, "4x4", m))
return out
# ------------------------------------------------------------------ cmds
def cmd_entities(w, args):
t0 = time.time()
inst = w.refresh()
print(f"# {len(inst)} live instances (scan {time.time()-t0:.2f}s)")
from collections import Counter
c = Counter(n for _, _, n in inst)
for nm, k in c.most_common():
print(f" {k:4d} {nm}")
def cmd_orient(w, args):
inst = w.refresh()
want = args[0] if args else None
for va, off, nm in inst:
if want and want not in nm:
continue
buf = w.read_off(off, WINDOW)
rots = find_rotations(buf)
print(f"\n{va:#010x} {nm} -> {len(rots)} rotation-like block(s)")
for o, kind, m in rots[:6]:
print(f" +{o:#05x} {kind} "
+ " ".join(f"{v:+.3f}" for v in m))
def cmd_probe(w, args):
secs = float(args[0]) if args else 6.0
hz = float(args[1]) if len(args) > 1 else 5.0
out = args[2] if len(args) > 2 else "probe.bin"
inst = w.refresh()
print(f"# probing {len(inst)} instances for {secs}s at {hz}Hz -> {out}")
with open(out, "wb") as f:
f.write(struct.pack("<II", len(inst), WINDOW))
for va, off, nm in inst:
nb = nm.encode()[:63].ljust(64, b"\0")
f.write(struct.pack("<II", va, off) + nb)
n = int(secs * hz)
for i in range(n):
t = time.time()
f.write(struct.pack("<d", t))
for va, off, nm in inst:
f.write(w.read_off(off, WINDOW).ljust(WINDOW, b"\0"))
time.sleep(max(0, 1.0 / hz - (time.time() - t)))
print(f"# wrote {n} frames")
def main():
if len(sys.argv) < 2:
sys.exit(__doc__)
w = World()
cmd, args = sys.argv[1], sys.argv[2:]
{"entities": cmd_entities, "orient": cmd_orient, "probe": cmd_probe}[cmd](w, args)
if __name__ == "__main__":
main()

Some files were not shown because too many files have changed in this diff Show More