fabi/Syplheed-Reborn
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

re: solve the runtime Unit (craft/vessel) struct from live guest memory

Browse Source 
Craft stats were the last parked Route-B target: the Hangar exposes only a
weight class, and in menus the flight-model object is not instantiated at all.
It is instantiated in-mission, so this reads it there.

The definition class is discovered rather than assumed (unit_discover.py):
scanning for pointers to the disc ID strings and tallying the word behind each
pointer site picks out vtable 0x820af844 -- one object per unit ID, name-record
pointer at +0x04, string at +0x10, exactly the Weapon shape. The neighbouring
class 0x820af030 is the spawned entity, not the definition; the two are told
apart by one-object-per-ID and by 14/14 objects being byte-identical across two
snapshots 12 minutes and one firefight apart.

22 fields bind with zero contradictions on >=3 distinct values. Beyond that,
the Maneuver sub-record turns out to be laid out in schema declaration order,
4 bytes per field, base 0x9c with a two-slot gap after AA_Roll_Min -- 29
independently solved anchors fit two exact bases with no conflicts. That rule
pins five fields NO unit table ever values (PitchDragFactor, RollDragFactor,
DragFactorThreshold, ArterBurner_Acc, DecPitchFactor); PitchDrag == RollDrag ==
the disc's YawDrag on 18/18 units corroborates the interpolation.

Angles are float32 radians at runtime and degrees on disc.

Coverage is 18 of 110 units: unlike weapons, unit definitions are instantiated
per stage, so it grows by visiting missions. The AI-behaviour tail of Maneuver
is explicitly NOT resolved and marked tentative in the CSV.

Captured from Xenia Canary (BASIC CONTROLS tutorial + Stage 02 from save 01).
This commit is contained in:
Claude (auto-RE)
2026-07-29 17:27:53 +00:00
parent 6a41525c41
commit 2fa4c33d1a
8 changed files with 2326 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/re/INDEX.md
View File

@@ -22,6 +22,7 @@ Promote to a prose `structures/…md` file when a format needs behavioural notes
| XBG7 mesh | 🟡/❔ | `sylpheed-formats/src/mesh.rs` + `tests/mesh_disc.rs` ([xbg7](structures/xbg7-mesh.md)) | weapons/props: declaration-driven variable stride (36 models), GPU-confirmed. **Stage containers: 5662 sub-models across 22 stages** via content-anchored grouped pools (`stage_models`). Quantized hero bodies (DeltaSaber `f004`) still declined |
| Capital-ship part placement | 🟡 | `sylpheed-formats/src/ship.rs` (static) + [runtime capture](ship-placement-runtime-capture.md) | hull placement static-exact; external parts approximate statically. **Runtime capture** (Canary F10 → VS-constant WorldView) gives ground truth — validated on `e106` destroyer; not yet baked into the viewer |
| Weapon fields defaulted on disc | ✅ | [runtime struct](structures/weapon-struct-runtime.md) · [DATA SHEET route](weapon-datasheet-runtime.md) | **Solved.** Canary maps guest RAM into `/dev/shm`, so the parsed `Weapon`/`Shell` objects are readable live; their layout is solved against disc ground truth (zero contradictions over 100+ records). All 126 weapons, exact numbers, no story progress needed — [4 393 values](captures/weapon-runtime-fields.csv) the disc does not carry. Supersedes the letter-bucket limit of the DATA SHEET route, which now serves as the independent cross-check |
| Unit (craft/vessel) fields defaulted on disc | ✅/🟡 | [runtime struct](structures/unit-struct-runtime.md) | The parsed `unit\UN_*.tbl` definition object, vtable `0x820af844`, ≥`0x380` bytes, one per unit — **discovered, not assumed** (`unit_discover.py`), and distinguished from the spawned-entity class `0x820af030` by being one-per-ID and byte-constant across snapshots. 22 fields ✅; the `Maneuver` block is **schema declaration order, 4 bytes/field, base `0x9c` with a two-slot gap after `AA_Roll_Min`** (29 anchors, 0 conflicts), which also pins 5 fields *no* disc record ever values. Angles are **radians at runtime, degrees on disc**. Unlike weapons, unit definitions are instantiated **per stage**, so coverage (18/110) grows by visiting missions — [values](captures/unit-runtime-fields.csv) |
| UI screen layout (`.rat`) | ✅/🟡 | [ui-rat-layout](structures/ui-rat-layout.md) | One pak per UI screen; each RATC = one (context × language) build; every `<name>.t32` sprite has a `<name>.rat` **layout record** (BE u32; 1280×720 design space; scale/tint/X/Y, keyframes for animated elements, `opt ` link to the focused state). **The tutorial PAUSE menu and the title main menu both rebuild pixel-accurately from the disc.** `loop1.rat` (screen-level draw order) not yet decoded |
## Runtime / dynamic-capture technique

1531
docs/re/captures/unit-runtime-fields.csv Normal file
View File

File diff suppressed because it is too large Load Diff

222
docs/re/structures/unit-struct-runtime.md Normal file
View File

@@ -0,0 +1,222 @@
# Runtime `Unit` struct (craft / vessel definitions) — read from live guest memory
**Confidence: ✅ CONFIRMED** for the 22 fields marked ✅ below (each binding is
reproduced by 10–18 independent disc records, on ≥3 distinct values, with
**zero** contradictions), plus the Maneuver block's declaration-order layout
(29 anchors, two bases, no conflicts). 🟡 PROBABLE for the fields interpolated
between confirmed anchors. 🟡/❔ for the thin single-value bindings, which are
listed but must not be trusted yet.
Captured 2026-07-29 from Xenia Canary running the retail disc in the sylph-re
container: the **BASIC CONTROLS tutorial** (4 units) and **Stage 02 "Declaration
of War"** loaded from save slot 01 (14 units). 18 of the disc's 110 units.
## Why this exists
Craft stats were the one Route-B target the previous two passes could not
reach. The Hangar exposes only `Gross Weight` (a class, not a number), and
[the menu route](../weapon-datasheet-runtime.md) has no surface at all for the
flight model. From the memory side, menus were equally useless: **only the
player craft's name string is resident there — the definition objects do not
exist until a mission loads.**
They do exist in-mission. This documents their layout and the values the disc
leaves defaulted.
## Finding the class (discovered, not assumed)
[`tools/re-capture/unit_discover.py`](../../../tools/re-capture/unit_discover.py)
takes no vtable as input. It locates every disc unit-ID string in a memory
snapshot, finds every aligned word pointing at `string_va - d` for a range of
`d`, and tallies the word at `pointer_site - k` across **distinct** unit IDs.
One `(d, k, word)` combination wins by a wide margin:
| δ (name-record → string) | ID pointer at | word | distinct units |
|---|---|---|---|
| `0x10` | object `+0x04` | `0x820af030` | 4 |
| `0x10` | object `+0x04` | `0x820af844` | 4 |
— i.e. exactly the `Weapon` shape: the object holds a name-record pointer at
`+0x04`, and the ID string sits at `name_record + 0x10`.
The two vtables are **two different things**, and telling them apart matters:
| vtable | what it is | evidence |
|---|---|---|
| `0x820af030` | **spawned entity instance** — live state | 12 objects for 4 IDs; the same ID appears many times (one per box in the scene); irregular spacing |
| `0x820af844` | **parsed definition** — the `.tbl` | exactly one object per distinct unit ID; minimum spacing `0x380` |
Only `0x820af844` is used. It is the runtime image of the `.tbl`, and its
objects are **byte-identical across two snapshots taken ~12 minutes apart with
combat in between** (14/14 objects, 0 differing bytes) — definition data, not
state.
One `.tbl` → **one** object. A unit table is several sub-records (`Generic`,
`Maneuver`, `Shield`, `Explosion`, `Mass`, `Effect`, `SE`, `Turret_00N`), and
they are all flattened into that single ≥`0x380`-byte object — unlike weapons,
where `Weapon` and `Shell` are separate arrays.
## Solving the layout
Same discipline as
[`weapon_runtime.py`](../../../tools/re-capture/weapon_runtime.py): score every
`(field, byte-offset, encoding)` triple against the disc records and accept a
binding only on **zero contradictions**, requiring ≥3 distinct values so a
field whose samples are all one number cannot match any offset holding that
constant.
```bash
python3 tools/re-capture/unit_runtime.py unit_tokens.txt snap_a.bin snap_b.bin --csv
```
Snapshots are **unioned** — each mission instantiates only the units in its own
stage, so coverage grows by visiting stages. `cp --sparse=always` a copy of
`/dev/shm/xenia_memory_*` first (~2 s); the running emulator pegs every core
under lavapipe and makes repeated live reads flaky.
### Encoding note — angles are radians at runtime
Every `AV_*` / `AA_*` / `*Bank*` / `Turn_AngularVelocity` field is stored as
**float32 radians**, while the disc writes **degrees**. The solver needed a
`rad` encoding (`degrees(f32)`) to bind them at all; 15 units agree on
`AV_PitchPlus_Max` alone. A reimplementation reading the `.tbl` must convert.
## Confirmed layout
✅ = ≥10 disc records agree on ≥3 distinct values, zero contradict.
| offset | enc | field | agree | distinct |
|---|---|---|---:|---:|
| `+0x030` | f32 | `Size_X` | 18 | 13 |
| `+0x038` | f32 | `Size_Z` | 16 | 13 |
| `+0x040` | f32 | `Color_R` | 18 | 5 |
| `+0x044` | f32 | `Color_G` | 16 | 6 |
| `+0x048` | f32 | `Color_B` | 14 | 5 |
| `+0x050` | f32 | `Size_Radius` | 11 | 10 |
| `+0x054` | f32 | `HP` | 17 | 10 |
| `+0x074` | f32 | `ResistanceToOptics` | 10 | 3 |
| `+0x08c` | f32 | `ScorePoint` | 18 | 9 |
| `+0x0a0` | f32 | `MaximumVelocity` | 16 | 8 |
| `+0x0a4` | f32 | `CruisingVelocity` | 14 | 7 |
| `+0x0a8` | f32 | `Acceleration` | 15 | 5 |
| `+0x0ac` | f32 | `Deceleration` | 14 | 5 |
| `+0x0b0` | rad | `AV_PitchPlus_Max` | 15 | 7 |
| `+0x0f8` | f32 | `SideThrustVelocity_Max` | 12 | 3 |
| `+0x238` | f32 | `MaxValue` (Shield) | 12 | 6 |
| `+0x244` | f32 | `ChargeSpeed` (Shield) | 11 | 6 |
| `+0x270` | f32 | `DestroyMotionTime` | 16 | 7 |
| `+0x2a0` | f32 | `RadarRange` | 15 | 8 |
| `+0x2a4` | f32 | `FCSRange` | 10 | 6 |
| `+0x2b4` | f32 | `AttackVesselPoint` | 11 | 8 |
| `+0x2bc` | f32 | `DefencePoint` | 10 | 7 |
`Size_Y +0x034`, `MassScore +0x094`, `HQRatio +0x058`, `ShieldRatio +0x05c`,
`ThrusterRatio +0x060`, `ResistanceToShell +0x078`,
`ResistanceToExplosion +0x07c`, `ResistanceToPlayer +0x080`,
`ResistanceParalyze +0x084`, `BridgeCount +0x070` (i32),
`MinimumVelocity +0x09c`, `DryMass +0x274`, `GrossMass +0x278`,
`LowerHPThresholdRatio +0x298`, `AttackCraftPoint +0x2b8` bind with zero
contradictions on fewer records or fewer distinct values — 🟡 PROBABLE. The
full solver output is in
[`captures/unit-runtime-fields.csv`](../captures/unit-runtime-fields.csv)
(1 440 values, `conf` column).
## The Maneuver block is laid out in schema declaration order
This is the strongest structural result and it is independent of any single
field's agreement count.
[`schema_order.py`](../../../tools/re-capture/schema_order.py) merges the
`Maneuver` field-name order from **all 113 unit tables** by topological sort
over their pairwise "k[i] precedes k[i+1]" constraints. The merge is acyclic and
**every one of the 113 tables is a subsequence of the merged 102-field order** —
so that order is the schema's.
Against it, the solved offsets fall into two exact runs:
| declaration indices | offset rule | anchors that fit |
|---|---|---|
| 0 … 20 (`MinimumVelocity` … `AA_Roll_Min`) | `0x09c + 4·i` | 21 / 21 |
| 21 … 32 (`SideThrustVelocity_Max` … `AccPitchFactor`) | `0x0a4 + 4·i` | 8 / 8 |
One 4-byte slot per field, with a **two-slot gap after `AA_Roll_Min`**
(`0x0f0`–`0x0f4`, purpose unknown). 29 independently-derived anchors, zero
conflicts, across a 0x9c–0x125 span.
### Fields NO disc record ever values — 🟡 PROBABLE
Five `Maneuver` fields are declared by the schema but left at their default by
*every* one of the 110 unit tables, so no amount of disc analysis can ever
reach them. The declaration-order rule pins them between confirmed anchors
(`YawDragFactor +0x104` … `ArterBurner_Vc +0x114`, and
`ReverseThrust_Vc +0x118` … `ReverseThrust_Acc +0x120`):
| offset | field | craft (`f00*`, `e0*`) | capital ships (`*1**`, `e2*`) | inert (`SchlosBase`, `Box`) |
|---|---|---:|---:|---:|
| `+0x108` | `PitchDragFactor` | 3 | 2 | 0 |
| `+0x10c` | `RollDragFactor` | 3 | 2 | 0 |
| `+0x110` | `DragFactorThreshold` | 0.5 | 0.5 | 0 |
| `+0x11c` | `ArterBurner_Acc` | 2 | 2 | 0 |
| `+0x128` | `DecPitchFactor` | 30 | 30 | 0 |
Corroboration beyond the interpolation, checked over all 18 units:
* `PitchDragFactor == RollDragFactor == YawDragFactor` holds **18/18** — and
`YawDragFactor` is *disc-supplied* (3.0 for craft, 2.0 for warships), so two
interpolated offsets reproduce a known number, per unit, every time.
* `DecPitchFactor == AccPitchFactor` holds **17/18**. The exception is
`UN_e015_ADAN_Puppy` (`AccPitchFactor` 1, `DecPitchFactor` 0.5) — which
defaults *both* on disc, so it is two independent fields that happen to be
set equal elsewhere, not a broken binding.
`UN_e015_ADAN_Puppy` also breaks the craft/warship bucketing above for
`DecPitchFactor` (0.5, not 30); the per-unit values are in the CSV.
The tail of the `Maneuver` block (the AI-behaviour fields — `SideRoll_*`,
`BarrelRoll_*`, `TurnAttack_*`, `HoldPosition_*`, `Slalom_*`, `Through_*`,
`SolidCutoff_*`, and the `AxisMode` / `AB_*` sub-block) is **NOT resolved**.
Those fields are declared by only a handful of tables and almost always with a
single distinct value, so the solver's bindings there are coincidences: it
placed `Slalom_CutoffRatio` at `+0x00c` and `TurnAttack_DoubleTimeMin` at
`+0x110`, both of which the declaration-order rule contradicts. They are marked
`tentative` in the CSV. **NEEDS-HUMAN / needs more coverage** — more stages
would give those fields distinct values and settle it.
## The player craft, `UN_f001_TCAF_DeltaSaber_T_Player`
30 of its fields are defaulted on disc. Notable recovered values:
| field | value | note |
|---|---|---|
| `Size_Radius` | 10 | ✅ |
| `FCSRange` | 500000 | ✅ — same as `RadarRange` |
| `ChargeSpeed` (shield) | 25 | ✅ |
| `ResistanceToOptics` | 1 | ✅ |
| `HQRatio` / `ShieldRatio` / `ThrusterRatio` | 1 / 1 / 1 | 🟡 |
| `ResistanceToShell` / `ResistanceToExplosion` | 1 / 1 | 🟡 |
| `DryMass` | 100 | 🟡 (`GrossMass` 250 is on disc) |
| `LowerHPThresholdRatio` | 0.3 | 🟡 |
| `ChargeDelay_Break` | 10 | 🟡 |
| `MassScore` | 0 | 🟡 |
Every AI-behaviour field the solver bound reads **0** for the player craft,
which is the expected shape (the player is not AI-driven) — but see the caveat
above: those offsets are not settled, so treat the zeros as consistent, not
proven.
The `…Ratio` family that the Route-B target list parked is **1.0 for almost
every unit**, with real exceptions that only the runtime shows:
`UN_e105_ADAN_Cruiser` `HQRatio` = 0.2, `UN_bf001_TCAF_SchlosBase` and
`UN_e106_ADAN_Destroyer` `ThrusterRatio` = 0.2,
`UN_n001_TTRL_Box` `ShieldRatio` = 0.3.
## Coverage and how to extend it
18 of 110 units. Unlike weapons — where one snapshot held all 126 — **unit
definitions are instantiated per stage**, so coverage is bounded by the stages
reachable from the save (slot 01 is at 5 %, Stage 02). Each further mission or
tutorial adds its own units; `unit_runtime.py` unions any number of snapshots
and re-solves, and more units directly promote the 🟡 bindings to ✅ by adding
distinct values.
Stages captured so far: `Ttrl` (BASIC CONTROLS), Stage 02.

14
tools/re-capture/README.md
View File

@@ -18,3 +18,17 @@ Canary + lavapipe, headless. They assume the container helpers `screenshot`,
auto-repeat (two rows per press).
Findings produced with these: [`docs/re/weapon-datasheet-runtime.md`](../../docs/re/weapon-datasheet-runtime.md).
## Guest-memory tools (no screenshots)
| Script | What it does |
|---|---|
| `gmem.py` | Read the live guest address space out of `/dev/shm/xenia_memory_*` (Xenia's backing file), addressed by guest VA. `find` / `read` / `words`. |
| `weapon_runtime.py` | Solve the `Weapon`/`Shell` struct layouts against the disc records and read the fields the disc defaults. |
| `unit_discover.py` | Find *which* runtime class carries a set of ID strings, assuming no vtable: tallies the word at `pointer_site - k` across distinct IDs. |
| `unit_runtime.py` | Same solver for the `unit\UN_*.tbl` definition objects (vtable `0x820af844`). Unions several snapshots — unit definitions are per-stage. |
| `schema_order.py` | Merge a sub-record's field-declaration order across all tables (topological sort); the layout check that pins fields no table ever values. |
| `order_check.py` | Test `offset = base + 4*index` for one table's sub-record against solver output. |
Snapshot first — `cp --sparse=always /dev/shm/xenia_memory_* snap.bin` (~2 s) — and
point `$GMEM_FILE` at the copy; the running emulator pegs every core under lavapipe.

95
tools/re-capture/order_check.py Normal file
View File

@@ -0,0 +1,95 @@
#!/usr/bin/env python3
"""Test the hypothesis that a unit sub-record's runtime fields are laid out in
*schema declaration order*, one 4-byte slot each.
The IDXD string pool lists a sub-record's field names in declaration order,
including the ones this .tbl leaves at their default (they appear as a bare key
with no preceding value). If the runtime struct is `base + 4*index` over that
list, then the offsets the solver found independently must all satisfy it.
That is a much stronger check than any single field's agreement count: one base
offset has to explain dozens of independently-derived anchors at once.
Usage: order_check.py <raw-token-dump> <tbl-hash> <SubRecord> <solved.txt>
"""
import re
import sys
NUMERIC = re.compile(r"^[-+]?[0-9]*\.?[0-9]+[fF]?$|^0x[0-9a-fA-F]+$")
VALUE_WORDS = {"Yes", "No", "YES", "NO", "On", "Off", "ON", "OFF"}
def section(dump, tbl, sub, subs):
"""Ordered token list of one sub-record of one .tbl."""
toks, on = [], False
for line in open(dump):
if line.startswith("RAW "):
on = line.split()[1] == tbl
continue
if on and line.startswith("T "):
toks.append(line[2:].rstrip("\n"))
starts = [i for i, t in enumerate(toks) if t in subs or t.lstrip("(") in subs]
for n, i in enumerate(starts):
name = toks[i].lstrip("(")
if name != sub:
continue
end = starts[n + 1] if n + 1 < len(starts) else len(toks)
return toks[i + 1 : end]
return []
def keys_in_order(toks):
"""Field names, in declaration order: every token that is not a value.
In the numeric sub-records every value is a number literal, so a token that
is not numeric and not a Yes/No word is a key.
"""
out = []
for t in toks:
if NUMERIC.match(t) or t in VALUE_WORDS:
continue
if t not in out:
out.append(t)
return out
def solved_offsets(path):
out = {}
for line in open(path):
m = re.match(r"\s*(0x[0-9a-f]+)\s+(\w+)\s+(\w+)\s+(\d+)\s+(\d+)\s+(\d+)\s*$", line)
if m:
off, enc, field, agree, dist, bad = m.groups()
out[field] = (int(off, 16), enc, int(agree), int(dist), int(bad))
return out
def main():
dump, tbl, sub, solvedf = sys.argv[1:5]
subs = {"Generic", "Maneuver", "Shield", "Explosion", "Mass", "Effect", "SE"}
keys = keys_in_order(section(dump, tbl, sub, subs))
solved = solved_offsets(solvedf)
anchors = [(i, k, solved[k]) for i, k in enumerate(keys) if k in solved]
if not anchors:
sys.exit(f"no solved field lands in {sub}")
# base that the most anchors agree on
votes = {}
for i, k, (off, *_) in anchors:
votes.setdefault(off - 4 * i, []).append(k)
base, winners = max(votes.items(), key=lambda kv: len(kv[1]))
print(f"# {sub}: {len(keys)} declared fields, {len(anchors)} solved anchors")
print(f"# best base = {base:#x} -> {len(winners)}/{len(anchors)} anchors fit "
f"offset = base + 4*index\n")
for i, k in enumerate(keys):
off = base + 4 * i
if k in solved:
so, enc, agree, dist, bad = solved[k]
mark = "fits" if so == off else f"CONFLICT solver={so:#x}"
print(f" +{off:#05x} [{i:3d}] {k:<34} {enc} agree={agree:<3} "
f"dist={dist:<3} {mark}")
else:
print(f" +{off:#05x} [{i:3d}] {k:<34} — (never valued on disc)")
if __name__ == "__main__":
main()

80
tools/re-capture/schema_order.py Normal file
View File

@@ -0,0 +1,80 @@
#!/usr/bin/env python3
"""Recover a sub-record's FULL schema field order by merging every unit .tbl.
Each .tbl's string pool lists only the fields that .tbl declares, in schema
order. Merging them is a topological sort over the pairwise "k[i] precedes
k[i+1]" constraints of all 110 tables: any single table is a subsequence of the
schema, so the union order is the schema order — provided the constraints are
acyclic, which is itself the check that the premise holds.
Usage: schema_order.py <raw-token-dump> <SubRecord> [solved.txt]
"""
import sys
from collections import defaultdict
sys.path.insert(0, ".")
from order_check import section, keys_in_order, solved_offsets # noqa: E402
SUBS = {"Generic", "Maneuver", "Shield", "Explosion", "Mass", "Effect", "SE"}
def merged_order(dump, sub):
tbls = [l.split()[1] for l in open(dump) if l.startswith("RAW ")]
succ, indeg, nodes = defaultdict(set), defaultdict(int), []
seqs = []
for t in tbls:
ks = keys_in_order(section(dump, t, sub, SUBS))
if ks:
seqs.append(ks)
for k in ks:
if k not in nodes:
nodes.append(k)
for a, b in zip(ks, ks[1:]):
if b not in succ[a]:
succ[a].add(b)
indeg[b] += 1
# Kahn, breaking ties by first-seen order so the result is deterministic
ready = [n for n in nodes if indeg[n] == 0]
out, cycles = [], False
while ready:
ready.sort(key=nodes.index)
n = ready.pop(0)
out.append(n)
for m in sorted(succ[n], key=nodes.index):
indeg[m] -= 1
if indeg[m] == 0:
ready.append(m)
if len(out) != len(nodes):
cycles = True
out += [n for n in nodes if n not in out]
return out, seqs, cycles
def main():
dump, sub = sys.argv[1], sys.argv[2]
order, seqs, cycles = merged_order(dump, sub)
print(f"# {sub}: {len(order)} fields merged from {len(seqs)} tables"
f"{' !! CYCLE — order is not consistent' if cycles else ''}")
# every table must be a subsequence of the merged order — the real check
pos = {k: i for i, k in enumerate(order)}
bad = [s for s in seqs if [pos[k] for k in s] != sorted(pos[k] for k in s)]
print(f"# tables that are NOT a subsequence of the merged order: {len(bad)}")
if len(sys.argv) > 3:
solved = solved_offsets(sys.argv[3])
anchors = [(i, k, solved[k]) for i, k in enumerate(order) if k in solved]
votes = defaultdict(list)
for i, k, (off, *_) in anchors:
votes[off - 4 * i].append((i, k))
print(f"# {len(anchors)} solved anchors; base votes:")
for b, ks in sorted(votes.items(), key=lambda kv: -len(kv[1])):
idx = sorted(i for i, _ in ks)
print(f"# base {b:#07x}: {len(ks):3d} anchors, indices {idx[0]}..{idx[-1]}")
for i, k in enumerate(order):
s = solved.get(k)
note = f"solver={s[0]:#x} {s[1]} agree={s[2]} dist={s[3]}" if s else ""
print(f" [{i:3d}] {k:<34} {note}")
if __name__ == "__main__":
main()

123
tools/re-capture/unit_discover.py Normal file
View File

@@ -0,0 +1,123 @@
#!/usr/bin/env python3
"""Discover the runtime class that carries the parsed `UN_*` unit definitions.
No vtable VA is assumed. Procedure:
1. locate every disc unit-ID string in the snapshot;
2. find every 4-byte-aligned word in RAM that points at (string_va - d) for a
few plausible name-record deltas;
3. for each such pointer site P and each small back-offset k, read the word at
P-k and tally it across *distinct* unit IDs.
A word that appears at the same (d, k) for many different units is that class's
vtable pointer, and k is the ID-pointer offset inside the object.
"""
import os
import re
import struct
import sys
from collections import defaultdict
sys.path.insert(0, "/home/fabi/RE - Project Sylpheed/sylpheed-reborn/tools/re-capture")
import gmem # noqa: E402
DELTAS = [0x00, 0x10, 0x08, 0x04, 0x0C, 0x14, 0x18, 0x20]
BACK = list(range(0, 0x60, 4))
def load_ids(path):
ids = []
for line in open(path):
if line.startswith("REC "):
p = line.split()
if p[3] == "Generic" and p[4].startswith("UN_"):
ids.append(p[4])
return sorted(set(ids))
def scan_strings(f, size, ids):
"""{id: [va,...]} — every NUL-terminated occurrence of each id string."""
pats = {i: (i.encode() + b"\0") for i in ids}
rx = re.compile(b"|".join(re.escape(p) for p in pats.values()))
out = defaultdict(list)
for a, b in gmem.extents(f.fileno(), size):
f.seek(a)
blob = f.read(b - a)
for m in rx.finditer(blob):
off = a + m.start()
va = gmem.primary_va(off)
if va is not None:
out[m.group(0)[:-1].decode()].append(va)
return out
def scan_pointers(f, size, targets):
"""{target_va: [site_va,...]} for every aligned word equal to a target."""
want = {struct.pack(">I", t): t for t in targets}
rx = re.compile(b"|".join(re.escape(k) for k in want))
out = defaultdict(list)
for a, b in gmem.extents(f.fileno(), size):
f.seek(a)
blob = f.read(b - a)
for m in rx.finditer(blob):
off = a + m.start()
if off % 4:
continue
va = gmem.primary_va(off)
if va is not None:
out[want[m.group(0)]].append(va)
return out
def main():
ids = load_ids(sys.argv[1])
path = gmem.mem_path()
size = os.path.getsize(path)
with open(path, "rb") as f:
strs = scan_strings(f, size, ids)
print(f"# {len(ids)} disc unit IDs, {len(strs)} found in RAM "
f"({sum(len(v) for v in strs.values())} occurrences)", file=sys.stderr)
# every (id, string_va - delta) we want pointers to
targets, owner = set(), defaultdict(set)
for uid, vas in strs.items():
for va in vas:
for d in DELTAS:
t = va - d
targets.add(t)
owner[t].add((uid, d))
ptrs = scan_pointers(f, size, targets)
print(f"# {len(ptrs)} of {len(targets)} candidate targets are pointed at",
file=sys.stderr)
# tally (delta, back-offset, word) over distinct unit IDs
tally = defaultdict(set)
sites = defaultdict(list)
for tgt, plist in ptrs.items():
for uid, d in owner[tgt]:
for p in plist:
for k in BACK:
base = p - k
try:
f.seek(gmem.va_to_off(base))
except ValueError:
continue
w = f.read(4)
if len(w) < 4:
continue
(vt,) = struct.unpack(">I", w)
if 0x82000000 <= vt < 0x83000000:
tally[(d, k, vt)].add(uid)
sites[(d, k, vt)].append((base, uid))
rank = sorted(tally.items(), key=lambda kv: -len(kv[1]))
print(f"{'delta':>6} {'idoff':>6} {'word':>10} {'#ids':>5}")
for (d, k, vt), s in rank[:25]:
print(f"{d:#6x} {k:#6x} {vt:#010x} {len(s):5d} e.g. {sorted(s)[:2]}")
if rank:
best = rank[0][0]
print("\n# object bases for the top candidate (first 20):", file=sys.stderr)
for base, uid in sorted(set(sites[best]))[:20]:
print(f"# {base:#010x} {uid}", file=sys.stderr)
if __name__ == "__main__":
main()

260
tools/re-capture/unit_runtime.py Normal file
View File

@@ -0,0 +1,260 @@
#!/usr/bin/env python3
"""Solve the runtime `Unit` (craft/vessel definition) struct layout against the
disc's `unit\\UN_*.tbl` records, then read the fields the disc leaves defaulted.
Same method as weapon_runtime.py, with two differences forced by the data:
* the runtime class is *discovered*, not assumed — see unit_discover.py; the
definition objects carry vtable 0x820af844 and hold a pointer to their name
record at +0x04 (string at name+0x10), exactly like `Weapon`;
* a unit `.tbl` is one object made of several sub-records (Generic, Maneuver,
Shield, Explosion, Mass, Effect, SE, Turret_*), all flattened into ONE
runtime object, so the disc side is merged per unit ID.
Only the *definition* objects are used. The other class found in RAM
(0x820af030) is the spawned-entity instance — same ID string, live state, not
definition data — and is deliberately ignored.
Usage: unit_runtime.py <tokens.txt> <snapshot...> (snapshots are unioned)
"""
import math
import os
import re
import struct
import sys
from collections import defaultdict
sys.path.insert(0, "/home/fabi/RE - Project Sylpheed/sylpheed-reborn/tools/re-capture")
import gmem # noqa: E402
DEF_VTABLE = 0x820AF844
STRIDE = 0x380
SUBRECS = ("Generic", "Maneuver", "Shield", "Explosion", "Mass", "Effect", "SE")
# `Maneuver` fields that EVERY unit table leaves at its default, so the solver
# can never bind them: there is no disc value anywhere to score against. Their
# offsets come from the declaration-order rule (see schema_order.py), which 29
# solved anchors confirm, and each of these sits *between* two of those anchors
# -- `YawDragFactor +0x104` .. `ArterBurner_Vc +0x114` and
# `ReverseThrust_Vc +0x118` .. `ReverseThrust_Acc +0x120`. Reported separately
# and marked `interpolated`, never mixed into the solved set.
INTERPOLATED = [
(0x108, "PitchDragFactor", "f32"),
(0x10C, "RollDragFactor", "f32"),
(0x110, "DragFactorThreshold", "f32"),
(0x11C, "ArterBurner_Acc", "f32"),
(0x128, "DecPitchFactor", "f32"),
]
# ---------------------------------------------------------------- disc side
def read_tokens(path):
"""{unit_id: {field: value}} merged over the tbl's sub-records.
A field name that two sub-records of the same unit define with different
values is ambiguous *within* the object, so it is dropped rather than
silently resolved.
"""
per_hash = defaultdict(lambda: {"id": None, "f": defaultdict(set)})
cur = None
for line in open(path):
if line.startswith("REC "):
_, h, _, ty, rid = line.split()
cur = per_hash[h]
if ty == "Generic" and rid.startswith("UN_"):
cur["id"] = rid
elif line.startswith("F ") and cur is not None:
_, k, v = line.rstrip("\n").split(" ", 2)
cur["f"][k].add(v)
out, ambiguous = {}, {}
for rec in per_hash.values():
if not rec["id"]:
continue
fields, bad = {}, []
for k, vs in rec["f"].items():
if len(vs) == 1:
fields[k] = next(iter(vs))
else:
bad.append(k)
out[rec["id"]] = fields
if bad:
ambiguous[rec["id"]] = sorted(bad)
return out, ambiguous
# ------------------------------------------------------------- runtime side
def scan_vtable(f, size, vt):
pat = struct.pack(">I", vt)
hits = []
for a, b in gmem.extents(f.fileno(), size):
f.seek(a)
blob = f.read(b - a)
for m in re.finditer(re.escape(pat), blob):
off = a + m.start()
if off % 4 == 0:
hits.append(off)
return sorted(set(hits))
def runtime_objects(paths):
"""{id: raw} unioned over snapshots; identical IDs must agree byte-for-byte."""
objs, conflicts = {}, []
for path in paths:
size = os.path.getsize(path)
with open(path, "rb") as f:
for off in scan_vtable(f, size, DEF_VTABLE):
f.seek(off)
raw = f.read(STRIDE)
(p,) = struct.unpack_from(">I", raw, 4)
try:
f.seek(gmem.va_to_off(p + 0x10))
except ValueError:
continue
nm = f.read(96).split(b"\0")[0]
try:
nm = nm.decode("ascii")
except UnicodeDecodeError:
continue
if not nm.startswith("UN_"):
continue
if nm in objs and objs[nm] != raw:
conflicts.append((nm, os.path.basename(path)))
objs[nm] = raw
return objs, conflicts
# ------------------------------------------------------------------- solver
ENC = {
"f32": lambda raw, o: struct.unpack_from(">f", raw, o)[0],
"u32": lambda raw, o: float(struct.unpack_from(">I", raw, o)[0]),
"i32": lambda raw, o: float(struct.unpack_from(">i", raw, o)[0]),
"rad": lambda raw, o: math.degrees(struct.unpack_from(">f", raw, o)[0]),
}
def near(a, b):
if a == b:
return True
return abs(a - b) <= 2e-4 * max(abs(a), abs(b), 1e-6)
def solve(disc, run):
numeric = {}
for rid, fields in disc.items():
if rid not in run:
continue
for k, v in fields.items():
try:
numeric.setdefault(k, {})[rid] = float(v.rstrip("fF"))
except ValueError:
pass
solved = {}
for field, samples in numeric.items():
if len(samples) < 2:
continue
cands = []
# +0x00 is the vtable and +0x04 the name-record pointer; no field lives
# there. They are excluded explicitly because a huge-exponent pointer
# word read as f32 is a denormal, which the tolerance test happily
# "matches" against any disc value of 0.0.
for off in range(8, STRIDE - 3, 4):
for enc, fn in ENC.items():
agree, bad = 0, []
for rid, want in samples.items():
got = fn(run[rid], off)
if math.isfinite(got) and near(got, want):
agree += 1
else:
bad.append((rid, want, got))
if agree >= 2:
cands.append((len(bad), -agree, off, enc, agree, bad))
if not cands:
continue
cands.sort()
nbad, _, off, enc, agree, bad = cands[0]
if len(bad) > 0.2 * len(samples):
continue
good = {rid for rid in samples} - {b[0] for b in bad}
distinct = len({round(samples[r], 6) for r in good})
solved[field] = (off, enc, agree, bad, distinct)
# two fields cannot share one byte offset -- and the offset alone is the
# identity, not (offset, encoding): the same word read as f32 and as i32 is
# still one field.
best_at = {}
for field, (off, enc, agree, bad, distinct) in solved.items():
score = (distinct, agree, -len(bad))
if off not in best_at or score > best_at[off][1]:
best_at[off] = (field, score)
winners = {f for f, _ in best_at.values()}
return {f: v for f, v in solved.items() if f in winners}, numeric
def main():
tokens = sys.argv[1]
snaps = [a for a in sys.argv[2:] if not a.startswith("--")]
disc, ambiguous = read_tokens(tokens)
run, conflicts = runtime_objects(snaps)
matched = sorted(set(disc) & set(run))
print(f"# disc units: {len(disc)} runtime objects: {len(run)} matched: {len(matched)}")
if conflicts:
print(f"# !! objects that differ between snapshots: {conflicts}")
unknown = sorted(set(run) - set(disc))
if unknown:
print(f"# runtime IDs with no disc record: {unknown}")
solved, numeric = solve(disc, run)
print(f"# numeric disc fields on matched units: {len(numeric)} solved: {len(solved)}\n")
print(f"{'offset':>8} {'enc':>4} {'field':<32} {'agree':>5} {'dist':>4} {'bad':>3}")
for field, (off, enc, agree, bad, distinct) in sorted(solved.items(), key=lambda kv: kv[1][0]):
print(f"{off:#8x} {enc:>4} {field:<32} {agree:5d} {distinct:4d} {len(bad):3d}")
if "--csv" in sys.argv:
import csv
w = csv.writer(open("unit-runtime-fields.csv", "w", newline=""))
w.writerow(["unit", "field", "offset", "enc", "value", "source", "conf"])
for rid in matched:
for field, (off, enc, agree, bad, distinct) in sorted(
solved.items(), key=lambda kv: kv[1][0]
):
conf = "confirmed" if (not bad and agree >= 10 and distinct >= 3) else "tentative"
src = "disc" if field in disc[rid] else "defaulted-on-disc"
w.writerow([rid, field, f"{off:#05x}", enc,
f"{ENC[enc](run[rid], off):g}", src, conf])
for off, field, enc in INTERPOLATED:
w.writerow([rid, field, f"{off:#05x}", enc,
f"{ENC[enc](run[rid], off):g}",
"never-valued-on-disc", "interpolated"])
for arg in sys.argv:
if arg.startswith("--unit="):
rid = arg.split("=", 1)[1]
print(f"\n# every solved field of {rid}")
for field, (off, enc, agree, bad, distinct) in sorted(
solved.items(), key=lambda kv: kv[1][0]
):
conf = "OK " if (not bad and agree >= 10 and distinct >= 3) else "thin"
src = "disc" if field in disc.get(rid, {}) else "DEFAULTED"
print(f" +{off:#05x} {enc} {conf} {field:<32} "
f"{ENC[enc](run[rid], off):>14g} {src}")
if "--values" in sys.argv:
print("\n# defaulted-on-disc values read from the runtime objects")
for field, (off, enc, agree, bad, distinct) in sorted(solved.items(), key=lambda kv: kv[1][0]):
if bad:
continue
miss = [(r, ENC[enc](run[r], off)) for r in matched if field not in disc[r]]
if miss:
print(f"\n{field} (+{off:#x}, {enc}):")
for r, v in miss:
print(f" {r:<48} {v:g}")
if __name__ == "__main__":
main()