fabi/Syplheed-Reborn
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

re: solve the runtime Unit (craft/vessel) struct from live guest memory

Browse Source 
Craft stats were the last parked Route-B target: the Hangar exposes only a
weight class, and in menus the flight-model object is not instantiated at all.
It is instantiated in-mission, so this reads it there.

The definition class is discovered rather than assumed (unit_discover.py):
scanning for pointers to the disc ID strings and tallying the word behind each
pointer site picks out vtable 0x820af844 -- one object per unit ID, name-record
pointer at +0x04, string at +0x10, exactly the Weapon shape. The neighbouring
class 0x820af030 is the spawned entity, not the definition; the two are told
apart by one-object-per-ID and by 14/14 objects being byte-identical across two
snapshots 12 minutes and one firefight apart.

22 fields bind with zero contradictions on >=3 distinct values. Beyond that,
the Maneuver sub-record turns out to be laid out in schema declaration order,
4 bytes per field, base 0x9c with a two-slot gap after AA_Roll_Min -- 29
independently solved anchors fit two exact bases with no conflicts. That rule
pins five fields NO unit table ever values (PitchDragFactor, RollDragFactor,
DragFactorThreshold, ArterBurner_Acc, DecPitchFactor); PitchDrag == RollDrag ==
the disc's YawDrag on 18/18 units corroborates the interpolation.

Angles are float32 radians at runtime and degrees on disc.

Coverage is 18 of 110 units: unlike weapons, unit definitions are instantiated
per stage, so it grows by visiting missions. The AI-behaviour tail of Maneuver
is explicitly NOT resolved and marked tentative in the CSV.

Captured from Xenia Canary (BASIC CONTROLS tutorial + Stage 02 from save 01).
This commit is contained in:
Claude (auto-RE)
2026-07-29 17:27:53 +00:00
parent 6a41525c41
commit 2fa4c33d1a
8 changed files with 2326 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
tools/re-capture/README.md
View File

@@ -18,3 +18,17 @@ Canary + lavapipe, headless. They assume the container helpers `screenshot`,
auto-repeat (two rows per press).
Findings produced with these: [`docs/re/weapon-datasheet-runtime.md`](../../docs/re/weapon-datasheet-runtime.md).
## Guest-memory tools (no screenshots)
| Script | What it does |
|---|---|
| `gmem.py` | Read the live guest address space out of `/dev/shm/xenia_memory_*` (Xenia's backing file), addressed by guest VA. `find` / `read` / `words`. |
| `weapon_runtime.py` | Solve the `Weapon`/`Shell` struct layouts against the disc records and read the fields the disc defaults. |
| `unit_discover.py` | Find *which* runtime class carries a set of ID strings, assuming no vtable: tallies the word at `pointer_site - k` across distinct IDs. |
| `unit_runtime.py` | Same solver for the `unit\UN_*.tbl` definition objects (vtable `0x820af844`). Unions several snapshots — unit definitions are per-stage. |
| `schema_order.py` | Merge a sub-record's field-declaration order across all tables (topological sort); the layout check that pins fields no table ever values. |
| `order_check.py` | Test `offset = base + 4*index` for one table's sub-record against solver output. |
Snapshot first — `cp --sparse=always /dev/shm/xenia_memory_* snap.bin` (~2 s) — and
point `$GMEM_FILE` at the copy; the running emulator pegs every core under lavapipe.

95
tools/re-capture/order_check.py Normal file
View File

@@ -0,0 +1,95 @@
#!/usr/bin/env python3
"""Test the hypothesis that a unit sub-record's runtime fields are laid out in
*schema declaration order*, one 4-byte slot each.
The IDXD string pool lists a sub-record's field names in declaration order,
including the ones this .tbl leaves at their default (they appear as a bare key
with no preceding value). If the runtime struct is `base + 4*index` over that
list, then the offsets the solver found independently must all satisfy it.
That is a much stronger check than any single field's agreement count: one base
offset has to explain dozens of independently-derived anchors at once.
Usage: order_check.py <raw-token-dump> <tbl-hash> <SubRecord> <solved.txt>
"""
import re
import sys
NUMERIC = re.compile(r"^[-+]?[0-9]*\.?[0-9]+[fF]?$|^0x[0-9a-fA-F]+$")
VALUE_WORDS = {"Yes", "No", "YES", "NO", "On", "Off", "ON", "OFF"}
def section(dump, tbl, sub, subs):
"""Ordered token list of one sub-record of one .tbl."""
toks, on = [], False
for line in open(dump):
if line.startswith("RAW "):
on = line.split()[1] == tbl
continue
if on and line.startswith("T "):
toks.append(line[2:].rstrip("\n"))
starts = [i for i, t in enumerate(toks) if t in subs or t.lstrip("(") in subs]
for n, i in enumerate(starts):
name = toks[i].lstrip("(")
if name != sub:
continue
end = starts[n + 1] if n + 1 < len(starts) else len(toks)
return toks[i + 1 : end]
return []
def keys_in_order(toks):
"""Field names, in declaration order: every token that is not a value.
In the numeric sub-records every value is a number literal, so a token that
is not numeric and not a Yes/No word is a key.
"""
out = []
for t in toks:
if NUMERIC.match(t) or t in VALUE_WORDS:
continue
if t not in out:
out.append(t)
return out
def solved_offsets(path):
out = {}
for line in open(path):
m = re.match(r"\s*(0x[0-9a-f]+)\s+(\w+)\s+(\w+)\s+(\d+)\s+(\d+)\s+(\d+)\s*$", line)
if m:
off, enc, field, agree, dist, bad = m.groups()
out[field] = (int(off, 16), enc, int(agree), int(dist), int(bad))
return out
def main():
dump, tbl, sub, solvedf = sys.argv[1:5]
subs = {"Generic", "Maneuver", "Shield", "Explosion", "Mass", "Effect", "SE"}
keys = keys_in_order(section(dump, tbl, sub, subs))
solved = solved_offsets(solvedf)
anchors = [(i, k, solved[k]) for i, k in enumerate(keys) if k in solved]
if not anchors:
sys.exit(f"no solved field lands in {sub}")
# base that the most anchors agree on
votes = {}
for i, k, (off, *_) in anchors:
votes.setdefault(off - 4 * i, []).append(k)
base, winners = max(votes.items(), key=lambda kv: len(kv[1]))
print(f"# {sub}: {len(keys)} declared fields, {len(anchors)} solved anchors")
print(f"# best base = {base:#x} -> {len(winners)}/{len(anchors)} anchors fit "
f"offset = base + 4*index\n")
for i, k in enumerate(keys):
off = base + 4 * i
if k in solved:
so, enc, agree, dist, bad = solved[k]
mark = "fits" if so == off else f"CONFLICT solver={so:#x}"
print(f" +{off:#05x} [{i:3d}] {k:<34} {enc} agree={agree:<3} "
f"dist={dist:<3} {mark}")
else:
print(f" +{off:#05x} [{i:3d}] {k:<34} — (never valued on disc)")
if __name__ == "__main__":
main()

80
tools/re-capture/schema_order.py Normal file
View File

@@ -0,0 +1,80 @@
#!/usr/bin/env python3
"""Recover a sub-record's FULL schema field order by merging every unit .tbl.
Each .tbl's string pool lists only the fields that .tbl declares, in schema
order. Merging them is a topological sort over the pairwise "k[i] precedes
k[i+1]" constraints of all 110 tables: any single table is a subsequence of the
schema, so the union order is the schema order — provided the constraints are
acyclic, which is itself the check that the premise holds.
Usage: schema_order.py <raw-token-dump> <SubRecord> [solved.txt]
"""
import sys
from collections import defaultdict
sys.path.insert(0, ".")
from order_check import section, keys_in_order, solved_offsets # noqa: E402
SUBS = {"Generic", "Maneuver", "Shield", "Explosion", "Mass", "Effect", "SE"}
def merged_order(dump, sub):
tbls = [l.split()[1] for l in open(dump) if l.startswith("RAW ")]
succ, indeg, nodes = defaultdict(set), defaultdict(int), []
seqs = []
for t in tbls:
ks = keys_in_order(section(dump, t, sub, SUBS))
if ks:
seqs.append(ks)
for k in ks:
if k not in nodes:
nodes.append(k)
for a, b in zip(ks, ks[1:]):
if b not in succ[a]:
succ[a].add(b)
indeg[b] += 1
# Kahn, breaking ties by first-seen order so the result is deterministic
ready = [n for n in nodes if indeg[n] == 0]
out, cycles = [], False
while ready:
ready.sort(key=nodes.index)
n = ready.pop(0)
out.append(n)
for m in sorted(succ[n], key=nodes.index):
indeg[m] -= 1
if indeg[m] == 0:
ready.append(m)
if len(out) != len(nodes):
cycles = True
out += [n for n in nodes if n not in out]
return out, seqs, cycles
def main():
dump, sub = sys.argv[1], sys.argv[2]
order, seqs, cycles = merged_order(dump, sub)
print(f"# {sub}: {len(order)} fields merged from {len(seqs)} tables"
f"{' !! CYCLE — order is not consistent' if cycles else ''}")
# every table must be a subsequence of the merged order — the real check
pos = {k: i for i, k in enumerate(order)}
bad = [s for s in seqs if [pos[k] for k in s] != sorted(pos[k] for k in s)]
print(f"# tables that are NOT a subsequence of the merged order: {len(bad)}")
if len(sys.argv) > 3:
solved = solved_offsets(sys.argv[3])
anchors = [(i, k, solved[k]) for i, k in enumerate(order) if k in solved]
votes = defaultdict(list)
for i, k, (off, *_) in anchors:
votes[off - 4 * i].append((i, k))
print(f"# {len(anchors)} solved anchors; base votes:")
for b, ks in sorted(votes.items(), key=lambda kv: -len(kv[1])):
idx = sorted(i for i, _ in ks)
print(f"# base {b:#07x}: {len(ks):3d} anchors, indices {idx[0]}..{idx[-1]}")
for i, k in enumerate(order):
s = solved.get(k)
note = f"solver={s[0]:#x} {s[1]} agree={s[2]} dist={s[3]}" if s else ""
print(f" [{i:3d}] {k:<34} {note}")
if __name__ == "__main__":
main()

123
tools/re-capture/unit_discover.py Normal file
View File

@@ -0,0 +1,123 @@
#!/usr/bin/env python3
"""Discover the runtime class that carries the parsed `UN_*` unit definitions.
No vtable VA is assumed. Procedure:
1. locate every disc unit-ID string in the snapshot;
2. find every 4-byte-aligned word in RAM that points at (string_va - d) for a
few plausible name-record deltas;
3. for each such pointer site P and each small back-offset k, read the word at
P-k and tally it across *distinct* unit IDs.
A word that appears at the same (d, k) for many different units is that class's
vtable pointer, and k is the ID-pointer offset inside the object.
"""
import os
import re
import struct
import sys
from collections import defaultdict
sys.path.insert(0, "/home/fabi/RE - Project Sylpheed/sylpheed-reborn/tools/re-capture")
import gmem # noqa: E402
DELTAS = [0x00, 0x10, 0x08, 0x04, 0x0C, 0x14, 0x18, 0x20]
BACK = list(range(0, 0x60, 4))
def load_ids(path):
ids = []
for line in open(path):
if line.startswith("REC "):
p = line.split()
if p[3] == "Generic" and p[4].startswith("UN_"):
ids.append(p[4])
return sorted(set(ids))
def scan_strings(f, size, ids):
"""{id: [va,...]} — every NUL-terminated occurrence of each id string."""
pats = {i: (i.encode() + b"\0") for i in ids}
rx = re.compile(b"|".join(re.escape(p) for p in pats.values()))
out = defaultdict(list)
for a, b in gmem.extents(f.fileno(), size):
f.seek(a)
blob = f.read(b - a)
for m in rx.finditer(blob):
off = a + m.start()
va = gmem.primary_va(off)
if va is not None:
out[m.group(0)[:-1].decode()].append(va)
return out
def scan_pointers(f, size, targets):
"""{target_va: [site_va,...]} for every aligned word equal to a target."""
want = {struct.pack(">I", t): t for t in targets}
rx = re.compile(b"|".join(re.escape(k) for k in want))
out = defaultdict(list)
for a, b in gmem.extents(f.fileno(), size):
f.seek(a)
blob = f.read(b - a)
for m in rx.finditer(blob):
off = a + m.start()
if off % 4:
continue
va = gmem.primary_va(off)
if va is not None:
out[want[m.group(0)]].append(va)
return out
def main():
ids = load_ids(sys.argv[1])
path = gmem.mem_path()
size = os.path.getsize(path)
with open(path, "rb") as f:
strs = scan_strings(f, size, ids)
print(f"# {len(ids)} disc unit IDs, {len(strs)} found in RAM "
f"({sum(len(v) for v in strs.values())} occurrences)", file=sys.stderr)
# every (id, string_va - delta) we want pointers to
targets, owner = set(), defaultdict(set)
for uid, vas in strs.items():
for va in vas:
for d in DELTAS:
t = va - d
targets.add(t)
owner[t].add((uid, d))
ptrs = scan_pointers(f, size, targets)
print(f"# {len(ptrs)} of {len(targets)} candidate targets are pointed at",
file=sys.stderr)
# tally (delta, back-offset, word) over distinct unit IDs
tally = defaultdict(set)
sites = defaultdict(list)
for tgt, plist in ptrs.items():
for uid, d in owner[tgt]:
for p in plist:
for k in BACK:
base = p - k
try:
f.seek(gmem.va_to_off(base))
except ValueError:
continue
w = f.read(4)
if len(w) < 4:
continue
(vt,) = struct.unpack(">I", w)
if 0x82000000 <= vt < 0x83000000:
tally[(d, k, vt)].add(uid)
sites[(d, k, vt)].append((base, uid))
rank = sorted(tally.items(), key=lambda kv: -len(kv[1]))
print(f"{'delta':>6} {'idoff':>6} {'word':>10} {'#ids':>5}")
for (d, k, vt), s in rank[:25]:
print(f"{d:#6x} {k:#6x} {vt:#010x} {len(s):5d} e.g. {sorted(s)[:2]}")
if rank:
best = rank[0][0]
print("\n# object bases for the top candidate (first 20):", file=sys.stderr)
for base, uid in sorted(set(sites[best]))[:20]:
print(f"# {base:#010x} {uid}", file=sys.stderr)
if __name__ == "__main__":
main()

260
tools/re-capture/unit_runtime.py Normal file
View File

@@ -0,0 +1,260 @@
#!/usr/bin/env python3
"""Solve the runtime `Unit` (craft/vessel definition) struct layout against the
disc's `unit\\UN_*.tbl` records, then read the fields the disc leaves defaulted.
Same method as weapon_runtime.py, with two differences forced by the data:
* the runtime class is *discovered*, not assumed — see unit_discover.py; the
definition objects carry vtable 0x820af844 and hold a pointer to their name
record at +0x04 (string at name+0x10), exactly like `Weapon`;
* a unit `.tbl` is one object made of several sub-records (Generic, Maneuver,
Shield, Explosion, Mass, Effect, SE, Turret_*), all flattened into ONE
runtime object, so the disc side is merged per unit ID.
Only the *definition* objects are used. The other class found in RAM
(0x820af030) is the spawned-entity instance — same ID string, live state, not
definition data — and is deliberately ignored.
Usage: unit_runtime.py <tokens.txt> <snapshot...> (snapshots are unioned)
"""
import math
import os
import re
import struct
import sys
from collections import defaultdict
sys.path.insert(0, "/home/fabi/RE - Project Sylpheed/sylpheed-reborn/tools/re-capture")
import gmem # noqa: E402
DEF_VTABLE = 0x820AF844
STRIDE = 0x380
SUBRECS = ("Generic", "Maneuver", "Shield", "Explosion", "Mass", "Effect", "SE")
# `Maneuver` fields that EVERY unit table leaves at its default, so the solver
# can never bind them: there is no disc value anywhere to score against. Their
# offsets come from the declaration-order rule (see schema_order.py), which 29
# solved anchors confirm, and each of these sits *between* two of those anchors
# -- `YawDragFactor +0x104` .. `ArterBurner_Vc +0x114` and
# `ReverseThrust_Vc +0x118` .. `ReverseThrust_Acc +0x120`. Reported separately
# and marked `interpolated`, never mixed into the solved set.
INTERPOLATED = [
(0x108, "PitchDragFactor", "f32"),
(0x10C, "RollDragFactor", "f32"),
(0x110, "DragFactorThreshold", "f32"),
(0x11C, "ArterBurner_Acc", "f32"),
(0x128, "DecPitchFactor", "f32"),
]
# ---------------------------------------------------------------- disc side
def read_tokens(path):
"""{unit_id: {field: value}} merged over the tbl's sub-records.
A field name that two sub-records of the same unit define with different
values is ambiguous *within* the object, so it is dropped rather than
silently resolved.
"""
per_hash = defaultdict(lambda: {"id": None, "f": defaultdict(set)})
cur = None
for line in open(path):
if line.startswith("REC "):
_, h, _, ty, rid = line.split()
cur = per_hash[h]
if ty == "Generic" and rid.startswith("UN_"):
cur["id"] = rid
elif line.startswith("F ") and cur is not None:
_, k, v = line.rstrip("\n").split(" ", 2)
cur["f"][k].add(v)
out, ambiguous = {}, {}
for rec in per_hash.values():
if not rec["id"]:
continue
fields, bad = {}, []
for k, vs in rec["f"].items():
if len(vs) == 1:
fields[k] = next(iter(vs))
else:
bad.append(k)
out[rec["id"]] = fields
if bad:
ambiguous[rec["id"]] = sorted(bad)
return out, ambiguous
# ------------------------------------------------------------- runtime side
def scan_vtable(f, size, vt):
pat = struct.pack(">I", vt)
hits = []
for a, b in gmem.extents(f.fileno(), size):
f.seek(a)
blob = f.read(b - a)
for m in re.finditer(re.escape(pat), blob):
off = a + m.start()
if off % 4 == 0:
hits.append(off)
return sorted(set(hits))
def runtime_objects(paths):
"""{id: raw} unioned over snapshots; identical IDs must agree byte-for-byte."""
objs, conflicts = {}, []
for path in paths:
size = os.path.getsize(path)
with open(path, "rb") as f:
for off in scan_vtable(f, size, DEF_VTABLE):
f.seek(off)
raw = f.read(STRIDE)
(p,) = struct.unpack_from(">I", raw, 4)
try:
f.seek(gmem.va_to_off(p + 0x10))
except ValueError:
continue
nm = f.read(96).split(b"\0")[0]
try:
nm = nm.decode("ascii")
except UnicodeDecodeError:
continue
if not nm.startswith("UN_"):
continue
if nm in objs and objs[nm] != raw:
conflicts.append((nm, os.path.basename(path)))
objs[nm] = raw
return objs, conflicts
# ------------------------------------------------------------------- solver
ENC = {
"f32": lambda raw, o: struct.unpack_from(">f", raw, o)[0],
"u32": lambda raw, o: float(struct.unpack_from(">I", raw, o)[0]),
"i32": lambda raw, o: float(struct.unpack_from(">i", raw, o)[0]),
"rad": lambda raw, o: math.degrees(struct.unpack_from(">f", raw, o)[0]),
}
def near(a, b):
if a == b:
return True
return abs(a - b) <= 2e-4 * max(abs(a), abs(b), 1e-6)
def solve(disc, run):
numeric = {}
for rid, fields in disc.items():
if rid not in run:
continue
for k, v in fields.items():
try:
numeric.setdefault(k, {})[rid] = float(v.rstrip("fF"))
except ValueError:
pass
solved = {}
for field, samples in numeric.items():
if len(samples) < 2:
continue
cands = []
# +0x00 is the vtable and +0x04 the name-record pointer; no field lives
# there. They are excluded explicitly because a huge-exponent pointer
# word read as f32 is a denormal, which the tolerance test happily
# "matches" against any disc value of 0.0.
for off in range(8, STRIDE - 3, 4):
for enc, fn in ENC.items():
agree, bad = 0, []
for rid, want in samples.items():
got = fn(run[rid], off)
if math.isfinite(got) and near(got, want):
agree += 1
else:
bad.append((rid, want, got))
if agree >= 2:
cands.append((len(bad), -agree, off, enc, agree, bad))
if not cands:
continue
cands.sort()
nbad, _, off, enc, agree, bad = cands[0]
if len(bad) > 0.2 * len(samples):
continue
good = {rid for rid in samples} - {b[0] for b in bad}
distinct = len({round(samples[r], 6) for r in good})
solved[field] = (off, enc, agree, bad, distinct)
# two fields cannot share one byte offset -- and the offset alone is the
# identity, not (offset, encoding): the same word read as f32 and as i32 is
# still one field.
best_at = {}
for field, (off, enc, agree, bad, distinct) in solved.items():
score = (distinct, agree, -len(bad))
if off not in best_at or score > best_at[off][1]:
best_at[off] = (field, score)
winners = {f for f, _ in best_at.values()}
return {f: v for f, v in solved.items() if f in winners}, numeric
def main():
tokens = sys.argv[1]
snaps = [a for a in sys.argv[2:] if not a.startswith("--")]
disc, ambiguous = read_tokens(tokens)
run, conflicts = runtime_objects(snaps)
matched = sorted(set(disc) & set(run))
print(f"# disc units: {len(disc)} runtime objects: {len(run)} matched: {len(matched)}")
if conflicts:
print(f"# !! objects that differ between snapshots: {conflicts}")
unknown = sorted(set(run) - set(disc))
if unknown:
print(f"# runtime IDs with no disc record: {unknown}")
solved, numeric = solve(disc, run)
print(f"# numeric disc fields on matched units: {len(numeric)} solved: {len(solved)}\n")
print(f"{'offset':>8} {'enc':>4} {'field':<32} {'agree':>5} {'dist':>4} {'bad':>3}")
for field, (off, enc, agree, bad, distinct) in sorted(solved.items(), key=lambda kv: kv[1][0]):
print(f"{off:#8x} {enc:>4} {field:<32} {agree:5d} {distinct:4d} {len(bad):3d}")
if "--csv" in sys.argv:
import csv
w = csv.writer(open("unit-runtime-fields.csv", "w", newline=""))
w.writerow(["unit", "field", "offset", "enc", "value", "source", "conf"])
for rid in matched:
for field, (off, enc, agree, bad, distinct) in sorted(
solved.items(), key=lambda kv: kv[1][0]
):
conf = "confirmed" if (not bad and agree >= 10 and distinct >= 3) else "tentative"
src = "disc" if field in disc[rid] else "defaulted-on-disc"
w.writerow([rid, field, f"{off:#05x}", enc,
f"{ENC[enc](run[rid], off):g}", src, conf])
for off, field, enc in INTERPOLATED:
w.writerow([rid, field, f"{off:#05x}", enc,
f"{ENC[enc](run[rid], off):g}",
"never-valued-on-disc", "interpolated"])
for arg in sys.argv:
if arg.startswith("--unit="):
rid = arg.split("=", 1)[1]
print(f"\n# every solved field of {rid}")
for field, (off, enc, agree, bad, distinct) in sorted(
solved.items(), key=lambda kv: kv[1][0]
):
conf = "OK " if (not bad and agree >= 10 and distinct >= 3) else "thin"
src = "disc" if field in disc.get(rid, {}) else "DEFAULTED"
print(f" +{off:#05x} {enc} {conf} {field:<32} "
f"{ENC[enc](run[rid], off):>14g} {src}")
if "--values" in sys.argv:
print("\n# defaulted-on-disc values read from the runtime objects")
for field, (off, enc, agree, bad, distinct) in sorted(solved.items(), key=lambda kv: kv[1][0]):
if bad:
continue
miss = [(r, ENC[enc](run[r], off)) for r in matched if field not in disc[r]]
if miss:
print(f"\n{field} (+{off:#x}, {enc}):")
for r, v in miss:
print(f" {r:<48} {v:g}")
if __name__ == "__main__":
main()