re: widen unit coverage to all six tutorials, and prove the fields are run-invariant

Captures the five remaining tutorials (grab_tutorial.sh, one cold boot each)
and re-solves over seven snapshots from seven separate emulator runs. Coverage
18 -> 21 units, confirmed fields 22 -> 27 (Size_Y, MassScore, MinimumVelocity,
AV_PitchPlus_Min, AV_PitchMinus_Min join the zero-contradiction set).

The multi-run union exposed something the single-run check could not: the same
unit's object is NOT byte-identical between runs. unit_runtime.py --crosscheck
pins down why -- exactly 15 words differ, 13 of them holding guest heap
pointers, and none of them is a solved or interpolated field offset. So every
value reported is run-invariant, which is a stronger statement than the
within-run identity check that came before it.

The two non-pointer stragglers are a real caveat, now documented rather than
smoothed over: +0x2c8 reads 8000.0 for UN_f001_TCAF_DeltaSaber_T_Ttrl in two
tutorials and 10000.0 in a third, with a 0/1 flag at +0x2d0. A couple of words
in the object are set per stage, so it is mostly but not entirely the parsed
table. Unidentified, marked NEEDS-HUMAN.

Coverage honesty: all six tutorials together add only 3 units the missions do
not already have -- they reuse one training box, one drone and the player
craft. Further coverage needs story progress, not more tutorials.

Two navigation facts encoded in grab_tutorial.sh, both learned by breaking
them: the main menu is not input-ready for ~10 s after the title tap and early
d-pad presses are dropped (which sends the A to NEW GAME); and NEW GAME is not
a shortcut to Stage 01 -- it gates on DIFFICULTY then plays the prologue. A
NEW GAME excursion to the READY ROOM leaves game01/savedata byte-identical,
verified by diff against a backup.
This commit is contained in:
2026-07-29 18:22:47 +00:00
parent 2196ac112d
commit 1ee2d5e406
6 changed files with 577 additions and 130 deletions

View File

@@ -22,7 +22,7 @@ Promote to a prose `structures/…md` file when a format needs behavioural notes
| XBG7 mesh | 🟡/❔ | `sylpheed-formats/src/mesh.rs` + `tests/mesh_disc.rs` ([xbg7](structures/xbg7-mesh.md)) | weapons/props: declaration-driven variable stride (36 models), GPU-confirmed. **Stage containers: 5662 sub-models across 22 stages** via content-anchored grouped pools (`stage_models`). Quantized hero bodies (DeltaSaber `f004`) still declined |
| Capital-ship part placement | 🟡 | `sylpheed-formats/src/ship.rs` (static) + [runtime capture](ship-placement-runtime-capture.md) | hull placement static-exact; external parts approximate statically. **Runtime capture** (Canary F10 → VS-constant WorldView) gives ground truth — validated on `e106` destroyer; not yet baked into the viewer |
| Weapon fields defaulted on disc | ✅ | [runtime struct](structures/weapon-struct-runtime.md) · [DATA SHEET route](weapon-datasheet-runtime.md) | **Solved.** Canary maps guest RAM into `/dev/shm`, so the parsed `Weapon`/`Shell` objects are readable live; their layout is solved against disc ground truth (zero contradictions over 100+ records). All 126 weapons, exact numbers, no story progress needed — [4 393 values](captures/weapon-runtime-fields.csv) the disc does not carry. Supersedes the letter-bucket limit of the DATA SHEET route, which now serves as the independent cross-check |
| Unit (craft/vessel) fields defaulted on disc | ✅/🟡 | [runtime struct](structures/unit-struct-runtime.md) | The parsed `unit\UN_*.tbl` definition object, vtable `0x820af844`, ≥`0x380` bytes, one per unit — **discovered, not assumed** (`unit_discover.py`), and distinguished from the spawned-entity class `0x820af030` by being one-per-ID and byte-constant across snapshots. 22 fields ✅; the `Maneuver` block is **schema declaration order, 4 bytes/field, base `0x9c` with a two-slot gap after `AA_Roll_Min`** (29 anchors, 0 conflicts), which also pins 5 fields *no* disc record ever values. Angles are **radians at runtime, degrees on disc**. Unlike weapons, unit definitions are instantiated **per stage**, so coverage (18/110) grows by visiting missions — [values](captures/unit-runtime-fields.csv) |
| Unit (craft/vessel) fields defaulted on disc | ✅/🟡 | [runtime struct](structures/unit-struct-runtime.md) | The parsed `unit\UN_*.tbl` definition object, vtable `0x820af844`, ≥`0x380` bytes, one per unit — **discovered, not assumed** (`unit_discover.py`), and distinguished from the spawned-entity class `0x820af030` by being one-per-ID and byte-constant within a run. Across runs only pointer words move — `--crosscheck` proves **no reported field offset is run-dependent** (two words, `+0x2c8`/`+0x2d0`, are stage-dependent and remain unidentified). 27 fields ✅ (21 units, 7 runs); the `Maneuver` block is **schema declaration order, 4 bytes/field, base `0x9c` with a two-slot gap after `AA_Roll_Min`** (29 anchors, 0 conflicts), which also pins 5 fields *no* disc record ever values. Angles are **radians at runtime, degrees on disc**. Unlike weapons, unit definitions are instantiated **per stage**, so coverage (21/110) grows by visiting missions — [values](captures/unit-runtime-fields.csv) |
| UI screen layout (`.rat`) | ✅/🟡 | [ui-rat-layout](structures/ui-rat-layout.md) | One pak per UI screen; each RATC = one (context × language) build; every `<name>.t32` sprite has a `<name>.rat` **layout record** (BE u32; 1280×720 design space; scale/tint/X/Y, keyframes for animated elements, `opt ` link to the focused state). **The tutorial PAUSE menu and the title main menu both rebuild pixel-accurately from the disc.** `loop1.rat` (screen-level draw order) not yet decoded |
## Runtime / dynamic-capture technique

File diff suppressed because it is too large Load Diff

View File

@@ -1,15 +1,16 @@
# Runtime `Unit` struct (craft / vessel definitions) — read from live guest memory
**Confidence: ✅ CONFIRMED** for the 22 fields marked ✅ below (each binding is
reproduced by 1018 independent disc records, on ≥3 distinct values, with
**Confidence: ✅ CONFIRMED** for the 27 fields marked ✅ below (each binding is
reproduced by 1021 independent disc records, on ≥3 distinct values, with
**zero** contradictions), plus the Maneuver block's declaration-order layout
(29 anchors, two bases, no conflicts). 🟡 PROBABLE for the fields interpolated
between confirmed anchors. 🟡/❔ for the thin single-value bindings, which are
listed but must not be trusted yet.
Captured 2026-07-29 from Xenia Canary running the retail disc in the sylph-re
container: the **BASIC CONTROLS tutorial** (4 units) and **Stage 02 "Declaration
of War"** loaded from save slot 01 (14 units). 18 of the disc's 110 units.
container: **all six tutorials** and **Stage 02 "Declaration of War"** loaded
from save slot 01. 21 of the disc's 110 units, over 7 snapshots from 7 separate
emulator runs.
## Why this exists
@@ -46,10 +47,23 @@ The two vtables are **two different things**, and telling them apart matters:
| `0x820af030` | **spawned entity instance** — live state | 12 objects for 4 IDs; the same ID appears many times (one per box in the scene); irregular spacing |
| `0x820af844` | **parsed definition** — the `.tbl` | exactly one object per distinct unit ID; minimum spacing `0x380` |
Only `0x820af844` is used. It is the runtime image of the `.tbl`, and its
objects are **byte-identical across two snapshots taken ~12 minutes apart with
combat in between** (14/14 objects, 0 differing bytes) — definition data, not
state.
Only `0x820af844` is used. It is the runtime image of the `.tbl`:
* **Within one run** it is byte-identical across two snapshots taken ~12 minutes
apart with combat in between (14/14 objects, 0 differing bytes) — definition
data, not live state.
* **Across runs** the same unit is *not* byte-identical, and that had to be
explained rather than waved away. `unit_runtime.py --crosscheck` compares
every unit that appears in more than one snapshot (5 of them, over 7 runs):
exactly **15 words differ**, and 13 of them hold guest pointers
(`0x8xxxxxxx`/`0xbxxxxxxx` — heap addresses, which move per process).
**No solved or interpolated field offset is among the 15** — every value
reported here is run-invariant.
* The two non-pointer stragglers, `+0x2c8` and `+0x2d0`, are **stage-dependent**:
for one and the same unit (`UN_f001_TCAF_DeltaSaber_T_Ttrl`) `+0x2c8` reads
`8000.0` in two tutorials and `10000.0` in a third, with `+0x2d0` a 0/1 flag
beside it. So the object is *mostly* but not *entirely* the parsed table —
a couple of words are set per stage. Unidentified; **NEEDS-HUMAN**.
One `.tbl`**one** object. A unit table is several sub-records (`Generic`,
`Maneuver`, `Shield`, `Explosion`, `Mass`, `Effect`, `SE`, `Turret_00N`), and
@@ -78,7 +92,7 @@ under lavapipe and makes repeated live reads flaky.
Every `AV_*` / `AA_*` / `*Bank*` / `Turn_AngularVelocity` field is stored as
**float32 radians**, while the disc writes **degrees**. The solver needed a
`rad` encoding (`degrees(f32)`) to bind them at all; 15 units agree on
`rad` encoding (`degrees(f32)`) to bind them at all; 18 units agree on
`AV_PitchPlus_Max` alone. A reimplementation reading the `.tbl` must convert.
## Confirmed layout
@@ -87,39 +101,44 @@ Every `AV_*` / `AA_*` / `*Bank*` / `Turn_AngularVelocity` field is stored as
| offset | enc | field | agree | distinct |
|---|---|---|---:|---:|
| `+0x030` | f32 | `Size_X` | 18 | 13 |
| `+0x038` | f32 | `Size_Z` | 16 | 13 |
| `+0x040` | f32 | `Color_R` | 18 | 5 |
| `+0x044` | f32 | `Color_G` | 16 | 6 |
| `+0x048` | f32 | `Color_B` | 14 | 5 |
| `+0x050` | f32 | `Size_Radius` | 11 | 10 |
| `+0x054` | f32 | `HP` | 17 | 10 |
| `+0x074` | f32 | `ResistanceToOptics` | 10 | 3 |
| `+0x08c` | f32 | `ScorePoint` | 18 | 9 |
| `+0x0a0` | f32 | `MaximumVelocity` | 16 | 8 |
| `+0x0a4` | f32 | `CruisingVelocity` | 14 | 7 |
| `+0x0a8` | f32 | `Acceleration` | 15 | 5 |
| `+0x0ac` | f32 | `Deceleration` | 14 | 5 |
| `+0x0b0` | rad | `AV_PitchPlus_Max` | 15 | 7 |
| `+0x0f8` | f32 | `SideThrustVelocity_Max` | 12 | 3 |
| `+0x238` | f32 | `MaxValue` (Shield) | 12 | 6 |
| `+0x244` | f32 | `ChargeSpeed` (Shield) | 11 | 6 |
| `+0x270` | f32 | `DestroyMotionTime` | 16 | 7 |
| `+0x2a0` | f32 | `RadarRange` | 15 | 8 |
| `+0x2a4` | f32 | `FCSRange` | 10 | 6 |
| `+0x2b4` | f32 | `AttackVesselPoint` | 11 | 8 |
| `+0x2bc` | f32 | `DefencePoint` | 10 | 7 |
| `+0x030` | f32 | `Size_X` | 21 | 13 |
| `+0x034` | f32 | `Size_Y` | 12 | 7 |
| `+0x038` | f32 | `Size_Z` | 19 | 13 |
| `+0x040` | f32 | `Color_R` | 21 | 5 |
| `+0x044` | f32 | `Color_G` | 19 | 6 |
| `+0x048` | f32 | `Color_B` | 17 | 5 |
| `+0x050` | f32 | `Size_Radius` | 12 | 10 |
| `+0x054` | f32 | `HP` | 19 | 10 |
| `+0x074` | f32 | `ResistanceToOptics` | 11 | 3 |
| `+0x08c` | f32 | `ScorePoint` | 21 | 9 |
| `+0x094` | f32 | `MassScore` | 10 | 7 |
| `+0x09c` | f32 | `MinimumVelocity` | 11 | 3 |
| `+0x0a0` | f32 | `MaximumVelocity` | 19 | 8 |
| `+0x0a4` | f32 | `CruisingVelocity` | 17 | 7 |
| `+0x0a8` | f32 | `Acceleration` | 18 | 5 |
| `+0x0ac` | f32 | `Deceleration` | 17 | 5 |
| `+0x0b0` | rad | `AV_PitchPlus_Max` | 18 | 7 |
| `+0x0b4` | rad | `AV_PitchPlus_Min` | 10 | 6 |
| `+0x0c4` | rad | `AV_PitchMinus_Min` | 10 | 5 |
| `+0x0f8` | f32 | `SideThrustVelocity_Max` | 14 | 3 |
| `+0x238` | f32 | `MaxValue` (Shield) | 13 | 6 |
| `+0x244` | f32 | `ChargeSpeed` (Shield) | 13 | 6 |
| `+0x270` | f32 | `DestroyMotionTime` | 19 | 7 |
| `+0x2a0` | f32 | `RadarRange` | 18 | 9 |
| `+0x2a4` | f32 | `FCSRange` | 12 | 8 |
| `+0x2b4` | f32 | `AttackVesselPoint` | 14 | 8 |
| `+0x2bc` | f32 | `DefencePoint` | 12 | 7 |
`Size_Y +0x034`, `MassScore +0x094`, `HQRatio +0x058`, `ShieldRatio +0x05c`,
`HQRatio +0x058`, `ShieldRatio +0x05c`,
`ThrusterRatio +0x060`, `ResistanceToShell +0x078`,
`ResistanceToExplosion +0x07c`, `ResistanceToPlayer +0x080`,
`ResistanceParalyze +0x084`, `BridgeCount +0x070` (i32),
`MinimumVelocity +0x09c`, `DryMass +0x274`, `GrossMass +0x278`,
`DryMass +0x274`, `GrossMass +0x278`,
`LowerHPThresholdRatio +0x298`, `AttackCraftPoint +0x2b8` bind with zero
contradictions on fewer records or fewer distinct values — 🟡 PROBABLE. The
full solver output is in
[`captures/unit-runtime-fields.csv`](../captures/unit-runtime-fields.csv)
(1 440 values, `conf` column).
(1 827 values, `conf` column).
## The Maneuver block is laid out in schema declaration order
@@ -212,12 +231,32 @@ every unit**, with real exceptions that only the runtime shows:
## Coverage and how to extend it
18 of 110 units. Unlike weapons — where one snapshot held all 126 — **unit
21 of 110 units. Unlike weapons — where one snapshot held all 126 — **unit
definitions are instantiated per stage**, so coverage is bounded by the stages
reachable from the save (slot 01 is at 5 %, Stage 02). Each further mission or
tutorial adds its own units; `unit_runtime.py` unions any number of snapshots
and re-solves, and more units directly promote the 🟡 bindings to ✅ by adding
distinct values.
reachable from the save (slot 01 is at 5 %, Stage 02). `unit_runtime.py` unions
any number of snapshots and re-solves, and more units directly promote the 🟡
bindings to ✅ by adding distinct values — the six tutorials took the confirmed
set from 22 fields to 27.
The tutorials are nearly exhausted as a source: all six together contribute only
3 units the missions do not already have (`UN_f001_TCAF_DeltaSaber_T_Ttrl`,
`UN_e015_ADAN_Puppy_2`, `UN_f001_TCAF_DeltaSaber_T_Player_Ttrl2`) — they reuse
one training box, one target drone and the player craft. **Real coverage now
needs real missions**, i.e. story progress on the save.
`tools/re-capture/grab_tutorial.sh` captures one tutorial per invocation
(cold boot → menu → Nth entry → snapshot, ~2.5 min). It cold-boots for each
because backing out of a loaded mission via PAUSE → BACK TO MENU wedges the
emulator. Two timing facts it encodes, both learned the hard way: the main menu
is **not input-ready for ~10 s** after the title tap, and d-pad presses before
that are silently dropped — which sends the A to `NEW GAME` instead of
`TUTORIAL`. And `NEW GAME` is not a cheap way to reach Stage 01: it gates on a
DIFFICULTY menu and then plays the prologue movie.
A NEW GAME excursion as far as the READY ROOM leaves
`535107D4/00000001/game01/savedata` byte-identical — only the profile `.gpd`
achievement files change — so it does not endanger the 5 % save. Verified by
diff against a backup, not assumed.
**A stage's whole unit set is parsed at load, not as waves spawn** — checked by
counting the definition objects at three points in Stage 02: immediately after

View File

@@ -29,6 +29,7 @@ Findings produced with these: [`docs/re/weapon-datasheet-runtime.md`](../../docs
| `unit_runtime.py` | Same solver for the `unit\UN_*.tbl` definition objects (vtable `0x820af844`). Unions several snapshots — unit definitions are per-stage. |
| `schema_order.py` | Merge a sub-record's field-declaration order across all tables (topological sort); the layout check that pins fields no table ever values. |
| `order_check.py` | Test `offset = base + 4*index` for one table's sub-record against solver output. |
| `grab_tutorial.sh` | Cold-boot Canary, walk to the Nth TUTORIAL entry, wait for the stage load, snapshot guest RAM. One emulator per capture — backing out of a loaded mission wedges it. |
Snapshot first — `cp --sparse=always /dev/shm/xenia_memory_* snap.bin` (~2 s) — and
point `$GMEM_FILE` at the copy; the running emulator pegs every core under lavapipe.

View File

@@ -0,0 +1,66 @@
#!/usr/bin/env bash
# Capture one tutorial's unit definitions: cold-boot Canary, walk
# title -> TUTORIAL -> the Nth entry, wait for the stage to load, snapshot RAM.
#
# One emulator per tutorial on purpose: backing out of a loaded mission via
# PAUSE -> BACK TO MENU wedged the emulator (log stops, CPU spins), while a
# cold boot here is ~15 s. The unit definitions are all parsed at stage load,
# so a single snapshot right after the load is everything this needs.
set -u
N="${1:?usage: grab_tutorial.sh <index 0..5> <outfile>}"
OUT="${2:?}"
export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy DISPLAY=:98
SD="$(cd "$(dirname "$0")" && pwd)"
RC="/home/fabi/RE - Project Sylpheed/sylpheed-reborn/tools/re-capture"
# The container entrypoint's Xvfb/openbox do NOT restart on their own, and a
# dead Xvfb leaves a <defunct> entry that still matches `pgrep` -- so test the
# display with xdpyinfo, never by process name.
ensure_display(){
if ! xdpyinfo -display "$DISPLAY" >/dev/null 2>&1; then
echo "[grab] $DISPLAY is down -> restarting Xvfb + openbox"
setsid nohup Xvfb "$DISPLAY" -screen 0 1280x720x24 -ac -nolisten tcp \
+extension GLX +extension RANDR </dev/null >/tmp/xvfb98.log 2>&1 &
for _ in $(seq 1 50); do xdpyinfo -display "$DISPLAY" >/dev/null 2>&1 && break; sleep 0.2; done
setsid nohup env DISPLAY="$DISPLAY" HOME=/sylph-home openbox </dev/null >/tmp/openbox98.log 2>&1 &
sleep 1
fi
xdpyinfo -display "$DISPLAY" >/dev/null 2>&1 || { echo "DISPLAY UNAVAILABLE"; exit 1; }
}
step(){ vgamepad dpad "$1"; sleep 0.2; vgamepad dpad center; sleep 0.6; }
shot(){ screenshot "$SD/$1" >/dev/null 2>&1; }
pkill -x xenia_canary 2>/dev/null; sleep 2
pgrep -x xenia_canary >/dev/null && { pkill -9 -x xenia_canary; sleep 2; }
rm -f /dev/shm/xenia_memory_* /dev/shm/xenia_code_cache_* 2>/dev/null
ensure_display
cd /sylph-home/re
setsid nohup run-canary --audio --apu=sdl --log_mask=13 \
--logged_profile_slot_0_xuid=E0300000EFBEA3D4 </dev/null >/dev/null 2>&1 &
sleep 5
"$RC/skip_intro.sh" 600 || { echo "BOOT FAILED"; exit 1; }
# The main menu is not input-ready for ~10 s after the title tap under lavapipe;
# d-pad presses before that are silently dropped, and the A then lands on NEW
# GAME instead of TUTORIAL. This wait is the whole fix.
sleep 14
shot "nav-$N-menu.png"
step down; step down # NEW GAME -> TUTORIAL
vgamepad tap A 250; sleep 8
shot "nav-$N-list.png"
i=0; while [ "$i" -lt "$N" ]; do step down; i=$((i+1)); done
shot "nav-$N-pick.png"
vgamepad tap A 250
# the stage load takes ~25-45 s under lavapipe; wait for the screen to settle
sleep 50
shot "nav-$N-loaded.png"
MEM=$(ls /dev/shm/xenia_memory_* 2>/dev/null | head -1)
[ -n "$MEM" ] || { echo "NO SHM"; exit 1; }
cp --sparse=always "$MEM" "$SD/$OUT" || exit 1
echo "SNAPSHOT $OUT ($(du -h "$SD/$OUT" | cut -f1))"

View File

@@ -195,6 +195,47 @@ def solve(disc, run):
return {f: v for f, v in solved.items() if f in winners}, numeric
def crosscheck(paths, solved):
"""Which words of a unit's object vary between snapshots, and are any of
them fields we claim to have solved?
Within one run the objects are byte-identical, but across runs the same
unit differs -- pointer words hold heap addresses, which move. This
separates "the emulator relocated it" from "the value is not definition
data", and it is the check that keeps the solved set honest.
"""
per = {}
for p in paths:
for k, v in runtime_objects([p])[0].items():
per.setdefault(k, {})[os.path.basename(p)] = v
diff = set()
shared = 0
for k, d in per.items():
if len(d) < 2:
continue
shared += 1
vs = list(d.values())
for v in vs[1:]:
diff |= {i // 4 * 4 for i in range(min(len(v), len(vs[0]))) if v[i] != vs[0][i]}
def looks_ptr(off):
vals = [struct.unpack_from(">I", v, off)[0]
for d in per.values() for v in d.values() if len(v) > off + 3]
return sum(1 for x in vals if 0x80000000 <= x < 0xC0000000), len(vals)
print(f"\n# cross-run check: {shared} unit(s) appear in more than one snapshot")
print(f"# words that differ between runs: {len(diff)}")
for off in sorted(diff):
p, n = looks_ptr(off)
kind = "guest pointer" if p == n else ("mixed" if p else "non-pointer")
print(f"# +{off:#05x} {kind} ({p}/{n} look like pointers)")
offs = {o for o, _, _, _, _ in solved.values()} | {o for o, _, _ in INTERPOLATED}
clash = sorted(offs & diff)
print("# solved/interpolated offsets that vary between runs: "
+ (", ".join(f"{o:#05x}" for o in clash) if clash
else "NONE — every reported field is run-invariant"))
def main():
tokens = sys.argv[1]
snaps = [a for a in sys.argv[2:] if not a.startswith("--")]
@@ -214,6 +255,9 @@ def main():
for field, (off, enc, agree, bad, distinct) in sorted(solved.items(), key=lambda kv: kv[1][0]):
print(f"{off:#8x} {enc:>4} {field:<32} {agree:5d} {distinct:4d} {len(bad):3d}")
if "--crosscheck" in sys.argv:
crosscheck(snaps, solved)
if "--csv" in sys.argv:
import csv