Claude (auto-RE)
0776beb6f4
Two follow-ups to the pad-scoring fix, both driven by the same invariant (a correctly located index run has no degenerate triangles): - anchor_grouped_meshes picked its pad by first-match too; scoring the pivot run the same way cleared every remaining ptc_pack composite (f102/f104/e107). - anchor_pool_mesh now prefers a degenerate-free candidate over an earlier dirty one. examples/better_home.rs showed the last two resources each had exactly one degenerate-free, pool-covering block, sitting later in file order than the lookalike we took. First-match order is kept for every clean hit, and a dirty block is still used if nothing clean exists, so coverage cannot regress. degenerate index runs, disc-wide: 582 -> 11 -> 1 captured index runs identical: 93/93 (unchanged) resources decoded / misses: 6 209 / 85 (unchanged) index runs changed / anchors moved: 590 / 10 (_rou_f402_dead x8, e201_bdy_03_m x2) Cross-container minority decodes 89 -> 96, and that is progress: all seven new rows are _rou_f402_dead, which now has a majority (32x25x8) for the first time, so its seven wrong copies are named instead of hidden behind "no majority". The last dirty run (_rou_f402_dead in Stage_S09) is blocked by distinct assignment — its clean block is claimed by e_rou_f003_Near, both 24-vertex bounding boxes. A winding-floor escalation for that case was written, measured to fire for nothing, and reverted; the reasoning is kept as a comment. Regression threshold tightened to 1. Suite green with --include-ignored apart from the pre-existing known-failing cross-container consistency test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NptfmpjdpNCKEez6d2xvA9
48 lines
17 KiB
Markdown
48 lines
17 KiB
Markdown
# RE knowledge index
|
||
|
||
Confidence: ✅ `CONFIRMED` · 🟡 `PROBABLE` · ❔ `HYPOTHESIS`. See [README](README.md).
|
||
|
||
Formats we've already reversed are, for now, **documented by their parser + disc round-trip
|
||
tests** (the executable spec) rather than a prose file — the "Spec" column points there.
|
||
Promote to a prose `structures/…md` file when a format needs behavioural notes beyond layout.
|
||
|
||
## Data structures / formats
|
||
|
||
| Format | Conf. | Spec (parser + tests) | Notes |
|
||
|--------|-------|-----------------------|-------|
|
||
| IPFB `.pak` archive | ✅ | `sylpheed-formats/src/pak.rs` + `tests/pak_idxd_disc.rs` | header + 12-byte TOC, Z1/zlib payloads |
|
||
| name-hash (TOC keys) | ✅ | `sylpheed-formats/src/hash.rs` | Barrett-reduction hash; recovers original paths |
|
||
| IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` | self-describing; ship/weapon stats verified vs known values |
|
||
| XPR2 texture + cubemap | 🟡/✅ | `sylpheed-formats/src/texture.rs` + [colour check](xpr2-colour-check.md) | de-tile + A8R8G8B8 and DXT1. **Channel order ✅ confirmed against the running game**: the Delta Saber's decoded atlas is orange-dominant (median saturated hue 23.3°, *zero* cool pixels) and the game renders the same hull at 9.3° — a red↔blue swap would sit at ≈200°. Exact fidelity (gamma/sRGB curve, premultiplied alpha, per-channel scale) is 🟡 untested, since a hue comparison cannot see it; cubemap face ordering ❔ |
|
||
| T8aD 2D texture | ✅ | `sylpheed-formats/src/t8ad.rs` | **100 % of the disc decodes** (19 216/19 216, measured). The "~15 % deferred variants" were a wrong model, not a variant: a surface is a list of **arbitrary sub-rectangles**, each with a 16-byte header of `dst X, dst Y, width, height`, not a 256×256 grid — `0x1c` is the **rectangle count**. Uncovered area stays transparent. **Colours ✅ CONFIRMED** ([k8888](structures/texture-color-k8888.md)) |
|
||
| RATC bundle | ✅ | `sylpheed-formats/src/ratc.rs` | child listing confirmed. **"One level deep" is not a limitation — there is nothing deeper**: 2 859 bundles hold 18 002 children at depth 1 and **0 at depth 2**, with no parse failures. Nested RATC blobs are **leaf records that reference siblings by name** (`opt `, the sprite name): 3 311 leaves, all embedding sibling names, **10 144 of 10 148 references resolve**. The 4 that do not are one dangling asset — `pmbase.rat` → `pmbase.t32` in `GP_STAGE_CLEAR.pak`'s four language builds, and `pmbase.t32` is **on the disc nowhere** |
|
||
| LSTA sprite list | ✅ | `sylpheed-formats/src/lsta.rs` | A display list of inline elements: **T8aD sprites and `PRMD` primitives**. The `count` at `0x04` is **exact and counts both** — `count == T8aD + PRMD` for **64/64** lists on the disc, which retires the old "a few entries disagree" note (it compared sprites against a total including primitives). **All 1 281 sprite frames decode** after the T8aD rectangle-list fix |
|
||
| IXUD subtitle | 🟡/✅ | `sylpheed-formats/src/ixud.rs` + [movie link](movie-subtitle-link.md) | timed cues. **The movie↔subtitle↔voice link is solved — statically**, from the movie config record in `tables.pak` (schema `0x067025b9`), not from the running game as this row previously assumed: [101 movies mapped](captures/movie-subtitle-voice-map.csv), 94 with subtitles, 83 with voice, 21 with a telop overlay. 93 of 94 subtitle refs resolve in the language paks; **`SUBTITLE_S12B.tbl` is missing from all six languages** — a dangling reference on the disc. Naming is `SUBTITLE_<base>.tbl` / `VOICE_<base>` with six documented exceptions. The record's ~104 **script ids** are ❔ — positional pairing drifts by three because the IDXD pool dedupes repeated values |
|
||
| Fonts (ttf/otf/ttc) | ✅ | `sylpheed-formats/src/font.rs` | standard OpenType, parsed via ttf-parser |
|
||
| XBG7 mesh | ✅/🟡 | `sylpheed-formats/src/mesh.rs` + `tests/mesh_disc.rs` ([xbg7](structures/xbg7-mesh.md)) | **6 294 resources, 6 209 decode (98.7 %), 82 searched-and-missed** (2026-08-12, up from 5 480 / 87.1 %). Five evidence-driven fixes got there: **distinct anchor assignment** (no two resources may claim one buffer — proved by a capture showing the container holds both mirrored `e106` hull halves), the connectivity cap replaced by a **winding-consistency gate at 0.70**, **structural requirements on pre-pivot sub-meshes** (index range, then exact pool coverage), and **filtering after the assignment** so a subset query cannot differ from the full decode. Validated against a runtime capture that names the file offset of every buffer the engine drew: **46/46 drawn buffers claimed, 45 anchored exactly**. **No real mesh now decodes differently in different containers** — all 89 remaining cross-container disagreements are interchangeable 24-vertex bounding boxes, which no anchoring rule can pin (monotone order re-tested and refuted). Remaining misses attribute to the degeneracy/extent gate (42), winding (31) and coverage (9); the first was probed and its "obvious" fix refuted. Every decoded sub-mesh covers its own vertex pool. **The `[index buffer][vertex buffer]` layout is now runtime-verified** (2026-08-13): with the F10 capture extended to log each draw's index buffer, all **42** drawn `Stage_S02` buffers match our decoded index count exactly, all 42 have their index union cover the pool exactly, and the 30 single-block cases all sit at `pad ≤ 3` — so `e106_eng_02_l`'s old rejection was the connectivity gate, not a misplaced index buffer. The `indices=` mystery was the capture keeping only the **first of several index batches** per buffer. **And comparing index VALUES found the biggest silent defect yet**: the anchor took the first `pad` that validated, so a block whose index data sits at pad 2 was read **one element late** — 76/93 captured runs matched, all 17 differences a one-element shift. Scoring pads by degenerate triangles + winding fixes it: **93/93** captured runs now match byte for byte, disc-wide degenerate runs **582 → 1** (the grouped path had the same bug; and two resources were anchored on a degenerate lookalike earlier in file order), **590 of 8 850** sub-meshes re-wired with 10 vertex anchors moved, resources decoded unchanged at 6 209. Cross-container minority decodes 89 → 96 — *because* the decoder improved: `_rou_f402_dead` now has a majority (32×25×8) so its seven wrong copies are named instead of hidden. One dirty run remains, blocked by distinct assignment on a 24-vertex box |
|
||
| Capital-ship part placement | ✅ | `sylpheed-formats/src/ship.rs` (static) + [runtime capture](ship-placement-runtime-capture.md) | Placement is **sound** (hull static-exact against the `e106` capture; cross-id mounting genuinely narrow, 2 pairs across 335 ships). The XBG7 mis-decode this row used to blame for "ships assemble wrong" — a shared turret ~100× too large in some containers — is **fixed** (2026-08-12, the exact-coverage requirement): `e303_wep_01` now decodes 49×23×42 everywhere and places at ±179 on the `e106` hull, and no real mesh disagrees across containers. A composite-node audit confirmed the assembler itself never applied a bad scale (all nodes scale 1.0, orthonormal). Still open: `static_assembly_matches_runtime_capture` walks capture parts only, so **extra** static placements cannot fail it |
|
||
| Weapon fields defaulted on disc | ✅ | [runtime struct](structures/weapon-struct-runtime.md) · [DATA SHEET route](weapon-datasheet-runtime.md) | **Solved.** Canary maps guest RAM into `/dev/shm`, so the parsed `Weapon`/`Shell` objects are readable live; their layout is solved against disc ground truth (zero contradictions over 100+ records). All 126 weapons, exact numbers, no story progress needed — [4 393 values](captures/weapon-runtime-fields.csv) the disc does not carry. Supersedes the letter-bucket limit of the DATA SHEET route, which now serves as the independent cross-check |
|
||
| Unit (craft/vessel) fields defaulted on disc | ✅/🟡 | [runtime struct](structures/unit-struct-runtime.md) | The parsed `unit\UN_*.tbl` definition object, vtable `0x820af844`, ≥`0x380` bytes, one per unit — **discovered, not assumed** (`unit_discover.py`), and distinguished from the spawned-entity class `0x820af030` by being one-per-ID and byte-constant within a run. Across runs only pointer words move — `--crosscheck` proves **no reported field offset is run-dependent** (two words, `+0x2c8`/`+0x2d0`, are stage-dependent and remain unidentified). 27 fields ✅ (21 units, 7 runs); the `Maneuver` block is **schema declaration order, 4 bytes/field, base `0x9c` with a two-slot gap after `AA_Roll_Min`** (29 anchors, 0 conflicts), which also pins 5 fields *no* disc record ever values. Angles are **radians at runtime, degrees on disc**. **Re-derived independently 2026-08-13 from the loader's own key strings** (`sub_82341A20`; the field name for each store is a string in the image): **159 fields**, agreeing with this solver on **25 of 25 shared offsets**, verified at **406 values matching the disc and 0 disagreeing** over 11 live objects spanning UNIT and VESSEL — landed as `data/unit_definition_layout.txt` + `sylpheed_formats::unit_layout` + a no-emulator test, with **121 defaulted fields** read out ([live-unit-definitions](live-unit-definitions.md)). Unlike weapons, unit definitions are instantiated **per stage**, so coverage (21/110) grows by visiting missions — but a defaulted field is **not** a global constant: `Size_Y` provably inherits `Size_X` (7 independent units, 6 distinct values), and three more sibling rules are recorded ❔, recovering 65 values in units never visited — [values](captures/unit-runtime-fields.csv) |
|
||
| Arsenal develop economy | ✅/❔ | [arsenal-develop-economy](arsenal-develop-economy.md) + [conditions](captures/arsenal-develop-conditions.csv) | The Arsenal reads `weapon.tbl` (item ids, in the 8-category display order) and `strings.tbl` (names, descriptions, and a **"Conditions to obtain"** block per item) out of `GP_HANGAR_ARSENAL.pak`. All **60** conditions are extracted: gates are stage completion, a predecessor item, or an **ace kill**; costs run 3 000–350 000 P and **20 items are free** once gated. `weapon.tbl`'s first record reproduces the in-game DATA SHEET exactly (Range D / Power E / Speed – / Weight 0.3 = Light / 4000 P) — later records are unreadable from the string pool alone because IDXD **dedupes repeated values**. Used to identify the save blob's index space, now **solved**: the blob follows **`strings.tbl`'s** order — the display order *plus* the cut items only the localisation file lists (`Adhesive Mine B2A`, `Ballista GSH`, …) — pinned by four hand-written probe saves (9 Stiletto, 21 Falcon, 39 Tomahawk, 48 Jamming System) and closing exactly at index 53. `weapon.tbl`'s id list is **not** the index space; that it is also 54 long is a coincidence, and the two agree only to index 32. The retail save's five unexplained owned entries are the cut items, shipped owned and never rendered |
|
||
| UI screen layout (`.rat`) | ✅/🟡 | [ui-rat-layout](structures/ui-rat-layout.md) | One pak per UI screen; each RATC = one (context × language) build; every `<name>.t32` sprite has a `<name>.rat` **layout record** (BE u32; 1280×720 design space; scale/tint/X/Y, keyframes for animated elements, `opt ` link to the focused state). **The tutorial PAUSE menu and the title main menu both rebuild pixel-accurately from the disc.** `loop1.rat` is decoded — it is a **looping sprite animation**, not a composition. The **screen's draw list is the RATC bundle's own declaration table** (elements in back-to-front order, including the `eff*`/`deli*`/`msg` sprites that have no `.rat`, and excluding focused button variants reached via `opt `); its entry also carries a **parent element index** at `+32`. **A screen is fully reconstructible from its bundle**: the placement region right after the declaration table gives every element a keyframe group (header = element index + keyframe count, then 40-byte blocks of scale/tint/X/Y), including the `.rat`-less sprites — verified 11/11 on the tutorial pause bundle, with `pgp_ttrl_btn10`'s inline (546,288) matching its own record exactly |
|
||
| Save file (`savedata`) | ✅/❔ | [savegame-format](structures/savegame-format.md) + [`tools/re-capture/savegame.py`](../../tools/re-capture/savegame.py) | `GDHA` container, zlib payload, chunk stream (`GDAA` / phase name / `GHAD` 122 B progress block / 16×20 B slot table / trailer). **Container and layout read off the title's own serializer `0x822C00E8` and verified by a byte-identical round-trip**; the whole save is 545 B. Payload offsets are also the live save object's offsets (`save+8` GHAD, `save+136` slots). A second save made in-game names **Points** (+24), **flight time in ms** (+4) and **clear ratio %** (+8) off the game's own Details panel; the payload is a **pure function of game state** (same state saved twice = byte-identical, only the header FILETIME and its uninitialised pointer padding move), and the 16 `SHAB` records are **not** the UI's 20 save slots. Difficulty vs stage is undecided — three fields hold 2. **A third save, taken after developing exactly one Arsenal weapon** (Light Machine Gun MG I, 4000 P), moves exactly three things: `+24` Points 4101→101 (which **separates it from `+28`**, that did not move), `+8` clear ratio 5→6 (so the ratio counts *collection*, not only stages), and two entries of the 54-byte blob — `2→4` for the item bought and `0→2` for the successor the game announced as newly developable, giving the blob its alphabet ✅ *0 locked / 2 developable / 4 developed* (only the `4`s are stored — `2` is re-derived at load). **Saves can also be written back**: three derived header fields (length at `+0x30`, payload length at `+0x8c`, `adler32` at `+0x8e`) are all that stand between a parse and a hand-written save that the title loads, and [`savegame_edit.py`](../../tools/re-capture/savegame_edit.py) re-wraps a real save byte-identically. That turned the blob's index space from blocked-on-story-progress into four probe saves — see the [economy note](arsenal-develop-economy.md) |
|
||
|
||
## Runtime / dynamic-capture technique
|
||
|
||
| Technique | Conf. | Spec | Notes |
|
||
|-----------|-------|------|-------|
|
||
| Live guest-memory read | ✅ | [`tools/re-capture/gmem.py`](../../tools/re-capture/gmem.py) | Canary backs the guest address space with `/dev/shm/xenia_memory_*`; guest VAs map in through Xenia's fixed table. Full-RAM search ~0.2 s (sparse, `SEEK_DATA`). No debugger, no emulator patch, game keeps running |
|
||
| IDXD object layout solver | ✅ | [`tools/re-capture/weapon_runtime.py`](../../tools/re-capture/weapon_runtime.py) | Scan RAM for a class's vtable → enumerate its objects → brute-force `(field, offset, encoding)` against the disc records. Accepts a binding only on **zero** contradictions. Generalizes to any IDXD-backed definition |
|
||
| Live entity state, anchored on the definition | ✅ | [`tools/re-capture/own_state.py`](../../tools/re-capture/own_state.py) · [autopilot](autopilot-memory-driven.md) | An undamaged craft holds its definition's own numbers, so a *solved definition field* locates the matching live field without a value scan: definition `HP` (1500) → **hull at `position+0x154`**, confirmed by a trace across a death (30/60/90 per hit, negative at 0). Reusable for any live counter whose maximum the definition carries |
|
||
| Mission / escort state, every entity's hull | ✅ | [`tools/re-capture/mission_state.py`](../../tools/re-capture/mission_state.py) · [escort state](mission-escort-state.md) | `hull = position + 0x154` is a property of the **entity class**, not of the player object: at t=0 it equals each entity's own definition `HP` across 7 classes and 5 distinct HP values (turret 100, fighter 500, destroyer 10000, cruiser 30000, **ACROPOLIS 25000**), falls under fire (780 damage events in 240 s), goes negative at death, and the object then leaves the heap. So an escort objective is scoreable live — `UN_f101_TCAF_Acropolis` measured at 25000 → 23038 over 240 s, attack starting only at t≈170 s. `REMAINING OB` counts objectives, not hostiles (012 on the HUD vs 118 live ADAN); its address is still ❔ |
|
||
| In-flight control mapping | ✅/🟡 | [`tools/re-capture/fire_probe.sh`](../../tools/re-capture/fire_probe.sh) · [controls](flight-controls-runtime.md) | Measured by holding each pad input and photographing the HUD ammo counters: **`RB` = nose gun** (6000→5956 in 4 s, ~11 rounds/s, HEAT rises), **`Y` = main mount** (missiles, 300→299), d-pad = **tactical map** overlay, nothing else moves a counter. No target-cycle input exists — the `TARGET` marker is present with nothing pressed, so targeting is automatic and a missile lock is **time-on-target**. That, not target choice or ballistics, is what caps lethality at 2 kills per 98 missiles |
|
||
| Input → dynamics calibration | ✅ | [`tools/re-capture/ctrl_probe.py`](../../tools/re-capture/ctrl_probe.py) · [`binq.py`](../../tools/re-capture/binq.py) | Hold each pad input in turn and measure the craft's speed as displacement/s of its own position triple — no speed field needed first. Settled the throttle: **`RT` accelerates, `LT` brakes, and the setting persists** (488 → 1510 → 174 units/s), overturning an earlier field-scan conclusion |
|
||
|
||
## Functions / code paths
|
||
|
||
_None documented yet — populated during the dynamic-RE phase._
|
||
|
||
| Function | Conf. | Reimpl. | Summary |
|
||
|----------|-------|---------|---------|
|
||
| — | — | — | — |
|