From d1b50dc1e63b7dae06b34f2f7603388315dcfd23 Mon Sep 17 00:00:00 2001 From: "Claude (auto-RE)" Date: Thu, 13 Aug 2026 12:50:19 +0000 Subject: [PATCH] =?UTF-8?q?re(xbg7):=20characterise=20the=20residual=2047?= =?UTF-8?q?=20misses=20=E2=80=94=2099.25%=20of=20resources=20decode?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit examples/why_missed.rs reports the furthest gate per undecoded resource. Disc-wide that is 47 rows / 43 distinct names, and they are mostly not ship geometry: 30 e_rou_/_rou_ pose & proxy composites (24-vertex marker boxes, extent 0.010) 6 .DAT particle composites in ptc_pack 8 damage/LOD variants (e101_bdy_02_d, e901_wing_05_*_m, ...) 3 props/other (g005 extent 0.196, _rou_f001_wep_05, e_rou_e005) By gate: extent 32, coverage 8, winding 7. The extent bucket is almost entirely the pose-proxy boxes, and lowering that floor was measured and refuted earlier. The one genuinely interesting residual is e901_wing_05_L/R at winding 0.587/0.570 against the 0.70 floor β€” the signature of a thin double-sided sheet, unproven without a capture with the boss on screen (flying stage 16 puts the container in memory but the unit never appeared). So: 6 247 / 6 294 resources decode (99.25 %), and every real mesh drawn in three captured missions decodes at the GPU's own offsets with index runs matching byte for byte (93/93 and 128/128). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NptfmpjdpNCKEez6d2xvA9 --- .../sylpheed-formats/examples/why_missed.rs | 19 +++++++++++++ docs/re/INDEX.md | 2 +- docs/re/structures/xbg7-mesh.md | 27 +++++++++++++++++++ 3 files changed, 47 insertions(+), 1 deletion(-) create mode 100644 crates/sylpheed-formats/examples/why_missed.rs diff --git a/crates/sylpheed-formats/examples/why_missed.rs b/crates/sylpheed-formats/examples/why_missed.rs new file mode 100644 index 0000000..08cb339 --- /dev/null +++ b/crates/sylpheed-formats/examples/why_missed.rs @@ -0,0 +1,19 @@ +//! Why does a named resource never decode? Reports the furthest gate its best +//! candidate reached. Usage: why_missed [name-substring] +use sylpheed_formats::mesh::{debug_best_rejection, xbg7_resource_names, Xbg7Model}; +use std::collections::HashSet; +fn main() { + let a: Vec = std::env::args().collect(); + let bytes = std::fs::read(&a[1]).unwrap(); + let filter = a.get(2).cloned().unwrap_or_default(); + let decoded: HashSet = Xbg7Model::stage_models(&bytes).into_iter().map(|m| m.name).collect(); + for n in xbg7_resource_names(&bytes) { + if decoded.contains(&n) || (!filter.is_empty() && !n.contains(&filter)) { + continue; + } + match debug_best_rejection(&bytes, &n) { + Some((rank, why)) => println!("{n:<28} rank {rank} {why}"), + None => println!("{n:<28} (no candidate reached any gate / skipped before anchoring)"), + } + } +} diff --git a/docs/re/INDEX.md b/docs/re/INDEX.md index 5a5d10e..16fef47 100644 --- a/docs/re/INDEX.md +++ b/docs/re/INDEX.md @@ -19,7 +19,7 @@ Promote to a prose `structures/…md` file when a format needs behavioural notes | LSTA sprite list | βœ… | `sylpheed-formats/src/lsta.rs` | A display list of inline elements: **T8aD sprites and `PRMD` primitives**. The `count` at `0x04` is **exact and counts both** β€” `count == T8aD + PRMD` for **64/64** lists on the disc, which retires the old "a few entries disagree" note (it compared sprites against a total including primitives). **All 1 281 sprite frames decode** after the T8aD rectangle-list fix | | IXUD subtitle | 🟑/βœ… | `sylpheed-formats/src/ixud.rs` + [movie link](movie-subtitle-link.md) | timed cues. **The movie↔subtitle↔voice link is solved β€” statically**, from the movie config record in `tables.pak` (schema `0x067025b9`), not from the running game as this row previously assumed: [101 movies mapped](captures/movie-subtitle-voice-map.csv), 94 with subtitles, 83 with voice, 21 with a telop overlay. 93 of 94 subtitle refs resolve in the language paks; **`SUBTITLE_S12B.tbl` is missing from all six languages** β€” a dangling reference on the disc. Naming is `SUBTITLE_.tbl` / `VOICE_` with six documented exceptions. The record's ~104 **script ids** are ❔ β€” positional pairing drifts by three because the IDXD pool dedupes repeated values | | Fonts (ttf/otf/ttc) | βœ… | `sylpheed-formats/src/font.rs` | standard OpenType, parsed via ttf-parser | -| XBG7 mesh | βœ…/🟑 | `sylpheed-formats/src/mesh.rs` + `tests/mesh_disc.rs` ([xbg7](structures/xbg7-mesh.md)) | **6 294 resources, 6 209 decode (98.7 %), 82 searched-and-missed** (2026-08-12, up from 5 480 / 87.1 %). Five evidence-driven fixes got there: **distinct anchor assignment** (no two resources may claim one buffer β€” proved by a capture showing the container holds both mirrored `e106` hull halves), the connectivity cap replaced by a **winding-consistency gate at 0.70**, **structural requirements on pre-pivot sub-meshes** (index range, then exact pool coverage), and **filtering after the assignment** so a subset query cannot differ from the full decode. Validated against a runtime capture that names the file offset of every buffer the engine drew: **46/46 drawn buffers claimed, 45 anchored exactly**. **No real mesh now decodes differently in different containers** β€” all 89 remaining cross-container disagreements are interchangeable 24-vertex bounding boxes, which no anchoring rule can pin (monotone order re-tested and refuted). Remaining misses attribute to the degeneracy/extent gate (42), winding (31) and coverage (9); the first was probed and its "obvious" fix refuted. Every decoded sub-mesh covers its own vertex pool. **The `[index buffer][vertex buffer]` layout is now runtime-verified** (2026-08-13): with the F10 capture extended to log each draw's index buffer, all **42** drawn `Stage_S02` buffers match our decoded index count exactly, all 42 have their index union cover the pool exactly, and the 30 single-block cases all sit at `pad ≀ 3` β€” so `e106_eng_02_l`'s old rejection was the connectivity gate, not a misplaced index buffer. The `indices=` mystery was the capture keeping only the **first of several index batches** per buffer. **And comparing index VALUES found the biggest silent defect yet**: the anchor took the first `pad` that validated, so a block whose index data sits at pad 2 was read **one element late** β€” 76/93 captured runs matched, all 17 differences a one-element shift. Scoring pads by degenerate triangles + winding fixes it: **93/93** captured runs now match byte for byte, disc-wide degenerate runs **582 β†’ 1** (the grouped path had the same bug; and two resources were anchored on a degenerate lookalike earlier in file order), **590 of 8 850** sub-meshes re-wired with 10 vertex anchors moved, resources decoded unchanged at 6 209. Cross-container minority decodes 89 β†’ 96 β€” *because* the decoder improved: `_rou_f402_dead` now has a majority (32Γ—25Γ—8) so its seven wrong copies are named instead of hidden. One dirty run remains, blocked by distinct assignment on a 24-vertex box | +| XBG7 mesh | βœ…/🟑 | `sylpheed-formats/src/mesh.rs` + `tests/mesh_disc.rs` ([xbg7](structures/xbg7-mesh.md)) | **6 294 resources, 6 209 decode (98.7 %), 82 searched-and-missed** (2026-08-12, up from 5 480 / 87.1 %). Five evidence-driven fixes got there: **distinct anchor assignment** (no two resources may claim one buffer β€” proved by a capture showing the container holds both mirrored `e106` hull halves), the connectivity cap replaced by a **winding-consistency gate at 0.70**, **structural requirements on pre-pivot sub-meshes** (index range, then exact pool coverage), and **filtering after the assignment** so a subset query cannot differ from the full decode. Validated against a runtime capture that names the file offset of every buffer the engine drew: **46/46 drawn buffers claimed, 45 anchored exactly**. **No real mesh now decodes differently in different containers** β€” all 89 remaining cross-container disagreements are interchangeable 24-vertex bounding boxes, which no anchoring rule can pin (monotone order re-tested and refuted). Remaining misses attribute to the degeneracy/extent gate (42), winding (31) and coverage (9); the first was probed and its "obvious" fix refuted. Every decoded sub-mesh covers its own vertex pool. **The `[index buffer][vertex buffer]` layout is now runtime-verified** (2026-08-13): with the F10 capture extended to log each draw's index buffer, all **42** drawn `Stage_S02` buffers match our decoded index count exactly, all 42 have their index union cover the pool exactly, and the 30 single-block cases all sit at `pad ≀ 3` β€” so `e106_eng_02_l`'s old rejection was the connectivity gate, not a misplaced index buffer. The `indices=` mystery was the capture keeping only the **first of several index batches** per buffer. **And comparing index VALUES found the biggest silent defect yet**: the anchor took the first `pad` that validated, so a block whose index data sits at pad 2 was read **one element late** β€” 76/93 captured runs matched, all 17 differences a one-element shift. Scoring pads by degenerate triangles + winding fixes it: **93/93** captured runs now match byte for byte, disc-wide degenerate runs **582 β†’ 1** (the grouped path had the same bug; and two resources were anchored on a degenerate lookalike earlier in file order), **590 of 8 850** sub-meshes re-wired with 10 vertex anchors moved, resources decoded unchanged at 6 209. Cross-container minority decodes 89 β†’ 96 β€” *because* the decoder improved: `_rou_f402_dead` now has a majority (32Γ—25Γ—8) so its seven wrong copies are named instead of hidden. One dirty run remains, blocked by distinct assignment on a 24-vertex box. **Then the descriptor gave up its last structural secret**: it declares a vertex layout **per sub-mesh** (`n201_01` β†’ strides 24/24/24/**28**, capture-confirmed), and grouped selection must prefer the candidate explaining the **whole** pool rather than the first whose pivot validates β€” together they take never-decoding resources **85 β†’ 47** (**6 247 / 6 294 = 99.25 %** decode), put `n201_01` on all four capture-proven offsets and raise the stage-05 capture oracle to **128/128**. The residual 47 is 30 pose/proxy composites (0.010-unit marker boxes), 6 `.DAT` particle composites, 8 damage/LOD variants and 3 props β€” not a threshold away | | Capital-ship part placement | βœ… | `sylpheed-formats/src/ship.rs` (static) + [runtime capture](ship-placement-runtime-capture.md) | Placement is **sound** (hull static-exact against the `e106` capture; cross-id mounting genuinely narrow, 2 pairs across 335 ships). The XBG7 mis-decode this row used to blame for "ships assemble wrong" β€” a shared turret ~100Γ— too large in some containers β€” is **fixed** (2026-08-12, the exact-coverage requirement): `e303_wep_01` now decodes 49Γ—23Γ—42 everywhere and places at Β±179 on the `e106` hull, and no real mesh disagrees across containers. A composite-node audit confirmed the assembler itself never applied a bad scale (all nodes scale 1.0, orthonormal). Still open: `static_assembly_matches_runtime_capture` walks capture parts only, so **extra** static placements cannot fail it | | Weapon fields defaulted on disc | βœ… | [runtime struct](structures/weapon-struct-runtime.md) Β· [DATA SHEET route](weapon-datasheet-runtime.md) | **Solved.** Canary maps guest RAM into `/dev/shm`, so the parsed `Weapon`/`Shell` objects are readable live; their layout is solved against disc ground truth (zero contradictions over 100+ records). All 126 weapons, exact numbers, no story progress needed β€” [4 393 values](captures/weapon-runtime-fields.csv) the disc does not carry. Supersedes the letter-bucket limit of the DATA SHEET route, which now serves as the independent cross-check | | Unit (craft/vessel) fields defaulted on disc | βœ…/🟑 | [runtime struct](structures/unit-struct-runtime.md) | The parsed `unit\UN_*.tbl` definition object, vtable `0x820af844`, β‰₯`0x380` bytes, one per unit β€” **discovered, not assumed** (`unit_discover.py`), and distinguished from the spawned-entity class `0x820af030` by being one-per-ID and byte-constant within a run. Across runs only pointer words move β€” `--crosscheck` proves **no reported field offset is run-dependent** (two words, `+0x2c8`/`+0x2d0`, are stage-dependent and remain unidentified). 27 fields βœ… (21 units, 7 runs); the `Maneuver` block is **schema declaration order, 4 bytes/field, base `0x9c` with a two-slot gap after `AA_Roll_Min`** (29 anchors, 0 conflicts), which also pins 5 fields *no* disc record ever values. Angles are **radians at runtime, degrees on disc**. **Re-derived independently 2026-08-13 from the loader's own key strings** (`sub_82341A20`; the field name for each store is a string in the image): **159 fields**, agreeing with this solver on **25 of 25 shared offsets**, verified at **406 values matching the disc and 0 disagreeing** over 11 live objects spanning UNIT and VESSEL β€” landed as `data/unit_definition_layout.txt` + `sylpheed_formats::unit_layout` + a no-emulator test, with **121 defaulted fields** read out ([live-unit-definitions](live-unit-definitions.md)). Unlike weapons, unit definitions are instantiated **per stage**, so coverage (21/110) grows by visiting missions β€” but a defaulted field is **not** a global constant: `Size_Y` provably inherits `Size_X` (7 independent units, 6 distinct values), and three more sibling rules are recorded ❔, recovering 65 values in units never visited β€” [values](captures/unit-runtime-fields.csv) | diff --git a/docs/re/structures/xbg7-mesh.md b/docs/re/structures/xbg7-mesh.md index ae7fa60..ae8c2a2 100644 --- a/docs/re/structures/xbg7-mesh.md +++ b/docs/re/structures/xbg7-mesh.md @@ -1700,3 +1700,30 @@ The last row is the one that matters most: the four buffers that capture could n name are the `n201` family, and now they decode **and** their index runs match the GPU byte for byte. Suite green (`twin_pairs_do_not_share_a_buffer` included) apart from the pre-existing known-failing cross-container consistency test. + +### Where the anchor work stands: 6 247 / 6 294 (99.25 %), and what the residual 47 is + +`examples/why_missed.rs` reports the furthest gate for every resource a container +fails to decode. Disc-wide that is **47 rows / 43 distinct names**, and they are +not a threshold away β€” they are mostly **not ship geometry**: + +| class | rows | what they are | +|---|---|---| +| `e_rou_` / `_rou_` pose & proxy composites | **30** | scene-composite records like `e_rou_e901_attackA_start`, `…_guard_end`, `…_stand` β€” 24-vertex marker boxes whose extent is **0.010** | +| `.DAT` particle composites (`ptc_pack`) | 6 | `h309.DAT`, `m001_n.DAT`, … β€” the marker lists that provably do not map onto the stored blocks | +| damage / LOD variants | 8 | `e101_bdy_02_d` (indices reach 13 171 of 15 430 vertices), `e901_wing_05_*_m`, … | +| props / other | 3 | `g005` (extent 0.196), `_rou_f001_wep_05`, `e_rou_e005` | + +By gate: **extent 32 Β· coverage 8 Β· winding 7**. The extent bucket is almost +entirely the pose-proxy boxes β€” a 0.010-unit cube is a *marker*, not a mesh, and +lowering the floor to admit them was measured before and refuted (it bought 2 +resources and cost cross-container consistency). The `e901_wing_05_L/R` pair is the +one genuinely interesting residual: winding **0.587 / 0.570** against the 0.70 floor, +which is what a **thin double-sided sheet** looks like when face normals are compared +to averaged vertex normals β€” plausible but unproven, and the only way to settle it is +a capture with the boss on screen (flying stage 16 was not enough: the container is +resident but the unit never appeared). + +So the anchor line of work is **done for practical purposes**: every real mesh the +engine draws in three captured missions decodes, at the offsets the GPU used, with +index runs matching byte for byte (93/93 and 128/128).