fabi/Syplheed-Reborn
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(formats): declaration-driven XBG7 decode (variable stride)

Browse Source 
Dynamic-RE follow-up: capture Canary's GPU vertex-fetch + draw calls and
feed the ground truth back into the static decoder.

The GPU capture confirmed the reverse-engineered layout exactly — meshes
draw as triangle LISTs (prim=4) with pos f32x3 @0, normal f16x4 @0x0C,
uv f16x2 @0x14 — and revealed the XBG7 vertex format is NOT fixed-stride:
models omit elements (stride 20 = pos+normal, no UV; 24 = pos+normal+uv).

- mesh.rs: parse the descriptor's vertex declaration ({offset, format-code,
  usage} triples; 0x2A23B9=pos f32x3, 0x1A2360=normal f16x4, 0x2C235F=uv
  f16x2) to drive per-model stride + element offsets, instead of assuming
  stride 24. Coverage 25 -> 36 fully-validated models (e.g. the Stage_S*
  props, which are pos+normal only). Same index-range + unit-normal safety
  gates; complex/mismatched layouts still declined.
- Endianness note (documented): the capture's fetch endian=k8in32 describes
  the GPU's guest-memory copy, NOT the .xpr file bytes — reading the file
  with k8in32 breaks the normals (|n|->1.33); naive big-endian per element
  is correct (the game rearranges vertex data on load).
- tests: a coverage-regression test (>=35 models) + the existing weapon /
  body-declined disc tests still pass.

Confirmed against a Canary draw-log capture; see docs/re/structures/
xbg7-mesh.md (evidence log + declaration table).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-12 18:42:21 +02:00
parent ef6e448268
commit 4096b2d2a5
4 changed files with 244 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files

193
crates/sylpheed-formats/src/mesh.rs
View File

@@ -17,23 +17,25 @@
//!
//! ## The "simple" layout decoded here (CONFIRMED)
//!
//! For single-stream models (weapons, simple props — ~40 of the 166 disc
//! models) the data section is a straight sequence of sub-meshes, each:
//! For single-stream models (weapons, simple props — 36 of the 166 disc models)
//! the data section is a straight sequence of sub-meshes, each:
//!
//! ```text
//! [ index buffer : idx_count × u16 big-endian ] triangle list
//! [ 12-byte vertex-buffer header (contents undecoded) ]
//! [ vertex buffer : vtx_count × 24 bytes ] (declaration below)
//! offset 0x00 POSITION : f32 × 3 big-endian (model units)
//! offset 0x0C NORMAL : f16 × 4 big-endian (x, y, z, w; unit)
//! offset 0x14 TEXCOORD : f16 × 2 big-endian (u, v)
//! [ vertex buffer : vtx_count × stride bytes ] (declaration-driven)
//! (pad to 16 bytes → next sub-mesh)
//! ```
//!
//! The per-vertex element offsets / usages come from a **vertex declaration**
//! in the descriptor (usage `0x00` POSITION, `0x03` NORMAL, `0x05` TEXCOORD),
//! identical across the decoded models. Correct alignment is pinned by the
//! recovered normals being exactly unit-length.
//! The vertex layout is **not fixed** — it comes from a **vertex declaration**
//! in the descriptor: a table of `{offset, format-code, usage}` triples (usage
//! `0x00` POSITION `f32×3`, `0x03` NORMAL `f16×4`, `0x05` TEXCOORD `f16×2`).
//! Models omit UV or use fewer elements, so stride varies (20 = pos+normal,
//! 24 = pos+normal+uv, …). Each element is read in naive big-endian component
//! order. Correct alignment is pinned by the recovered normals being exactly
//! unit-length. **Cross-checked against a Canary GPU vertex-fetch capture**,
//! which confirmed the primitive type (triangle list), formats, and offsets —
//! see `docs/re/structures/xbg7-mesh.md`.
//!
//! `vtx_count` / `idx_count` come from per-sub-mesh records in the descriptor:
//! a `[vtx_count:u32][0:u32][idx_count:u32][tail:u32]` tuple (big-endian), read
@@ -130,6 +132,13 @@ impl Xbg7Model {
let name = read_cstr(bytes, xbg.name_offset as usize + DIR_BASE)
.unwrap_or_else(|| "XBG7".to_string());
// Parse the vertex declaration (element offsets/formats + stride). The
// XBG7 layout is NOT fixed-stride — models omit UV or use fewer elements
// (stride 20 = pos+normal, stride 24 = pos+normal+uv, …). Confirmed
// against a Canary GPU vertex-fetch capture (see docs/re/xbg7-mesh.md).
let decl = parse_vertex_decl(&bytes[desc..desc_end])
.ok_or(MeshError::UnsupportedLayout)?;
// Extract the ordered list of sub-mesh (vtx_count, idx_count) records.
let records = submesh_records(&bytes[desc..desc_end]);
if records.is_empty() {
@@ -147,7 +156,7 @@ impl Xbg7Model {
// header (a normal length of exactly 1.0 pins this offset across
// every decoded model — `align`-based guesses landed 4 bytes early).
let vb = ie + VERTEX_BUFFER_GAP;
let ve = vb + vtx_count * VERTEX_STRIDE;
let ve = vb + vtx_count * decl.stride;
if ve > bytes.len() {
return Err(MeshError::UnsupportedLayout);
}
@@ -162,43 +171,53 @@ impl Xbg7Model {
indices.push(i);
}
// Vertex layout (stride 24), per the descriptor's vertex declaration
// (usage codes: 0x00 POSITION, 0x03 NORMAL, 0x05 TEXCOORD):
// +0x00 POSITION f32 × 3 (big-endian)
// +0x0C NORMAL f16 × 4 (x, y, z, w; use xyz — unit length)
// +0x14 TEXCOORD f16 × 2 (u, v)
// Vertices per the declaration. The .xpr stores each element in
// naive big-endian component order (f32 / f16 read at consecutive
// offsets) — the GPU's `k8in32` fetch endianness applies to the
// rearranged guest-memory copy, not to these file bytes.
let mut positions = Vec::with_capacity(vtx_count);
let mut normals = Vec::with_capacity(vtx_count);
let mut uvs = Vec::with_capacity(vtx_count);
let mut normal_len_sum = 0.0f32;
for v in 0..vtx_count {
let o = vb + v * VERTEX_STRIDE;
let x = bef(bytes, o);
let y = bef(bytes, o + 4);
let z = bef(bytes, o + 8);
let o = vb + v * decl.stride;
// POSITION: f32×3 big-endian.
let p = o + decl.pos_offset;
let x = bef(bytes, p);
let y = bef(bytes, p + 4);
let z = bef(bytes, p + 8);
if !(x.is_finite() && y.is_finite() && z.is_finite()) {
return Err(MeshError::UnsupportedLayout);
}
positions.push([x, y, z]);
let nx = half(bytes, o + 0x0C);
let ny = half(bytes, o + 0x0E);
let nz = half(bytes, o + 0x10);
normal_len_sum += (nx * nx + ny * ny + nz * nz).sqrt();
normals.push([nx, ny, nz]);
// NORMAL: f16×4 (use xyz).
if let Some(no) = decl.normal_offset {
let nb = o + no;
let nx = half(bytes, nb);
let ny = half(bytes, nb + 2);
let nz = half(bytes, nb + 4);
normal_len_sum += (nx * nx + ny * ny + nz * nz).sqrt();
normals.push([nx, ny, nz]);
}
let u = half(bytes, o + 0x14);
let vv = half(bytes, o + 0x16);
uvs.push([u, vv]);
// TEXCOORD: f16×2.
if let Some(uo) = decl.uv_offset {
let ub = o + uo;
uvs.push([half(bytes, ub), half(bytes, ub + 2)]);
}
}
// Sanity gate: a correctly-aligned vertex buffer in this layout has
// unit-length normals. If the mean is far off, the model does not
// fit the simple layout (wrong padding / different format) — decline
// rather than emit garbage. (Catches e.g. `Stage_S*` placeholders.)
let mean_normal_len = normal_len_sum / vtx_count.max(1) as f32;
if !(0.5..=2.0).contains(&mean_normal_len) {
return Err(MeshError::UnsupportedLayout);
// Sanity gate: when the declaration has a normal element, a correctly
// aligned vertex buffer yields unit-length normals. A mean far from 1
// means the layout does not fit (wrong stride / offset) — decline
// rather than emit garbage.
if decl.normal_offset.is_some() {
let mean = normal_len_sum / vtx_count.max(1) as f32;
if !(0.5..=2.0).contains(&mean) {
return Err(MeshError::UnsupportedLayout);
}
}
meshes.push(GameMesh {
@@ -217,13 +236,111 @@ impl Xbg7Model {
// ── Layout constants ────────────────────────────────────────────────────────
/// Bytes per vertex: `pos f32×3 (12) + normal f16×4 (8) + uv f16×2 (4)`.
const VERTEX_STRIDE: usize = 24;
/// Fixed byte gap between the end of a sub-mesh's index buffer and the start of
/// its vertex buffer (a small vertex-buffer header — contents not yet decoded).
const VERTEX_BUFFER_GAP: usize = 12;
// ── Vertex declaration ───────────────────────────────────────────────────────
/// The vertex layout for one XBG7 resource, parsed from the descriptor's
/// declaration table (shared by all its sub-meshes).
struct VertexDecl {
/// Bytes per vertex.
stride: usize,
/// Byte offset of the POSITION element (`f32×3`) within a vertex.
pos_offset: usize,
/// Byte offset of the NORMAL element (`f16×4`), if present.
normal_offset: Option<usize>,
/// Byte offset of the TEXCOORD element (`f16×2`), if present.
uv_offset: Option<usize>,
}
/// Known element format codes → element size in bytes (from the GPU capture:
/// POSITION `f32×3`, NORMAL `f16×4`, TEXCOORD `f16×2`).
fn decl_code_size(code: u32) -> Option<usize> {
match code {
0x2A_23B9 => Some(12), // f32×3 (POSITION)
0x1A_2360 => Some(8), // f16×4 (NORMAL)
0x2C_235F => Some(4), // f16×2 (TEXCOORD)
_ => None,
}
}
/// Parse the XBG7 vertex declaration: a table of `{offset:u32, code:u32,
/// usage<<16:u32}` big-endian triples that follows the `(index_bytes,
/// index_count)` marker, terminated by an `offset == 0x00FF0000` /
/// `code == 0xFFFFFFFF` sentinel. Usage codes: `0` POSITION, `3` NORMAL,
/// `5` TEXCOORD. Stride is the max element extent; unknown element sizes are
/// inferred from the next element's offset.
fn parse_vertex_decl(desc: &[u8]) -> Option<VertexDecl> {
// Locate the (index_bytes, index_count) marker: index_bytes == count*2.
let mut mk = None;
let mut rel = 0usize;
while rel + 40 <= desc.len() {
let a = be32(desc, rel);
let c = be32(desc, rel + 4);
if c >= 3 && c % 3 == 0 && c < 200_000 && a == c * 2 {
mk = Some(rel);
break;
}
rel += 4;
}
let mk = mk?;
// Read declaration triples.
let mut elems: Vec<(usize, u32, u32)> = Vec::new(); // (offset, code, usage)
let mut r = mk + 8;
for _ in 0..16 {
if r + 12 > desc.len() {
break;
}
let off = be32(desc, r);
let code = be32(desc, r + 4);
let usage = be32(desc, r + 8) >> 16;
if off == 0x00FF_0000 || code == 0xFFFF_FFFF {
break;
}
if off as usize > 0x1000 {
break; // out-of-range offset — not a real element
}
elems.push((off as usize, code & 0x00FF_FFFF, usage));
r += 12;
}
if elems.is_empty() {
return None;
}
let mut stride = 0usize;
for (i, &(off, code, _)) in elems.iter().enumerate() {
let size = decl_code_size(code).unwrap_or_else(|| {
if i + 1 < elems.len() {
elems[i + 1].0.saturating_sub(off)
} else {
4
}
});
stride = stride.max(off + size);
}
if stride == 0 || stride > 256 {
return None;
}
let pos_offset = elems
.iter()
.find(|&&(_, c, u)| c == 0x2A_23B9 || u == 0)
.map(|&(o, _, _)| o)
.unwrap_or(0);
let normal_offset = elems.iter().find(|&&(_, _, u)| u == 3).map(|&(o, _, _)| o);
let uv_offset = elems.iter().find(|&&(_, _, u)| u == 5).map(|&(o, _, _)| o);
Some(VertexDecl {
stride,
pos_offset,
normal_offset,
uv_offset,
})
}
// ── Descriptor sub-mesh record scan ─────────────────────────────────────────
/// Scan an XBG7 descriptor for the ordered list of per-sub-mesh

36
crates/sylpheed-formats/tests/mesh_disc.rs
View File

@@ -66,6 +66,42 @@ fn weapon_model_decodes_to_expected_geometry() {
);
}
#[test]
#[ignore = "requires extracted disc models — set SYLPHEED_RES3D"]
fn declaration_driven_decode_covers_expected_model_count() {
let Some(dir) = res3d_dir() else {
return;
};
let mut ok = 0usize;
let mut total = 0usize;
for entry in std::fs::read_dir(&dir).unwrap() {
let path = entry.unwrap().path();
if path.extension().and_then(|e| e.to_str()) != Some("xpr") {
continue;
}
total += 1;
let bytes = std::fs::read(&path).unwrap();
if let Ok(model) = Xbg7Model::from_xpr2(&bytes) {
if !model.meshes.is_empty() {
// Every decoded model must be self-consistent (indices in range,
// and unit normals where present).
for m in &model.meshes {
assert!(
m.indices.iter().all(|&i| (i as usize) < m.positions.len()),
"{path:?}: index out of range"
);
}
ok += 1;
}
}
}
eprintln!("XBG7 declaration-driven decode: {ok}/{total} models");
// Variable-stride declaration parsing lifted coverage vs the old fixed
// stride-24 decoder (25 → 36 fully-validated models; multi-sub-mesh models
// whose later sub-mesh offset isn't yet handled are still declined whole).
assert!(ok >= 35, "coverage regressed: only {ok}/{total} decoded");
}
#[test]
#[ignore = "requires extracted disc models — set SYLPHEED_RES3D"]
fn complex_body_mesh_is_declined_not_garbage() {

2
docs/re/INDEX.md
View File

@@ -19,7 +19,7 @@ Promote to a prose `structures/…md` file when a format needs behavioural notes
| LSTA sprite list | 🟡 | `sylpheed-formats/src/lsta.rs` | inline T8aD frames |
| IXUD subtitle | 🟡 | `sylpheed-formats/src/ixud.rs` | timed cues; **movie↔track link unknown** (dynamic item) |
| Fonts (ttf/otf/ttc) | ✅ | `sylpheed-formats/src/font.rs` | standard OpenType, parsed via ttf-parser |
| XBG7 mesh | 🟡/❔ | `sylpheed-formats/src/mesh.rs` + `tests/mesh_disc.rs` ([xbg7](structures/xbg7-mesh.md)) | single-stream layout decoded (~25 models, pos+normal+UV via vertex declaration); complex body layout + other vertex layouts declined |
| XBG7 mesh | 🟡/❔ | `sylpheed-formats/src/mesh.rs` + `tests/mesh_disc.rs` ([xbg7](structures/xbg7-mesh.md)) | single-stream, declaration-driven variable stride (36 models); triangle-list + formats GPU-confirmed; multi-submesh + body meshes still declined |
## Functions / code paths

81
docs/re/structures/xbg7-mesh.md
View File

@@ -38,49 +38,63 @@ Read in file order by a sliding 4-byte scan; each is a big-endian tuple:
(For `rou_f001_wep_00`: `vtx_count=215`, `idx_count=1092` — matches the recovered geometry exactly.)
## The single-stream data layout — 🟡 PROBABLE (decoded)
## The single-stream data layout — 🟡 PROBABLE (decoded, GPU-cross-checked)
For ~25 of the 166 models (weapons `rou_f001_wep_00/02/03/04/09/…`) the data section is a straight
sequence of sub-meshes, carved from `header_size` in record order:
For 36 of the 166 models (weapons, simple props) the data section is a straight sequence of
sub-meshes, carved from `header_size` in record order:
```
per sub-mesh:
[ index buffer : idx_count × u16 BE ] triangle list
[ index buffer : idx_count × u16 BE ] triangle list (prim=4, GPU-confirmed)
[ 12-byte vertex-buffer header (contents undecoded) ]
[ vertex buffer : vtx_count × 24 bytes ] ← declaration below
[ vertex buffer : vtx_count × stride bytes ] ← declaration-driven
(pad to 16 bytes → next sub-mesh)
```
**Vertex declaration** (from the descriptor; usage codes `0x00` POSITION, `0x03` NORMAL,
`0x05` TEXCOORD — identical across decoded models):
**Vertex declaration.** The layout is **not fixed-stride**. The descriptor holds a declaration table
(right after the `(index_bytes, index_count)` marker) of `{offset:u32, format-code:u32, usage<<16:u32}`
big-endian triples, terminated by `offset == 0x00FF0000` / `code == 0xFFFFFFFF`:
```
+0x00 POSITION f32 × 3 BE model units
+0x0C NORMAL f16 × 4 BE (x, y, z, w); use xyz — unit length
+0x14 TEXCOORD f16 × 2 BE (u, v)
```
| usage | element | format code | format | size |
|-------|----------|-------------|--------|------|
| 0x00 | POSITION | `0x2A23B9` | f32×3 | 12 B |
| 0x03 | NORMAL | `0x1A2360` | f16×4 (use xyz) | 8 B |
| 0x05 | TEXCOORD | `0x2C235F` | f16×2 (u,v) | 4 B |
**Alignment is pinned by the normals.** The vertex buffer starts at `index_end + 12` (a fixed
12-byte header — NOT `align16`, which lands 4 bytes early on most models and silently corrupts every
field). The correct offset is the unique one where the recovered normals are exactly unit-length; an
`align`-based guess produced the small stray-triangle artefacts seen in the first viewer pass.
Stride = max element extent. Models **omit elements** → variable stride (20 = pos+normal, no UV;
24 = pos+normal+uv). Each element is read in **naive big-endian component order** (the raw file
bytes; see the endianness note below). Assuming a fixed stride-24 was why the old decoder mis-aligned
and declined the pos+normal-only models.
**Safety gate:** every index is validated `< vtx_count`, **and** the mean recovered `|normal|` must be
≈1 (in `[0.5, 2.0]`). A model failing either is rejected (`MeshError::UnsupportedLayout`) rather than
emitting garbage — this correctly declines `Stage_S*` placeholders and models using a different
vertex layout.
**Alignment pinned by the normals.** The vertex buffer starts at `index_end + 12` (a fixed 12-byte
header — NOT `align16`, which lands 4 bytes early on most models). The correct offset is the unique
one where recovered normals are exactly unit-length.
**Safety gate:** every index is validated `< vtx_count`, and when the declaration has a normal
element the mean recovered `|normal|` must be ≈1 (`[0.5, 2.0]`). A model failing either is rejected
(`MeshError::UnsupportedLayout`) rather than emitting garbage.
### Endianness — file bytes are naive-BE, `k8in32` is a red herring ✅
The Canary GPU capture reports every vertex stream with fetch `endian = 2` (`k8in32`, each 32-bit
word byte-reversed). This describes the **guest-memory** copy the GPU fetches — **not** the `.xpr`
file bytes. Reading the file with a `k8in32` transform breaks the normals (mean `|normal|` → 1.33);
the plain per-element big-endian read yields exactly unit normals. So the game **rearranges** the
vertex data between the on-disc `.xpr` and the uploaded buffer; the decoder reads the file directly
and must use naive BE.
## Not yet decoded — ❔ the complex body layout
The hero-ship *body* meshes (`DeltaSaber_A/_T/_W.xpr` resource `f004`, and ~100 other models) use a
**multi-stream** layout: the data section does **not** start with an index buffer; the real
geometry sits at descriptor-addressed offsets deep in a multi-MB data section, as **separate
position / attribute streams** (a plain `f32×3` position stream at stride 12 was located at a high
offset, with attributes in other streams), and positions appear partly quantized. `DeltaSaber_A`'s
5 XBG7 blocks are `f004` (body) + `_rou_f004_mnv01_L/_R`, `_mnv02`, `_turn180` (maneuver / pose
variants). Decoding this requires parsing the descriptor's vertex-declaration + stream table
(offsets `0x0C`/`0x14` with format codes were seen but not resolved). These models are cleanly
declined today.
The hero-ship *body* meshes (`DeltaSaber_A/_T/_W.xpr` resource `f004`, and ~100 other models) still
decline. Two open sub-problems: (1) **multi-sub-mesh models** whose *first* sub-mesh decodes but a
later one's inter-mesh offset isn't yet handled (the `align16` advance is a guess) — these are
declined whole; (2) the big body meshes, where the data section does not start with an index buffer
and the geometry sits at descriptor-addressed offsets. NB the GPU capture showed **every** rendered
mesh is *single-stream* (just wider strides, e.g. 44 bytes = pos + f32×3 + colour + 2×f16×4), so the
body is likely single-stream-with-a-richer-declaration rather than the "separate streams" first
guessed — it was simply not rendered in the captured session (menu only). A capture taken *in a
mission* (where `DeltaSaber` renders) would hand over its exact declaration directly. `DeltaSaber_A`'s
5 XBG7 blocks are `f004` (body) + `_rou_f004_mnv01_L/_R`, `_mnv02`, `_turn180` (maneuver / pose).
## Evidence log
@@ -98,6 +112,15 @@ declined today.
gives median `|normal| = 1.000` on every weapon model (`wep_00/02/03/04`), vs `align16` landing 4
bytes early. UVs then land in `[0,1]`. Adding the normal gate: **25 models decode clean +
normal-valid** (the rest — incl. `Stage_S*` degenerate blobs — correctly declined).
- 2026-07-12 (DYNAMIC) — added a cvar-gated draw logger to Canary
(`command_processor.cc::LogDrawForRE`, cvar `log_draws`), captured the Ready Room / Briefings.
**GPU ground truth confirmed the static layout exactly**: primitive `prim=4` = **triangle LIST**
(settles the list-vs-strip question), and a stream with `f32x3 @offset0` + `f16x4 @3dw` +
`f16x2 @5dw`, stride 6 dwords = 24 bytes — matching `POSITION@0, NORMAL@0x0C, TEXCOORD@0x14`.
Revealed **stride varies** (24, 20, 28, 44 …) and that all streams are **single-stream** →
parsed the declaration for variable stride: coverage **25 → 36** (e.g. `wep_05` is pos+normal,
stride 20, no UV — previously mis-aligned). Also confirmed the endianness note above: fetch
`endian=2` (k8in32) is the *guest* copy; file stays naive-BE. `Stage_S*` now decode (stride 20).
- 2026-07-12 — `DeltaSaber_A.xpr` body: data does **not** begin with indices; plain-`f32×3` runs
with ship-scale extent (span ≈27–34, matching bbox 30.0) found only at high offsets
(`data+0x28634C`, …) → multi-stream, **undecoded**.