diff --git a/docs/re/INDEX.md b/docs/re/INDEX.md index 2cdb80d8..3443212a 100644 --- a/docs/re/INDEX.md +++ b/docs/re/INDEX.md @@ -94,7 +94,7 @@ files, which is how the same ground got covered twice. | [`idxd-legacy-reader-audit.md`](idxd-legacy-reader-audit.md) | The legacy IDXD string-pool reader vs the real field table β€” what the old numbers got wrong | 🟑 shape CONFIRMED by hand (`FCSRange`, `ShieldRatio`, hangar `Model`); disc-wide rates are single-source | | [`structures/idxd-container.md`](structures/idxd-container.md) | The IDXD/IXUD container β€” record/field table, and the two beliefs it withdraws | βœ… CONFIRMED disc-wide, 7 750/7 750 objects and 1 271 462/1 271 462 named fields, zero failures | | [`structures/hud-glyph-quad.md`](structures/hud-glyph-quad.md) | The HUD's glyph quad β€” vtable `0x820B2A64` | βœ… CONFIRMED for the object layout and the atlas size, read live off | -| [`structures/slb-data-offset.md`](structures/slb-data-offset.md) | `.slb` leading-stream offset is `first_riff % 2048`, not the constant 1392 | βœ… CONFIRMED by decoding β€” 85 of 140 sampled banks yield more audio (median 70Γ—), 54 identical controls; offset recovery for `RIFF`-less banks is 99.95 % on the labelled set | +| [`structures/slb-data-offset.md`](structures/slb-data-offset.md) | `.slb` data offset = `(cumulative start of the `.pNN` segment) mod 2048`; the leading bytes are the previous bank's audio, not a header | βœ… CONFIRMED β€” exact for 8 783/8 783 banks, 0 mismatches; the four values are the running sums of the five segment sizes mod 2048; supersedes the earlier scan/`seek` heuristics | | [`structures/sound-pak-contents.md`](structures/sound-pak-contents.md) | Census of `sound.pak`, and the limit of the leading-region rule | βœ… CONFIRMED, 5 135/5 135 names hash into the TOC and **9 519/9 519** entries accounted for; ⚠️ leading-region rule holds for 1 571/4 382 eng and 0/5 100 jpn | | [`structures/sound-cue-table.md`](structures/sound-cue-table.md) | The cue index in `tables.pak` β€” message id -> cue -> sound id -> `.slb` bank | βœ… CONFIRMED, 1 326/1 338 script message ids bind to a bank; SOUNDS and FILES agree on the same 12 absentees, 0 orphan files | | [`structures/cutscene-message-table.md`](structures/cutscene-message-table.md) | Cutscene dialogue β€” speaker, portrait, on-screen seconds, audio cue per page | βœ… CONFIRMED, field count = 9Β·PageCount+2 for all 7 PageCounts, 1 252/1 252 caption keys match, 138 ids close both ways | diff --git a/docs/re/structures/slb-data-offset.md b/docs/re/structures/slb-data-offset.md index 16fb743a..5b044cc7 100644 --- a/docs/re/structures/slb-data-offset.md +++ b/docs/re/structures/slb-data-offset.md @@ -1,5 +1,14 @@ # `.slb` leading-stream data offset β€” 1392 was never a constant +> ⚠️ **Read the last section first.** A second pass on 2026-08-26 closed the ❔ +> at the bottom of the original writeup: the offset is +> `(cumulative start of the `.pNN` segment holding the wave) mod 2048`, exact for +> 8 783 of 8 783 banks. The `first_riff % 2048` rule below is a *consequence* of +> that and stays correct where a `RIFF` exists; the packet-plausibility scan and +> the `seek`-residue heuristic are superseded, and the 🟑 "most banks declare more +> `data` than they store" finding is **withdrawn**. Jump to +> [*X is the `.pNN` segment's grid phase*](#-settled-2026-08-26-second-pass--x-is-the-pnn-segments-grid-phase). + **βœ… Settled 2026-08-26, verified by decoding.** A bank's leading headerless packet stream does not start at a fixed offset. It starts at **`first_riff % 2048`**. `HEADERLESS_DATA_OFFSET = 1392` is the value that @@ -256,3 +265,295 @@ that opens a bank by name and seeks to its data; the constant, or the table it indexes, should be visible there. That is static PE work (`/work/*.pe`, offset = VA βˆ’ 0x82000000), not another pass over the archive β€” this page has taken the byte-level evidence about as far as it goes. + +--- + +# βœ… SETTLED 2026-08-26 (second pass) β€” X is the `.pNN` segment's grid phase + +The open ❔ above ("why the offset takes those four values, and what is in those +bytes") is now closed, exactly, with no residual heuristic: + +> **`X = (cumulative start of the `.pNN` segment file that holds the wave) mod 2048.`** + + segment size size % 2048 cum_start X + sound.p00 267 930 992 1392 0 0 + sound.p01 268 404 812 76 267 930 992 1392 + sound.p02 268 404 868 132 536 335 804 1468 + sound.p03 268 384 384 128 804 740 672 1600 + sound.p04 14 903 296 0 1 073 125 056 1728 + +**The four values are the running sums of the segment sizes mod 2048.** +`1392 = |p00| mod 2048`; `1468 = 1392 + 76`; `1600 = 1468 + 132`; +`1728 = 1600 + 128`. That identity is not a fit β€” it is arithmetic, and it comes +out of five file sizes that nothing in this analysis chose. + +Reproduce with `tools/re-capture/slb_segment_phase.py phases`. + +## Why + +The XMA1 packet grid is 2048-aligned **inside each individual `.pNN` file**: a +segment file starts at its own offset 0 with a bank header and everything after +is on that file's own 2048 grid. But the `.pak` **TOC addresses entries in the +flat *concatenation*** of `p00..p04`, and every TOC offset is itself a multiple +of 2048 (**checked: 0 of 9 519 entries are misaligned**). The segment files are +*not* multiples of 2048 long, so each join shifts the grid by +`size % 2048`, and an entry inside segment *k* sees the accumulated shift. + +Nothing in the game computes 1392. It is a **build-time artifact of where the +packer chose to cut a ~1.07 GB stream into five ~256 MB files.** + +### Verification β€” 100 %, no exceptions + +`tools/re-capture/slb_segment_phase.py verify`: + +| | banks | agree | disagree | +|---|---|---|---| +| has a `RIFF/WAVE` β€” X read off the wave header | 7 620 | **7 620** | **0** | +| `RIFF`-less β€” X read off the leading `seek` chunk | 1 163 | **1 163** | **0** | + +This **replaces both heuristics** in the section above. The packet-plausibility +scan (99.62 %) and the `seek`-residue rule (99.97 %, combined 99.95 %) are no +longer needed for anything: the offset is now *derivable* for every bank, +including the 1 495 `RIFF`-less ones and including the 28 ties the scan could +not break. Those sections stand as an honest record of how the number was +narrowed, not as the recommended method. + +### The prediction that could have refuted it, and did not + +If X really is a per-segment property rather than a per-bank one, then an entry +that **straddles** a segment boundary must show **two different phases inside +one file**. Exactly 3 of 9 518 entries straddle, and they do: + + eng\Voice\VOICE_ACRO_010.slb window starts 11 708 B before the p01β†’p02 join + seek chunk @11 632 ≑ 1392 (mod 2048) <- p01 phase + ---- sound.p01 / sound.p02 join at 11 708 ---- + RIFF @21 948 ≑ 1468 (mod 2048) <- p02 phase + seek @52 668 ≑ 1468 (mod 2048) + + jpn\etc\VOICE_D_149.slb (1468 -> 1600) jpn\Voice\VOICE_TCAF_577.slb (1600 -> 1728) + +A single-offset-per-bank model cannot produce that. ⚠️ It also means **"the data +offset of a bank" is not well defined for those three entries** β€” any decoder +that stores one offset per file will decode part of them mid-packet. + +## βœ… What is in those X bytes: audio, from the *previous* bank + +Not a header. They are the tail of the preceding bank's XMA1 packet stream, +carried into this window because the TOC window boundary and the bank boundary +are different things. + +Three independent lines: + +1. **Statistics are identical to known audio.** Over 167 `eng\etc` banks, the + number of distinct byte values seen at each fixed offset averages **101.06** + across `[0, 1392)` and **101.90** across `[1392, 3440)` β€” a region that is + certainly one XMA packet. Not one position in `[0, 3600)` takes ≀8 distinct + values, i.e. **there is no fixed field anywhere in the header**. A real + header would show constants. + +2. **The tiling arithmetic is exact.** A wave's `seek` (XMA `dpds`) chunk sits + immediately after its data and holds one `u32 LE` per packet, so it names its + own packet count and therefore where its data started. Entry *K*'s last wave + overruns the entry; entry *K+1* opens with a `seek` whose packet count + matches, at exactly the overrun distance measured **past the TOC window + rounded up to 2048**: + + VOICE_D_451 window 67 704 -> padded 69 632 last wave ends at 73 072 + VOICE_D_452 leading seek @ 3 440 (14 packets) 73 072 - 69 632 = 3 440 βœ… + VOICE_D_452 window 67 704 -> padded 69 632 last wave ends at 103 792 + VOICE_D_453 leading seek @34 160 (20 packets) 103 792 - 69 632 = 34 160 βœ… + + `tools/re-capture/slb_segment_phase.py chain 'eng\etc\VOICE_D_450.slb' 4`. + +3. **The bytes the TOC skips are audio too.** Between the end of + `VOICE_D_451`'s stored 67 704 bytes and the start of `VOICE_D_452` sit 1 928 + bytes of `.pNN` that no entry claims. **1 903 of the 1 928 are non-zero** β€” + dense audio, not padding. The stream is continuous through them. + +### ❌ Withdrawn: "the header is unique per bank, so it is not shared data" + +An earlier step here searched all 1 088 MB of `p00..p04` for one bank's exact +1 392-byte header and found **one** occurrence, its own β€” and I briefly read +that as ruling out any shared-stream model. It does not. The windows **tile**, +they do not overlap: the bytes appear once because they are stored once. The +measurement was right, the inference from it was wrong. + +## βœ… The real `.slb` bank layout (this is what the entries contain) + +Each bank is self-contained and starts on its segment's 2048 grid. All fields +big-endian. + + +0x00 u32 bank / cue id (BGM_001 -> 1001, BGM_105 -> 1105; voice ids run + consecutively in stream order, e.g. 7228, 7229, …) + +0x04 u32 0x11 (17) constant on every bank seen + +0x08 u32 0x20 (32) constant + +0x0C u32 1 constant + +0x10 u32 0x48 (72) constant + +0x14 u32 0 + +0x18 u32 0x800 BLOCK SIZE β€” the 2048 alignment unit + +0x1C u32 data size in bytes ❔ does not equal the sum of the waves; see below + +0x20 u32 bank id again + +0x24 u32 header size in BLOCKS always 5 -> the first wave is at +10240 + +0x28 u16 bits per sample (16) u16 channels (2 for BGM, 4 for voice banks) + +0x2C f32 } three floats, 1.0 / 0.5 / 0.1 on BGM_001, 1.0 / 0.4 on BGM_105 + +0x30 f32 } ❔ volume / mix, not identified + +0x34 f32 } + ... zero fill to 0x2800 (10240) + +then, repeated per wave: + + RIFF/WAVE exactly 4096 bytes = 2 blocks: + 'fmt ' 32 bytes standard little-endian XMAWAVEFORMAT + 'Dmmy' 4028 bytes of zero pad, so that… + 'data' …audio begins at riff+4096, on the grid + size is EXACTLY packets * 2048 + 'seek' 8 + 4*packets u32 LE cumulative decoded-sample counts + +`+0x18 == 0x800` plus `bank[0x00] == bank[0x20]` is a reliable signature: scanned +at the correct phase it finds every bank and nothing else. + +`+0x24` is always 5 on this disc, so **a bank's own audio starts 10 240 bytes +after its header** β€” but that header is generally *not* at offset 0 of the pak +entry (see the next section), which is why the value never showed up as a +constant. + +### The `fmt ` chunk decodes cleanly as XMA1 `XMAWAVEFORMAT` + +`eng\etc\VOICE_D_452.slb` first wave: tag `0x0165`, 16 bits, `NumStreams` 1, +`LoopCount` 0, `Version` 2, `PsuedoBytesPerSec` 12 212, `SampleRate` 48 000, +loop 0..0, `Channels` 1, `ChannelMask` 1 β€” mono. `BGM_105.slb`: +`LoopCount` 0xFF, `PsuedoBytesPerSec` 35 879, 48 000 Hz, `LoopStart` 0x003D72B1, +`LoopEnd` 0x0168BDBA, `Channels` 2. So the format was never in doubt; only the +framing was. + +## ❌ Withdrawn: "🟑 Most banks declare more `data` than they store" + +The section above reports that 5 296 of 7 586 banks (69.8 %) declare a `data` +size larger than the entry holds, and calls the decoder's "honest per sub-wave" +comment a documentation defect. **The measurement was right and the reading was +wrong β€” and the comment it accused was correct.** + +Declared `data` sizes are exact. The bytes are simply *outside the TOC window*, +still present in the `.pNN` stream. Taking 400 random `eng\`/`jpn\` banks and, +for every last wave that overruns its window, reading the declared range +straight from the segment stream: the wave's `seek` chunk lands at exactly the +declared end in **260 of 260** cases, 0 failures. + +So `eng\Voice\VOICE_TCAF_608.slb` is **not** "truncated, 99 % short", and that is +not why it decodes badly β€” its wave continues into the next window. The +counterexample section above needs revisiting on that basis. + +`data` size is always an exact multiple of 2048, which is the packet count. + +## ❔ Still open β€” the TOC window is not the bank + +The `.pak` TOC entry for a cue is a window that **contains** the cue's bank but +is not aligned to it, and the offset drifts entry to entry: + + eng\etc\VOICE_D_450.slb bank ids 7212 @ 7 536, 7226 @ 60 784 + eng\etc\VOICE_D_451.slb bank id 7228 @ 30 064 + eng\etc\VOICE_D_452.slb bank ids 7229 @ 5 488, 7230 @ 48 496 + eng\etc\VOICE_D_453.slb bank id 7231 @ 36 208 + eng\etc\VOICE_D_454.slb bank id 7232 @ 50 544 + +Bank ids run consecutively in pak-offset order, so a nameβ†’bank mapping exists, +but **which bank in a window belongs to the entry's name is not established** β€” +some windows hold two. A window can also cut a bank in half. Since a bank header +can only sit at `≑ X (mod 2048)` while a TOC offset is `≑ 0`, **a bank header can +never be at entry offset 0**; the leading region is structural, not accidental. + +❔ `+0x1C` (data size) does not match the sum of the wave `data` chunks β€” +`VOICE_D_452`'s bank 7229 declares 38 980 for a 26 624-byte wave. Unexplained. + +## 🟑 The loader β€” found, mapped, and it does *not* contain X (null result) + +Static search of `default.xex` (`/work/*.pe`, `/work/xenia-rs/sylpheed.db`), +done as the deliberate refutation attempt: if some code computes X, it should be +visible. **It is not, and under the segment-phase explanation it should not be.** + +**What was searched, all negative:** + +* **1392 / 1468 / 1600 / 1728 as immediates** (`li`/`addi`/`subi`/`cmpwi`/ + `cmplwi`/`ori`/`lis`, and the negations) over all 25 481 functions: 15 / 1 / 7 / + 10 hits, every one accounted for and none audio-related. Only two functions + hold β‰₯2 of the values and both are `subi rN,r1,K` / `addi r1,rN,K` stack-frame + pairs (`sub_82766DB0`, `sub_821DB270`, both D3D/shader-compiler code). + The most promising-looking hit β€” `cmplwi cr6,r31,0x570` at **0x8217FC68**, + next door to the sound manager β€” is **`ERROR_FILE_CORRUPT`**, in a run of + `0x7B`/`0x5`/`0xB7`/`0x48F` = `ERROR_INVALID_NAME`/`ACCESS_DENIED`/ + `ALREADY_EXISTS`/`DEVICE_NOT_CONNECTED`: storage-device retry code, not sizes. + `li r7,1468` at 0x827D3BC4 is a `__LINE__` for an HLSL-compiler assert. +* **A table of the four values** anywhere in the 9 568 256-byte image: 32-bit BE + aligned and unaligned, 32-bit LE, 16-bit both endians, and `float32`. Zero + clusters holding β‰₯2 of the four. The **only** 4-byte-aligned BE occurrence of + any of them in the whole image is one `1600` at 0x820489A0, in a zlib-adjacent + globals blob. `0x8202D668` looks like a hit (`… 0570 05BC …`) but is a + compiler message-offset table β€” it continues `0x664`, `0x6B8`, not `0x640`, + `0x6C0` β€” and is immediately followed by `"internal error: unknown "`. +* **`.slb` / `slb` / `XACT` / `XWB` / `xma` / `wavebank` as strings** β€” absent + from the image in ASCII and UTF-16. No 4CC-shaped immediates in the audio code. + +**The sound subsystem, for whoever picks this up next** (all located by residual +`__FILE__`/debug-`printf` strings β€” the image has **no user symbols at all**: +25 310 of 25 481 functions are `sub_XXXXXXXX`, RTTI stripped, `demangled_names` +empty): + +| range / address | what | +|---|---| +| 0x82175E00–0x8217DC00 | `silph::SoundManager` game layer | +| **0x821774A0** | BGM bank-slot manager β€” holds both the `BankSlots::get_bgm_data_area` (0x820A18B0) and `BankSlots::setup_bgm` (0x820A1938) failure strings | +| 0x82179988 | `SoundManager::Impl::SetMovieMode`, called from the movie handler 0x821B4968 | +| 0x82178F60 | sound config loader β€” refs `SOUNDS`/`SETTINGS`/`game:\` at 0x820A1878 | +| 0x82604A00–0x8260E900 | `gsfw` sound_framework middleware, 383 functions; public entries 0x826069E8 (play/register, `r3=15`), 0x82606A38 (stop/release), 0x826062D8, 0x82605028; assert sites 0x82608120, 0x8260DF28, 0x8260D8C0, 0x8260D9D0 β†’ the `gsfw_object.h` / `gsfw_objectcore.cpp` strings | +| 0x824D1000–0x824DC400 | XMA/XAudio driver β€” the only kernel audio imports (`XMACreateContext` wrapper 0x824D3BA8, `XAudioRegisterRenderDriverClient` 0x824DC280, …) | + +The one size-related constant on the bank path is in **`sub_821774A0`**: a +128 KB (`0x20000`) threshold on the loaded resource size, then +`addi r4,r29,2047` / `addi r11,r3,2047` / `clrrwi r11,r11,11` β€” allocate +`size + 2047` and round the pointer **up to 2048** β€” stored at `+76` of the slot +struct and handed to a gsfw vtable slot. So the runtime **does** honour the 2048 +grid; it just never needs to name 1392, because the 2048-alignment it applies is +to its own buffer, and the phase is baked into the file at pack time. + +🟑 rather than βœ… because **the walk from "a bank is opened" to "the first packet +is submitted" was not completed** β€” `sub_821774A0` is the BGM path, and no code +was traced that consumes a voice `.slb`. The claim proven here is the narrower +one: none of the four values exists as a constant or a table anywhere in the +executable. That is consistent with, but does not by itself prove, "the game +never needs X". + +## Consequences for our decoder + +`crates/sylpheed-formats/src/slb.rs` currently takes the offset from +`first_riff % 2048` (right, where a `RIFF` exists) and from `scan_data_offset` +otherwise (99.6 % right). Both can be replaced by the exact rule, which needs +one new input: **the pak offset of the entry, plus the segment start table** β€” +`PakArchive` already knows both. Suggested shape: + +* `PakArchive` exposes `segment_phase(concat_offset) -> usize`. +* `slb::to_xma_riff` takes the phase instead of deriving it. +* the 3 straddling entries need the phase looked up **per wave**, not per file. + +⚠️ Also worth revisiting: because the declared `data` sizes are honest and the +window is not the bank, the reader currently clamps away audio that is really +there. Reading a cue's full audio means reading past the TOC entry into the +`.pNN` stream. + +Not implemented here β€” this pass was static RE only, and the API change reaches +every caller. + +## Evidence log + +* 2026-08-26 β€” `X = segment_cum_start % 2048`. `CONFIRMED`. 7 620/7 620 banks + with a `RIFF` and 1 163/1 163 `RIFF`-less banks via `seek`, 0 mismatches; + independently, the four values are the running sums of the five `.pNN` sizes + mod 2048; independently, the 3 straddling entries show the phase step *inside* + one file at the segment join. `tools/re-capture/slb_segment_phase.py`. +* 2026-08-26 β€” leading X bytes are the previous bank's audio. `CONFIRMED`. + Byte-diversity identical to a known packet (101.06 vs 101.90, no fixed field); + `seek` packet counts chain exactly across three consecutive entries; the + unclaimed inter-entry `.pNN` bytes are 1 903/1 928 non-zero. +* 2026-08-26 β€” declared `data` sizes are exact, 260/260 checked. **Demotes** the + 🟑 "most banks declare more than they store" reading above. +* 2026-08-26 β€” no `.slb` data offset exists in `default.xex`. `PROBABLE` + (exhaustive constant/table/string search; the loader walk itself is unfinished).