From 2aa7f9a5a5d3c77bbec7f0f84f3306beb69e38cd Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 05:01:26 +0000 Subject: [PATCH] re: the .slb data offset is the .pNN segment's 2048 grid phase MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the open question at the bottom of slb-data-offset.md. X = (cumulative start of the .pNN segment holding the wave) mod 2048 The XMA packet grid is 2048-aligned inside each individual segment file, but the .pak TOC addresses entries in the flat concatenation at offsets that are themselves multiples of 2048. The segment files are not multiples of 2048 long, so each join shifts the grid by size % 2048 β€” and the four disc-wide values are exactly the running sums: 1392 = |p00| % 2048; 1468 = +76; 1600 = +132; 1728 = +128 Exact for 7620/7620 banks with a RIFF and 1163/1163 RIFF-less ones via their seek chunk, 0 mismatches. Supersedes both heuristics (the 99.62 % packet scan and the 99.97 % seek-residue rule) and dissolves the 28 ties. The refutation test β€” an entry straddling a segment join must show two phases in one file β€” passes on all 3 straddlers. The leading bytes are the previous bank's audio, not a header: byte diversity per offset is indistinguishable from a known packet (101.06 vs 101.90, no fixed field anywhere), the seek packet counts chain exactly across consecutive entries, and the inter-entry bytes no TOC entry claims are 1903/1928 non-zero. Also recorded: the real bank header layout (id, block size 0x800, header size in blocks, XMAWAVEFORMAT), and the loader search β€” a null result. None of the four values exists as an immediate, a table or a float anywhere in default.xex, which is what a pack-time artifact predicts. Sound subsystem addresses mapped for the next pass. Withdraws the 🟑 "most banks declare more data than they store" finding: declared data sizes are exact (260/260), the bytes are just outside the TOC window. Also withdraws my own "the header is unique, so nothing is shared" inference β€” the windows tile, they do not overlap. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE --- docs/re/INDEX.md | 2 +- docs/re/structures/slb-data-offset.md | 301 ++++++++++++++++++++++++++ 2 files changed, 302 insertions(+), 1 deletion(-) diff --git a/docs/re/INDEX.md b/docs/re/INDEX.md index 2cdb80d8..3443212a 100644 --- a/docs/re/INDEX.md +++ b/docs/re/INDEX.md @@ -94,7 +94,7 @@ files, which is how the same ground got covered twice. | [`idxd-legacy-reader-audit.md`](idxd-legacy-reader-audit.md) | The legacy IDXD string-pool reader vs the real field table β€” what the old numbers got wrong | 🟑 shape CONFIRMED by hand (`FCSRange`, `ShieldRatio`, hangar `Model`); disc-wide rates are single-source | | [`structures/idxd-container.md`](structures/idxd-container.md) | The IDXD/IXUD container β€” record/field table, and the two beliefs it withdraws | βœ… CONFIRMED disc-wide, 7 750/7 750 objects and 1 271 462/1 271 462 named fields, zero failures | | [`structures/hud-glyph-quad.md`](structures/hud-glyph-quad.md) | The HUD's glyph quad β€” vtable `0x820B2A64` | βœ… CONFIRMED for the object layout and the atlas size, read live off | -| [`structures/slb-data-offset.md`](structures/slb-data-offset.md) | `.slb` leading-stream offset is `first_riff % 2048`, not the constant 1392 | βœ… CONFIRMED by decoding β€” 85 of 140 sampled banks yield more audio (median 70Γ—), 54 identical controls; offset recovery for `RIFF`-less banks is 99.95 % on the labelled set | +| [`structures/slb-data-offset.md`](structures/slb-data-offset.md) | `.slb` data offset = `(cumulative start of the `.pNN` segment) mod 2048`; the leading bytes are the previous bank's audio, not a header | βœ… CONFIRMED β€” exact for 8 783/8 783 banks, 0 mismatches; the four values are the running sums of the five segment sizes mod 2048; supersedes the earlier scan/`seek` heuristics | | [`structures/sound-pak-contents.md`](structures/sound-pak-contents.md) | Census of `sound.pak`, and the limit of the leading-region rule | βœ… CONFIRMED, 5 135/5 135 names hash into the TOC and **9 519/9 519** entries accounted for; ⚠️ leading-region rule holds for 1 571/4 382 eng and 0/5 100 jpn | | [`structures/sound-cue-table.md`](structures/sound-cue-table.md) | The cue index in `tables.pak` β€” message id -> cue -> sound id -> `.slb` bank | βœ… CONFIRMED, 1 326/1 338 script message ids bind to a bank; SOUNDS and FILES agree on the same 12 absentees, 0 orphan files | | [`structures/cutscene-message-table.md`](structures/cutscene-message-table.md) | Cutscene dialogue β€” speaker, portrait, on-screen seconds, audio cue per page | βœ… CONFIRMED, field count = 9Β·PageCount+2 for all 7 PageCounts, 1 252/1 252 caption keys match, 138 ids close both ways | diff --git a/docs/re/structures/slb-data-offset.md b/docs/re/structures/slb-data-offset.md index 16fb743a..5b044cc7 100644 --- a/docs/re/structures/slb-data-offset.md +++ b/docs/re/structures/slb-data-offset.md @@ -1,5 +1,14 @@ # `.slb` leading-stream data offset β€” 1392 was never a constant +> ⚠️ **Read the last section first.** A second pass on 2026-08-26 closed the ❔ +> at the bottom of the original writeup: the offset is +> `(cumulative start of the `.pNN` segment holding the wave) mod 2048`, exact for +> 8 783 of 8 783 banks. The `first_riff % 2048` rule below is a *consequence* of +> that and stays correct where a `RIFF` exists; the packet-plausibility scan and +> the `seek`-residue heuristic are superseded, and the 🟑 "most banks declare more +> `data` than they store" finding is **withdrawn**. Jump to +> [*X is the `.pNN` segment's grid phase*](#-settled-2026-08-26-second-pass--x-is-the-pnn-segments-grid-phase). + **βœ… Settled 2026-08-26, verified by decoding.** A bank's leading headerless packet stream does not start at a fixed offset. It starts at **`first_riff % 2048`**. `HEADERLESS_DATA_OFFSET = 1392` is the value that @@ -256,3 +265,295 @@ that opens a bank by name and seeks to its data; the constant, or the table it indexes, should be visible there. That is static PE work (`/work/*.pe`, offset = VA βˆ’ 0x82000000), not another pass over the archive β€” this page has taken the byte-level evidence about as far as it goes. + +--- + +# βœ… SETTLED 2026-08-26 (second pass) β€” X is the `.pNN` segment's grid phase + +The open ❔ above ("why the offset takes those four values, and what is in those +bytes") is now closed, exactly, with no residual heuristic: + +> **`X = (cumulative start of the `.pNN` segment file that holds the wave) mod 2048.`** + + segment size size % 2048 cum_start X + sound.p00 267 930 992 1392 0 0 + sound.p01 268 404 812 76 267 930 992 1392 + sound.p02 268 404 868 132 536 335 804 1468 + sound.p03 268 384 384 128 804 740 672 1600 + sound.p04 14 903 296 0 1 073 125 056 1728 + +**The four values are the running sums of the segment sizes mod 2048.** +`1392 = |p00| mod 2048`; `1468 = 1392 + 76`; `1600 = 1468 + 132`; +`1728 = 1600 + 128`. That identity is not a fit β€” it is arithmetic, and it comes +out of five file sizes that nothing in this analysis chose. + +Reproduce with `tools/re-capture/slb_segment_phase.py phases`. + +## Why + +The XMA1 packet grid is 2048-aligned **inside each individual `.pNN` file**: a +segment file starts at its own offset 0 with a bank header and everything after +is on that file's own 2048 grid. But the `.pak` **TOC addresses entries in the +flat *concatenation*** of `p00..p04`, and every TOC offset is itself a multiple +of 2048 (**checked: 0 of 9 519 entries are misaligned**). The segment files are +*not* multiples of 2048 long, so each join shifts the grid by +`size % 2048`, and an entry inside segment *k* sees the accumulated shift. + +Nothing in the game computes 1392. It is a **build-time artifact of where the +packer chose to cut a ~1.07 GB stream into five ~256 MB files.** + +### Verification β€” 100 %, no exceptions + +`tools/re-capture/slb_segment_phase.py verify`: + +| | banks | agree | disagree | +|---|---|---|---| +| has a `RIFF/WAVE` β€” X read off the wave header | 7 620 | **7 620** | **0** | +| `RIFF`-less β€” X read off the leading `seek` chunk | 1 163 | **1 163** | **0** | + +This **replaces both heuristics** in the section above. The packet-plausibility +scan (99.62 %) and the `seek`-residue rule (99.97 %, combined 99.95 %) are no +longer needed for anything: the offset is now *derivable* for every bank, +including the 1 495 `RIFF`-less ones and including the 28 ties the scan could +not break. Those sections stand as an honest record of how the number was +narrowed, not as the recommended method. + +### The prediction that could have refuted it, and did not + +If X really is a per-segment property rather than a per-bank one, then an entry +that **straddles** a segment boundary must show **two different phases inside +one file**. Exactly 3 of 9 518 entries straddle, and they do: + + eng\Voice\VOICE_ACRO_010.slb window starts 11 708 B before the p01β†’p02 join + seek chunk @11 632 ≑ 1392 (mod 2048) <- p01 phase + ---- sound.p01 / sound.p02 join at 11 708 ---- + RIFF @21 948 ≑ 1468 (mod 2048) <- p02 phase + seek @52 668 ≑ 1468 (mod 2048) + + jpn\etc\VOICE_D_149.slb (1468 -> 1600) jpn\Voice\VOICE_TCAF_577.slb (1600 -> 1728) + +A single-offset-per-bank model cannot produce that. ⚠️ It also means **"the data +offset of a bank" is not well defined for those three entries** β€” any decoder +that stores one offset per file will decode part of them mid-packet. + +## βœ… What is in those X bytes: audio, from the *previous* bank + +Not a header. They are the tail of the preceding bank's XMA1 packet stream, +carried into this window because the TOC window boundary and the bank boundary +are different things. + +Three independent lines: + +1. **Statistics are identical to known audio.** Over 167 `eng\etc` banks, the + number of distinct byte values seen at each fixed offset averages **101.06** + across `[0, 1392)` and **101.90** across `[1392, 3440)` β€” a region that is + certainly one XMA packet. Not one position in `[0, 3600)` takes ≀8 distinct + values, i.e. **there is no fixed field anywhere in the header**. A real + header would show constants. + +2. **The tiling arithmetic is exact.** A wave's `seek` (XMA `dpds`) chunk sits + immediately after its data and holds one `u32 LE` per packet, so it names its + own packet count and therefore where its data started. Entry *K*'s last wave + overruns the entry; entry *K+1* opens with a `seek` whose packet count + matches, at exactly the overrun distance measured **past the TOC window + rounded up to 2048**: + + VOICE_D_451 window 67 704 -> padded 69 632 last wave ends at 73 072 + VOICE_D_452 leading seek @ 3 440 (14 packets) 73 072 - 69 632 = 3 440 βœ… + VOICE_D_452 window 67 704 -> padded 69 632 last wave ends at 103 792 + VOICE_D_453 leading seek @34 160 (20 packets) 103 792 - 69 632 = 34 160 βœ… + + `tools/re-capture/slb_segment_phase.py chain 'eng\etc\VOICE_D_450.slb' 4`. + +3. **The bytes the TOC skips are audio too.** Between the end of + `VOICE_D_451`'s stored 67 704 bytes and the start of `VOICE_D_452` sit 1 928 + bytes of `.pNN` that no entry claims. **1 903 of the 1 928 are non-zero** β€” + dense audio, not padding. The stream is continuous through them. + +### ❌ Withdrawn: "the header is unique per bank, so it is not shared data" + +An earlier step here searched all 1 088 MB of `p00..p04` for one bank's exact +1 392-byte header and found **one** occurrence, its own β€” and I briefly read +that as ruling out any shared-stream model. It does not. The windows **tile**, +they do not overlap: the bytes appear once because they are stored once. The +measurement was right, the inference from it was wrong. + +## βœ… The real `.slb` bank layout (this is what the entries contain) + +Each bank is self-contained and starts on its segment's 2048 grid. All fields +big-endian. + + +0x00 u32 bank / cue id (BGM_001 -> 1001, BGM_105 -> 1105; voice ids run + consecutively in stream order, e.g. 7228, 7229, …) + +0x04 u32 0x11 (17) constant on every bank seen + +0x08 u32 0x20 (32) constant + +0x0C u32 1 constant + +0x10 u32 0x48 (72) constant + +0x14 u32 0 + +0x18 u32 0x800 BLOCK SIZE β€” the 2048 alignment unit + +0x1C u32 data size in bytes ❔ does not equal the sum of the waves; see below + +0x20 u32 bank id again + +0x24 u32 header size in BLOCKS always 5 -> the first wave is at +10240 + +0x28 u16 bits per sample (16) u16 channels (2 for BGM, 4 for voice banks) + +0x2C f32 } three floats, 1.0 / 0.5 / 0.1 on BGM_001, 1.0 / 0.4 on BGM_105 + +0x30 f32 } ❔ volume / mix, not identified + +0x34 f32 } + ... zero fill to 0x2800 (10240) + +then, repeated per wave: + + RIFF/WAVE exactly 4096 bytes = 2 blocks: + 'fmt ' 32 bytes standard little-endian XMAWAVEFORMAT + 'Dmmy' 4028 bytes of zero pad, so that… + 'data' …audio begins at riff+4096, on the grid + size is EXACTLY packets * 2048 + 'seek' 8 + 4*packets u32 LE cumulative decoded-sample counts + +`+0x18 == 0x800` plus `bank[0x00] == bank[0x20]` is a reliable signature: scanned +at the correct phase it finds every bank and nothing else. + +`+0x24` is always 5 on this disc, so **a bank's own audio starts 10 240 bytes +after its header** β€” but that header is generally *not* at offset 0 of the pak +entry (see the next section), which is why the value never showed up as a +constant. + +### The `fmt ` chunk decodes cleanly as XMA1 `XMAWAVEFORMAT` + +`eng\etc\VOICE_D_452.slb` first wave: tag `0x0165`, 16 bits, `NumStreams` 1, +`LoopCount` 0, `Version` 2, `PsuedoBytesPerSec` 12 212, `SampleRate` 48 000, +loop 0..0, `Channels` 1, `ChannelMask` 1 β€” mono. `BGM_105.slb`: +`LoopCount` 0xFF, `PsuedoBytesPerSec` 35 879, 48 000 Hz, `LoopStart` 0x003D72B1, +`LoopEnd` 0x0168BDBA, `Channels` 2. So the format was never in doubt; only the +framing was. + +## ❌ Withdrawn: "🟑 Most banks declare more `data` than they store" + +The section above reports that 5 296 of 7 586 banks (69.8 %) declare a `data` +size larger than the entry holds, and calls the decoder's "honest per sub-wave" +comment a documentation defect. **The measurement was right and the reading was +wrong β€” and the comment it accused was correct.** + +Declared `data` sizes are exact. The bytes are simply *outside the TOC window*, +still present in the `.pNN` stream. Taking 400 random `eng\`/`jpn\` banks and, +for every last wave that overruns its window, reading the declared range +straight from the segment stream: the wave's `seek` chunk lands at exactly the +declared end in **260 of 260** cases, 0 failures. + +So `eng\Voice\VOICE_TCAF_608.slb` is **not** "truncated, 99 % short", and that is +not why it decodes badly β€” its wave continues into the next window. The +counterexample section above needs revisiting on that basis. + +`data` size is always an exact multiple of 2048, which is the packet count. + +## ❔ Still open β€” the TOC window is not the bank + +The `.pak` TOC entry for a cue is a window that **contains** the cue's bank but +is not aligned to it, and the offset drifts entry to entry: + + eng\etc\VOICE_D_450.slb bank ids 7212 @ 7 536, 7226 @ 60 784 + eng\etc\VOICE_D_451.slb bank id 7228 @ 30 064 + eng\etc\VOICE_D_452.slb bank ids 7229 @ 5 488, 7230 @ 48 496 + eng\etc\VOICE_D_453.slb bank id 7231 @ 36 208 + eng\etc\VOICE_D_454.slb bank id 7232 @ 50 544 + +Bank ids run consecutively in pak-offset order, so a nameβ†’bank mapping exists, +but **which bank in a window belongs to the entry's name is not established** β€” +some windows hold two. A window can also cut a bank in half. Since a bank header +can only sit at `≑ X (mod 2048)` while a TOC offset is `≑ 0`, **a bank header can +never be at entry offset 0**; the leading region is structural, not accidental. + +❔ `+0x1C` (data size) does not match the sum of the wave `data` chunks β€” +`VOICE_D_452`'s bank 7229 declares 38 980 for a 26 624-byte wave. Unexplained. + +## 🟑 The loader β€” found, mapped, and it does *not* contain X (null result) + +Static search of `default.xex` (`/work/*.pe`, `/work/xenia-rs/sylpheed.db`), +done as the deliberate refutation attempt: if some code computes X, it should be +visible. **It is not, and under the segment-phase explanation it should not be.** + +**What was searched, all negative:** + +* **1392 / 1468 / 1600 / 1728 as immediates** (`li`/`addi`/`subi`/`cmpwi`/ + `cmplwi`/`ori`/`lis`, and the negations) over all 25 481 functions: 15 / 1 / 7 / + 10 hits, every one accounted for and none audio-related. Only two functions + hold β‰₯2 of the values and both are `subi rN,r1,K` / `addi r1,rN,K` stack-frame + pairs (`sub_82766DB0`, `sub_821DB270`, both D3D/shader-compiler code). + The most promising-looking hit β€” `cmplwi cr6,r31,0x570` at **0x8217FC68**, + next door to the sound manager β€” is **`ERROR_FILE_CORRUPT`**, in a run of + `0x7B`/`0x5`/`0xB7`/`0x48F` = `ERROR_INVALID_NAME`/`ACCESS_DENIED`/ + `ALREADY_EXISTS`/`DEVICE_NOT_CONNECTED`: storage-device retry code, not sizes. + `li r7,1468` at 0x827D3BC4 is a `__LINE__` for an HLSL-compiler assert. +* **A table of the four values** anywhere in the 9 568 256-byte image: 32-bit BE + aligned and unaligned, 32-bit LE, 16-bit both endians, and `float32`. Zero + clusters holding β‰₯2 of the four. The **only** 4-byte-aligned BE occurrence of + any of them in the whole image is one `1600` at 0x820489A0, in a zlib-adjacent + globals blob. `0x8202D668` looks like a hit (`… 0570 05BC …`) but is a + compiler message-offset table β€” it continues `0x664`, `0x6B8`, not `0x640`, + `0x6C0` β€” and is immediately followed by `"internal error: unknown "`. +* **`.slb` / `slb` / `XACT` / `XWB` / `xma` / `wavebank` as strings** β€” absent + from the image in ASCII and UTF-16. No 4CC-shaped immediates in the audio code. + +**The sound subsystem, for whoever picks this up next** (all located by residual +`__FILE__`/debug-`printf` strings β€” the image has **no user symbols at all**: +25 310 of 25 481 functions are `sub_XXXXXXXX`, RTTI stripped, `demangled_names` +empty): + +| range / address | what | +|---|---| +| 0x82175E00–0x8217DC00 | `silph::SoundManager` game layer | +| **0x821774A0** | BGM bank-slot manager β€” holds both the `BankSlots::get_bgm_data_area` (0x820A18B0) and `BankSlots::setup_bgm` (0x820A1938) failure strings | +| 0x82179988 | `SoundManager::Impl::SetMovieMode`, called from the movie handler 0x821B4968 | +| 0x82178F60 | sound config loader β€” refs `SOUNDS`/`SETTINGS`/`game:\` at 0x820A1878 | +| 0x82604A00–0x8260E900 | `gsfw` sound_framework middleware, 383 functions; public entries 0x826069E8 (play/register, `r3=15`), 0x82606A38 (stop/release), 0x826062D8, 0x82605028; assert sites 0x82608120, 0x8260DF28, 0x8260D8C0, 0x8260D9D0 β†’ the `gsfw_object.h` / `gsfw_objectcore.cpp` strings | +| 0x824D1000–0x824DC400 | XMA/XAudio driver β€” the only kernel audio imports (`XMACreateContext` wrapper 0x824D3BA8, `XAudioRegisterRenderDriverClient` 0x824DC280, …) | + +The one size-related constant on the bank path is in **`sub_821774A0`**: a +128 KB (`0x20000`) threshold on the loaded resource size, then +`addi r4,r29,2047` / `addi r11,r3,2047` / `clrrwi r11,r11,11` β€” allocate +`size + 2047` and round the pointer **up to 2048** β€” stored at `+76` of the slot +struct and handed to a gsfw vtable slot. So the runtime **does** honour the 2048 +grid; it just never needs to name 1392, because the 2048-alignment it applies is +to its own buffer, and the phase is baked into the file at pack time. + +🟑 rather than βœ… because **the walk from "a bank is opened" to "the first packet +is submitted" was not completed** β€” `sub_821774A0` is the BGM path, and no code +was traced that consumes a voice `.slb`. The claim proven here is the narrower +one: none of the four values exists as a constant or a table anywhere in the +executable. That is consistent with, but does not by itself prove, "the game +never needs X". + +## Consequences for our decoder + +`crates/sylpheed-formats/src/slb.rs` currently takes the offset from +`first_riff % 2048` (right, where a `RIFF` exists) and from `scan_data_offset` +otherwise (99.6 % right). Both can be replaced by the exact rule, which needs +one new input: **the pak offset of the entry, plus the segment start table** β€” +`PakArchive` already knows both. Suggested shape: + +* `PakArchive` exposes `segment_phase(concat_offset) -> usize`. +* `slb::to_xma_riff` takes the phase instead of deriving it. +* the 3 straddling entries need the phase looked up **per wave**, not per file. + +⚠️ Also worth revisiting: because the declared `data` sizes are honest and the +window is not the bank, the reader currently clamps away audio that is really +there. Reading a cue's full audio means reading past the TOC entry into the +`.pNN` stream. + +Not implemented here β€” this pass was static RE only, and the API change reaches +every caller. + +## Evidence log + +* 2026-08-26 β€” `X = segment_cum_start % 2048`. `CONFIRMED`. 7 620/7 620 banks + with a `RIFF` and 1 163/1 163 `RIFF`-less banks via `seek`, 0 mismatches; + independently, the four values are the running sums of the five `.pNN` sizes + mod 2048; independently, the 3 straddling entries show the phase step *inside* + one file at the segment join. `tools/re-capture/slb_segment_phase.py`. +* 2026-08-26 β€” leading X bytes are the previous bank's audio. `CONFIRMED`. + Byte-diversity identical to a known packet (101.06 vs 101.90, no fixed field); + `seek` packet counts chain exactly across three consecutive entries; the + unclaimed inter-entry `.pNN` bytes are 1 903/1 928 non-zero. +* 2026-08-26 β€” declared `data` sizes are exact, 260/260 checked. **Demotes** the + 🟑 "most banks declare more than they store" reading above. +* 2026-08-26 β€” no `.slb` data offset exists in `default.xex`. `PROBABLE` + (exhaustive constant/table/string search; the loader walk itself is unfinished).