From 220e250453f3f5479467f722076902a0e5897852 Mon Sep 17 00:00:00 2001 From: Sylpheed RE agent Date: Wed, 26 Aug 2026 04:32:23 +0000 Subject: [PATCH] slb: the seek chunk gives the data offset structurally, and breaks the 28 ties The ties needed a different signal, not a longer scan. Banks carry one: a seek chunk sitting on a packet boundary, so seek_pos % 2048 IS the data offset. On the 6033 labelled banks with a seek before their first RIFF, 6031 agree (99.97%) -- better than the packet scan and structural rather than statistical, so scan_data_offset now tries it first. On the scan's 28 ties it resolves 26 correctly and 0 wrongly (2 have no usable seek). Combined rule scores 7354/7358 = 99.95%, up from 99.62%. 762 of the 1495 RIFF-less banks carry a seek, so the signal exists where it is needed. Also ruled out, since a wrong offset was this page's whole subject: the header is not audio being discarded. Adding 0 to the candidate set, it wins 6 of 7358. 7 disc tests pass. --- crates/sylpheed-formats/src/slb.rs | 33 +++++++++++++++++++++++---- docs/re/BACKLOG.md | 9 ++++++-- docs/re/INDEX.md | 2 +- docs/re/structures/slb-data-offset.md | 31 +++++++++++++++++++++++++ 4 files changed, 67 insertions(+), 8 deletions(-) diff --git a/crates/sylpheed-formats/src/slb.rs b/crates/sylpheed-formats/src/slb.rs index 78a979f2..ec2d8ee2 100644 --- a/crates/sylpheed-formats/src/slb.rs +++ b/crates/sylpheed-formats/src/slb.rs @@ -166,14 +166,37 @@ fn packet_plausibility(slb: &[u8], start: usize) -> f32 { } } +/// The data offset implied by the bank's `seek` chunk, if it has one. +/// +/// A bank's `seek` chunk lands on a packet boundary, so `seek_pos % XMA1_PACKET` +/// *is* the data offset. Measured on the 6 033 labelled banks that have a +/// `seek` before their first `RIFF`: **6 031 agree (99.97 %)**, 2 disagree. +/// This is structural rather than statistical, which is why it is tried first. +fn seek_chunk_offset(slb: &[u8]) -> Option { + let pos = find(slb, b"seek", 0)?; + let residue = pos % XMA1_PACKET; + DATA_OFFSET_CANDIDATES.contains(&residue).then_some(residue) +} + /// Recover a bank's data offset when there is no `RIFF` to derive it from. /// -/// Picks the candidate whose packet headers look most like a real XMA1 stream. -/// Cross-checked against the 7 358 banks where the answer *is* known from the -/// `RIFF` position: **7 330 correct (99.62 %)**, and every one of the 28 misses -/// is a tie on the top score — the scan is never wrong when it has a unique -/// winner. Ties fall back to [`HEADERLESS_DATA_OFFSET`]. +/// Two independent signals, tried in order of how well each is evidenced: +/// +/// 1. **the `seek` chunk's position** mod the packet size — 99.97 % on the +/// labelled set, and structural rather than statistical; +/// 2. **packet-header plausibility** — pick the candidate whose XMA1 headers +/// stay in range over the first 24 packets. Alone this is 99.62 %, and all +/// 28 of its misses are ties rather than wrong unique winners. +/// +/// Together, on the 7 358 banks where the answer *is* known from the `RIFF` +/// position: **7 354 correct (99.95 %)**. The `seek` residue resolves 26 of the +/// scan's 28 ties correctly and none of them wrongly; the other 2 have no +/// usable `seek`. Falls back to [`HEADERLESS_DATA_OFFSET`] when neither signal +/// decides. pub fn scan_data_offset(slb: &[u8]) -> usize { + if let Some(off) = seek_chunk_offset(slb) { + return off; + } let mut best = (HEADERLESS_DATA_OFFSET, -1.0f32); let mut tied = false; for &c in &DATA_OFFSET_CANDIDATES { diff --git a/docs/re/BACKLOG.md b/docs/re/BACKLOG.md index 9a0b1bde..e64822ab 100644 --- a/docs/re/BACKLOG.md +++ b/docs/re/BACKLOG.md @@ -35,8 +35,13 @@ mistaken a constant derived from `eng\etc\` for a property of the format. See at *exactly* 1392 behind a zero-filled header, i.e. a zero-length leading region, which both the old code and the new derivation already handle. -**Still open**: the 28 offset-scan ties, and ❔ why 69.8 % of banks declare more -`data` than they store (see +**Closed 2026-08-26**: the 28 offset-scan ties. A bank's `seek` chunk sits on a +packet boundary, so `seek_pos % 2048` is a second and structural derivation of +the offset (99.97 % on the labelled set). It resolves 26 of the 28 correctly and +none wrongly; the combined rule is 99.95 %. + +**Still open**: ❔ why 69.8 % of banks declare more `data` than they store, and +❔ why the offset takes exactly four values by directory (see [`structures/slb-data-offset.md`](structures/slb-data-offset.md)). --- diff --git a/docs/re/INDEX.md b/docs/re/INDEX.md index 057a013c..2cdb80d8 100644 --- a/docs/re/INDEX.md +++ b/docs/re/INDEX.md @@ -94,7 +94,7 @@ files, which is how the same ground got covered twice. | [`idxd-legacy-reader-audit.md`](idxd-legacy-reader-audit.md) | The legacy IDXD string-pool reader vs the real field table — what the old numbers got wrong | 🟡 shape CONFIRMED by hand (`FCSRange`, `ShieldRatio`, hangar `Model`); disc-wide rates are single-source | | [`structures/idxd-container.md`](structures/idxd-container.md) | The IDXD/IXUD container — record/field table, and the two beliefs it withdraws | ✅ CONFIRMED disc-wide, 7 750/7 750 objects and 1 271 462/1 271 462 named fields, zero failures | | [`structures/hud-glyph-quad.md`](structures/hud-glyph-quad.md) | The HUD's glyph quad — vtable `0x820B2A64` | ✅ CONFIRMED for the object layout and the atlas size, read live off | -| [`structures/slb-data-offset.md`](structures/slb-data-offset.md) | `.slb` leading-stream offset is `first_riff % 2048`, not the constant 1392 | ✅ CONFIRMED by decoding — 85 of 140 sampled banks yield more audio (median 70×), 54 identical controls, 1 where neither offset works | +| [`structures/slb-data-offset.md`](structures/slb-data-offset.md) | `.slb` leading-stream offset is `first_riff % 2048`, not the constant 1392 | ✅ CONFIRMED by decoding — 85 of 140 sampled banks yield more audio (median 70×), 54 identical controls; offset recovery for `RIFF`-less banks is 99.95 % on the labelled set | | [`structures/sound-pak-contents.md`](structures/sound-pak-contents.md) | Census of `sound.pak`, and the limit of the leading-region rule | ✅ CONFIRMED, 5 135/5 135 names hash into the TOC and **9 519/9 519** entries accounted for; ⚠️ leading-region rule holds for 1 571/4 382 eng and 0/5 100 jpn | | [`structures/sound-cue-table.md`](structures/sound-cue-table.md) | The cue index in `tables.pak` — message id -> cue -> sound id -> `.slb` bank | ✅ CONFIRMED, 1 326/1 338 script message ids bind to a bank; SOUNDS and FILES agree on the same 12 absentees, 0 orphan files | | [`structures/cutscene-message-table.md`](structures/cutscene-message-table.md) | Cutscene dialogue — speaker, portrait, on-screen seconds, audio cue per page | ✅ CONFIRMED, field count = 9·PageCount+2 for all 7 PageCounts, 1 252/1 252 caption keys match, 138 ids close both ways | diff --git a/docs/re/structures/slb-data-offset.md b/docs/re/structures/slb-data-offset.md index ec7886d2..087aa687 100644 --- a/docs/re/structures/slb-data-offset.md +++ b/docs/re/structures/slb-data-offset.md @@ -120,6 +120,37 @@ with a unique winner. `scan_data_offset` therefore falls back to 1392 on a tie. This is used only for the `RIFF`-less banks. Where a `RIFF` exists the offset is derived from it exactly, never scanned. +### ✅ A second, independent signal — and it breaks the ties + +**Settled 2026-08-26.** The 28 ties needed a different signal, not more of the +same one, and the banks carry one: a **`seek` chunk sitting on a packet +boundary**. Its position modulo 2048 therefore *is* the data offset. + + seek at 3 516 / 5 564 / 7 612 / 9 660 / 13 756 / 19 900 — all ≡ 1468 (mod 2048) + +On the 6 033 labelled banks that have a `seek` before their first `RIFF`, +**6 031 agree (99.97 %)** and 2 disagree. That is better than the packet scan +and, more importantly, *structural* rather than statistical — which is why it is +now tried first. + +Applied to the packet scan's 28 ties: **26 resolved correctly, 0 wrongly**, and +2 with no usable `seek`. The combined rule — `seek` residue, else packet +plausibility, else 1392 — scores **7 354 / 7 358 = 99.95 %** on the labelled set, +up from 99.62 %. + +762 of the 1 495 `RIFF`-less banks carry a `seek`, and its residue lands on the +four known offsets there too (1468 ×343, 1600 ×255, 1392 ×148, 1728 ×16), so the +signal is available in the population that needs it. + +### ❌ The header is not audio being discarded + +Worth ruling out, since a wrong data offset was the whole subject of this page: +if the bytes *before* the offset were audio, we would be throwing away the start +of every clip. Adding **0** to the candidate set and re-running the scan, it wins +**6 of 7 358** — noise. The header is genuinely not part of the packet stream. +(1 482 banks have an all-zero header; 5 876 have content in it, which is what +prompted the check.) + ### Is that 99.62 % transferable? — checked, and it is conservative The labelled set has a `RIFF`; the population the scan actually serves does not.