slb: the headerless path was decoding stereo at a fixed offset; both are wrong
1495 banks carry no RIFF and take a separate path that hardcoded both the offset and stereo. Across a random 48-bank sample there was NOT ONE where the old stereo-at-1392 pair beat the best mono offset; median gain 184x, individual banks going from 0-4816 decoded bytes to 180000-380000. Stereo shows the same stop-after-one-frame signature already recorded for the leading segment. With no RIFF the offset cannot be derived, so scan_data_offset picks among the four disc offsets by XMA1 packet-header plausibility. Validated on the LABELLED set -- all 7358 banks that do have a RIFF, where the answer is forced: 7330 correct (99.62%), and all 28 misses are ties on the top score, never a wrong unique winner. Ties fall back to 1392. The winning offsets also reproduce, by directory, the distribution measured independently from the RIFF-bearing banks. jpn\etc splits 1468/1600, so path alone is not sufficient -- which is why this is a scan and not a lookup table. 7 disc tests pass (build-reborn test -p sylpheed-formats --test slb_leading_segment_disc, SYLPHEED_DISC wired up).
This commit is contained in:
@@ -131,6 +131,70 @@ fn parse_voice_clip(name: &str) -> VoiceClip {
|
|||||||
/// that yields the full track for segment banks while the clamp drops the
|
/// that yields the full track for segment banks while the clamp drops the
|
||||||
/// duplicate takes for alternate-take banks. (Dynamic RE via Canary file-I/O
|
/// duplicate takes for alternate-take banks. (Dynamic RE via Canary file-I/O
|
||||||
/// tracing confirmed the movie→voice binding; this fixes the *decode* of `RT*`.)
|
/// tracing confirmed the movie→voice binding; this fixes the *decode* of `RT*`.)
|
||||||
|
/// The four data offsets that occur on the disc, in ascending order.
|
||||||
|
///
|
||||||
|
/// Measured over all 7 358 banks whose offset is *known* (they carry a `RIFF`,
|
||||||
|
/// so the offset is forced to `first_riff % XMA1_PACKET`): no other value
|
||||||
|
/// occurs. They are all of the form `1392 + 4k`.
|
||||||
|
pub const DATA_OFFSET_CANDIDATES: [usize; 4] = [1392, 1468, 1600, 1728];
|
||||||
|
|
||||||
|
/// How plausible a candidate offset is, judged by XMA1 packet headers alone.
|
||||||
|
///
|
||||||
|
/// Each 2048-byte packet opens with a big-endian header: 6 bits frame count,
|
||||||
|
/// 15 bits frame-offset-in-bits, 3 bits metadata, 8 bits packet-skip. At the
|
||||||
|
/// true offset those fields stay in range packet after packet; one byte off and
|
||||||
|
/// they do not. Returns the fraction of the first `LIMIT` packets that look
|
||||||
|
/// sane, so 1.0 is a clean stream.
|
||||||
|
fn packet_plausibility(slb: &[u8], start: usize) -> f32 {
|
||||||
|
const LIMIT: usize = 24;
|
||||||
|
let (mut seen, mut ok, mut pos) = (0usize, 0usize, start);
|
||||||
|
while pos + XMA1_PACKET <= slb.len() && seen < LIMIT {
|
||||||
|
let h = u32::from_be_bytes([slb[pos], slb[pos + 1], slb[pos + 2], slb[pos + 3]]);
|
||||||
|
let frame_offset_bits = (h >> 11) & 0x7FFF;
|
||||||
|
let metadata = (h >> 8) & 0x7;
|
||||||
|
let packet_skip = h & 0xFF;
|
||||||
|
if frame_offset_bits as usize <= XMA1_PACKET * 8 && metadata <= 1 && packet_skip <= 8 {
|
||||||
|
ok += 1;
|
||||||
|
}
|
||||||
|
seen += 1;
|
||||||
|
pos += XMA1_PACKET;
|
||||||
|
}
|
||||||
|
if seen == 0 {
|
||||||
|
0.0
|
||||||
|
} else {
|
||||||
|
ok as f32 / seen as f32
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Recover a bank's data offset when there is no `RIFF` to derive it from.
|
||||||
|
///
|
||||||
|
/// Picks the candidate whose packet headers look most like a real XMA1 stream.
|
||||||
|
/// Cross-checked against the 7 358 banks where the answer *is* known from the
|
||||||
|
/// `RIFF` position: **7 330 correct (99.62 %)**, and every one of the 28 misses
|
||||||
|
/// is a tie on the top score — the scan is never wrong when it has a unique
|
||||||
|
/// winner. Ties fall back to [`HEADERLESS_DATA_OFFSET`].
|
||||||
|
pub fn scan_data_offset(slb: &[u8]) -> usize {
|
||||||
|
let mut best = (HEADERLESS_DATA_OFFSET, -1.0f32);
|
||||||
|
let mut tied = false;
|
||||||
|
for &c in &DATA_OFFSET_CANDIDATES {
|
||||||
|
if c >= slb.len() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let score = packet_plausibility(slb, c);
|
||||||
|
if score > best.1 {
|
||||||
|
best = (c, score);
|
||||||
|
tied = false;
|
||||||
|
} else if (score - best.1).abs() < f32::EPSILON {
|
||||||
|
tied = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if tied {
|
||||||
|
HEADERLESS_DATA_OFFSET
|
||||||
|
} else {
|
||||||
|
best.0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Where a bank's leading headerless packet stream starts.
|
/// Where a bank's leading headerless packet stream starts.
|
||||||
///
|
///
|
||||||
/// The stream is a whole number of 2048-byte XMA1 packets ending at the first
|
/// The stream is a whole number of 2048-byte XMA1 packets ending at the first
|
||||||
@@ -148,10 +212,19 @@ pub fn to_xma_riffs(slb: &[u8]) -> Vec<Vec<u8>> {
|
|||||||
let mut out = Vec::new();
|
let mut out = Vec::new();
|
||||||
let first_riff = find(slb, b"RIFF", 0);
|
let first_riff = find(slb, b"RIFF", 0);
|
||||||
if first_riff.is_none() {
|
if first_riff.is_none() {
|
||||||
// Headerless single-stream bank.
|
// Headerless single-stream bank. Two things here were wrong, and the
|
||||||
if let Some(data) = slb.get(HEADERLESS_DATA_OFFSET..) {
|
// corrections are measured (docs/re/structures/slb-data-offset.md):
|
||||||
|
//
|
||||||
|
// * the offset is not the constant — with no `RIFF` to derive it from,
|
||||||
|
// scan the four candidates by packet plausibility;
|
||||||
|
// * the stream is **mono**. At two channels a 48-bank sample yielded
|
||||||
|
// 0..4 816 bytes; at one, 180 000..380 000. There was not one bank
|
||||||
|
// where the old stereo/1392 pair beat the scanned mono pair, and the
|
||||||
|
// median gain was 184x.
|
||||||
|
let start = scan_data_offset(slb);
|
||||||
|
if let Some(data) = slb.get(start..) {
|
||||||
if !data.is_empty() {
|
if !data.is_empty() {
|
||||||
out.push(build_riff(&synth_xma1_fmt(2, 2, 48000), data));
|
out.push(build_riff(&synth_xma1_fmt(1, 0, 48000), data));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return out;
|
return out;
|
||||||
|
|||||||
@@ -153,3 +153,69 @@ fn derived_offset_recovers_voice_banks_without_regressing_etc() {
|
|||||||
b.windows(4).position(|w| w == b"RIFF").unwrap()),
|
b.windows(4).position(|w| w == b"RIFF").unwrap()),
|
||||||
slb::HEADERLESS_DATA_OFFSET);
|
slb::HEADERLESS_DATA_OFFSET);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The scan agrees with the truth wherever the truth is knowable.
|
||||||
|
///
|
||||||
|
/// A bank carrying a `RIFF` has its offset *forced* to `first_riff % 2048`, so
|
||||||
|
/// those banks are a labelled set for a rule meant to serve the ones without a
|
||||||
|
/// `RIFF`. Over the whole labelled set the scan is right 99.6 % of the time and
|
||||||
|
/// its only failures are ties. This test walks a slice of it.
|
||||||
|
#[test]
|
||||||
|
fn scan_data_offset_agrees_with_the_riff_derived_answer() {
|
||||||
|
skip_without_disc!(root);
|
||||||
|
let snd = PakArchive::open(root.join("dat/sound.pak")).expect("sound.pak");
|
||||||
|
let mut checked = 0usize;
|
||||||
|
let mut agreed = 0usize;
|
||||||
|
for lang in ["eng", "jpn"] {
|
||||||
|
for (dir, lo, hi) in [("Voice", 1u32, 120u32), ("etc", 1, 120)] {
|
||||||
|
for n in lo..hi {
|
||||||
|
let path = format!("{lang}\\{dir}\\VOICE_TCAF_{n:03}.slb");
|
||||||
|
let Some(entry) = snd.find_by_name(&path) else { continue };
|
||||||
|
let Ok(b) = snd.read(entry) else { continue };
|
||||||
|
let Some(ri) = b.windows(4).position(|w| w == b"RIFF") else { continue };
|
||||||
|
if ri <= slb::HEADERLESS_DATA_OFFSET {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if !b[slb::HEADERLESS_DATA_OFFSET..ri].iter().any(|v| *v != 0) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
checked += 1;
|
||||||
|
if slb::scan_data_offset(&b) == slb::leading_data_offset(ri) {
|
||||||
|
agreed += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert!(checked >= 20, "expected a usable labelled set, got {checked}");
|
||||||
|
// The whole-disc rate is 99.62%; allow a little slack for a small slice.
|
||||||
|
let rate = agreed as f64 / checked as f64;
|
||||||
|
assert!(
|
||||||
|
rate >= 0.95,
|
||||||
|
"scan agreed on {agreed}/{checked} ({:.1}%), expected >=95%",
|
||||||
|
rate * 100.0
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Every offset the scan can return is one of the four seen on disc.
|
||||||
|
#[test]
|
||||||
|
fn scan_only_returns_known_offsets() {
|
||||||
|
skip_without_disc!(root);
|
||||||
|
let snd = PakArchive::open(root.join("dat/sound.pak")).expect("sound.pak");
|
||||||
|
let mut seen = 0usize;
|
||||||
|
for n in 1u32..200 {
|
||||||
|
for path in [
|
||||||
|
format!("eng\\Voice\\VOICE_ADAN_{n:03}.slb"),
|
||||||
|
format!("jpn\\Voice\\VOICE_ADAN_{n:03}.slb"),
|
||||||
|
] {
|
||||||
|
let Some(entry) = snd.find_by_name(&path) else { continue };
|
||||||
|
let Ok(b) = snd.read(entry) else { continue };
|
||||||
|
let got = slb::scan_data_offset(&b);
|
||||||
|
assert!(
|
||||||
|
slb::DATA_OFFSET_CANDIDATES.contains(&got),
|
||||||
|
"{path}: scan returned {got}, not a known offset"
|
||||||
|
);
|
||||||
|
seen += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert!(seen >= 20, "expected banks to test, saw {seen}");
|
||||||
|
}
|
||||||
|
|||||||
@@ -75,13 +75,58 @@ the sample the format was derived from.
|
|||||||
The same error was hiding a defect in the **English** set too: 1 873 `eng\Voice`
|
The same error was hiding a defect in the **English** set too: 1 873 `eng\Voice`
|
||||||
banks sit at 1468 and were being decoded mid-packet just as badly.
|
banks sit at 1468 and were being decoded mid-packet just as badly.
|
||||||
|
|
||||||
|
## The `RIFF`-less banks had the same bug, plus a worse one
|
||||||
|
|
||||||
|
**✅ Settled 2026-08-26.** 1 495 banks (799 `jpn`, 696 `eng`) carry no `RIFF` at
|
||||||
|
all and take a separate code path. That path was wrong twice over:
|
||||||
|
|
||||||
|
1. it used the constant offset, with no `RIFF` to derive from; and
|
||||||
|
2. it built a **stereo** `fmt` chunk.
|
||||||
|
|
||||||
|
Decoded across a random 48-bank sample:
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| banks where the old stereo-at-1392 beat the best mono offset | **0 of 48** |
|
||||||
|
| median gain | **184×** |
|
||||||
|
| range | 25× – 489 344× |
|
||||||
|
|
||||||
|
Stereo is the same failure signature recorded for the leading segment: it stops
|
||||||
|
after one frame. Individual banks went from 0–4 816 bytes to 180 000–380 000.
|
||||||
|
|
||||||
|
The winning offsets fall out **by directory**, and they reproduce the
|
||||||
|
distribution measured independently from the `RIFF`-bearing banks — which is the
|
||||||
|
cross-check that makes this more than curve-fitting:
|
||||||
|
|
||||||
|
eng\etc 1392 (11/11) eng\Voice 1468 (9/9) eng\Briefing 1392 (2/2)
|
||||||
|
jpn\Voice 1600 (12/13) jpn\etc 1468 (8/12), 1600 (4)
|
||||||
|
|
||||||
|
Note `jpn\etc` splits, so the **path alone is not enough** to pick the offset.
|
||||||
|
|
||||||
|
### Picking the offset without a decoder
|
||||||
|
|
||||||
|
An XMA1 packet opens with a big-endian header — 6 bits frame count, 15 bits
|
||||||
|
frame-offset-in-bits, 3 bits metadata, 8 bits packet-skip. At the true offset
|
||||||
|
those fields stay in range packet after packet; one byte off and they do not.
|
||||||
|
Scoring the first 24 packets and taking the best candidate:
|
||||||
|
|
||||||
|
**7 330 of 7 358 (99.62 %)** on the labelled set — every bank that *has* a
|
||||||
|
`RIFF`, where the answer is forced and therefore known. All **28** misses are
|
||||||
|
ties on the top score; there is not a single case where the scan picks wrongly
|
||||||
|
with a unique winner. `scan_data_offset` therefore falls back to 1392 on a tie.
|
||||||
|
|
||||||
|
This is used only for the `RIFF`-less banks. Where a `RIFF` exists the offset is
|
||||||
|
derived from it exactly, never scanned.
|
||||||
|
|
||||||
## What this does not settle
|
## What this does not settle
|
||||||
|
|
||||||
* **Why the offset takes those four values**, and what the bytes before it are.
|
* **Why the offset takes those four values**, and what the bytes before it are.
|
||||||
There is no length field in the first 64 bytes — banks open on high-entropy
|
There is no length field in the first 64 bytes — banks open on high-entropy
|
||||||
data — so the offset is derived, not read.
|
data — so the offset is derived, not read.
|
||||||
* **`eng\Voice\VOICE_TCAF_608.slb`**, above.
|
* **`eng\Voice\VOICE_TCAF_608.slb`**, above.
|
||||||
* **The 799 jpn / 696 eng banks with no `RIFF` at all** are untouched by this;
|
* **The 28 ties.** The scan cannot separate them and falls back to 1392, which
|
||||||
they go down the headerless path and were not re-examined.
|
is right for roughly a third of that population and wrong for the rest.
|
||||||
|
* **Why the offset takes exactly these four values by directory** is still
|
||||||
|
unexplained — see above.
|
||||||
* Nothing here was run **in the game** — this is a decoder-side result measured
|
* Nothing here was run **in the game** — this is a decoder-side result measured
|
||||||
with FFmpeg as the oracle.
|
with FFmpeg as the oracle.
|
||||||
|
|||||||
Reference in New Issue
Block a user