I read the seek chunk's packet count big-endian; it is little-endian at seek+12,
with size == 8 + 4*count. And a seek sits immediately AFTER its own data, so an
entry's first seek usually belongs to the PREVIOUS bank (implied start -25232
for D_452, -145988 for TCAF_608). I was comparing an entry's first seek against
its first data -- different waves by construction, which is why no reading lined
up.
With that fixed, the declared sizes are honest: every RIFF-bearing entry on the
disc has seek magic at exactly data_at + declared_size with count*2048 ==
declared. 7620/7620, zero failures.
VOICE_TCAF_608 is not truncated. Its Channels is 2 and I decoded it as mono;
read as stereo it gives 6520176 bytes = 33.96 s, agreeing with both length
signals in the bank (33.88 s from cumulative samples, 33.97 s from
PsuedoBytesPerSec). 170 of 8021 banks (2.12%) are stereo -- exactly the rate of
my 1-in-60 outlier.
This is the mono/stereo trap already documented on this very page, met from the
other direction: I had written 'at two channels every bank yields one frame' and
then spent several passes blaming missing data for a one-frame decode.
Code fix: to_xma_riffs built the leading segment with a hard-wired mono fmt. It
now reads Channels from the bank's first RIFF. 7 disc tests pass.
Identified the structure: a little-endian size field after the tag, then
0x01000000, a varying word, a zero, then a strictly ascending table. Sizes 64 /
108 / 348 bytes for three sample banks.
Neither obvious reading gives the declared packet count. Entry count minus the
three header words is exactly right for VOICE_D_452 (13) and wrong for the other
two (24 vs 17, 84 vs 371) -- a one-of-three fit, recorded as FAILED rather than
as a rule with exceptions, because two measurements today already returned
plausible numbers for questions they could not answer.
The ascending values step by 1.5-2.2 million, far too large to be packet
indices, so they are on some other scale. Recorded what the next attempt should
know, including that auto/slb-loader chains seek packet counts successfully by
some field that is not either of the two tried here.
Generalising the TCAF_608 test across the 5296 over-declaring banks first looked
like a clean reversal: 59 of 60 gained audio (median 2.10x) from reading the full
declared size instead of stopping at the TOC window.
Then I tested whether the declared size is an honest boundary by reading TWICE
it. 33 of 40 yield >1.5x more audio, median ratio 1.64. The decoder keeps going
indefinitely, because XMA1 packets are self-contained -- the next bank's packets
decode perfectly well and are simply not this bank's. A byte count cannot tell
them apart.
So the 59/60 is withdrawn as evidence for anything. Same error as the unanchored
'data' search earlier on this page: a measurement returning a plausible number
for a question it cannot answer.
What survives: TCAF_608 is genuinely anomalous (86x more input, zero extra
output, where the typical bank yields without limit); the honest-size question
is unresolved and needs a boundary signal independent of the decoder, such as
the seek-chunk packet chain; and the only measured fact is that the declared
size exceeds the TOC window for 5296 of 7586 banks.
Read VOICE_TCAF_608's full declared 759808 bytes out of the flat stream ignoring
window boundaries. It decodes to the same 896 bytes (0.01 s) as stopping at the
window end -- 86x more input, zero extra samples. Control VOICE_D_452, whose
declared size fits, decodes to 2.18 s normally. No segment join is crossed, so
the packing phase is not a confound.
So 'the bytes are outside the window but still in the stream' does not hold for
the one bank the other branch named as not truncated. I still withdraw my word
'truncated' as an over-claim about the other 5295 banks: I measured that their
declared size exceeds their window, not what lies beyond it, and have now tested
exactly one. Recorded the three claims at their actual strengths.
Reproduced independently: X = (cumulative start of the .pNN segment) mod 2048.
The running sums of the five sound.pNN file sizes are 0, 1392, 1468, 1600, 1728
-- exactly the four values measured, plus zero. The XMA grid is 2048-aligned
inside each segment file, the segments are not multiples of 2048 long, so each
join shifts the phase.
Two of my explanations on this page were wrong even though the measurements
stand. 'Varies by language and subdirectory' was correlation, not cause --
directories cluster into segments. And there is no header at all: those bytes
are the PREVIOUS bank's audio, which is why they had no length field and looked
like data.
Also records a disagreement I could not resolve. auto/slb-loader withdraws my
'69.8% over-declare' finding as declared-sizes-are-exact, but VOICE_TCAF_608
declares 759808 bytes of data with 55296 bytes to the next TOC entry. Both
readings agree the declared size exceeds the window; mine said 'truncated',
which I withdraw as an over-claim, but 'exact' requires a wave to span ~14 named
entries. Recorded with the experiment that would decide it.
Probed the bytes before the data offset at all four values. Not a length field
(no word in the first 64 bytes matches the offset, the RIFF position or the entry
size, either endianness). Not a seek table or any ascending index (~50% of
consecutive BE word pairs are non-decreasing, i.e. random; all words distinct,
none zero). Not zero padding in general (5876 of 7358 have content). Not audio
being discarded (offset 0 wins 6 of 7358).
High-entropy content whose size is constant per language and subdirectory, with
no self-describing length -- which points at a loader that knows the size a
priori. Recorded the next step as static PE work rather than another archive
pass; the byte-level evidence here is about exhausted.
The ties needed a different signal, not a longer scan. Banks carry one: a seek
chunk sitting on a packet boundary, so seek_pos % 2048 IS the data offset. On
the 6033 labelled banks with a seek before their first RIFF, 6031 agree
(99.97%) -- better than the packet scan and structural rather than statistical,
so scan_data_offset now tries it first.
On the scan's 28 ties it resolves 26 correctly and 0 wrongly (2 have no usable
seek). Combined rule scores 7354/7358 = 99.95%, up from 99.62%. 762 of the 1495
RIFF-less banks carry a seek, so the signal exists where it is needed.
Also ruled out, since a wrong offset was this page's whole subject: the header
is not audio being discarded. Adding 0 to the candidate set, it wins 6 of 7358.
7 disc tests pass.
The labelled set has a RIFF and the scan is unbounded, so it reads past the RIFF
there -- the headline number could have been borrowing discrimination that a
RIFF-less bank cannot offer. Confining the scan to the leading region gives
69.98%, which looks like exactly that problem.
It is not. Split by how much leading audio there is: on the 989 banks with >=24
packets of it, the scan is 100% correct with ZERO ties, whether or not the RIFF
is in range. The 69.98% is an artifact of short leading regions, where two or
three packets are not enough to separate candidates.
A RIFF-less bank is a whole pak entry, so 24 packets are always available. The
99.62% is conservative for the population the scan serves, not optimistic.
Of the 7586 banks with a RIFF and a data chunk after it, 5296 declare a data
size larger than the pak entry holds; 2290 declare less (the ordinary
multi-sub-wave case); NONE declare exactly what they hold. This contradicts the
decoder comment claiming the declared size 'is honest per sub-wave'. The code
clamps, so it is a documentation defect, not a crash.
It also closes the loose end from the offset work: eng\Voice\VOICE_TCAF_608,
the single bank where neither offset decoded, is 99% short -- there is nothing
there to decode.
Method note recorded: my first pass searched for 'data' from offset 0, which can
match by chance inside the leading audio region. Anchoring the search after the
first RIFF moved the count 5038 -> 5296.
Separately, the 55 'early RIFF' English banks are not an anomaly: all 55 sit at
exactly 1392 behind a zero-filled header -- a zero-length leading region, which
both the old and new code already handle correctly.
1495 banks carry no RIFF and take a separate path that hardcoded both the
offset and stereo. Across a random 48-bank sample there was NOT ONE where the
old stereo-at-1392 pair beat the best mono offset; median gain 184x, individual
banks going from 0-4816 decoded bytes to 180000-380000. Stereo shows the same
stop-after-one-frame signature already recorded for the leading segment.
With no RIFF the offset cannot be derived, so scan_data_offset picks among the
four disc offsets by XMA1 packet-header plausibility. Validated on the LABELLED
set -- all 7358 banks that do have a RIFF, where the answer is forced: 7330
correct (99.62%), and all 28 misses are ties on the top score, never a wrong
unique winner. Ties fall back to 1392.
The winning offsets also reproduce, by directory, the distribution measured
independently from the RIFF-bearing banks. jpn\etc splits 1468/1600, so path
alone is not sufficient -- which is why this is a scan and not a lookup table.
7 disc tests pass (build-reborn test -p sylpheed-formats --test
slb_leading_segment_disc, SYLPHEED_DISC wired up).
HEADERLESS_DATA_OFFSET is the value the offset takes in <lang>\etc\, not a
property of the format. The leading stream is a whole number of 2048-byte XMA1
packets ending at the first RIFF, so its start is first_riff % XMA1_PACKET.
Disc-wide that takes four values -- 1392, 1468, 1600, 1728 -- varying by
language and subdirectory.
Verified by decoding, not by arithmetic: on a random 140-bank sample with a
non-empty leading region, the derived offset yields more audio in 85, identical
in 54 (the eng\etc controls, where it must and does reproduce the old
behaviour) and less in 1. Median gain among the improved is 70x --
eng\Voice\VOICE_TCAF_592 goes 1506 -> 97152 bytes, jpn 2910 -> 127178.
This withdraws my own claim from earlier today that the Japanese banks were a
different undecoded layout. They are the same format with a different offset;
I had treated a constant derived from one subdirectory as a property of the
format. The same error was hiding the identical defect in 1873 eng\Voice banks.
