Last iteration I noticed BR09_04's block id names BR10_03, itself the next
exception entry, and flagged it as suggestive but untested. Tested: sorting all
282 by offset, entry[i].field equals entry[i+1]'s own id in 8 of 281 (2.8%),
indistinguishable from chance -- and BR09_04 was one of the eight.
Recorded rather than dropped because it is exactly the kind of pattern that
reads as a discovery when spotted in a sample of one.
The offset from the entry's own id is broadly spread, 7 to 14 and beyond with a
peak at 9-11, so the field is near the entry's id without standing in any fixed
relation to it.
Not noise -- a small structured record in three runs at about +1790, +3840 and
+7940. Across all 282: the leading 16-bit value names a real SOUNDS cue
282/282, the block carries two IEEE 1.0 floats in 281, and an 01/02/02/0x64 tail
in 281. Two unit floats and a 100 are the shape of volume, pitch and priority,
so this reads as an XACT cue record -- marked yellow, since every value is a
default and nothing varies enough to prove it.
The id is NOT this entry's own cue: it is consistently a different, higher one
(8501 -> 8504 = BR02_01; 5027 -> 5036 = VOICE_A_036), with a varying offset so
not a fixed stride. BR09_04's field names BR10_03, which is itself the next such
entry -- suggestive of a chain, but one observation and untested.
The practical point, which is settled: the block is a populated metadata record
naming a real cue, not padding.
Both, 282/282 exact: w0 == table[-2], w1 == table[-1]. The trailer's first two
words are a verbatim copy of the tail of the wave's own seek table.
That closes the question and kills the 'usable length or loop end' reading of
w0 -- it is just the previous seek point. It also explains why w1-w0 is always a
multiple of 512 without needing a separate fact: consecutive seek points advance
by whole 512-sample frames, so adjacent entries always differ by a whole number
of them. What I had recorded as a meaningful constraint was a property of the
table the values were copied from.
Neither word is the wave's true length -- w1 lands within 512 of the
PsuedoBytesPerSec length in only 9 of 282, consistent with the correction above.
Still unidentified: the ~40 scattered non-zero bytes elsewhere in the block.
Chasing w0 I decoded leading waves and the output matched neither trailer word.
Following that into RIFF waves, where the extent is certain, the decoded sample
count exceeds the seek table's last cumulative sample by a median 9.7%.
The tiebreak is the bank's own PsuedoBytesPerSec: over 14 banks it agrees with
an actual FFmpeg decode to a mean of 0.007 s and with the seek-table duration
only to 0.287 s. Two independent quantities agree with each other and both
disagree with the seek total, so that last entry is the last SEEK POINT, not the
sample count.
Both artifacts regenerated with data_bytes / PsuedoBytesPerSec. Total audio is
408.3 minutes, not the 390.9 I published -- 4.3% in aggregate. Dialogue timings
all rise, e.g. 'They got Leader!' from 1.78/2.46 s to 2.25/2.93 s.
What made this hard to catch is worth recording: the seek total gave BGM of 2.4
minutes, chatter of 2.8 seconds and cutscenes of 11 minutes, and I cited that as
the chain validating itself. It validated the structure, not the scale -- a
uniform 10% error preserves every ratio I checked.
The trailer opens with two little-endian u32 words. w1 is the leading wave's
total sample count -- 282/282 exact, the same number as the last entry of that
wave's own seek table. And w0 < w1 with w1-w0 always a whole multiple of 512
(282/282), i.e. a whole number of XMA1 frames, 9 to 91 with a median of 14.
What w0 means is left at yellow: trailing the total by a whole number of frames
is the shape of a usable-length or loop-end field, but that is a reading of the
shape rather than a measurement, and nothing here separates the candidates.
Also corrects my own description: I called the region after the seek chunk 'zero
padding'. It is not padding -- about 48 non-zero bytes are scattered thinly
across the 12 KiB, roughly 17 per KiB in blocks 1, 3 and 7. That is a sparse
table, still unidentified, and the distinction matters to anyone skipping it.
Dumped it. In all 282 exceptions the region from the earlier wave's seek to the
first RIFF is exactly 12288 bytes: the seek chunk (240-260 bytes), then 47-57
non-zero trailer bytes, then zero padding out to the block size.
So an entry may hold a leading wave, a 12288-byte padded block, then its RIFF
wave -- and the assignment rule completes: first-or-second wave at/after the
entry offset is 7620/7620 = 100.00%, with ZERO unexplained.
This overturns my own refutation from one iteration ago. I proposed these were
leading segments, tested it as 'the seek should sit AT the first RIFF', got 0 of
282, and recorded the hypothesis refuted. It was right; my predicate was wrong
by exactly the padded block, a constant I had not yet found. A negative result is
only as good as the predicate it tests.
Still unidentified: what the 47-57 trailer bytes hold.
Scanning the flat stream for seek chunks satisfying size == 8+4*count with a
non-negative implied start yields 9661 waves and rejects NOTHING -- no false
positives in a gigabyte of audio. Each wave's extent is
[seek_pos - count*2048, seek_pos).
Assignment rule 'an entry names the first wave starting at or after its offset'
is 96.30% correct (7338/7620) against entries whose answer is independently
known from their RIFF.
The 282 exceptions are one class: the gap between the earlier wave's seek and
the first RIFF is EXACTLY 12288 bytes in all 282. I guessed they were leading
segments, which would put that seek AT the RIFF; refuted, 0 of 282.
Recorded at yellow. Correcting by 12288 would reach 100% on this set but is
curve-fitting until the region is identified, and the rule cannot yet unblock
headerless durations -- there is nothing to check it against there.
I read the seek chunk's packet count big-endian; it is little-endian at seek+12,
with size == 8 + 4*count. And a seek sits immediately AFTER its own data, so an
entry's first seek usually belongs to the PREVIOUS bank (implied start -25232
for D_452, -145988 for TCAF_608). I was comparing an entry's first seek against
its first data -- different waves by construction, which is why no reading lined
up.
With that fixed, the declared sizes are honest: every RIFF-bearing entry on the
disc has seek magic at exactly data_at + declared_size with count*2048 ==
declared. 7620/7620, zero failures.
VOICE_TCAF_608 is not truncated. Its Channels is 2 and I decoded it as mono;
read as stereo it gives 6520176 bytes = 33.96 s, agreeing with both length
signals in the bank (33.88 s from cumulative samples, 33.97 s from
PsuedoBytesPerSec). 170 of 8021 banks (2.12%) are stereo -- exactly the rate of
my 1-in-60 outlier.
This is the mono/stereo trap already documented on this very page, met from the
other direction: I had written 'at two channels every bank yields one frame' and
then spent several passes blaming missing data for a one-frame decode.
Code fix: to_xma_riffs built the leading segment with a hard-wired mono fmt. It
now reads Channels from the bank's first RIFF. 7 disc tests pass.
Identified the structure: a little-endian size field after the tag, then
0x01000000, a varying word, a zero, then a strictly ascending table. Sizes 64 /
108 / 348 bytes for three sample banks.
Neither obvious reading gives the declared packet count. Entry count minus the
three header words is exactly right for VOICE_D_452 (13) and wrong for the other
two (24 vs 17, 84 vs 371) -- a one-of-three fit, recorded as FAILED rather than
as a rule with exceptions, because two measurements today already returned
plausible numbers for questions they could not answer.
The ascending values step by 1.5-2.2 million, far too large to be packet
indices, so they are on some other scale. Recorded what the next attempt should
know, including that auto/slb-loader chains seek packet counts successfully by
some field that is not either of the two tried here.
Generalising the TCAF_608 test across the 5296 over-declaring banks first looked
like a clean reversal: 59 of 60 gained audio (median 2.10x) from reading the full
declared size instead of stopping at the TOC window.
Then I tested whether the declared size is an honest boundary by reading TWICE
it. 33 of 40 yield >1.5x more audio, median ratio 1.64. The decoder keeps going
indefinitely, because XMA1 packets are self-contained -- the next bank's packets
decode perfectly well and are simply not this bank's. A byte count cannot tell
them apart.
So the 59/60 is withdrawn as evidence for anything. Same error as the unanchored
'data' search earlier on this page: a measurement returning a plausible number
for a question it cannot answer.
What survives: TCAF_608 is genuinely anomalous (86x more input, zero extra
output, where the typical bank yields without limit); the honest-size question
is unresolved and needs a boundary signal independent of the decoder, such as
the seek-chunk packet chain; and the only measured fact is that the declared
size exceeds the TOC window for 5296 of 7586 banks.
Read VOICE_TCAF_608's full declared 759808 bytes out of the flat stream ignoring
window boundaries. It decodes to the same 896 bytes (0.01 s) as stopping at the
window end -- 86x more input, zero extra samples. Control VOICE_D_452, whose
declared size fits, decodes to 2.18 s normally. No segment join is crossed, so
the packing phase is not a confound.
So 'the bytes are outside the window but still in the stream' does not hold for
the one bank the other branch named as not truncated. I still withdraw my word
'truncated' as an over-claim about the other 5295 banks: I measured that their
declared size exceeds their window, not what lies beyond it, and have now tested
exactly one. Recorded the three claims at their actual strengths.
Reproduced independently: X = (cumulative start of the .pNN segment) mod 2048.
The running sums of the five sound.pNN file sizes are 0, 1392, 1468, 1600, 1728
-- exactly the four values measured, plus zero. The XMA grid is 2048-aligned
inside each segment file, the segments are not multiples of 2048 long, so each
join shifts the phase.
Two of my explanations on this page were wrong even though the measurements
stand. 'Varies by language and subdirectory' was correlation, not cause --
directories cluster into segments. And there is no header at all: those bytes
are the PREVIOUS bank's audio, which is why they had no length field and looked
like data.
Also records a disagreement I could not resolve. auto/slb-loader withdraws my
'69.8% over-declare' finding as declared-sizes-are-exact, but VOICE_TCAF_608
declares 759808 bytes of data with 55296 bytes to the next TOC entry. Both
readings agree the declared size exceeds the window; mine said 'truncated',
which I withdraw as an over-claim, but 'exact' requires a wave to span ~14 named
entries. Recorded with the experiment that would decide it.
Probed the bytes before the data offset at all four values. Not a length field
(no word in the first 64 bytes matches the offset, the RIFF position or the entry
size, either endianness). Not a seek table or any ascending index (~50% of
consecutive BE word pairs are non-decreasing, i.e. random; all words distinct,
none zero). Not zero padding in general (5876 of 7358 have content). Not audio
being discarded (offset 0 wins 6 of 7358).
High-entropy content whose size is constant per language and subdirectory, with
no self-describing length -- which points at a loader that knows the size a
priori. Recorded the next step as static PE work rather than another archive
pass; the byte-level evidence here is about exhausted.
The ties needed a different signal, not a longer scan. Banks carry one: a seek
chunk sitting on a packet boundary, so seek_pos % 2048 IS the data offset. On
the 6033 labelled banks with a seek before their first RIFF, 6031 agree
(99.97%) -- better than the packet scan and structural rather than statistical,
so scan_data_offset now tries it first.
On the scan's 28 ties it resolves 26 correctly and 0 wrongly (2 have no usable
seek). Combined rule scores 7354/7358 = 99.95%, up from 99.62%. 762 of the 1495
RIFF-less banks carry a seek, so the signal exists where it is needed.
Also ruled out, since a wrong offset was this page's whole subject: the header
is not audio being discarded. Adding 0 to the candidate set, it wins 6 of 7358.
7 disc tests pass.
The labelled set has a RIFF and the scan is unbounded, so it reads past the RIFF
there -- the headline number could have been borrowing discrimination that a
RIFF-less bank cannot offer. Confining the scan to the leading region gives
69.98%, which looks like exactly that problem.
It is not. Split by how much leading audio there is: on the 989 banks with >=24
packets of it, the scan is 100% correct with ZERO ties, whether or not the RIFF
is in range. The 69.98% is an artifact of short leading regions, where two or
three packets are not enough to separate candidates.
A RIFF-less bank is a whole pak entry, so 24 packets are always available. The
99.62% is conservative for the population the scan serves, not optimistic.
Of the 7586 banks with a RIFF and a data chunk after it, 5296 declare a data
size larger than the pak entry holds; 2290 declare less (the ordinary
multi-sub-wave case); NONE declare exactly what they hold. This contradicts the
decoder comment claiming the declared size 'is honest per sub-wave'. The code
clamps, so it is a documentation defect, not a crash.
It also closes the loose end from the offset work: eng\Voice\VOICE_TCAF_608,
the single bank where neither offset decoded, is 99% short -- there is nothing
there to decode.
Method note recorded: my first pass searched for 'data' from offset 0, which can
match by chance inside the leading audio region. Anchoring the search after the
first RIFF moved the count 5038 -> 5296.
Separately, the 55 'early RIFF' English banks are not an anomaly: all 55 sit at
exactly 1392 behind a zero-filled header -- a zero-length leading region, which
both the old and new code already handle correctly.
1495 banks carry no RIFF and take a separate path that hardcoded both the
offset and stereo. Across a random 48-bank sample there was NOT ONE where the
old stereo-at-1392 pair beat the best mono offset; median gain 184x, individual
banks going from 0-4816 decoded bytes to 180000-380000. Stereo shows the same
stop-after-one-frame signature already recorded for the leading segment.
With no RIFF the offset cannot be derived, so scan_data_offset picks among the
four disc offsets by XMA1 packet-header plausibility. Validated on the LABELLED
set -- all 7358 banks that do have a RIFF, where the answer is forced: 7330
correct (99.62%), and all 28 misses are ties on the top score, never a wrong
unique winner. Ties fall back to 1392.
The winning offsets also reproduce, by directory, the distribution measured
independently from the RIFF-bearing banks. jpn\etc splits 1468/1600, so path
alone is not sufficient -- which is why this is a scan and not a lookup table.
7 disc tests pass (build-reborn test -p sylpheed-formats --test
slb_leading_segment_disc, SYLPHEED_DISC wired up).
HEADERLESS_DATA_OFFSET is the value the offset takes in <lang>\etc\, not a
property of the format. The leading stream is a whole number of 2048-byte XMA1
packets ending at the first RIFF, so its start is first_riff % XMA1_PACKET.
Disc-wide that takes four values -- 1392, 1468, 1600, 1728 -- varying by
language and subdirectory.
Verified by decoding, not by arithmetic: on a random 140-bank sample with a
non-empty leading region, the derived offset yields more audio in 85, identical
in 54 (the eng\etc controls, where it must and does reproduce the old
behaviour) and less in 1. Median gain among the improved is 70x --
eng\Voice\VOICE_TCAF_592 goes 1506 -> 97152 bytes, jpn 2910 -> 127178.
This withdraws my own claim from earlier today that the Japanese banks were a
different undecoded layout. They are the same format with a different offset;
I had treated a constant derived from one subdirectory as a property of the
format. The same error was hiding the identical defect in 1873 eng\Voice banks.
