Chasing w0 I decoded leading waves and the output matched neither trailer word.
Following that into RIFF waves, where the extent is certain, the decoded sample
count exceeds the seek table's last cumulative sample by a median 9.7%.
The tiebreak is the bank's own PsuedoBytesPerSec: over 14 banks it agrees with
an actual FFmpeg decode to a mean of 0.007 s and with the seek-table duration
only to 0.287 s. Two independent quantities agree with each other and both
disagree with the seek total, so that last entry is the last SEEK POINT, not the
sample count.
Both artifacts regenerated with data_bytes / PsuedoBytesPerSec. Total audio is
408.3 minutes, not the 390.9 I published -- 4.3% in aggregate. Dialogue timings
all rise, e.g. 'They got Leader!' from 1.78/2.46 s to 2.25/2.93 s.
What made this hard to catch is worth recording: the seek total gave BGM of 2.4
minutes, chatter of 2.8 seconds and cutscenes of 11 minutes, and I cited that as
the chain validating itself. It validated the structure, not the scale -- a
uniform 10% error preserves every ratio I checked.
The trailer opens with two little-endian u32 words. w1 is the leading wave's
total sample count -- 282/282 exact, the same number as the last entry of that
wave's own seek table. And w0 < w1 with w1-w0 always a whole multiple of 512
(282/282), i.e. a whole number of XMA1 frames, 9 to 91 with a median of 14.
What w0 means is left at yellow: trailing the total by a whole number of frames
is the shape of a usable-length or loop-end field, but that is a reading of the
shape rather than a measurement, and nothing here separates the candidates.
Also corrects my own description: I called the region after the seek chunk 'zero
padding'. It is not padding -- about 48 non-zero bytes are scattered thinly
across the 12 KiB, roughly 17 per KiB in blocks 1, 3 and 7. That is a sparse
table, still unidentified, and the distinction matters to anyone skipping it.
Dumped it. In all 282 exceptions the region from the earlier wave's seek to the
first RIFF is exactly 12288 bytes: the seek chunk (240-260 bytes), then 47-57
non-zero trailer bytes, then zero padding out to the block size.
So an entry may hold a leading wave, a 12288-byte padded block, then its RIFF
wave -- and the assignment rule completes: first-or-second wave at/after the
entry offset is 7620/7620 = 100.00%, with ZERO unexplained.
This overturns my own refutation from one iteration ago. I proposed these were
leading segments, tested it as 'the seek should sit AT the first RIFF', got 0 of
282, and recorded the hypothesis refuted. It was right; my predicate was wrong
by exactly the padded block, a constant I had not yet found. A negative result is
only as good as the predicate it tests.
Still unidentified: what the 47-57 trailer bytes hold.
Scanning the flat stream for seek chunks satisfying size == 8+4*count with a
non-negative implied start yields 9661 waves and rejects NOTHING -- no false
positives in a gigabyte of audio. Each wave's extent is
[seek_pos - count*2048, seek_pos).
Assignment rule 'an entry names the first wave starting at or after its offset'
is 96.30% correct (7338/7620) against entries whose answer is independently
known from their RIFF.
The 282 exceptions are one class: the gap between the earlier wave's seek and
the first RIFF is EXACTLY 12288 bytes in all 282. I guessed they were leading
segments, which would put that seek AT the RIFF; refuted, 0 of 282.
Recorded at yellow. Correcting by 12288 would reach 100% on this set but is
curve-fitting until the region is identified, and the rule cannot yet unblock
headerless durations -- there is nothing to check it against there.
343 English headerless banks carry a valid seek chunk, which would lift dialogue
duration coverage from 62% to ~75%, with a median 3.25 s that looks exactly like
in-mission voice.
It is wrong. A seek sits immediately after its OWN data, so the implied wave
start is seek_pos - packets*2048 -- and for 277 of the 343 that start lies
BEFORE the entry begins, meaning the seek describes the previous bank's wave.
Applying it would attach a neighbour's duration to 81% of the recovered rows.
The 66 that start inside the entry are not proven to be that entry's wave
either; 'starts inside' is necessary, not sufficient.
Recorded as refused rather than applied with a caveat, because a plausible
median is exactly what has misled this work twice today. Recovering these needs
the bank-to-wave assignment settled first.
Joins the ISL script's built-in 64 call sites, the caption table and the sound
bank seek tables. Duration is the last cumulative sample over the sample rate --
read, never decoded. 1663 of 2683 call sites (62.0%) have an English duration,
2198 (81.9%) a Japanese one; the gap is the headerless banks, and which language
lacks one differs per line, so the columns are complementary.
Two columns deliberately: FILES names the jpn banks and the game substitutes the
language directory, so a line's English and Japanese recordings are different
files of different lengths -- 'They got Leader!' is 1.78 s in English and 2.46 s
in Japanese. The captions are English, so eng is the matching column. The first
version of this artifact had one column and would have attached Japanese timings
to English text; that is why it was regenerated.
With the wave boundary exact and Channels read rather than assumed, every field
describing a bank can be read: path, cue, sound id, channels, rate, data bytes,
packets, samples, seconds. Duration comes from the last cumulative sample in the
seek table over the sample rate -- arithmetic, no decoding.
Nothing was fitted to an expectation, yet every category lands where its content
says it should: BGM mean 146 s, Movie mean 79 s with an 11-minute maximum,
Briefing 10.8 s, in-mission Voice 2.79 s never exceeding 7.3. If the boundary
rule, the channel field or the seek table were misread, the numbers would not
sort into those five shapes.
Incidental: 4 banks run at 44100 Hz against 48000 everywhere else, and the BGM
tracks are the stereo ones. 1021 of the 5135 FILES paths have no RIFF and are
excluded as headerless.
The declared sizes are honest (seek magic at data_at + declared_size, 7620/7620)
and VOICE_TCAF_608 was stereo decoded as mono, not truncated. The four offsets
are a segment-packing phase, not a per-directory header size. Neither was closed
by finding something new; both were closed by correcting a mistake of mine.
I read the seek chunk's packet count big-endian; it is little-endian at seek+12,
with size == 8 + 4*count. And a seek sits immediately AFTER its own data, so an
entry's first seek usually belongs to the PREVIOUS bank (implied start -25232
for D_452, -145988 for TCAF_608). I was comparing an entry's first seek against
its first data -- different waves by construction, which is why no reading lined
up.
With that fixed, the declared sizes are honest: every RIFF-bearing entry on the
disc has seek magic at exactly data_at + declared_size with count*2048 ==
declared. 7620/7620, zero failures.
VOICE_TCAF_608 is not truncated. Its Channels is 2 and I decoded it as mono;
read as stereo it gives 6520176 bytes = 33.96 s, agreeing with both length
signals in the bank (33.88 s from cumulative samples, 33.97 s from
PsuedoBytesPerSec). 170 of 8021 banks (2.12%) are stereo -- exactly the rate of
my 1-in-60 outlier.
This is the mono/stereo trap already documented on this very page, met from the
other direction: I had written 'at two channels every bank yields one frame' and
then spent several passes blaming missing data for a one-frame decode.
Code fix: to_xma_riffs built the leading segment with a hard-wired mono fmt. It
now reads Channels from the bank's first RIFF. 7 disc tests pass.
The '3 or 4 wrapped lines' sentence is in isl-message-dialogue-link.md itself,
not in cutscene-message-table.md as my correction claimed. Corrected the
sentence in place -- a page has four line SLOTS, of which an English page fills
1 to 4 -- and fixed the misattribution in the correction note rather than
quietly repointing it.
The five European packs have byte-for-byte identical MSG key sets: 18813 keys,
4091 ids, zero differences in any direction. Japanese has 14295 keys, which
looks like 4521 missing captions and is not: its id set matches to within 5, its
page count is the same (4765 vs 4758), and its mean lines per page is 1.52
against English's 1.85 with no 4-line pages at all. It carries the same dialogue
and wraps into fewer lines, which is what a language without inter-word spaces
does in a fixed-width box.
Also corrects my own wording in cutscene-message-table.md, which said a page
'holds 3 or 4 wrapped lines'. That counted key SLOTS, which MSG_DEMO records
allocate four at a time. Counting lines with text, an English page holds 1 to 4
(1574/2439/632/113). And it reconciles the 8800 figure used throughout: that is
distinct English keys with non-empty text, out of 18813 keys total.
Identified the structure: a little-endian size field after the tag, then
0x01000000, a varying word, a zero, then a strictly ascending table. Sizes 64 /
108 / 348 bytes for three sample banks.
Neither obvious reading gives the declared packet count. Entry count minus the
three header words is exactly right for VOICE_D_452 (13) and wrong for the other
two (24 vs 17, 84 vs 371) -- a one-of-three fit, recorded as FAILED rather than
as a rule with exceptions, because two measurements today already returned
plausible numbers for questions they could not answer.
The ascending values step by 1.5-2.2 million, far too large to be packet
indices, so they are on some other scale. Recorded what the next attempt should
know, including that auto/slb-loader chains seek packet counts successfully by
some field that is not either of the two tried here.
Generalising the TCAF_608 test across the 5296 over-declaring banks first looked
like a clean reversal: 59 of 60 gained audio (median 2.10x) from reading the full
declared size instead of stopping at the TOC window.
Then I tested whether the declared size is an honest boundary by reading TWICE
it. 33 of 40 yield >1.5x more audio, median ratio 1.64. The decoder keeps going
indefinitely, because XMA1 packets are self-contained -- the next bank's packets
decode perfectly well and are simply not this bank's. A byte count cannot tell
them apart.
So the 59/60 is withdrawn as evidence for anything. Same error as the unanchored
'data' search earlier on this page: a measurement returning a plausible number
for a question it cannot answer.
What survives: TCAF_608 is genuinely anomalous (86x more input, zero extra
output, where the typical bank yields without limit); the honest-size question
is unresolved and needs a boundary signal independent of the decoder, such as
the seek-chunk packet chain; and the only measured fact is that the declared
size exceeds the TOC window for 5296 of 7586 banks.
Read VOICE_TCAF_608's full declared 759808 bytes out of the flat stream ignoring
window boundaries. It decodes to the same 896 bytes (0.01 s) as stopping at the
window end -- 86x more input, zero extra samples. Control VOICE_D_452, whose
declared size fits, decodes to 2.18 s normally. No segment join is crossed, so
the packing phase is not a confound.
So 'the bytes are outside the window but still in the stream' does not hold for
the one bank the other branch named as not truncated. I still withdraw my word
'truncated' as an over-claim about the other 5295 banks: I measured that their
declared size exceeds their window, not what lies beyond it, and have now tested
exactly one. Recorded the three claims at their actual strengths.
Reproduced independently: X = (cumulative start of the .pNN segment) mod 2048.
The running sums of the five sound.pNN file sizes are 0, 1392, 1468, 1600, 1728
-- exactly the four values measured, plus zero. The XMA grid is 2048-aligned
inside each segment file, the segments are not multiples of 2048 long, so each
join shifts the phase.
Two of my explanations on this page were wrong even though the measurements
stand. 'Varies by language and subdirectory' was correlation, not cause --
directories cluster into segments. And there is no header at all: those bytes
are the PREVIOUS bank's audio, which is why they had no length field and looked
like data.
Also records a disagreement I could not resolve. auto/slb-loader withdraws my
'69.8% over-declare' finding as declared-sizes-are-exact, but VOICE_TCAF_608
declares 759808 bytes of data with 55296 bytes to the next TOC entry. Both
readings agree the declared size exceeds the window; mine said 'truncated',
which I withdraw as an over-claim, but 'exact' requires a wave to span ~14 named
entries. Recorded with the experiment that would decide it.
Built-in 12 activate_unit returns 0 when the live object is NULL: it registers an
object that already exists and cannot create one. So a wave arrival is a craft
reaching a point on its route, not a new record, and a count of entity records
was never going to move.
Stage 02's UnitGroup totals Count = 116 across 111 squadrons, matching the live
probe's flat 116. The natural objection is that 116 might match by accident --
it does not: across all 28 stages the roster totals run 2..116, and 116 is the
MAXIMUM and unique to Stage 02, with nothing else within 5.
Three independent things agree: the handler cannot spawn, the roster totals 116,
and the live count sits at 116 while 22 of 113 routed squadrons have a route
starting after t=0.
Still n=1 for the per-member identity (only a Stage 02 save exists), and this
cannot settle timetable-vs-event -- that needs a position probe, not a count.
The pooled cross-tab gave p ~= 0.02 for 'no Yes-flagged squadron has a delayed
route', recorded at a cautious yellow. Breaking it down by stage removes it:
all 55 Yes routes live in 2 stages, and 45 are in S14, which has NO delayed
route at all -- those 45 cannot be evidence either way. Only S04 contains both
a Yes squadron and a delayed route.
Within S04 the exact test is P = C(47,5)/C(57,5) = 0.37, expected 0.88 observed
0. Unremarkable. The pooled p treated 1023 routes as independent draws when they
are two clusters.
The doc had already flagged this as the kind of near-fit that misled the corpus
before; that caution was right and did not go far enough.
Route keyframes are (time, quat x4, pos x3), so each consecutive pair gives an
implied speed. The flight-speed law was measured live against a real-time clock
(cruise ~420, max ~1530 world units/s), which makes it an independent yardstick
in known units.
Across all 28 stage tables, 331 Route_* records and 1104 keyframe pairs: under
SECONDS not one pair exceeds the 1530 ceiling (median 147); under frames at 30
or 60 fps, 89% and 94% of the game's routes would demand impossible speeds.
Two corrections recorded. A first pass silently skipped non-numeric fields while
flattening records, shifting every later value -- it reported a median implied
speed of 65534, suspiciously 2^16, which is what exposed it. And the remaining
outliers were Frame_S<NN>_Asteroid records, not routes; the final measurement
filters by record kind rather than dropping whole stages.
Probed the bytes before the data offset at all four values. Not a length field
(no word in the first 64 bytes matches the offset, the RIFF position or the entry
size, either endianness). Not a seek table or any ascending index (~50% of
consecutive BE word pairs are non-decreasing, i.e. random; all words distinct,
none zero). Not zero padding in general (5876 of 7358 have content). Not audio
being discarded (offset 0 wins 6 of 7358).
High-entropy content whose size is constant per language and subdirectory, with
no self-describing length -- which points at a loader that knows the size a
priori. Recorded the next step as static PE work rather than another archive
pass; the byte-level evidence here is about exhausted.
The ties needed a different signal, not a longer scan. Banks carry one: a seek
chunk sitting on a packet boundary, so seek_pos % 2048 IS the data offset. On
the 6033 labelled banks with a seek before their first RIFF, 6031 agree
(99.97%) -- better than the packet scan and structural rather than statistical,
so scan_data_offset now tries it first.
On the scan's 28 ties it resolves 26 correctly and 0 wrongly (2 have no usable
seek). Combined rule scores 7354/7358 = 99.95%, up from 99.62%. 762 of the 1495
RIFF-less banks carry a seek, so the signal exists where it is needed.
Also ruled out, since a wrong offset was this page's whole subject: the header
is not audio being discarded. Adding 0 to the candidate set, it wins 6 of 7358.
7 disc tests pass.
Having found that the cue-less cutscene pages bind their audio through the movie
path, I tested the same explanation on the twelve script lines with no cue and
no bank. It fails: matching their caption text against every line in the
cutscene message table gives 0 of 14 hits, so nothing carries their audio by
another route.
They are ordinary story dialogue ('Katana, wait for me in hell'), which makes a
deliberate subtitle-only design unlikely -- but that is a judgement, not a
measurement, and why they are unvoiced is still open. Also noted: D_609 and
D_610 are equally unvoiced and simply are not among the twelve, because the
twelve counts script-called names and nothing calls them.
16 of the 17 pages with an empty slot 4 belong to MSG_DEMO_600..604, whose
caption text matches, word for word, the five resupply lines documented from the
movie side as VOICE_D_450..454. Slot 4 is empty on purpose: that audio binds
through the movie path, not through a DEMO_nnn cue.
That also disposes of the '5 ids with more than one record' item -- each appears
once per stage that uses it, and the repeat counts equal the movie-slot counts
recorded independently for the matching banks: 3/3/4/4/2 both ways, 5 of 5. The
11 extra records over 138 ids are these repeats.
Worth more than closing two items: the cutscene message table and the movie
subtitle chain were decoded from different containers by different routes and
agree on the same five lines with the same multiplicities.
MSG_DEMO_500 remains: one page, KATANA, 2s, no text and no cue.
The labelled set has a RIFF and the scan is unbounded, so it reads past the RIFF
there -- the headline number could have been borrowing discrimination that a
RIFF-less bank cannot offer. Confining the scan to the leading region gives
69.98%, which looks like exactly that problem.
It is not. Split by how much leading audio there is: on the 989 banks with >=24
packets of it, the scan is 100% correct with ZERO ties, whether or not the RIFF
is in range. The 69.98% is an artifact of short leading regions, where two or
three packets are not enough to separate candidates.
A RIFF-less bank is a whole pak entry, so 24 packets are always available. The
99.62% is conservative for the population the scan serves, not optimistic.
Of the 7586 banks with a RIFF and a data chunk after it, 5296 declare a data
size larger than the pak entry holds; 2290 declare less (the ordinary
multi-sub-wave case); NONE declare exactly what they hold. This contradicts the
decoder comment claiming the declared size 'is honest per sub-wave'. The code
clamps, so it is a documentation defect, not a crash.
It also closes the loose end from the offset work: eng\Voice\VOICE_TCAF_608,
the single bank where neither offset decoded, is 99% short -- there is nothing
there to decode.
Method note recorded: my first pass searched for 'data' from offset 0, which can
match by chance inside the leading audio region. Anchoring the search after the
first RIFF moved the count 5038 -> 5296.
Separately, the 55 'early RIFF' English banks are not an anomaly: all 55 sit at
exactly 1392 behind a zero-filled header -- a zero-length leading region, which
both the old and new code already handle correctly.
static.slb (8970240 bytes, the shared SE bank) and Pj_Silph.xgs (533 bytes, XACT
global settings) both hash into the TOC. Their names come from the BANK_SE and
SETTINGS records of the very IDXD object this page documents -- I had printed
them at the top of the write-up and then reported the entries as unidentified.
9519 of 9519 now: 5100 jpn + 4382 eng + 35 root + these 2.
1495 banks carry no RIFF and take a separate path that hardcoded both the
offset and stereo. Across a random 48-bank sample there was NOT ONE where the
old stereo-at-1392 pair beat the best mono offset; median gain 184x, individual
banks going from 0-4816 decoded bytes to 180000-380000. Stereo shows the same
stop-after-one-frame signature already recorded for the leading segment.
With no RIFF the offset cannot be derived, so scan_data_offset picks among the
four disc offsets by XMA1 packet-header plausibility. Validated on the LABELLED
set -- all 7358 banks that do have a RIFF, where the answer is forced: 7330
correct (99.62%), and all 28 misses are ties on the top score, never a wrong
unique winner. Ties fall back to 1392.
The winning offsets also reproduce, by directory, the distribution measured
independently from the RIFF-bearing banks. jpn\etc splits 1468/1600, so path
alone is not sufficient -- which is why this is a scan and not a lookup table.
7 disc tests pass (build-reborn test -p sylpheed-formats --test
slb_leading_segment_disc, SYLPHEED_DISC wired up).
HEADERLESS_DATA_OFFSET is the value the offset takes in <lang>\etc\, not a
property of the format. The leading stream is a whole number of 2048-byte XMA1
packets ending at the first RIFF, so its start is first_riff % XMA1_PACKET.
Disc-wide that takes four values -- 1392, 1468, 1600, 1728 -- varying by
language and subdirectory.
Verified by decoding, not by arithmetic: on a random 140-bank sample with a
non-empty leading region, the derived offset yields more audio in 85, identical
in 54 (the eng\etc controls, where it must and does reproduce the old
behaviour) and less in 1. Median gain among the improved is 70x --
eng\Voice\VOICE_TCAF_592 goes 1506 -> 97152 bytes, jpn 2910 -> 127178.
This withdraws my own claim from earlier today that the Japanese banks were a
different undecoded layout. They are the same format with a different offset;
I had treated a constant derived from one subdirectory as a property of the
format. The same error was hiding the identical defect in 1873 eng\Voice banks.
Independently reproduced across all 33 paks: 7750 IDXD objects, 1485577 unnamed
field entries, 7094 distinct never-named keys splitting cleanly into 7052 in an
ordinal band (<=0x2198, 94.6% equal to their own field index) and 42 hash-shaped
(>=0x2677C), with ZERO keys in the gap between. The 42 carry exactly 504
entries -- six language copies of one object times two records.
So the preimage target was 42, not 504, and my earlier wording invited the
misreading. Cross-referenced to the idxd-unnamed-keys write-up, which shows the
42 belong to <lang>\script\ID.tbl and cannot be recovered from a 24-bit hash.
Every FILES path hashes straight into the sound.pak TOC (5135/5135, path as
stored). The archive is then almost fully accounted for: 5100 jpn + 4382 eng +
35 BGM/JNGL = 9517 of 9519. Only eng and jpn resolve -- deu/fra/ita/esp give
zero, which is the archive-side measurement behind 'two voice sets shipped'.
718 Japanese banks have no English counterpart.
More importantly this bounds an existing result. The leading-region rule from
voice-bank-leading-region.md reproduces its five documented banks EXACTLY under
eng\etc\ (16384/2048/14336/45056/59392), so the pipeline is sound -- but it
holds for only 1571 of 4382 English banks and for ZERO of 5100 Japanese ones.
It is a property of some English banks, not of .slb in general, and the Japanese
layout is undecoded.
I nearly misread this as a broken extractor, because FILES lists jpn paths and
those gave non-multiples of 2048. Reproducing the recorded figures exactly under
eng is what distinguished a wrong assumption from a wrong reader.
My sound-cue write-up said the table 'does not explain' the VOICE_D_452
rejection and implied it was still open. It is not: voice-bank-leading-region.md
settled it earlier -- the bank holds one generic line reused across four stages,
so the binding is correct and the expectation was wrong. The narrower true
statement, which is what the cue table actually adds, is that the reuse is a
movie-slot -> bank relation and not a cue-level one.
dat/tables.pak holds a 5798-entry SOUNDS record (cue name -> sound id) and a
5135-entry FILES record (.slb bank paths). Cue names are the join key, so a
script message id now resolves all the way to the bank that voices it:
MSG_VOICE_D_257 -> VOICE_D_257 -> 6945 -> jpn\etc\VOICE_D_257.slb.
The prefix rule is MSG_ -> VOICE_, not strip-MSG_. My first rule was the
latter; it left 88 names unresolved and I was about to write those families up
as text-only announcements, until VOICE_TCAF_592.slb turned up in FILES and
refuted it. Corrected rule resolves 1326 of 1338, and SOUNDS and FILES agree on
exactly the same 12 absentees.
Separately, MSG_DEMO_* is driven by its own IDXD tables in the language packs,
which carry speaker, portrait, on-screen seconds and audio cue per page. Field
count is 9*PageCount+2 for all 7 distinct PageCounts; 1252/1252 caption-key
slots match <ID>_<page>_<line>; the 78 multi-page records equal the 78 counted
independently from the caption side; 138 ids close exactly against the caption
table both ways.
Does not settle the known VOICE_D_452 wrong-recording case -- every cue id is
distinct, so bank sharing is not happening at this layer.
Measured the <id>_<page>_<line> key structure: a page is one subtitle box of
3 or 4 wrapped lines, and successive pages are successive utterances by
possibly different speakers. 452 of 4091 ids span more than one page, up to 8.
That refutes the isl_dialogue.py committed two commits ago, which read only
page 000 -- 356 of the 1338 script message names are multi-page, so a quarter
of its output was truncated to the opening utterance. Tool now walks pages
until one is empty; Stage 02 sample regenerated (43 of 213 calls multi-page).
The 2683/2683 resolution figure is unaffected: it counted ids that have text,
and every id does have a page 000. What was wrong was the rendered text.
Built-in 64's slot-0 operand is a symbol-table-1 type-6 message id, and every
one of them now has caption text: 2683 of 2683 call sites across the 28 stage
scripts, 1338 distinct names, no residue of any kind.
This only became reachable once build_caption_text was switched to the IXUD
field table (537 -> 8800 lines); before that most of these names had nothing
to resolve to.
Adds isl_dialogue.py plus a committed Stage 02 sample. Does not settle which
recording plays for a given line, multi-page captions, or the other five
languages.
The row still described ixud.rs as a cue reader and said nothing about the
record table or the caption families. Rewritten with the measured numbers:
* the IXUD record/field table is decoded and wired in -- IxudObject mirrors
IdxdObject, uniform 16-byte records, 12-byte fields, every offset in CHARS,
and the word at 0x08 is record 0's hash rather than a schema id. Verified
1104/1104 objects, 1476/1476 records, 628165/628165 named fields.
* caption text goes 537 -> 8800 lines, which is 8800 of 8800 distinct keys,
in two steps: generalising the key parser to all eight families took it to
8074, and switching from token adjacency to record fields finished it.
The row also carries my correction rather than quietly dropping it: the
earlier "1.3% of the game's text" counted occurrences across blocks, and the
honest denominator is 8800 distinct keys, so the starting point was 6.1%.
And it keeps the DEMO control, which is the part a reader should remember:
token adjacency finds 537 lines in that family, fields find 541. The old
reader was dropping lines in the one family it was written for.
Status moved from 🟡/✅ to ✅.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
ixud.rs now has an IdxdObject-shaped reader, IxudObject, and build_caption_text
reads captions as FIELDS instead of pairing them with whatever token follows in
the pool.
build_demo_text token adjacency 134 ids 537 lines
build_caption_text token adjacency 3721 8074
build_caption_text record fields 4085 8800 = all of them
Verified over the whole disc by tests/ixud_records_disc.rs: 1104/1104 objects
parse, 1476/1476 records and 628165/628165 named fields reproduce their
ixud_hash, 48 positional, zero failures. The header word at 0x08 is record 0's
hash, asserted per object -- there is no schema field, exactly as for IDXD. The
module doc described a 12-byte record directory and a "schema/type hash"; both
were wrong and are corrected.
I also have to correct my own number from the previous commit. "1.3% of the
game's text" counted OCCURRENCES: each family lives in 24-45 IXUD blocks and
the same key repeats across them. Distinct text-bearing MSG_* keys number 8800,
not 44579, and every one has the <id>_<page>_<line> shape. So the real coverage
was 537/8800 = 6.1%, and I overstated the gap about fivefold. Direction right,
magnitude wrong.
The DEMO control is the sharpest evidence for the change: token adjacency finds
537 lines there, the field reader 541. It was dropping lines even in the one
family it was written for -- which is why the test now asserts "must not lose
lines" rather than "must be identical".
Same lesson twice in one session: pool adjacency is a consequence of how
records are written, not a rule of the format.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
build_caption_text generalises the key parser from MSG_DEMO_* to all eight
families. The shapes are uniform and each family is 100% consistent with its
own: seven use MSG_<FAM>_<id>_<page>_<line>, and VOICE alone inserts a family
letter before the id.
ids lines
build_demo_text 134 537
build_caption_text 3721 8074
The DEMO family comes out identical through both readers -- 537 lines either
way -- which is the control that generalising changed nothing that already
worked. Pinned by tests/caption_families_disc.rs, along with VOICE ids keeping
their family letter.
But this does NOT close the gap, and the write-up says so: 8074 against the
44579 text-bearing fields the record-level scan counts is about 18%.
The reason is the same lesson this session already learned once.
build_caption_text pairs a value with the key that happens to follow it in the
raw UTF-16 token stream -- the adjacency heuristic that was wrong for IDXD and
is wrong here for the same reason. ixud.rs has no record/field reader at all.
The IXUD record table IS decoded and verified disc-wide (1104/1104 objects,
628165/628165 fields reproducing their key) and was simply never wired into
the crate.
Next step recorded: give ixud.rs an IdxdObject-shaped reader and read captions
as fields rather than adjacent tokens. The decode exists; only the plumbing is
missing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
movie_subtitle handles MSG_DEMO_*, the cutscene captions. Counting every IXUD
block in GP_MAIN_GAME_E.pak, that is the SMALLEST of eight families:
MSG_ADAN 23236 keys 9801 with text ADAN combat chatter
MSG_RHIN 21196 8509 Rhino squadron
MSG_TCAF 17148 6728 TCAF
MSG_VOICE 13060 6776 in-mission scripted dialogue
MSG_BIRD 14036 5834 Bird squadron
MSG_ADPL 12640 4127 ADAN pilots
MSG_ACRO 4804 2244 Acropolis
MSG_DEMO 1252 560 cutscene captions <- the only one read
total 107372 44579
560 of 44579 text-bearing keys = 1.3%. I report the text-bearing column rather
than raw keys because only 41.5% of keys carry text -- the rest are the empty
line slots this container pads with, and counting those would flatter the
denominator.
MSG_VOICE_* is the family the message tables reference -- the dialogue whose
voice bindings this file now analyses in detail -- and nothing in crates/
parses it. So the corpus knows which bank plays for a line it cannot read.
First step recorded: build_demo_text already pairs a text value with the
MSG_DEMO_<demo>_<page>_<line> key that follows it, and the other seven
families use the same <id>_<page>_<line> shape, so generalising the key parser
is most of the work. With a warning attached: do NOT assume the id spaces
relate, since the voice-bank id and the caption id within one message page are
different numbers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
The 60 tokens absent from BOTH voice languages, itemised:
45 VOICE_E_ family -- 44 numeric [0..43] plus the lettered VOICE_E_012B
13 VOICE_C_ at 421, 423-426, 430, 432, 447-450, 470, 471 -- INSIDE the
listed range [0..489], so interior gaps rather than a truncated tail
2 VOICE_D_182 and _183, adjacent
These lines were written and captioned; only the audio is missing. Their
caption keys resolve to real text in the IXUD blocks:
MSG_VOICE_C_355_000_00 "What are you doing? Quit wasting..."
MSG_VOICE_C_367_000_00 "The final defense weapon is..."
MSG_VOICE_C_347_000_00 (Japanese)
MSG_VOICE_D_152_000_00 (Japanese)
MSG_VOICE_E_044_000_00 (Japanese)
Three of the five sampled are still Japanese INSIDE the English pak --
captioned but never translated, matching the untranslated entries already
noted for the localised-text container.
Separately, a trap worth its own heading: within one message page the voice
bank token and the caption keys use DIFFERENT numbering.
Message_106 voice VOICE_C_468 lines MSG_VOICE_C_385_000_00..02
Message_129 voice VOICE_D_182 lines MSG_VOICE_D_152_000_00..02
Message_044 voice VOICE_E_012B ID MSG_VOICE_E_044
Same family letter, different index space. Deriving one id from the other
will silently mis-pair audio with text -- which is the same class of mistake
as the demo-id voice binding this corpus already had to reject in-game.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Last iteration I read the scattered missing runs as cut dialogue. Checking the
other languages refutes that for most of them.
Only two voice languages exist: tables.pak holds eng\sounds.tbl (4418 banks)
and jpn\sounds.tbl (5136), and NO fra/deu/esp/ita. The other four
localisations are subtitle-only, which is itself a fact the corpus did not
record.
The English set is a strict subset of the Japanese one:
eng jpn
listed 4418 5136
resolved 4175 4383
missing 268 60
lacked by
the other 0 718
Every id missing in Japanese is also missing in English, and NONE is missing
in Japanese but present in English. So the 268 split cleanly:
208 exist in Japanese, not in English -- recorded in the original and
never dubbed. A localisation gap, not cut content.
60 absent from both -- genuinely cut, including all 44 VOICE_E_ (listed
0 times in either language) plus 16 others.
Per family eng/jpn: A 683/771, B 383/423, C 366/474, D 387/546. The 718
Japanese-only banks include whole families English never had -- VOICE_ADAN_
(110), VOICE_ZZZZ_ (104), VOICE_TCAF_ (41), VOICE_RHIN_ (27).
So VOICE_E_ survives as the one genuinely-cut family, and my "short runs of
cut dialogue" reading was wrong: those lines are on the disc, in Japanese. The
previous text is kept above the correction.
This is why the "measured on English only" limit was worth writing down rather
than quietly generalising -- the second language changed the conclusion.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Following up the incidental gap from the DEMO_* work. Resolving every VOICE_*
token the way the engine does -- token, then eng\sounds.tbl for its directory,
then sound.pak -- gives 4175 resolved and 268 unresolved of 4443 distinct.
The gap is in the manifest, not the archive: the unresolved tokens are not
listed in sounds.tbl at all, and of the 4175 that ARE listed, every one is
present in the pak. Nothing is listed-then-missing.
By family, script-used ids against sounds.tbl-listed ids:
VOICE_A_ 696 [0..772] listed 683 missing 14
VOICE_B_ 391 [0..422] listed 383 missing 8
VOICE_C_ 429 [0..489] listed 366 missing 63
VOICE_D_ 508 [0..598] listed 387 missing 126
VOICE_E_ 44 [0..43] listed 0 missing 44 <- the whole family
Two different shapes, and the distinction matters. A/B/C/D lose short runs
scattered through an otherwise dense range -- 11,12,13 / 56,57 / 181,182,183
/ 389-392 -- which is what cut dialogue looks like. VOICE_E_ is absent
entirely: 44 ids used, none listed, no VOICE_E_* anywhere in the manifest. A
whole speaker or category, cut.
For the port: a missing voice bank is a RETAIL condition, not a decode bug. A
reimplementation must tolerate an unresolvable VOICE_* reference, the same way
it must tolerate SUBTITLE_S12B.tbl resolving in no language.
Limit stated: measured on GP_MAIN_GAME_E.pak and eng\sounds.tbl only; whether
the other five languages drop the same ids is untested.
An earlier probe of mine checked only three sound.pak directories and would
have mis-attributed this; sounds.tbl lists five, including Briefing. Resolving
through the manifest rather than guessing directories is what makes the 268
trustworthy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
An open question from the game_data migration: 132 dialogue lines carry an
unprefixed DEMO_* token where a VOICE_* one is expected, and whether DEMO_*
names a bank was untested. It does not.
VOICE_* 4443 distinct, 4175 resolve in sound.pak
(2360 eng\Voice\, 1815 eng\etc\)
DEMO_* 296 distinct, 0 resolve
Zero resolve under any of the three directories, and looking them up through
the convention -- VOICE_DEMO_nnn -- resolves 0 of 296 as well. So the slot is
not "the voice bank"; it can hold either a bank reference or a demo one.
The tokens run DEMO_000 to DEMO_310, 287 distinct in [0, 310] -- a dense
sequential id space, which is the shape of the demo id this file already uses
in MSG_DEMO_<demo>_<page>_<line> caption keys. I did NOT confirm the overlap
and say so: those keys live in IXUD wide-string blocks while my scan walked
IDXD objects only, so it found 0 of them. That needs an IXUD-aware sweep.
Incidental and left open: 268 of the 4443 VOICE_* tokens do not resolve
either. That is a separate gap and the DEMO_* finding does not explain it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
The last file still resting on the "defaulted on disc" premise. Re-checked
through the record table over every unit-like object on the disc -- 114 with
a Generic.Type of Craft or Vessel:
present in ALL 114 objects 50
present in some 49
present in NONE 2 <- FiringRange, SELength
So only FiringRange and SELength are genuinely absent, out of 101 listed.
FiringRange was already recorded correctly elsewhere (runtime 0 is the
accessor's miss value); SELength joins it.
One pattern worth keeping rather than dismissing: 15 of the fields sit in
exactly 5 objects each -- AB_* (afterburner), PowerCut*, AA_/AV_AxisMode_*.
That is the Delta Saber family, so those fields are player-craft-only rather
than defaulted, which is a different fact than either the file or my
correction to it would suggest on its own.
The runtime values themselves stay -- they are real measurements. What is
withdrawn is the claim that the disc does not carry the fields.
That completes the sweep: weapon-datasheet-runtime.md, unit-struct-runtime.md,
live-unit-definitions.md, INDEX.md and now this capture all carried the same
wrong premise from the same blind reader.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
The two headline rows for weapons and units were the last place still telling
a reader that these values are not on the disc. Both corrected in place, with
numbers I measured rather than adopted.
Weapons: re-reading captures/weapon-runtime-fields.csv through the record
table, 1514 of the 4393 rows labelled defaulted-on-disc DO have a value on
disc; 2879 are genuinely absent. I state that as an upper bound -- my matcher
counts a field when it appears in ANY record of the object, and a per-record
count gives ~1448. Either way the headline "4393 values the disc does not
carry" is wrong by about a third.
The spot checks are exact rather than aggregate: wep_05/wep_60
TriggerShotCount 4, wep_02 Power 100.0, wep_60 Power 1000.0 (refuting the
recorded "C band 150-500" bracket), wep_25 MaximumRange 4000.0,
wep_11/28/36/70 LoadingCount 6/5/5/0.
Units: the ~30-field player-craft table is on disc at exactly the values the
runtime "recovered", spread across the Generic / Shield / Mass / SE records
-- which is why a reader that could not name a record saw them as absent. And
"18 of 23 vessel records are missing at least one of Size_X/Y/Z/HP" is false:
0 of 114 objects with a Generic.Type (43 Craft + 71 Vessel) miss any of them.
Status markers moved from ✅ and ✅/🟡 to ✅/❌ so the rows no longer read as
settled-and-correct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
unit-struct-runtime.md lists ~30 player-craft fields as defaulted on disc with
values "recovered from the running game". All of them are on the disc at
exactly those values. Verified myself for UN_f001_TCAF_DeltaSaber_T:
Size_Radius 10.0, FCSRange 500000.0, ResistanceToOptics/HQRatio/
ShieldRatio/ThrusterRatio/ResistanceToShell/ResistanceToExplosion 1.0,
MassScore 0 [Generic]
ChargeSpeed 25.0, ChargeDelay_Break 10.0 [Shield]
DryMass 100.0 [Mass]
LowerHPThresholdRatio 0.3 [SE]
The record column is the explanation: the values are spread across Generic,
Shield, Mass and SE. That is exactly why the old flat reader missed them --
it could not say which record it meant, so a per-record field read as absent.
Also wrong, and measured: "18 of the 23 vessel records are missing at least
one of Size_X/Y/Z/HP, nearly always Size_Y". Over every object carrying a
Generic.Type -- 43 Craft plus 71 Vessel, 114 objects -- ZERO are missing any
of the four.
live-unit-definitions.md's "Route B: 13 defaulted fields read out of the
running game" goes with it: all 13 Size_Y values are on disc, each exactly
equal to the value the runtime run "recovered". Nothing was recovered that
the disc did not hold; the run confirms only that the parser reads the disc.
As with the weapon sheet, the runtime captures stand as measurements. What is
withdrawn is the premise that they were reaching values the disc lacks.
Correction banners at the top of both; original text intact beneath.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Every field weapon-datasheet-runtime.md calls defaulted is on the disc. The
record table reads them directly; the old string-pool reader could not see a
value whose string is shared or reordered and reported it absent. Verified
against the disc myself rather than adopted:
wep_05_ASMissile TriggerShotCount (defaulted) -> 4
wep_60_ASMissile TriggerShotCount (defaulted) -> 4
wep_02_Missile Power (defaulted) -> 100.0 [Shell]
wep_60_ASMissile Power (defaulted) -> 1000.0 [Shell]
wep_25_HBeam MaximumRange (defaulted) -> 4000.0 [Shell]
wep_11/28/36/70 LoadingCount (defaulted) -> 6/5/5/0
Two consequences recorded:
* The hypothesis "the title-code default for TriggerShotCount is 4" answers a
question that does not exist -- both weapons state 4 explicitly.
* The bracket "wep_60_ASMissile's defaulted Power sits in the C band
(150-500)" is refuted: it is 1000.0, outside the band. That also puts a
question over the bucket model or over that row's identification.
What is NOT withdrawn: the runtime capture itself. Every number it read
matches the disc. What goes is the reason for taking it -- the Gallery panel
was not recovering values the disc lacks, it was reading values our parser
was missing. Still useful as an oracle, redundant as a source.
Correction banner at the top; the original document is left intact beneath it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
