The declared sizes are honest (seek magic at data_at + declared_size, 7620/7620)
and VOICE_TCAF_608 was stereo decoded as mono, not truncated. The four offsets
are a segment-packing phase, not a per-directory header size. Neither was closed
by finding something new; both were closed by correcting a mistake of mine.
I read the seek chunk's packet count big-endian; it is little-endian at seek+12,
with size == 8 + 4*count. And a seek sits immediately AFTER its own data, so an
entry's first seek usually belongs to the PREVIOUS bank (implied start -25232
for D_452, -145988 for TCAF_608). I was comparing an entry's first seek against
its first data -- different waves by construction, which is why no reading lined
up.
With that fixed, the declared sizes are honest: every RIFF-bearing entry on the
disc has seek magic at exactly data_at + declared_size with count*2048 ==
declared. 7620/7620, zero failures.
VOICE_TCAF_608 is not truncated. Its Channels is 2 and I decoded it as mono;
read as stereo it gives 6520176 bytes = 33.96 s, agreeing with both length
signals in the bank (33.88 s from cumulative samples, 33.97 s from
PsuedoBytesPerSec). 170 of 8021 banks (2.12%) are stereo -- exactly the rate of
my 1-in-60 outlier.
This is the mono/stereo trap already documented on this very page, met from the
other direction: I had written 'at two channels every bank yields one frame' and
then spent several passes blaming missing data for a one-frame decode.
Code fix: to_xma_riffs built the leading segment with a hard-wired mono fmt. It
now reads Channels from the bank's first RIFF. 7 disc tests pass.
The '3 or 4 wrapped lines' sentence is in isl-message-dialogue-link.md itself,
not in cutscene-message-table.md as my correction claimed. Corrected the
sentence in place -- a page has four line SLOTS, of which an English page fills
1 to 4 -- and fixed the misattribution in the correction note rather than
quietly repointing it.
The five European packs have byte-for-byte identical MSG key sets: 18813 keys,
4091 ids, zero differences in any direction. Japanese has 14295 keys, which
looks like 4521 missing captions and is not: its id set matches to within 5, its
page count is the same (4765 vs 4758), and its mean lines per page is 1.52
against English's 1.85 with no 4-line pages at all. It carries the same dialogue
and wraps into fewer lines, which is what a language without inter-word spaces
does in a fixed-width box.
Also corrects my own wording in cutscene-message-table.md, which said a page
'holds 3 or 4 wrapped lines'. That counted key SLOTS, which MSG_DEMO records
allocate four at a time. Counting lines with text, an English page holds 1 to 4
(1574/2439/632/113). And it reconciles the 8800 figure used throughout: that is
distinct English keys with non-empty text, out of 18813 keys total.
Identified the structure: a little-endian size field after the tag, then
0x01000000, a varying word, a zero, then a strictly ascending table. Sizes 64 /
108 / 348 bytes for three sample banks.
Neither obvious reading gives the declared packet count. Entry count minus the
three header words is exactly right for VOICE_D_452 (13) and wrong for the other
two (24 vs 17, 84 vs 371) -- a one-of-three fit, recorded as FAILED rather than
as a rule with exceptions, because two measurements today already returned
plausible numbers for questions they could not answer.
The ascending values step by 1.5-2.2 million, far too large to be packet
indices, so they are on some other scale. Recorded what the next attempt should
know, including that auto/slb-loader chains seek packet counts successfully by
some field that is not either of the two tried here.
Generalising the TCAF_608 test across the 5296 over-declaring banks first looked
like a clean reversal: 59 of 60 gained audio (median 2.10x) from reading the full
declared size instead of stopping at the TOC window.
Then I tested whether the declared size is an honest boundary by reading TWICE
it. 33 of 40 yield >1.5x more audio, median ratio 1.64. The decoder keeps going
indefinitely, because XMA1 packets are self-contained -- the next bank's packets
decode perfectly well and are simply not this bank's. A byte count cannot tell
them apart.
So the 59/60 is withdrawn as evidence for anything. Same error as the unanchored
'data' search earlier on this page: a measurement returning a plausible number
for a question it cannot answer.
What survives: TCAF_608 is genuinely anomalous (86x more input, zero extra
output, where the typical bank yields without limit); the honest-size question
is unresolved and needs a boundary signal independent of the decoder, such as
the seek-chunk packet chain; and the only measured fact is that the declared
size exceeds the TOC window for 5296 of 7586 banks.
Read VOICE_TCAF_608's full declared 759808 bytes out of the flat stream ignoring
window boundaries. It decodes to the same 896 bytes (0.01 s) as stopping at the
window end -- 86x more input, zero extra samples. Control VOICE_D_452, whose
declared size fits, decodes to 2.18 s normally. No segment join is crossed, so
the packing phase is not a confound.
So 'the bytes are outside the window but still in the stream' does not hold for
the one bank the other branch named as not truncated. I still withdraw my word
'truncated' as an over-claim about the other 5295 banks: I measured that their
declared size exceeds their window, not what lies beyond it, and have now tested
exactly one. Recorded the three claims at their actual strengths.
Reproduced independently: X = (cumulative start of the .pNN segment) mod 2048.
The running sums of the five sound.pNN file sizes are 0, 1392, 1468, 1600, 1728
-- exactly the four values measured, plus zero. The XMA grid is 2048-aligned
inside each segment file, the segments are not multiples of 2048 long, so each
join shifts the phase.
Two of my explanations on this page were wrong even though the measurements
stand. 'Varies by language and subdirectory' was correlation, not cause --
directories cluster into segments. And there is no header at all: those bytes
are the PREVIOUS bank's audio, which is why they had no length field and looked
like data.
Also records a disagreement I could not resolve. auto/slb-loader withdraws my
'69.8% over-declare' finding as declared-sizes-are-exact, but VOICE_TCAF_608
declares 759808 bytes of data with 55296 bytes to the next TOC entry. Both
readings agree the declared size exceeds the window; mine said 'truncated',
which I withdraw as an over-claim, but 'exact' requires a wave to span ~14 named
entries. Recorded with the experiment that would decide it.
Closes the open question at the bottom of slb-data-offset.md.
X = (cumulative start of the .pNN segment holding the wave) mod 2048
The XMA packet grid is 2048-aligned inside each individual segment file,
but the .pak TOC addresses entries in the flat concatenation at offsets
that are themselves multiples of 2048. The segment files are not multiples
of 2048 long, so each join shifts the grid by size % 2048 — and the four
disc-wide values are exactly the running sums:
1392 = |p00| % 2048; 1468 = +76; 1600 = +132; 1728 = +128
Exact for 7620/7620 banks with a RIFF and 1163/1163 RIFF-less ones via
their seek chunk, 0 mismatches. Supersedes both heuristics (the 99.62 %
packet scan and the 99.97 % seek-residue rule) and dissolves the 28 ties.
The refutation test — an entry straddling a segment join must show two
phases in one file — passes on all 3 straddlers.
The leading bytes are the previous bank's audio, not a header: byte
diversity per offset is indistinguishable from a known packet (101.06 vs
101.90, no fixed field anywhere), the seek packet counts chain exactly
across consecutive entries, and the inter-entry bytes no TOC entry claims
are 1903/1928 non-zero.
Also recorded: the real bank header layout (id, block size 0x800, header
size in blocks, XMAWAVEFORMAT), and the loader search — a null result.
None of the four values exists as an immediate, a table or a float
anywhere in default.xex, which is what a pack-time artifact predicts.
Sound subsystem addresses mapped for the next pass.
Withdraws the 🟡 "most banks declare more data than they store" finding:
declared data sizes are exact (260/260), the bytes are just outside the
TOC window. Also withdraws my own "the header is unique, so nothing is
shared" inference — the windows tile, they do not overlap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Built-in 12 activate_unit returns 0 when the live object is NULL: it registers an
object that already exists and cannot create one. So a wave arrival is a craft
reaching a point on its route, not a new record, and a count of entity records
was never going to move.
Stage 02's UnitGroup totals Count = 116 across 111 squadrons, matching the live
probe's flat 116. The natural objection is that 116 might match by accident --
it does not: across all 28 stages the roster totals run 2..116, and 116 is the
MAXIMUM and unique to Stage 02, with nothing else within 5.
Three independent things agree: the handler cannot spawn, the roster totals 116,
and the live count sits at 116 while 22 of 113 routed squadrons have a route
starting after t=0.
Still n=1 for the per-member identity (only a Stage 02 save exists), and this
cannot settle timetable-vs-event -- that needs a position probe, not a count.
The pooled cross-tab gave p ~= 0.02 for 'no Yes-flagged squadron has a delayed
route', recorded at a cautious yellow. Breaking it down by stage removes it:
all 55 Yes routes live in 2 stages, and 45 are in S14, which has NO delayed
route at all -- those 45 cannot be evidence either way. Only S04 contains both
a Yes squadron and a delayed route.
Within S04 the exact test is P = C(47,5)/C(57,5) = 0.37, expected 0.88 observed
0. Unremarkable. The pooled p treated 1023 routes as independent draws when they
are two clusters.
The doc had already flagged this as the kind of near-fit that misled the corpus
before; that caution was right and did not go far enough.
Route keyframes are (time, quat x4, pos x3), so each consecutive pair gives an
implied speed. The flight-speed law was measured live against a real-time clock
(cruise ~420, max ~1530 world units/s), which makes it an independent yardstick
in known units.
Across all 28 stage tables, 331 Route_* records and 1104 keyframe pairs: under
SECONDS not one pair exceeds the 1530 ceiling (median 147); under frames at 30
or 60 fps, 89% and 94% of the game's routes would demand impossible speeds.
Two corrections recorded. A first pass silently skipped non-numeric fields while
flattening records, shifting every later value -- it reported a median implied
speed of 65534, suspiciously 2^16, which is what exposed it. And the remaining
outliers were Frame_S<NN>_Asteroid records, not routes; the final measurement
filters by record kind rather than dropping whole stages.
Probed the bytes before the data offset at all four values. Not a length field
(no word in the first 64 bytes matches the offset, the RIFF position or the entry
size, either endianness). Not a seek table or any ascending index (~50% of
consecutive BE word pairs are non-decreasing, i.e. random; all words distinct,
none zero). Not zero padding in general (5876 of 7358 have content). Not audio
being discarded (offset 0 wins 6 of 7358).
High-entropy content whose size is constant per language and subdirectory, with
no self-describing length -- which points at a loader that knows the size a
priori. Recorded the next step as static PE work rather than another archive
pass; the byte-level evidence here is about exhausted.
The ties needed a different signal, not a longer scan. Banks carry one: a seek
chunk sitting on a packet boundary, so seek_pos % 2048 IS the data offset. On
the 6033 labelled banks with a seek before their first RIFF, 6031 agree
(99.97%) -- better than the packet scan and structural rather than statistical,
so scan_data_offset now tries it first.
On the scan's 28 ties it resolves 26 correctly and 0 wrongly (2 have no usable
seek). Combined rule scores 7354/7358 = 99.95%, up from 99.62%. 762 of the 1495
RIFF-less banks carry a seek, so the signal exists where it is needed.
Also ruled out, since a wrong offset was this page's whole subject: the header
is not audio being discarded. Adding 0 to the candidate set, it wins 6 of 7358.
7 disc tests pass.
Having found that the cue-less cutscene pages bind their audio through the movie
path, I tested the same explanation on the twelve script lines with no cue and
no bank. It fails: matching their caption text against every line in the
cutscene message table gives 0 of 14 hits, so nothing carries their audio by
another route.
They are ordinary story dialogue ('Katana, wait for me in hell'), which makes a
deliberate subtitle-only design unlikely -- but that is a judgement, not a
measurement, and why they are unvoiced is still open. Also noted: D_609 and
D_610 are equally unvoiced and simply are not among the twelve, because the
twelve counts script-called names and nothing calls them.
16 of the 17 pages with an empty slot 4 belong to MSG_DEMO_600..604, whose
caption text matches, word for word, the five resupply lines documented from the
movie side as VOICE_D_450..454. Slot 4 is empty on purpose: that audio binds
through the movie path, not through a DEMO_nnn cue.
That also disposes of the '5 ids with more than one record' item -- each appears
once per stage that uses it, and the repeat counts equal the movie-slot counts
recorded independently for the matching banks: 3/3/4/4/2 both ways, 5 of 5. The
11 extra records over 138 ids are these repeats.
Worth more than closing two items: the cutscene message table and the movie
subtitle chain were decoded from different containers by different routes and
agree on the same five lines with the same multiplicities.
MSG_DEMO_500 remains: one page, KATANA, 2s, no text and no cue.
The labelled set has a RIFF and the scan is unbounded, so it reads past the RIFF
there -- the headline number could have been borrowing discrimination that a
RIFF-less bank cannot offer. Confining the scan to the leading region gives
69.98%, which looks like exactly that problem.
It is not. Split by how much leading audio there is: on the 989 banks with >=24
packets of it, the scan is 100% correct with ZERO ties, whether or not the RIFF
is in range. The 69.98% is an artifact of short leading regions, where two or
three packets are not enough to separate candidates.
A RIFF-less bank is a whole pak entry, so 24 packets are always available. The
99.62% is conservative for the population the scan serves, not optimistic.
Of the 7586 banks with a RIFF and a data chunk after it, 5296 declare a data
size larger than the pak entry holds; 2290 declare less (the ordinary
multi-sub-wave case); NONE declare exactly what they hold. This contradicts the
decoder comment claiming the declared size 'is honest per sub-wave'. The code
clamps, so it is a documentation defect, not a crash.
It also closes the loose end from the offset work: eng\Voice\VOICE_TCAF_608,
the single bank where neither offset decoded, is 99% short -- there is nothing
there to decode.
Method note recorded: my first pass searched for 'data' from offset 0, which can
match by chance inside the leading audio region. Anchoring the search after the
first RIFF moved the count 5038 -> 5296.
Separately, the 55 'early RIFF' English banks are not an anomaly: all 55 sit at
exactly 1392 behind a zero-filled header -- a zero-length leading region, which
both the old and new code already handle correctly.
static.slb (8970240 bytes, the shared SE bank) and Pj_Silph.xgs (533 bytes, XACT
global settings) both hash into the TOC. Their names come from the BANK_SE and
SETTINGS records of the very IDXD object this page documents -- I had printed
them at the top of the write-up and then reported the entries as unidentified.
9519 of 9519 now: 5100 jpn + 4382 eng + 35 root + these 2.
1495 banks carry no RIFF and take a separate path that hardcoded both the
offset and stereo. Across a random 48-bank sample there was NOT ONE where the
old stereo-at-1392 pair beat the best mono offset; median gain 184x, individual
banks going from 0-4816 decoded bytes to 180000-380000. Stereo shows the same
stop-after-one-frame signature already recorded for the leading segment.
With no RIFF the offset cannot be derived, so scan_data_offset picks among the
four disc offsets by XMA1 packet-header plausibility. Validated on the LABELLED
set -- all 7358 banks that do have a RIFF, where the answer is forced: 7330
correct (99.62%), and all 28 misses are ties on the top score, never a wrong
unique winner. Ties fall back to 1392.
The winning offsets also reproduce, by directory, the distribution measured
independently from the RIFF-bearing banks. jpn\etc splits 1468/1600, so path
alone is not sufficient -- which is why this is a scan and not a lookup table.
7 disc tests pass (build-reborn test -p sylpheed-formats --test
slb_leading_segment_disc, SYLPHEED_DISC wired up).
HEADERLESS_DATA_OFFSET is the value the offset takes in <lang>\etc\, not a
property of the format. The leading stream is a whole number of 2048-byte XMA1
packets ending at the first RIFF, so its start is first_riff % XMA1_PACKET.
Disc-wide that takes four values -- 1392, 1468, 1600, 1728 -- varying by
language and subdirectory.
Verified by decoding, not by arithmetic: on a random 140-bank sample with a
non-empty leading region, the derived offset yields more audio in 85, identical
in 54 (the eng\etc controls, where it must and does reproduce the old
behaviour) and less in 1. Median gain among the improved is 70x --
eng\Voice\VOICE_TCAF_592 goes 1506 -> 97152 bytes, jpn 2910 -> 127178.
This withdraws my own claim from earlier today that the Japanese banks were a
different undecoded layout. They are the same format with a different offset;
I had treated a constant derived from one subdirectory as a property of the
format. The same error was hiding the identical defect in 1873 eng\Voice banks.
The three unnamed built-ins are one family, and the chain from the dispatch
table to the field write is now followed for all of them:
builtin -> ScriptPhase vtable slot -> interpreter command word 0xAB<op>BA
-> command-table thunk -> opcode handler -> unit message 0xED08nnDE
-> the GROUP pump sub_8232C4C0, which rebroadcasts to each child as
0xED09nnDE
-> the entity base handler sub_82398CC0, which writes the field.
26 -> [unit+532] = min(max(def.HP * pct, 0), def.HP) = set_unit_hp_pct
29 -> [unit+676] = pct, a multiplier on damage TAKEN = set_unit_damage_taken_pct
28 -> [unit+672] = pct, default 1.0 = 🟡 damage DEALT
101 -> the same message as 29 with a hard-wired 0.0, broadcast to every unit
= all_units_invulnerable
`damage_unit` is WITHDRAWN for 26. The handler sets an absolute value rather
than subtracting one, and 100 heals to full -- which no damage primitive does.
It is pinned three ways: [unit+496] is the unit definition (the constructor
sub_82393868 fills it from the same std::map::find built-in 15 uses), [def+84]
is HP in unit_definition_layout.txt and the constructor seeds [unit+532] from
it, and crossing zero loads [def+584] = Delay and raises a flag, i.e. the
destruction sequence. So 0 destroys, with the datasheet's own death delay.
29 is the strongest of the three: [unit+676] has three independent readers
(sub_8237B020, sub_823800A8, sub_82398CC0) and every one multiplies a damage
amount immediately before it reduces [unit+532].
28 is deliberately left 🟡. The write and the 1.0 default are certain, but the
field has exactly ONE reader in the whole image -- the craft update's projectile
spawn, where it ends up as a multiplicative term in the damage message. That is
the mirror of 29 and it is tidy, which is exactly the shape that produced the
wrong names this file has already had to withdraw. What is not established is
that it reaches every weapon; the sibling damage sender sub_82388FF8 has no
+672 term at all.
Three usage tests, all measured over the 28 stages:
* operand ceilings -- 26 is 97/97 inside [0,100] and 29 is 164/164, while 28
(identical signature, identical x0.01 conversion) reaches 2000;
* the craft cross-tab -- 26 splits cleanly into disposable props at 0,
warships at 30-80 and the tutorial player craft at 100; 29 lands on the
player, the tutorial boxes and the escorted TCAF hulls; 28 orders
boss > ace > elite > line > prop;
* the setup idiom -- activate_unit, then 15/29/28 as a speed/toughness/
firepower trio, with 26 added wherever a unit must arrive pre-damaged.
Refutations attempted are recorded, including the two that turned into
confirmations (Stage 28 makes each tutorial box invulnerable with 29 and then
removes it with 26) and the offset-search trap that produced three false
readers, because projectiles have their own fields at 672 and 676.
Also corrected: the state guards. 26 rejects states 3 and 4; 28 and 29 reject
1, 3 and 4. This file said otherwise for both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
804 rows across all 28 stages, each joined to its squadron's craft through
`UnitGroup_S<NN>.tbl` -- 671/671 of the three unit-taking built-ins resolve,
none unknown. Regenerable from the committed tool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Independently reproduced across all 33 paks: 7750 IDXD objects, 1485577 unnamed
field entries, 7094 distinct never-named keys splitting cleanly into 7052 in an
ordinal band (<=0x2198, 94.6% equal to their own field index) and 42 hash-shaped
(>=0x2677C), with ZERO keys in the gap between. The 42 carry exactly 504
entries -- six language copies of one object times two records.
So the preimage target was 42, not 504, and my earlier wording invited the
misreading. Cross-referenced to the idxd-unnamed-keys write-up, which shows the
42 belong to <lang>\script\ID.tbl and cannot be recovered from a 24-bit hash.
All 42 live in six byte-identical copies of <lang>\script\ID.tbl inside
GP_READY_ROOM.pak, the artefact of the pass the executable announces as
"PrepareScript - isl script prescanning start". Two records, FILE and
OFFSET, carry the same 42 keys: symbol hash -> (.isb file, offset).
Confirmed independently, not inferred: 41 of the 42 keys occur verbatim as
little-endian words inside the .isb bytecode, and where they occur is a
coherent call graph -- main.isb calls the 30 stage/challenge/tutorial
symbols, those call the five helpers in function.isb, the tutorials call the
six in function_tutorial.isb, and main.isb's own symbol is called by nothing.
Ordinals do not appear as call targets in code; hashes do.
30 of the 42 names have their last characters PINNED by the hash's own
algebra. Bumping the character k places from the end moves the low 24 bits
by 256^k mod M and the top byte by 1, and the measured deltas are exactly
+0x01000001 across stage01..09, stage10..16 and challenge01..06, and
+0x01010000 across tutorial0101..0601. So those names end in the same digits
as their .isb filename. The prefixes are not recovered.
NULL RESULT on the names, and quantified rather than asserted. Seven attacks
(1.2M disc + executable strings, printf substitution, a hand guess list,
two-token composition over a mined vocabulary at S = 3.5e8, exhaustive
meet-in-the-middle preimage search) produced zero hits above the noise floor.
The search is not broken -- fed tag_hash("Stage01") it returns "Stage01" plus
three collisions, as 63^6 / 2^32 predicts. It returns nothing for the real
targets at <=6 characters, so the prefixes are longer, and at 7 characters
one target already has 1 176 measured preimages. A 24-bit modulus cannot
name an 8+ character identifier uniquely; only a corpus holding the actual
string can, and the disc does not hold it.
Refutations kept rather than deleted: the filename-stem hypothesis (direct),
the shared-prefix-plus-stem hypothesis (algebraic -- A's byte sum is pinned
twice and the two values disagree, 9 pairs, 9 contradictions), names inside
the .isb payloads (there are no ASCII names there at all), the hash constants
sitting in the executable (absent in both endiannesses), and IXUD holding
more of the same (8 unnamed tags, all ordinals 0..7).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Walks every IDXD object on the disc (7 750 objects, 190 782 records,
2 757 039 field entries, zero parse failures) and separates the field keys
that carry no name in the pool from those that do.
The headline correction: the "504 fields disc-wide that are hash-keyed with
no name" is a count of field ENTRIES. It is 42 distinct keys x 6 language
copies of one object x 2 records (FILE and OFFSET) = 504. The brute-force
target was never 504 names.
7 094 distinct keys are never named anywhere. 7 052 of them are not hashes
at all: they are author-assigned element ids, equal to the field's own index
in 1 404 924 of 1 485 577 cases and hand-numbered with gaps elsewhere.
Refuted directly rather than assumed -- tag_hash("BGM_001") is 0xC662435B
while the key of the field valued "BGM_001.slb" is 0x000003E9.
The split is measured, not stipulated: every key that DOES carry a name and
has a zero checksum byte sits at 0x0002677C or above, the ordinal band tops
out at 0x2198, and not one key falls in the gap between.
tag_hash reproduces 1 271 462 / 1 271 462 named field keys disc-wide
(`idxd_unnamed_keys.py selftest`), which is the gate everything else rests on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Every FILES path hashes straight into the sound.pak TOC (5135/5135, path as
stored). The archive is then almost fully accounted for: 5100 jpn + 4382 eng +
35 BGM/JNGL = 9517 of 9519. Only eng and jpn resolve -- deu/fra/ita/esp give
zero, which is the archive-side measurement behind 'two voice sets shipped'.
718 Japanese banks have no English counterpart.
More importantly this bounds an existing result. The leading-region rule from
voice-bank-leading-region.md reproduces its five documented banks EXACTLY under
eng\etc\ (16384/2048/14336/45056/59392), so the pipeline is sound -- but it
holds for only 1571 of 4382 English banks and for ZERO of 5100 Japanese ones.
It is a property of some English banks, not of .slb in general, and the Japanese
layout is undecoded.
I nearly misread this as a broken extractor, because FILES lists jpn paths and
those gave non-multiples of 2048. Reproducing the recorded figures exactly under
eng is what distinguished a wrong assumption from a wrong reader.
My sound-cue write-up said the table 'does not explain' the VOICE_D_452
rejection and implied it was still open. It is not: voice-bank-leading-region.md
settled it earlier -- the bank holds one generic line reused across four stages,
so the binding is correct and the expectation was wrong. The narrower true
statement, which is what the cue table actually adds, is that the reuse is a
movie-slot -> bank relation and not a cue-level one.
dat/tables.pak holds a 5798-entry SOUNDS record (cue name -> sound id) and a
5135-entry FILES record (.slb bank paths). Cue names are the join key, so a
script message id now resolves all the way to the bank that voices it:
MSG_VOICE_D_257 -> VOICE_D_257 -> 6945 -> jpn\etc\VOICE_D_257.slb.
The prefix rule is MSG_ -> VOICE_, not strip-MSG_. My first rule was the
latter; it left 88 names unresolved and I was about to write those families up
as text-only announcements, until VOICE_TCAF_592.slb turned up in FILES and
refuted it. Corrected rule resolves 1326 of 1338, and SOUNDS and FILES agree on
exactly the same 12 absentees.
Separately, MSG_DEMO_* is driven by its own IDXD tables in the language packs,
which carry speaker, portrait, on-screen seconds and audio cue per page. Field
count is 9*PageCount+2 for all 7 distinct PageCounts; 1252/1252 caption-key
slots match <ID>_<page>_<line>; the 78 multi-page records equal the 78 counted
independently from the caption side; 138 ids close exactly against the caption
table both ways.
Does not settle the known VOICE_D_452 wrong-recording case -- every cue id is
distinct, so bank sharing is not happening at this layer.
Measured the <id>_<page>_<line> key structure: a page is one subtitle box of
3 or 4 wrapped lines, and successive pages are successive utterances by
possibly different speakers. 452 of 4091 ids span more than one page, up to 8.
That refutes the isl_dialogue.py committed two commits ago, which read only
page 000 -- 356 of the 1338 script message names are multi-page, so a quarter
of its output was truncated to the opening utterance. Tool now walks pages
until one is empty; Stage 02 sample regenerated (43 of 213 calls multi-page).
The 2683/2683 resolution figure is unaffected: it counted ids that have text,
and every id does have a page 000. What was wrong was the rendered text.
Built-in 64's slot-0 operand is a symbol-table-1 type-6 message id, and every
one of them now has caption text: 2683 of 2683 call sites across the 28 stage
scripts, 1338 distinct names, no residue of any kind.
This only became reachable once build_caption_text was switched to the IXUD
field table (537 -> 8800 lines); before that most of these names had nothing
to resolve to.
Adds isl_dialogue.py plus a committed Stage 02 sample. Does not settle which
recording plays for a given line, multi-page captions, or the other five
languages.
The row still described ixud.rs as a cue reader and said nothing about the
record table or the caption families. Rewritten with the measured numbers:
* the IXUD record/field table is decoded and wired in -- IxudObject mirrors
IdxdObject, uniform 16-byte records, 12-byte fields, every offset in CHARS,
and the word at 0x08 is record 0's hash rather than a schema id. Verified
1104/1104 objects, 1476/1476 records, 628165/628165 named fields.
* caption text goes 537 -> 8800 lines, which is 8800 of 8800 distinct keys,
in two steps: generalising the key parser to all eight families took it to
8074, and switching from token adjacency to record fields finished it.
The row also carries my correction rather than quietly dropping it: the
earlier "1.3% of the game's text" counted occurrences across blocks, and the
honest denominator is 8800 distinct keys, so the starting point was 6.1%.
And it keeps the DEMO control, which is the part a reader should remember:
token adjacency finds 537 lines in that family, fields find 541. The old
reader was dropping lines in the one family it was written for.
Status moved from 🟡/✅ to ✅.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
ixud.rs now has an IdxdObject-shaped reader, IxudObject, and build_caption_text
reads captions as FIELDS instead of pairing them with whatever token follows in
the pool.
build_demo_text token adjacency 134 ids 537 lines
build_caption_text token adjacency 3721 8074
build_caption_text record fields 4085 8800 = all of them
Verified over the whole disc by tests/ixud_records_disc.rs: 1104/1104 objects
parse, 1476/1476 records and 628165/628165 named fields reproduce their
ixud_hash, 48 positional, zero failures. The header word at 0x08 is record 0's
hash, asserted per object -- there is no schema field, exactly as for IDXD. The
module doc described a 12-byte record directory and a "schema/type hash"; both
were wrong and are corrected.
I also have to correct my own number from the previous commit. "1.3% of the
game's text" counted OCCURRENCES: each family lives in 24-45 IXUD blocks and
the same key repeats across them. Distinct text-bearing MSG_* keys number 8800,
not 44579, and every one has the <id>_<page>_<line> shape. So the real coverage
was 537/8800 = 6.1%, and I overstated the gap about fivefold. Direction right,
magnitude wrong.
The DEMO control is the sharpest evidence for the change: token adjacency finds
537 lines there, the field reader 541. It was dropping lines even in the one
family it was written for -- which is why the test now asserts "must not lose
lines" rather than "must be identical".
Same lesson twice in one session: pool adjacency is a consequence of how
records are written, not a rule of the format.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
build_caption_text generalises the key parser from MSG_DEMO_* to all eight
families. The shapes are uniform and each family is 100% consistent with its
own: seven use MSG_<FAM>_<id>_<page>_<line>, and VOICE alone inserts a family
letter before the id.
ids lines
build_demo_text 134 537
build_caption_text 3721 8074
The DEMO family comes out identical through both readers -- 537 lines either
way -- which is the control that generalising changed nothing that already
worked. Pinned by tests/caption_families_disc.rs, along with VOICE ids keeping
their family letter.
But this does NOT close the gap, and the write-up says so: 8074 against the
44579 text-bearing fields the record-level scan counts is about 18%.
The reason is the same lesson this session already learned once.
build_caption_text pairs a value with the key that happens to follow it in the
raw UTF-16 token stream -- the adjacency heuristic that was wrong for IDXD and
is wrong here for the same reason. ixud.rs has no record/field reader at all.
The IXUD record table IS decoded and verified disc-wide (1104/1104 objects,
628165/628165 fields reproducing their key) and was simply never wired into
the crate.
Next step recorded: give ixud.rs an IdxdObject-shaped reader and read captions
as fields rather than adjacent tokens. The decode exists; only the plumbing is
missing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
movie_subtitle handles MSG_DEMO_*, the cutscene captions. Counting every IXUD
block in GP_MAIN_GAME_E.pak, that is the SMALLEST of eight families:
MSG_ADAN 23236 keys 9801 with text ADAN combat chatter
MSG_RHIN 21196 8509 Rhino squadron
MSG_TCAF 17148 6728 TCAF
MSG_VOICE 13060 6776 in-mission scripted dialogue
MSG_BIRD 14036 5834 Bird squadron
MSG_ADPL 12640 4127 ADAN pilots
MSG_ACRO 4804 2244 Acropolis
MSG_DEMO 1252 560 cutscene captions <- the only one read
total 107372 44579
560 of 44579 text-bearing keys = 1.3%. I report the text-bearing column rather
than raw keys because only 41.5% of keys carry text -- the rest are the empty
line slots this container pads with, and counting those would flatter the
denominator.
MSG_VOICE_* is the family the message tables reference -- the dialogue whose
voice bindings this file now analyses in detail -- and nothing in crates/
parses it. So the corpus knows which bank plays for a line it cannot read.
First step recorded: build_demo_text already pairs a text value with the
MSG_DEMO_<demo>_<page>_<line> key that follows it, and the other seven
families use the same <id>_<page>_<line> shape, so generalising the key parser
is most of the work. With a warning attached: do NOT assume the id spaces
relate, since the voice-bank id and the caption id within one message page are
different numbers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
The 60 tokens absent from BOTH voice languages, itemised:
45 VOICE_E_ family -- 44 numeric [0..43] plus the lettered VOICE_E_012B
13 VOICE_C_ at 421, 423-426, 430, 432, 447-450, 470, 471 -- INSIDE the
listed range [0..489], so interior gaps rather than a truncated tail
2 VOICE_D_182 and _183, adjacent
These lines were written and captioned; only the audio is missing. Their
caption keys resolve to real text in the IXUD blocks:
MSG_VOICE_C_355_000_00 "What are you doing? Quit wasting..."
MSG_VOICE_C_367_000_00 "The final defense weapon is..."
MSG_VOICE_C_347_000_00 (Japanese)
MSG_VOICE_D_152_000_00 (Japanese)
MSG_VOICE_E_044_000_00 (Japanese)
Three of the five sampled are still Japanese INSIDE the English pak --
captioned but never translated, matching the untranslated entries already
noted for the localised-text container.
Separately, a trap worth its own heading: within one message page the voice
bank token and the caption keys use DIFFERENT numbering.
Message_106 voice VOICE_C_468 lines MSG_VOICE_C_385_000_00..02
Message_129 voice VOICE_D_182 lines MSG_VOICE_D_152_000_00..02
Message_044 voice VOICE_E_012B ID MSG_VOICE_E_044
Same family letter, different index space. Deriving one id from the other
will silently mis-pair audio with text -- which is the same class of mistake
as the demo-id voice binding this corpus already had to reject in-game.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Last iteration I read the scattered missing runs as cut dialogue. Checking the
other languages refutes that for most of them.
Only two voice languages exist: tables.pak holds eng\sounds.tbl (4418 banks)
and jpn\sounds.tbl (5136), and NO fra/deu/esp/ita. The other four
localisations are subtitle-only, which is itself a fact the corpus did not
record.
The English set is a strict subset of the Japanese one:
eng jpn
listed 4418 5136
resolved 4175 4383
missing 268 60
lacked by
the other 0 718
Every id missing in Japanese is also missing in English, and NONE is missing
in Japanese but present in English. So the 268 split cleanly:
208 exist in Japanese, not in English -- recorded in the original and
never dubbed. A localisation gap, not cut content.
60 absent from both -- genuinely cut, including all 44 VOICE_E_ (listed
0 times in either language) plus 16 others.
Per family eng/jpn: A 683/771, B 383/423, C 366/474, D 387/546. The 718
Japanese-only banks include whole families English never had -- VOICE_ADAN_
(110), VOICE_ZZZZ_ (104), VOICE_TCAF_ (41), VOICE_RHIN_ (27).
So VOICE_E_ survives as the one genuinely-cut family, and my "short runs of
cut dialogue" reading was wrong: those lines are on the disc, in Japanese. The
previous text is kept above the correction.
This is why the "measured on English only" limit was worth writing down rather
than quietly generalising -- the second language changed the conclusion.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Following up the incidental gap from the DEMO_* work. Resolving every VOICE_*
token the way the engine does -- token, then eng\sounds.tbl for its directory,
then sound.pak -- gives 4175 resolved and 268 unresolved of 4443 distinct.
The gap is in the manifest, not the archive: the unresolved tokens are not
listed in sounds.tbl at all, and of the 4175 that ARE listed, every one is
present in the pak. Nothing is listed-then-missing.
By family, script-used ids against sounds.tbl-listed ids:
VOICE_A_ 696 [0..772] listed 683 missing 14
VOICE_B_ 391 [0..422] listed 383 missing 8
VOICE_C_ 429 [0..489] listed 366 missing 63
VOICE_D_ 508 [0..598] listed 387 missing 126
VOICE_E_ 44 [0..43] listed 0 missing 44 <- the whole family
Two different shapes, and the distinction matters. A/B/C/D lose short runs
scattered through an otherwise dense range -- 11,12,13 / 56,57 / 181,182,183
/ 389-392 -- which is what cut dialogue looks like. VOICE_E_ is absent
entirely: 44 ids used, none listed, no VOICE_E_* anywhere in the manifest. A
whole speaker or category, cut.
For the port: a missing voice bank is a RETAIL condition, not a decode bug. A
reimplementation must tolerate an unresolvable VOICE_* reference, the same way
it must tolerate SUBTITLE_S12B.tbl resolving in no language.
Limit stated: measured on GP_MAIN_GAME_E.pak and eng\sounds.tbl only; whether
the other five languages drop the same ids is untested.
An earlier probe of mine checked only three sound.pak directories and would
have mis-attributed this; sounds.tbl lists five, including Briefing. Resolving
through the manifest rather than guessing directories is what makes the 268
trustworthy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
An open question from the game_data migration: 132 dialogue lines carry an
unprefixed DEMO_* token where a VOICE_* one is expected, and whether DEMO_*
names a bank was untested. It does not.
VOICE_* 4443 distinct, 4175 resolve in sound.pak
(2360 eng\Voice\, 1815 eng\etc\)
DEMO_* 296 distinct, 0 resolve
Zero resolve under any of the three directories, and looking them up through
the convention -- VOICE_DEMO_nnn -- resolves 0 of 296 as well. So the slot is
not "the voice bank"; it can hold either a bank reference or a demo one.
The tokens run DEMO_000 to DEMO_310, 287 distinct in [0, 310] -- a dense
sequential id space, which is the shape of the demo id this file already uses
in MSG_DEMO_<demo>_<page>_<line> caption keys. I did NOT confirm the overlap
and say so: those keys live in IXUD wide-string blocks while my scan walked
IDXD objects only, so it found 0 of them. That needs an IXUD-aware sweep.
Incidental and left open: 268 of the 4443 VOICE_* tokens do not resolve
either. That is a separate gap and the DEMO_* finding does not explain it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
The last file still resting on the "defaulted on disc" premise. Re-checked
through the record table over every unit-like object on the disc -- 114 with
a Generic.Type of Craft or Vessel:
present in ALL 114 objects 50
present in some 49
present in NONE 2 <- FiringRange, SELength
So only FiringRange and SELength are genuinely absent, out of 101 listed.
FiringRange was already recorded correctly elsewhere (runtime 0 is the
accessor's miss value); SELength joins it.
One pattern worth keeping rather than dismissing: 15 of the fields sit in
exactly 5 objects each -- AB_* (afterburner), PowerCut*, AA_/AV_AxisMode_*.
That is the Delta Saber family, so those fields are player-craft-only rather
than defaulted, which is a different fact than either the file or my
correction to it would suggest on its own.
The runtime values themselves stay -- they are real measurements. What is
withdrawn is the claim that the disc does not carry the fields.
That completes the sweep: weapon-datasheet-runtime.md, unit-struct-runtime.md,
live-unit-definitions.md, INDEX.md and now this capture all carried the same
wrong premise from the same blind reader.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
The two headline rows for weapons and units were the last place still telling
a reader that these values are not on the disc. Both corrected in place, with
numbers I measured rather than adopted.
Weapons: re-reading captures/weapon-runtime-fields.csv through the record
table, 1514 of the 4393 rows labelled defaulted-on-disc DO have a value on
disc; 2879 are genuinely absent. I state that as an upper bound -- my matcher
counts a field when it appears in ANY record of the object, and a per-record
count gives ~1448. Either way the headline "4393 values the disc does not
carry" is wrong by about a third.
The spot checks are exact rather than aggregate: wep_05/wep_60
TriggerShotCount 4, wep_02 Power 100.0, wep_60 Power 1000.0 (refuting the
recorded "C band 150-500" bracket), wep_25 MaximumRange 4000.0,
wep_11/28/36/70 LoadingCount 6/5/5/0.
Units: the ~30-field player-craft table is on disc at exactly the values the
runtime "recovered", spread across the Generic / Shield / Mass / SE records
-- which is why a reader that could not name a record saw them as absent. And
"18 of 23 vessel records are missing at least one of Size_X/Y/Z/HP" is false:
0 of 114 objects with a Generic.Type (43 Craft + 71 Vessel) miss any of them.
Status markers moved from ✅ and ✅/🟡 to ✅/❌ so the rows no longer read as
settled-and-correct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
unit-struct-runtime.md lists ~30 player-craft fields as defaulted on disc with
values "recovered from the running game". All of them are on the disc at
exactly those values. Verified myself for UN_f001_TCAF_DeltaSaber_T:
Size_Radius 10.0, FCSRange 500000.0, ResistanceToOptics/HQRatio/
ShieldRatio/ThrusterRatio/ResistanceToShell/ResistanceToExplosion 1.0,
MassScore 0 [Generic]
ChargeSpeed 25.0, ChargeDelay_Break 10.0 [Shield]
DryMass 100.0 [Mass]
LowerHPThresholdRatio 0.3 [SE]
The record column is the explanation: the values are spread across Generic,
Shield, Mass and SE. That is exactly why the old flat reader missed them --
it could not say which record it meant, so a per-record field read as absent.
Also wrong, and measured: "18 of the 23 vessel records are missing at least
one of Size_X/Y/Z/HP, nearly always Size_Y". Over every object carrying a
Generic.Type -- 43 Craft plus 71 Vessel, 114 objects -- ZERO are missing any
of the four.
live-unit-definitions.md's "Route B: 13 defaulted fields read out of the
running game" goes with it: all 13 Size_Y values are on disc, each exactly
equal to the value the runtime run "recovered". Nothing was recovered that
the disc did not hold; the run confirms only that the parser reads the disc.
As with the weapon sheet, the runtime captures stand as measurements. What is
withdrawn is the premise that they were reaching values the disc lacks.
Correction banners at the top of both; original text intact beneath.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Every field weapon-datasheet-runtime.md calls defaulted is on the disc. The
record table reads them directly; the old string-pool reader could not see a
value whose string is shared or reordered and reported it absent. Verified
against the disc myself rather than adopted:
wep_05_ASMissile TriggerShotCount (defaulted) -> 4
wep_60_ASMissile TriggerShotCount (defaulted) -> 4
wep_02_Missile Power (defaulted) -> 100.0 [Shell]
wep_60_ASMissile Power (defaulted) -> 1000.0 [Shell]
wep_25_HBeam MaximumRange (defaulted) -> 4000.0 [Shell]
wep_11/28/36/70 LoadingCount (defaulted) -> 6/5/5/0
Two consequences recorded:
* The hypothesis "the title-code default for TriggerShotCount is 4" answers a
question that does not exist -- both weapons state 4 explicitly.
* The bracket "wep_60_ASMissile's defaulted Power sits in the C band
(150-500)" is refuted: it is 1000.0, outside the band. That also puts a
question over the bucket model or over that row's identification.
What is NOT withdrawn: the runtime capture itself. Every number it read
matches the disc. What goes is the reason for taking it -- the Gallery panel
was not recovering values the disc lacks, it was reading values our parser
was missing. Still useful as an oracle, redundant as a source.
Correction banner at the top; the original document is left intact beneath it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
The refutation used the wrong hash function. A record key is tag_hash of the
record's own name, not the pak-TOC name_hash which lowercases and uses a
different modulus:
tag_hash("Formation_4_Bird") = 0x22a5eeed <- exactly the record key
name_hash("Formation_4_Bird") = 0x6286edad <- the wrong hash, hence
the recorded "0 of 16"
Formation_4_Bird IS a record name in FormationSet_S02.tbl and every record in
that table keys by tag_hash of its name. Across all 28 UnitGroup_S<NN>.tbl the
same holds for field keys: 6960 of 6960 are tag_hash(field name), 0 are
name_hash. So FormationID resolves by a direct hash lookup and the roster
indirection is NOT required for it.
This is exactly the trap idxd-container.md warns about, and it cost this file
a spurious refutation. The original text is kept below the correction.
The squadron half of the claim still stands and is marked so: a squadron
record's key is not tag_hash of the squadron id either -- tag_hash("TCN001")
is 0x760cdeaa against keys running 0x659aff47 -- because the record's NAME is
not the squadron id. That roster indirection is real.
Also closes the file's standing "the key derivation stays open, a second hash
function is unidentified": that function is tag_hash, recovered since.
Found while checking a subagent's claim rather than taking it -- it reported
the hash correction, and verifying it turned up that the FormationID half was
a stronger result than reported.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
