diff --git a/docs/re/structures/idxd-unnamed-keys.md b/docs/re/structures/idxd-unnamed-keys.md new file mode 100644 index 00000000..e4dceee8 --- /dev/null +++ b/docs/re/structures/idxd-unnamed-keys.md @@ -0,0 +1,253 @@ +# IDXD field keys that carry no name + +- **Confidence:** ✅ the census and the identification of the hash-keyed set · + ✅ the last characters of 30 of the 42 names · ❌ **no name was recovered** +- **Tool:** `tools/re-capture/idxd_unnamed_keys.py` (`census` / `idtbl` / `crack` / `selftest`) +- **Data:** [`docs/re/data/idxd-unnamed-field-keys.txt`](../data/idxd-unnamed-field-keys.txt) +- **Seen in:** every `dat/*.pak` + `hidden/*.pak` on the disc; the interesting 42 + are in `dat/GP_READY_ROOM.pak`, file `\script\ID.tbl` +- **Depends on:** the IDXD record/field table and `tag_hash`, which live on branch + `auto/re-idxd-container` — **not on `main`**, so this note transcribes what it + needs and the tool carries its own copy of `tag_hash` rather than importing one. + +An IDXD field entry is `(key, name_off, value_off)`. Usually `name_off` points at +the field's own name in the string pool and `key == tag_hash(name)`. When +`name_off` is `0xFFFFFFFF` the pool holds no name and the reader has to know what +`key` means from somewhere else. This note answers: how many such keys are there, +what are they, and can the names be inverted out of the hash? + +**Short answer: 7 094 distinct keys, of which 7 052 are not hashes at all and 42 +are. None of the 42 names was recovered — but they are now identified, their call +graph is reconstructed, and 30 of them have their last two-to-four characters +pinned algebraically.** + +## ✅ The census + +Measured by walking all 33 `dat/*.pak` plus `hidden/DefTables.pak` and +`hidden/MiscBin.pak` (`.pNN` segments joined, `Z1` payloads inflated): + +| | | +|---|---| +| IDXD objects / records / field entries | 7 750 / 190 782 / **2 757 039** | +| named field entries | 1 271 462 | +| unnamed field entries (`name_off == 0xFFFFFFFF`) | 1 485 577 | +| distinct keys named *somewhere* on disc | 12 174 | +| **distinct keys never named anywhere** | **7 094** | +| … below `0x00010000` | 7 052 | +| … in `[0x10000, 0x01000000)` | **0** | +| … at or above `0x01000000` | **42** | + +Parse failures: 0. `tag_hash` reproduces the key of **1 271 462 / 1 271 462** +named field entries disc-wide (`idxd_unnamed_keys.py selftest`), which is the +validation gate for everything below — far past the 200-pair bar this +investigation was asked to clear. + +### ✅ "504 hash-keyed fields disc-wide" is 504 **entries**, not 504 keys + +`INDEX.md` (on `auto/re-idxd-container`) records *504 fields disc-wide that are +hash-keyed with no name*. That number is exactly reproduced here — and it counts +**field entries**, not distinct keys: + +``` +42 distinct keys x 6 language copies of the same object x 2 records = 504 +``` + +The three unnamed populations add up: +`504` (hash-keyed) `+ 1 380 324` (ordinal) `+ 104 749` (key `0x00000000`, i.e. +ordinal 0, held apart only because `tag_hash("") == 0` makes it look "named" +somewhere) `= 1 485 577`. So the brute-force target was never 504 names; it was +**42**. + +### ✅ The gap at `[0x10000, 0x01000000)` is what separates hashes from ordinals + +`tag_hash` puts the byte-sum checksum of the name in the **top byte**. A real +hash therefore falls below `0x01000000` only when the name's byte sum is `0 mod +256` — about 1 name in 256 — *and* its low 24 bits are also tiny. Of the 12 174 +keys that do carry a name, 75 have a zero top byte, and the smallest of those is +`0x0002677C`; the never-named ordinal band tops out at `0x2198`. There is a clean +three-order-of-magnitude gap and **not one key in it**. The split is measured, +not assumed. + +## ✅ The 7 052 ordinals are not hashes and never were + +They are author-assigned element ids written straight into the tag slot. For +1 404 924 of the 1 485 577 unnamed entries the key **equals the field's index +within its record**; a further 42 750 are index + 1. The rest are hand-numbered +with deliberate gaps, e.g. the hangar `WEAPONS` record runs +`…0x30 0x31 … 0x35`, jumps to `0x50 … 0x55`, then `0x61` — an id space, not a +counter. `tables.pak`'s `FILES` record numbers 5 135 resources from `0x3E9` +(1001) upwards. + +**Do not feed these to a reverse lookup.** Confirmed by the strongest possible +test: `tag_hash("BGM_001") = 0xC662435B`, while the key of the field whose value +is `BGM_001.slb` is `0x000003E9`. Zero of 5 135 `FILES` keys match a hash of +their own value under any stem/case/extension variant. + +Full per-key listing: Part 3 of the data file. Per-schema value profiles: Part 2. + +## ✅ What the 42 hash-shaped keys are + +All 42 live in **six byte-identical copies** of one object — `eng jpn fra deu ita +esp \script\ID.tbl` inside `dat/GP_READY_ROOM.pak` (resolved through the IPFB TOC +by `name_hash("eng\\script\\ID.tbl") = 0xC40BC120`, and pointed at by the +`SCRIPT_PATH` field of `tables.pak`'s `BASE_INFO`, whose value is +`dat\GP_READY_ROOM.pak+eng\script\`). + +The object has exactly two records, `FILE` and `OFFSET`, carrying **the same 42 +keys in the same order** — it is a column store: + +``` +key(tag_hash of an ISL script symbol) -> FILE = the .isb it is defined in + OFFSET = where in that file +``` + +It is the artefact of the pass the executable announces as +`silph::GamePart_ReadyRoom::Impl::PrepareScript - isl script prescanning start.` +— a link map from symbol hash to definition site. The 34 `.isb` files it names +are all present in the same pak (`eng\script\stage01.isb` etc.). + +### ✅ Independent confirmation: the keys appear in the bytecode + +41 of the 42 keys occur verbatim as **little-endian 32-bit words** inside the +`.isb` bytecode, and where they occur reconstructs a coherent call graph +(Part 4 of the data file): + +| defined in | referenced from | +|---|---| +| `stage01..16.isb`, `challenge01..06.isb`, `tutorial0N01.isb` (30 symbols) | `main.isb` only | +| `function.isb` (5 symbols) | the stage and challenge scripts | +| `function_tutorial.isb` (6 symbols) | the six tutorial scripts | +| `main.isb` (1 symbol) | **nothing** — the root entry point | + +That is exactly the shape a launcher + shared-helper-library arrangement should +have, and it is the reason these keys can be called hashes with confidence rather +than "large ordinals": ordinals are not stored as call targets in code. + +🟡 `OFFSET` is **base 36**. Every value uses only `[0-9a-z]`, and read as base 36 +they are small ascending numbers per file: `function.isb` → 1, 9, 19, 35, 58; +`function_tutorial.isb` → 1, 12, 23, 34, 45, 50; `tutorial0101.isb` → 1, 61, 191. +The first symbol of every file is 1. Whether the unit is a statement index, a +line, or a word offset is ❔ — not settled here. + +## ✅ 30 of the 42 names end in the digits of their own filename + +This is *forced* by the hash, not guessed. `tag_hash`'s low 24 bits are a +base-256 polynomial mod `0x00FFFFDF`, so bumping the character `k` places from +the end changes the low bits by `256^k mod M` (1, `0x100`, `0x10000`, `0x21`, …) +and the top byte by 1. Measured: + +| family | observed key deltas | forced conclusion | +|---|---|---| +| `stage01..09` | `+0x01000001` × 8 | last character increments by 1 | +| `stage10..16` | `+0x01000001` × 6 | ditto | +| `stage09 → stage10` | `0x8FA349EF → 0x87A34AE6` | low bits `+0xF7 = 0x100 − 9`, top `−8`: the last two characters go `"09" → "10"` | +| `challenge01..06` | `+0x01000001` × 5 | last character increments by 1 | +| `tutorial0101 → 0201 … 0601` | `+0x01010000` × 5 | the character **two** from the end increments | +| `tutorial0101`'s two siblings | `+0x01000001`, `+0x02000002` | last character `+1`, `+2` | + +So the sixteen stage symbols end `"01" … "16"`, the six challenge symbols end +`"01" … "06"`, and the eight tutorial symbols end `"0101" "0102" "0103" "0201" +"0301" "0401" "0501" "0601"` — matching their `.isb` filenames exactly. The +remaining 12 (`function.isb` × 5, `function_tutorial.isb` × 6, `main.isb` × 1) are +unconstrained. + +That is the whole of the recovered name information. **The prefixes are not +recovered.** + +## ❌ Cracking: a null result, and the arithmetic that says why + +`tag_hash` maps onto `256 × 0x00FFFFDF ≈ 2^32` values, so a search over `S` +candidates yields on average `S / 2^32` **false** preimages *per target*. That is +the number every attempt below is judged against. + +| attack | candidates (space `S`) | E[false] per target | hits | +|---|---|---|---| +| every distinct disc string (IDXD names, record names, field values) | 105 393 | 0.000025 | **0** | +| every string in the executable (`.pe`) | 75 912 | 0.000018 | **0** | +| every ASCII run ≥3 in every pak payload except `sound.pak` (incl. `.ssb` symbol tables, XML, XPR) | 1 023 671 | 0.00024 | **0** | +| all three combined, hashed against all 42 | 1 204 976 | 0.00028 | **0** | +| `printf`-style substitution: 49 139 format strings × 80 numeric substitutions | 161 200 | 0.00004 | **0** | +| hand-built guess list (`main`/`Stage`/`Mission`/`ReadyRoom`/… × prefixes × decorations × numbers) | 506 520 | 0.00012 | **0** | +| two-token composition over a vocabulary mined from disc + exe identifiers (`t1 + sep + t2`, sep ∈ `"" _ -`) — this is what `idxd_unnamed_keys.py crack --pe …` runs | 3.48 × 10⁸ | 0.081 | **0** | +| the same, with each family's forced digit suffix appended | 1.18 × 10¹⁰ across the sweep | 2.8 total | 1 (`"Cj-format 01"` — noise, and the same string for all six challenge keys, i.e. **one** false prefix) | +| meet-in-the-middle exhaustive preimage, `[A-Za-z0-9_]`, ≤ 6 chars before the known suffix | 6.3 × 10¹⁰ | ~15 | noise only | + +The last row is the important one. Exhaustive search **succeeds** — it is not +that the search is broken. Fed `tag_hash("Stage01")` it returns +`['F4D_kl01', 'Stage01', 'be_zT01', 'cd_Yu01']`: the right answer plus three +collisions, which is what `63^5 / 2^32 ≈ 0.2` … `63^6 / 2^32 ≈ 15` predicts. Fed +the real target `0x87A349E7` with suffix `"01"` it returns **nothing**, so the +prefix is **longer than 6 characters** over that alphabet. + +And that is where it stops being useful. Measured, not extrapolated: at 7 +characters target `0xA40C6DF7` has **1 176** preimages (`'1T1E1B6'`, +`'24A7O0G'`, `'4quqkkC'`, …). At 8 the expectation is ≈ 58 000. A hash with a +24-bit modulus simply does not have enough entropy to name an 8+ character +identifier uniquely, so **no amount of compute recovers these names from the hash +alone** — only a corpus containing the actual string can, and the disc does not +contain it. + +### 🔴 Refuted along the way (kept, not deleted) + +* **"The symbol is the filename stem, in some case/decoration."** Refuted + directly: `tag_hash` of `stage01`, `Stage01`, `STAGE01`, `stage_01`, + `mission01`, and 30-odd relatives is not `0x87A349E7`. +* **"The symbol is ` + + `."** Refuted + *algebraically*, without searching. If `name = A + "stage01"` and + `name = A + "challenge01"` share one `A`, then `A`'s byte sum is pinned twice + and the two values must agree. They do not, for lowercase, Capitalised or + UPPER stems, with or without a `_`/`-` separator — 9 pairs tested, 9 + contradictions. +* **"The name is somewhere in the `.isb` files."** Refuted: the `.isb` payloads + contain no ASCII symbol names at all. They reference symbols purely by hash — + which is *why* the prescan link map exists. +* **"The hash constants might be in the executable next to a name table."** + Refuted: none of the 42 values occurs in the `.pe` image in either endianness. +* **"IXUD might hold more hash-keyed nameless fields."** Refuted by measurement: + 534 IXUD objects, 624 488 field entries, **8** distinct unnamed tags — all of + them ordinals `0…7` (the weapon `CATEGORY_DESC` list). Zero hash-shaped. +* **⚠️ "There are ~504 keys to crack."** Withdrawn — 504 is the *entry* count. + See above. + +## Coverage / limits + +- The census covers every IDXD object the disc has (7 750). `sound.pak` holds no + IDXD objects and `dat/movie/*.wmv` and `hidden/resource3d/*.xpr` are not + containers; strings from `resource3d` were nevertheless folded into the corpus + attack. +- The 42 targets are fully characterised as *references*; their *spelling* is + unrecovered and, per the arithmetic above, unrecoverable from the disc. + +## Evidence log (append-only; newest last) + +- `2026-08-26` — Disc-wide walk of 7 750 IDXD objects / 2 757 039 field entries + with `tag_hash` self-check at 1 271 462/1 271 462. Census → confidence + `CONFIRMED` for the counts and for the ordinal/hash split. +- `2026-08-26` — 42 hash-shaped keys localised to six copies of + `\script\ID.tbl`; cross-checked against the `.isb` bytecode, where 41 of + 42 appear as little-endian call targets forming a consistent call graph → + `CONFIRMED` that they are ISL script-symbol hashes. +- `2026-08-26` — Key-delta arithmetic pins the trailing digits of 30 of the 42 + names → `CONFIRMED` (forced by the hash's polynomial structure, three + independent families agreeing). +- `2026-08-26` — Seven corpus/composition/brute-force attacks, none producing a + hit above the noise floor; exhaustive search validated on a known name and + shown to return nothing at ≤ 6 characters → the names are **not recovered**, + and the negative result is quantified rather than asserted. + +## Open questions / what would raise confidence + +- **The 12 unconstrained names** (`function.isb` ×5, `function_tutorial.isb` ×6, + `main.isb` ×1) have no digit structure to exploit at all. +- **What would actually work:** a build artefact containing ISL source or a symbol + list (none on the retail disc); or observing the game *construct* one of these + strings at runtime — `PrepareScript` builds the table from `.isb` contents, but + whatever `main.isb`'s root symbol is called, the name existed only in the + compiler's input. A `--mem-watch` on the ReadyRoom lookup path would show + whether any caller ever passes a literal name rather than a precomputed hash; + that is the one remaining honest route and it was **not** attempted here (this + was a static-only investigation). +- **`OFFSET`'s unit** — base-36 is settled, what it counts is not. +- The `.isb` instruction encoding is only sketched here (a length byte at + `byte[2]`, call payloads leading with the target hash). It is not the `.ssb` + ISL bytecode documented elsewhere and deserves its own note.