Merge branch 'auto/idxd-unnamed-keys'

# Conflicts:
#	docs/re/INDEX.md

docs/re/structures/idxd-unnamed-keys.md

@@ -0,0 +1,253 @@
+# IDXD field keys that carry no name
+
+- **Confidence:** ✅ the census and the identification of the hash-keyed set ·
+  ✅ the last characters of 30 of the 42 names · ❌ **no name was recovered**
+- **Tool:** `tools/re-capture/idxd_unnamed_keys.py` (`census` / `idtbl` / `crack` / `selftest`)
+- **Data:** [`docs/re/data/idxd-unnamed-field-keys.txt`](../data/idxd-unnamed-field-keys.txt)
+- **Seen in:** every `dat/*.pak` + `hidden/*.pak` on the disc; the interesting 42
+  are in `dat/GP_READY_ROOM.pak`, file `<lang>\script\ID.tbl`
+- **Depends on:** the IDXD record/field table and `tag_hash`, which live on branch
+  `auto/re-idxd-container` — **not on `main`**, so this note transcribes what it
+  needs and the tool carries its own copy of `tag_hash` rather than importing one.
+
+An IDXD field entry is `(key, name_off, value_off)`. Usually `name_off` points at
+the field's own name in the string pool and `key == tag_hash(name)`. When
+`name_off` is `0xFFFFFFFF` the pool holds no name and the reader has to know what
+`key` means from somewhere else. This note answers: how many such keys are there,
+what are they, and can the names be inverted out of the hash?
+
+**Short answer: 7 094 distinct keys, of which 7 052 are not hashes at all and 42
+are. None of the 42 names was recovered — but they are now identified, their call
+graph is reconstructed, and 30 of them have their last two-to-four characters
+pinned algebraically.**
+
+## ✅ The census
+
+Measured by walking all 33 `dat/*.pak` plus `hidden/DefTables.pak` and
+`hidden/MiscBin.pak` (`.pNN` segments joined, `Z1` payloads inflated):
+
+| | |
+|---|---|
+| IDXD objects / records / field entries | 7 750 / 190 782 / **2 757 039** |
+| named field entries | 1 271 462 |
+| unnamed field entries (`name_off == 0xFFFFFFFF`) | 1 485 577 |
+| distinct keys named *somewhere* on disc | 12 174 |
+| **distinct keys never named anywhere** | **7 094** |
+|  … below `0x00010000` | 7 052 |
+|  … in `[0x10000, 0x01000000)` | **0** |
+|  … at or above `0x01000000` | **42** |
+
+Parse failures: 0. `tag_hash` reproduces the key of **1 271 462 / 1 271 462**
+named field entries disc-wide (`idxd_unnamed_keys.py selftest`), which is the
+validation gate for everything below — far past the 200-pair bar this
+investigation was asked to clear.
+
+### ✅ "504 hash-keyed fields disc-wide" is 504 **entries**, not 504 keys
+
+`INDEX.md` (on `auto/re-idxd-container`) records *504 fields disc-wide that are
+hash-keyed with no name*. That number is exactly reproduced here — and it counts
+**field entries**, not distinct keys:
+
+```
+42 distinct keys  x  6 language copies of the same object  x  2 records = 504
+```
+
+The three unnamed populations add up:
+`504` (hash-keyed) `+ 1 380 324` (ordinal) `+ 104 749` (key `0x00000000`, i.e.
+ordinal 0, held apart only because `tag_hash("") == 0` makes it look "named"
+somewhere) `= 1 485 577`. So the brute-force target was never 504 names; it was
+**42**.
+
+### ✅ The gap at `[0x10000, 0x01000000)` is what separates hashes from ordinals
+
+`tag_hash` puts the byte-sum checksum of the name in the **top byte**. A real
+hash therefore falls below `0x01000000` only when the name's byte sum is `0 mod
+256` — about 1 name in 256 — *and* its low 24 bits are also tiny. Of the 12 174
+keys that do carry a name, 75 have a zero top byte, and the smallest of those is
+`0x0002677C`; the never-named ordinal band tops out at `0x2198`. There is a clean
+three-order-of-magnitude gap and **not one key in it**. The split is measured,
+not assumed.
+
+## ✅ The 7 052 ordinals are not hashes and never were
+
+They are author-assigned element ids written straight into the tag slot. For
+1 404 924 of the 1 485 577 unnamed entries the key **equals the field's index
+within its record**; a further 42 750 are index + 1. The rest are hand-numbered
+with deliberate gaps, e.g. the hangar `WEAPONS` record runs
+`…0x30 0x31 … 0x35`, jumps to `0x50 … 0x55`, then `0x61` — an id space, not a
+counter. `tables.pak`'s `FILES` record numbers 5 135 resources from `0x3E9`
+(1001) upwards.
+
+**Do not feed these to a reverse lookup.** Confirmed by the strongest possible
+test: `tag_hash("BGM_001") = 0xC662435B`, while the key of the field whose value
+is `BGM_001.slb` is `0x000003E9`. Zero of 5 135 `FILES` keys match a hash of
+their own value under any stem/case/extension variant.
+
+Full per-key listing: Part 3 of the data file. Per-schema value profiles: Part 2.
+
+## ✅ What the 42 hash-shaped keys are
+
+All 42 live in **six byte-identical copies** of one object — `eng jpn fra deu ita
+esp \script\ID.tbl` inside `dat/GP_READY_ROOM.pak` (resolved through the IPFB TOC
+by `name_hash("eng\\script\\ID.tbl") = 0xC40BC120`, and pointed at by the
+`SCRIPT_PATH` field of `tables.pak`'s `BASE_INFO`, whose value is
+`dat\GP_READY_ROOM.pak+eng\script\`).
+
+The object has exactly two records, `FILE` and `OFFSET`, carrying **the same 42
+keys in the same order** — it is a column store:
+
+```
+key(tag_hash of an ISL script symbol)  ->  FILE   = the .isb it is defined in
+                                           OFFSET = where in that file
+```
+
+It is the artefact of the pass the executable announces as
+`silph::GamePart_ReadyRoom::Impl::PrepareScript - isl script prescanning start.`
+— a link map from symbol hash to definition site. The 34 `.isb` files it names
+are all present in the same pak (`eng\script\stage01.isb` etc.).
+
+### ✅ Independent confirmation: the keys appear in the bytecode
+
+41 of the 42 keys occur verbatim as **little-endian 32-bit words** inside the
+`.isb` bytecode, and where they occur reconstructs a coherent call graph
+(Part 4 of the data file):
+
+| defined in | referenced from |
+|---|---|
+| `stage01..16.isb`, `challenge01..06.isb`, `tutorial0N01.isb` (30 symbols) | `main.isb` only |
+| `function.isb` (5 symbols) | the stage and challenge scripts |
+| `function_tutorial.isb` (6 symbols) | the six tutorial scripts |
+| `main.isb` (1 symbol) | **nothing** — the root entry point |
+
+That is exactly the shape a launcher + shared-helper-library arrangement should
+have, and it is the reason these keys can be called hashes with confidence rather
+than "large ordinals": ordinals are not stored as call targets in code.
+
+🟡 `OFFSET` is **base 36**. Every value uses only `[0-9a-z]`, and read as base 36
+they are small ascending numbers per file: `function.isb` → 1, 9, 19, 35, 58;
+`function_tutorial.isb` → 1, 12, 23, 34, 45, 50; `tutorial0101.isb` → 1, 61, 191.
+The first symbol of every file is 1. Whether the unit is a statement index, a
+line, or a word offset is ❔ — not settled here.
+
+## ✅ 30 of the 42 names end in the digits of their own filename
+
+This is *forced* by the hash, not guessed. `tag_hash`'s low 24 bits are a
+base-256 polynomial mod `0x00FFFFDF`, so bumping the character `k` places from
+the end changes the low bits by `256^k mod M` (1, `0x100`, `0x10000`, `0x21`, …)
+and the top byte by 1. Measured:
+
+| family | observed key deltas | forced conclusion |
+|---|---|---|
+| `stage01..09` | `+0x01000001` × 8 | last character increments by 1 |
+| `stage10..16` | `+0x01000001` × 6 | ditto |
+| `stage09 → stage10` | `0x8FA349EF → 0x87A34AE6` | low bits `+0xF7 = 0x100 − 9`, top `−8`: the last two characters go `"09" → "10"` |
+| `challenge01..06` | `+0x01000001` × 5 | last character increments by 1 |
+| `tutorial0101 → 0201 … 0601` | `+0x01010000` × 5 | the character **two** from the end increments |
+| `tutorial0101`'s two siblings | `+0x01000001`, `+0x02000002` | last character `+1`, `+2` |
+
+So the sixteen stage symbols end `"01" … "16"`, the six challenge symbols end
+`"01" … "06"`, and the eight tutorial symbols end `"0101" "0102" "0103" "0201"
+"0301" "0401" "0501" "0601"` — matching their `.isb` filenames exactly. The
+remaining 12 (`function.isb` × 5, `function_tutorial.isb` × 6, `main.isb` × 1) are
+unconstrained.
+
+That is the whole of the recovered name information. **The prefixes are not
+recovered.**
+
+## ❌ Cracking: a null result, and the arithmetic that says why
+
+`tag_hash` maps onto `256 × 0x00FFFFDF ≈ 2^32` values, so a search over `S`
+candidates yields on average `S / 2^32` **false** preimages *per target*. That is
+the number every attempt below is judged against.
+
+| attack | candidates (space `S`) | E[false] per target | hits |
+|---|---|---|---|
+| every distinct disc string (IDXD names, record names, field values) | 105 393 | 0.000025 | **0** |
+| every string in the executable (`.pe`) | 75 912 | 0.000018 | **0** |
+| every ASCII run ≥3 in every pak payload except `sound.pak` (incl. `.ssb` symbol tables, XML, XPR) | 1 023 671 | 0.00024 | **0** |
+| all three combined, hashed against all 42 | 1 204 976 | 0.00028 | **0** |
+| `printf`-style substitution: 49 139 format strings × 80 numeric substitutions | 161 200 | 0.00004 | **0** |
+| hand-built guess list (`main`/`Stage`/`Mission`/`ReadyRoom`/… × prefixes × decorations × numbers) | 506 520 | 0.00012 | **0** |
+| two-token composition over a vocabulary mined from disc + exe identifiers (`t1 + sep + t2`, sep ∈ `"" _ -`) — this is what `idxd_unnamed_keys.py crack --pe …` runs | 3.48 × 10⁸ | 0.081 | **0** |
+| the same, with each family's forced digit suffix appended | 1.18 × 10¹⁰ across the sweep | 2.8 total | 1 (`"Cj-format 01"` — noise, and the same string for all six challenge keys, i.e. **one** false prefix) |
+| meet-in-the-middle exhaustive preimage, `[A-Za-z0-9_]`, ≤ 6 chars before the known suffix | 6.3 × 10¹⁰ | ~15 | noise only |
+
+The last row is the important one. Exhaustive search **succeeds** — it is not
+that the search is broken. Fed `tag_hash("Stage01")` it returns
+`['F4D_kl01', 'Stage01', 'be_zT01', 'cd_Yu01']`: the right answer plus three
+collisions, which is what `63^5 / 2^32 ≈ 0.2` … `63^6 / 2^32 ≈ 15` predicts. Fed
+the real target `0x87A349E7` with suffix `"01"` it returns **nothing**, so the
+prefix is **longer than 6 characters** over that alphabet.
+
+And that is where it stops being useful. Measured, not extrapolated: at 7
+characters target `0xA40C6DF7` has **1 176** preimages (`'1T1E1B6'`,
+`'24A7O0G'`, `'4quqkkC'`, …). At 8 the expectation is ≈ 58 000. A hash with a
+24-bit modulus simply does not have enough entropy to name an 8+ character
+identifier uniquely, so **no amount of compute recovers these names from the hash
+alone** — only a corpus containing the actual string can, and the disc does not
+contain it.
+
+### 🔴 Refuted along the way (kept, not deleted)
+
+* **"The symbol is the filename stem, in some case/decoration."** Refuted
+  directly: `tag_hash` of `stage01`, `Stage01`, `STAGE01`, `stage_01`,
+  `mission01`, and 30-odd relatives is not `0x87A349E7`.
+* **"The symbol is `<common prefix> + <filename stem> + <number>`."** Refuted
+  *algebraically*, without searching. If `name = A + "stage01"` and
+  `name = A + "challenge01"` share one `A`, then `A`'s byte sum is pinned twice
+  and the two values must agree. They do not, for lowercase, Capitalised or
+  UPPER stems, with or without a `_`/`-` separator — 9 pairs tested, 9
+  contradictions.
+* **"The name is somewhere in the `.isb` files."** Refuted: the `.isb` payloads
+  contain no ASCII symbol names at all. They reference symbols purely by hash —
+  which is *why* the prescan link map exists.
+* **"The hash constants might be in the executable next to a name table."**
+  Refuted: none of the 42 values occurs in the `.pe` image in either endianness.
+* **"IXUD might hold more hash-keyed nameless fields."** Refuted by measurement:
+  534 IXUD objects, 624 488 field entries, **8** distinct unnamed tags — all of
+  them ordinals `0…7` (the weapon `CATEGORY_DESC` list). Zero hash-shaped.
+* **⚠️ "There are ~504 keys to crack."** Withdrawn — 504 is the *entry* count.
+  See above.
+
+## Coverage / limits
+
+- The census covers every IDXD object the disc has (7 750). `sound.pak` holds no
+  IDXD objects and `dat/movie/*.wmv` and `hidden/resource3d/*.xpr` are not
+  containers; strings from `resource3d` were nevertheless folded into the corpus
+  attack.
+- The 42 targets are fully characterised as *references*; their *spelling* is
+  unrecovered and, per the arithmetic above, unrecoverable from the disc.
+
+## Evidence log (append-only; newest last)
+
+- `2026-08-26` — Disc-wide walk of 7 750 IDXD objects / 2 757 039 field entries
+  with `tag_hash` self-check at 1 271 462/1 271 462. Census → confidence
+  `CONFIRMED` for the counts and for the ordinal/hash split.
+- `2026-08-26` — 42 hash-shaped keys localised to six copies of
+  `<lang>\script\ID.tbl`; cross-checked against the `.isb` bytecode, where 41 of
+  42 appear as little-endian call targets forming a consistent call graph →
+  `CONFIRMED` that they are ISL script-symbol hashes.
+- `2026-08-26` — Key-delta arithmetic pins the trailing digits of 30 of the 42
+  names → `CONFIRMED` (forced by the hash's polynomial structure, three
+  independent families agreeing).
+- `2026-08-26` — Seven corpus/composition/brute-force attacks, none producing a
+  hit above the noise floor; exhaustive search validated on a known name and
+  shown to return nothing at ≤ 6 characters → the names are **not recovered**,
+  and the negative result is quantified rather than asserted.
+
+## Open questions / what would raise confidence
+
+- **The 12 unconstrained names** (`function.isb` ×5, `function_tutorial.isb` ×6,
+  `main.isb` ×1) have no digit structure to exploit at all.
+- **What would actually work:** a build artefact containing ISL source or a symbol
+  list (none on the retail disc); or observing the game *construct* one of these
+  strings at runtime — `PrepareScript` builds the table from `.isb` contents, but
+  whatever `main.isb`'s root symbol is called, the name existed only in the
+  compiler's input. A `--mem-watch` on the ReadyRoom lookup path would show
+  whether any caller ever passes a literal name rather than a precomputed hash;
+  that is the one remaining honest route and it was **not** attempted here (this
+  was a static-only investigation).
+- **`OFFSET`'s unit** — base-36 is settled, what it counts is not.
+- The `.isb` instruction encoding is only sketched here (a length byte at
+  `byte[2]`, call payloads leading with the target hash). It is not the `.ssb`
+  ISL bytecode documented elsewhere and deserves its own note.