Merge branch 'auto/idxd-unnamed-keys'

# Conflicts:
#	docs/re/INDEX.md

docs/re/INDEX.md

@@ -12,7 +12,12 @@ Promote to a prose `structures/…md` file when a format needs behavioural notes
 |--------|-------|-----------------------|-------|
 | IPFB `.pak` archive | ✅ | `sylpheed-formats/src/pak.rs` + `tests/pak_idxd_disc.rs` | header + 12-byte TOC, Z1/zlib payloads |
 | name-hash (TOC keys) | ✅ | `sylpheed-formats/src/hash.rs` | Barrett-reduction hash; recovers original paths |
+<<<<<<< HEAD
 | IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` + `tests/idxd_records_disc.rs` ([container](structures/idxd-container.md)) | **The binary record/index region in front of the string pool is DECODED** (2026-08-25), closing the parser's long-standing "not yet decoded" note. Uniform 16-byte records `{name_hash, name_off, field_begin, field_end}` sorted by hash and binary-searched, then a field count, 12-byte fields `{key, name_off, value_off}` sorted by key, a pool size, and the string pool; the trailing `pool_size == file_len - pool_base` identity makes the layout self-checking. Verified over the **whole disc** with **zero** failures: 7 750/7 750 objects, 190 782/190 782 records reproducing their stored `tag_hash`, 1 271 462/1 271 462 named fields reproducing their key — and `IXUD` is the same container with `ixud_hash`, UTF-16BE and all offsets in **chars** (1 104/1 104 objects, 628 165/628 165 fields). **Field names are stored on disc** — a field's middle word points at its own name — so nothing needs preimage search except the **504** field entries disc-wide that are hash-keyed with no name — which are only **42 distinct keys**, each in 12 places (the page for these was never written — the finding is in this row), and are provably unrecoverable from the hash alone; the other 1 485 073 nameless fields are *positional*, keyed by a literal integer (line slots, movie ids). ⚠️ **Two long-held beliefs WITHDRAWN**: the word at `0x08` is **not a schema hash**, it is record 0's `name_hash` (7 750/7 750) — the header has no type field at all, so an object's kind is known only from the caller that loads it; and the field's middle word is **not** an always-`0xFFFFFFFF` flags word. The first was caught by a test asserting that every movie id names a real record: `1005 -> STAGE10_PHASE01` failed because `tag_hash("STAGE10_PHASE01")` **is** `0x067025B9`, that table's supposed schema id. 🟡 the legacy value-before-key string-pool reader is now known to be an *approximation* of the real table, and every number derived from it is re-checkable but not yet re-checked |
+=======
+| IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` | self-describing; ship/weapon stats verified vs known values |
+| IDXD nameless field keys | ✅/❌ | [idxd-unnamed-keys](structures/idxd-unnamed-keys.md) + [`tools/re-capture/idxd_unnamed_keys.py`](../../tools/re-capture/idxd_unnamed_keys.py) | Census of every field entry whose `name_off` is `0xFFFFFFFF`, disc-wide: **7 750 objects, 2 757 039 field entries, 0 parse failures**, `tag_hash` reproducing **1 271 462/1 271 462** named keys. **7 094 distinct keys are never named — and 7 052 of them are not hashes at all**, but author-assigned element ids (equal to the field's own index in 1 404 924 of 1 485 577 cases; `tag_hash("BGM_001")` is `0xC662435B` while the key valued `BGM_001.slb` is `0x000003E9`). ⚠️ **The "504 hash-keyed nameless fields" figure is 504 ENTRIES, not 504 names** — 42 distinct keys × 6 language copies × 2 records. All 42 are **ISL script-symbol hashes** in `<lang>\script\ID.tbl` (GP_READY_ROOM.pak), the link map built by `PrepareScript`'s "isl script prescanning"; 41 of 42 appear as little-endian call targets inside the `.isb` bytecode, forming a coherent launcher/helper call graph. The hash's own algebra pins the **trailing digits of 30 of the 42 names** (deltas of exactly `+0x01000001` across `stage01..09`, `stage10..16`, `challenge01..06`; `+0x01010000` across `tutorial0101..0601`). ❌ **No name was cracked, and the negative is quantified**: seven attacks up to a 3.5×10⁸ composition space found nothing above the noise floor; exhaustive preimage search recovers `"Stage01"` from its own hash but returns nothing for the real targets at ≤6 characters, and at 7 characters one target already has **1 176** preimages — a 24-bit modulus cannot name an 8+ character identifier uniquely |
+>>>>>>> auto/idxd-unnamed-keys
 | XPR2 texture + cubemap | 🟡/✅ | `sylpheed-formats/src/texture.rs` + [colour check](xpr2-colour-check.md) | de-tile + A8R8G8B8 and DXT1. **Channel order ✅ confirmed against the running game**: the Delta Saber's decoded atlas is orange-dominant (median saturated hue 23.3°, *zero* cool pixels) and the game renders the same hull at 9.3° — a red↔blue swap would sit at ≈200°. Exact fidelity (gamma/sRGB curve, premultiplied alpha, per-channel scale) is 🟡 untested, since a hue comparison cannot see it; cubemap face ordering ❔ |
 | T8aD 2D texture | ✅ | `sylpheed-formats/src/t8ad.rs` | **100 % of the disc decodes** (19 216/19 216, measured). The "~15 % deferred variants" were a wrong model, not a variant: a surface is a list of **arbitrary sub-rectangles**, each with a 16-byte header of `dst X, dst Y, width, height`, not a 256×256 grid — `0x1c` is the **rectangle count**. Uncovered area stays transparent. **Colours ✅ CONFIRMED** ([k8888](structures/texture-color-k8888.md)) |
 | RATC bundle | ✅ | `sylpheed-formats/src/ratc.rs` | child listing confirmed. **"One level deep" is not a limitation — there is nothing deeper**: 2 859 bundles hold 18 002 children at depth 1 and **0 at depth 2**, with no parse failures. Nested RATC blobs are **leaf records that reference siblings by name** (`opt `, the sprite name): 3 311 leaves, all embedding sibling names, **10 144 of 10 148 references resolve**. The 4 that do not are one dangling asset — `pmbase.rat` → `pmbase.t32` in `GP_STAGE_CLEAR.pak`'s four language builds, and `pmbase.t32` is **on the disc nowhere** |

docs/re/structures/idxd-unnamed-keys.md

@@ -0,0 +1,253 @@
+# IDXD field keys that carry no name
+
+- **Confidence:** ✅ the census and the identification of the hash-keyed set ·
+  ✅ the last characters of 30 of the 42 names · ❌ **no name was recovered**
+- **Tool:** `tools/re-capture/idxd_unnamed_keys.py` (`census` / `idtbl` / `crack` / `selftest`)
+- **Data:** [`docs/re/data/idxd-unnamed-field-keys.txt`](../data/idxd-unnamed-field-keys.txt)
+- **Seen in:** every `dat/*.pak` + `hidden/*.pak` on the disc; the interesting 42
+  are in `dat/GP_READY_ROOM.pak`, file `<lang>\script\ID.tbl`
+- **Depends on:** the IDXD record/field table and `tag_hash`, which live on branch
+  `auto/re-idxd-container` — **not on `main`**, so this note transcribes what it
+  needs and the tool carries its own copy of `tag_hash` rather than importing one.
+
+An IDXD field entry is `(key, name_off, value_off)`. Usually `name_off` points at
+the field's own name in the string pool and `key == tag_hash(name)`. When
+`name_off` is `0xFFFFFFFF` the pool holds no name and the reader has to know what
+`key` means from somewhere else. This note answers: how many such keys are there,
+what are they, and can the names be inverted out of the hash?
+
+**Short answer: 7 094 distinct keys, of which 7 052 are not hashes at all and 42
+are. None of the 42 names was recovered — but they are now identified, their call
+graph is reconstructed, and 30 of them have their last two-to-four characters
+pinned algebraically.**
+
+## ✅ The census
+
+Measured by walking all 33 `dat/*.pak` plus `hidden/DefTables.pak` and
+`hidden/MiscBin.pak` (`.pNN` segments joined, `Z1` payloads inflated):
+
+| | |
+|---|---|
+| IDXD objects / records / field entries | 7 750 / 190 782 / **2 757 039** |
+| named field entries | 1 271 462 |
+| unnamed field entries (`name_off == 0xFFFFFFFF`) | 1 485 577 |
+| distinct keys named *somewhere* on disc | 12 174 |
+| **distinct keys never named anywhere** | **7 094** |
+|  … below `0x00010000` | 7 052 |
+|  … in `[0x10000, 0x01000000)` | **0** |
+|  … at or above `0x01000000` | **42** |
+
+Parse failures: 0. `tag_hash` reproduces the key of **1 271 462 / 1 271 462**
+named field entries disc-wide (`idxd_unnamed_keys.py selftest`), which is the
+validation gate for everything below — far past the 200-pair bar this
+investigation was asked to clear.
+
+### ✅ "504 hash-keyed fields disc-wide" is 504 **entries**, not 504 keys
+
+`INDEX.md` (on `auto/re-idxd-container`) records *504 fields disc-wide that are
+hash-keyed with no name*. That number is exactly reproduced here — and it counts
+**field entries**, not distinct keys:
+
+```
+42 distinct keys  x  6 language copies of the same object  x  2 records = 504
+```
+
+The three unnamed populations add up:
+`504` (hash-keyed) `+ 1 380 324` (ordinal) `+ 104 749` (key `0x00000000`, i.e.
+ordinal 0, held apart only because `tag_hash("") == 0` makes it look "named"
+somewhere) `= 1 485 577`. So the brute-force target was never 504 names; it was
+**42**.
+
+### ✅ The gap at `[0x10000, 0x01000000)` is what separates hashes from ordinals
+
+`tag_hash` puts the byte-sum checksum of the name in the **top byte**. A real
+hash therefore falls below `0x01000000` only when the name's byte sum is `0 mod
+256` — about 1 name in 256 — *and* its low 24 bits are also tiny. Of the 12 174
+keys that do carry a name, 75 have a zero top byte, and the smallest of those is
+`0x0002677C`; the never-named ordinal band tops out at `0x2198`. There is a clean
+three-order-of-magnitude gap and **not one key in it**. The split is measured,
+not assumed.
+
+## ✅ The 7 052 ordinals are not hashes and never were
+
+They are author-assigned element ids written straight into the tag slot. For
+1 404 924 of the 1 485 577 unnamed entries the key **equals the field's index
+within its record**; a further 42 750 are index + 1. The rest are hand-numbered
+with deliberate gaps, e.g. the hangar `WEAPONS` record runs
+`…0x30 0x31 … 0x35`, jumps to `0x50 … 0x55`, then `0x61` — an id space, not a
+counter. `tables.pak`'s `FILES` record numbers 5 135 resources from `0x3E9`
+(1001) upwards.
+
+**Do not feed these to a reverse lookup.** Confirmed by the strongest possible
+test: `tag_hash("BGM_001") = 0xC662435B`, while the key of the field whose value
+is `BGM_001.slb` is `0x000003E9`. Zero of 5 135 `FILES` keys match a hash of
+their own value under any stem/case/extension variant.
+
+Full per-key listing: Part 3 of the data file. Per-schema value profiles: Part 2.
+
+## ✅ What the 42 hash-shaped keys are
+
+All 42 live in **six byte-identical copies** of one object — `eng jpn fra deu ita
+esp \script\ID.tbl` inside `dat/GP_READY_ROOM.pak` (resolved through the IPFB TOC
+by `name_hash("eng\\script\\ID.tbl") = 0xC40BC120`, and pointed at by the
+`SCRIPT_PATH` field of `tables.pak`'s `BASE_INFO`, whose value is
+`dat\GP_READY_ROOM.pak+eng\script\`).
+
+The object has exactly two records, `FILE` and `OFFSET`, carrying **the same 42
+keys in the same order** — it is a column store:
+
+```
+key(tag_hash of an ISL script symbol)  ->  FILE   = the .isb it is defined in
+                                           OFFSET = where in that file
+```
+
+It is the artefact of the pass the executable announces as
+`silph::GamePart_ReadyRoom::Impl::PrepareScript - isl script prescanning start.`
+— a link map from symbol hash to definition site. The 34 `.isb` files it names
+are all present in the same pak (`eng\script\stage01.isb` etc.).
+
+### ✅ Independent confirmation: the keys appear in the bytecode
+
+41 of the 42 keys occur verbatim as **little-endian 32-bit words** inside the
+`.isb` bytecode, and where they occur reconstructs a coherent call graph
+(Part 4 of the data file):
+
+| defined in | referenced from |
+|---|---|
+| `stage01..16.isb`, `challenge01..06.isb`, `tutorial0N01.isb` (30 symbols) | `main.isb` only |
+| `function.isb` (5 symbols) | the stage and challenge scripts |
+| `function_tutorial.isb` (6 symbols) | the six tutorial scripts |
+| `main.isb` (1 symbol) | **nothing** — the root entry point |
+
+That is exactly the shape a launcher + shared-helper-library arrangement should
+have, and it is the reason these keys can be called hashes with confidence rather
+than "large ordinals": ordinals are not stored as call targets in code.
+
+🟡 `OFFSET` is **base 36**. Every value uses only `[0-9a-z]`, and read as base 36
+they are small ascending numbers per file: `function.isb` → 1, 9, 19, 35, 58;
+`function_tutorial.isb` → 1, 12, 23, 34, 45, 50; `tutorial0101.isb` → 1, 61, 191.
+The first symbol of every file is 1. Whether the unit is a statement index, a
+line, or a word offset is ❔ — not settled here.
+
+## ✅ 30 of the 42 names end in the digits of their own filename
+
+This is *forced* by the hash, not guessed. `tag_hash`'s low 24 bits are a
+base-256 polynomial mod `0x00FFFFDF`, so bumping the character `k` places from
+the end changes the low bits by `256^k mod M` (1, `0x100`, `0x10000`, `0x21`, …)
+and the top byte by 1. Measured:
+
+| family | observed key deltas | forced conclusion |
+|---|---|---|
+| `stage01..09` | `+0x01000001` × 8 | last character increments by 1 |
+| `stage10..16` | `+0x01000001` × 6 | ditto |
+| `stage09 → stage10` | `0x8FA349EF → 0x87A34AE6` | low bits `+0xF7 = 0x100 − 9`, top `−8`: the last two characters go `"09" → "10"` |
+| `challenge01..06` | `+0x01000001` × 5 | last character increments by 1 |
+| `tutorial0101 → 0201 … 0601` | `+0x01010000` × 5 | the character **two** from the end increments |
+| `tutorial0101`'s two siblings | `+0x01000001`, `+0x02000002` | last character `+1`, `+2` |
+
+So the sixteen stage symbols end `"01" … "16"`, the six challenge symbols end
+`"01" … "06"`, and the eight tutorial symbols end `"0101" "0102" "0103" "0201"
+"0301" "0401" "0501" "0601"` — matching their `.isb` filenames exactly. The
+remaining 12 (`function.isb` × 5, `function_tutorial.isb` × 6, `main.isb` × 1) are
+unconstrained.
+
+That is the whole of the recovered name information. **The prefixes are not
+recovered.**
+
+## ❌ Cracking: a null result, and the arithmetic that says why
+
+`tag_hash` maps onto `256 × 0x00FFFFDF ≈ 2^32` values, so a search over `S`
+candidates yields on average `S / 2^32` **false** preimages *per target*. That is
+the number every attempt below is judged against.
+
+| attack | candidates (space `S`) | E[false] per target | hits |
+|---|---|---|---|
+| every distinct disc string (IDXD names, record names, field values) | 105 393 | 0.000025 | **0** |
+| every string in the executable (`.pe`) | 75 912 | 0.000018 | **0** |
+| every ASCII run ≥3 in every pak payload except `sound.pak` (incl. `.ssb` symbol tables, XML, XPR) | 1 023 671 | 0.00024 | **0** |
+| all three combined, hashed against all 42 | 1 204 976 | 0.00028 | **0** |
+| `printf`-style substitution: 49 139 format strings × 80 numeric substitutions | 161 200 | 0.00004 | **0** |
+| hand-built guess list (`main`/`Stage`/`Mission`/`ReadyRoom`/… × prefixes × decorations × numbers) | 506 520 | 0.00012 | **0** |
+| two-token composition over a vocabulary mined from disc + exe identifiers (`t1 + sep + t2`, sep ∈ `"" _ -`) — this is what `idxd_unnamed_keys.py crack --pe …` runs | 3.48 × 10⁸ | 0.081 | **0** |
+| the same, with each family's forced digit suffix appended | 1.18 × 10¹⁰ across the sweep | 2.8 total | 1 (`"Cj-format 01"` — noise, and the same string for all six challenge keys, i.e. **one** false prefix) |
+| meet-in-the-middle exhaustive preimage, `[A-Za-z0-9_]`, ≤ 6 chars before the known suffix | 6.3 × 10¹⁰ | ~15 | noise only |
+
+The last row is the important one. Exhaustive search **succeeds** — it is not
+that the search is broken. Fed `tag_hash("Stage01")` it returns
+`['F4D_kl01', 'Stage01', 'be_zT01', 'cd_Yu01']`: the right answer plus three
+collisions, which is what `63^5 / 2^32 ≈ 0.2` … `63^6 / 2^32 ≈ 15` predicts. Fed
+the real target `0x87A349E7` with suffix `"01"` it returns **nothing**, so the
+prefix is **longer than 6 characters** over that alphabet.
+
+And that is where it stops being useful. Measured, not extrapolated: at 7
+characters target `0xA40C6DF7` has **1 176** preimages (`'1T1E1B6'`,
+`'24A7O0G'`, `'4quqkkC'`, …). At 8 the expectation is ≈ 58 000. A hash with a
+24-bit modulus simply does not have enough entropy to name an 8+ character
+identifier uniquely, so **no amount of compute recovers these names from the hash
+alone** — only a corpus containing the actual string can, and the disc does not
+contain it.
+
+### 🔴 Refuted along the way (kept, not deleted)
+
+* **"The symbol is the filename stem, in some case/decoration."** Refuted
+  directly: `tag_hash` of `stage01`, `Stage01`, `STAGE01`, `stage_01`,
+  `mission01`, and 30-odd relatives is not `0x87A349E7`.
+* **"The symbol is `<common prefix> + <filename stem> + <number>`."** Refuted
+  *algebraically*, without searching. If `name = A + "stage01"` and
+  `name = A + "challenge01"` share one `A`, then `A`'s byte sum is pinned twice
+  and the two values must agree. They do not, for lowercase, Capitalised or
+  UPPER stems, with or without a `_`/`-` separator — 9 pairs tested, 9
+  contradictions.
+* **"The name is somewhere in the `.isb` files."** Refuted: the `.isb` payloads
+  contain no ASCII symbol names at all. They reference symbols purely by hash —
+  which is *why* the prescan link map exists.
+* **"The hash constants might be in the executable next to a name table."**
+  Refuted: none of the 42 values occurs in the `.pe` image in either endianness.
+* **"IXUD might hold more hash-keyed nameless fields."** Refuted by measurement:
+  534 IXUD objects, 624 488 field entries, **8** distinct unnamed tags — all of
+  them ordinals `0…7` (the weapon `CATEGORY_DESC` list). Zero hash-shaped.
+* **⚠️ "There are ~504 keys to crack."** Withdrawn — 504 is the *entry* count.
+  See above.
+
+## Coverage / limits
+
+- The census covers every IDXD object the disc has (7 750). `sound.pak` holds no
+  IDXD objects and `dat/movie/*.wmv` and `hidden/resource3d/*.xpr` are not
+  containers; strings from `resource3d` were nevertheless folded into the corpus
+  attack.
+- The 42 targets are fully characterised as *references*; their *spelling* is
+  unrecovered and, per the arithmetic above, unrecoverable from the disc.
+
+## Evidence log (append-only; newest last)
+
+- `2026-08-26` — Disc-wide walk of 7 750 IDXD objects / 2 757 039 field entries
+  with `tag_hash` self-check at 1 271 462/1 271 462. Census → confidence
+  `CONFIRMED` for the counts and for the ordinal/hash split.
+- `2026-08-26` — 42 hash-shaped keys localised to six copies of
+  `<lang>\script\ID.tbl`; cross-checked against the `.isb` bytecode, where 41 of
+  42 appear as little-endian call targets forming a consistent call graph →
+  `CONFIRMED` that they are ISL script-symbol hashes.
+- `2026-08-26` — Key-delta arithmetic pins the trailing digits of 30 of the 42
+  names → `CONFIRMED` (forced by the hash's polynomial structure, three
+  independent families agreeing).
+- `2026-08-26` — Seven corpus/composition/brute-force attacks, none producing a
+  hit above the noise floor; exhaustive search validated on a known name and
+  shown to return nothing at ≤ 6 characters → the names are **not recovered**,
+  and the negative result is quantified rather than asserted.
+
+## Open questions / what would raise confidence
+
+- **The 12 unconstrained names** (`function.isb` ×5, `function_tutorial.isb` ×6,
+  `main.isb` ×1) have no digit structure to exploit at all.
+- **What would actually work:** a build artefact containing ISL source or a symbol
+  list (none on the retail disc); or observing the game *construct* one of these
+  strings at runtime — `PrepareScript` builds the table from `.isb` contents, but
+  whatever `main.isb`'s root symbol is called, the name existed only in the
+  compiler's input. A `--mem-watch` on the ReadyRoom lookup path would show
+  whether any caller ever passes a literal name rather than a precomputed hash;
+  that is the one remaining honest route and it was **not** attempted here (this
+  was a static-only investigation).
+- **`OFFSET`'s unit** — base-36 is settled, what it counts is not.
+- The `.isb` instruction encoding is only sketched here (a length byte at
+  `byte[2]`, call payloads leading with the target hash). It is not the `.ssb`
+  ISL bytecode documented elsewhere and deserves its own note.