From 53084a11cae363581dc937599379da6159eaf174 Mon Sep 17 00:00:00 2001 From: "Claude (auto)" Date: Thu, 27 Aug 2026 19:59:20 +0000 Subject: [PATCH] re: the Enumerate declaration tables close DefTables (130 -> 1413 of 1425 IDXD) The string harvest could never name DefTables because those names are not spelled out as "something.tbl" anywhere on the disc -- they are declared. An IDXD object whose single record is named Enumerate is a DECLARATION TABLE: its field NAMES are the names of other objects, each resolving as name_hash(".tbl"). EnumLODSet_test.tbl declares 676, EnumGameModel_test.tbl 360; the disc holds 144 such objects (138 in DefTables, one in each GP_MAIN_GAME_*) declaring 1298 distinct names. Route 2 names +1283 entries route 1 could not. 130 + 1283 + 12 = 1425, no overlap, 99.2 % coverage. Zero partials: of 5 suffixes x 6 prefixes, ('', .tbl) scored 1036/1036 and every other combination scored 0. Residual in full: 8 declaration tables nothing declares, 2 LOD sets (Model rou_e004 / rou_e013), 2 motion sets; 15 declared names have no pak entry at all. REFUTED alongside it: the 40 XPR2 manifests are not the naming source -- their 82 Name= values and 82 DataFile/Source paths resolve 0 entries under any of the 5 suffixes. They share the MODEL namespace only: 40 of the 82 appear as the Model field value inside the tables. Artefact +17/-18, every line paired, byte-identical across two runs. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE --- docs/re/BACKLOG.md | 58 +++++++++++++++++++++++++++ docs/re/INDEX.md | 2 +- docs/re/data/archive-naming.txt | 35 ++++++++-------- docs/re/structures/archive-naming.md | 60 ++++++++++++++++++++++++++++ tools/re-capture/archive_naming.py | 37 +++++++++++++++++ 5 files changed, 173 insertions(+), 19 deletions(-) diff --git a/docs/re/BACKLOG.md b/docs/re/BACKLOG.md index c4f1282..18452e0 100644 --- a/docs/re/BACKLOG.md +++ b/docs/re/BACKLOG.md @@ -4793,3 +4793,61 @@ manifests name meshes like `machines\rou_e104\objects\…xbg` — whether those names hash to the LOD/motion table keys is untested). The 337 unnamed IDXD in each `GP_MAIN_GAME_*` are also uncharacterised — the same census would say what they are, and was not run this iteration. + +--- + +## ✅✅ 2026-08-27 — the `Enumerate` declaration tables close `DefTables` (99.2 %) + +Item taken: "do the XPR2 manifest model names hash to the `DefTables` LOD/motion +tables?" — **refuted**, and the real naming source found next to it. + +**Docs checked first:** `archive-naming.md` (mine); `isl-condition-builtins.md` +owns the 40 XPR2 manifests; `stage-definition-table.md` / `stage-mission-tables.md` +own the names `EnumLODSet_test.tbl` / `EnumGameModel_test.tbl` and record that +they live in `DefTables.pak` — but **neither says what is inside them**. + +🔴 **REFUTED — the XPR2 manifests are not a naming source.** Their 82 `Name=` +values and 82 `DataFile`/`Source` paths, hashed bare and under +`.tbl`/`.xbg`/`.xpr`/`.prt`/`game:\`, resolve **0** `DefTables` entries. They do +share the *model* namespace: 40 of the 82 appear as the `Model` field VALUE +inside the LOD/motion tables (1 272 distinct `Model` values). ⚠️ The first +attempt regexed `Name="` and found 0 values — the manifests write `Name = "…"` +with spaces; a regex miss looks exactly like a null result. + +✅✅ **An IDXD object whose single record is named `Enumerate` is a DECLARATION +TABLE: its FIELD NAMES are the names of other objects**, each resolving as +`name_hash(".tbl")`. `EnumLODSet_test.tbl` declares 676 names, +`EnumGameModel_test.tbl` 360; the disc holds **144** such objects (138 in +`DefTables`, one in each `GP_MAIN_GAME_*`) declaring **1 298** distinct names. + +| | | +|---|---:| +| `DefTables` named by the string harvest (route 1) | 130 | +| **named by the declaration tables (route 2)** | **+1 283** | +| still unnamed | **12** | +| **coverage** | **1 413 / 1 425 IDXD = 99.2 %** | + +`130 + 1283 + 12 = 1425`, no overlap between routes. ✅ **Zero partials and the +suffix is exact**: of 5 suffixes × 6 prefixes, `('', '.tbl')` scored 1 036/1 036 +on the `_test` declarations and every other combination scored 0. + +**Residual, in full:** 8 declaration tables nothing declares (the other 130 are +named by route 1, from the stage tables that reference them), 2 +`Generic+Level_0` LOD sets whose `Model` reads `rou_e004`/`rou_e013`, and 2 +motion sets. **15 declared names have no pak entry** — content that did not ship. + +`archive_naming.py` extended with route 2; artefact **+17/−18**, every line +paired (the DefTables census row 130→1413 and its shape list collapsing 17→3). +Byte-identical across two runs. + +🟡 Not settled: **why those 12 are unreachable** — no attempt was made to name +the 8 orphan declaration tables or the 4 stragglers. And the six +`GP_MAIN_GAME_*` `Enumerate` objects named **nothing** there; what they declare +and where it lives was not chased. + +❔ Handed forward: with `DefTables` closed, the largest unnamed block is the +identical **337 IDXD in each `GP_MAIN_GAME_*`**, 53 shapes led by `Shell+Weapon` +(90), `Faces+Generic` (64), the unit shape +`Effect+Explosion+Generic+Maneuver+Mass+SE+Shield+StructureCount` (44) and a +chatter table (8) — weapon/unit/chatter families the corpus already documents +under other names. Characterising them is the next item. diff --git a/docs/re/INDEX.md b/docs/re/INDEX.md index 4d62c47..592554d 100644 --- a/docs/re/INDEX.md +++ b/docs/re/INDEX.md @@ -102,7 +102,7 @@ files, which is how the same ground got covered twice. | [`structures/mission-scoring.md`](structures/mission-scoring.md) | Mission scoring and the S/A/B/C/D rank thresholds — 24 tables x 3 difficulties | ✅ CONFIRMED, one 22-field schema over 72 records; difficulty moves 10 of 22 fields and never the RankScore thresholds; ✅ all 24 named stage\StageParameter_S.tbl (+ _Tutorial shared by six, + _Test) | | [`structures/stage-settings-table.md`](structures/stage-settings-table.md) | The per-stage settings object — Camera rigs, Player limits, Difficulty_* damage record | ✅ CONFIRMED, 24 settings objects vs 29 resource objects both using Phase_1/2/3; Camera is 13/14 fields constant; GravityFactor non-zero in 4 stages; IsBoss16Enable identifies one object; the Phase block is 94 fields of which 68 never vary between phases; all 24 objects named StageParameter_S | | [`structures/isl-mission-timer.md`](structures/isl-mission-timer.md) | The mission timer — two sequential countdowns, not a limit and a warning | ✅ CONFIRMED — read from `sub_822639B8`; the threshold reading is | -| [`structures/archive-naming.md`](structures/archive-naming.md) | Which archives the disc can name — 100% for menu paks, **0.0%** for the six 2D paks and `GP_READY_ROOM` | ✅ CONFIRMED — 6027 names × 16 prefixes, bimodal; unnamed entries split by content — three UI paks have ZERO unnamed IDXD (artwork only), DefTables is the only real data gap (1295 LOD/motion tables) | +| [`structures/archive-naming.md`](structures/archive-naming.md) | Which archives the disc can name — 100% for menu paks, **0.0%** for the six 2D paks and `GP_READY_ROOM` | ✅ CONFIRMED — 6027 names × 16 prefixes, bimodal; unnamed entries split by content — three UI paks have ZERO unnamed IDXD (artwork only), DefTables is the only real data gap (1295 LOD/motion tables); **route 2 = the `Enumerate` declaration tables** closes DefTables to 1413/1425 IDXD (99.2 %), 12 left; XPR2 manifests refuted as a naming source | | [`structures/hud-config.md`](structures/hud-config.md) | The in-game HUD configuration — 16 records, 419 asset paths, and a per-stage `ResourceTable` | ✅ CONFIRMED — 0 of 419 config paths resolve as pak entries | | [`structures/mission-script-manifest.md`](structures/mission-script-manifest.md) | `Stage\script.tbl`'s 11 non-`MISSION` fields, and the mission dialogue table they name | ✅ CONFIRMED — 1 of 7 values resolves (`message\MissionDialogMessage.tbl`, 200 records), controls in the same sweep | | [`structures/stage-numbering-and-player-craft.md`](structures/stage-numbering-and-player-craft.md) | The disc's stage numbering (S01–S16, S18–S29) and the player's craft per stage | ✅✅ CONFIRMED — two independent files close on 6 + 15 + 1 + 6 = 28 | diff --git a/docs/re/data/archive-naming.txt b/docs/re/data/archive-naming.txt index e2def5a..63350c9 100644 --- a/docs/re/data/archive-naming.txt +++ b/docs/re/data/archive-naming.txt @@ -1752,10 +1752,23 @@ GP_TUTORIAL.pak 2 named, 1 families x2 puscr.prt +## ROUTE 2 the `Enumerate` declaration tables + declaration objects found: 144 distinct names they declare: 1298 + x138 DefTables.pak + x1 GP_MAIN_GAME_D.pak + x1 GP_MAIN_GAME_E.pak + x1 GP_MAIN_GAME_F.pak + x1 GP_MAIN_GAME_I.pak + x1 GP_MAIN_GAME_J.pak + x1 GP_MAIN_GAME_S.pak + DefTables.pak +1283 newly named + entries named by route 2 that route 1 could not: 1283 + declared names with NO pak entry: 15 ['GameModel_eff_e0033.tbl', 'GameModel_eff_e0058.tbl', 'GameModel_eff_e0059.tbl', 'GameModel_eff_e0060.tbl', 'GameModel_rou_e106_wep_02.tbl', 'LOD_Effect_eff_e0033.tbl'] + ## What the UNNAMED entries are (magic of every entry not resolved above) archive IDXD named unnamedI unnamedUI LSTA GP_HANGAR_ARSENAL.pak 180 180 0 1149 42 - DefTables.pak 1425 130 1295 0 0 + DefTables.pak 1425 1413 12 0 0 GP_MAIN_GAME_D.pak 1004 667 337 0 0 GP_MAIN_GAME_E.pak 1004 667 337 0 0 GP_MAIN_GAME_F.pak 1004 667 337 0 0 @@ -1779,24 +1792,10 @@ GP_MISSION_LOG.pak 0 0 0 0 2 record-name sets of the UNNAMED IDXD entries, per archive: - DefTables.pak 1295 unnamed IDXD in 17 shapes - x294 Generic Level_0 Level_1 - x247 Generic Level_0 - x212 Default EnumMotions Generic ReferenceFrames - x209 Generic Level_0 Level_1 Level_2 Level_3 - x182 Default EnumMotions Generic Motion_break Motion_dead ReferenceFrames - x61 Default EnumMotions Generic Motion_break ReferenceFrames - x57 Generic Level_0 Level_1 Level_2 - x8 Default EnumMotions Generic Motion_break Motion_down ReferenceFrames - x8 Default EnumMotions Generic Motion_break Motion_selected ReferenceFrames + DefTables.pak 12 unnamed IDXD in 3 shapes x8 Enumerate - x3 Default EnumMotions Generic Motion_FlipTurn Motion_ShieldAttack Motion_SideRoll_L Motion_SideRol - x1 Damage Damage2 Default EnumMotions Generic ReferenceFrames - x1 Default Default2 Default3 EnumMotions Generic ReferenceFrames - x1 Default EnumMotions Generic Motion_AfterBurner Motion_break ReferenceFrames - x1 Default EnumMotions Generic Motion_dead ReferenceFrames - x1 Default EnumMotions Generic Motion_down ReferenceFrames - x1 EnumMotions Generic Motion_attackA_charge Motion_attackA_charge_b1 Motion_attackA_charge_b2 Moti + x2 Default EnumMotions Generic Motion_break Motion_dead ReferenceFrames + x2 Generic Level_0 GP_MAIN_GAME_D.pak 337 unnamed IDXD in 53 shapes x90 Shell Weapon x64 Faces Generic diff --git a/docs/re/structures/archive-naming.md b/docs/re/structures/archive-naming.md index aee8757..ba120cd 100644 --- a/docs/re/structures/archive-naming.md +++ b/docs/re/structures/archive-naming.md @@ -67,6 +67,66 @@ redo the work. is now iterated **sorted** — a plain `set` gave a different winner for collided hashes on each run and the artefact failed its own byte-identical check. +## ✅✅ ROUTE 2 — the `Enumerate` declaration tables close `DefTables` (2026-08-27) + +The string harvest could never name `DefTables`, because **those names are not +spelled out as `something.tbl` anywhere on the disc**. They are declared instead. + +🔑 **An IDXD object whose single record is named `Enumerate` is a DECLARATION +TABLE: its FIELD NAMES are the names of other objects, and each one resolves as +`name_hash(".tbl")`.** `EnumLODSet_test.tbl` declares 676 names, +`EnumGameModel_test.tbl` 360 — and the disc holds **144** such objects (138 in +`DefTables`, one in each of the six `GP_MAIN_GAME_*`), declaring **1 298** +distinct names between them. + +| | | +|---|---:| +| entries `DefTables` named by the string harvest (route 1) | 130 | +| **entries named by the declaration tables (route 2)** | **+1 283** | +| still unnamed | **12** | +| **`DefTables` IDXD coverage** | **1 413 / 1 425 = 99.2 %** | + +`130 + 1283 + 12 = 1425` — the arithmetic closes, with **no overlap** between +the two routes. + +✅ **Zero partials, and the suffix is exact.** Every one of the 1 036 +`_test`-declared names resolves, and it resolves **only** as `.tbl` with +**no** path prefix — of the 5 suffixes × 6 prefixes tried, `('', '.tbl')` scored +1 036 and every other combination scored 0. + +**What the residual is**, in full: 8 declaration tables that nothing declares (the +other 130 are named by route 1, from the stage tables that reference them), plus +**4** ordinary tables — two `Generic+Level_0` LOD sets whose `Model` fields read +`rou_e004` and `rou_e013`, and two motion sets. **15 declared names have no pak +entry** (`GameModel_eff_e0033`, `…e0058`–`e0060`, `rou_e106_wep_02`, and their +`LOD_Effect_*` twins) — declarations for content that did not ship. + +### 🔴 The XPR2 manifests are NOT the naming source — refuted + +The obvious guess was the 40 XPR2 manifests, which name meshes +(``). +Their **82** `Name=` values and 82 `DataFile`/`Source` paths, hashed bare and +under `.tbl` / `.xbg` / `.xpr` / `.prt` / `game:\`, resolve **0** `DefTables` +entries. + +They are not unrelated, though: **40 of the 82 manifest names appear as the +`Model` field VALUE inside the LOD/motion tables** (1 272 distinct `Model` +values in `DefTables`). Manifest and table share the *model* namespace; the pak +key is a different namespace, and only the declaration tables bridge it. + +⚠️ First attempt used `Name="` and found **0 values** — the manifests write +`Name = "…"` with spaces. A regex miss looks exactly like a null result. + +### ⚠️ Bonus, not this item: the `GP_MAIN_GAME_*` residual is already-owned data + +With `DefTables` closed, the largest remaining unnamed block is the identical +**337 IDXD in each of the six `GP_MAIN_GAME_*`**, in 53 record-name shapes whose +top entries are `Shell+Weapon` (90), `Faces+Generic` (64), +`Effect+Explosion+Generic+Maneuver+Mass+SE+Shield+StructureCount` (44) and a +chatter table (`ATTACK_IS_AVOIDED …`, 8) — i.e. **weapon, unit and chatter +tables**, all families the corpus already documents. Characterising them +properly is still open. + ## ✅ WHY each archive is low-coverage — two different reasons (2026-08-27) Splitting every **unnamed** entry by its magic turns the percentage table into an diff --git a/tools/re-capture/archive_naming.py b/tools/re-capture/archive_naming.py index d347c81..2570939 100644 --- a/tools/re-capture/archive_naming.py +++ b/tools/re-capture/archive_naming.py @@ -19,6 +19,8 @@ PRE = ['', '2d\\', 'ui\\', 'hud\\', 'Stage\\', 'stage\\', 'message\\', 'language def main(): paks = sorted(glob.glob('/work/sylph_extract/**/*.pak', recursive=True)) names = set() + decl = set() + nenum = collections.Counter() toc = {} for pk in paks: ents = pak_entries(pk) @@ -26,6 +28,18 @@ def main(): for h, b in ents: for m in re.finditer((r'[A-Za-z0-9_\\.\-]{4,60}\.' + EXT).encode(), b): names.add(m.group(0).decode('latin-1')) + # ROUTE 2 seed: an object whose single record is named `Enumerate` + # is a DECLARATION TABLE — its field names are other objects' names. + if b[:4] == b'IDXD' and b'Enumerate' in b: + try: + recs = U.parse(b) + except Exception: + continue + if len(recs) == 1 and recs[0]['squadron'] == 'Enumerate': + nenum[os.path.basename(pk)] += 1 + for _t, fn, _v in recs[0]['fields']: + if fn: + decl.add(fn) H = set() for n in names: @@ -62,6 +76,29 @@ def main(): for k, c in sorted(fam.items(), key=lambda kv: (-kv[1], kv[0])): print(" x%-4d %s" % (c, k)) + # ROUTE 2 -- the declaration tables. An `Enumerate` object lists, as FIELD + # NAMES, the names of other objects; each resolves as `.tbl`. This is + # the naming source the string harvest cannot see, because these names are + # never spelled out as `something.tbl` anywhere on the disc. + DECL = {} + for n in sorted(decl): + DECL.setdefault(U.name_hash(n + '.tbl'), n + '.tbl') + print("\n## ROUTE 2 the `Enumerate` declaration tables") + print(" declaration objects found: %d distinct names they declare: %d" + % (sum(nenum.values()), len(decl))) + for nm, c in sorted(nenum.items(), key=lambda kv: (-kv[1], kv[0])): + print(" x%-5d %s" % (c, nm)) + gained = 0 + for nm, ent in rows: + g = sum(1 for h in ent if h in DECL and h not in NAMES) + if g: + print(" %-30s +%d newly named" % (nm, g)) + gained += g + print(" entries named by route 2 that route 1 could not: %d" % gained) + miss = [v for k, v in sorted(DECL.items()) if not any(k in ent for ent in toc.values())] + print(" declared names with NO pak entry: %d %s" % (len(miss), sorted(miss)[:6])) + NAMES.update({k: v for k, v in DECL.items() if k not in NAMES}) + # WHY an archive is low-coverage: split its UNNAMED entries by content. print("\n## What the UNNAMED entries are (magic of every entry not resolved above)") print(" %-28s %6s %6s %8s %9s %6s" % (