diff --git a/docs/re/BACKLOG.md b/docs/re/BACKLOG.md index de21c40..7d573af 100644 --- a/docs/re/BACKLOG.md +++ b/docs/re/BACKLOG.md @@ -827,6 +827,42 @@ unknown, what evidence exists, and what the first step would be. Move an item in field; the stat keys in records not opened here (`MegaTons`, `BulletLimit`, `CrewCount`, `HP_CLASS`); why the roster lists 59 of 131. +* โœ… **(2026-08-27) WHICH ARCHIVES THE DISC CAN NAME โ€” 100 % for eleven menu + paks, **0.0 %** for all six 2D paks AND `GP_READY_ROOM`.** + [structures/archive-naming](structures/archive-naming.md), artefact + `data/archive-naming.txt`, regenerator `tools/re-capture/archive_naming.py`. + Chasing more prefixes for the 419 HUD paths would have been the same mistake + twice, so this censuses the whole disc: harvest **every** plausible asset-name + string from **every** archive (**6 027**), hash each under the 16 known + prefixes, and ask per archive what fraction of its TOC that explains. + `idxd-container.md`/`idxd-tag-hash.md` own the hash; neither says which + archives are REACHABLE by it. + ๐Ÿ”‘ **The result is bimodal.** `GP_TITLE` 16/16, `GP_PAUSE_MENU` 11/11, + `GP_STAGE_CLEAR` 44/44, `GP_CHALLENGE` 151/151, `GP_MOVIE_THEATER` 56/56, + `MiscBin` 40/40, `GP_GAMEOVER`, `GP_BUNK`, `GP_SYSTEM`, `GP_TUTORIAL`, `fonts` + โ€” **100 %**; `tables.pak` 78/79, `GP_DIALOG` 139/140, the six language paks + 115/117, `GP_SAVE_LOAD`/`GP_LEADERBOARD`/`GP_OPTIONS`/`GP_MISSION_LOG` 91-94 %; + `GP_MAIN_GAME_*` 751/1119 = 67 %. Then the cliff: **the six + `GP_MAIN_GAME_*2D.pak` at 0 of 711 EACH**, and **`GP_READY_ROOM` at 6 of + 1 106 = 0.5 %** โ€” the largest UI pak on the disc, and not previously noted + anywhere in `docs/re/`. + ๐Ÿงช Eleven paks at 100 % in the same run is the control that makes 0.0 % a + finding rather than a failed guess: the method works; these archives are + outside it. + ๐Ÿ”‘ **So the 419 HUD paths are not "missing assets"** โ€” nothing in the 2D paks + is reachable by name from the disc's own strings at all. Those TOC keys hash + names that are **not written anywhere readable**: composed by the tool chain or + by the executable, or listed outside these archives. + โŒ Also refuted first: the 419 config values under **13 name transformations** + (as-is, forward slashes, basename, basename-no-ext, no-ext, UPPER, `.rat` + substituted, `2d\`/`eng\`/`hud\`/`GP_MAIN_GAME_2D\` prefixed, first-directory + stripped) โ€” **every one scored 0**, against the E2D 711 and against all 16 630 + entries on the disc. + ๐ŸŸก Not settled: what those names are. The one lever left is the hash's shape โ€” + the top byte is the character-sum checksum, so candidates are cheap to reject โ€” + but that needs a name corpus the disc does not contain. Also unexplained: + `DefTables` 8.9 % and `GP_HANGAR_ARSENAL` 22.6 % sitting in the middle. + * โœ… **(2026-08-27) THE WHOLE IN-GAME HUD CONFIG IS OPEN โ€” 16 records, 419 asset paths, a per-stage `ResourceTable` โ€” and NONE of it is name-hash addressed.** [structures/hud-config](structures/hud-config.md), artefact diff --git a/docs/re/INDEX.md b/docs/re/INDEX.md index c3aa52f..9466600 100644 --- a/docs/re/INDEX.md +++ b/docs/re/INDEX.md @@ -99,6 +99,7 @@ files, which is how the same ground got covered twice. | [`structures/sound-cue-table.md`](structures/sound-cue-table.md) | The cue index in `tables.pak` โ€” message id -> cue -> sound id -> `.slb` bank | โœ… CONFIRMED, 1 326/1 338 script message ids bind to a bank; SOUNDS and FILES agree on the same 12 absentees, 0 orphan files | | [`structures/cutscene-message-table.md`](structures/cutscene-message-table.md) | Cutscene dialogue โ€” speaker, portrait, on-screen seconds, audio cue per page | โœ… CONFIRMED, field count = 9ยทPageCount+2 for all 7 PageCounts, 1 252/1 252 caption keys match, 138 ids close both ways | | [`structures/isl-mission-timer.md`](structures/isl-mission-timer.md) | The mission timer โ€” two sequential countdowns, not a limit and a warning | โœ… CONFIRMED โ€” read from `sub_822639B8`; the threshold reading is | +| [`structures/archive-naming.md`](structures/archive-naming.md) | Which archives the disc can name โ€” 100% for menu paks, **0.0%** for the six 2D paks and `GP_READY_ROOM` | โœ… CONFIRMED โ€” 6027 names ร— 16 prefixes, bimodal | | [`structures/hud-config.md`](structures/hud-config.md) | The in-game HUD configuration โ€” 16 records, 419 asset paths, and a per-stage `ResourceTable` | โœ… CONFIRMED โ€” 0 of 419 config paths resolve as pak entries | | [`structures/mission-script-manifest.md`](structures/mission-script-manifest.md) | `Stage\script.tbl`'s 11 non-`MISSION` fields, and the mission dialogue table they name | โœ… CONFIRMED โ€” 1 of 7 values resolves (`message\MissionDialogMessage.tbl`, 200 records), controls in the same sweep | | [`structures/stage-numbering-and-player-craft.md`](structures/stage-numbering-and-player-craft.md) | The disc's stage numbering (S01โ€“S16, S18โ€“S29) and the player's craft per stage | โœ…โœ… CONFIRMED โ€” two independent files close on 6 + 15 + 1 + 6 = 28 | diff --git a/docs/re/data/archive-naming.txt b/docs/re/data/archive-naming.txt new file mode 100644 index 0000000..d9e4e78 --- /dev/null +++ b/docs/re/data/archive-naming.txt @@ -0,0 +1,49 @@ +# How much of each pak's TOC can be named from the disc's own strings +# Regenerate: python3 tools/re-capture/archive_naming.py +# See docs/re/structures/archive-naming.md + + candidate asset-name strings harvested: 6027 + path prefixes tried: 16 + + archive entries named pct + sound.pak 9519 1 0.0% + GP_HANGAR_ARSENAL.pak 1538 347 22.6% + DefTables.pak 1465 130 8.9% + GP_MAIN_GAME_D.pak 1119 751 67.1% + GP_MAIN_GAME_E.pak 1119 751 67.1% + GP_MAIN_GAME_F.pak 1119 751 67.1% + GP_MAIN_GAME_I.pak 1119 751 67.1% + GP_MAIN_GAME_J.pak 1119 751 67.1% + GP_MAIN_GAME_S.pak 1119 751 67.1% + GP_READY_ROOM.pak 1106 6 0.5% + GP_MAIN_GAME_D2D.pak 711 0 0.0% + GP_MAIN_GAME_E2D.pak 711 0 0.0% + GP_MAIN_GAME_F2D.pak 711 0 0.0% + GP_MAIN_GAME_I2D.pak 711 0 0.0% + GP_MAIN_GAME_J2D.pak 711 0 0.0% + GP_MAIN_GAME_S2D.pak 711 0 0.0% + GP_DEBRIEFING_PILOTLOG.pak 234 78 33.3% + GP_CHALLENGE.pak 151 151 100.0% + GP_DIALOG.pak 140 139 99.3% + deu.pak 117 115 98.3% + eng.pak 117 115 98.3% + esp.pak 117 115 98.3% + fra.pak 117 115 98.3% + ita.pak 117 115 98.3% + jpn.pak 117 115 98.3% + GP_SAVE_LOAD.pak 108 102 94.4% + GP_MISSION_SELECT.pak 88 22 25.0% + GP_LEADERBOARD.pak 87 81 93.1% + tables.pak 79 78 98.7% + GP_MOVIE_THEATER.pak 56 56 100.0% + GP_STAGE_CLEAR.pak 44 44 100.0% + MiscBin.pak 40 40 100.0% + GP_OPTIONS.pak 26 24 92.3% + GP_MISSION_LOG.pak 24 22 91.7% + GP_TITLE.pak 16 16 100.0% + GP_GAMEOVER.pak 12 12 100.0% + GP_PAUSE_MENU.pak 11 11 100.0% + GP_BUNK.pak 10 10 100.0% + fonts.pak 3 3 100.0% + GP_SYSTEM.pak 2 2 100.0% + GP_TUTORIAL.pak 2 2 100.0% diff --git a/docs/re/structures/archive-naming.md b/docs/re/structures/archive-naming.md new file mode 100644 index 0000000..e77eef2 --- /dev/null +++ b/docs/re/structures/archive-naming.md @@ -0,0 +1,66 @@ +# โœ… Which archives the disc can name โ€” and the two it cannot + +The previous pass found that **0 of 419** in-game HUD config paths hash to a pak +entry ([hud-config](hud-config.md)). Chasing more prefixes would have been the +same mistake twice, so this censuses the whole disc instead: harvest **every** +plausible asset-name string from **every** archive, hash each under the 16 known +path prefixes, and ask per archive **what fraction of its TOC that explains**. + +`idxd-container.md` and `idxd-tag-hash.md` own the hash (`name_hash` = a +checksum byte over a 24-bit modular value, case-insensitive, distinct from +`tag_hash`); neither says which archives are *reachable* by it. Artefact +`../data/archive-naming.txt`, regenerator `tools/re-capture/archive_naming.py` +(**6 027** candidate names ร— 16 prefixes). + +## The result is bimodal + +| archive | entries | named | | +|---|---:|---:|---| +| `GP_TITLE`, `GP_PAUSE_MENU`, `GP_STAGE_CLEAR`, `GP_CHALLENGE`, `GP_MOVIE_THEATER`, `GP_GAMEOVER`, `GP_BUNK`, `GP_SYSTEM`, `GP_TUTORIAL`, `MiscBin`, `fonts` | 2โ€“151 | **all** | **100 %** | +| `tables.pak` | 79 | 78 | 98.7 % | +| `GP_DIALOG` | 140 | 139 | 99.3 % | +| `deu/eng/esp/fra/ita/jpn.pak` | 117 | 115 | 98.3 % | +| `GP_SAVE_LOAD` / `GP_LEADERBOARD` / `GP_OPTIONS` / `GP_MISSION_LOG` | 24โ€“108 | | 91โ€“94 % | +| `GP_MAIN_GAME_{D,E,F,I,J,S}` | 1 119 | 751 | 67.1 % | +| `GP_DEBRIEFING_PILOTLOG` | 234 | 78 | 33.3 % | +| `GP_HANGAR_ARSENAL` | 1 538 | 347 | 22.6 % | +| `DefTables` | 1 465 | 130 | 8.9 % | +| `GP_MISSION_SELECT` | 88 | 22 | 25.0 % | +| **`GP_READY_ROOM`** | **1 106** | **6** | **0.5 %** | +| **`GP_MAIN_GAME_{D,E,F,I,J,S}2D`** | **711** each | **0** | **0.0 %** | +| `sound.pak` | 9 519 | 1 | 0.0 % (a separate format โ€” [sound-pak-contents](sound-pak-contents.md)) | + +๐Ÿ”‘ **The six 2D paks are at exactly 0.0 %, all six, 711 entries each** โ€” while +eleven menu paks are at 100 % by the same method in the same run. That is the +control that makes it a finding rather than a failed guess: the method works, and +these archives are outside it. + +๐Ÿ”‘ **`GP_READY_ROOM` is a second such archive** โ€” 1 106 entries, **6** named. +Not previously noted anywhere in `docs/re/`; it is the largest UI pak on the +disc and its names are as unreachable as the 2D paks'. + +## What that means for the 419 + +The in-game HUD's asset paths are not "missing". **Nothing in the 2D paks is +reachable by name from the disc's own strings**, whatever the extension. So the +2D and READY_ROOM TOC keys are hashes of names that are **not written anywhere +we can read** โ€” built by the tool chain, or from a string the executable +composes, or from a name list held outside these archives. + +## ๐Ÿงช Also refuted here โ€” 13 name transformations + +Before the census, the 419 config values were re-hashed under 13 rewrites: +as-is, forward slashes, basename, basename without extension, without extension, +uppercased, `.rat` substituted, `2d\`/`eng\`/`hud\`/`GP_MAIN_GAME_2D\` prefixed, +and first-directory stripped. **Every one scored 0, both against the E2D pak's +711 entries and against all 16 630 entries on the disc.** + +## ๐ŸŸก Not settled + +* What names the 2D and `GP_READY_ROOM` TOCs actually hash. The one lever left is + the hash's own shape โ€” the top byte is the character-sum checksum, so any + candidate is cheap to reject โ€” but that needs a name corpus this disc does not + contain. +* Why `DefTables` (8.9 %) and `GP_HANGAR_ARSENAL` (22.6 %) sit in the middle; + probably just that most of their entries are tables referenced by records this + harvest does not treat as filenames. Not chased. diff --git a/tools/re-capture/archive_naming.py b/tools/re-capture/archive_naming.py new file mode 100644 index 0000000..40c5bc3 --- /dev/null +++ b/tools/re-capture/archive_naming.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +"""How much of each pak's TOC can be named from the asset strings on the disc. + +Harvests every plausible asset-name string from every archive, hashes each under +the known path prefixes, and reports per archive how many of its entries are +explained. Regenerates docs/re/data/archive-naming.txt. +""" +import sys, os, glob, re, collections + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import unitgroup as U +from unit_substructures import pak_entries + +EXT = r'(prt|t32|rat|tbl|xpr|ttc|ttf|TTC|TTF|bin|col|rgn|prm)' +PRE = ['', '2d\\', 'ui\\', 'hud\\', 'Stage\\', 'stage\\', 'message\\', 'language\\', + 'prt\\', 'view\\'] + [l + '\\' for l in ('eng', 'jpn', 'fra', 'deu', 'ita', 'esp')] + +def main(): + paks = sorted(glob.glob('/work/sylph_extract/**/*.pak', recursive=True)) + names = set() + toc = {} + for pk in paks: + ents = pak_entries(pk) + toc[os.path.basename(pk)] = [h for h, b in ents] + for h, b in ents: + for m in re.finditer((r'[A-Za-z0-9_\\.\-]{4,60}\.' + EXT).encode(), b): + names.add(m.group(0).decode('latin-1')) + + H = set() + for n in names: + for p in PRE: + H.add(U.name_hash(p + n)) + + print("# How much of each pak's TOC can be named from the disc's own strings") + print("# Regenerate: python3 tools/re-capture/archive_naming.py") + print("# See docs/re/structures/archive-naming.md") + print("\n candidate asset-name strings harvested: %d" % len(names)) + print(" path prefixes tried: %d" % len(PRE)) + print("\n archive entries named pct") + rows = sorted(((k, v) for k, v in toc.items()), key=lambda r: -len(r[1])) + for nm, ent in rows: + nd = sum(1 for h in ent if h in H) + print(" %-30s %6d %6d %5.1f%%" % (nm, len(ent), nd, 100.0 * nd / len(ent) if ent else 0)) + +if __name__ == "__main__": + main()