Merge branch 'auto/idxd-unnamed-keys'
# Conflicts: # docs/re/INDEX.md
This commit is contained in:
Sylpheed RE agent
@@ -12,7 +12,12 @@ Promote to a prose `structures/…md` file when a format needs behavioural notes
|
||||
|--------|-------|-----------------------|-------|
|
||||
| IPFB `.pak` archive | ✅ | `sylpheed-formats/src/pak.rs` + `tests/pak_idxd_disc.rs` | header + 12-byte TOC, Z1/zlib payloads |
|
||||
| name-hash (TOC keys) | ✅ | `sylpheed-formats/src/hash.rs` | Barrett-reduction hash; recovers original paths |
|
||||
<<<<<<< HEAD
|
||||
| IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` + `tests/idxd_records_disc.rs` ([container](structures/idxd-container.md)) | **The binary record/index region in front of the string pool is DECODED** (2026-08-25), closing the parser's long-standing "not yet decoded" note. Uniform 16-byte records `{name_hash, name_off, field_begin, field_end}` sorted by hash and binary-searched, then a field count, 12-byte fields `{key, name_off, value_off}` sorted by key, a pool size, and the string pool; the trailing `pool_size == file_len - pool_base` identity makes the layout self-checking. Verified over the **whole disc** with **zero** failures: 7 750/7 750 objects, 190 782/190 782 records reproducing their stored `tag_hash`, 1 271 462/1 271 462 named fields reproducing their key — and `IXUD` is the same container with `ixud_hash`, UTF-16BE and all offsets in **chars** (1 104/1 104 objects, 628 165/628 165 fields). **Field names are stored on disc** — a field's middle word points at its own name — so nothing needs preimage search except the **504** field entries disc-wide that are hash-keyed with no name — which are only **42 distinct keys**, each in 12 places (the page for these was never written — the finding is in this row), and are provably unrecoverable from the hash alone; the other 1 485 073 nameless fields are *positional*, keyed by a literal integer (line slots, movie ids). ⚠️ **Two long-held beliefs WITHDRAWN**: the word at `0x08` is **not a schema hash**, it is record 0's `name_hash` (7 750/7 750) — the header has no type field at all, so an object's kind is known only from the caller that loads it; and the field's middle word is **not** an always-`0xFFFFFFFF` flags word. The first was caught by a test asserting that every movie id names a real record: `1005 -> STAGE10_PHASE01` failed because `tag_hash("STAGE10_PHASE01")` **is** `0x067025B9`, that table's supposed schema id. 🟡 the legacy value-before-key string-pool reader is now known to be an *approximation* of the real table, and every number derived from it is re-checkable but not yet re-checked |
|
||||
=======
|
||||
| IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` | self-describing; ship/weapon stats verified vs known values |
|
||||
| IDXD nameless field keys | ✅/❌ | [idxd-unnamed-keys](structures/idxd-unnamed-keys.md) + [`tools/re-capture/idxd_unnamed_keys.py`](../../tools/re-capture/idxd_unnamed_keys.py) | Census of every field entry whose `name_off` is `0xFFFFFFFF`, disc-wide: **7 750 objects, 2 757 039 field entries, 0 parse failures**, `tag_hash` reproducing **1 271 462/1 271 462** named keys. **7 094 distinct keys are never named — and 7 052 of them are not hashes at all**, but author-assigned element ids (equal to the field's own index in 1 404 924 of 1 485 577 cases; `tag_hash("BGM_001")` is `0xC662435B` while the key valued `BGM_001.slb` is `0x000003E9`). ⚠️ **The "504 hash-keyed nameless fields" figure is 504 ENTRIES, not 504 names** — 42 distinct keys × 6 language copies × 2 records. All 42 are **ISL script-symbol hashes** in `<lang>\script\ID.tbl` (GP_READY_ROOM.pak), the link map built by `PrepareScript`'s "isl script prescanning"; 41 of 42 appear as little-endian call targets inside the `.isb` bytecode, forming a coherent launcher/helper call graph. The hash's own algebra pins the **trailing digits of 30 of the 42 names** (deltas of exactly `+0x01000001` across `stage01..09`, `stage10..16`, `challenge01..06`; `+0x01010000` across `tutorial0101..0601`). ❌ **No name was cracked, and the negative is quantified**: seven attacks up to a 3.5×10⁸ composition space found nothing above the noise floor; exhaustive preimage search recovers `"Stage01"` from its own hash but returns nothing for the real targets at ≤6 characters, and at 7 characters one target already has **1 176** preimages — a 24-bit modulus cannot name an 8+ character identifier uniquely |
|
||||
>>>>>>> auto/idxd-unnamed-keys
|
||||
| XPR2 texture + cubemap | 🟡/✅ | `sylpheed-formats/src/texture.rs` + [colour check](xpr2-colour-check.md) | de-tile + A8R8G8B8 and DXT1. **Channel order ✅ confirmed against the running game**: the Delta Saber's decoded atlas is orange-dominant (median saturated hue 23.3°, *zero* cool pixels) and the game renders the same hull at 9.3° — a red↔blue swap would sit at ≈200°. Exact fidelity (gamma/sRGB curve, premultiplied alpha, per-channel scale) is 🟡 untested, since a hue comparison cannot see it; cubemap face ordering ❔ |
|
||||
| T8aD 2D texture | ✅ | `sylpheed-formats/src/t8ad.rs` | **100 % of the disc decodes** (19 216/19 216, measured). The "~15 % deferred variants" were a wrong model, not a variant: a surface is a list of **arbitrary sub-rectangles**, each with a 16-byte header of `dst X, dst Y, width, height`, not a 256×256 grid — `0x1c` is the **rectangle count**. Uncovered area stays transparent. **Colours ✅ CONFIRMED** ([k8888](structures/texture-color-k8888.md)) |
|
||||
| RATC bundle | ✅ | `sylpheed-formats/src/ratc.rs` | child listing confirmed. **"One level deep" is not a limitation — there is nothing deeper**: 2 859 bundles hold 18 002 children at depth 1 and **0 at depth 2**, with no parse failures. Nested RATC blobs are **leaf records that reference siblings by name** (`opt `, the sprite name): 3 311 leaves, all embedding sibling names, **10 144 of 10 148 references resolve**. The 4 that do not are one dangling asset — `pmbase.rat` → `pmbase.t32` in `GP_STAGE_CLEAR.pak`'s four language builds, and `pmbase.t32` is **on the disc nowhere** |
|
||||
|
||||
7549
docs/re/data/idxd-unnamed-field-keys.txt
Normal file
7549
docs/re/data/idxd-unnamed-field-keys.txt
Normal file
File diff suppressed because it is too large
Load Diff
253
docs/re/structures/idxd-unnamed-keys.md
Normal file
253
docs/re/structures/idxd-unnamed-keys.md
Normal file
@@ -0,0 +1,253 @@
|
||||
# IDXD field keys that carry no name
|
||||
|
||||
- **Confidence:** ✅ the census and the identification of the hash-keyed set ·
|
||||
✅ the last characters of 30 of the 42 names · ❌ **no name was recovered**
|
||||
- **Tool:** `tools/re-capture/idxd_unnamed_keys.py` (`census` / `idtbl` / `crack` / `selftest`)
|
||||
- **Data:** [`docs/re/data/idxd-unnamed-field-keys.txt`](../data/idxd-unnamed-field-keys.txt)
|
||||
- **Seen in:** every `dat/*.pak` + `hidden/*.pak` on the disc; the interesting 42
|
||||
are in `dat/GP_READY_ROOM.pak`, file `<lang>\script\ID.tbl`
|
||||
- **Depends on:** the IDXD record/field table and `tag_hash`, which live on branch
|
||||
`auto/re-idxd-container` — **not on `main`**, so this note transcribes what it
|
||||
needs and the tool carries its own copy of `tag_hash` rather than importing one.
|
||||
|
||||
An IDXD field entry is `(key, name_off, value_off)`. Usually `name_off` points at
|
||||
the field's own name in the string pool and `key == tag_hash(name)`. When
|
||||
`name_off` is `0xFFFFFFFF` the pool holds no name and the reader has to know what
|
||||
`key` means from somewhere else. This note answers: how many such keys are there,
|
||||
what are they, and can the names be inverted out of the hash?
|
||||
|
||||
**Short answer: 7 094 distinct keys, of which 7 052 are not hashes at all and 42
|
||||
are. None of the 42 names was recovered — but they are now identified, their call
|
||||
graph is reconstructed, and 30 of them have their last two-to-four characters
|
||||
pinned algebraically.**
|
||||
|
||||
## ✅ The census
|
||||
|
||||
Measured by walking all 33 `dat/*.pak` plus `hidden/DefTables.pak` and
|
||||
`hidden/MiscBin.pak` (`.pNN` segments joined, `Z1` payloads inflated):
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| IDXD objects / records / field entries | 7 750 / 190 782 / **2 757 039** |
|
||||
| named field entries | 1 271 462 |
|
||||
| unnamed field entries (`name_off == 0xFFFFFFFF`) | 1 485 577 |
|
||||
| distinct keys named *somewhere* on disc | 12 174 |
|
||||
| **distinct keys never named anywhere** | **7 094** |
|
||||
| … below `0x00010000` | 7 052 |
|
||||
| … in `[0x10000, 0x01000000)` | **0** |
|
||||
| … at or above `0x01000000` | **42** |
|
||||
|
||||
Parse failures: 0. `tag_hash` reproduces the key of **1 271 462 / 1 271 462**
|
||||
named field entries disc-wide (`idxd_unnamed_keys.py selftest`), which is the
|
||||
validation gate for everything below — far past the 200-pair bar this
|
||||
investigation was asked to clear.
|
||||
|
||||
### ✅ "504 hash-keyed fields disc-wide" is 504 **entries**, not 504 keys
|
||||
|
||||
`INDEX.md` (on `auto/re-idxd-container`) records *504 fields disc-wide that are
|
||||
hash-keyed with no name*. That number is exactly reproduced here — and it counts
|
||||
**field entries**, not distinct keys:
|
||||
|
||||
```
|
||||
42 distinct keys x 6 language copies of the same object x 2 records = 504
|
||||
```
|
||||
|
||||
The three unnamed populations add up:
|
||||
`504` (hash-keyed) `+ 1 380 324` (ordinal) `+ 104 749` (key `0x00000000`, i.e.
|
||||
ordinal 0, held apart only because `tag_hash("") == 0` makes it look "named"
|
||||
somewhere) `= 1 485 577`. So the brute-force target was never 504 names; it was
|
||||
**42**.
|
||||
|
||||
### ✅ The gap at `[0x10000, 0x01000000)` is what separates hashes from ordinals
|
||||
|
||||
`tag_hash` puts the byte-sum checksum of the name in the **top byte**. A real
|
||||
hash therefore falls below `0x01000000` only when the name's byte sum is `0 mod
|
||||
256` — about 1 name in 256 — *and* its low 24 bits are also tiny. Of the 12 174
|
||||
keys that do carry a name, 75 have a zero top byte, and the smallest of those is
|
||||
`0x0002677C`; the never-named ordinal band tops out at `0x2198`. There is a clean
|
||||
three-order-of-magnitude gap and **not one key in it**. The split is measured,
|
||||
not assumed.
|
||||
|
||||
## ✅ The 7 052 ordinals are not hashes and never were
|
||||
|
||||
They are author-assigned element ids written straight into the tag slot. For
|
||||
1 404 924 of the 1 485 577 unnamed entries the key **equals the field's index
|
||||
within its record**; a further 42 750 are index + 1. The rest are hand-numbered
|
||||
with deliberate gaps, e.g. the hangar `WEAPONS` record runs
|
||||
`…0x30 0x31 … 0x35`, jumps to `0x50 … 0x55`, then `0x61` — an id space, not a
|
||||
counter. `tables.pak`'s `FILES` record numbers 5 135 resources from `0x3E9`
|
||||
(1001) upwards.
|
||||
|
||||
**Do not feed these to a reverse lookup.** Confirmed by the strongest possible
|
||||
test: `tag_hash("BGM_001") = 0xC662435B`, while the key of the field whose value
|
||||
is `BGM_001.slb` is `0x000003E9`. Zero of 5 135 `FILES` keys match a hash of
|
||||
their own value under any stem/case/extension variant.
|
||||
|
||||
Full per-key listing: Part 3 of the data file. Per-schema value profiles: Part 2.
|
||||
|
||||
## ✅ What the 42 hash-shaped keys are
|
||||
|
||||
All 42 live in **six byte-identical copies** of one object — `eng jpn fra deu ita
|
||||
esp \script\ID.tbl` inside `dat/GP_READY_ROOM.pak` (resolved through the IPFB TOC
|
||||
by `name_hash("eng\\script\\ID.tbl") = 0xC40BC120`, and pointed at by the
|
||||
`SCRIPT_PATH` field of `tables.pak`'s `BASE_INFO`, whose value is
|
||||
`dat\GP_READY_ROOM.pak+eng\script\`).
|
||||
|
||||
The object has exactly two records, `FILE` and `OFFSET`, carrying **the same 42
|
||||
keys in the same order** — it is a column store:
|
||||
|
||||
```
|
||||
key(tag_hash of an ISL script symbol) -> FILE = the .isb it is defined in
|
||||
OFFSET = where in that file
|
||||
```
|
||||
|
||||
It is the artefact of the pass the executable announces as
|
||||
`silph::GamePart_ReadyRoom::Impl::PrepareScript - isl script prescanning start.`
|
||||
— a link map from symbol hash to definition site. The 34 `.isb` files it names
|
||||
are all present in the same pak (`eng\script\stage01.isb` etc.).
|
||||
|
||||
### ✅ Independent confirmation: the keys appear in the bytecode
|
||||
|
||||
41 of the 42 keys occur verbatim as **little-endian 32-bit words** inside the
|
||||
`.isb` bytecode, and where they occur reconstructs a coherent call graph
|
||||
(Part 4 of the data file):
|
||||
|
||||
| defined in | referenced from |
|
||||
|---|---|
|
||||
| `stage01..16.isb`, `challenge01..06.isb`, `tutorial0N01.isb` (30 symbols) | `main.isb` only |
|
||||
| `function.isb` (5 symbols) | the stage and challenge scripts |
|
||||
| `function_tutorial.isb` (6 symbols) | the six tutorial scripts |
|
||||
| `main.isb` (1 symbol) | **nothing** — the root entry point |
|
||||
|
||||
That is exactly the shape a launcher + shared-helper-library arrangement should
|
||||
have, and it is the reason these keys can be called hashes with confidence rather
|
||||
than "large ordinals": ordinals are not stored as call targets in code.
|
||||
|
||||
🟡 `OFFSET` is **base 36**. Every value uses only `[0-9a-z]`, and read as base 36
|
||||
they are small ascending numbers per file: `function.isb` → 1, 9, 19, 35, 58;
|
||||
`function_tutorial.isb` → 1, 12, 23, 34, 45, 50; `tutorial0101.isb` → 1, 61, 191.
|
||||
The first symbol of every file is 1. Whether the unit is a statement index, a
|
||||
line, or a word offset is ❔ — not settled here.
|
||||
|
||||
## ✅ 30 of the 42 names end in the digits of their own filename
|
||||
|
||||
This is *forced* by the hash, not guessed. `tag_hash`'s low 24 bits are a
|
||||
base-256 polynomial mod `0x00FFFFDF`, so bumping the character `k` places from
|
||||
the end changes the low bits by `256^k mod M` (1, `0x100`, `0x10000`, `0x21`, …)
|
||||
and the top byte by 1. Measured:
|
||||
|
||||
| family | observed key deltas | forced conclusion |
|
||||
|---|---|---|
|
||||
| `stage01..09` | `+0x01000001` × 8 | last character increments by 1 |
|
||||
| `stage10..16` | `+0x01000001` × 6 | ditto |
|
||||
| `stage09 → stage10` | `0x8FA349EF → 0x87A34AE6` | low bits `+0xF7 = 0x100 − 9`, top `−8`: the last two characters go `"09" → "10"` |
|
||||
| `challenge01..06` | `+0x01000001` × 5 | last character increments by 1 |
|
||||
| `tutorial0101 → 0201 … 0601` | `+0x01010000` × 5 | the character **two** from the end increments |
|
||||
| `tutorial0101`'s two siblings | `+0x01000001`, `+0x02000002` | last character `+1`, `+2` |
|
||||
|
||||
So the sixteen stage symbols end `"01" … "16"`, the six challenge symbols end
|
||||
`"01" … "06"`, and the eight tutorial symbols end `"0101" "0102" "0103" "0201"
|
||||
"0301" "0401" "0501" "0601"` — matching their `.isb` filenames exactly. The
|
||||
remaining 12 (`function.isb` × 5, `function_tutorial.isb` × 6, `main.isb` × 1) are
|
||||
unconstrained.
|
||||
|
||||
That is the whole of the recovered name information. **The prefixes are not
|
||||
recovered.**
|
||||
|
||||
## ❌ Cracking: a null result, and the arithmetic that says why
|
||||
|
||||
`tag_hash` maps onto `256 × 0x00FFFFDF ≈ 2^32` values, so a search over `S`
|
||||
candidates yields on average `S / 2^32` **false** preimages *per target*. That is
|
||||
the number every attempt below is judged against.
|
||||
|
||||
| attack | candidates (space `S`) | E[false] per target | hits |
|
||||
|---|---|---|---|
|
||||
| every distinct disc string (IDXD names, record names, field values) | 105 393 | 0.000025 | **0** |
|
||||
| every string in the executable (`.pe`) | 75 912 | 0.000018 | **0** |
|
||||
| every ASCII run ≥3 in every pak payload except `sound.pak` (incl. `.ssb` symbol tables, XML, XPR) | 1 023 671 | 0.00024 | **0** |
|
||||
| all three combined, hashed against all 42 | 1 204 976 | 0.00028 | **0** |
|
||||
| `printf`-style substitution: 49 139 format strings × 80 numeric substitutions | 161 200 | 0.00004 | **0** |
|
||||
| hand-built guess list (`main`/`Stage`/`Mission`/`ReadyRoom`/… × prefixes × decorations × numbers) | 506 520 | 0.00012 | **0** |
|
||||
| two-token composition over a vocabulary mined from disc + exe identifiers (`t1 + sep + t2`, sep ∈ `"" _ -`) — this is what `idxd_unnamed_keys.py crack --pe …` runs | 3.48 × 10⁸ | 0.081 | **0** |
|
||||
| the same, with each family's forced digit suffix appended | 1.18 × 10¹⁰ across the sweep | 2.8 total | 1 (`"Cj-format 01"` — noise, and the same string for all six challenge keys, i.e. **one** false prefix) |
|
||||
| meet-in-the-middle exhaustive preimage, `[A-Za-z0-9_]`, ≤ 6 chars before the known suffix | 6.3 × 10¹⁰ | ~15 | noise only |
|
||||
|
||||
The last row is the important one. Exhaustive search **succeeds** — it is not
|
||||
that the search is broken. Fed `tag_hash("Stage01")` it returns
|
||||
`['F4D_kl01', 'Stage01', 'be_zT01', 'cd_Yu01']`: the right answer plus three
|
||||
collisions, which is what `63^5 / 2^32 ≈ 0.2` … `63^6 / 2^32 ≈ 15` predicts. Fed
|
||||
the real target `0x87A349E7` with suffix `"01"` it returns **nothing**, so the
|
||||
prefix is **longer than 6 characters** over that alphabet.
|
||||
|
||||
And that is where it stops being useful. Measured, not extrapolated: at 7
|
||||
characters target `0xA40C6DF7` has **1 176** preimages (`'1T1E1B6'`,
|
||||
`'24A7O0G'`, `'4quqkkC'`, …). At 8 the expectation is ≈ 58 000. A hash with a
|
||||
24-bit modulus simply does not have enough entropy to name an 8+ character
|
||||
identifier uniquely, so **no amount of compute recovers these names from the hash
|
||||
alone** — only a corpus containing the actual string can, and the disc does not
|
||||
contain it.
|
||||
|
||||
### 🔴 Refuted along the way (kept, not deleted)
|
||||
|
||||
* **"The symbol is the filename stem, in some case/decoration."** Refuted
|
||||
directly: `tag_hash` of `stage01`, `Stage01`, `STAGE01`, `stage_01`,
|
||||
`mission01`, and 30-odd relatives is not `0x87A349E7`.
|
||||
* **"The symbol is `<common prefix> + <filename stem> + <number>`."** Refuted
|
||||
*algebraically*, without searching. If `name = A + "stage01"` and
|
||||
`name = A + "challenge01"` share one `A`, then `A`'s byte sum is pinned twice
|
||||
and the two values must agree. They do not, for lowercase, Capitalised or
|
||||
UPPER stems, with or without a `_`/`-` separator — 9 pairs tested, 9
|
||||
contradictions.
|
||||
* **"The name is somewhere in the `.isb` files."** Refuted: the `.isb` payloads
|
||||
contain no ASCII symbol names at all. They reference symbols purely by hash —
|
||||
which is *why* the prescan link map exists.
|
||||
* **"The hash constants might be in the executable next to a name table."**
|
||||
Refuted: none of the 42 values occurs in the `.pe` image in either endianness.
|
||||
* **"IXUD might hold more hash-keyed nameless fields."** Refuted by measurement:
|
||||
534 IXUD objects, 624 488 field entries, **8** distinct unnamed tags — all of
|
||||
them ordinals `0…7` (the weapon `CATEGORY_DESC` list). Zero hash-shaped.
|
||||
* **⚠️ "There are ~504 keys to crack."** Withdrawn — 504 is the *entry* count.
|
||||
See above.
|
||||
|
||||
## Coverage / limits
|
||||
|
||||
- The census covers every IDXD object the disc has (7 750). `sound.pak` holds no
|
||||
IDXD objects and `dat/movie/*.wmv` and `hidden/resource3d/*.xpr` are not
|
||||
containers; strings from `resource3d` were nevertheless folded into the corpus
|
||||
attack.
|
||||
- The 42 targets are fully characterised as *references*; their *spelling* is
|
||||
unrecovered and, per the arithmetic above, unrecoverable from the disc.
|
||||
|
||||
## Evidence log (append-only; newest last)
|
||||
|
||||
- `2026-08-26` — Disc-wide walk of 7 750 IDXD objects / 2 757 039 field entries
|
||||
with `tag_hash` self-check at 1 271 462/1 271 462. Census → confidence
|
||||
`CONFIRMED` for the counts and for the ordinal/hash split.
|
||||
- `2026-08-26` — 42 hash-shaped keys localised to six copies of
|
||||
`<lang>\script\ID.tbl`; cross-checked against the `.isb` bytecode, where 41 of
|
||||
42 appear as little-endian call targets forming a consistent call graph →
|
||||
`CONFIRMED` that they are ISL script-symbol hashes.
|
||||
- `2026-08-26` — Key-delta arithmetic pins the trailing digits of 30 of the 42
|
||||
names → `CONFIRMED` (forced by the hash's polynomial structure, three
|
||||
independent families agreeing).
|
||||
- `2026-08-26` — Seven corpus/composition/brute-force attacks, none producing a
|
||||
hit above the noise floor; exhaustive search validated on a known name and
|
||||
shown to return nothing at ≤ 6 characters → the names are **not recovered**,
|
||||
and the negative result is quantified rather than asserted.
|
||||
|
||||
## Open questions / what would raise confidence
|
||||
|
||||
- **The 12 unconstrained names** (`function.isb` ×5, `function_tutorial.isb` ×6,
|
||||
`main.isb` ×1) have no digit structure to exploit at all.
|
||||
- **What would actually work:** a build artefact containing ISL source or a symbol
|
||||
list (none on the retail disc); or observing the game *construct* one of these
|
||||
strings at runtime — `PrepareScript` builds the table from `.isb` contents, but
|
||||
whatever `main.isb`'s root symbol is called, the name existed only in the
|
||||
compiler's input. A `--mem-watch` on the ReadyRoom lookup path would show
|
||||
whether any caller ever passes a literal name rather than a precomputed hash;
|
||||
that is the one remaining honest route and it was **not** attempted here (this
|
||||
was a static-only investigation).
|
||||
- **`OFFSET`'s unit** — base-36 is settled, what it counts is not.
|
||||
- The `.isb` instruction encoding is only sketched here (a length byte at
|
||||
`byte[2]`, call payloads leading with the target hash). It is not the `.ssb`
|
||||
ISL bytecode documented elsewhere and deserves its own note.
|
||||
536
tools/re-capture/idxd_unnamed_keys.py
Normal file
536
tools/re-capture/idxd_unnamed_keys.py
Normal file
@@ -0,0 +1,536 @@
|
||||
#!/usr/bin/env python3
|
||||
r"""Census -- and preimage attack -- for IDXD field keys that carry no name.
|
||||
|
||||
Pure static work: reads the extracted disc, runs no emulator.
|
||||
|
||||
python3 tools/re-capture/idxd_unnamed_keys.py census > docs/re/data/idxd-unnamed-field-keys.txt
|
||||
python3 tools/re-capture/idxd_unnamed_keys.py crack # corpus + brute-force attack
|
||||
python3 tools/re-capture/idxd_unnamed_keys.py idtbl # dump the 42 hash-shaped keys in context
|
||||
|
||||
An IDXD field entry is `(key, name_off, value_off)`. When `name_off` is
|
||||
0xFFFFFFFF the field has no name in the pool, and the reader must know what the
|
||||
`key` means from somewhere else. This tool answers "how many such keys are
|
||||
there, what do they look like, and can the names be recovered from the hash?"
|
||||
|
||||
The headline measurement (see docs/re/structures/idxd-unnamed-keys.md):
|
||||
|
||||
2 757 039 field entries in 7 750 IDXD objects
|
||||
1 271 462 named, 1 485 577 unnamed
|
||||
7 094 distinct keys that are never named ANYWHERE on the disc
|
||||
7 052 of them are < 0x10000 -- author-assigned ordinals / element ids
|
||||
42 of them are >= 0x01000000 -- genuine tag_hash values
|
||||
|
||||
`SYLPHEED_DISC` (default /work/sylph_extract) points at the extracted disc.
|
||||
"""
|
||||
import argparse, collections, glob, os, re, struct, sys, zlib
|
||||
|
||||
DISC = os.environ.get('SYLPHEED_DISC', '/work/sylph_extract')
|
||||
|
||||
# --------------------------------------------------------------------------- hashes
|
||||
|
||||
TAG_MODULUS = 0x00FFFFDF # 2^24 - 33, prime
|
||||
TAG_MAGIC = 0x2101 # floor(2^56/M)+1, the guest's divide magic
|
||||
|
||||
|
||||
def tag_hash(s):
|
||||
"""IDXD record key / field tag -- `sub_82447DF0`.
|
||||
|
||||
Transcribed in docs/re/structures/idxd-tag-hash.md and implemented three
|
||||
times in this tree (here, tools/re-capture/unitgroup.py, and
|
||||
sylpheed_formats::hash::tag_hash). Case-SENSITIVE, bytes sign-extended.
|
||||
"""
|
||||
a = b = 0
|
||||
for byte in s.encode('latin-1', 'replace'):
|
||||
c = (byte - 256) if byte > 127 else byte # extsb
|
||||
a = ((a << 8) & 0xFFFFFFFF)
|
||||
a = (a + c) & 0xFFFFFFFF
|
||||
b = (b + c) & 0xFFFFFFFF
|
||||
hi = ((a * TAG_MAGIC) >> 32) & 0xFFFFFFFF # mulhwu
|
||||
q = ((hi + (((a - hi) & 0xFFFFFFFF) >> 1)) & 0xFFFFFFFF) >> 23
|
||||
a = (a - q * TAG_MODULUS) & 0xFFFFFFFF
|
||||
return (((b << 24) & 0xFF000000) | (a & 0x00FFFFFF)) & 0xFFFFFFFF
|
||||
|
||||
|
||||
NAME_MODULUS, NAME_RECIP = 0x00FFF9D7, 0x80031493
|
||||
|
||||
|
||||
def _rotl(v, n):
|
||||
return ((v << n) | (v >> (32 - n))) & 0xFFFFFFFF
|
||||
|
||||
|
||||
def name_hash(s):
|
||||
"""IPFB TOC path hash -- lowercased, a different modulus. `sub_82455C78`."""
|
||||
bs = bytearray(s.encode('latin-1', 'replace'))
|
||||
for i, x in enumerate(bs):
|
||||
if 65 <= x <= 90:
|
||||
bs[i] = x + 0x20
|
||||
a = b = 0
|
||||
for byte in bs:
|
||||
c = ((byte - 256) if byte > 127 else byte) & 0xFFFFFFFF
|
||||
a = ((_rotl(a, 8) & 0xFFFFFF00) + c) & 0xFFFFFFFF
|
||||
b = (b + c) & 0xFFFFFFFF
|
||||
q = _rotl(((a * NAME_RECIP) >> 32) & 0xFFFFFFFF, 9) & 0x1FF
|
||||
a = (a - (q * NAME_MODULUS)) & 0xFFFFFFFF
|
||||
return (((b << 24) & 0xFF000000) | (a & 0x00FFFFFF)) & 0xFFFFFFFF
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- pak / IDXD
|
||||
|
||||
def pak_entries(pak):
|
||||
"""yield (toc_key, payload) for every entry of an IPFB pak + its .pNN segments."""
|
||||
idx = open(pak, 'rb').read()
|
||||
data = b''.join(open(p, 'rb').read()
|
||||
for p in sorted(glob.glob(pak[:-4] + '.p[0-9][0-9]')))
|
||||
for i in range(struct.unpack_from('>I', idx, 4)[0]):
|
||||
k, off, csize = struct.unpack_from('>III', idx, 16 + i * 12)
|
||||
s = data[off:off + csize]
|
||||
if s[:2] == b'Z1':
|
||||
try:
|
||||
s = zlib.decompress(s[10:])
|
||||
except zlib.error:
|
||||
continue
|
||||
yield k, s
|
||||
|
||||
|
||||
def parse_idxd(b):
|
||||
"""-> [(record_key, record_name, [(tag, name|None, value, field_index)])]"""
|
||||
u = lambda o: struct.unpack_from('>I', b, o)[0]
|
||||
nrec = u(0x04)
|
||||
npool_off = 0x08 + nrec * 16
|
||||
npool = u(npool_off)
|
||||
pool = npool_off + 4
|
||||
strsize_off = pool + npool * 12
|
||||
STR = strsize_off + 4
|
||||
strsize = u(strsize_off)
|
||||
if STR + strsize != len(b):
|
||||
raise ValueError('string-pool trailer mismatch')
|
||||
|
||||
def s(o):
|
||||
if o >= strsize:
|
||||
raise ValueError('string offset out of pool')
|
||||
return b[STR + o:b.index(b'\x00', STR + o)].decode('latin-1')
|
||||
|
||||
out = []
|
||||
for i in range(nrec):
|
||||
key, nm, lo, hi = (u(0x08 + i * 16 + j * 4) for j in range(4))
|
||||
fields = []
|
||||
for j in range(lo, hi):
|
||||
o = pool + j * 12
|
||||
tag, noff, voff = u(o), u(o + 4), u(o + 8)
|
||||
fields.append((tag, s(noff) if noff != 0xFFFFFFFF else None, s(voff), j - lo))
|
||||
out.append((key, s(nm), fields))
|
||||
return out
|
||||
|
||||
|
||||
def all_paks():
|
||||
return sorted(glob.glob(DISC + '/dat/*.pak') + glob.glob(DISC + '/hidden/*.pak'))
|
||||
|
||||
|
||||
def walk(counter=None):
|
||||
"""yield (pak_basename, obj_toc_key, record_key, record_name, fields)."""
|
||||
for pak in all_paks():
|
||||
base = os.path.basename(pak)
|
||||
for h, payload in pak_entries(pak):
|
||||
if payload[:4] != b'IDXD':
|
||||
continue
|
||||
try:
|
||||
recs = parse_idxd(payload)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print('PARSE FAIL %s %08x %s' % (base, h, exc), file=sys.stderr)
|
||||
continue
|
||||
if counter is not None:
|
||||
counter['objects'] += 1
|
||||
schema = recs[0][1] if recs else ''
|
||||
for key, rname, fields in recs:
|
||||
yield base, h, schema, key, rname, fields
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- value profile
|
||||
|
||||
_FLOAT = re.compile(r'^-?(\d+\.\d*|\.\d+|\d+)([eE][-+]?\d+)?$')
|
||||
_INT = re.compile(r'^-?\d+$')
|
||||
_HEX = re.compile(r'^(0[xX])?[0-9a-fA-F]+$')
|
||||
_FILE = re.compile(r'^[\w\\/. -]+\.[A-Za-z0-9]{2,4}$')
|
||||
|
||||
|
||||
def classify(values):
|
||||
"""A coarse type label for a bag of field values."""
|
||||
if not values:
|
||||
return 'empty'
|
||||
kinds = set()
|
||||
for v in values:
|
||||
if v == '':
|
||||
kinds.add('empty')
|
||||
elif _INT.match(v):
|
||||
kinds.add('int')
|
||||
elif _FLOAT.match(v):
|
||||
kinds.add('float')
|
||||
elif _FILE.match(v):
|
||||
kinds.add('file')
|
||||
elif v in ('True', 'False', 'Yes', 'No', 'ON', 'OFF'):
|
||||
kinds.add('bool')
|
||||
elif _HEX.match(v) and len(v) >= 6:
|
||||
kinds.add('hex')
|
||||
else:
|
||||
kinds.add('id')
|
||||
order = ['file', 'id', 'hex', 'bool', 'float', 'int', 'empty']
|
||||
return '+'.join(k for k in order if k in kinds)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- census
|
||||
|
||||
def census():
|
||||
named = collections.defaultdict(collections.Counter)
|
||||
unnamed = collections.Counter()
|
||||
key_objs = collections.defaultdict(set)
|
||||
key_recs = collections.defaultdict(collections.Counter)
|
||||
key_idx = collections.defaultdict(lambda: [1 << 30, -1])
|
||||
key_vals = collections.defaultdict(list)
|
||||
key_nval = collections.Counter()
|
||||
# per (object-schema, record-name) family, for the ordinal population
|
||||
fam = collections.defaultdict(lambda: {'keys': set(), 'n': 0, 'vals': [], 'recs': 0})
|
||||
tot = collections.Counter()
|
||||
schema_of = {}
|
||||
|
||||
for base, h, schema, rkey, rname, fields in walk(tot):
|
||||
tot['records'] += 1
|
||||
# the object's schema hash is the key of its first record; use record name
|
||||
fam_id = None
|
||||
for tag, name, val, fidx in fields:
|
||||
tot['fields'] += 1
|
||||
if name is not None:
|
||||
tot['named'] += 1
|
||||
named[tag][name] += 1
|
||||
else:
|
||||
tot['unnamed'] += 1
|
||||
unnamed[tag] += 1
|
||||
key_objs[tag].add((base, h))
|
||||
key_recs[tag][rname] += 1
|
||||
lohi = key_idx[tag]
|
||||
lohi[0] = min(lohi[0], fidx)
|
||||
lohi[1] = max(lohi[1], fidx)
|
||||
key_nval[tag] += 1
|
||||
if len(key_vals[tag]) < 40:
|
||||
key_vals[tag].append(val)
|
||||
if fam_id is None:
|
||||
fam_id = (base, schema)
|
||||
f = fam[fam_id]
|
||||
f['keys'].add(tag)
|
||||
f['n'] += 1
|
||||
if len(f['vals']) < 60:
|
||||
f['vals'].append(val)
|
||||
if fam_id is not None:
|
||||
fam[fam_id]['recs'] += 1
|
||||
|
||||
return dict(named=named, unnamed=unnamed, key_objs=key_objs, key_recs=key_recs,
|
||||
key_idx=key_idx, key_vals=key_vals, key_nval=key_nval, fam=fam, tot=tot)
|
||||
|
||||
|
||||
def report(c, out=sys.stdout):
|
||||
named, unnamed = c['named'], c['unnamed']
|
||||
never = sorted(set(unnamed) - set(named))
|
||||
hashy = [k for k in never if k >= 0x01000000]
|
||||
ordinal = [k for k in never if k < 0x01000000]
|
||||
both = sorted(set(unnamed) & set(named))
|
||||
w = out.write
|
||||
|
||||
w('# IDXD field keys that carry no name\n')
|
||||
w('#\n')
|
||||
w('# Generated by tools/re-capture/idxd_unnamed_keys.py census\n')
|
||||
w('# Disc: %s (all dat/*.pak + hidden/*.pak, .pNN segments joined, Z1 inflated)\n' % DISC)
|
||||
w('# Companion write-up: docs/re/structures/idxd-unnamed-keys.md\n')
|
||||
w('#\n')
|
||||
w('# A field entry is (key, name_off, value_off). name_off == 0xFFFFFFFF means the\n')
|
||||
w('# pool holds no name for it, so the reader must already know what `key` means.\n')
|
||||
w('\n')
|
||||
w('== TOTALS ==\n')
|
||||
w('IDXD objects walked : %d\n' % c['tot']['objects'])
|
||||
w('IDXD records walked : %d\n' % c['tot']['records'])
|
||||
w('field entries : %d\n' % c['tot']['fields'])
|
||||
w(' named (name_off != 0xFFFFFFFF) : %d\n' % c['tot']['named'])
|
||||
w(' unnamed (name_off == 0xFFFFFFFF) : %d\n' % c['tot']['unnamed'])
|
||||
w('distinct keys seen named somewhere : %d\n' % len(named))
|
||||
w('distinct keys seen unnamed somewhere : %d\n' % len(unnamed))
|
||||
w('distinct keys NEVER named anywhere on disc : %d\n' % len(never))
|
||||
w(' of those, < 0x00010000 (ordinal-shaped) : %d\n' % len([k for k in never if k < 0x10000]))
|
||||
w(' of those, in [0x10000, 0x1000000) : %d\n' % len([k for k in never if 0x10000 <= k < 0x1000000]))
|
||||
w(' of those, >= 0x01000000 (hash-shaped) : %d\n' % len(hashy))
|
||||
w('keys that are named in one place and unnamed in another: %d %s\n'
|
||||
% (len(both), ['%08x' % k for k in both]))
|
||||
w('unnamed FIELD ENTRIES whose key is hash-shaped : %d\n'
|
||||
% sum(c['key_nval'][k] for k in hashy))
|
||||
w('unnamed FIELD ENTRIES whose key is an ordinal : %d\n'
|
||||
% sum(c['key_nval'][k] for k in ordinal))
|
||||
w('unnamed FIELD ENTRIES with key 0x00000000 : %d (ordinal 0; the key is\n'
|
||||
% sum(c['key_nval'][k] for k in both))
|
||||
w('# "named" elsewhere only because tag_hash("") == 0, so it is counted apart)\n')
|
||||
w('# The first of those two is the "504 hash-keyed fields disc-wide" already in\n')
|
||||
w('# INDEX.md. It is 504 ENTRIES, not 504 distinct keys: %d keys x 6 language\n' % len(hashy))
|
||||
w('# copies of the same object x 2 records (FILE and OFFSET) = %d.\n'
|
||||
% sum(c['key_nval'][k] for k in hashy))
|
||||
w('\n')
|
||||
w('# Why the split at 0x01000000 is not arbitrary: a tag_hash puts the byte-sum\n')
|
||||
w('# checksum in the top byte, so a hash lands below 0x01000000 only when the name\n')
|
||||
w('# byte-sum is 0 mod 256 (1/256 of names) AND its low 24 bits are also tiny. Of\n')
|
||||
zerotop = sorted(k for k in named if (k >> 24) == 0 and k != 0)
|
||||
w('# the %d keys that DO carry a name, %d have a zero top byte, and the smallest of\n'
|
||||
% (len(named), len(zerotop)))
|
||||
w('# those is 0x%06x -- far above the ordinal band, which tops out at 0x%x.\n'
|
||||
% (zerotop[0], max(ordinal)))
|
||||
w('\n')
|
||||
|
||||
w('== PART 1: the %d HASH-SHAPED never-named keys ==\n' % len(hashy))
|
||||
w('# key n objects record field value\n')
|
||||
for k in hashy:
|
||||
recs = ','.join(sorted(c['key_recs'][k]))
|
||||
lo, hi = c['key_idx'][k]
|
||||
vals = sorted(set(c['key_vals'][k]))
|
||||
w('%08x %3d %2d %-13s %2d-%-2d %s\n'
|
||||
% (k, c['key_nval'][k], len(c['key_objs'][k]), recs, lo, hi, '|'.join(vals)))
|
||||
w('\n')
|
||||
w('# All %d live in the six copies of <lang>\\script\\ID.tbl inside\n' % len(hashy))
|
||||
w('# dat/GP_READY_ROOM.pak (eng jpn fra deu ita esp). Each copy has two records,\n')
|
||||
w('# FILE and OFFSET, with the SAME 42 keys in the same order -- i.e. the object is\n')
|
||||
w('# a column store: key -> (which .isb file, what offset inside it).\n')
|
||||
w('\n')
|
||||
|
||||
w('== PART 2: the ordinal population, by (pak, object schema) ==\n')
|
||||
w('# The schema is the name of the object\'s FIRST record -- the string whose\n')
|
||||
w('# tag_hash is the object header word. `recs` counts records with >=1 unnamed field.\n')
|
||||
w('# pak schema recs fields keys keymin-keymax valuetype sample\n')
|
||||
rows = []
|
||||
for (base, rname), f in c['fam'].items():
|
||||
ks = f['keys']
|
||||
rows.append((base, rname, f['recs'], f['n'], len(ks), min(ks), max(ks),
|
||||
classify(f['vals']), f['vals'][0] if f['vals'] else ''))
|
||||
for r in sorted(rows, key=lambda r: (-r[3], r[0], r[1])):
|
||||
w('%-28s %-17s %5d %7d %5d %6x-%-6x %-10s %s\n'
|
||||
% (r[0], r[1], r[2], r[3], r[4], r[5], r[6], r[7], r[8][:40]))
|
||||
w('\n')
|
||||
|
||||
w('== PART 3: every never-named ordinal key ==\n')
|
||||
w('# key occurrences distinct-records fieldidx-lo-hi valuetype sample-value\n')
|
||||
for k in ordinal:
|
||||
lo, hi = c['key_idx'][k]
|
||||
w('%08x %8d %6d %4d-%-4d %-10s %s\n'
|
||||
% (k, c['key_nval'][k], len(c['key_recs'][k]), lo, hi,
|
||||
classify(c['key_vals'][k]), (c['key_vals'][k] or [''])[0][:40]))
|
||||
w('\n')
|
||||
w('== PART 4: the ID.tbl script-symbol call graph ==\n')
|
||||
try:
|
||||
idtbl(out)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
w('# unavailable: %s\n' % exc)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- ID.tbl
|
||||
|
||||
READY_ROOM = DISC + '/dat/GP_READY_ROOM.pak'
|
||||
LANGS = ('eng', 'jpn', 'fra', 'deu', 'ita', 'esp')
|
||||
|
||||
|
||||
_RR_CACHE = {}
|
||||
|
||||
|
||||
def readyroom_file(path):
|
||||
"""Read one file out of GP_READY_ROOM.pak by its (lowercased) path."""
|
||||
if not _RR_CACHE:
|
||||
_RR_CACHE.update(dict(pak_entries(READY_ROOM)))
|
||||
return _RR_CACHE.get(name_hash(path))
|
||||
|
||||
|
||||
def idtbl(out=sys.stdout):
|
||||
"""The 42 keys with their FILE/OFFSET columns, plus the .isb cross-check."""
|
||||
payload = readyroom_file('eng\\script\\ID.tbl')
|
||||
recs = {r[1]: r[2] for r in parse_idxd(payload)}
|
||||
files = recs['FILE']
|
||||
offs = recs['OFFSET']
|
||||
out.write("# eng\\script\\ID.tbl -- key -> (defining .isb file, offset), plus every\n")
|
||||
out.write('# .isb whose bytecode contains the key as a little-endian word (= a call site).\n')
|
||||
out.write('# idx key defined-in offset referenced-from\n')
|
||||
isb = {}
|
||||
for name in sorted({v for _, _, v, _ in files}):
|
||||
b = readyroom_file('eng\\script\\' + name)
|
||||
if b:
|
||||
isb[name] = {struct.unpack_from('<I', b, o)[0] for o in range(0, len(b) - 3)}
|
||||
for i, (tag, _n, val, _f) in enumerate(files):
|
||||
seen = sorted(f for f, s in isb.items() if tag in s)
|
||||
out.write('%3d %08x %-24s %-6s %s\n'
|
||||
% (i, tag, val, offs[i][2], ','.join(seen) or '-'))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- attacks
|
||||
|
||||
def _poly(s):
|
||||
a = 0
|
||||
for ch in s.encode('latin-1', 'replace'):
|
||||
a = (a * 256 + ch) % TAG_MODULUS
|
||||
return a
|
||||
|
||||
|
||||
def corpus_attack(targets, words):
|
||||
"""Straight dictionary attack. Returns {tag: [names]} and the corpus size."""
|
||||
hits = collections.defaultdict(list)
|
||||
n = 0
|
||||
for w in words:
|
||||
n += 1
|
||||
h = tag_hash(w)
|
||||
if h in targets:
|
||||
hits[h].append(w)
|
||||
return hits, n
|
||||
|
||||
|
||||
def compose_attack(targets, tokens, suffixes=('',), seps=('', '_', '-')):
|
||||
"""Two-token composition attack: name = t1 + sep + t2 + suffix.
|
||||
|
||||
Returns (hits, effective search space). The space is what matters: with a
|
||||
32-bit tag, expected FALSE hits per target is space / 2**32.
|
||||
"""
|
||||
right = collections.defaultdict(list)
|
||||
for t in tokens:
|
||||
for sep in seps:
|
||||
for suf in suffixes:
|
||||
r = sep + t + suf
|
||||
right[(len(r), _poly(r))].append(r)
|
||||
lens = sorted({k[0] for k in right})
|
||||
p256 = {l: pow(256, l, TAG_MODULUS) for l in lens}
|
||||
hits = collections.defaultdict(list)
|
||||
for left in tokens:
|
||||
pl = _poly(left)
|
||||
for l in lens:
|
||||
for tag in targets:
|
||||
want = ((tag & 0xFFFFFF) - pl * p256[l]) % TAG_MODULUS
|
||||
for r in right.get((l, want), ()):
|
||||
cand = left + r
|
||||
if tag_hash(cand) == tag:
|
||||
hits[tag].append(cand)
|
||||
space = len(tokens) * sum(len(v) for v in right.values())
|
||||
return hits, space
|
||||
|
||||
|
||||
def brute_force(target, suffix='', maxlen=6, alphabet=None):
|
||||
"""Meet-in-the-middle preimage search: all P over `alphabet` with
|
||||
tag_hash(P + suffix) == target and len(P) <= maxlen.
|
||||
|
||||
Needs numpy. The honest limit is maxlen 6: the expected number of FALSE
|
||||
preimages is |alphabet|**maxlen / 2**32, which is ~15 at 63**6 and ~58 000
|
||||
at 63**8, so anything past 6 returns noise, not names.
|
||||
"""
|
||||
import numpy as np
|
||||
if alphabet is None:
|
||||
alphabet = ([chr(c) for c in range(48, 58)] + [chr(c) for c in range(65, 91)]
|
||||
+ [chr(c) for c in range(97, 123)] + ['_'])
|
||||
A = len(alphabet)
|
||||
codes = np.array([ord(c) for c in alphabet], dtype=np.int64)
|
||||
tabs = []
|
||||
polys = np.zeros(1, dtype=np.int64)
|
||||
sums = np.zeros(1, dtype=np.int64)
|
||||
tabs.append((polys, sums))
|
||||
for _ in range(maxlen // 2 + maxlen % 2):
|
||||
polys = ((polys[:, None] * 256 + codes[None, :]) % TAG_MODULUS).ravel()
|
||||
sums = (sums[:, None] + codes[None, :]).ravel()
|
||||
tabs.append((polys, sums))
|
||||
|
||||
def decode(idx, k):
|
||||
s = ''
|
||||
for _ in range(k):
|
||||
s = alphabet[idx % A] + s
|
||||
idx //= A
|
||||
return s
|
||||
|
||||
ps = _poly(suffix)
|
||||
inv = pow(pow(256, len(suffix), TAG_MODULUS), -1, TAG_MODULUS)
|
||||
need_sum = ((target >> 24) - sum(suffix.encode('latin-1'))) & 0xFF
|
||||
out = []
|
||||
for L in range(maxlen + 1):
|
||||
v = L // 2
|
||||
u = L - v
|
||||
if u >= len(tabs) or v >= len(tabs):
|
||||
continue
|
||||
pu, su = tabs[u]
|
||||
pv, sv = tabs[v]
|
||||
order = np.argsort(pv, kind='stable')
|
||||
pv_s = pv[order]
|
||||
need = (((target & 0xFFFFFF) - ps) * inv) % TAG_MODULUS
|
||||
want = (need - (pu * pow(256, v, TAG_MODULUS)) % TAG_MODULUS) % TAG_MODULUS
|
||||
lo = np.searchsorted(pv_s, want, side='left')
|
||||
hi = np.searchsorted(pv_s, want, side='right')
|
||||
for i in np.nonzero(hi > lo)[0]:
|
||||
for j in range(lo[i], hi[i]):
|
||||
vi = order[j]
|
||||
if ((su[i] + sv[vi]) & 0xFF) != need_sum:
|
||||
continue
|
||||
cand = decode(int(i), u) + decode(int(vi), v) + suffix
|
||||
if tag_hash(cand) == target:
|
||||
out.append(cand)
|
||||
return out
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- main
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument('mode', choices=('census', 'idtbl', 'crack', 'selftest'))
|
||||
ap.add_argument('--maxlen', type=int, default=6)
|
||||
ap.add_argument('--pe', default=os.environ.get('SYLPHEED_PE'),
|
||||
help='flat VA dump of default.xex; its strings widen the corpus')
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.mode == 'census':
|
||||
report(census())
|
||||
elif args.mode == 'idtbl':
|
||||
idtbl()
|
||||
elif args.mode == 'selftest':
|
||||
# tag_hash must reproduce the key of every NAMED field on the disc.
|
||||
ok = bad = 0
|
||||
for _b, _h, _sc, _rk, _rn, fields in walk():
|
||||
for tag, name, _v, _i in fields:
|
||||
if name is None:
|
||||
continue
|
||||
if tag_hash(name) == tag:
|
||||
ok += 1
|
||||
else:
|
||||
bad += 1
|
||||
if bad < 5:
|
||||
print('MISMATCH %r %08x != %08x' % (name, tag, tag_hash(name)))
|
||||
print('tag_hash reproduces %d / %d named field keys (%d wrong)' % (ok, ok + bad, bad))
|
||||
else:
|
||||
c = census()
|
||||
targets = {k for k in set(c['unnamed']) - set(c['named']) if k >= 0x01000000}
|
||||
print('targets: %d hash-shaped never-named keys' % len(targets))
|
||||
words = set()
|
||||
for b, h, _sc, _rk, rname, fields in walk():
|
||||
words.add(rname)
|
||||
for _t, name, val, _i in fields:
|
||||
if name:
|
||||
words.add(name)
|
||||
words.add(val)
|
||||
if args.pe and os.path.exists(args.pe):
|
||||
blob = open(args.pe, 'rb').read()
|
||||
words |= {m.decode('latin-1') for m in re.findall(rb'[ -~]{3,}', blob)}
|
||||
hits, n = corpus_attack(targets, words)
|
||||
print('disc-string corpus : %d strings, %d hits, E[false]=%.4f'
|
||||
% (n, len(hits), n * len(targets) / 2 ** 32))
|
||||
for t, v in hits.items():
|
||||
print(' %08x %s' % (t, v[:6]))
|
||||
toks = collections.Counter()
|
||||
for w in words:
|
||||
for part in re.split(r'[^A-Za-z0-9]+', w):
|
||||
for p in re.findall(r'[A-Z]+(?![a-z])|[A-Z][a-z]+|[a-z]+', part):
|
||||
if 2 <= len(p) <= 14:
|
||||
toks[p] += 1
|
||||
tokens = [t for t, _ in toks.most_common()]
|
||||
hits, space = compose_attack(targets, tokens)
|
||||
print('2-token composition: space=%d, %d hits, E[false]=%.3f'
|
||||
% (space, len(hits), space * len(targets) / 2 ** 32))
|
||||
for t, v in hits.items():
|
||||
print(' %08x %s' % (t, v[:6]))
|
||||
for t in sorted(targets)[:3]:
|
||||
r = brute_force(t, '', args.maxlen)
|
||||
print('brute force <=%d chars %08x: %d candidates %s'
|
||||
% (args.maxlen, t, len(r), r[:6]))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user