fabi/Sylpheed
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

re: retract the paint-key census and redo it over all 21 184 sprites

Browse Source 
The census filtered pak entries whose own first four bytes are T8aD.  A sprite
is usually a child of a RATC bundle, and a bundle entry's magic is RATC, so a
top-level magic filter cannot see one:

    top-level T8aD entries (counted)    4 525 sprites,  45 keys
    T8aD inside RATC bundles (missed)  16 659 sprites, 204 keys
    both                               21 184 sprites, 216 keys

171 of the 216 keys exist only inside bundles.  The sharpest statement of the
error: that census never saw GP_TITLE.pak at all -- the pak holding both of the
screens this page's entire evidence comes from.

Retracted: "45 values", "the keys are pak-local", "each auxiliary pak occupies
its own narrow high-byte band".  On the full population 68/216 keys (31%, not
9%) cross a pak family and the per-pak ranges overlap heavily -- GP_BUNK
0x8000-0xa110, GP_TITLE 0x8000-0xc150, GP_LEADERBOARD 0x8000-0xf100.  The tidy
banding was an artifact of seeing one or two keys per pak.  So the key looks
like a shared vocabulary, which is the opposite of what I published.

Survives, now on the full population: the field is a u16 at +0x0A (upper half
zero 21 184/21 184), and it is an enumeration (216 values for 21 184 sprites).

Three wrong numbers on this page now, all the same shape -- a statistic computed
over a population I had not checked was the population in question.  Stated once
at the end of the section rather than three times: check the sampling frame
before the statistic.
This commit is contained in:
Sylpheed RE agent
2026-08-26 10:06:23 +00:00
parent dbeab6aa9c
commit 0a64728fae
4 changed files with 145 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
docs/re/BACKLOG.md
View File

@@ -143,17 +143,20 @@ where translucent sprites overlap.
* ❔ What the field's bits mean — `0x8000`/`0x80a0`/`0xa110` look like flag words
with a layer in some bits, not a plain depth. Sorting the whole word works on
both measured screens; which bits carry the layer is unknown.
**Censused 2026-08-26** over all **4 525** disc sprites
**Censused 2026-08-26** over **all 21 184** disc sprites
(`tools/re-capture/paint_key_census.py`): ✅ the field is a **`u16` at `+0x0A`**
(upper half zero **4 525/4 525**), ✅ it is an **enumeration — 45 values**, and
🔴 **it is pak-local, so it is not a global layer vocabulary**: only **4/45**
keys cross a pak family and **33/45** live only in `GP_MAIN_GAME_2D`, each
other pak owning a narrow high-byte band (`0x90`–`0x94` overlays, `0xa4`
mission log, `0xb1`–`0xb2` save/load). Supports "group id in the high bits,
order in the low bits" but does **not** test it — both measured screens are
inside one pak. ⚠️ My first count said 37/45 shared; that was the six
*language* copies of `GP_MAIN_GAME_2D` (identical key sets) counted as six
paks, and it pointed at the opposite conclusion.
(upper half zero **21 184/21 184**) and ✅ an **enumeration — 216 values**.
🔴 **"the keys are pak-local" is REFUTED** — **68/216 (31 %)** cross a pak
family and the per-pak ranges overlap heavily, so it looks like a shared
vocabulary, not a per-screen depth. ❔ Which bits carry the layer is still
unknown.
⚠️ **A first version of this census was retracted.** It filtered pak entries on
a `T8aD` magic, but sprites are usually **`RATC` children** — it saw 4 525 of
21 184 sprites, 45 of 216 keys, and **not `GP_TITLE.pak` at all**, the pak both
measured screens come from. It reported "45 values" and "pak-local", both wrong.
A separate earlier slip on the same page counted the six *language* copies of
`GP_MAIN_GAME_2D` as six paks. Same shape each time: a statistic computed over
an unverified sampling frame.
* ❔ A third measured permutation, to promote "holds on two" to a rule. The
cheapest is a screen whose object is resident at the same time as the title's.
* ❔ 341 builds now composite in an order no capture has checked.

48
docs/re/data/paint-key-census.txt
View File

@@ -1,17 +1,35 @@
pak families: {'GP_CHALLENGE.pak': 2, 'GP_DEBRIEFING_PILOTLOG.pak': 2, 'GP_HANGAR_ARSENAL.pak': 3, 'GP_LEADERBOARD.pak': 3, 'GP_MAIN_GAME_2D.pak': 34, 'GP_MISSION_LOG.pak': 1, 'GP_MISSION_SELECT.pak': 1, 'GP_OPTIONS.pak': 1, 'GP_SAVE_LOAD.pak': 3, 'GP_STAGE_CLEAR.pak': 1}
ALL T8aD sprites (top-level + RATC children): 21184
upper half of the +0x08 word is zero : 21184/21184
distinct u16 keys at +0x0A : 216
keys in more than one pak family : 68/216 = 31%
keys confined to GP_MAIN_GAME_2D : 52/216
the six language 2D paks have identical key sets: True (6 paks)
keys in more than one FAMILY: 4/45 -> ['0000', '9092', '9110', '9200']
keys confined to GP_MAIN_GAME_2D alone: 33/45
per family, its keys:
GP_CHALLENGE.pak: ['9110', '9200']
GP_DEBRIEFING_PILOTLOG.pak: ['9091', '9092']
GP_HANGAR_ARSENAL.pak: ['9200', '9300', '9400']
GP_LEADERBOARD.pak: ['9108', '9110', '9200']
GP_MAIN_GAME_2D.pak: ['0000', '7d00', '7e00', '7e10', '7e20', '7e40', '7e50', '7e51', '7e52', '7e80', '7e90', '7f00', '7f10', '7f20', '7f30', '7f40', '7f60', '7f61', '8000', '8010', '8020', '8030', '8040', '8080', '8100', '8108', '8110', '8200', '8210', '8300', '8800', '8880', '8900', '9820']
GP_MISSION_LOG.pak: ['a410']
GP_MISSION_SELECT.pak: ['9110']
GP_OPTIONS.pak: ['0000']
GP_SAVE_LOAD.pak: ['b100', 'b110', 'b210']
GP_STAGE_CLEAR.pak: ['9092']
keys per family:
GP_BUNK.pak: 16 keys 8000..a110
GP_CHALLENGE.pak: 9 keys 8000..9200
GP_DEBRIEFING_PILOTLOG.pak: 17 keys 8200..9900
GP_DIALOG.pak: 22 keys 8100..f102
GP_GAMEOVER.pak: 6 keys d840..d862
GP_HANGAR_ARSENAL.pak: 20 keys 8000..9412
GP_LEADERBOARD.pak: 14 keys 8000..f100
GP_MAIN_GAME_2D.pak: 72 keys 0000..9820
GP_MISSION_LOG.pak: 5 keys a400..a420
GP_MISSION_SELECT.pak: 13 keys 8300..9820
GP_MOVIE_THEATER.pak: 7 keys 8300..9122
GP_OPTIONS.pak: 21 keys 0000..b900
GP_PAUSE_MENU.pak: 8 keys 8020..a112
GP_READY_ROOM.pak: 45 keys 0000..9fff
GP_SAVE_LOAD.pak: 27 keys 8300..f100
GP_STAGE_CLEAR.pak: 12 keys 8200..9112
GP_SYSTEM.pak: 7 keys a010..a112
GP_TITLE.pak: 28 keys 8000..c150
GP_TUTORIAL.pak: 5 keys 8210..8312
deu.pak: 5 keys 8000..8010
eng.pak: 5 keys 8000..8010
esp.pak: 5 keys 8000..8010
fra.pak: 5 keys 8000..8010
ita.pak: 5 keys 8000..8010
jpn.pak: 5 keys 8000..8010
commonest 12 keys: [('8000', 1768), ('8100', 1524), ('9230', 1454), ('9400', 857), ('9220', 776), ('9402', 720), ('8010', 689), ('9110', 527), ('9100', 467), ('8900', 450), ('8400', 426), ('a100', 423)]

108
docs/re/structures/ui-paint-order-key.md
View File

@@ -326,72 +326,72 @@ X/Y, the `T8aD` header words, and `kind`.
## The corpus-wide census (2026-08-26)
Everything above rests on twelve values from two screens. This walks **4 525
`T8aD` sprites on the disc**, so the open question can be asked of a corpus.
`tools/re-capture/paint_key_census.py`, output in
[`../data/paint-key-census.txt`](../data/paint-key-census.txt).
[`../data/paint-key-census.txt`](../data/paint-key-census.txt). **All 21 184
`T8aD` sprites on the disc** — pak entries *and* `RATC` children.
> ⚠️ **Read the next paragraph before using these numbers.** The census counts
> pak entries whose *own* first four bytes are `T8aD`. But the sprites this page
> measures paint order on are **children of a `RATC` bundle** — `ui_layout.rs`
> reaches them through `ratc::parse`, and a bundle entry's magic is `RATC`, so a
> child `T8aD` is invisible to a top-level magic filter. The two populations may
> overlap partly, wholly, or not at all, and **which one the 45 keys describe is
> being measured, not assumed.** Until that lands, treat every count below as a
> statement about top-level `T8aD` entries only.
>
> This is the same failure shape as the 37/45 note further down: a number
> computed over a population I had not checked was the population in question.
### ⚠️ The first version of this section counted 4 525 sprites, and was retracted
It filtered pak entries whose own first four bytes are `T8aD`. But a sprite is
usually a **child of a `RATC` bundle**, and a bundle entry's magic is `RATC`, so
a top-level magic filter cannot see one:
| | sprites | distinct keys |
|---|---|---|
| top-level `T8aD` entries (what was counted) | 4 525 | 45 |
| `T8aD` inside `RATC` bundles (what was missed) | **16 659** | **204** |
| both | **21 184** | **216** |
**171 of the 216 keys exist only inside bundles.** The sharpest way to put it:
that census never saw **`GP_TITLE.pak` at all** — the pak holding both screens
this page's evidence comes from. Every number it produced described a fifth of
the corpus, chosen by an accident of container nesting.
Retracted from it: "45 values", "the keys are pak-local", and "each auxiliary pak
occupies its own narrow high-byte band". What survives is below.
### ✅ The field is a `u16` at `+0x0A`, not a word at `+0x08`
The upper half of the 32-bit word this page reads is **zero in 4 525 / 4 525**
sprites. Nothing above changes — `0x00008100` sorts identically to `0x8100` —
but the port should read two bytes, and a future value with the high half set
would mean something has been misread rather than that the layer got deeper.
The upper half of the 32-bit word this page reads is **zero in 21 184 / 21 184**
sprites. Nothing above changes — `0x00008100` sorts identically to `0x8100` — but
the port should read two bytes, and a value with the high half set would mean
something has been misread rather than that the layer got deeper.
### ✅ It is an enumeration: 45 values for 4 525 sprites
### ✅ It is an enumeration: 216 values for 21 184 sprites
Not a per-sprite depth. The commonest key alone (`0x8100`) covers 1 188 sprites,
and the whole range is `0x0000`, `0x7d00`–`0x9820`, then `0xa410`, `0xb100`–`0xb210`.
Not a per-sprite depth. The commonest key (`0x8000`) covers 1 768 sprites,
`0x8100` another 1 524.
### 🔴 The keys are **pak-local** — so this is not a global layer vocabulary
### 🔴 Refuted: the keys are **not** partitioned by pak
That was the reading worth trying, and it fails. Collapsing the six language
variants of `GP_MAIN_GAME_*2D.pak` into one family:
This was the previous section's headline and the fuller population kills it.
| | |
|---|---|
| keys appearing in more than one pak family | **4 / 45** (`0x0000`, `0x9092`, `0x9110`, `0x9200`) |
| keys confined to `GP_MAIN_GAME_2D` alone | **33 / 45** |
| | 4 525-sprite version | **all 21 184** |
|---|---|---|
| keys crossing a pak family | 4 / 45 = 9 % | **68 / 216 = 31 %** |
| keys confined to `GP_MAIN_GAME_2D` | 33 / 45 | **52 / 216** |
and each auxiliary pak occupies its own narrow high-byte band:
And the per-pak *ranges overlap heavily* rather than banding: `GP_BUNK`
`0x8000`–`0xa110`, `GP_CHALLENGE` `0x8000`–`0x9200`, `GP_HANGAR_ARSENAL`
`0x8000`–`0x9412`, `GP_LEADERBOARD` `0x8000`–`0xf100`, `GP_TITLE`
`0x8000`–`0xc150`. The tidy "each pak owns a high-byte band" picture was an
artifact of seeing one or two keys per pak. Only three paks are actually narrow
(`GP_GAMEOVER` `0xd840`–`0xd862`, `GP_MISSION_LOG` `0xa400`–`0xa420`,
`GP_SYSTEM` `0xa010`–`0xa112`).
| pak | its keys |
|---|---|
| `GP_MAIN_GAME_2D` | `0x7d00`–`0x9820` (34 keys) |
| `GP_DEBRIEFING_PILOTLOG` | `0x9091`, `0x9092` |
| `GP_LEADERBOARD` | `0x9108`, `0x9110`, `0x9200` |
| `GP_CHALLENGE` | `0x9110`, `0x9200` |
| `GP_HANGAR_ARSENAL` | `0x9200`, `0x9300`, `0x9400` |
| `GP_STAGE_CLEAR` | `0x9092` |
| `GP_MISSION_SELECT` | `0x9110` |
| `GP_MISSION_LOG` | `0xa410` |
| `GP_SAVE_LOAD` | `0xb100`, `0xb110`, `0xb210` |
So the key is closer to a **shared vocabulary** than to a per-screen depth — the
opposite of what the under-sampled census said. ❔ Which bits carry the layer
remains unknown, and is now a question about 216 values rather than 45.
A screen that owns one or two keys is not using them to order itself internally —
it has nothing to order against. So the high bits look like a **group or owner
id** and the low bits like the ordering within it, which is the shape this page
already suspected. 🟡 That remains a *description of 45 numbers*: paint order has
been measured on two screens only, both inside `GP_MAIN_GAME_2D`, so there is no
ground truth anywhere else and the split between "group" and "order" bits is
still untested.
### ✅ The language paks are one screen set six times
### ⚠️ The first count I got was 37 / 45, and it was about localisation
The six `GP_MAIN_GAME_*2D.pak` have **byte-for-byte identical key sets**. This is
why counting paks rather than pak *families* first reported "37 of 45 keys appear
in more than one pak" — a statement about localisation, not about the format.
Counting paks rather than pak *families* makes every `GP_MAIN_GAME_2D` key look
like it appears in six paks. The six are the same screens in six languages — and
their key sets are **byte-for-byte identical** (verified: one distinct set across
all six). The inflated number would have supported exactly the wrong conclusion,
that the keys form a vocabulary shared across the whole UI. Recorded because the
shape recurs: a corpus with near-duplicate members will manufacture agreement.
### The recurring failure, stated once
Three published numbers on this page were wrong the same way: each was computed
over a population I had not checked was the population in question — the six
language copies, then the top-level-only sprites, then the size of that gap. The
analysis was never the problem. **Check the sampling frame before the statistic.**