diff --git a/docs/port/HANDOFF.md b/docs/port/HANDOFF.md index 9bb26395..8e5686c4 100644 --- a/docs/port/HANDOFF.md +++ b/docs/port/HANDOFF.md @@ -219,6 +219,22 @@ authored version can be deleted. ⚠️ An earlier version of this bullet said "the title screen loops at β‰ˆ 2.2 s" and attributed it to `ptloop01/02`. Both halves were wrong: it is the **plate**, and it is a different build. +* 🟑 **Our composite is brighter than the emulator's frame β€” measured, and you + would be *authoring* if you apply it.** Alignment is exact (best offset + dy=0 dx=0, correlation **0.9466**), so only the tone differs. Fitting on 16Γ—16 + patches that are flat in **both** images: `capture β‰ˆ 255Β·(render/255)^Ξ³` with + Ξ³ = **1.491** (main menu), **1.493** (`EXTRAS`), **1.338** (title). + ⚠️ **Narrow reach.** Those flat patches span only render values ~0–60, where a + gamma and a plain scale are nearly indistinguishable β€” on both menus the errors + are 0.28 vs 0.34 and 0.22 vs 0.28. Only the title separates them (1.08 vs + 9.02). Nothing here constrains midtones or highlights. + πŸ”΄ The held-out control **failed to discriminate**: the splash's 2 918 flat + patches are pure black (render 0–4), so every model scores β‰ˆ 0. + ❔ **It may be the emulator, not the game.** Canary applies + `kernel_display_gamma_type = 2` (BT.709) on output. The discriminating run β€” + set it to `0` and re-fit β€” was not done. + [`structures/ui-render-tone-curve.md`](../re/structures/ui-render-tone-curve.md) + * 🟑 **`screen render` silently drops one full-screen element per screen β€” and you must NOT simply draw it.** Auditing what the composer omits on your five screens: everything is accounted for (`kind & 0x4` ghost instances, `.prm` diff --git a/docs/re/INDEX.md b/docs/re/INDEX.md index 9d6d2056..a1b4bd7a 100644 --- a/docs/re/INDEX.md +++ b/docs/re/INDEX.md @@ -19,7 +19,7 @@ Promote to a prose `structures/…md` file when a format needs behavioural notes | name-hash (TOC keys) | βœ… | `sylpheed-formats/src/hash.rs` | Barrett-reduction hash; recovers original paths | | IDXD object/table | βœ… | `sylpheed-formats/src/idxd.rs` + `tests/idxd_records_disc.rs` ([container](structures/idxd-container.md)) | **The binary record/index region in front of the string pool is DECODED** (2026-08-25), closing the parser's long-standing "not yet decoded" note. Uniform 16-byte records `{name_hash, name_off, field_begin, field_end}` sorted by hash and binary-searched, then a field count, 12-byte fields `{key, name_off, value_off}` sorted by key, a pool size, and the string pool; the trailing `pool_size == file_len - pool_base` identity makes the layout self-checking. Verified over the **whole disc** with **zero** failures: 7 750/7 750 objects, 190 782/190 782 records reproducing their stored `tag_hash`, 1 271 462/1 271 462 named fields reproducing their key β€” and `IXUD` is the same container with `ixud_hash`, UTF-16BE and all offsets in **chars** (1 104/1 104 objects, 628 165/628 165 fields). **Field names are stored on disc** β€” a field's middle word points at its own name β€” so nothing needs preimage search except the **504** field entries disc-wide that are hash-keyed with no name β€” which are only **42 distinct keys**, each in 12 places (the page for these was never written β€” the finding is in this row), and are provably unrecoverable from the hash alone; the other 1 485 073 nameless fields are *positional*, keyed by a literal integer (line slots, movie ids). ⚠️ **Two long-held beliefs WITHDRAWN**: the word at `0x08` is **not a schema hash**, it is record 0's `name_hash` (7 750/7 750) β€” the header has no type field at all, so an object's kind is known only from the caller that loads it; and the field's middle word is **not** an always-`0xFFFFFFFF` flags word. The first was caught by a test asserting that every movie id names a real record: `1005 -> STAGE10_PHASE01` failed because `tag_hash("STAGE10_PHASE01")` **is** `0x067025B9`, that table's supposed schema id. 🟑 the legacy value-before-key string-pool reader is now known to be an *approximation* of the real table, and every number derived from it is re-checkable but not yet re-checked | | IDXD nameless field keys | βœ…/❌ | [idxd-unnamed-keys](structures/idxd-unnamed-keys.md) + [`tools/re-capture/idxd_unnamed_keys.py`](../../tools/re-capture/idxd_unnamed_keys.py) | Census of every field entry whose `name_off` is `0xFFFFFFFF`, disc-wide: **7 750 objects, 2 757 039 field entries, 0 parse failures**, `tag_hash` reproducing **1 271 462/1 271 462** named keys. **7 094 distinct keys are never named β€” and 7 052 of them are not hashes at all**, but author-assigned element ids (equal to the field's own index in 1 404 924 of 1 485 577 cases; `tag_hash("BGM_001")` is `0xC662435B` while the key valued `BGM_001.slb` is `0x000003E9`). ⚠️ **The "504 hash-keyed nameless fields" figure is 504 ENTRIES, not 504 names** β€” 42 distinct keys Γ— 6 language copies Γ— 2 records. All 42 are **ISL script-symbol hashes** in `\script\ID.tbl` (GP_READY_ROOM.pak), the link map built by `PrepareScript`'s "isl script prescanning"; 41 of 42 appear as little-endian call targets inside the `.isb` bytecode, forming a coherent launcher/helper call graph. The hash's own algebra pins the **trailing digits of 30 of the 42 names** (deltas of exactly `+0x01000001` across `stage01..09`, `stage10..16`, `challenge01..06`; `+0x01010000` across `tutorial0101..0601`). ❌ **No name was cracked, and the negative is quantified**: seven attacks up to a 3.5Γ—10⁸ composition space found nothing above the noise floor; exhaustive preimage search recovers `"Stage01"` from its own hash but returns nothing for the real targets at ≀6 characters, and at 7 characters one target already has **1 176** preimages β€” a 24-bit modulus cannot name an 8+ character identifier uniquely | -| XPR2 texture + cubemap | 🟑/βœ… | `sylpheed-formats/src/texture.rs` + [colour check](xpr2-colour-check.md) | de-tile + A8R8G8B8 and DXT1. **Channel order βœ… confirmed against the running game**: the Delta Saber's decoded atlas is orange-dominant (median saturated hue 23.3Β°, *zero* cool pixels) and the game renders the same hull at 9.3Β° β€” a red↔blue swap would sit at β‰ˆ200Β°. Exact fidelity (gamma/sRGB curve, premultiplied alpha, per-channel scale) is 🟑 untested, since a hue comparison cannot see it; cubemap face ordering ❔ | +| XPR2 texture + cubemap | 🟑/βœ… | `sylpheed-formats/src/texture.rs` + [colour check](xpr2-colour-check.md) | de-tile + A8R8G8B8 and DXT1. **Channel order βœ… confirmed against the running game**: the Delta Saber's decoded atlas is orange-dominant (median saturated hue 23.3Β°, *zero* cool pixels) and the game renders the same hull at 9.3Β° β€” a red↔blue swap would sit at β‰ˆ200Β°. Exact fidelity (gamma/sRGB curve, premultiplied alpha, per-channel scale) is 🟑 untested, since a hue comparison cannot see it; cubemap face ordering ❔. **2026-08-29, for the UI path only:** our composite is brighter than the emulator's frame by a gamma of **β‰ˆ1.34–1.49** across three screens ([tone curve](structures/ui-render-tone-curve.md)) β€” 🟑 measured, not decoded, constrained only over render values ~0–60, and possibly canary's own `kernel_display_gamma_type = 2` (BT.709) output stage rather than the game's | | T8aD 2D texture | βœ… | `sylpheed-formats/src/t8ad.rs` | **100 % of the disc decodes** (19 216/19 216, measured). The "~15 % deferred variants" were a wrong model, not a variant: a surface is a list of **arbitrary sub-rectangles**, each with a 16-byte header of `dst X, dst Y, width, height`, not a 256Γ—256 grid β€” `0x1c` is the **rectangle count**. Uncovered area stays transparent. **Colours βœ… CONFIRMED** ([k8888](structures/texture-color-k8888.md)) | | RATC bundle | βœ… | `sylpheed-formats/src/ratc.rs` | child listing confirmed. **"One level deep" is not a limitation β€” there is nothing deeper**: 2 859 bundles hold 18 002 children at depth 1 and **0 at depth 2**, with no parse failures. Nested RATC blobs are **leaf records that reference siblings by name** (`opt `, the sprite name): 3 311 leaves, all embedding sibling names, **10 144 of 10 148 references resolve**. The 4 that do not are one dangling asset β€” `pmbase.rat` β†’ `pmbase.t32` in `GP_STAGE_CLEAR.pak`'s four language builds, and `pmbase.t32` is **on the disc nowhere** | | LSTA sprite list | βœ… | `sylpheed-formats/src/lsta.rs` | A display list of inline elements: **T8aD sprites and `PRMD` primitives**. The `count` at `0x04` is **exact and counts both** β€” `count == T8aD + PRMD` for **64/64** lists on the disc, which retires the old "a few entries disagree" note (it compared sprites against a total including primitives). **All 1 281 sprite frames decode** after the T8aD rectangle-list fix | diff --git a/docs/re/METHOD.md b/docs/re/METHOD.md index ce775665..b9d714ee 100644 --- a/docs/re/METHOD.md +++ b/docs/re/METHOD.md @@ -482,3 +482,17 @@ agent's loop prompt, i.e. nowhere durable. See [`README.md`](README.md) for the lookup by declared name misses. Before concluding a texture is absent, look at the bytes the link actually points at β€” the name in the declaration table and the name the sprite table keys on can differ. +* **A fit whose residual is large for every model is a broken method, not a + close call.** Comparing our composite to a capture pixel-wise gave mean abs + errors of 10–14 for gamma, for a linear scale, for everything β€” and a + non-monotonic transfer curve (render 96–127 mapping *brighter* than render + 128–159). The cause was edge misalignment: at correlation 0.947 a bright pixel + in one image routinely lands on a dark one in the other. Restricting to patches + that are flat in **both** images dropped the residual to 0.2–1.1. When every + candidate model fits badly, stop choosing between them and look at what the + comparison is actually measuring. +* **Say when a control failed to discriminate, rather than reporting it as a + pass.** The held-out screen for a tone-curve fit was the developer splash, + whose flat regions are pure black β€” every model scored β‰ˆ 0.00 error there. That + is not corroboration; it is a test with no power, and reporting the 0.00 as + agreement would have dressed an untested claim as a verified one. diff --git a/docs/re/structures/ui-render-tone-curve.md b/docs/re/structures/ui-render-tone-curve.md new file mode 100644 index 00000000..1a1e8611 --- /dev/null +++ b/docs/re/structures/ui-render-tone-curve.md @@ -0,0 +1,82 @@ +# 🟑 Our composite is brighter than the emulator's frame β€” measured, not decoded + +**Status:** 🟑 **measured, with a narrow reach and a live confound.** Closes an +observation left dangling by +[ui-8ax-fullres-background](ui-8ax-fullres-background.md) ("the capture is ~4Γ— +darker than the render"), and puts a number on the ❔ that +[INDEX](../INDEX.md)'s texture row already carried: *"exact fidelity +(gamma/sRGB curve, premultiplied alpha, per-channel scale) is untested, since a +hue comparison cannot see it."* + +⚠️ **This is not a decode.** A port applying it is authoring a value. + +## First: the geometry is right + +Cross-correlating `live-main-menu.png` against our render over Β±6 px finds the +best alignment at exactly **dy = 0, dx = 0**, correlation **0.9466**. So the +composite is in the right place at the right size and only the *tone* differs. +(The capture is 1279Γ—675 and top-aligned; that is the screenshot tool's crop.) + +## πŸ”΄ The first two methods were wrong, and both failed visibly + +* **Three dark patches** gave "capture β‰ˆ 4Γ— darker". Over the whole frame the + best linear scale is **0.914**. Three patches from one region are not a + transfer curve. +* **A pixel-wise fit** over 854 685 pixels produced a non-monotonic transfer + (render 96–127 β†’ capture *143*, brighter than render 128–159 β†’ 132). That is + the signature of **edge misalignment**, not of a tone curve: at a 0.947 + correlation a bright render pixel routinely lands on a dark capture pixel. + Mean abs error was 10–14 for every model, which is the tell that none of them + fit. + +Both are recorded because the second is the interesting failure β€” a fit whose +*residual* is large everywhere is not a model to choose between, it is a method +to throw away. + +## The method that works: flat patches only + +16Γ—16 patches where **both** images have `std < 8`, so local edges cannot +contribute. The threshold is not arbitrary β€” at `std < 3` there are **zero** +patches, and the count runs 0 / 83 / 404 / 1055 / 1788 for `std <` +3 / 5 / 8 / 12 / 20. + +| screen | flat patches | gamma exponent | mean abs err | best linear | its err | +|---|---|---|---|---|---| +| main menu | 404 | **1.491** | 0.28 | 0.276 | 0.34 | +| `EXTRAS` | 382 | **1.493** | 0.22 | 0.273 | 0.28 | +| title | 506 | **1.338** | 1.08 | 0.842 | 9.02 | + +So `capture β‰ˆ 255Β·(render/255)^Ξ³` with **Ξ³ β‰ˆ 1.34 – 1.49**. + +## ⚠️ The reach β€” and it is narrow + +* **The flat patches are almost all dark**: render values ~0–60. Over that range + a gamma and a linear scale are nearly indistinguishable β€” on the two menus the + errors are 0.28 vs 0.34 and 0.22 vs 0.28, which decides nothing. **Only the + title separates them** (1.08 vs 9.02), because its flat regions reach ~60. +* **The held-out control could not test it.** Running the same fit on the + developer splash gives 2 918 flat patches whose render range is **0–4** β€” pure + black. Every model scores β‰ˆ 0.00 there. That is a control that failed to + discriminate, not a control that passed. +* **Nothing here constrains midtones or highlights**, which is exactly where a + Ξ³ = 1.4 curve does its visible work. + +## ❔ The confound: this may be the emulator, not the game + +The comparison is our composite against **what Xenia Canary displays**, and +canary applies its own output transform: +`kernel_display_gamma_type = 2` β€” BT.709 (HDTV) β€” in this container's config +(`0` linear, `1` sRGB, `3` a power set by `kernel_display_gamma_power`). + +So the measured exponent may belong to canary's display stage rather than to the +game or the console. **The discriminating experiment is cheap and was not run:** +set `kernel_display_gamma_type = 0`, re-capture the same screen, and re-fit. If +the exponent collapses toward 1.0 it is canary's; if it survives it is upstream. +That needs one emulator run reaching the main menu. + +## What a port should do with this + +Treat it as **authored**, not transcribed. If the goal is to match the emulator β€” +which is what every capture in this corpus is β€” a Ξ³ β‰ˆ 1.4 darkening of the +composite gets closer, and is best applied where it was measured (the dark +background), not extrapolated to the whole range on this evidence.