diff --git a/docs/re/INDEX.md b/docs/re/INDEX.md new file mode 100644 index 0000000..d52a286 --- /dev/null +++ b/docs/re/INDEX.md @@ -0,0 +1,29 @@ +# RE knowledge index + +Confidence: ✅ `CONFIRMED` · 🟡 `PROBABLE` · ❔ `HYPOTHESIS`. See [README](README.md). + +Formats we've already reversed are, for now, **documented by their parser + disc round-trip +tests** (the executable spec) rather than a prose file — the "Spec" column points there. +Promote to a prose `structures/…md` file when a format needs behavioural notes beyond layout. + +## Data structures / formats + +| Format | Conf. | Spec (parser + tests) | Notes | +|--------|-------|-----------------------|-------| +| IPFB `.pak` archive | ✅ | `sylpheed-formats/src/pak.rs` + `tests/pak_idxd_disc.rs` | header + 12-byte TOC, Z1/zlib payloads | +| name-hash (TOC keys) | ✅ | `sylpheed-formats/src/hash.rs` | Barrett-reduction hash; recovers original paths | +| IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` | self-describing; ship/weapon stats verified vs known values | +| XPR2 texture + cubemap | 🟡 | `sylpheed-formats/src/texture.rs` | de-tile + A8R8G8B8; **colours unverified** (dynamic item) | +| T8aD 2D texture | 🟡 | `sylpheed-formats/src/t8ad.rs` | ~85% decode; **colours unverified**; ~15% variants deferred | +| RATC bundle | 🟡 | `sylpheed-formats/src/ratc.rs` | child listing confirmed; one level deep | +| LSTA sprite list | 🟡 | `sylpheed-formats/src/lsta.rs` | inline T8aD frames | +| IXUD subtitle | 🟡 | `sylpheed-formats/src/ixud.rs` | timed cues; **movie↔track link unknown** (dynamic item) | +| Fonts (ttf/otf/ttc) | ✅ | `sylpheed-formats/src/font.rs` | standard OpenType, parsed via ttf-parser | + +## Functions / code paths + +_None documented yet — populated during the dynamic-RE phase._ + +| Function | Conf. | Reimpl. | Summary | +|----------|-------|---------|---------| +| — | — | — | — | diff --git a/docs/re/README.md b/docs/re/README.md new file mode 100644 index 0000000..ff98138 --- /dev/null +++ b/docs/re/README.md @@ -0,0 +1,88 @@ +# Reverse-engineering knowledge base + +This directory is the **spec-side** of the clean-room: it records what the original +*Project Sylpheed* binary **does** (behaviour) and how its **data is laid out**, so that +the Rust port can be implemented **from these specs** without re-deriving anything and +without ever copying original code. + +It exists to answer one question fast: *"do we already know how X works, and how sure are we?"* + +--- + +## The one rule that matters + +> **Never document a claim more confidently than the evidence supports, and never +> paste original code here.** + +A wrong-but-confident note is worse than no note: someone builds on it and the bug hides +for weeks. Every entry therefore carries an explicit **confidence** and its **evidence**. +This mirrors the project method — *measure the oracle, never infer; refute before believing.* + +### Clean-room firewall + +- ✅ Allowed: behaviour descriptions, field offsets/types, formulas, state machines, + observed input→output pairs, and **references** to the original by address + (`sub_821B68C0`) or to `xenia-rs/sylpheed.db`. +- ❌ Forbidden here and in `crates/`: pasted decompiled C/C++ or verbatim disassembled + function bodies presented as the thing to reimplement. Cite the address; describe the + behaviour in your own words. Disassembly is a tool for *understanding*, not a source to copy. + +--- + +## Confidence levels + +| Level | Meaning | Bar to reach it | +|-------|---------|-----------------| +| `CONFIRMED` | Behaviour verified against ground truth. | ≥2 independent observations **or** one observation cross-checked against an oracle (canary framebuffer, a known-correct value, a second code path). | +| `PROBABLE` | Strong single-source inference. | One clean observation, or an unambiguous static read of the disassembly. | +| `HYPOTHESIS` | Educated guess, not yet tested. | Anything else. Must say what would confirm/refute it. | + +**Promotion requires new evidence, not re-reading the old evidence.** A `HYPOTHESIS` that +"looks right again" is still a `HYPOTHESIS`. Only an *independent* check promotes it. +If evidence later contradicts an entry, **demote it and record the contradiction** — do +not silently edit the conclusion. + +--- + +## When to document + +- **Right after** a function/structure crosses from `HYPOTHESIS` to at least `PROBABLE` — + before moving to the next code path, so the knowledge isn't lost or re-derived. +- **Whenever confidence changes** (up or down) — append to the Evidence log, don't overwrite. +- **Not** while it's still a pure guess with no evidence — a one-liner in the relevant + backlog/plan is enough until there's something to stand on. + +## What to document + +- **Functions/code paths** → `docs/re/functions/.md` (one file per function or tight cluster). +- **Data structures / formats** → `docs/re/structures/.md`. +- Keep the index in [`INDEX.md`](INDEX.md) (one line each: name · confidence · one-line summary). + +Use the templates: [`_TEMPLATE.function.md`](_TEMPLATE.function.md), +[`_TEMPLATE.structure.md`](_TEMPLATE.structure.md). + +--- + +## How we find and confirm code paths (the toolchain) + +Everything joins on the **guest virtual address (PC)** — code addresses are fixed by the +XEX load, identical across our emulator and canary. + +- **Static (cheap, try first):** `xenia-rs/sylpheed.db` (DuckDB: 25 481 functions, xrefs, + strings, vtables, imports). Query with `xenia-rs/zq.py` — `zq.py grep `, + `zq.py xref `, `zq.py dis `, `zq.py fn `. Entry points are usually a + **string** (`zq.py grep MSG_DEMO`) or an **import** (movie/XMA API) xref'd back to the loader. +- **Dynamic (when static is ambiguous):** run `xenia-rs` with its probe suite — + `--pc-probe` / `--audit-pc-probe-hex` (fires at block entry), `--mem-watch` (mid-block + reads/writes of a VA), `--lr-trace` (call/return chains), `--trace-instructions`, + `--dump-addr` (read guest memory). These already exist; prefer them over hacking canary. +- **Oracle (correctness ground truth):** canary — the **Wine cross-build** + `xenia-canary/build-cross/bin/Windows/Debug/xenia_canary.exe` (the native Linux ELF + **crashes / does not run** — do not use it). This is the only emulator that reaches the + in-game menu; our `xenia-rs` never got past the intro video. Use canary to *observe output* + (capture its framebuffer for texture colours), not usually to instrument code — though its + `build-cross` toolchain does compile, so small C++ probes + rebuild are possible when needed. + Run **muted, one emulator process at a time**, point it at the real ISO (not the symlink). + +> ⚠️ **VA-equality caveat:** join **code** by PC (fixed), but **never** assume a data VA +> holds the same bytes across emulators — allocators differ. Compare data by content/layout. diff --git a/docs/re/_TEMPLATE.function.md b/docs/re/_TEMPLATE.function.md new file mode 100644 index 0000000..90b22c9 --- /dev/null +++ b/docs/re/_TEMPLATE.function.md @@ -0,0 +1,23 @@ +# `` + +- **Address:** `0x________` (in `sylpheed.db`; `zq.py fn 0x________`) +- **Confidence:** `HYPOTHESIS | PROBABLE | CONFIRMED` +- **Reimplemented in:** `crates/…/…rs::fn` — or *not yet* +- **Related:** other RE entries, `[[memory-slug]]` + +## What it does +One-paragraph behavioural summary, in your own words. No pasted disassembly. + +## Signature / calling convention +Inputs (registers/args + meaning), outputs, side effects (memory it writes, globals, imports it calls). + +## Behaviour +Step-by-step of the observable behaviour — the *spec* to implement from. Formulas, branches, +state transitions. Cite addresses for detail (`the loop at 0x…`), don't transcribe code. + +## Evidence log (append-only; newest last) +- `YYYY-MM-DD` — *how we know*: static disasm read / `--lr-trace` from 0x… / canary output / + N observations. What it established. → confidence set to `…`. + +## Open questions / what would raise confidence +The specific check that would confirm or refute the current claim. diff --git a/docs/re/_TEMPLATE.structure.md b/docs/re/_TEMPLATE.structure.md new file mode 100644 index 0000000..aee7007 --- /dev/null +++ b/docs/re/_TEMPLATE.structure.md @@ -0,0 +1,29 @@ +# `` + +- **Confidence:** `HYPOTHESIS | PROBABLE | CONFIRMED` +- **Parser in:** `crates/sylpheed-formats/src/…rs` — or *not yet* +- **Seen in:** which pak / asset / memory region +- **Related:** other RE entries, `[[memory-slug]]` + +## Layout +```text +offset size type name notes +0x00 4 magic "____" +0x__ 4 u32be … +… +``` +Endianness, alignment, variable-length rules, how child/section sizes are derived. + +## Field semantics +What each field *means* and its observed value range. Distinguish **confirmed** fields from +**guessed** ones explicitly (a guessed field with a plausible value is still a guess). + +## Coverage / limits +What fraction of real instances this decodes; which variants are deferred and why. + +## Evidence log (append-only; newest last) +- `YYYY-MM-DD` — how established (forensic scan over N disc entries / round-trip test / + oracle comparison). → confidence `…`. + +## Open questions / what would raise confidence +e.g. "colours unverified until compared against canary's framebuffer." diff --git a/docs/re/structures/texture-color-k8888.md b/docs/re/structures/texture-color-k8888.md new file mode 100644 index 0000000..5e2f079 --- /dev/null +++ b/docs/re/structures/texture-color-k8888.md @@ -0,0 +1,38 @@ +# Texture colour interpretation — `k_8_8_8_8` (32bpp UI/HUD textures) + +- **Confidence:** mixed — see per-claim tags below. +- **Parser in:** `sylpheed-formats/src/t8ad.rs`, `src/texture.rs` (XPR2) +- **Applies to:** T8aD 2D UI textures, XPR2 32bpp surfaces — the "colours look a bit odd" concern. +- **Related:** [[reborn-dynamic-re-plan]], INDEX rows T8aD / XPR2. + +## What this resolves +Whether our decoded 32bpp textures use the right **sRGB-vs-linear** interpretation and the right +**channel order**. Ground truth = Canary's Vulkan texture cache, which is the code that actually +produces correct on-screen colour. + +## Findings + +### ✅ CONFIRMED — plain `k_8_8_8_8` is LINEAR (UNORM), never sRGB +Canary's host-format table maps plain `k_8_8_8_8` → `VK_FORMAT_R8G8B8A8_UNORM` with a straight +32-bit copy load shader (`kLoadShaderIndex32bpb`) and an **identity** format swizzle +(`XE_GPU_TEXTURE_SWIZZLE_RGBA`). There is **no** `..._SRGB` host format anywhere in the table; +gamma is a separate explicit path (`k_8_8_8_8_GAMMA_EDRAM` only). Therefore a plain 32bpp texture +is sampled as **raw UNORM bytes — apply no sRGB/gamma decode.** +→ *Implication:* if the viewer/engine applies an sRGB→linear (or linear→sRGB) step to these, that is +wrong. Show the bytes as-is (raw RGBA8), gamma only where the game explicitly requests it. + +### ❔ HYPOTHESIS — exact source **channel order** (needs measurement, do not infer) +Our decoders currently assume on-disk **A8R8G8B8** and reorder `[a,r,g,b] → [r,g,b,a]`. Canary +proves the *host* target is R8G8B8A8 identity-swizzled, but the *guest→host* byte order also depends +on the texture's runtime **endian** field (e.g. `8in32`), which is set by the game at fetch time and +**cannot be read statically** from the format table. So whether our reorder matches is unverified. +→ *What would confirm/refute it:* one **measured** decoded texel — either capture Canary's +framebuffer over a texture with distinct R≠G≠B pixels of known colour, or patch Canary's already- +present-but-unwired `TextureDump` (`src/xenia/gpu/texture_dump.cc`) to dump the decoded texture and +diff bytes against our decode. **Measure, don't guess** — a symmetric colour (white logo) can't +disambiguate channel order. + +## Evidence log +- `2026-07-11` — static read of Canary `src/xenia/gpu/vulkan/vulkan_texture_cache.cc` host-format + table + confirmed no `R8G8B8A8_SRGB` entry exists. → sRGB/UNORM claim `CONFIRMED`; channel-order + claim remains `HYPOTHESIS` pending a measured texel.