docs(re): add reverse-engineering knowledge base + conventions
Some checks failed
CI / Native — ubuntu-latest (push) Failing after 7m53s
CI / WASM — Web (push) Failing after 6m32s
CI / Formatting (push) Failing after 35s
CI / Native — macos-latest (push) Has been cancelled
CI / Native — windows-latest (push) Has been cancelled

Establish docs/re/ as the spec-side of the clean-room:
- README: confidence tiers (HYPOTHESIS/PROBABLE/CONFIRMED), promotion
  requires NEW independent evidence, append-only evidence logs,
  demote-on-contradiction, and a clean-room firewall (describe behaviour +
  cite addresses; never paste decompiled code). Documents the toolchain
  (sylpheed.db + zq.py, xenia-rs probes, Wine-Canary oracle) and the
  join-by-guest-PC method with the VA-equality caveat.
- Function/structure templates + a seeded INDEX (9 already-reversed formats).
- First entry: texture-color-k8888 — CONFIRMED plain k_8_8_8_8 is linear
  UNORM (no sRGB), channel order still to be measured.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-11 17:04:23 +02:00
parent e1ca95c0af
commit b7028471e9
5 changed files with 207 additions and 0 deletions

29
docs/re/INDEX.md Normal file
View File

@@ -0,0 +1,29 @@
# RE knowledge index
Confidence: ✅ `CONFIRMED` · 🟡 `PROBABLE` · ❔ `HYPOTHESIS`. See [README](README.md).
Formats we've already reversed are, for now, **documented by their parser + disc round-trip
tests** (the executable spec) rather than a prose file — the "Spec" column points there.
Promote to a prose `structures/…md` file when a format needs behavioural notes beyond layout.
## Data structures / formats
| Format | Conf. | Spec (parser + tests) | Notes |
|--------|-------|-----------------------|-------|
| IPFB `.pak` archive | ✅ | `sylpheed-formats/src/pak.rs` + `tests/pak_idxd_disc.rs` | header + 12-byte TOC, Z1/zlib payloads |
| name-hash (TOC keys) | ✅ | `sylpheed-formats/src/hash.rs` | Barrett-reduction hash; recovers original paths |
| IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` | self-describing; ship/weapon stats verified vs known values |
| XPR2 texture + cubemap | 🟡 | `sylpheed-formats/src/texture.rs` | de-tile + A8R8G8B8; **colours unverified** (dynamic item) |
| T8aD 2D texture | 🟡 | `sylpheed-formats/src/t8ad.rs` | ~85% decode; **colours unverified**; ~15% variants deferred |
| RATC bundle | 🟡 | `sylpheed-formats/src/ratc.rs` | child listing confirmed; one level deep |
| LSTA sprite list | 🟡 | `sylpheed-formats/src/lsta.rs` | inline T8aD frames |
| IXUD subtitle | 🟡 | `sylpheed-formats/src/ixud.rs` | timed cues; **movie↔track link unknown** (dynamic item) |
| Fonts (ttf/otf/ttc) | ✅ | `sylpheed-formats/src/font.rs` | standard OpenType, parsed via ttf-parser |
## Functions / code paths
_None documented yet — populated during the dynamic-RE phase._
| Function | Conf. | Reimpl. | Summary |
|----------|-------|---------|---------|
| — | — | — | — |

88
docs/re/README.md Normal file
View File

@@ -0,0 +1,88 @@
# Reverse-engineering knowledge base
This directory is the **spec-side** of the clean-room: it records what the original
*Project Sylpheed* binary **does** (behaviour) and how its **data is laid out**, so that
the Rust port can be implemented **from these specs** without re-deriving anything and
without ever copying original code.
It exists to answer one question fast: *"do we already know how X works, and how sure are we?"*
---
## The one rule that matters
> **Never document a claim more confidently than the evidence supports, and never
> paste original code here.**
A wrong-but-confident note is worse than no note: someone builds on it and the bug hides
for weeks. Every entry therefore carries an explicit **confidence** and its **evidence**.
This mirrors the project method — *measure the oracle, never infer; refute before believing.*
### Clean-room firewall
- ✅ Allowed: behaviour descriptions, field offsets/types, formulas, state machines,
observed input→output pairs, and **references** to the original by address
(`sub_821B68C0`) or to `xenia-rs/sylpheed.db`.
- ❌ Forbidden here and in `crates/`: pasted decompiled C/C++ or verbatim disassembled
function bodies presented as the thing to reimplement. Cite the address; describe the
behaviour in your own words. Disassembly is a tool for *understanding*, not a source to copy.
---
## Confidence levels
| Level | Meaning | Bar to reach it |
|-------|---------|-----------------|
| `CONFIRMED` | Behaviour verified against ground truth. | ≥2 independent observations **or** one observation cross-checked against an oracle (canary framebuffer, a known-correct value, a second code path). |
| `PROBABLE` | Strong single-source inference. | One clean observation, or an unambiguous static read of the disassembly. |
| `HYPOTHESIS` | Educated guess, not yet tested. | Anything else. Must say what would confirm/refute it. |
**Promotion requires new evidence, not re-reading the old evidence.** A `HYPOTHESIS` that
"looks right again" is still a `HYPOTHESIS`. Only an *independent* check promotes it.
If evidence later contradicts an entry, **demote it and record the contradiction** — do
not silently edit the conclusion.
---
## When to document
- **Right after** a function/structure crosses from `HYPOTHESIS` to at least `PROBABLE`
before moving to the next code path, so the knowledge isn't lost or re-derived.
- **Whenever confidence changes** (up or down) — append to the Evidence log, don't overwrite.
- **Not** while it's still a pure guess with no evidence — a one-liner in the relevant
backlog/plan is enough until there's something to stand on.
## What to document
- **Functions/code paths** → `docs/re/functions/<name>.md` (one file per function or tight cluster).
- **Data structures / formats** → `docs/re/structures/<name>.md`.
- Keep the index in [`INDEX.md`](INDEX.md) (one line each: name · confidence · one-line summary).
Use the templates: [`_TEMPLATE.function.md`](_TEMPLATE.function.md),
[`_TEMPLATE.structure.md`](_TEMPLATE.structure.md).
---
## How we find and confirm code paths (the toolchain)
Everything joins on the **guest virtual address (PC)** — code addresses are fixed by the
XEX load, identical across our emulator and canary.
- **Static (cheap, try first):** `xenia-rs/sylpheed.db` (DuckDB: 25 481 functions, xrefs,
strings, vtables, imports). Query with `xenia-rs/zq.py``zq.py grep <str>`,
`zq.py xref <addr>`, `zq.py dis <lo> <hi>`, `zq.py fn <pc>`. Entry points are usually a
**string** (`zq.py grep MSG_DEMO`) or an **import** (movie/XMA API) xref'd back to the loader.
- **Dynamic (when static is ambiguous):** run `xenia-rs` with its probe suite —
`--pc-probe` / `--audit-pc-probe-hex` (fires at block entry), `--mem-watch` (mid-block
reads/writes of a VA), `--lr-trace` (call/return chains), `--trace-instructions`,
`--dump-addr` (read guest memory). These already exist; prefer them over hacking canary.
- **Oracle (correctness ground truth):** canary — the **Wine cross-build**
`xenia-canary/build-cross/bin/Windows/Debug/xenia_canary.exe` (the native Linux ELF
**crashes / does not run** — do not use it). This is the only emulator that reaches the
in-game menu; our `xenia-rs` never got past the intro video. Use canary to *observe output*
(capture its framebuffer for texture colours), not usually to instrument code — though its
`build-cross` toolchain does compile, so small C++ probes + rebuild are possible when needed.
Run **muted, one emulator process at a time**, point it at the real ISO (not the symlink).
> ⚠️ **VA-equality caveat:** join **code** by PC (fixed), but **never** assume a data VA
> holds the same bytes across emulators — allocators differ. Compare data by content/layout.

View File

@@ -0,0 +1,23 @@
# `<function name / sub_XXXXXXXX>`
- **Address:** `0x________` (in `sylpheed.db`; `zq.py fn 0x________`)
- **Confidence:** `HYPOTHESIS | PROBABLE | CONFIRMED`
- **Reimplemented in:** `crates/…/…rs::fn` — or *not yet*
- **Related:** other RE entries, `[[memory-slug]]`
## What it does
One-paragraph behavioural summary, in your own words. No pasted disassembly.
## Signature / calling convention
Inputs (registers/args + meaning), outputs, side effects (memory it writes, globals, imports it calls).
## Behaviour
Step-by-step of the observable behaviour — the *spec* to implement from. Formulas, branches,
state transitions. Cite addresses for detail (`the loop at 0x…`), don't transcribe code.
## Evidence log (append-only; newest last)
- `YYYY-MM-DD`*how we know*: static disasm read / `--lr-trace` from 0x… / canary output /
N observations. What it established. → confidence set to `…`.
## Open questions / what would raise confidence
The specific check that would confirm or refute the current claim.

View File

@@ -0,0 +1,29 @@
# `<structure / format name>`
- **Confidence:** `HYPOTHESIS | PROBABLE | CONFIRMED`
- **Parser in:** `crates/sylpheed-formats/src/…rs` — or *not yet*
- **Seen in:** which pak / asset / memory region
- **Related:** other RE entries, `[[memory-slug]]`
## Layout
```text
offset size type name notes
0x00 4 magic "____"
0x__ 4 u32be …
```
Endianness, alignment, variable-length rules, how child/section sizes are derived.
## Field semantics
What each field *means* and its observed value range. Distinguish **confirmed** fields from
**guessed** ones explicitly (a guessed field with a plausible value is still a guess).
## Coverage / limits
What fraction of real instances this decodes; which variants are deferred and why.
## Evidence log (append-only; newest last)
- `YYYY-MM-DD` — how established (forensic scan over N disc entries / round-trip test /
oracle comparison). → confidence `…`.
## Open questions / what would raise confidence
e.g. "colours unverified until compared against canary's framebuffer."

View File

@@ -0,0 +1,38 @@
# Texture colour interpretation — `k_8_8_8_8` (32bpp UI/HUD textures)
- **Confidence:** mixed — see per-claim tags below.
- **Parser in:** `sylpheed-formats/src/t8ad.rs`, `src/texture.rs` (XPR2)
- **Applies to:** T8aD 2D UI textures, XPR2 32bpp surfaces — the "colours look a bit odd" concern.
- **Related:** [[reborn-dynamic-re-plan]], INDEX rows T8aD / XPR2.
## What this resolves
Whether our decoded 32bpp textures use the right **sRGB-vs-linear** interpretation and the right
**channel order**. Ground truth = Canary's Vulkan texture cache, which is the code that actually
produces correct on-screen colour.
## Findings
### ✅ CONFIRMED — plain `k_8_8_8_8` is LINEAR (UNORM), never sRGB
Canary's host-format table maps plain `k_8_8_8_8``VK_FORMAT_R8G8B8A8_UNORM` with a straight
32-bit copy load shader (`kLoadShaderIndex32bpb`) and an **identity** format swizzle
(`XE_GPU_TEXTURE_SWIZZLE_RGBA`). There is **no** `..._SRGB` host format anywhere in the table;
gamma is a separate explicit path (`k_8_8_8_8_GAMMA_EDRAM` only). Therefore a plain 32bpp texture
is sampled as **raw UNORM bytes — apply no sRGB/gamma decode.**
*Implication:* if the viewer/engine applies an sRGB→linear (or linear→sRGB) step to these, that is
wrong. Show the bytes as-is (raw RGBA8), gamma only where the game explicitly requests it.
### ❔ HYPOTHESIS — exact source **channel order** (needs measurement, do not infer)
Our decoders currently assume on-disk **A8R8G8B8** and reorder `[a,r,g,b] → [r,g,b,a]`. Canary
proves the *host* target is R8G8B8A8 identity-swizzled, but the *guest→host* byte order also depends
on the texture's runtime **endian** field (e.g. `8in32`), which is set by the game at fetch time and
**cannot be read statically** from the format table. So whether our reorder matches is unverified.
*What would confirm/refute it:* one **measured** decoded texel — either capture Canary's
framebuffer over a texture with distinct R≠G≠B pixels of known colour, or patch Canary's already-
present-but-unwired `TextureDump` (`src/xenia/gpu/texture_dump.cc`) to dump the decoded texture and
diff bytes against our decode. **Measure, don't guess** — a symmetric colour (white logo) can't
disambiguate channel order.
## Evidence log
- `2026-07-11` — static read of Canary `src/xenia/gpu/vulkan/vulkan_texture_cache.cc` host-format
table + confirmed no `R8G8B8A8_SRGB` entry exists. → sRGB/UNORM claim `CONFIRMED`; channel-order
claim remains `HYPOTHESIS` pending a measured texel.