docs(re): add reverse-engineering knowledge base + conventions
Some checks failed
Some checks failed
Establish docs/re/ as the spec-side of the clean-room: - README: confidence tiers (HYPOTHESIS/PROBABLE/CONFIRMED), promotion requires NEW independent evidence, append-only evidence logs, demote-on-contradiction, and a clean-room firewall (describe behaviour + cite addresses; never paste decompiled code). Documents the toolchain (sylpheed.db + zq.py, xenia-rs probes, Wine-Canary oracle) and the join-by-guest-PC method with the VA-equality caveat. - Function/structure templates + a seeded INDEX (9 already-reversed formats). - First entry: texture-color-k8888 — CONFIRMED plain k_8_8_8_8 is linear UNORM (no sRGB), channel order still to be measured. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
29
docs/re/INDEX.md
Normal file
29
docs/re/INDEX.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# RE knowledge index
|
||||
|
||||
Confidence: ✅ `CONFIRMED` · 🟡 `PROBABLE` · ❔ `HYPOTHESIS`. See [README](README.md).
|
||||
|
||||
Formats we've already reversed are, for now, **documented by their parser + disc round-trip
|
||||
tests** (the executable spec) rather than a prose file — the "Spec" column points there.
|
||||
Promote to a prose `structures/…md` file when a format needs behavioural notes beyond layout.
|
||||
|
||||
## Data structures / formats
|
||||
|
||||
| Format | Conf. | Spec (parser + tests) | Notes |
|
||||
|--------|-------|-----------------------|-------|
|
||||
| IPFB `.pak` archive | ✅ | `sylpheed-formats/src/pak.rs` + `tests/pak_idxd_disc.rs` | header + 12-byte TOC, Z1/zlib payloads |
|
||||
| name-hash (TOC keys) | ✅ | `sylpheed-formats/src/hash.rs` | Barrett-reduction hash; recovers original paths |
|
||||
| IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` | self-describing; ship/weapon stats verified vs known values |
|
||||
| XPR2 texture + cubemap | 🟡 | `sylpheed-formats/src/texture.rs` | de-tile + A8R8G8B8; **colours unverified** (dynamic item) |
|
||||
| T8aD 2D texture | 🟡 | `sylpheed-formats/src/t8ad.rs` | ~85% decode; **colours unverified**; ~15% variants deferred |
|
||||
| RATC bundle | 🟡 | `sylpheed-formats/src/ratc.rs` | child listing confirmed; one level deep |
|
||||
| LSTA sprite list | 🟡 | `sylpheed-formats/src/lsta.rs` | inline T8aD frames |
|
||||
| IXUD subtitle | 🟡 | `sylpheed-formats/src/ixud.rs` | timed cues; **movie↔track link unknown** (dynamic item) |
|
||||
| Fonts (ttf/otf/ttc) | ✅ | `sylpheed-formats/src/font.rs` | standard OpenType, parsed via ttf-parser |
|
||||
|
||||
## Functions / code paths
|
||||
|
||||
_None documented yet — populated during the dynamic-RE phase._
|
||||
|
||||
| Function | Conf. | Reimpl. | Summary |
|
||||
|----------|-------|---------|---------|
|
||||
| — | — | — | — |
|
||||
88
docs/re/README.md
Normal file
88
docs/re/README.md
Normal file
@@ -0,0 +1,88 @@
|
||||
# Reverse-engineering knowledge base
|
||||
|
||||
This directory is the **spec-side** of the clean-room: it records what the original
|
||||
*Project Sylpheed* binary **does** (behaviour) and how its **data is laid out**, so that
|
||||
the Rust port can be implemented **from these specs** without re-deriving anything and
|
||||
without ever copying original code.
|
||||
|
||||
It exists to answer one question fast: *"do we already know how X works, and how sure are we?"*
|
||||
|
||||
---
|
||||
|
||||
## The one rule that matters
|
||||
|
||||
> **Never document a claim more confidently than the evidence supports, and never
|
||||
> paste original code here.**
|
||||
|
||||
A wrong-but-confident note is worse than no note: someone builds on it and the bug hides
|
||||
for weeks. Every entry therefore carries an explicit **confidence** and its **evidence**.
|
||||
This mirrors the project method — *measure the oracle, never infer; refute before believing.*
|
||||
|
||||
### Clean-room firewall
|
||||
|
||||
- ✅ Allowed: behaviour descriptions, field offsets/types, formulas, state machines,
|
||||
observed input→output pairs, and **references** to the original by address
|
||||
(`sub_821B68C0`) or to `xenia-rs/sylpheed.db`.
|
||||
- ❌ Forbidden here and in `crates/`: pasted decompiled C/C++ or verbatim disassembled
|
||||
function bodies presented as the thing to reimplement. Cite the address; describe the
|
||||
behaviour in your own words. Disassembly is a tool for *understanding*, not a source to copy.
|
||||
|
||||
---
|
||||
|
||||
## Confidence levels
|
||||
|
||||
| Level | Meaning | Bar to reach it |
|
||||
|-------|---------|-----------------|
|
||||
| `CONFIRMED` | Behaviour verified against ground truth. | ≥2 independent observations **or** one observation cross-checked against an oracle (canary framebuffer, a known-correct value, a second code path). |
|
||||
| `PROBABLE` | Strong single-source inference. | One clean observation, or an unambiguous static read of the disassembly. |
|
||||
| `HYPOTHESIS` | Educated guess, not yet tested. | Anything else. Must say what would confirm/refute it. |
|
||||
|
||||
**Promotion requires new evidence, not re-reading the old evidence.** A `HYPOTHESIS` that
|
||||
"looks right again" is still a `HYPOTHESIS`. Only an *independent* check promotes it.
|
||||
If evidence later contradicts an entry, **demote it and record the contradiction** — do
|
||||
not silently edit the conclusion.
|
||||
|
||||
---
|
||||
|
||||
## When to document
|
||||
|
||||
- **Right after** a function/structure crosses from `HYPOTHESIS` to at least `PROBABLE` —
|
||||
before moving to the next code path, so the knowledge isn't lost or re-derived.
|
||||
- **Whenever confidence changes** (up or down) — append to the Evidence log, don't overwrite.
|
||||
- **Not** while it's still a pure guess with no evidence — a one-liner in the relevant
|
||||
backlog/plan is enough until there's something to stand on.
|
||||
|
||||
## What to document
|
||||
|
||||
- **Functions/code paths** → `docs/re/functions/<name>.md` (one file per function or tight cluster).
|
||||
- **Data structures / formats** → `docs/re/structures/<name>.md`.
|
||||
- Keep the index in [`INDEX.md`](INDEX.md) (one line each: name · confidence · one-line summary).
|
||||
|
||||
Use the templates: [`_TEMPLATE.function.md`](_TEMPLATE.function.md),
|
||||
[`_TEMPLATE.structure.md`](_TEMPLATE.structure.md).
|
||||
|
||||
---
|
||||
|
||||
## How we find and confirm code paths (the toolchain)
|
||||
|
||||
Everything joins on the **guest virtual address (PC)** — code addresses are fixed by the
|
||||
XEX load, identical across our emulator and canary.
|
||||
|
||||
- **Static (cheap, try first):** `xenia-rs/sylpheed.db` (DuckDB: 25 481 functions, xrefs,
|
||||
strings, vtables, imports). Query with `xenia-rs/zq.py` — `zq.py grep <str>`,
|
||||
`zq.py xref <addr>`, `zq.py dis <lo> <hi>`, `zq.py fn <pc>`. Entry points are usually a
|
||||
**string** (`zq.py grep MSG_DEMO`) or an **import** (movie/XMA API) xref'd back to the loader.
|
||||
- **Dynamic (when static is ambiguous):** run `xenia-rs` with its probe suite —
|
||||
`--pc-probe` / `--audit-pc-probe-hex` (fires at block entry), `--mem-watch` (mid-block
|
||||
reads/writes of a VA), `--lr-trace` (call/return chains), `--trace-instructions`,
|
||||
`--dump-addr` (read guest memory). These already exist; prefer them over hacking canary.
|
||||
- **Oracle (correctness ground truth):** canary — the **Wine cross-build**
|
||||
`xenia-canary/build-cross/bin/Windows/Debug/xenia_canary.exe` (the native Linux ELF
|
||||
**crashes / does not run** — do not use it). This is the only emulator that reaches the
|
||||
in-game menu; our `xenia-rs` never got past the intro video. Use canary to *observe output*
|
||||
(capture its framebuffer for texture colours), not usually to instrument code — though its
|
||||
`build-cross` toolchain does compile, so small C++ probes + rebuild are possible when needed.
|
||||
Run **muted, one emulator process at a time**, point it at the real ISO (not the symlink).
|
||||
|
||||
> ⚠️ **VA-equality caveat:** join **code** by PC (fixed), but **never** assume a data VA
|
||||
> holds the same bytes across emulators — allocators differ. Compare data by content/layout.
|
||||
23
docs/re/_TEMPLATE.function.md
Normal file
23
docs/re/_TEMPLATE.function.md
Normal file
@@ -0,0 +1,23 @@
|
||||
# `<function name / sub_XXXXXXXX>`
|
||||
|
||||
- **Address:** `0x________` (in `sylpheed.db`; `zq.py fn 0x________`)
|
||||
- **Confidence:** `HYPOTHESIS | PROBABLE | CONFIRMED`
|
||||
- **Reimplemented in:** `crates/…/…rs::fn` — or *not yet*
|
||||
- **Related:** other RE entries, `[[memory-slug]]`
|
||||
|
||||
## What it does
|
||||
One-paragraph behavioural summary, in your own words. No pasted disassembly.
|
||||
|
||||
## Signature / calling convention
|
||||
Inputs (registers/args + meaning), outputs, side effects (memory it writes, globals, imports it calls).
|
||||
|
||||
## Behaviour
|
||||
Step-by-step of the observable behaviour — the *spec* to implement from. Formulas, branches,
|
||||
state transitions. Cite addresses for detail (`the loop at 0x…`), don't transcribe code.
|
||||
|
||||
## Evidence log (append-only; newest last)
|
||||
- `YYYY-MM-DD` — *how we know*: static disasm read / `--lr-trace` from 0x… / canary output /
|
||||
N observations. What it established. → confidence set to `…`.
|
||||
|
||||
## Open questions / what would raise confidence
|
||||
The specific check that would confirm or refute the current claim.
|
||||
29
docs/re/_TEMPLATE.structure.md
Normal file
29
docs/re/_TEMPLATE.structure.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# `<structure / format name>`
|
||||
|
||||
- **Confidence:** `HYPOTHESIS | PROBABLE | CONFIRMED`
|
||||
- **Parser in:** `crates/sylpheed-formats/src/…rs` — or *not yet*
|
||||
- **Seen in:** which pak / asset / memory region
|
||||
- **Related:** other RE entries, `[[memory-slug]]`
|
||||
|
||||
## Layout
|
||||
```text
|
||||
offset size type name notes
|
||||
0x00 4 magic "____"
|
||||
0x__ 4 u32be …
|
||||
…
|
||||
```
|
||||
Endianness, alignment, variable-length rules, how child/section sizes are derived.
|
||||
|
||||
## Field semantics
|
||||
What each field *means* and its observed value range. Distinguish **confirmed** fields from
|
||||
**guessed** ones explicitly (a guessed field with a plausible value is still a guess).
|
||||
|
||||
## Coverage / limits
|
||||
What fraction of real instances this decodes; which variants are deferred and why.
|
||||
|
||||
## Evidence log (append-only; newest last)
|
||||
- `YYYY-MM-DD` — how established (forensic scan over N disc entries / round-trip test /
|
||||
oracle comparison). → confidence `…`.
|
||||
|
||||
## Open questions / what would raise confidence
|
||||
e.g. "colours unverified until compared against canary's framebuffer."
|
||||
38
docs/re/structures/texture-color-k8888.md
Normal file
38
docs/re/structures/texture-color-k8888.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# Texture colour interpretation — `k_8_8_8_8` (32bpp UI/HUD textures)
|
||||
|
||||
- **Confidence:** mixed — see per-claim tags below.
|
||||
- **Parser in:** `sylpheed-formats/src/t8ad.rs`, `src/texture.rs` (XPR2)
|
||||
- **Applies to:** T8aD 2D UI textures, XPR2 32bpp surfaces — the "colours look a bit odd" concern.
|
||||
- **Related:** [[reborn-dynamic-re-plan]], INDEX rows T8aD / XPR2.
|
||||
|
||||
## What this resolves
|
||||
Whether our decoded 32bpp textures use the right **sRGB-vs-linear** interpretation and the right
|
||||
**channel order**. Ground truth = Canary's Vulkan texture cache, which is the code that actually
|
||||
produces correct on-screen colour.
|
||||
|
||||
## Findings
|
||||
|
||||
### ✅ CONFIRMED — plain `k_8_8_8_8` is LINEAR (UNORM), never sRGB
|
||||
Canary's host-format table maps plain `k_8_8_8_8` → `VK_FORMAT_R8G8B8A8_UNORM` with a straight
|
||||
32-bit copy load shader (`kLoadShaderIndex32bpb`) and an **identity** format swizzle
|
||||
(`XE_GPU_TEXTURE_SWIZZLE_RGBA`). There is **no** `..._SRGB` host format anywhere in the table;
|
||||
gamma is a separate explicit path (`k_8_8_8_8_GAMMA_EDRAM` only). Therefore a plain 32bpp texture
|
||||
is sampled as **raw UNORM bytes — apply no sRGB/gamma decode.**
|
||||
→ *Implication:* if the viewer/engine applies an sRGB→linear (or linear→sRGB) step to these, that is
|
||||
wrong. Show the bytes as-is (raw RGBA8), gamma only where the game explicitly requests it.
|
||||
|
||||
### ❔ HYPOTHESIS — exact source **channel order** (needs measurement, do not infer)
|
||||
Our decoders currently assume on-disk **A8R8G8B8** and reorder `[a,r,g,b] → [r,g,b,a]`. Canary
|
||||
proves the *host* target is R8G8B8A8 identity-swizzled, but the *guest→host* byte order also depends
|
||||
on the texture's runtime **endian** field (e.g. `8in32`), which is set by the game at fetch time and
|
||||
**cannot be read statically** from the format table. So whether our reorder matches is unverified.
|
||||
→ *What would confirm/refute it:* one **measured** decoded texel — either capture Canary's
|
||||
framebuffer over a texture with distinct R≠G≠B pixels of known colour, or patch Canary's already-
|
||||
present-but-unwired `TextureDump` (`src/xenia/gpu/texture_dump.cc`) to dump the decoded texture and
|
||||
diff bytes against our decode. **Measure, don't guess** — a symmetric colour (white logo) can't
|
||||
disambiguate channel order.
|
||||
|
||||
## Evidence log
|
||||
- `2026-07-11` — static read of Canary `src/xenia/gpu/vulkan/vulkan_texture_cache.cc` host-format
|
||||
table + confirmed no `R8G8B8A8_SRGB` entry exists. → sRGB/UNORM claim `CONFIRMED`; channel-order
|
||||
claim remains `HYPOTHESIS` pending a measured texel.
|
||||
Reference in New Issue
Block a user