docs(re): add reverse-engineering knowledge base + conventions
Some checks failed
Some checks failed
Establish docs/re/ as the spec-side of the clean-room: - README: confidence tiers (HYPOTHESIS/PROBABLE/CONFIRMED), promotion requires NEW independent evidence, append-only evidence logs, demote-on-contradiction, and a clean-room firewall (describe behaviour + cite addresses; never paste decompiled code). Documents the toolchain (sylpheed.db + zq.py, xenia-rs probes, Wine-Canary oracle) and the join-by-guest-PC method with the VA-equality caveat. - Function/structure templates + a seeded INDEX (9 already-reversed formats). - First entry: texture-color-k8888 — CONFIRMED plain k_8_8_8_8 is linear UNORM (no sRGB), channel order still to be measured. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
29
docs/re/INDEX.md
Normal file
29
docs/re/INDEX.md
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
# RE knowledge index
|
||||||
|
|
||||||
|
Confidence: ✅ `CONFIRMED` · 🟡 `PROBABLE` · ❔ `HYPOTHESIS`. See [README](README.md).
|
||||||
|
|
||||||
|
Formats we've already reversed are, for now, **documented by their parser + disc round-trip
|
||||||
|
tests** (the executable spec) rather than a prose file — the "Spec" column points there.
|
||||||
|
Promote to a prose `structures/…md` file when a format needs behavioural notes beyond layout.
|
||||||
|
|
||||||
|
## Data structures / formats
|
||||||
|
|
||||||
|
| Format | Conf. | Spec (parser + tests) | Notes |
|
||||||
|
|--------|-------|-----------------------|-------|
|
||||||
|
| IPFB `.pak` archive | ✅ | `sylpheed-formats/src/pak.rs` + `tests/pak_idxd_disc.rs` | header + 12-byte TOC, Z1/zlib payloads |
|
||||||
|
| name-hash (TOC keys) | ✅ | `sylpheed-formats/src/hash.rs` | Barrett-reduction hash; recovers original paths |
|
||||||
|
| IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` | self-describing; ship/weapon stats verified vs known values |
|
||||||
|
| XPR2 texture + cubemap | 🟡 | `sylpheed-formats/src/texture.rs` | de-tile + A8R8G8B8; **colours unverified** (dynamic item) |
|
||||||
|
| T8aD 2D texture | 🟡 | `sylpheed-formats/src/t8ad.rs` | ~85% decode; **colours unverified**; ~15% variants deferred |
|
||||||
|
| RATC bundle | 🟡 | `sylpheed-formats/src/ratc.rs` | child listing confirmed; one level deep |
|
||||||
|
| LSTA sprite list | 🟡 | `sylpheed-formats/src/lsta.rs` | inline T8aD frames |
|
||||||
|
| IXUD subtitle | 🟡 | `sylpheed-formats/src/ixud.rs` | timed cues; **movie↔track link unknown** (dynamic item) |
|
||||||
|
| Fonts (ttf/otf/ttc) | ✅ | `sylpheed-formats/src/font.rs` | standard OpenType, parsed via ttf-parser |
|
||||||
|
|
||||||
|
## Functions / code paths
|
||||||
|
|
||||||
|
_None documented yet — populated during the dynamic-RE phase._
|
||||||
|
|
||||||
|
| Function | Conf. | Reimpl. | Summary |
|
||||||
|
|----------|-------|---------|---------|
|
||||||
|
| — | — | — | — |
|
||||||
88
docs/re/README.md
Normal file
88
docs/re/README.md
Normal file
@@ -0,0 +1,88 @@
|
|||||||
|
# Reverse-engineering knowledge base
|
||||||
|
|
||||||
|
This directory is the **spec-side** of the clean-room: it records what the original
|
||||||
|
*Project Sylpheed* binary **does** (behaviour) and how its **data is laid out**, so that
|
||||||
|
the Rust port can be implemented **from these specs** without re-deriving anything and
|
||||||
|
without ever copying original code.
|
||||||
|
|
||||||
|
It exists to answer one question fast: *"do we already know how X works, and how sure are we?"*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The one rule that matters
|
||||||
|
|
||||||
|
> **Never document a claim more confidently than the evidence supports, and never
|
||||||
|
> paste original code here.**
|
||||||
|
|
||||||
|
A wrong-but-confident note is worse than no note: someone builds on it and the bug hides
|
||||||
|
for weeks. Every entry therefore carries an explicit **confidence** and its **evidence**.
|
||||||
|
This mirrors the project method — *measure the oracle, never infer; refute before believing.*
|
||||||
|
|
||||||
|
### Clean-room firewall
|
||||||
|
|
||||||
|
- ✅ Allowed: behaviour descriptions, field offsets/types, formulas, state machines,
|
||||||
|
observed input→output pairs, and **references** to the original by address
|
||||||
|
(`sub_821B68C0`) or to `xenia-rs/sylpheed.db`.
|
||||||
|
- ❌ Forbidden here and in `crates/`: pasted decompiled C/C++ or verbatim disassembled
|
||||||
|
function bodies presented as the thing to reimplement. Cite the address; describe the
|
||||||
|
behaviour in your own words. Disassembly is a tool for *understanding*, not a source to copy.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Confidence levels
|
||||||
|
|
||||||
|
| Level | Meaning | Bar to reach it |
|
||||||
|
|-------|---------|-----------------|
|
||||||
|
| `CONFIRMED` | Behaviour verified against ground truth. | ≥2 independent observations **or** one observation cross-checked against an oracle (canary framebuffer, a known-correct value, a second code path). |
|
||||||
|
| `PROBABLE` | Strong single-source inference. | One clean observation, or an unambiguous static read of the disassembly. |
|
||||||
|
| `HYPOTHESIS` | Educated guess, not yet tested. | Anything else. Must say what would confirm/refute it. |
|
||||||
|
|
||||||
|
**Promotion requires new evidence, not re-reading the old evidence.** A `HYPOTHESIS` that
|
||||||
|
"looks right again" is still a `HYPOTHESIS`. Only an *independent* check promotes it.
|
||||||
|
If evidence later contradicts an entry, **demote it and record the contradiction** — do
|
||||||
|
not silently edit the conclusion.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## When to document
|
||||||
|
|
||||||
|
- **Right after** a function/structure crosses from `HYPOTHESIS` to at least `PROBABLE` —
|
||||||
|
before moving to the next code path, so the knowledge isn't lost or re-derived.
|
||||||
|
- **Whenever confidence changes** (up or down) — append to the Evidence log, don't overwrite.
|
||||||
|
- **Not** while it's still a pure guess with no evidence — a one-liner in the relevant
|
||||||
|
backlog/plan is enough until there's something to stand on.
|
||||||
|
|
||||||
|
## What to document
|
||||||
|
|
||||||
|
- **Functions/code paths** → `docs/re/functions/<name>.md` (one file per function or tight cluster).
|
||||||
|
- **Data structures / formats** → `docs/re/structures/<name>.md`.
|
||||||
|
- Keep the index in [`INDEX.md`](INDEX.md) (one line each: name · confidence · one-line summary).
|
||||||
|
|
||||||
|
Use the templates: [`_TEMPLATE.function.md`](_TEMPLATE.function.md),
|
||||||
|
[`_TEMPLATE.structure.md`](_TEMPLATE.structure.md).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How we find and confirm code paths (the toolchain)
|
||||||
|
|
||||||
|
Everything joins on the **guest virtual address (PC)** — code addresses are fixed by the
|
||||||
|
XEX load, identical across our emulator and canary.
|
||||||
|
|
||||||
|
- **Static (cheap, try first):** `xenia-rs/sylpheed.db` (DuckDB: 25 481 functions, xrefs,
|
||||||
|
strings, vtables, imports). Query with `xenia-rs/zq.py` — `zq.py grep <str>`,
|
||||||
|
`zq.py xref <addr>`, `zq.py dis <lo> <hi>`, `zq.py fn <pc>`. Entry points are usually a
|
||||||
|
**string** (`zq.py grep MSG_DEMO`) or an **import** (movie/XMA API) xref'd back to the loader.
|
||||||
|
- **Dynamic (when static is ambiguous):** run `xenia-rs` with its probe suite —
|
||||||
|
`--pc-probe` / `--audit-pc-probe-hex` (fires at block entry), `--mem-watch` (mid-block
|
||||||
|
reads/writes of a VA), `--lr-trace` (call/return chains), `--trace-instructions`,
|
||||||
|
`--dump-addr` (read guest memory). These already exist; prefer them over hacking canary.
|
||||||
|
- **Oracle (correctness ground truth):** canary — the **Wine cross-build**
|
||||||
|
`xenia-canary/build-cross/bin/Windows/Debug/xenia_canary.exe` (the native Linux ELF
|
||||||
|
**crashes / does not run** — do not use it). This is the only emulator that reaches the
|
||||||
|
in-game menu; our `xenia-rs` never got past the intro video. Use canary to *observe output*
|
||||||
|
(capture its framebuffer for texture colours), not usually to instrument code — though its
|
||||||
|
`build-cross` toolchain does compile, so small C++ probes + rebuild are possible when needed.
|
||||||
|
Run **muted, one emulator process at a time**, point it at the real ISO (not the symlink).
|
||||||
|
|
||||||
|
> ⚠️ **VA-equality caveat:** join **code** by PC (fixed), but **never** assume a data VA
|
||||||
|
> holds the same bytes across emulators — allocators differ. Compare data by content/layout.
|
||||||
23
docs/re/_TEMPLATE.function.md
Normal file
23
docs/re/_TEMPLATE.function.md
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
# `<function name / sub_XXXXXXXX>`
|
||||||
|
|
||||||
|
- **Address:** `0x________` (in `sylpheed.db`; `zq.py fn 0x________`)
|
||||||
|
- **Confidence:** `HYPOTHESIS | PROBABLE | CONFIRMED`
|
||||||
|
- **Reimplemented in:** `crates/…/…rs::fn` — or *not yet*
|
||||||
|
- **Related:** other RE entries, `[[memory-slug]]`
|
||||||
|
|
||||||
|
## What it does
|
||||||
|
One-paragraph behavioural summary, in your own words. No pasted disassembly.
|
||||||
|
|
||||||
|
## Signature / calling convention
|
||||||
|
Inputs (registers/args + meaning), outputs, side effects (memory it writes, globals, imports it calls).
|
||||||
|
|
||||||
|
## Behaviour
|
||||||
|
Step-by-step of the observable behaviour — the *spec* to implement from. Formulas, branches,
|
||||||
|
state transitions. Cite addresses for detail (`the loop at 0x…`), don't transcribe code.
|
||||||
|
|
||||||
|
## Evidence log (append-only; newest last)
|
||||||
|
- `YYYY-MM-DD` — *how we know*: static disasm read / `--lr-trace` from 0x… / canary output /
|
||||||
|
N observations. What it established. → confidence set to `…`.
|
||||||
|
|
||||||
|
## Open questions / what would raise confidence
|
||||||
|
The specific check that would confirm or refute the current claim.
|
||||||
29
docs/re/_TEMPLATE.structure.md
Normal file
29
docs/re/_TEMPLATE.structure.md
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
# `<structure / format name>`
|
||||||
|
|
||||||
|
- **Confidence:** `HYPOTHESIS | PROBABLE | CONFIRMED`
|
||||||
|
- **Parser in:** `crates/sylpheed-formats/src/…rs` — or *not yet*
|
||||||
|
- **Seen in:** which pak / asset / memory region
|
||||||
|
- **Related:** other RE entries, `[[memory-slug]]`
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
```text
|
||||||
|
offset size type name notes
|
||||||
|
0x00 4 magic "____"
|
||||||
|
0x__ 4 u32be …
|
||||||
|
…
|
||||||
|
```
|
||||||
|
Endianness, alignment, variable-length rules, how child/section sizes are derived.
|
||||||
|
|
||||||
|
## Field semantics
|
||||||
|
What each field *means* and its observed value range. Distinguish **confirmed** fields from
|
||||||
|
**guessed** ones explicitly (a guessed field with a plausible value is still a guess).
|
||||||
|
|
||||||
|
## Coverage / limits
|
||||||
|
What fraction of real instances this decodes; which variants are deferred and why.
|
||||||
|
|
||||||
|
## Evidence log (append-only; newest last)
|
||||||
|
- `YYYY-MM-DD` — how established (forensic scan over N disc entries / round-trip test /
|
||||||
|
oracle comparison). → confidence `…`.
|
||||||
|
|
||||||
|
## Open questions / what would raise confidence
|
||||||
|
e.g. "colours unverified until compared against canary's framebuffer."
|
||||||
38
docs/re/structures/texture-color-k8888.md
Normal file
38
docs/re/structures/texture-color-k8888.md
Normal file
@@ -0,0 +1,38 @@
|
|||||||
|
# Texture colour interpretation — `k_8_8_8_8` (32bpp UI/HUD textures)
|
||||||
|
|
||||||
|
- **Confidence:** mixed — see per-claim tags below.
|
||||||
|
- **Parser in:** `sylpheed-formats/src/t8ad.rs`, `src/texture.rs` (XPR2)
|
||||||
|
- **Applies to:** T8aD 2D UI textures, XPR2 32bpp surfaces — the "colours look a bit odd" concern.
|
||||||
|
- **Related:** [[reborn-dynamic-re-plan]], INDEX rows T8aD / XPR2.
|
||||||
|
|
||||||
|
## What this resolves
|
||||||
|
Whether our decoded 32bpp textures use the right **sRGB-vs-linear** interpretation and the right
|
||||||
|
**channel order**. Ground truth = Canary's Vulkan texture cache, which is the code that actually
|
||||||
|
produces correct on-screen colour.
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### ✅ CONFIRMED — plain `k_8_8_8_8` is LINEAR (UNORM), never sRGB
|
||||||
|
Canary's host-format table maps plain `k_8_8_8_8` → `VK_FORMAT_R8G8B8A8_UNORM` with a straight
|
||||||
|
32-bit copy load shader (`kLoadShaderIndex32bpb`) and an **identity** format swizzle
|
||||||
|
(`XE_GPU_TEXTURE_SWIZZLE_RGBA`). There is **no** `..._SRGB` host format anywhere in the table;
|
||||||
|
gamma is a separate explicit path (`k_8_8_8_8_GAMMA_EDRAM` only). Therefore a plain 32bpp texture
|
||||||
|
is sampled as **raw UNORM bytes — apply no sRGB/gamma decode.**
|
||||||
|
→ *Implication:* if the viewer/engine applies an sRGB→linear (or linear→sRGB) step to these, that is
|
||||||
|
wrong. Show the bytes as-is (raw RGBA8), gamma only where the game explicitly requests it.
|
||||||
|
|
||||||
|
### ❔ HYPOTHESIS — exact source **channel order** (needs measurement, do not infer)
|
||||||
|
Our decoders currently assume on-disk **A8R8G8B8** and reorder `[a,r,g,b] → [r,g,b,a]`. Canary
|
||||||
|
proves the *host* target is R8G8B8A8 identity-swizzled, but the *guest→host* byte order also depends
|
||||||
|
on the texture's runtime **endian** field (e.g. `8in32`), which is set by the game at fetch time and
|
||||||
|
**cannot be read statically** from the format table. So whether our reorder matches is unverified.
|
||||||
|
→ *What would confirm/refute it:* one **measured** decoded texel — either capture Canary's
|
||||||
|
framebuffer over a texture with distinct R≠G≠B pixels of known colour, or patch Canary's already-
|
||||||
|
present-but-unwired `TextureDump` (`src/xenia/gpu/texture_dump.cc`) to dump the decoded texture and
|
||||||
|
diff bytes against our decode. **Measure, don't guess** — a symmetric colour (white logo) can't
|
||||||
|
disambiguate channel order.
|
||||||
|
|
||||||
|
## Evidence log
|
||||||
|
- `2026-07-11` — static read of Canary `src/xenia/gpu/vulkan/vulkan_texture_cache.cc` host-format
|
||||||
|
table + confirmed no `R8G8B8A8_SRGB` entry exists. → sRGB/UNORM claim `CONFIRMED`; channel-order
|
||||||
|
claim remains `HYPOTHESIS` pending a measured texel.
|
||||||
Reference in New Issue
Block a user