Claude
d96b225e47
Issue #49: the repos carry code, tooling and docs only. Untracks 143 screenshots, 3 savegame blobs and `tools/re-capture/ob_digits.png` (a digit-template sheet cut from game frames) -- 146 files, 76.4 MB. They stay in the working tree and are gitignored, so the pages' relative links still resolve where the captures exist and nothing ships. Derived measurements (csv/tsv/txt/log/json/jsonl/npy) are our own numbers, not game content, and stay tracked -- they are what most claims rest on. `check-capture-citations` had its contract inverted, and it is the half worth reading: * presence now comes from the WORKING TREE, not `git ls-files`. The assets are deliberately untracked, so asking the index would report every screenshot as missing and fail all 203 citations. * a NEW failure: a game asset that IS tracked. A screenshot that sneaks back in is invisible in review -- a binary shows as "Bin 0 -> 1234567 bytes" -- and is permanent once merged, since removing it later needs a history rewrite. So that half has to be loud. Verified: * selftest 8/8, including the new rule * scan: 212 present, 212 cited, 0 dangling, 0 tracked -> exit 0 * force-add one PNG -> "game assets TRACKED: 1", exit 1, selftest red ⚠️ This does NOT remove the blobs from history; a clone still fetches them. That needs a filter-repo rewrite and a force-push, which is a separate, human-run step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
131 lines
6.8 KiB
Markdown
131 lines
6.8 KiB
Markdown
# Reverse-engineering knowledge base
|
||
|
||
This directory is the **spec-side** of the clean-room: it records what the original
|
||
*Project Sylpheed* binary **does** (behaviour) and how its **data is laid out**, so that
|
||
the Rust port can be implemented **from these specs** without re-deriving anything and
|
||
without ever copying original code.
|
||
|
||
It exists to answer one question fast: *"do we already know how X works, and how sure are we?"*
|
||
|
||
---
|
||
|
||
## The one rule that matters
|
||
|
||
> **Never document a claim more confidently than the evidence supports, and never
|
||
> paste original code here.**
|
||
|
||
A wrong-but-confident note is worse than no note: someone builds on it and the bug hides
|
||
for weeks. Every entry therefore carries an explicit **confidence** and its **evidence**.
|
||
This mirrors the project method — *measure the oracle, never infer; refute before believing.*
|
||
|
||
### Captures are local-only (issue #49)
|
||
|
||
The repository carries **code, tooling and docs**. Screenshots and savegame
|
||
blobs are game-derived, so since 2026-09-19 they live in `docs/re/captures/`
|
||
on disk and are **gitignored** — the pages' relative links still resolve on a
|
||
machine that has them, and nothing ships.
|
||
|
||
Derived measurements (`csv`, `tsv`, `txt`, `log`, `json`, `jsonl`, `npy`) are
|
||
our own numbers rather than game content, and stay tracked — they are what most
|
||
claims here actually rest on.
|
||
|
||
`tools/re/check-capture-citations` enforces both halves: a cited capture must be
|
||
**present**, and a game asset must **not be tracked**. ⚠️ A fresh clone has no
|
||
captures, so its citations will not resolve until the captures are copied in;
|
||
that is expected, and the checker is a local gate rather than a CI one.
|
||
|
||
### Clean-room firewall
|
||
|
||
- ✅ Allowed: behaviour descriptions, field offsets/types, formulas, state machines,
|
||
observed input→output pairs, and **references** to the original by address
|
||
(`sub_821B68C0`) or to `sylpheed.db`.
|
||
- ❌ Forbidden here and in `crates/`: pasted decompiled C/C++ or verbatim disassembled
|
||
function bodies presented as the thing to reimplement. Cite the address; describe the
|
||
behaviour in your own words. Disassembly is a tool for *understanding*, not a source to copy.
|
||
|
||
---
|
||
|
||
## Confidence levels
|
||
|
||
| Level | Meaning | Bar to reach it |
|
||
|-------|---------|-----------------|
|
||
| `CONFIRMED` | Behaviour verified against ground truth. | ≥2 independent observations **or** one observation cross-checked against an oracle (canary framebuffer, a known-correct value, a second code path). |
|
||
| `PROBABLE` | Strong single-source inference. | One clean observation, or an unambiguous static read of the disassembly. |
|
||
| `HYPOTHESIS` | Educated guess, not yet tested. | Anything else. Must say what would confirm/refute it. |
|
||
|
||
### The status markers
|
||
|
||
The table above is the *confidence* scale. The markers that appear in
|
||
`BACKLOG.md` and the `structures/` pages are a separate, and until now undefined,
|
||
vocabulary. They mean:
|
||
|
||
| Marker | Meaning |
|
||
|---|---|
|
||
| ✅ | Confirmed — verified against ground truth. |
|
||
| 🟡 | Partial: true as far as it goes, or true under a stated assumption. |
|
||
| ❔ | Open question. Nobody has answered it yet. |
|
||
| 🔴 | **Refuted** — shown false — **or blocked by something the container cannot do.** |
|
||
| ❌ | A specific claim that was tried and failed. Prefer 🔴. |
|
||
| 🚧 | Work started and not finished. |
|
||
|
||
**🔴 never means "we have not run it yet."** That is ❔ or 🚧. Reserve 🔴's
|
||
"blocked" sense for a real limit of the box — no push credentials, no hardware
|
||
Vulkan (lavapipe only), or a decision only the user can make. The box *can* run
|
||
the emulator, script input, screenshot, read guest memory, and build and test
|
||
Rust, so "needs a run" is never a blocker. This paragraph exists because the
|
||
marker was undefined for 98 uses and three of them were mislabelled that way.
|
||
|
||
**Promotion requires new evidence, not re-reading the old evidence.** A `HYPOTHESIS` that
|
||
"looks right again" is still a `HYPOTHESIS`. Only an *independent* check promotes it.
|
||
If evidence later contradicts an entry, **demote it and record the contradiction** — do
|
||
not silently edit the conclusion.
|
||
|
||
---
|
||
|
||
## When to document
|
||
|
||
- **Right after** a function/structure crosses from `HYPOTHESIS` to at least `PROBABLE` —
|
||
before moving to the next code path, so the knowledge isn't lost or re-derived.
|
||
- **Whenever confidence changes** (up or down) — append to the Evidence log, don't overwrite.
|
||
- **Not** while it's still a pure guess with no evidence — a one-liner in the relevant
|
||
backlog/plan is enough until there's something to stand on.
|
||
|
||
## What to document
|
||
|
||
- **Functions/code paths** → `docs/re/functions/<name>.md` (one file per function or tight cluster).
|
||
- **Data structures / formats** → `docs/re/structures/<name>.md`.
|
||
- Keep the index in [`INDEX.md`](INDEX.md) (one line each: name · confidence · one-line summary).
|
||
|
||
Use the templates: [`_TEMPLATE.function.md`](_TEMPLATE.function.md),
|
||
[`_TEMPLATE.structure.md`](_TEMPLATE.structure.md).
|
||
|
||
---
|
||
|
||
## How we find and confirm code paths (the toolchain)
|
||
|
||
Everything joins on the **guest virtual address (PC)** — code addresses are fixed by the
|
||
XEX load, identical across our emulator and canary.
|
||
|
||
- **Static (cheap, try first):** `sylpheed.db` at the repository root (DuckDB: 25 676
|
||
functions, xrefs, strings, vtables, imports; git-ignored — build it with
|
||
`sylph-xexdb dis <ISO> --db sylpheed.db --analyze sql --quiet`, then re-stamp names with
|
||
`tools/apply_re_symbols.sql`). Query with `tools/zq.py` — `zq.py grep <str>`,
|
||
`zq.py xref <addr>`, `zq.py dis <lo> <hi>`, `zq.py fn <pc>`. Entry points are usually a
|
||
**string** (`zq.py grep MSG_DEMO`) or an **import** (movie/XMA API) xref'd back to the loader.
|
||
- **Dynamic (when static is ambiguous):** Xenia Canary, below. The `xenia-rs` probe suite
|
||
this bullet used to recommend (`--pc-probe`, `--mem-watch`, `--lr-trace`,
|
||
`--dump-addr`) went with that emulator when it was retired on 2026-09-16. For static
|
||
`.rdata` reads that `--dump-addr` served, read the `.pe` directly: it is a flat VA dump,
|
||
file offset = `VA − 0x82000000`.
|
||
- **Oracle (correctness ground truth):** canary, in either of two builds — the **Wine
|
||
cross-build** in `xenia-canary/build-cross/bin/Windows/Debug/` (launched by
|
||
`run-canary-safe.sh`) or the **native Linux build** in `xenia-canary-native/` (launched by
|
||
`run-canary-native-safe.sh`). This is the only emulator that reaches the in-game menu; our
|
||
retired `xenia-rs` never got past the intro video. Use canary to *observe output*
|
||
(capture its framebuffer for texture colours), not usually to instrument code — though both
|
||
builds compile, so small C++ probes + rebuild are possible when needed.
|
||
Run **muted, one emulator process at a time**, point it at the real ISO.
|
||
|
||
> ⚠️ **VA-equality caveat:** join **code** by PC (fixed), but **never** assume a data VA
|
||
> holds the same bytes across emulators — allocators differ. Compare data by content/layout.
|