dedcf37867
The generator had not been able to run correctly since the manual moved into
`tools/ppc-manual/`: it computed the repository root as `HERE.parent.parent`,
which now names `tools/`, so the XML, Canary's emitters and xenia-rs all stopped
resolving — silently, because both scrapers skipped what they could not find.
Every page's references had been pointing at paths that exist nowhere.
What each source contributed, measured on the 350 pages before this change:
Operation (pseudocode) 251 pages: fixed boilerplate "derives from the xenia-rs
interpreter"; 99 carry real hand-written seeds
C translation 337 pages: the same kind of boilerplate
xenia-rs snapshot 336 pages: the interpreter arm, pasted in — the only
per-instruction semantics on unseeded pages
links xenia-rs opcode/decoder/interpreter + Canary emitter
Now:
* semantics come from **Xenia Canary**, the reference emulator, read through
`git show` at a pinned upstream commit (`origin/canary_experimental`,
f21ebd49e9). Not our checkout: it carries instrumentation and lacked
upstream's `mcrf` fix, so it would have published probes and a wrong `mcrf`.
Each page embeds the emitter (`InstrEmit_<mnem>`), and for the 128 pure
one-line delegations also the helper that holds the semantics.
* decode references point at `crates/sylpheed-ppc` — the decoder that
produces `sylpheed.db` — as in-repo relative links.
* the boilerplate now says what is true, and the C translation guide maps
Canary's actual HIR calls, checked against `ppc_hir_builder.h` (including
that `UpdateCR(n, v)` truncates to 32 bits).
* `rust_scraper.py` -> `decoder_scraper.py` (interpreter half dropped);
missing sources are now errors, not empty results.
Verified:
consistency checks 455 XML entries, 350 families, 598 index keys
hand-written tails 386/386 byte-identical after regeneration
xenia-rs in generated 0
pages with a snapshot 349/350 (was 336) — `dcbi` has no Canary emitter at all
in-repo decoder links 910/910 resolve to a line holding the identifier
emitter boundaries brace counter == column-0 `}` rule on 521/521;
preprocessor model unit-tested (#if 0/#else/#elif)
idempotency re-run: 0 pages updated, 0 working-tree changes
Hand-written notes (outside the generated regions) are not rewritten here:
* 110 links into `../../xenia-rs/...` were dead; they now point at the file in
the archived repository (git.mc02.dev/fabi/xenia-rs @ 8401d4d). Line anchors
were dropped because the notes predate that commit — 0 of 441 old line
ranges match it — and a precise-looking wrong anchor is worse than none. The
link text, which carries the author's line numbers, is unchanged.
* 140 prose claims about xenia-rs's behaviour remain. 23 are verified to hold
for Canary too (the 32-bit CR0 truncation, OE left unimplemented); the other
114 need checking one by one, and some invert — e.g. `divdx` notes a correct
64-bit CR0 update in xenia-rs where Canary's `UpdateCR` truncates. Left for
a deliberate pass rather than a blind substitution.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
135 lines
6.3 KiB
Markdown
135 lines
6.3 KiB
Markdown
# Manual generator
|
|
|
|
Python scripts that build the `tools/ppc-manual/` tree from:
|
|
|
|
- **Xenia Canary** — the reference emulator — read at a **pinned commit** through
|
|
`git show`, never from a working tree:
|
|
- `tools/ppc-instructions.xml` — metadata for all 455 Xbox 360 PPC instructions
|
|
(mnemonic, form, group, opcode, in/out fields, disasm template);
|
|
- `src/xenia/cpu/ppc/ppc_emit_*.cc` — each instruction's `InstrEmit_<mnem>`
|
|
emitter, quoted on its page as the semantic snapshot.
|
|
- **`crates/sylpheed-ppc`** — Sylpheed's own decoder, the one that produces the
|
|
disassembly in `sylpheed.db`; referenced by opcode and decoder line.
|
|
|
|
The pin is `origin/canary_experimental` by default. Our own Canary checkout is a
|
|
working branch: it carries instrumentation and can lag upstream by months — it
|
|
lacked upstream's `mcrf` fix when this was written — so quoting it would publish
|
|
probes and stale semantics as "Canary".
|
|
|
|
`xenia-rs` was a source until 2026-09-16. It is retired and archived; its
|
|
interpreter is no longer quoted. Hand-written notes that still discuss its
|
|
behaviour link to the archived repository.
|
|
|
|
## Files
|
|
|
|
| File | Purpose |
|
|
| --- | --- |
|
|
| `generate_manual.py` | Main entry point. Parses XML, builds families, renders pages, writes `index.json`. |
|
|
| `xml_model.py` | XML parser + `expand_runtime_variants()` (produces the set of Rc/OE/LK-expanded mnemonics a single XML entry covers). |
|
|
| `bit_layout.py` | Per-form bit-field tables (rendered into the Encoding section of every page and into `forms/*.md`). |
|
|
| `cxx_scraper.py` | `CanarySource` (one pinned commit) and `CxxScraper`: locates each `InstrEmit_<mnem>`, captures its full text, and follows pure one-line delegations to the helper that holds the semantics. Brace matching models `#if 0` / `#else`. |
|
|
| `decoder_scraper.py` | Locates each `PpcOpcode::<mnem>` variant and decoder producer in `crates/sylpheed-ppc`. |
|
|
|
|
## Running
|
|
|
|
From the repository root:
|
|
|
|
```bash
|
|
python3 tools/ppc-manual/generator/generate_manual.py # full generate
|
|
python3 tools/ppc-manual/generator/generate_manual.py --dry-run # parse + consistency checks only
|
|
python3 tools/ppc-manual/generator/generate_manual.py --canary ../xenia-canary --canary-ref origin/canary_experimental
|
|
python3 tools/ppc-manual/generator/generate_manual.py --out /tmp/out # alternate output root
|
|
```
|
|
|
|
`--canary` defaults to the `xenia-canary` checkout beside this repository.
|
|
Fetch upstream first (`git -C ../xenia-canary fetch origin`) to move the pin
|
|
forward; the commit used is printed, recorded in `index.json` under
|
|
`sources.canary.commit`, and stamped on every snapshot. A missing checkout, ref
|
|
or source file is an **error** — the previous generator skipped what it could
|
|
not find, which is how every reference in the manual rotted unnoticed when it
|
|
moved into `tools/`.
|
|
|
|
No third-party dependencies; Python 3.10+ standard library and `git`.
|
|
|
|
## Idempotency
|
|
|
|
The generator is re-runnable without data loss:
|
|
|
|
1. Each page has a pair of sentinel comments:
|
|
- `<!-- GENERATED: BEGIN -->`
|
|
- `<!-- GENERATED: END -->`
|
|
2. On re-run, only the text **between** the sentinels is rewritten.
|
|
Everything after `END` (Special Cases, Related Instructions, IBM
|
|
Reference) is preserved verbatim.
|
|
3. If the `END` sentinel is missing, the generator assumes a reviewer
|
|
has fully taken over the file and skips it entirely.
|
|
|
|
## Consistency checks (enforced by `--dry-run` as well)
|
|
|
|
- **XML entry count ≡ 455** — warns if the XML has been modified.
|
|
- **family membership total ≡ XML entry count** — every XML entry
|
|
must land in exactly one family.
|
|
- **index coverage ≡ runtime-expanded mnemonic count** — the JSON
|
|
index must contain a key for every runtime variant (`add`, `add.`,
|
|
`addo`, `addo.`, `bclr`, `bclrl`, …).
|
|
|
|
## Family grouping rules
|
|
|
|
Three rules applied in order (see `_family_head` in
|
|
`generate_manual.py`):
|
|
|
|
1. If a mnemonic ends in `128` and the non-128 sibling exists, it
|
|
joins the sibling's family. So `vaddfp128` is consolidated into
|
|
the `vaddfp` page.
|
|
2. For memory ops (group `m`), trailing `u`, `x`, or `ux` suffixes
|
|
are stripped when the base exists. So `lwz`, `lwzu`, `lwzx`,
|
|
`lwzux` all land on the `lwz` page.
|
|
3. Otherwise the mnemonic is its own family head.
|
|
|
|
All other flag variants (`Rc`, `OE`, `LK`) are **runtime** — they are
|
|
NOT separate XML entries; they are listed in the page's "Assembler
|
|
Mnemonics" table.
|
|
|
|
## Category mapping
|
|
|
|
| XML group | Category dir | Notes |
|
|
| --- | --- | --- |
|
|
| `i` (integer) | `alu/` | |
|
|
| `m` (memory) | `memory/` | |
|
|
| `b` (branch) | `branch/` | Includes `sc` and traps |
|
|
| `c` (control) | `control/` | CR logical, SPR, sync |
|
|
| `f` (fpu) | `fpu/` | |
|
|
| `v` (vector) | `vmx/` or `vmx128/` | Split by form: `VX128*` → `vmx128/` |
|
|
|
|
## Extending the generator
|
|
|
|
- **Pseudocode seeds.** The `PSEUDOCODE_SEEDS` dict in
|
|
`generate_manual.py` maps an XML mnemonic to a PPC-style pseudocode
|
|
block. Add entries here to pre-fill the Operation section for
|
|
additional mnemonics. Phase 2 reviewers can still override by
|
|
writing content outside the sentinels.
|
|
- **C translation seeds.** Similar dict of C snippets keyed by family
|
|
head.
|
|
- **Field descriptions.** `FIELD_DESCRIPTIONS` maps XML field names to
|
|
IBM-style prose. Missing entries are marked "_Phase 2: document
|
|
this field._"
|
|
|
|
## Known limitations
|
|
|
|
- Extended-opcode extraction in `xml_model.Instruction.extended_opcode`
|
|
is best-effort per form. For VMX128 variants the extracted value may
|
|
not match the exact pattern used by the decoder — the page still
|
|
shows it as a reference, but the `sylpheed-ppc` decoder line (linked
|
|
on every page) is authoritative.
|
|
- `cxx_scraper` delimits emitters with a brace counter that ignores comments,
|
|
string literals and preprocessor-dead lines. On the pinned commit it agrees with
|
|
the column-0 `}` convention on all 521 functions that follow it; the one that
|
|
does not, `InstrEmit_branch`, closes on a `}` carrying a stale
|
|
`// namespace ppc` comment. `memory/dcbi.md` has no snapshot because Canary has
|
|
no `InstrEmit_dcbi` at all.
|
|
- The generator treats mnemonics ending in `x` as xenia convention
|
|
("extended/XO form") and strips them for assembly display — except
|
|
for the memory group, where `x` is the natural indexed-form suffix.
|
|
If future xenia XML adds a new group where `x` is structural, the
|
|
heuristic in `xml_model.expand_runtime_variants` needs updating.
|