feat(xexdb): lift the DuckDB generator and disassembler out of the retired emulator #32

Merged

fabi merged 7 commits from feat/xexdb-tool into main 2026-09-15 19:15:22 +00:00

Conversation 0 Commits 7 Files Changed 61 +24812 -1

fabi commented 2026-09-13 19:09:32 +00:00

Owner

Copy Link

CONSOLIDATION.md Phase 3 — the DuckDB generator and disassembler, lifted out of the retired emulator.

⚠️ Stacked: this branch contains #30 (reference assets — build.rs needs the export table) and #31 (container docs). Merge those first, or merge this and close them.

What came, and what did not

crates/sylpheed-xex      2,462  XEX2 container + the disc image it lives in
crates/sylpheed-ppc      3,681  decode + disassembly, of xenia-cpu's 21,813
crates/sylpheed-xexdb   10,658  the analysis passes, the schema
src/bin/sylph-xexdb        849  five commands, of xenia-app's 7,214

≈17,200 lines of ~71,000. No interpreter, no JIT, no scheduler, no GPU, no kernel — possible because the coupling was three symbols: decoder::decode, disasm::DisasmItem, disasm::format.

🔴 The gate: regenerate the database and diff it

Schema is a superset — 22 new tables (RTTI, jump tables, XDBF, SQL views) and none lost. instructions (1,865,751), pdata_entries, imports, sections and all three eh_* identical to the row. Most others are slightly higher: +195 functions, +226 strings, +224 classes, from the "complete, correct and faster" work on that branch.

Two tables drop by ~1.8 M, and it is not a loss:

indirect_dispatch_candidates  1,827,297 -> 4,199
xrefs                         2,178,120 -> 356,534

I checked rather than assumed: the xref delta is entirely ind_call (−1,823,098; everything else nets +1,512, mostly new jt rows). The new schema records why — 6,556 sites truncated=true carrying 1,801,075 candidates, against 427 untruncated sites producing the 4,199. A bcctrl through this->vptr at offset 0 matches nearly every class, so the old database materialised a cross product. The count is now truthful and the rows are not invented.

Two defects fixed during the move, not after

The export table came from a sibling checkout. build.rs read Canary's xboxkrnl_table.inc through ../xenia-canary/src/…; when absent it printed a cargo:warning, returned an empty table, and every import in the database resolved to nothing while the build still succeeded. It now reads docs/reference/xbox360-exports.json from this repo — 2,913 ordinals, the exact count #30 adopted, printed in the build log — and a missing file fails the build.

zq.py's escape hatch pointed at a command that no longer exists (xenia-rs exec --dump-addr). Replaced with the two static routes: the .pe is a flat VA dump (offset = VA − 0x82000000), and sylph-xexdb extract regenerates it byte-identically.

Also corrected: the --db help said "SQLite" in two places. It is DuckDB.

Lint

check 0 · clippy 0 · fmt 0. Four #![allow]s, each scoped and argued rather than blanket:

• lzx.rs — index arithmetic is the algorithm; a decompressor that is merely idiomatic is worth nothing if it is not bit-exact.
• sylpheed-xexdb — nine sites index reg[r] where r is the PowerPC register number; plus argument counts and one return type on code that arrived whole from a retired repo, where a refactor would be an unreviewed edit dressed as a lint fix.
• the golden generator — identity_op, because << 0 keeps the encodings checkable by eye against the ISA field table, which is all a generator has.

Everything else clippy asked for is fixed, including all 14 doc-indent sites.

Tests: the three that failed were mine, not a port defect — disasm_goldens.rs resolves fixtures through a relative path and I had copied the test without the data. Fixtures moved, paths repointed. The full workspace test leg is running as I open this; I will comment with the number.

---

⏳→✅ The outstanding workspace cargo test count

CONSOLIDATION.md listed this as the one result never reported cleanly. Run 2026-09-15 on the
second machine, head 6048829, in sylph-ci:local (rustc 1.98.1 (48a229cea 2026-09-01),
byte-identical to the runner's):

	disc	suites	passed	failed	ignored
docker/ci/run cargo test --workspace --no-fail-fast	PRESENT	45	367	1	14
same tree + #35, CI-mirror (disc not mounted, as the runner)	ABSENT	45	377	0	14

The +9 between them is exactly the nine unit tests #35 adds in imports.rs (368 + 9 = 377).

🔴 The one failure is not this PR's. ui_prm_primitives_disc::the_derived_order_puts_primitives_last_and_that_wipes_screens
fails on main itself since #23 — bisected: passes at 4ca0b8e and 641be93, fails at
3977cc9, independent of corpus, branch and toolchain. #23 forces full-screen primitives first
(ui-forced-backdrop.md), so the wipe the test asserts still exists no longer happens. CI is
green because the runner has no disc.

Three things that make the number reproducible, all learned by it failing first:

• --no-fail-fast is required. Without it cargo test stops at the first failing binary —
  an earlier run reported "25 suites / 201 passed", which was about half the workspace.
• The default SYLPH_CI_MEM_GB=7 cannot finish on a 31 GB machine with the disc mounted.
  slb_leading_segment_disc is SIGKILLed: six test threads each load a large voice bank.
  Serial passes 10/10 in 35 s under the same 7 GB, so it is concurrency, not one allocation.
  The default is "half the machine" for a 15 GB box. This run used 14.
• Budget ~36 GB of disk for the sylph-ci-target volume on a cold build. It pushed this
  host to 95%; it has been deleted since.

CONSOLIDATION.md **Phase 3** — the DuckDB generator and disassembler, lifted out of the retired emulator. ⚠️ **Stacked:** this branch contains #30 (reference assets — `build.rs` needs the export table) and #31 (container docs). Merge those first, or merge this and close them. ### What came, and what did not ``` crates/sylpheed-xex 2,462 XEX2 container + the disc image it lives in crates/sylpheed-ppc 3,681 decode + disassembly, of xenia-cpu's 21,813 crates/sylpheed-xexdb 10,658 the analysis passes, the schema src/bin/sylph-xexdb 849 five commands, of xenia-app's 7,214 ``` ≈17,200 lines of ~71,000. No interpreter, no JIT, no scheduler, no GPU, no kernel — possible because the coupling was **three symbols**: `decoder::decode`, `disasm::DisasmItem`, `disasm::format`. ### 🔴 The gate: regenerate the database and diff it Schema is a **superset** — 22 new tables (RTTI, jump tables, XDBF, SQL views) and none lost. `instructions` (1,865,751), `pdata_entries`, `imports`, `sections` and all three `eh_*` identical to the row. Most others are slightly *higher*: +195 functions, +226 strings, +224 classes, from the "complete, correct and faster" work on that branch. Two tables drop by ~1.8 M, and it is not a loss: ``` indirect_dispatch_candidates 1,827,297 -> 4,199 xrefs 2,178,120 -> 356,534 ``` I checked rather than assumed: the xref delta is **entirely `ind_call`** (−1,823,098; everything else nets **+1,512**, mostly new `jt` rows). The new schema records why — **6,556 sites `truncated=true` carrying 1,801,075 candidates**, against 427 untruncated sites producing the 4,199. A `bcctrl` through `this->vptr` at offset 0 matches nearly every class, so the old database materialised a cross product. The count is now truthful and the rows are not invented. ### Two defects fixed during the move, not after **The export table came from a sibling checkout.** `build.rs` read Canary's `xboxkrnl_table.inc` through `../xenia-canary/src/…`; when absent it printed a `cargo:warning`, returned an empty table, and **every import in the database resolved to nothing while the build still succeeded**. It now reads `docs/reference/xbox360-exports.json` from this repo — **2,913 ordinals**, the exact count #30 adopted, printed in the build log — and a missing file **fails the build**. **`zq.py`'s escape hatch pointed at a command that no longer exists** (`xenia-rs exec --dump-addr`). Replaced with the two static routes: the `.pe` is a flat VA dump (`offset = VA − 0x82000000`), and `sylph-xexdb extract` regenerates it byte-identically. Also corrected: the `--db` help said **"SQLite"** in two places. It is DuckDB. ### Lint `check 0 · clippy 0 · fmt 0`. Four `#![allow]`s, each **scoped and argued** rather than blanket: * `lzx.rs` — index arithmetic *is* the algorithm; a decompressor that is merely idiomatic is worth nothing if it is not bit-exact. * `sylpheed-xexdb` — nine sites index `reg[r]` where `r` is the PowerPC register number; plus argument counts and one return type on code that arrived whole from a retired repo, where a refactor would be an unreviewed edit dressed as a lint fix. * the golden **generator** — `identity_op`, because `<< 0` keeps the encodings checkable by eye against the ISA field table, which is all a generator has. Everything else clippy asked for is fixed, including all 14 doc-indent sites. Tests: the three that failed were **mine, not a port defect** — `disasm_goldens.rs` resolves fixtures through a relative path and I had copied the test without the data. Fixtures moved, paths repointed. The full workspace test leg is running as I open this; I will comment with the number. --- ## ⏳→✅ The outstanding workspace `cargo test` count `CONSOLIDATION.md` listed this as the one result never reported cleanly. Run 2026-09-15 on the second machine, head `6048829`, in `sylph-ci:local` (`rustc 1.98.1 (48a229cea 2026-09-01)`, byte-identical to the runner's): | | disc | suites | passed | failed | ignored | |---|---|---|---|---|---| | `docker/ci/run cargo test --workspace --no-fail-fast` | **PRESENT** | 45 | **367** | **1** | 14 | | same tree + #35, CI-mirror (disc not mounted, as the runner) | ABSENT | 45 | 377 | 0 | 14 | The +9 between them is exactly the nine unit tests #35 adds in `imports.rs` (368 + 9 = 377). 🔴 **The one failure is not this PR's.** `ui_prm_primitives_disc::the_derived_order_puts_primitives_last_and_that_wipes_screens` fails on `main` itself since **#23** — bisected: passes at `4ca0b8e` and `641be93`, fails at `3977cc9`, independent of corpus, branch and toolchain. #23 forces full-screen primitives first (`ui-forced-backdrop.md`), so the wipe the test asserts still exists no longer happens. CI is green because the runner has no disc. Three things that make the number reproducible, all learned by it failing first: * **`--no-fail-fast` is required.** Without it `cargo test` stops at the first failing binary — an earlier run reported "25 suites / 201 passed", which was about half the workspace. * **The default `SYLPH_CI_MEM_GB=7` cannot finish on a 31 GB machine with the disc mounted.** `slb_leading_segment_disc` is SIGKILLed: six test threads each load a large voice bank. Serial passes 10/10 in 35 s under the same 7 GB, so it is concurrency, not one allocation. The default is "half the machine" for a 15 GB box. This run used 14. * **Budget ~36 GB of disk** for the `sylph-ci-target` volume on a cold build. It pushed this host to 95%; it has been deleted since.

fabi added 8 commits 2026-09-13 19:09:32 +00:00

docs(reference): adopt the four homeless reference files, and a LICENSE ... 3b90f1c29d

CONSOLIDATION.md Phase 2. Each of these existed only in a repository about to
be retired, or as an untracked file on one disk -- which, for anything that
has to survive a machine, is the same as not existing.

  xex2-format.md            39 KB   the XEX2 container format
  xbox360-exports.{json,md} 1.1 MB  2,913 exports: xboxkrnl, xam, xbdm
  ppc-instructions.{json,md} 340 KB the PowerPC instruction reference
  LICENSE                            MIT -- this repo had none

The first two are byte-identical to xex2tractor's copies, verified with cmp,
and byte-identical again to the loose untracked copies in the project root:
three copies, zero of them tracked.

docs/reference/ is deliberately its own directory with a README that says
what it is NOT: nothing here is a finding, and no docs/re/ page should cite it
as evidence. Reference material filed beside measurements is how a borrowed
table starts getting read as a result.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

wip: extract the xexdb tool closure 53d00e4e93

Merge branch 'harvest/xex2tractor-assets' into feat/xexdb-tool 8342007285

wip: tool README, DuckDB wording, zq.py escape hatch aa8a00d1ce

fix(xexdb): clear the lint gate on the imported crates ... 9c48c340bd

rustfmt, then clippy -D warnings across the three new crates. Mechanical,
except three decisions that are stated rather than silently allowed:

  * lzx.rs gets file-scoped needless_range_loop/explicit_counter_loop allows.
    Index arithmetic IS the algorithm -- LZX is defined over symbol indices,
    Huffman slots and window positions, and a decompressor that is merely
    idiomatic is worth nothing if it is not bit-exact.
  * sylpheed-xexdb gets crate-scoped allows for needless_range_loop (nine
    sites index reg[r] where r is the PowerPC register number -- the index is
    the meaning), too_many_arguments and type_complexity. This code arrived
    whole from a retired repository; a refactor here would be an unreviewed
    edit dressed as a lint fix.
  * Everything else clippy asked for is FIXED, including all 14 doc-indent
    sites, the let-else, and a Prepared type alias in the binary.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

docs(agents): how the containers and agents are actually set up ... 12d6439b2b

Written for the other machine, which asked. Everything in it was read off the
running host rather than remembered.

Also adds docker/ci/ -- the CI image recipe and a capped runner -- because the
image existed on exactly one host and its Dockerfile was in a scratch directory
under /tmp, which was swept. That is the same shape as every other thing this
consolidation has turned up: something correct that exists in one place.

  docker/ci/Dockerfile          rust 1.98.1 + the apt list copied from ci.yml
  docker/ci/Dockerfile.ffmpeg   + ffmpeg, which sylpheed-export shells out to
  docker/ci/run                 6 CPUs / 7 GB / NO SWAP, named cargo volumes

The rule the runner exists to enforce: every heavy command goes in the capped
container. CARGO_BUILD_JOBS caps codegen units, not rustc's threads, not the
linker, not the test harness -- a bare host build is unbounded and has frozen
this box repeatedly.

The document also records the two agent defects the other machine will meet:
a brief change does not reach a RESUMED session, and nothing brought an agent
back to its own red PR (fixed in #24, which is itself subject to the first).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

test(ppc): bring the disassembler's golden fixtures across ... e5e01e2ac2

Three tests failed on the extracted tree for a reason that was not a port
defect: disasm_goldens.rs resolves its fixtures through a relative path to
xenia-cpu/tests/golden/, and I had copied the test without the data.

The fixtures (base + extended mnemonics, VMX128 registers) and the generator
test now live in sylpheed-ppc, and the shim test in sylpheed-xexdb points at
them. serde/serde_json become dev-dependencies of sylpheed-ppc, which the
fixtures need and the library does not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

test(ppc): identity_op off in the golden generator, with the reason ... 60488297b1

The shifts clippy objects to are the point. These are instruction ENCODERS:
every field is written at its ISA position, so `(((vb >> 5) & 0x3) << 0)`
sits in a column with << 11, << 16, << 21 and reads straight against the
manual's field table, and a trailing `| 0` says 'the low field is zero
here', which is information.

Reduced to what clippy wants, the encodings stop being checkable by eye
against the ISA -- and eye-checkability is all a golden GENERATOR has. There
is nothing else to verify it against.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

fabi referenced this pull request 2026-09-13 19:18:52 +00:00

chore(tools): adopt the PPC manual and a canary launcher that works anywhere #34

fabi referenced this pull request 2026-09-13 19:23:07 +00:00

docs(agents): the plan to consolidate six repositories into two #29

fabi referenced this issue from a commit 2026-09-14 18:43:24 +00:00

feat(xexdb): port import-thunk naming + schema-wide foreign keys

fabi referenced this issue from a commit 2026-09-14 18:44:51 +00:00

docs(agents): the Phase 0 gate cannot see a working tree — it missed 468 lines

fabi referenced this pull request 2026-09-14 18:45:39 +00:00

feat(xexdb): port import-thunk naming + 16 schema foreign keys (rescued from an uncommitted tree) #35

fabi referenced this pull request 2026-09-14 18:46:03 +00:00

docs(agents): the gate that cannot see a working tree, and the Phase 5/7 checks re-run #36

fabi referenced this issue from a commit 2026-09-15 06:13:04 +00:00

docs(agents): the #32 workspace test count is in — 367 passed, and the one failure is main's

fabi merged commit d23f0b312d into main 2026-09-15 19:15:22 +00:00

fabi referenced this issue from a commit 2026-09-15 19:15:23 +00:00

Merge pull request 'feat(xexdb): lift the DuckDB generator and disassembler out of the retired emulator' (#32) from feat/xexdb-tool into main

fabi referenced this pull request 2026-09-15 19:18:14 +00:00

feat(xexdb): re-land #35 onto main — import-thunk naming, FKs, zq.py fixes, RE symbols #39

fabi referenced this issue from a commit 2026-09-19 20:29:49 +00:00

feat(xexdb): port import-thunk naming + schema-wide foreign keys

fabi referenced this issue from a commit 2026-09-19 20:29:49 +00:00

docs(agents): the Phase 0 gate cannot see a working tree — it missed 468 lines

fabi referenced this issue from a commit 2026-09-19 20:29:49 +00:00

docs(agents): the #32 workspace test count is in — 367 passed, and the one failure is main's

fabi referenced this issue from a commit 2026-09-19 20:29:49 +00:00

Merge pull request 'feat(xexdb): lift the DuckDB generator and disassembler out of the retired emulator' (#32) from feat/xexdb-tool into main

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

agent/decoder

Owned by the Decoder (disc to meaning; runs the emulator)

agent/port

Owned by the Port (disc to playable; no RE)

kind/ask

One agent asking the other for something it cannot answer in role

kind/bundle

A bundle the human defined; agents decompose it into items

kind/defect

Found by a play-test or a check

kind/item

One unit of work, small enough to finish in a single session

state/approved

Human approved the shape; an agent may start

state/blocked

Waiting on another item; use the Depends-On field, not prose

state/in-progress

An agent is working it now

state/needs-human

Done as far as an agent can tell -- a person must look. The body says what to look at

state/proposed

Agent proposed this item; awaiting the human's approval to start

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

sylph-decoder sylph-pi sylph-port fabi

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: fabi/Sylpheed#32