feat(xexdb): port import-thunk naming + 16 schema foreign keys (rescued from an uncommitted tree) #35

fabi · 2026-09-14T18:45:39Z

fabi commented

2026-09-14 18:45:39 +00:00

Carries xenia-rs harvest/import-thunk-naming (b4f19f1) into the lifted crate.

⚠️ Stacked on #32 — it ports into crates/sylpheed-xexdb, which only exists there.

Base is feat/xexdb-tool, not main, so this diff shows only the port instead of
burying it in #32's ~17,000 lines. Merge #32 first. If merging #32 deletes its branch,
retarget this to main — by then #32's content is in main and the diff is unchanged.

Why this exists

That work was found uncommitted in the retired repo's working tree on 2026-09-14
— 468 lines plus an untracked 338-line imports.rs, written 2026-09-10, in the very
crate Phase 3 lifted. It existed in exactly one place, and that place was not git:

	`imports.rs`	`import_address`	`zq.py impcalls`
#32, the lifted crate	✗	✗	✗
`iterate-4A` (committed)	✗	✗	✗
xenia-rs working tree	✓	✓	✓

CONSOLIDATION.md Phase 5 ends with "delete the local clone" and Phase 7 drops the
emulator, so a deletion was scheduled against it. The finding itself is written up in
docs/consolidation-dirty-tree-finding.

What arrives

Import-thunk recognition (imports.rs). An XEX import is not a PLT jump: the
linker emits a four-word thunk whose first two words are import records, rewritten
by the loader at module load. On disc they are still records, so a PowerPC-only
decoder prints two meaningless .longs in front of an indirect branch. This maps
every word of every thunk — and every direct branch into one — back to its imports
row, so disassembly says xboxkrnl.exe::RtlEnterCriticalSection instead of
.long 0x01010194. Shape-validated, not trusted: an entry is indexed only when the
four words it points at actually have the thunk shape.

Adds instructions.import_address (FK onto imports.address) and import_role
('record' | 'thunk' | 'call', NULL iff import_address is NULL), plus an index.
tools/zq.py gains imp and impcalls.

Sixteen schema-wide foreign keys, wherever a column is derived from another table.
functions is deliberately not an FK parent, and the golden test now asserts
zero inbound FKs onto it — DuckDB implements UPDATE as delete+insert, so one inbound
FK would make functions.name un-updatable and break apply_re_symbols.sql.

How it was ported — and the one conflict worth reading

Not a cherry-pick. The lift's base is exactly the harvest's parent (8401d4d), so
every file was merged three-way (lifted / base / harvest).

10 conflicts, all caused by the lift's reformatting pass. Nine are the harvest's
additions landing inside blocks rustfmt had rewrapped — content kept, re-indented to
the lift's style. One is genuinely semantic and is the reviewable part:

xdbf_achievements.image_id is now an FK onto xdbf_images(id), so the image rows
must be inserted before the achievement rows. The merge moved that block; the
conflict was the stale copy left behind at the old position.

🔴 Correction to the commit message: it says "9 of them pure rustfmt reflow". That
undersells them — they are reflow plus my content, not reflow alone. The table
above is the accurate description. Force-push is denied here, so the correction lives
in this description rather than in a rewritten commit.

Verification

cargo test -p sylpheed-xexdb                             10 passed / 0 failed
  └ db_schema_golden                                     41s, builds a real DuckDB
cargo clippy -p sylpheed-xexdb --all-targets -D warnings  clean
cargo fmt --all -- --check                                clean

db_schema_golden is the one that matters: it locks the 16-FK set and the
no-FK-onto-functions rule, and it builds an actual database rather than asserting
against a string.

The database write path needed no change in the binary — DbWriter builds its own
index inside ingest_instructions from info.import_libraries. Only the two output
paths (enrich_section for JSONL, write_asm) take it as an argument.

Phase 3 residue found while porting

tools/zq.py — fixed in 38cc170. It hardcoded one absolute path, pointing
inside xenia-rs: a repository Phase 5 archives and Phase 7 drops. It resolved on
exactly one machine, and on that machine it was days away from becoming a duckdb
exception about a missing file with nothing saying why.

Resolution is now $SYLPH_XEXDB → <repo root>/sylpheed.db → a refusal naming both.
The repo-root path is relative to the script, not the caller's cwd. No fallback
beyond that, deliberately: the database is an untracked build artefact of a few
hundred MB, so there is nothing to fall back to, and a default that silently
resolves to the wrong database is worse than no default — issue #16's lesson, which
was this same shape in the test suite. The stale xenia-rs dis regeneration hint is
now sylph-xexdb dis.

Verified: usage still prints with no database (rc 0); missing database (rc 1, names
the build command); $SYLPH_XEXDB set to a non-file (rc 1); and pointed at a real
336 MB database, where imp Rtl returns its 1,280 RtlLeaveCriticalSection call
sites.

📌 That database turns out to have been generated from the uncommitted tree this
branch ports — it already carries import_address, import_role and all 16
foreign keys. An independent confirmation of the schema the golden test now locks.

Still not fixed, and still reported rather than changed:

SCHEMA.md is headed "xenia-analysis schema reference" and cites xenia-rs dis --db — both naming the retired repo and binary.
Phase 3's "says SQLite, is DuckDB" correction was made where it counts (README
and the binary's module doc), but two inline comments at sylph-xexdb.rs:499 and
:874 still say SQLite.

🤖 Generated with Claude Code

CI stand-in: every leg green on the runner's own toolchain

This PR targets feat/xexdb-tool, and ci.yml only runs on PRs targeting main — so it has
no CI statuses at all. Run instead in sylph-ci:local, whose rustc 1.98.1 (48a229cea 2026-09-01) is byte-identical to the runner's, mirroring ci.yml step for step, disc
not mounted (as on the runner), head 38cc170:

leg	command	result
check	`cargo check --workspace`	✅
build	`cargo build --workspace`	✅
test	`cargo test --workspace`	✅ 45 suites, 377 passed, 0 failed, 14 ignored
clippy	`cargo clippy --workspace -- -D warnings`	✅
fmt	`cargo fmt --all -- --check`	✅
wasm	`cargo check --target wasm32-unknown-unknown -p sylpheed-viewer -p sylpheed-formats`	✅
trunk	`trunk build --release`	⬜ not run — trunk is not in the image

Corpus report: ABSENT for all three, so this is parser-only coverage, same as the runner.

⚠️ Unrelated to this PR, and worth knowing before merging anything: with the disc present,
main itself fails ui_prm_primitives_disc since #23 — bisected, CI cannot see it.

Two more commits (2026-09-15)

10150f7 — ignore /sylpheed.db. 38cc170 made zq.py default to <repo root>/sylpheed.db and
tell people to put one there, but nothing ignored that path. One git add -A would have put a
several-hundred-MB build artefact into public history. Anchored, so only the root file is ignored.
b57733a — harvest RE_SYMBOLS.md + apply_re_symbols.sql, which Phase 3 left in xenia-rs.
This PR's own FK rule exists because apply_re_symbols.sql re-stamps names onto functions —
Sylpheed referenced it three times and contained it on no branch. docs/re/RE_SYMBOLS.md is a
byte-identical copy; tools/apply_re_symbols.sql keeps every statement and re-paths two comments.
Run against the real 336 MB database (all 16 FKs) in a rolled-back transaction: every statement
runs, 208 functions named, database unchanged.

Carries `xenia-rs` `harvest/import-thunk-naming` (`b4f19f1`) into the lifted crate. ⚠️ **Stacked on #32** — it ports into `crates/sylpheed-xexdb`, which only exists there. **Base is `feat/xexdb-tool`, not `main`**, so this diff shows only the port instead of burying it in #32's ~17,000 lines. Merge #32 first. If merging #32 deletes its branch, retarget this to `main` — by then #32's content is in `main` and the diff is unchanged. ## Why this exists That work was found **uncommitted** in the retired repo's working tree on 2026-09-14 — 468 lines plus an untracked 338-line `imports.rs`, written 2026-09-10, in the very crate Phase 3 lifted. It existed in exactly one place, and that place was not git: | | `imports.rs` | `import_address` | `zq.py impcalls` | |---|---|---|---| | #32, the lifted crate | ✗ | ✗ | ✗ | | `iterate-4A` (committed) | ✗ | ✗ | ✗ | | **xenia-rs working tree** | **✓** | **✓** | **✓** | `CONSOLIDATION.md` Phase 5 ends with *"delete the local clone"* and Phase 7 drops the emulator, so a deletion was scheduled against it. The finding itself is written up in `docs/consolidation-dirty-tree-finding`. ## What arrives **Import-thunk recognition** (`imports.rs`). An XEX import is not a PLT jump: the linker emits a four-word thunk whose first two words are import *records*, rewritten by the loader at module load. On disc they are still records, so a PowerPC-only decoder prints two meaningless `.long`s in front of an indirect branch. This maps every word of every thunk — and every direct branch into one — back to its `imports` row, so disassembly says `xboxkrnl.exe::RtlEnterCriticalSection` instead of `.long 0x01010194`. Shape-validated, not trusted: an entry is indexed only when the four words it points at actually have the thunk shape. Adds `instructions.import_address` (FK onto `imports.address`) and `import_role` (`'record'` | `'thunk'` | `'call'`, NULL iff `import_address` is NULL), plus an index. `tools/zq.py` gains `imp` and `impcalls`. **Sixteen schema-wide foreign keys**, wherever a column is derived from another table. `functions` is deliberately **not** an FK parent, and the golden test now *asserts* zero inbound FKs onto it — DuckDB implements UPDATE as delete+insert, so one inbound FK would make `functions.name` un-updatable and break `apply_re_symbols.sql`. ## How it was ported — and the one conflict worth reading Not a cherry-pick. The lift's base is **exactly** the harvest's parent (`8401d4d`), so every file was merged three-way (lifted / base / harvest). **10 conflicts, all caused by the lift's reformatting pass.** Nine are the harvest's additions landing inside blocks rustfmt had rewrapped — content kept, re-indented to the lift's style. **One is genuinely semantic and is the reviewable part:** > `xdbf_achievements.image_id` is now an FK onto `xdbf_images(id)`, so the image rows > must be inserted **before** the achievement rows. The merge moved that block; the > conflict was the stale copy left behind at the old position. 🔴 *Correction to the commit message:* it says "9 of them pure rustfmt reflow". That undersells them — they are reflow **plus** my content, not reflow alone. The table above is the accurate description. Force-push is denied here, so the correction lives in this description rather than in a rewritten commit. ## Verification ``` cargo test -p sylpheed-xexdb 10 passed / 0 failed └ db_schema_golden 41s, builds a real DuckDB cargo clippy -p sylpheed-xexdb --all-targets -D warnings clean cargo fmt --all -- --check clean ``` `db_schema_golden` is the one that matters: it locks the 16-FK set and the no-FK-onto-`functions` rule, and it builds an actual database rather than asserting against a string. The database write path needed **no** change in the binary — `DbWriter` builds its own index inside `ingest_instructions` from `info.import_libraries`. Only the two output paths (`enrich_section` for JSONL, `write_asm`) take it as an argument. ## Phase 3 residue found while porting **`tools/zq.py` — fixed in `38cc170`.** It hardcoded one absolute path, pointing *inside* `xenia-rs`: a repository Phase 5 archives and Phase 7 drops. It resolved on exactly one machine, and on that machine it was days away from becoming a `duckdb` exception about a missing file with nothing saying why. Resolution is now `$SYLPH_XEXDB` → `<repo root>/sylpheed.db` → a refusal naming both. The repo-root path is relative to the *script*, not the caller's cwd. **No fallback beyond that, deliberately**: the database is an untracked build artefact of a few hundred MB, so there is nothing to fall back *to*, and a default that silently resolves to the wrong database is worse than no default — issue #16's lesson, which was this same shape in the test suite. The stale `xenia-rs dis` regeneration hint is now `sylph-xexdb dis`. Verified: usage still prints with no database (rc 0); missing database (rc 1, names the build command); `$SYLPH_XEXDB` set to a non-file (rc 1); and pointed at a real 336 MB database, where `imp Rtl` returns its 1,280 `RtlLeaveCriticalSection` call sites. 📌 That database turns out to have been generated **from the uncommitted tree this branch ports** — it already carries `import_address`, `import_role` and all **16** foreign keys. An independent confirmation of the schema the golden test now locks. **Still not fixed, and still reported rather than changed:** * `SCHEMA.md` is headed *"`xenia-analysis` schema reference"* and cites `xenia-rs dis --db` — both naming the retired repo and binary. * Phase 3's "says SQLite, is DuckDB" correction **was** made where it counts (README and the binary's module doc), but two inline comments at `sylph-xexdb.rs:499` and `:874` still say SQLite. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## CI stand-in: every leg green on the runner's own toolchain This PR targets `feat/xexdb-tool`, and `ci.yml` only runs on PRs targeting `main` — so it has **no CI statuses at all**. Run instead in `sylph-ci:local`, whose `rustc 1.98.1 (48a229cea 2026-09-01)` is byte-identical to the runner's, mirroring `ci.yml` step for step, disc **not** mounted (as on the runner), head `38cc170`: | leg | command | result | |---|---|---| | check | `cargo check --workspace` | ✅ | | build | `cargo build --workspace` | ✅ | | test | `cargo test --workspace` | ✅ **45 suites, 377 passed, 0 failed, 14 ignored** | | clippy | `cargo clippy --workspace -- -D warnings` | ✅ | | fmt | `cargo fmt --all -- --check` | ✅ | | wasm | `cargo check --target wasm32-unknown-unknown -p sylpheed-viewer -p sylpheed-formats` | ✅ | | trunk | `trunk build --release` | ⬜ **not run** — trunk is not in the image | Corpus report: ABSENT for all three, so this is parser-only coverage, same as the runner. ⚠️ Unrelated to this PR, and worth knowing before merging anything: with the disc present, `main` itself fails `ui_prm_primitives_disc` since #23 — bisected, CI cannot see it. --- ## Two more commits (2026-09-15) * **`10150f7` — ignore `/sylpheed.db`.** `38cc170` made `zq.py` default to `<repo root>/sylpheed.db` and tell people to put one there, but nothing ignored that path. One `git add -A` would have put a several-hundred-MB build artefact into public history. Anchored, so only the root file is ignored. * **`b57733a` — harvest `RE_SYMBOLS.md` + `apply_re_symbols.sql`**, which Phase 3 left in `xenia-rs`. This PR's own FK rule exists *because* `apply_re_symbols.sql` re-stamps names onto `functions` — Sylpheed referenced it three times and contained it on no branch. `docs/re/RE_SYMBOLS.md` is a byte-identical copy; `tools/apply_re_symbols.sql` keeps every statement and re-paths two comments. Run against the real 336 MB database (all 16 FKs) in a rolled-back transaction: every statement runs, 208 functions named, database unchanged.

fabi added 1 commit 2026-09-14 18:45:39 +00:00

feat(xexdb): port import-thunk naming + schema-wide foreign keys f0638dc50b

Carries `xenia-rs` `harvest/import-thunk-naming` (b4f19f1) across into the
lifted crate. That work was found UNCOMMITTED in the retired repo's working
tree on 2026-09-14 and exists nowhere else: absent from `iterate-4A`, absent
from this crate as lifted. CONSOLIDATION.md Phase 5 ends with "delete the local
clone" and Phase 7 drops the emulator, so it had a deletion scheduled against
it.

Not a cherry-pick. The lift's base is exactly the harvest's parent (8401d4d),
so each file was merged three-way -- lifted vs base vs harvest -- which is what
made the port reviewable: 10 conflicts, 9 of them pure rustfmt reflow from the
lift's formatting pass, and 1 semantic.

The semantic one is insertion ORDER. `xdbf_achievements.image_id` is now a
foreign key onto `xdbf_images(id)`, so the image rows must be inserted before
the achievement rows. The merge moved that block; the conflict was the stale
copy left at the old position.

What arrives:

* **Import-thunk recognition** (`imports.rs`, 338 lines). An XEX import is not
  a PLT jump: the linker emits a four-word thunk whose first two words are
  import RECORDS that the loader rewrites at module load. On disc they are
  still records, so a PowerPC-only decoder prints two meaningless `.long`s in
  front of an indirect branch. This maps every word of every thunk, and every
  direct branch into one, back to its `imports` row. Shape-validated rather
  than trusted: an entry is indexed only when the four words it points at
  actually have the thunk shape.

  Adds `instructions.import_address` (FK onto `imports.address`) and
  `import_role` (`'record'` | `'thunk'` | `'call'`, NULL iff import_address is
  NULL), plus an index. `tools/zq.py` gains `imp` and `impcalls`, and `dis`
  now names imports instead of printing `.long 0x01010194`.

* **Sixteen schema-wide foreign keys**, declared wherever a column is derived
  from another table. CREATE TABLE and insertion order become load-bearing.

  `functions` is deliberately NOT an FK parent and the golden test now asserts
  zero inbound FKs onto it: DuckDB implements UPDATE as delete+insert, so one
  inbound FK would make `functions.name` un-updatable and break
  `apply_re_symbols.sql`, which re-applies RE symbol names after every
  regeneration.

The database path needed no change in the binary: `DbWriter` builds its own
index inside `ingest_instructions` from `info.import_libraries`. Only the two
output paths (`enrich_section` for JSONL, `write_asm`) take it as an argument.

Verified: `cargo test -p sylpheed-xexdb` = 10 passed / 0 failed, including
`db_schema_golden` (41s, builds a real DuckDB) which locks the 16-FK set and
the no-FK-onto-functions rule. `cargo clippy -p sylpheed-xexdb --all-targets
-- -D warnings` clean; `cargo fmt --all --check` clean.

Stacked on #32 -- it ports into a crate that only exists there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

fabi referenced this pull request

2026-09-14 18:46:03 +00:00

docs(agents): the gate that cannot see a working tree, and the Phase 5/7 checks re-run #36

fabi added 1 commit 2026-09-14 18:49:50 +00:00

fix(zq): stop defaulting the database path into the repo being deleted 38cc170491

`zq.py` hardcoded one absolute path, and it pointed inside `xenia-rs` --
a repository CONSOLIDATION.md archives in Phase 5 and drops in Phase 7. It
resolved on exactly one machine, and on that machine it was days from becoming
a `duckdb` exception about a missing file, with nothing saying why.

Resolution order is now `$SYLPH_XEXDB`, else `<repo root>/sylpheed.db`, else a
refusal that names both. The repo-root path is taken relative to this script,
not the caller's cwd, because zq.py is run from wherever the investigation is.

No fallback beyond that, on purpose. The database is a build artefact of a few
hundred MB and is not tracked, so there is nothing to fall back *to*; a default
that silently resolves to the wrong database is worse than no default. That is
issue #16's lesson, which was about exactly this shape in the test suite.

Both failure paths say what to do and exit 1:

  $ zq.py fn 82000000            # nothing set, nothing at the repo root
  no database found.
    looked for: /…/Sylpheed/sylpheed.db
    set $SYLPH_XEXDB to an existing one, or build it with:
      sylph-xexdb dis <xex|iso> --db sylpheed.db --analyze sql

  $ SYLPH_XEXDB=/nope/missing.db zq.py fn 82000000
  $SYLPH_XEXDB is set but is not a file:
    /nope/missing.db

Also fixes the regeneration hint, which still named `xenia-rs dis` -- the
retired binary. It is `sylph-xexdb dis`.

Connecting is skipped when there is no subcommand, so `zq.py` still prints its
usage on a machine that has no database yet.

Verified: usage with no db (rc 0); missing db (rc 1); `$SYLPH_XEXDB` set to a
non-file (rc 1); and `$SYLPH_XEXDB` pointed at a real 336 MB database, where
`imp Rtl` returns its 1,280 `RtlLeaveCriticalSection` call sites. That database
turns out to have been generated from the uncommitted tree this branch ports:
it carries `import_address`, `import_role` and all 16 foreign keys -- an
independent confirmation of the schema the golden test now locks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

fabi referenced this issue from a commit

2026-09-14 18:50:45 +00:00

docs(agents): the zq.py database path is fixed, not outstanding

fabi referenced this pull request

2026-09-15 06:12:32 +00:00

feat(xexdb): lift the DuckDB generator and disassembler out of the retired emulator #32

fabi referenced this issue from a commit

2026-09-15 06:13:04 +00:00

docs(agents): the #32 workspace test count is in — 367 passed, and the one failure is main's

fabi added 1 commit 2026-09-15 17:20:39 +00:00

chore: ignore /sylpheed.db — zq.py now defaults to it 10150f719d

The previous commit on this branch made `tools/zq.py` look for its database at
`<repo root>/sylpheed.db`, and its refusal message tells people to put one there.
Nothing ignored that path. The database is a build artefact of several hundred
MB, so a single `git add -A` would have committed it into a public repository's
history — the class of blob CONSOLIDATION.md's history-rewrite fork exists to
argue about.

Anchored (`/sylpheed.db`, plus DuckDB's `.wal`) so it covers only the default
location, not any file of that name elsewhere. Verified by creating the files:
both ignored, `git status` silent, and a `sylpheed.db` in a subdirectory is not
swallowed by the rule.

Found while writing the Phase 7 steps: moving the 336 MB database out of
`xenia-rs` before that clone is deleted is exactly how someone would put it here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

fabi added 1 commit 2026-09-15 17:23:08 +00:00

harvest: RE_SYMBOLS.md and apply_re_symbols.sql, which Phase 3 left behind

CI / Native — linux (pull_request) Successful in 1h54m9s

Details

CI / WASM — Web (pull_request) Successful in 30m5s

Details

CI / Formatting (pull_request) Successful in 1m19s

Details

b57733aade

Phase 3 lifted the DuckDB tool out of `xenia-rs` but not the two files that go
with it, and this branch's own code already depends on one of them: the FK rule
in `db.rs` and `db_schema_golden.rs` exists *because* `apply_re_symbols.sql`
re-stamps reverse-engineered names onto `functions` after every regeneration.
Sylpheed referenced that file three times and contained it nowhere — on no
branch. CONSOLIDATION.md's end state is "Sylpheed holds everything", and the
`xenia-rs` clone is scheduled for deletion.

  docs/re/RE_SYMBOLS.md       byte-identical to xenia-rs/RE_SYMBOLS.md (cmp)
  tools/apply_re_symbols.sql  every statement identical; the two comment lines
                              that name paths now name this repo's paths

Verified against the real 336 MB database, which carries all 16 foreign keys,
inside a transaction that was rolled back (the database is unchanged):
every statement runs, and 208 functions carry reverse-engineered names. Run
from both the old location and this one.

Not harvested: `xenia-rs/zq_dis.py`, an untracked six-line subset of
`zq.py dis` with a hardcoded path — superseded, nothing to keep.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

fabi merged commit 93504b7d5f into feat/xexdb-tool

2026-09-15 19:15:35 +00:00

fabi referenced this issue from a commit

2026-09-15 19:15:37 +00:00

Merge pull request 'feat(xexdb): port import-thunk naming + 16 schema foreign keys (rescued from an uncommitted tree)' (#35) from feat/xexdb-import-naming into feat/xexdb-tool

fabi referenced this pull request

2026-09-15 19:18:14 +00:00

feat(xexdb): re-land #35 onto main — import-thunk naming, FKs, zq.py fixes, RE symbols #39

fabi referenced this issue from a commit

2026-09-16 05:09:16 +00:00

Merge pull request 'feat(xexdb): re-land #35 onto main — import-thunk naming, FKs, zq.py fixes, RE symbols' (#39) from feat/xexdb-import-naming into main

fabi referenced this issue from a commit