feat(xexdb): re-land #35 onto main — import-thunk naming, FKs, zq.py fixes, RE symbols #39

fabi · 2026-09-15T19:18:14Z

fabi commented

2026-09-15 19:18:14 +00:00

Re-lands #35 onto main. #35 was merged into its stack parent feat/xexdb-tool — after #32
had already merged that branch into main — so its four commits reached feat/xexdb-tool and
never main. Nothing was lost and no history needs rewriting: this PR has the same head
branch as #35, and its diff against main is exactly #35's content.

b57733a harvest: RE_SYMBOLS.md and apply_re_symbols.sql, which Phase 3 left behind
10150f7 chore: ignore /sylpheed.db — zq.py now defaults to it
38cc170 fix(zq): stop defaulting the database path into the repo being deleted
f0638dc feat(xexdb): port import-thunk naming + schema-wide foreign keys

13 files, +1030 / −111. Verified before opening: all four commits are absent from main and
present on feat/xexdb-tool, and a local --no-ff merge onto main @ d23f0b3 is clean.

The full description, the verification tables and the review notes are on #35 — nothing
about the change itself differs. One thing does improve: #35 targeted a non-main branch and so
got no CI at all; this one targets main, so the runner checks it for real.

Merge with "Create merge commit", as usual.

🤖 Generated with Claude Code

**Re-lands #35 onto `main`.** #35 was merged into its stack parent `feat/xexdb-tool` — after #32 had already merged that branch into `main` — so its four commits reached `feat/xexdb-tool` and never `main`. Nothing was lost and no history needs rewriting: this PR has the **same head branch** as #35, and its diff against `main` is exactly #35's content. ``` b57733a harvest: RE_SYMBOLS.md and apply_re_symbols.sql, which Phase 3 left behind 10150f7 chore: ignore /sylpheed.db — zq.py now defaults to it 38cc170 fix(zq): stop defaulting the database path into the repo being deleted f0638dc feat(xexdb): port import-thunk naming + schema-wide foreign keys ``` 13 files, +1030 / −111. Verified before opening: all four commits are absent from `main` and present on `feat/xexdb-tool`, and a local `--no-ff` merge onto `main` @ `d23f0b3` is clean. **The full description, the verification tables and the review notes are on #35** — nothing about the change itself differs. One thing does improve: #35 targeted a non-`main` branch and so got **no CI at all**; this one targets `main`, so the runner checks it for real. Merge with **"Create merge commit"**, as usual. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

fabi added 4 commits 2026-09-15 19:18:14 +00:00

feat(xexdb): port import-thunk naming + schema-wide foreign keys f0638dc50b

Carries `xenia-rs` `harvest/import-thunk-naming` (b4f19f1) across into the
lifted crate. That work was found UNCOMMITTED in the retired repo's working
tree on 2026-09-14 and exists nowhere else: absent from `iterate-4A`, absent
from this crate as lifted. CONSOLIDATION.md Phase 5 ends with "delete the local
clone" and Phase 7 drops the emulator, so it had a deletion scheduled against
it.

Not a cherry-pick. The lift's base is exactly the harvest's parent (8401d4d),
so each file was merged three-way -- lifted vs base vs harvest -- which is what
made the port reviewable: 10 conflicts, 9 of them pure rustfmt reflow from the
lift's formatting pass, and 1 semantic.

The semantic one is insertion ORDER. `xdbf_achievements.image_id` is now a
foreign key onto `xdbf_images(id)`, so the image rows must be inserted before
the achievement rows. The merge moved that block; the conflict was the stale
copy left at the old position.

What arrives:

* **Import-thunk recognition** (`imports.rs`, 338 lines). An XEX import is not
  a PLT jump: the linker emits a four-word thunk whose first two words are
  import RECORDS that the loader rewrites at module load. On disc they are
  still records, so a PowerPC-only decoder prints two meaningless `.long`s in
  front of an indirect branch. This maps every word of every thunk, and every
  direct branch into one, back to its `imports` row. Shape-validated rather
  than trusted: an entry is indexed only when the four words it points at
  actually have the thunk shape.

  Adds `instructions.import_address` (FK onto `imports.address`) and
  `import_role` (`'record'` | `'thunk'` | `'call'`, NULL iff import_address is
  NULL), plus an index. `tools/zq.py` gains `imp` and `impcalls`, and `dis`
  now names imports instead of printing `.long 0x01010194`.

* **Sixteen schema-wide foreign keys**, declared wherever a column is derived
  from another table. CREATE TABLE and insertion order become load-bearing.

  `functions` is deliberately NOT an FK parent and the golden test now asserts
  zero inbound FKs onto it: DuckDB implements UPDATE as delete+insert, so one
  inbound FK would make `functions.name` un-updatable and break
  `apply_re_symbols.sql`, which re-applies RE symbol names after every
  regeneration.

The database path needed no change in the binary: `DbWriter` builds its own
index inside `ingest_instructions` from `info.import_libraries`. Only the two
output paths (`enrich_section` for JSONL, `write_asm`) take it as an argument.

Verified: `cargo test -p sylpheed-xexdb` = 10 passed / 0 failed, including
`db_schema_golden` (41s, builds a real DuckDB) which locks the 16-FK set and
the no-FK-onto-functions rule. `cargo clippy -p sylpheed-xexdb --all-targets
-- -D warnings` clean; `cargo fmt --all --check` clean.

Stacked on #32 -- it ports into a crate that only exists there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

fix(zq): stop defaulting the database path into the repo being deleted 38cc170491

`zq.py` hardcoded one absolute path, and it pointed inside `xenia-rs` --
a repository CONSOLIDATION.md archives in Phase 5 and drops in Phase 7. It
resolved on exactly one machine, and on that machine it was days from becoming
a `duckdb` exception about a missing file, with nothing saying why.

Resolution order is now `$SYLPH_XEXDB`, else `<repo root>/sylpheed.db`, else a
refusal that names both. The repo-root path is taken relative to this script,
not the caller's cwd, because zq.py is run from wherever the investigation is.

No fallback beyond that, on purpose. The database is a build artefact of a few
hundred MB and is not tracked, so there is nothing to fall back *to*; a default
that silently resolves to the wrong database is worse than no default. That is
issue #16's lesson, which was about exactly this shape in the test suite.

Both failure paths say what to do and exit 1:

  $ zq.py fn 82000000            # nothing set, nothing at the repo root
  no database found.
    looked for: /…/Sylpheed/sylpheed.db
    set $SYLPH_XEXDB to an existing one, or build it with:
      sylph-xexdb dis <xex|iso> --db sylpheed.db --analyze sql

  $ SYLPH_XEXDB=/nope/missing.db zq.py fn 82000000
  $SYLPH_XEXDB is set but is not a file:
    /nope/missing.db

Also fixes the regeneration hint, which still named `xenia-rs dis` -- the
retired binary. It is `sylph-xexdb dis`.

Connecting is skipped when there is no subcommand, so `zq.py` still prints its
usage on a machine that has no database yet.

Verified: usage with no db (rc 0); missing db (rc 1); `$SYLPH_XEXDB` set to a
non-file (rc 1); and `$SYLPH_XEXDB` pointed at a real 336 MB database, where
`imp Rtl` returns its 1,280 `RtlLeaveCriticalSection` call sites. That database
turns out to have been generated from the uncommitted tree this branch ports:
it carries `import_address`, `import_role` and all 16 foreign keys -- an
independent confirmation of the schema the golden test now locks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

chore: ignore /sylpheed.db — zq.py now defaults to it 10150f719d

The previous commit on this branch made `tools/zq.py` look for its database at
`<repo root>/sylpheed.db`, and its refusal message tells people to put one there.
Nothing ignored that path. The database is a build artefact of several hundred
MB, so a single `git add -A` would have committed it into a public repository's
history — the class of blob CONSOLIDATION.md's history-rewrite fork exists to
argue about.

Anchored (`/sylpheed.db`, plus DuckDB's `.wal`) so it covers only the default
location, not any file of that name elsewhere. Verified by creating the files:
both ignored, `git status` silent, and a `sylpheed.db` in a subdirectory is not
swallowed by the rule.

Found while writing the Phase 7 steps: moving the 336 MB database out of
`xenia-rs` before that clone is deleted is exactly how someone would put it here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

harvest: RE_SYMBOLS.md and apply_re_symbols.sql, which Phase 3 left behind

CI / Native — linux (pull_request) Successful in 1h54m9s

Details

CI / WASM — Web (pull_request) Successful in 30m5s

Details

CI / Formatting (pull_request) Successful in 1m19s

Details

b57733aade

Phase 3 lifted the DuckDB tool out of `xenia-rs` but not the two files that go
with it, and this branch's own code already depends on one of them: the FK rule
in `db.rs` and `db_schema_golden.rs` exists *because* `apply_re_symbols.sql`
re-stamps reverse-engineered names onto `functions` after every regeneration.
Sylpheed referenced that file three times and contained it nowhere — on no
branch. CONSOLIDATION.md's end state is "Sylpheed holds everything", and the
`xenia-rs` clone is scheduled for deletion.

  docs/re/RE_SYMBOLS.md       byte-identical to xenia-rs/RE_SYMBOLS.md (cmp)
  tools/apply_re_symbols.sql  every statement identical; the two comment lines
                              that name paths now name this repo's paths

Verified against the real 336 MB database, which carries all 16 foreign keys,
inside a transaction that was rolled back (the database is unchanged):
every statement runs, and 208 functions carry reverse-engineered names. Run
from both the old location and this one.

Not harvested: `xenia-rs/zq_dis.py`, an untracked six-line subset of
`zq.py dis` with a hardcoded path — superseded, nothing to keep.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

fabi merged commit e0187d6fd9 into main

2026-09-16 05:09:15 +00:00

fabi referenced this issue from a commit

2026-09-16 05:09:16 +00:00

Merge pull request 'feat(xexdb): re-land #35 onto main — import-thunk naming, FKs, zq.py fixes, RE symbols' (#39) from feat/xexdb-import-naming into main

fabi referenced this issue from a commit

2026-09-16 18:07:26 +00:00

fix(tools): every sylpheed.db reader honours $SYLPHEED_DB, and none needs a retired checkout

fabi referenced this pull request

2026-09-16 18:07:46 +00:00

fix(tools): every sylpheed.db reader honours $SYLPHEED_DB, and none needs a retired checkout #41

fabi referenced this issue from a commit