xenia-rs/crates/xenia-analysis/SCHEMA.md

xenia-analysis schema reference

Authoritative documentation for the DuckDB tables and SQL views produced by xenia-rs dis --db sylpheed.db. Track schema changes here alongside any update to the db_schema_golden test fixture.

The base + disasm tables (metadata, sections, imports, functions, labels, instructions, xrefs, opt-in exec_trace / import_calls / branch_trace) are documented inline in src/db.rs doc comment. This file collects layered analysis additions and forward-work notes.

---

Layer M1 — .pdata boundary correction (landed)

Schema additions

• functions.pdata_validated BOOLEAN NOT NULL — true when the row's address matches a RUNTIME_FUNCTION.BeginAddress from .pdata. Linker ground truth.
• functions.pdata_length BIGINT NULL — function_length (bytes) from the matching pdata entry; NULL when the row is prologue-only.
• New table pdata_entries(begin_address BIGINT PRIMARY KEY, end_address BIGINT, function_length BIGINT, prolog_length BIGINT, flags BIGINT) — every parsed .pdata RUNTIME_FUNCTION entry (raw, before any merge with prologue analysis).
• Index idx_functions_pdata_validated on functions(pdata_validated).

What this layer does

• Parses .pdata 8-byte RUNTIME_FUNCTION entries (PowerPC PE32 layout): word 0 BeginAddress (absolute VA), word 1 packed {prolog_length:8, function_length:22, flags:2}, both big-endian.
• Unions pdata BeginAddress values into the function-candidate set fed to the prologue walker, so functions our prologue heuristic missed still get rows.
• When pdata supplies a longer function_length than the prologue walk found, extends end_address to the pdata-implied end (catches mis-split where the walker stopped at an early blr).
• After the walker, performs a forward pass that trims function.end to the next start when they overlap (catches mis-merge where one row spanned two prologues — the audit-031 sub_824D23B0 / sub_824D29F0 case).

What this layer does NOT do

• Does not adjust prolog-derived frame_size / saved_gprs from .pdata's prolog_length field — those remain prologue-only inferences.
• Does not classify functions further than the existing is_leaf / is_saverestore columns. Class membership is M3.
• Does not detect functions whose entries are missing from BOTH .pdata and the bl-target scan (extremely rare; would require executable-byte linear sweep).

Reference docs

• Microsoft PE32+ exception data spec for PowerPC RUNTIME_FUNCTION.
• xenia-canary src/xenia/cpu/xex_module.cc:1570-1587 — canary's reference parser (extracts BeginAddress only; we additionally decode word 1).

Validation queries

-- All pdata entries found
SELECT COUNT(*) FROM pdata_entries;            -- ~23073 for Sylpheed
-- Functions cross-validated against pdata
SELECT COUNT(*) FROM functions WHERE pdata_validated;
-- Functions detected ONLY by prologue (orphans of pdata)
SELECT COUNT(*) FROM functions WHERE NOT pdata_validated;
-- Pdata orphans NOT yet in functions (should be 0 after this layer)
SELECT COUNT(*) FROM pdata_entries p
LEFT JOIN functions f ON f.address = p.begin_address
WHERE f.address IS NULL;
-- Audit-031 mis-merge resolved: 0x824D29F0 should have its own row
SELECT name FROM functions WHERE address = 2186674160;  -- 0x824D29F0

---

Layer M2 — MSVC C++ name demangler (landed)

Schema additions

• New table demangled_names(address BIGINT NULL, mangled VARCHAR NOT NULL, raw_demangled VARCHAR NOT NULL, namespace_path VARCHAR NULL, class_name VARCHAR NULL, method_name VARCHAR NULL, params_signature VARCHAR NULL).
• Indices on address, class_name, method_name.

What this layer does

• Wraps msvc_demangler::demangle (a Rust port of LLVM's MicrosoftDemangle.cpp) and splits the formatted output into structured fields via a heuristic top-level parser (handles templates and nested parens correctly).
• Populates demangled_names from any label whose name starts with ? plus any import name that happens to be mangled (defensive — typical kernel imports use C names).

What this layer does NOT do

• Does not parse the AST returned by msvc_demangler::parse — uses the formatted string and a heuristic split. Adequate for typical class member functions and RTTI strings; exotic template / lambda forms still get raw_demangled populated but may have NULL structured fields.
• Does not yet ingest RTTI strings discovered in .rdata — that's M3's job; M3 will append rows to this table at the addresses where it finds RTTI TypeDescriptors.

Reference docs

• msvc-demangler crate (https://docs.rs/msvc-demangler/0.11).
• LLVM MicrosoftDemangle.cpp (the parser this crate ports).

Layer M3 — Vtable + RTTI detection (landed)

Schema additions

• vtables(address PK, length, col_address NULL, class_name, rtti_present, base_classes_json NULL) — every detected static vtable.
• methods(vtable_address, slot, function_address, mangled_name NULL, demangled_name NULL, PRIMARY KEY (vtable_address, slot)) — one row per method slot.
• classes(name PK, vtable_address, rtti_present, base_classes_json NULL) — deduped by class name (first-detected vtable wins).
• Indices: methods.function_address, classes.rtti_present.

What this layer does

• Walks .rdata and .data looking for runs of ≥3 consecutive 4-byte BE values where each value is a known function start (from M1's corrected functions table). Single-2-method vtables are intentionally rejected to control false-positive rate.
• Attempts the MSVC RTTI walk vtable[-1] → CompleteObjectLocator → TypeDescriptor for each candidate. When successful, the demangled class ClassName string fills class_name and a best-effort RTTIClassHierarchyDescriptor walk fills base_classes_json (JSON array of base class names).
• Falls back to ANON_Class_<8-hex> keyed by FNV-1a hash of the sorted method-PC tuple when RTTI is absent (typical for shipped game binaries). Identical vtables across the binary (multiple instances) collapse to the same anonymous name.

What this layer does NOT do

• Vtables built at runtime in heap-allocated memory (e.g. by ctors copying static templates) are out of scope — only static .rdata/.data content.
• Multiple-inheritance "extra" vftables (one per base subobject) are detected as independent vtables with no link between them.
• Inheritance-tree walking beyond RTTIClassHierarchyDescriptor's direct base list is not attempted.

Reference docs

• openrce.org "Reversing Microsoft Visual C++" — RTTI layout articles (CompleteObjectLocator at vtable[-1]; TypeDescriptor at COL+0xC; mangled name at TD+0x8).

Layer M4 — Class-aware probe targeting (landed)

CLI extension only — no schema changes. The probe-token grammar adds three symbolic forms on top of the existing 0xADDR literal:

• Class::method — joins classes × methods × demangled_names to find every PC whose vtable belongs to that class and whose demangled method_name matches.
• Class::* — joins classes × methods to find every method PC of that class.
• function_name — falls back to functions.name lookup for free functions / saverestore stubs / labels.

Numeric tokens never touch the DB (preserves zero-IO fast path; lockstep digest unaffected). Symbolic tokens require the DuckDB at --probe-db PATH or XENIA_PROBE_DB; default is sylpheed.db next to the .iso when present.

Resolution happens BEFORE guest exec begins, so it cannot affect the lockstep digest.

See crates/xenia-analysis/src/lookup.rs.

---

Layer M5 — Indirect-dispatch reachability (landed)

Schema additions

• New value 'ind_call' in the xrefs.kind set.
• New SQL view v_indirect_reachability_from_entry — strict superset of v_reachability_from_entry, taking ind_call edges in the BFS.

What this layer does

• Walks each FuncAnalysis.functions entry with a per-basic-block register tracker. Recognises the canonical static-vtable pattern: lis+addi → lwz off(rA) → mtctr → bcctrl, where rA ends up holding a known vtable's start address from M3.
• Honours the PowerPC ABI: bl-style calls (op 18 / 16 with LK=1) clobber volatile r0..r12 + ctr but preserve non-volatile r13..r31, so a vtable pointer parked in r30/r31 before a call survives.
• Treats every M3 loc_* label as a basic-block boundary (kills register state) so jump-IN paths cannot induce false positives.

What this layer does NOT do (and observed impact)

• Vtable pointer loaded from a this-pointer field (lwz r_vt, off(rA) where rA = this) — by far the dominant pattern in real C++ — is unresolvable without alias / points-to analysis.
• On Sylpheed: the layer detects 0 edges. The binary's 1,001 lis+addi references into vtables are mostly constructor-side vptr writes (stw rVtable, vptr_offset(this)), not direct dispatches. The renderer hunt's audit-009 cluster therefore needs a future M5.5 with this-flow tracking before this layer surfaces it.

Reference docs

• IBM PowerPC ABI: register-save convention (volatile r0..r12 + ctr, non-volatile r13..r31).

Layer M7 — String / constant-pool detection (landed)

Schema additions

• New table strings(address PK, encoding, length, content).
• Index idx_strings_encoding.

What this layer does

• Scans .rdata for runs of length ≥ 6 of printable ASCII bytes followed by a NUL terminator.
• Scans .rdata for UTF-16LE runs of length ≥ 6 code units (printable-ASCII basic plane only) followed by a u16 NUL terminator.
• Cross-reference is implicit: existing xrefs.kind='ref' rows whose target falls in strings.address's exact match set name the referencing PCs. SQL: SELECT s.content, x.source FROM xrefs x JOIN strings s ON s.address = x.target WHERE x.kind='ref'.

What this layer does NOT do

• No UTF-8 multibyte / non-ASCII basic plane in either encoding.
• No .data scan (read-only-section bias).
• No multi-byte CJK encodings — Japanese text in localised builds may be represented in shift_jis / utf-8 with non-printable bytes that this scanner skips.

Sylpheed yield

• 6,311 ASCII strings (including full embedded HLSL shader source).
• 0 UTF-16LE strings (binary uses ASCII / native CJK encoding).
• 9,132 lis+addi sites cross-reference into the detected strings — names the source PCs that reference each string.

Layer M6 — Extended store-class xrefs + addr_mode column (landed)

Schema additions

• xrefs.addr_mode VARCHAR NULL — sub-classifies how the source instruction computes its target. NULL for control-flow edges (call / ind_call / j / br); one of the following tags for data edges:
  • d_form — standard signed-16 displacement (lwz/stw/lfs/stfs/etc.)
  • lis_addi — address materialised via lis + addi register tracking
  • lis_ori — address materialised via lis + ori
  • multiword — lmw / stmw (one xref per slot; up to 32-rS slots)
  • x_form_indexed — stwx / stbx / sthx / stwux / stbux / sthux / stdx / stdux / lwzx / lbzx / lhzx / lhax / lwzux / lbzux / lhzux / lhaux / ldx / ldux — emitted only when both rA and rB are tracked constants
  • x_form_byterev — stwbrx / sthbrx / lwbrx / lhbrx
  • atomic — stwcx. / stdcx. reservation-conditional stores
  • dcbz — cache-line clear (32-byte zero at rA+rB)
• Index idx_xrefs_addr_mode.

What this layer does

• Tags every existing data xref with its addressing mode (d_form for the bulk; lis_addi / lis_ori for the lift-and-add cases that produce DataRef rows).
• Adds new dispatch for opcode 47 (stmw) and 46 (lmw), expanding to per-slot DataWrite / DataRead rows.
• Adds new dispatch for opcode 31 X-form: stores, atomic, byte-reverse, dcbz. X-form rows are emitted ONLY when both rA and rB resolve to known constants (otherwise the address is runtime-dependent and we skip).

What this layer does NOT do

• VMX / VMX128 vector stores (opcode 31 with vector XO codes) are not emitted — they always have register-indexed addresses that the lis+addi tracker can't usually resolve, and detecting them adds noise without improving target resolution.
• The dominant runtime-of-stwx pattern (rA = base, rB = runtime index) is not resolved — by design; mem-watch covers the runtime side per VERIFY-B.

Sylpheed yield

• 28,834 lis_addi refs, 18,485 d_form reads, 3,288 d_form writes — the existing baseline now properly tagged.
• 442 newly-detected x_form_indexed reads — primarily lwzx/lhzx reads from in-table dispatch (each pair (rA,rB) resolved statically).
• 40 newly-detected atomic writes — every stwcx. site with a resolvable address; useful for reservation-table audits.
• 9 lis_ori refs.
• 0 multiword / dcbz / byterev — these instructions exist in the binary but are not in lis+addi-tracked code paths.

Layer M8 + M11 — Function-pointer arrays beyond vtables (landed)

Schema additions

• New table function_pointer_arrays(address PK, length, kind) where kind is 'vtable' (M3 re-emit), 'dispatch_table' (M8), or 'static_init' (M11).
• New table function_pointer_array_entries(array_address, slot, function_address, PRIMARY KEY (array_address, slot)) — one row per slot of every detected array (vtable + non-vtable).
• Indices on function_pointer_arrays.kind and function_pointer_array_entries.function_address.

What this layer does

• Walks .rdata (only — .data produces too many false positives) for runs of ≥ 2 consecutive 4-byte BE values where each value is a known function entry from M1's functions table.
• Skips runs whose start matches an M3 vtable head — those are re-emitted in this table with kind='vtable' for unified queries but not re-classified.
• Heuristically classifies non-vtable runs:
  • static_init (M11): every entry's first instruction is mfspr r12, LR AND the next is stwu r1, -N(r1) with N ≤ 0x80 (or a save-stub bl). Mirrors the typical C++ static-initialiser prologue.
  • dispatch_table (M8): everything else.

What this layer does NOT do

• Does not parse symbol-table-bracketed regions like __xc_a / __xc_z / __xi_a / __xi_z directly — Sylpheed's symbol table is stripped.
• Does not chain multi-segment static-init drivers; future M11.5 could walk the entry-point's static-init driver call chain to surface ground-truth ctor PCs.
• 2-slot runs in .rdata may be false positives where two struct fields happen to alias function VAs; downstream queries should use a length filter (WHERE length >= 3) when high precision matters.

Sylpheed yield

• 722 vtables (M3 re-emit) + 388 dispatch_tables = 1,110 arrays in function_pointer_arrays.
• 0 static_init detected — Sylpheed's ctors don't all match the conservative prologue heuristic. Lengths concentrate at 2 slots (typical of switch-case jump tables).

Layer M9 — has_eh from .pdata exception flag (landed)

Schema additions

• functions.has_eh BOOLEAN NOT NULL — true when .pdata's exception- handler-present bit (bit 31 of word 1, the high bit) is set.
• Index idx_functions_has_eh.

What this layer does

• Derived directly from M1's already-parsed pdata.flags bit field (no new parsing). The bit was always available in pdata_entries.flags; this layer surfaces it as a first-class column on functions.

What this layer does NOT do

• Does not parse the actual __CxxFrameHandler / __C_specific_handler scope-table records that the exception bit gates. Walking those tables would let us name try/catch ranges and per-state cleanup actions, but is out of scope for a derive-only milestone.

Sylpheed yield

• 2,975 of 23,073 pdata-validated functions have has_eh=true (12.9%) — plausible MSVC C++ EH coverage rate. Largest EH function: 26,328 bytes (sub_823518F0).

Layer M10 — .tls section / TLS directory (landed)

Schema additions

• New table tls_info(raw_data_start, raw_data_end, index_address, callback_array, zero_fill_size, characteristics) — at most one row (the IMAGE_TLS_DIRECTORY32).
• New table tls_callbacks(slot PK, address) — one row per resolved TLS callback function.

What this layer does

• Reads the first 24 bytes of the .tls section as an IMAGE_TLS_DIRECTORY32 and walks the zero-terminated callback array.
• All addresses stored as absolute VAs.

What this layer does NOT do

• Does not parse the raw TLS template content (the variable initialiser block); just records its start/end VAs.

Sylpheed yield

• 0 rows — Sylpheed has no .tls section. Infrastructure ready for any binary that uses __declspec(thread) storage.

Layer M12 — --lr-trace runtime canary-diff harness (landed)

Runtime additions (no DB)

• New CLI flag --lr-trace=PC[,PC,...] on exec — comma-separated PCs to capture as JSONL records on every fire. Symbolic tokens (Class::method) resolve via M4's lookup against --probe-db. Settable via XENIA_LR_TRACE.
• New CLI flag --lr-trace-out=PATH — writes JSONL to a file (one record per line). Stdout when omitted. Settable via XENIA_LR_TRACE_OUT.
• New kernel state fields lr_trace_pcs: HashSet<u32> + lr_trace_writer: Option<Mutex<File>> and helper KernelState::fire_lr_trace_if_match(hw_id) invoked from the per-instruction probe slot.

JSONL record fields

pc, tid, hw, cycle, r3, r4, r5, r6, lr — superset of what xenia-canary's --log_lr_on_pc patch emits, with a cycle counter added for cross-run reproducibility.

What this layer does NOT do

• Does not capture VMX / FP register state (only GPRs r3..r6).
• Does not buffer / batch records — one write_all per fire. For high-frequency probes (e.g. tight loops at >1M fires/sec), redirect to a file and use a SSD.

Determinism

Lockstep digest unaffected: probe firing happens after the per-instr hooks for ctor/branch probes and only emits side-channel output. Verified end-of-session: check sylpheed.iso --stable-digest -n 2M ×2 produced byte-identical digests (instructions=2000005).

---

Forward work (not yet landed)

• M5.5 — this-flow extension to M5. Resolve vtable dispatches via lwz vt, off(this) patterns by tracing constructor-side vptr writes. Highest-value future work for the audit-009 cluster renderer hunt.
• M9.5 — full __CxxFrameHandler scope-table parsing (try/catch range names, per-state cleanup actions).
• M11.5 — walk the static-initialiser driver call chain from the entry point to surface ground-truth ctor PCs.
• VMX/VMX128 vector-store xref emission (M6 follow-up).
• UTF-8 / shift_jis localised-string detection in .rdata (M7 follow-up).