2f838040ca
Instructions and generated text that still sent readers to `xenia-rs`, which is
archived and deleted locally. Provenance and dated research records are left as
they are — "lifted from xenia-rs", or a finding stating what database it was
measured against, is still true, and rewriting it would falsify the record.
Changed, because each one tells someone what to do today or writes stale text:
docs/re/README.md the RE toolchain guide queried `xenia-rs/sylpheed.db`
with `xenia-rs/zq.py`, and its "Dynamic" bullet said
to prefer xenia-rs's probe suite OVER Canary. Now:
`sylpheed.db` at the repo root (how to build it and
re-stamp names), `tools/zq.py`, dynamic = Canary, and
`.rdata` reads from the `.pe` (offset = VA - 0x82000000)
in place of `--dump-addr`.
challenge-mission-gate.md, structures/achievements.md
reproduction commands `python3 xenia-rs/zq.py dis …`
crates/sylpheed-xexdb/SCHEMA.md
titled after the retired `xenia-analysis` crate,
citing `xenia-rs dis`, pointing at a `xenia-analysis`
source path
sylpheed-xexdb/src/formatter.rs
WROTE "generated by xenia-rs" into every disassembly
it produced (no test pins the banner)
sylph-xexdb.rs, db.rs "SQLite" / "DuckDB writer for xenia-rs"
sylpheed-formats/src/hash.rs
pointed at `xenia-rs/RE_SYMBOLS.md`, now at
`docs/re/RE_SYMBOLS.md`
Verified: every rewritten reproduction command was extracted from the doc and
run as written against the regenerated database — all 5 exit 0 and return the
disassembly they describe. The README's function count (25 676) is the database's
own. `cargo fmt --all -- --check` clean.
Not changed, deliberately: the README's Oracle bullet says Canary's native Linux
ELF "crashes / does not run". That is about Canary, not xenia-rs, and it conflicts
with a July note that the native build works — unverified either way here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2365 lines
96 KiB
Rust
2365 lines
96 KiB
Rust
//! DuckDB writer for `sylph-xexdb`.
|
||
//!
|
||
//! Layered, streaming writes shared by `extract`, `dis`, and `exec`.
|
||
//! Each command's output is a superset of the previous:
|
||
//! - `extract --db` -> base tables (metadata, sections, imports)
|
||
//! - `dis --db` -> base + disasm tables (functions, labels, instructions, xrefs)
|
||
//! - `exec --db` -> base + disasm + opt-in trace tables (exec_trace, import_calls, branch_trace)
|
||
//!
|
||
//! Bulk inserts use the DuckDB Appender API, which bypasses the SQL layer and
|
||
//! writes directly to columnar storage — no transaction batching required.
|
||
//!
|
||
//! Trace kind values for `branch_trace.kind`:
|
||
//! - `"call"` : any branch with LK set (raw & 1 == 1)
|
||
//! - `"return"` : bclrx without LK
|
||
//! - `"jump"` : bcctrx without LK
|
||
//! - `"branch"` : bx/bcx without LK
|
||
//!
|
||
//! # Referential integrity
|
||
//!
|
||
//! The schema declares foreign keys wherever a column is *derived from* the
|
||
//! row it points at, so a build that would produce a dangling reference fails
|
||
//! loudly instead of shipping a broken database. Two consequences:
|
||
//!
|
||
//! - **Insert order is load-bearing.** DuckDB checks each row as it is
|
||
//! appended, so a parent must be populated before its children, and
|
||
//! `CREATE TABLE` statements must be ordered so a table exists before
|
||
//! anything references it. `write_analysis_results` documents its order.
|
||
//! - **`functions` is deliberately NOT a foreign-key parent.** DuckDB
|
||
//! implements `UPDATE` as delete-then-insert, so *any* FK onto
|
||
//! `functions(address)` makes the table un-updatable — even an update that
|
||
//! does not touch `address`. That would break `apply_re_symbols.sql`, which
|
||
//! re-stamps reverse-engineered names onto `functions.name` after every
|
||
//! regeneration. `xrefs.source_func`, `vptr_writes.writer_function` and
|
||
//! `jump_tables.function` are therefore plain columns. The same restriction
|
||
//! applies to any table a human post-edits.
|
||
//!
|
||
//! Some references are intentionally left undeclared because the relationship
|
||
//! genuinely does not hold: see `rtti_locators.vtable_address` (41 RTTI-named
|
||
//! vftables the M3 scanner misses), `methods.function_address` and
|
||
//! `indirect_dispatch_candidates.method_address` (vtable slots pointing at
|
||
//! code no `.pdata` record or `bl` ever declared), and `xrefs.target` (which
|
||
//! is frequently data). Those orphan sets are findings, not noise — querying
|
||
//! them is how the gaps stay visible.
|
||
//!
|
||
//! # Schema
|
||
//!
|
||
//! ## `metadata`
|
||
//! Key-value table, one row per XEX header field; values are strings. Beyond
|
||
//! the five columns tabulated below it also carries the module/image flag words
|
||
//! (raw + decoded), image size and load address, encryption + compression type,
|
||
//! disc number/count, per-import-library SDK versions
|
||
//! (`import_lib.<name>.version_cur`), and one `xex_optional_header.0x…` row for
|
||
//! every optional header present, so nothing in the XEX is silently dropped.
|
||
//!
|
||
//! | key | value format | meaning |
|
||
//! |--------------------|------------------|----------------------------------------------------|
|
||
//! | `image_base` | `"0xXXXXXXXX"` | Virtual address where the PE image is mapped |
|
||
//! | `entry_point` | `"0xXXXXXXXX"` | Absolute VA of the XEX entry point |
|
||
//! | `original_pe_name` | string | Original PE filename from XEX optional headers |
|
||
//! | `title_id` | `"0xXXXXXXXX"` | Xbox 360 Title ID (identifies the game) |
|
||
//! | `media_id` | `"0xXXXXXXXX"` | Disc/media ID (identifies the specific disc build) |
|
||
//!
|
||
//! ## `sections`
|
||
//! One row per PE section (`.text`, `.data`, etc.).
|
||
//! - `name` — PE section name
|
||
//! - `virtual_address` — RVA relative to `image_base` where the section is mapped in memory
|
||
//! - `virtual_size` — Size in memory; may exceed `raw_size` due to BSS zero-fill
|
||
//! - `raw_offset` — Byte offset of section data within the XEX/PE file
|
||
//! - `raw_size` — Size of section data on disk
|
||
//! - `flags` — `IMAGE_SCN_*` characteristics bit field
|
||
//! - `is_code` — `true` if `IMAGE_SCN_CNT_CODE` is set
|
||
//!
|
||
//! ## `imports`
|
||
//! One row per import record from the XEX import descriptor table.
|
||
//! - `library` — Module name (e.g. `xboxkrnl.exe`, `xam.xex`)
|
||
//! - `ordinal` — Numeric ordinal identifying the export within the library
|
||
//! - `name` — Resolved human-readable symbol name; `NULL` if not in symbol table
|
||
//! - `record_type` — XEX import record type: `0` = variable, `1` = function thunk
|
||
//! - `address` — Absolute VA of the import thunk or variable in the binary.
|
||
//! UNIQUE, because `instructions.import_address` is a foreign key onto it.
|
||
//!
|
||
//! ## `functions`
|
||
//! One row per detected function. Candidates are `bl` targets ∪ `.pdata`
|
||
//! `BeginAddress`es ∪ tail-call targets ∪ the entry point. `.pdata` is the
|
||
//! linker's own function table and is treated as authoritative: where it
|
||
//! covers a function, `end_address` is its declared end rather than a
|
||
//! prologue-walk guess.
|
||
//! - `address` — Absolute VA of the function entry point (PK)
|
||
//! - `name` — Symbol name, or `sub_XXXXXXXX` if unresolved
|
||
//! - `end_address` — Absolute VA of last instruction + 4 (exclusive end)
|
||
//! - `frame_size` — Stack frame size in bytes (from prologue)
|
||
//! - `saved_gprs` — Bitmask of GPRs saved in prologue (bit N set ⇒ rN is saved)
|
||
//! - `is_leaf` — `true` if the function has no outgoing calls (no `bl`/`blr`)
|
||
//! - `is_saverestore` — `true` if this is a `__savegprlr_*`/`__restgprlr_*` compiler stub
|
||
//! - `pdata_validated` — `true` when `.pdata` declares a function at this VA
|
||
//! - `pdata_length` — Declared size in bytes; `NULL` when prologue-only
|
||
//! - `prolog_length` — Declared prolog size in bytes; `NULL` when prologue-only
|
||
//! - `has_eh` — `.pdata` exception-handler bit; function has C++ EH/SEH
|
||
//!
|
||
//! ## `pdata_entries`
|
||
//! The raw `.pdata` `RUNTIME_FUNCTION` table, one row per entry, so a query
|
||
//! can distinguish linker ground truth from this crate's inferences.
|
||
//!
|
||
//! ## `labels`
|
||
//! One row per named address; superset of functions.
|
||
//! - `address` — Absolute VA (PK)
|
||
//! - `name` — Symbol name
|
||
//! - `kind` — One of: `function`, `import`, `saverestore`, `local`, `data`, `other`
|
||
//!
|
||
//! ## `instructions`
|
||
//! One row per disassembled instruction.
|
||
//! - `address` — Absolute VA (PK)
|
||
//! - `raw` — 4-byte big-endian instruction word as integer
|
||
//! - `mnemonic` — Base mnemonic (e.g. `stw`, `bl`, `cmpwi`)
|
||
//! - `operands` — Operand string from base disassembly
|
||
//! - `disasm` — Full base disassembly string (`mnemonic + " " + operands`)
|
||
//! - `ext_mnemonic` — Simplified mnemonic (e.g. `mr` for `or rX,rY,rY`); `NULL` if none
|
||
//! - `ext_operands` — Operands for the extended form; `NULL` if none
|
||
//! - `ext_disasm` — Full extended disassembly string; `NULL` if none
|
||
//! - `target_hex` — Resolved absolute branch target for `b`/`bc` (and link/AA variants); `NULL` for indirect or non-branch instructions. SQL views (`v_branch_xrefs`) self-join on this column.
|
||
//! - `section` — Name of the PE section containing this instruction
|
||
//! - `function` — VA of the enclosing function; `NULL` if not inside a detected function
|
||
//! - `label` — Label name at this address; `NULL` if none
|
||
//! - `is_data` — `true` when this word is data embedded in a code section (a
|
||
//! recovered jump table or index map, or an XEX import record). The decoded
|
||
//! `mnemonic` / `disasm` columns are meaningless on such rows; filter them
|
||
//! out (`WHERE NOT is_data`) for any instruction-level analysis.
|
||
//! - `import_address` — FK onto `imports.address` (a thunk head) when this word is
|
||
//! part of an XEX import thunk, or is a direct branch into one; `NULL`
|
||
//! otherwise. Join it to name the callee.
|
||
//! - `import_role` — Which of the three this row is: `'record'` (one of the two
|
||
//! loader-patched words at the thunk head — also `is_data`), `'thunk'` (the
|
||
//! `mtctr r11` / `bctr` body), or `'call'` (a `bl`/`b` elsewhere in the binary
|
||
//! whose target is that thunk). `NULL` iff `import_address` is `NULL`. Without
|
||
//! these columns an import call reads as a branch to a bare address and the
|
||
//! thunk head as two meaningless `.long`s.
|
||
//!
|
||
//! ## `jump_tables` / `jump_table_entries` / `data_in_code`
|
||
//! Recovered `switch` dispatches (see [`crate::jumptables`]). `jump_tables` has
|
||
//! one row per resolved `bctr`; `jump_table_entries` one row per case value in
|
||
//! case order. `data_in_code` lists the byte ranges those tables occupy inside
|
||
//! code sections — every word listed there is also flagged `instructions.is_data`.
|
||
//!
|
||
//! ## `rtti_type_descriptors` / `rtti_locators` / `rtti_base_classes`
|
||
//! The MSVC RTTI walk (see [`crate::rtti`]). These are the authoritative source
|
||
//! of C++ class identity: `rtti_type_descriptors.demangled_name` is the name
|
||
//! the linker wrote, not a heuristic guess. `rtti_locators.vtable_address`
|
||
//! binds a class to its vftable (`subobject_offset` separates the primary
|
||
//! vftable from the extra ones a multiply-inheriting class emits), and
|
||
//! `rtti_base_classes` is the linearised inheritance list with the PMD
|
||
//! displacement triple for each base.
|
||
//!
|
||
//! ## `xrefs`
|
||
//! One row per cross-reference edge (call, jump, data access).
|
||
//! - `source` — Absolute VA of the instruction making the reference
|
||
//! - `target` — Absolute VA of the referenced destination
|
||
//! - `kind` — Reference type as the short tag from [`crate::xref::XrefKind::tag`]:
|
||
//! `call`, `ind_call` (resolved vtable `bcctrl`),
|
||
//! `jt` (recovered `switch` case), `j` (jump),
|
||
//! `br` (branch), `read` (data_read),
|
||
//! `write` (data_write), `ref` (data_ref).
|
||
//! Note: this is a different convention from `branch_trace.kind`,
|
||
//! which uses the long names (`call` / `return` / `jump` / `branch`).
|
||
//! - `instruction` — Mnemonic of the source instruction; `NULL` if address is not in binary
|
||
//! - `source_func` — VA of the function containing `source`; `NULL` if unknown
|
||
//! - `source_label` — Label at `source`; `NULL` if none
|
||
//! - `target_label` — Label at `target`; `NULL` if none
|
||
//!
|
||
//! ## `exec_trace` *(opt-in: `--trace-instructions`)*
|
||
//! One row per executed instruction.
|
||
//! - `address` — Absolute VA of the instruction
|
||
//! - `cycle` — Monotonic instruction counter (execution order)
|
||
//! - `r3`, `r4`, `lr`, `sp` — Snapshot of key GPRs at time of execution
|
||
//!
|
||
//! ## `import_calls` *(opt-in: `--trace-imports`)*
|
||
//! One row per intercepted kernel/import call.
|
||
//! - `address` — VA of the import thunk
|
||
//! - `cycle` — Instruction counter at point of interception
|
||
//! - `module` — Library name (e.g. `xboxkrnl.exe`)
|
||
//! - `ordinal` — Numeric ordinal within the module
|
||
//! - `name` — Resolved symbol name
|
||
//! - `arg_r3`–`arg_r6` — First four call arguments (PowerPC ABI: r3–r6)
|
||
//! - `return_value` — Value in r3 after the call returns
|
||
//!
|
||
//! ## `branch_trace` *(opt-in: `--trace-branches`)*
|
||
//! One row per taken branch.
|
||
//! - `cycle` — Instruction counter
|
||
//! - `source` — VA of the branch instruction
|
||
//! - `target` — VA of the branch destination
|
||
//! - `kind` — `call`, `return`, `jump`, or `branch` (see top-level doc)
|
||
//! - `lr` — Link register value at time of branch
|
||
|
||
use std::collections::{HashMap, HashSet};
|
||
use std::path::Path;
|
||
|
||
use duckdb::{Connection, params};
|
||
|
||
use crate::formatter::DisasmInfo;
|
||
use crate::func::FuncAnalysis;
|
||
use crate::xref::{XrefMap, resolve_source_label};
|
||
|
||
const DEFAULT_BATCH_SIZE: u64 = 100_000;
|
||
|
||
/// Rows per trace buffer flush. Configurable via `XENIA_DB_BATCH_SIZE` env var (default 100_000).
|
||
/// Applies to `exec_trace` and `branch_trace` buffer thresholds.
|
||
/// `import_calls` always flushes at 1000 — low volume, not worth scaling.
|
||
fn batch_size() -> u64 {
|
||
use std::sync::OnceLock;
|
||
static CACHED: OnceLock<u64> = OnceLock::new();
|
||
*CACHED.get_or_init(|| {
|
||
std::env::var("XENIA_DB_BATCH_SIZE")
|
||
.ok()
|
||
.and_then(|s| s.parse::<u64>().ok())
|
||
.filter(|&n| n > 0)
|
||
.unwrap_or(DEFAULT_BATCH_SIZE)
|
||
})
|
||
}
|
||
|
||
pub struct ExecTraceEntry {
|
||
pub address: u32,
|
||
pub cycle: u64,
|
||
pub r3: u64,
|
||
pub r4: u64,
|
||
pub lr: u64,
|
||
pub sp: u64,
|
||
}
|
||
|
||
pub struct ImportCallEntry {
|
||
pub address: u32,
|
||
pub cycle: u64,
|
||
pub module: String,
|
||
pub ordinal: u16,
|
||
pub name: String,
|
||
pub arg_r3: u64,
|
||
pub arg_r4: u64,
|
||
pub arg_r5: u64,
|
||
pub arg_r6: u64,
|
||
pub return_value: u64,
|
||
}
|
||
|
||
pub struct BranchTraceEntry {
|
||
pub source: u32,
|
||
pub target: u32,
|
||
pub cycle: u64,
|
||
pub kind: &'static str,
|
||
pub lr: u64,
|
||
}
|
||
|
||
pub struct DbWriter {
|
||
conn: Connection,
|
||
exec_buffer: Vec<ExecTraceEntry>,
|
||
import_buffer: Vec<ImportCallEntry>,
|
||
branch_buffer: Vec<BranchTraceEntry>,
|
||
exec_count: u64,
|
||
import_count: u64,
|
||
branch_count: u64,
|
||
trace_instructions: bool,
|
||
trace_imports: bool,
|
||
trace_branches: bool,
|
||
/// Import-thunk index, built during `ingest_instructions` and reused when
|
||
/// `write_analysis_results` records the thunk records in `data_in_code`.
|
||
/// Empty until then — with no instruction rows there is nothing to
|
||
/// cross-reference it against.
|
||
import_sites: crate::imports::ImportSites,
|
||
}
|
||
|
||
impl DbWriter {
|
||
/// Open a fresh database at `path`, removing any existing file first.
|
||
pub fn open_fresh(path: &Path) -> anyhow::Result<Self> {
|
||
if path.exists() {
|
||
std::fs::remove_file(path)?;
|
||
}
|
||
let conn = Connection::open(path)?;
|
||
let cap = batch_size() as usize;
|
||
Ok(Self {
|
||
conn,
|
||
exec_buffer: Vec::with_capacity(cap),
|
||
import_buffer: Vec::with_capacity(1024),
|
||
branch_buffer: Vec::with_capacity(cap),
|
||
exec_count: 0,
|
||
import_count: 0,
|
||
branch_count: 0,
|
||
trace_instructions: false,
|
||
trace_imports: false,
|
||
trace_branches: false,
|
||
import_sites: crate::imports::ImportSites::default(),
|
||
})
|
||
}
|
||
|
||
// ── Base layer (written by extract/dis/exec) ─────────────────────────────
|
||
|
||
/// Write metadata, sections, imports tables and their indices.
|
||
#[tracing::instrument(skip_all, name = "db.write_base")]
|
||
pub fn write_base(&mut self, info: &DisasmInfo) -> anyhow::Result<()> {
|
||
self.conn.execute_batch(
|
||
"
|
||
CREATE TABLE metadata (
|
||
key VARCHAR PRIMARY KEY, -- header field name
|
||
value VARCHAR NOT NULL -- hex-formatted or plain string value
|
||
);
|
||
|
||
CREATE TABLE sections (
|
||
name VARCHAR NOT NULL, -- PE section name (e.g. .text, .rdata)
|
||
virtual_address BIGINT NOT NULL, -- RVA relative to image_base
|
||
virtual_size BIGINT NOT NULL, -- size in memory; may exceed raw_size (BSS)
|
||
raw_offset BIGINT NOT NULL, -- byte offset of section data in the file
|
||
raw_size BIGINT NOT NULL, -- size of section data on disk
|
||
flags BIGINT NOT NULL, -- IMAGE_SCN_* characteristics bit field
|
||
is_code BOOLEAN NOT NULL -- true if IMAGE_SCN_CNT_CODE is set
|
||
);
|
||
|
||
CREATE TABLE imports (
|
||
library VARCHAR NOT NULL, -- module name (e.g. xboxkrnl.exe, xam.xex)
|
||
ordinal BIGINT NOT NULL, -- ordinal identifying the export within the library
|
||
name VARCHAR, -- resolved symbol name; NULL if not in symbol table
|
||
-- 0 = variable (a pointer slot in .rdata the loader fills in),
|
||
-- 1 = function thunk (four words of code in .text). This was
|
||
-- documented the wrong way round; `ImportEntry::record_type`
|
||
-- in xenia-xex has always had it right.
|
||
record_type BIGINT NOT NULL,
|
||
-- Absolute VA of the thunk or variable. UNIQUE because
|
||
-- `instructions.import_address` is a foreign key onto it.
|
||
address BIGINT NOT NULL UNIQUE
|
||
);
|
||
",
|
||
)?;
|
||
|
||
insert_metadata(&self.conn, info)?;
|
||
insert_sections(&self.conn, info.sections)?;
|
||
insert_imports(&self.conn, info)?;
|
||
|
||
self.conn.execute_batch(
|
||
"
|
||
CREATE INDEX idx_imports_library ON imports(library);
|
||
CREATE INDEX idx_imports_name ON imports(name);
|
||
",
|
||
)?;
|
||
Ok(())
|
||
}
|
||
|
||
// ── Disasm layer (written by dis/exec) ───────────────────────────────────
|
||
|
||
/// Phase-3 ingest pass — purely mechanical disasm rows. Creates the
|
||
/// `instructions` table (and its indices) and streams every code-section
|
||
/// instruction through the iterator + DuckDB sink. Does NOT touch
|
||
/// `functions` / `labels` / `xrefs` — that's [`Self::write_analysis_results`].
|
||
///
|
||
/// `func_analysis` and `labels` are still required at this layer because
|
||
/// each row carries the rolling-window `function` and `label` columns for
|
||
/// downstream queries.
|
||
#[tracing::instrument(skip_all, name = "db.ingest_instructions")]
|
||
pub fn ingest_instructions(
|
||
&mut self,
|
||
pe: &[u8],
|
||
info: &DisasmInfo,
|
||
func_analysis: &FuncAnalysis,
|
||
labels: &HashMap<u32, String>,
|
||
data_words: &std::collections::BTreeSet<u32>,
|
||
) -> anyhow::Result<()> {
|
||
self.conn.execute_batch("
|
||
CREATE TABLE instructions (
|
||
address BIGINT PRIMARY KEY, -- absolute VA
|
||
raw BIGINT NOT NULL, -- 4-byte big-endian instruction word as integer
|
||
mnemonic VARCHAR NOT NULL, -- base mnemonic (e.g. stw, bl, cmpwi)
|
||
operands VARCHAR NOT NULL, -- operand string from base disassembly
|
||
disasm VARCHAR NOT NULL, -- full base disassembly (mnemonic + operands)
|
||
ext_mnemonic VARCHAR, -- simplified mnemonic (e.g. mr); NULL if none
|
||
ext_operands VARCHAR, -- operands for the extended form; NULL if none
|
||
ext_disasm VARCHAR, -- full extended disassembly string; NULL if none
|
||
target_hex BIGINT, -- resolved absolute target for direct branches; NULL for indirect/non-branch
|
||
section VARCHAR NOT NULL, -- PE section name containing this instruction
|
||
function BIGINT, -- VA of the enclosing function; NULL if unknown
|
||
label VARCHAR, -- label at this address; NULL if none
|
||
is_data BOOLEAN NOT NULL, -- M12: word is data embedded in code (jump table / index map / import record), NOT an instruction
|
||
-- XEX import this word references, as a foreign key into `imports`
|
||
-- (the `record_type = 1` row, whose address is the thunk head).
|
||
-- NULL for everything that is not part of, or a branch into, an
|
||
-- import thunk. See `import_role` for which of the three it is.
|
||
import_address BIGINT REFERENCES imports(address),
|
||
import_role VARCHAR -- 'record' | 'thunk' | 'call'; NULL iff import_address IS NULL
|
||
);
|
||
")?;
|
||
|
||
// Built here rather than passed in: everything it needs (the parsed
|
||
// import table and the loaded image) is already on `info`/`pe`, and
|
||
// the index is only ever consumed by this one streaming pass.
|
||
self.import_sites =
|
||
crate::imports::ImportSites::build(info.import_libraries, pe, info.image_base);
|
||
|
||
insert_instructions_streaming(
|
||
&self.conn,
|
||
pe,
|
||
info,
|
||
func_analysis,
|
||
labels,
|
||
data_words,
|
||
&self.import_sites,
|
||
)?;
|
||
|
||
let indices = [
|
||
(
|
||
"idx_instructions_function",
|
||
"CREATE INDEX idx_instructions_function ON instructions(function)",
|
||
),
|
||
(
|
||
"idx_instructions_mnemonic",
|
||
"CREATE INDEX idx_instructions_mnemonic ON instructions(mnemonic)",
|
||
),
|
||
(
|
||
"idx_instructions_ext_mnemonic",
|
||
"CREATE INDEX idx_instructions_ext_mnemonic ON instructions(ext_mnemonic)",
|
||
),
|
||
(
|
||
"idx_instructions_section",
|
||
"CREATE INDEX idx_instructions_section ON instructions(section)",
|
||
),
|
||
(
|
||
"idx_instructions_label",
|
||
"CREATE INDEX idx_instructions_label ON instructions(label)",
|
||
),
|
||
(
|
||
"idx_instructions_target_hex",
|
||
"CREATE INDEX idx_instructions_target_hex ON instructions(target_hex)",
|
||
),
|
||
(
|
||
"idx_instructions_is_data",
|
||
"CREATE INDEX idx_instructions_is_data ON instructions(is_data)",
|
||
),
|
||
(
|
||
"idx_instructions_import",
|
||
"CREATE INDEX idx_instructions_import ON instructions(import_address)",
|
||
),
|
||
];
|
||
for (name, sql) in indices {
|
||
tracing::debug!(index = name, "creating instructions index");
|
||
self.conn.execute_batch(sql)?;
|
||
}
|
||
Ok(())
|
||
}
|
||
|
||
/// Phase-3 analyze pass — writes the Rust-pass-derived tables
|
||
/// (`functions`, `labels`, `xrefs`) and their indices. Always executes
|
||
/// in `--analyze=rust` and `--analyze=both` modes; skipped only when
|
||
/// the caller deliberately chooses a Rust-free DB layout.
|
||
///
|
||
/// `vtables` is the M3 result; pass an empty slice when the caller has
|
||
/// not run the vtable scan (the tables are still created, just empty).
|
||
/// `strings` is the M7 result; same convention. `funcptr_arrays` is the
|
||
/// M8/M11 result. `typed_ind` is the M5.5 result. `eh_records` is the
|
||
/// M9.5 result. `xdbf` is the embedded title package, `None` when the XEX
|
||
/// declares no resource.
|
||
#[tracing::instrument(skip_all, name = "db.write_analysis_results")]
|
||
pub fn write_analysis_results(
|
||
&mut self,
|
||
pe: &[u8],
|
||
info: &DisasmInfo,
|
||
func_analysis: &FuncAnalysis,
|
||
labels: &HashMap<u32, String>,
|
||
xrefs: &XrefMap,
|
||
vtables: &[crate::vtables::Vtable],
|
||
strings: &[crate::strings::DetectedString],
|
||
funcptr_arrays: &[crate::funcptr_arrays::FuncPtrArray],
|
||
typed_ind: Option<&crate::ind_dispatch_typed::TypedIndirectResult>,
|
||
eh_records: &[crate::eh_scope::EhFuncInfo],
|
||
jump_tables: &[crate::jumptables::JumpTable],
|
||
rtti: &crate::rtti::RttiResult,
|
||
xdbf: Option<&crate::xdbf::Xdbf>,
|
||
) -> anyhow::Result<()> {
|
||
self.conn.execute_batch("
|
||
CREATE TABLE functions (
|
||
address BIGINT PRIMARY KEY, -- absolute VA of entry point
|
||
name VARCHAR NOT NULL, -- symbol name, or sub_XXXXXXXX if unresolved
|
||
end_address BIGINT NOT NULL, -- VA of last instruction + 4 (exclusive end)
|
||
frame_size BIGINT NOT NULL, -- stack frame size in bytes (from prologue)
|
||
saved_gprs BIGINT NOT NULL, -- bitmask of GPRs saved in prologue (bit N = rN)
|
||
is_leaf BOOLEAN NOT NULL, -- true if the function has no outgoing calls
|
||
is_saverestore BOOLEAN NOT NULL, -- true if __savegprlr_* / __restgprlr_* stub
|
||
pdata_validated BOOLEAN NOT NULL, -- true if .pdata RUNTIME_FUNCTION exists at this VA
|
||
pdata_length BIGINT, -- length in bytes per .pdata; NULL if no pdata entry
|
||
prolog_length BIGINT, -- prolog size in bytes per .pdata; NULL if no pdata entry
|
||
has_eh BOOLEAN NOT NULL -- M9: pdata exception-flag bit set; function has C++ EH/SEH
|
||
);
|
||
|
||
CREATE TABLE pdata_entries (
|
||
begin_address BIGINT PRIMARY KEY, -- absolute VA of function start (RUNTIME_FUNCTION.BeginAddress)
|
||
end_address BIGINT NOT NULL, -- begin_address + function_length (exclusive)
|
||
function_length BIGINT NOT NULL, -- function size in bytes
|
||
prolog_length BIGINT NOT NULL, -- prolog size in bytes
|
||
flags BIGINT NOT NULL -- raw 2-bit flags (bit 1=32-bit-code, bit 0=exception)
|
||
);
|
||
|
||
CREATE TABLE labels (
|
||
address BIGINT PRIMARY KEY, -- absolute VA
|
||
name VARCHAR NOT NULL, -- symbol name
|
||
kind VARCHAR NOT NULL -- function | import | saverestore | local | data | other
|
||
);
|
||
|
||
-- M13 — MSVC RTTI. `rtti_type_descriptors` is the authoritative
|
||
-- source of C++ class identity: the linker wrote these names, they
|
||
-- are not inferred.
|
||
CREATE TABLE rtti_type_descriptors (
|
||
address BIGINT PRIMARY KEY, -- VA of the TypeDescriptor
|
||
mangled_name VARCHAR NOT NULL, -- decorated name, e.g. .?AVSilph@silph@@
|
||
demangled_name VARCHAR NOT NULL -- readable form, e.g. silph::Silph
|
||
);
|
||
|
||
-- M13 — RTTICompleteObjectLocator. One per emitted vftable; the
|
||
-- `subobject_offset` column separates a class's primary vftable (0)
|
||
-- from the extra vftables it emits for secondary base subobjects.
|
||
CREATE TABLE rtti_locators (
|
||
address BIGINT PRIMARY KEY,
|
||
subobject_offset BIGINT NOT NULL, -- this-offset of the subobject this vftable serves
|
||
cd_offset BIGINT NOT NULL, -- constructor-displacement offset
|
||
type_descriptor BIGINT NOT NULL REFERENCES rtti_type_descriptors(address),
|
||
class_hierarchy BIGINT NOT NULL, -- VA of the RTTIClassHierarchyDescriptor
|
||
-- VA of vftable[0]; NULL if no referencing word was found.
|
||
-- Deliberately NOT a foreign key: on the reference title 41
|
||
-- locators name a vftable the M3 scanner did not detect (see
|
||
-- `crate::vtables`), so declaring one would fail the build on
|
||
-- a real gap rather than reporting it. Query the orphans with
|
||
-- SELECT * FROM rtti_locators l WHERE NOT EXISTS
|
||
-- (SELECT 1 FROM vtables v WHERE v.address = l.vtable_address)
|
||
vtable_address BIGINT
|
||
);
|
||
|
||
-- M13 — linearised RTTIBaseClassArray. Index 0 is the class itself;
|
||
-- the remaining rows are its bases in MSVC's depth-first order,
|
||
-- each with the PMD displacement triple needed to locate the base
|
||
-- subobject inside an instance.
|
||
CREATE TABLE rtti_base_classes (
|
||
class_hierarchy BIGINT NOT NULL, -- VA of the deriving class's hierarchy descriptor
|
||
base_index BIGINT NOT NULL, -- position in the base-class array
|
||
type_descriptor BIGINT NOT NULL REFERENCES rtti_type_descriptors(address),
|
||
name VARCHAR NOT NULL,
|
||
num_contained_bases BIGINT NOT NULL,
|
||
mdisp BIGINT NOT NULL, -- member displacement
|
||
pdisp BIGINT NOT NULL, -- vbtable displacement (-1 = non-virtual base)
|
||
vdisp BIGINT NOT NULL, -- displacement inside the vbtable
|
||
attributes BIGINT NOT NULL,
|
||
PRIMARY KEY (class_hierarchy, base_index)
|
||
);
|
||
|
||
CREATE TABLE vtables (
|
||
address BIGINT PRIMARY KEY, -- absolute VA of vtable[0]
|
||
length BIGINT NOT NULL, -- number of method slots
|
||
-- VA of the CompleteObjectLocator; NULL when the class has no RTTI.
|
||
col_address BIGINT REFERENCES rtti_locators(address),
|
||
class_name VARCHAR NOT NULL, -- demangled class name OR ANON_Class_<hash> when stripped
|
||
rtti_present BOOLEAN NOT NULL, -- true when COL → TypeDescriptor walk succeeded
|
||
base_classes_json VARCHAR -- JSON array of base class names (NULL if none / parse failure)
|
||
);
|
||
|
||
CREATE TABLE methods (
|
||
vtable_address BIGINT NOT NULL REFERENCES vtables(address), -- vtable this slot belongs to
|
||
slot BIGINT NOT NULL, -- 0-based slot index
|
||
function_address BIGINT NOT NULL, -- VA of the function this slot points at
|
||
mangled_name VARCHAR, -- raw label name when mangled (?...)
|
||
demangled_name VARCHAR, -- LLVM-style demangled output
|
||
PRIMARY KEY (vtable_address, slot)
|
||
);
|
||
|
||
CREATE TABLE classes (
|
||
name VARCHAR PRIMARY KEY, -- class name (demangled or ANON_*)
|
||
vtable_address BIGINT NOT NULL REFERENCES vtables(address), -- representative vtable (first detected)
|
||
rtti_present BOOLEAN NOT NULL,
|
||
base_classes_json VARCHAR -- JSON of base class names (NULL when stripped)
|
||
);
|
||
|
||
CREATE TABLE strings (
|
||
address BIGINT PRIMARY KEY, -- absolute VA of first byte
|
||
encoding VARCHAR NOT NULL, -- 'ascii' | 'utf16le' | 'shift_jis' | 'utf8'
|
||
length BIGINT NOT NULL, -- length in bytes (excluding NUL terminator)
|
||
content VARCHAR NOT NULL, -- UTF-8 representation of the string
|
||
section VARCHAR NOT NULL -- PE section the string lives in (.rdata / .data)
|
||
);
|
||
|
||
CREATE TABLE tls_info (
|
||
raw_data_start BIGINT NOT NULL, -- VA of TLS template start
|
||
raw_data_end BIGINT NOT NULL, -- VA one-past-end of TLS template
|
||
index_address BIGINT NOT NULL, -- VA of u32 the loader writes the assigned slot index into
|
||
callback_array BIGINT NOT NULL, -- VA of zero-terminated callback array (0 if none)
|
||
zero_fill_size BIGINT NOT NULL, -- bytes of zero-fill appended after raw template
|
||
characteristics BIGINT NOT NULL -- IMAGE_TLS_DIRECTORY characteristics flags
|
||
);
|
||
|
||
CREATE TABLE tls_callbacks (
|
||
slot BIGINT PRIMARY KEY, -- 0-based index in the callback array
|
||
address BIGINT NOT NULL -- VA of callback function
|
||
);
|
||
|
||
CREATE TABLE function_pointer_arrays (
|
||
address BIGINT PRIMARY KEY, -- absolute VA of the array's first slot
|
||
length BIGINT NOT NULL, -- number of slots
|
||
kind VARCHAR NOT NULL -- 'vtable' (M3) | 'dispatch_table' (M8) | 'static_init' (M11)
|
||
);
|
||
|
||
CREATE TABLE function_pointer_array_entries (
|
||
array_address BIGINT NOT NULL REFERENCES function_pointer_arrays(address),
|
||
slot BIGINT NOT NULL, -- 0-based slot index
|
||
function_address BIGINT NOT NULL, -- VA of the function this slot points at
|
||
PRIMARY KEY (array_address, slot)
|
||
);
|
||
|
||
-- M5.5 — typed indirect-dispatch resolutions. Each row is one
|
||
-- bcctrl site that matched the canonical lwz vt, off(this);
|
||
-- lwz fn, slot(vt); mtctr; bcctrl pattern. candidate_count > 1
|
||
-- means the analysis could not pick a single class; downstream
|
||
-- queries should treat such rows as reachability-only. When
|
||
-- `truncated` is set the site had more candidates than the
|
||
-- ceiling and none were materialised — the call is virtual and
|
||
-- unresolved, and `candidate_count` says how unresolved.
|
||
CREATE TABLE indirect_dispatch_sites (
|
||
dispatch_pc BIGINT PRIMARY KEY,
|
||
vptr_offset BIGINT NOT NULL,
|
||
slot BIGINT NOT NULL,
|
||
candidate_count BIGINT NOT NULL, -- candidates that matched, materialised or not
|
||
truncated BOOLEAN NOT NULL -- true => candidate_count exceeded the ceiling,
|
||
-- so no rows in indirect_dispatch_candidates
|
||
);
|
||
|
||
-- M5.5 — one row per (dispatch site × candidate vtable). The
|
||
-- ind_call xref edges in the `xrefs` table are derived from
|
||
-- this; this view lets you join back to vtable / method info.
|
||
CREATE TABLE indirect_dispatch_candidates (
|
||
dispatch_pc BIGINT NOT NULL REFERENCES indirect_dispatch_sites(dispatch_pc),
|
||
vtable_address BIGINT NOT NULL REFERENCES vtables(address),
|
||
-- No FK: a vtable slot may point at code `.pdata` never declared
|
||
-- and no `bl` ever targets, so this is not always in `functions`.
|
||
method_address BIGINT NOT NULL,
|
||
PRIMARY KEY (dispatch_pc, vtable_address)
|
||
);
|
||
|
||
-- M5.5 — every detected `stw rVtable, vptr_off(rThis)` writer
|
||
-- found in any function. Useful for diagnosing why a class
|
||
-- has (or does not have) coverage in the dispatch resolver.
|
||
CREATE TABLE vptr_writes (
|
||
writer_pc BIGINT NOT NULL,
|
||
vtable_address BIGINT NOT NULL REFERENCES vtables(address),
|
||
vptr_offset BIGINT NOT NULL,
|
||
writer_function BIGINT NOT NULL,
|
||
PRIMARY KEY (writer_pc, vtable_address, vptr_offset)
|
||
);
|
||
|
||
-- M9.5 — MSVC __CxxFrameHandler scope-table records found by
|
||
-- magic-number scan in .rdata.
|
||
CREATE TABLE eh_funcinfo (
|
||
address BIGINT PRIMARY KEY,
|
||
magic BIGINT NOT NULL, -- 0x19930520/21/22
|
||
max_state BIGINT NOT NULL,
|
||
p_unwind_map BIGINT NOT NULL,
|
||
n_try_blocks BIGINT NOT NULL,
|
||
p_try_block_map BIGINT NOT NULL,
|
||
n_ip_map_entries BIGINT NOT NULL,
|
||
p_ip_to_state_map BIGINT NOT NULL,
|
||
p_es_type_list BIGINT,
|
||
eh_flags BIGINT
|
||
);
|
||
|
||
CREATE TABLE eh_unwind_map (
|
||
funcinfo_address BIGINT NOT NULL REFERENCES eh_funcinfo(address),
|
||
state_index BIGINT NOT NULL,
|
||
to_state BIGINT NOT NULL,
|
||
action_pc BIGINT NOT NULL,
|
||
PRIMARY KEY (funcinfo_address, state_index)
|
||
);
|
||
|
||
CREATE TABLE eh_try_blocks (
|
||
funcinfo_address BIGINT NOT NULL REFERENCES eh_funcinfo(address),
|
||
try_index BIGINT NOT NULL,
|
||
try_low BIGINT NOT NULL,
|
||
try_high BIGINT NOT NULL,
|
||
catch_high BIGINT NOT NULL,
|
||
n_catches BIGINT NOT NULL,
|
||
p_handler_array BIGINT NOT NULL,
|
||
PRIMARY KEY (funcinfo_address, try_index)
|
||
);
|
||
|
||
-- XDBF/SPA package embedded in the XEX (see `crate::xdbf`).
|
||
-- One row per entry of the container's entry table.
|
||
CREATE TABLE xdbf_entries (
|
||
namespace BIGINT NOT NULL, -- 1=metadata, 2=image, 3=string table
|
||
namespace_name VARCHAR NOT NULL,
|
||
id BIGINT NOT NULL, -- fourcc / language / image id per namespace
|
||
body_offset BIGINT NOT NULL, -- offset of the body within the image buffer
|
||
size BIGINT NOT NULL,
|
||
magic VARCHAR, -- leading fourcc of the body, when printable
|
||
PRIMARY KEY (namespace, id)
|
||
);
|
||
|
||
CREATE TABLE xdbf_images (
|
||
id BIGINT PRIMARY KEY, -- image id referenced by achievements
|
||
is_title_icon BOOLEAN NOT NULL, -- id 0x8000 — the title's own icon
|
||
body_offset BIGINT NOT NULL, -- offset within the image buffer
|
||
size BIGINT NOT NULL,
|
||
format VARCHAR NOT NULL -- 'png' when the body carries the PNG signature
|
||
);
|
||
|
||
CREATE TABLE xdbf_achievements (
|
||
id BIGINT PRIMARY KEY, -- 1-based achievement id
|
||
name VARCHAR, -- resolved via the default language's string table
|
||
unlocked_desc VARCHAR,
|
||
locked_desc VARCHAR,
|
||
label_id BIGINT NOT NULL, -- string ids, for joining other languages
|
||
description_id BIGINT NOT NULL,
|
||
unachieved_id BIGINT NOT NULL,
|
||
image_id BIGINT NOT NULL REFERENCES xdbf_images(id),
|
||
gamerscore BIGINT NOT NULL,
|
||
flags BIGINT NOT NULL
|
||
);
|
||
|
||
-- Every localized string in the package. This is where the title
|
||
-- name, mission titles, game-phase labels and leaderboard names live.
|
||
CREATE TABLE xdbf_strings (
|
||
language BIGINT NOT NULL, -- XLanguage value
|
||
language_name VARCHAR NOT NULL,
|
||
string_id BIGINT NOT NULL,
|
||
value VARCHAR NOT NULL,
|
||
PRIMARY KEY (language, string_id)
|
||
);
|
||
|
||
CREATE TABLE demangled_names (
|
||
address BIGINT, -- VA the mangled name is associated with; NULL when from a non-address source (e.g. RTTI-only string)
|
||
mangled VARCHAR NOT NULL, -- original mangled symbol (e.g. ?Foo@Bar@@QEAAXXZ)
|
||
raw_demangled VARCHAR NOT NULL, -- LLVM-style demangled output (or mangled string on parse failure)
|
||
namespace_path VARCHAR, -- e.g. xe::apu (NULL = global / parser failure)
|
||
class_name VARCHAR, -- e.g. AudioSystem (NULL = free function / parser failure)
|
||
method_name VARCHAR, -- e.g. Setup (NULL on parser failure)
|
||
params_signature VARCHAR -- contents of the outermost (...) (NULL = not a function)
|
||
);
|
||
|
||
-- M12 — recovered `switch` dispatches. One row per `bctr` whose
|
||
-- jump table the analyzer could resolve and validate.
|
||
CREATE TABLE jump_tables (
|
||
bctr_pc BIGINT PRIMARY KEY, -- VA of the dispatching bctr
|
||
function BIGINT, -- VA of the enclosing function
|
||
table_address BIGINT NOT NULL, -- VA of the absolute-target table
|
||
entry_count BIGINT NOT NULL, -- number of case values (after index-map expansion)
|
||
table_slots BIGINT NOT NULL, -- 4-byte slots occupied by the target table itself
|
||
index_map_address BIGINT, -- VA of the byte-wide index map (sparse switch only)
|
||
index_map_count BIGINT, -- bytes read from the index map
|
||
case_bound BIGINT, -- largest valid case index per the cmplwi bound check
|
||
kind VARCHAR NOT NULL -- 'direct' (table[idx]) | 'indexed' (table[map[idx]])
|
||
);
|
||
|
||
-- M12 — one row per case value, in case order. `target_address`
|
||
-- repeats when several case values share a body.
|
||
CREATE TABLE jump_table_entries (
|
||
bctr_pc BIGINT NOT NULL REFERENCES jump_tables(bctr_pc),
|
||
case_index BIGINT NOT NULL, -- 0-based case value
|
||
target_address BIGINT NOT NULL, -- VA of the case body
|
||
PRIMARY KEY (bctr_pc, case_index)
|
||
);
|
||
|
||
-- M12 — byte ranges inside code sections that hold data, not
|
||
-- instructions. Anything listed here is a decoding hazard: linear
|
||
-- disassembly of these words produces garbage rows.
|
||
CREATE TABLE data_in_code (
|
||
address BIGINT PRIMARY KEY REFERENCES instructions(address), -- VA of the first byte
|
||
length BIGINT NOT NULL, -- byte length
|
||
kind VARCHAR NOT NULL -- 'jump_table' | 'jump_index_map' | 'import_record'
|
||
);
|
||
|
||
CREATE TABLE xrefs (
|
||
source BIGINT NOT NULL REFERENCES instructions(address),
|
||
-- No FK: a target may be data (.rdata/.data) or an address with
|
||
-- no decoded word, so it is not always an `instructions` row.
|
||
target BIGINT NOT NULL,
|
||
kind VARCHAR NOT NULL, -- call | ind_call | j | br | read | write | ref
|
||
addr_mode VARCHAR, -- M6 sub-classification of how source computes target (NULL for control-flow)
|
||
instruction VARCHAR, -- mnemonic of source instruction; NULL if not in binary
|
||
source_func BIGINT, -- VA of the function containing source; NULL if unknown
|
||
source_label VARCHAR, -- label at source; NULL if none
|
||
target_label VARCHAR -- label at target; NULL if none
|
||
);
|
||
")?;
|
||
|
||
// Every table above a few thousand rows goes through the DuckDB
|
||
// Appender rather than a row-at-a-time `INSERT`.
|
||
//
|
||
// This is not a micro-optimisation. DuckDB autocommits each statement,
|
||
// so a per-row `INSERT` loop pays a transaction + WAL flush per row:
|
||
// the 221k rows across `functions` / `labels` / `pdata_entries` alone
|
||
// took 20 minutes, and the 1.8M `indirect_dispatch_candidates` rows
|
||
// took ~59 more — 81 minutes for one database. Wrapping the lot in a
|
||
// single explicit transaction fixes the time but not the cause: DuckDB
|
||
// buffers per-statement, so the uncommitted set grew to ~16 GB RSS.
|
||
// The Appender writes directly to columnar storage in bounded chunks,
|
||
// which is both fast and flat in memory. It bypasses the SQL layer,
|
||
// so `ON CONFLICT DO NOTHING` is unavailable and each converted sink
|
||
// documents why its key cannot collide (or dedupes explicitly).
|
||
// Insertion order is load-bearing: the schema declares foreign keys,
|
||
// and DuckDB checks them per row, so a parent table must be populated
|
||
// before any child that references it. `insert_rtti` runs ahead of
|
||
// `insert_vtables` because `vtables.col_address` points into
|
||
// `rtti_locators`; everything else follows the natural dependency
|
||
// order (functions -> vtables -> methods/classes -> dispatch).
|
||
insert_functions(&self.conn, func_analysis, labels)?;
|
||
insert_pdata_entries(&self.conn, &func_analysis.pdata_entries)?;
|
||
insert_labels(&self.conn, labels)?;
|
||
insert_demangled_from_labels(&self.conn, labels, info.import_libraries)?;
|
||
insert_rtti(&self.conn, rtti)?;
|
||
insert_vtables(&self.conn, vtables, pe, info.image_base)?;
|
||
insert_methods_and_classes(&self.conn, vtables, labels)?;
|
||
insert_strings(&self.conn, strings)?;
|
||
insert_funcptr_arrays(&self.conn, funcptr_arrays)?;
|
||
insert_eh_records(&self.conn, eh_records)?;
|
||
insert_jump_tables(&self.conn, jump_tables)?;
|
||
insert_import_record_extents(&self.conn, &self.import_sites)?;
|
||
insert_xdbf(&self.conn, xdbf)?;
|
||
if let Some(t) = typed_ind {
|
||
insert_typed_ind_dispatch(&self.conn, t)?;
|
||
}
|
||
insert_xrefs_streaming(
|
||
&self.conn,
|
||
xrefs,
|
||
pe,
|
||
info.image_base,
|
||
func_analysis,
|
||
labels,
|
||
)?;
|
||
|
||
let indices = [
|
||
(
|
||
"idx_functions_name",
|
||
"CREATE INDEX idx_functions_name ON functions(name)",
|
||
),
|
||
(
|
||
"idx_functions_pdata_validated",
|
||
"CREATE INDEX idx_functions_pdata_validated ON functions(pdata_validated)",
|
||
),
|
||
(
|
||
"idx_functions_has_eh",
|
||
"CREATE INDEX idx_functions_has_eh ON functions(has_eh)",
|
||
),
|
||
(
|
||
"idx_labels_kind",
|
||
"CREATE INDEX idx_labels_kind ON labels(kind)",
|
||
),
|
||
(
|
||
"idx_labels_name",
|
||
"CREATE INDEX idx_labels_name ON labels(name)",
|
||
),
|
||
(
|
||
"idx_demangled_address",
|
||
"CREATE INDEX idx_demangled_address ON demangled_names(address)",
|
||
),
|
||
(
|
||
"idx_demangled_class",
|
||
"CREATE INDEX idx_demangled_class ON demangled_names(class_name)",
|
||
),
|
||
(
|
||
"idx_demangled_method",
|
||
"CREATE INDEX idx_demangled_method ON demangled_names(method_name)",
|
||
),
|
||
(
|
||
"idx_methods_function",
|
||
"CREATE INDEX idx_methods_function ON methods(function_address)",
|
||
),
|
||
(
|
||
"idx_classes_rtti",
|
||
"CREATE INDEX idx_classes_rtti ON classes(rtti_present)",
|
||
),
|
||
(
|
||
"idx_strings_encoding",
|
||
"CREATE INDEX idx_strings_encoding ON strings(encoding)",
|
||
),
|
||
(
|
||
"idx_xrefs_addr_mode",
|
||
"CREATE INDEX idx_xrefs_addr_mode ON xrefs(addr_mode)",
|
||
),
|
||
(
|
||
"idx_fparrays_kind",
|
||
"CREATE INDEX idx_fparrays_kind ON function_pointer_arrays(kind)",
|
||
),
|
||
(
|
||
"idx_fpentries_function",
|
||
"CREATE INDEX idx_fpentries_function ON function_pointer_array_entries(function_address)",
|
||
),
|
||
(
|
||
"idx_indcand_method",
|
||
"CREATE INDEX idx_indcand_method ON indirect_dispatch_candidates(method_address)",
|
||
),
|
||
(
|
||
"idx_indcand_vtable",
|
||
"CREATE INDEX idx_indcand_vtable ON indirect_dispatch_candidates(vtable_address)",
|
||
),
|
||
(
|
||
"idx_indsites_offset_slot",
|
||
"CREATE INDEX idx_indsites_offset_slot ON indirect_dispatch_sites(vptr_offset, slot)",
|
||
),
|
||
(
|
||
"idx_vptrw_vtable",
|
||
"CREATE INDEX idx_vptrw_vtable ON vptr_writes(vtable_address)",
|
||
),
|
||
(
|
||
"idx_vptrw_offset",
|
||
"CREATE INDEX idx_vptrw_offset ON vptr_writes(vptr_offset)",
|
||
),
|
||
(
|
||
"idx_xrefs_target",
|
||
"CREATE INDEX idx_xrefs_target ON xrefs(target)",
|
||
),
|
||
(
|
||
"idx_xrefs_source",
|
||
"CREATE INDEX idx_xrefs_source ON xrefs(source)",
|
||
),
|
||
(
|
||
"idx_xrefs_source_func",
|
||
"CREATE INDEX idx_xrefs_source_func ON xrefs(source_func)",
|
||
),
|
||
(
|
||
"idx_xrefs_kind",
|
||
"CREATE INDEX idx_xrefs_kind ON xrefs(kind)",
|
||
),
|
||
(
|
||
"idx_xrefs_instruction",
|
||
"CREATE INDEX idx_xrefs_instruction ON xrefs(instruction)",
|
||
),
|
||
(
|
||
"idx_xrefs_target_label",
|
||
"CREATE INDEX idx_xrefs_target_label ON xrefs(target_label)",
|
||
),
|
||
];
|
||
for (name, sql) in indices {
|
||
tracing::debug!(index = name, "creating analysis index");
|
||
self.conn.execute_batch(sql)?;
|
||
}
|
||
Ok(())
|
||
}
|
||
|
||
/// Back-compat wrapper for callers that want the full pre-Phase-3
|
||
/// "everything in one shot" behaviour. Equivalent to
|
||
/// `ingest_instructions` + `write_analysis_results` with no M3 vtables /
|
||
/// M7 strings.
|
||
#[tracing::instrument(skip_all, name = "db.write_disasm")]
|
||
pub fn write_disasm(
|
||
&mut self,
|
||
pe: &[u8],
|
||
info: &DisasmInfo,
|
||
func_analysis: &FuncAnalysis,
|
||
labels: &HashMap<u32, String>,
|
||
xrefs: &XrefMap,
|
||
) -> anyhow::Result<()> {
|
||
let empty = std::collections::BTreeSet::new();
|
||
self.ingest_instructions(pe, info, func_analysis, labels, &empty)?;
|
||
self.write_analysis_results(
|
||
pe,
|
||
info,
|
||
func_analysis,
|
||
labels,
|
||
xrefs,
|
||
&[],
|
||
&[],
|
||
&[],
|
||
None,
|
||
&[],
|
||
&[],
|
||
&crate::rtti::RttiResult::default(),
|
||
None,
|
||
)?;
|
||
Ok(())
|
||
}
|
||
|
||
/// M10 — write the parsed `.tls` directory + callback array. No-op
|
||
/// when `tls` is `None` (binary has no `.tls` section).
|
||
#[tracing::instrument(skip_all, name = "db.write_tls")]
|
||
pub fn write_tls(&mut self, tls: Option<&sylpheed_xex::tls::TlsInfo>) -> anyhow::Result<()> {
|
||
let Some(t) = tls else {
|
||
return Ok(());
|
||
};
|
||
self.conn.execute(
|
||
"INSERT INTO tls_info (raw_data_start, raw_data_end, index_address,
|
||
callback_array, zero_fill_size, characteristics)
|
||
VALUES (?, ?, ?, ?, ?, ?)",
|
||
params![
|
||
t.raw_data_start as i64,
|
||
t.raw_data_end as i64,
|
||
t.index_address as i64,
|
||
t.callback_array as i64,
|
||
t.zero_fill_size as i64,
|
||
t.characteristics as i64,
|
||
],
|
||
)?;
|
||
let mut stmt = self
|
||
.conn
|
||
.prepare("INSERT INTO tls_callbacks (slot, address) VALUES (?, ?)")?;
|
||
for (i, cb) in t.callbacks.iter().enumerate() {
|
||
stmt.execute(params![i as i64, cb.address as i64])?;
|
||
}
|
||
metrics::counter!("db.rows", "table" => "tls_callbacks")
|
||
.increment(t.callbacks.len() as u64);
|
||
tracing::info!(
|
||
rows = t.callbacks.len(),
|
||
table = "tls_callbacks",
|
||
"tls write complete"
|
||
);
|
||
Ok(())
|
||
}
|
||
|
||
/// Phase-3 SQL-views layer — defines additive read-only views over
|
||
/// `instructions` (and optionally `xrefs`/`functions`/`labels`).
|
||
/// See [`crate::sql_views`] for the SQL definitions.
|
||
///
|
||
/// Called when `--analyze=sql` or `--analyze=both` is in effect.
|
||
#[tracing::instrument(skip_all, name = "db.create_sql_views")]
|
||
pub fn create_sql_views(&mut self) -> anyhow::Result<()> {
|
||
for (name, sql) in crate::sql_views::ALL_VIEWS {
|
||
tracing::debug!(view = name, "creating SQL view");
|
||
self.conn.execute_batch(sql)?;
|
||
}
|
||
Ok(())
|
||
}
|
||
|
||
/// Cross-check: count branch xrefs found by the SQL view that are absent
|
||
/// from the Rust-pass `xrefs` table (and vice versa). Returns
|
||
/// `(sql_only, rust_only)` row counts. Both should be zero — the two
|
||
/// surfaces produce identical edges by construction. A non-zero count
|
||
/// signals drift between the formatter's `mnemonic` column and
|
||
/// `xref.rs`'s opcode classification, and is logged as a warning by the
|
||
/// caller.
|
||
#[tracing::instrument(skip_all, name = "db.cross_check_branch_xrefs")]
|
||
pub fn cross_check_branch_xrefs(&self) -> anyhow::Result<(u64, u64)> {
|
||
let sql_only: i64 = self.conn.query_row(
|
||
"SELECT COUNT(*) FROM v_branch_xrefs vb \
|
||
LEFT JOIN xrefs x \
|
||
ON x.source = vb.source AND x.target = vb.target AND x.kind = vb.kind \
|
||
WHERE x.source IS NULL",
|
||
[],
|
||
|row| row.get(0),
|
||
)?;
|
||
let rust_only: i64 = self.conn.query_row(
|
||
"SELECT COUNT(*) FROM xrefs x \
|
||
LEFT JOIN v_branch_xrefs vb \
|
||
ON vb.source = x.source AND vb.target = x.target AND vb.kind = x.kind \
|
||
WHERE x.kind IN ('call','j','br') AND vb.source IS NULL",
|
||
[],
|
||
|row| row.get(0),
|
||
)?;
|
||
Ok((sql_only as u64, rust_only as u64))
|
||
}
|
||
|
||
// ── Trace layer (written by exec when flags enabled) ─────────────────────
|
||
|
||
/// Create the opt-in trace tables. No-op if all flags are false.
|
||
pub fn prepare_trace_tables(
|
||
&mut self,
|
||
trace_instructions: bool,
|
||
trace_imports: bool,
|
||
trace_branches: bool,
|
||
) -> anyhow::Result<()> {
|
||
self.trace_instructions = trace_instructions;
|
||
self.trace_imports = trace_imports;
|
||
self.trace_branches = trace_branches;
|
||
|
||
if trace_instructions {
|
||
self.conn.execute_batch(
|
||
"
|
||
CREATE TABLE exec_trace (
|
||
address BIGINT NOT NULL, -- absolute VA of the instruction
|
||
cycle BIGINT NOT NULL, -- monotonic instruction counter (execution order)
|
||
r3 BIGINT NOT NULL, -- r3 at time of execution
|
||
r4 BIGINT NOT NULL, -- r4 at time of execution
|
||
lr BIGINT NOT NULL, -- link register
|
||
sp BIGINT NOT NULL -- stack pointer
|
||
);
|
||
",
|
||
)?;
|
||
}
|
||
|
||
if trace_imports {
|
||
self.conn.execute_batch(
|
||
"
|
||
CREATE TABLE import_calls (
|
||
address BIGINT NOT NULL, -- VA of the import thunk
|
||
cycle BIGINT NOT NULL, -- instruction counter at interception
|
||
module VARCHAR NOT NULL, -- library name (e.g. xboxkrnl.exe)
|
||
ordinal BIGINT NOT NULL, -- ordinal within the module
|
||
name VARCHAR NOT NULL, -- resolved symbol name
|
||
arg_r3 BIGINT NOT NULL, -- first argument (r3)
|
||
arg_r4 BIGINT NOT NULL, -- second argument (r4)
|
||
arg_r5 BIGINT NOT NULL, -- third argument (r5)
|
||
arg_r6 BIGINT NOT NULL, -- fourth argument (r6)
|
||
return_value BIGINT NOT NULL -- r3 after the call returns
|
||
);
|
||
",
|
||
)?;
|
||
}
|
||
|
||
if trace_branches {
|
||
self.conn.execute_batch(
|
||
"
|
||
CREATE TABLE branch_trace (
|
||
cycle BIGINT NOT NULL, -- instruction counter
|
||
source BIGINT NOT NULL, -- VA of the branch instruction
|
||
target BIGINT NOT NULL, -- VA of the branch destination
|
||
kind VARCHAR NOT NULL, -- call | return | jump | branch
|
||
lr BIGINT NOT NULL -- link register at time of branch
|
||
);
|
||
",
|
||
)?;
|
||
}
|
||
|
||
Ok(())
|
||
}
|
||
|
||
pub fn log_instruction(&mut self, entry: ExecTraceEntry) {
|
||
if !self.trace_instructions {
|
||
return;
|
||
}
|
||
self.exec_buffer.push(entry);
|
||
if self.exec_buffer.len() as u64 >= batch_size() {
|
||
self.flush_exec();
|
||
}
|
||
}
|
||
|
||
pub fn log_import_call(&mut self, entry: ImportCallEntry) {
|
||
if !self.trace_imports {
|
||
return;
|
||
}
|
||
self.import_buffer.push(entry);
|
||
if self.import_buffer.len() >= 1000 {
|
||
self.flush_imports();
|
||
}
|
||
}
|
||
|
||
pub fn log_branch(&mut self, entry: BranchTraceEntry) {
|
||
if !self.trace_branches {
|
||
return;
|
||
}
|
||
self.branch_buffer.push(entry);
|
||
if self.branch_buffer.len() as u64 >= batch_size() {
|
||
self.flush_branches();
|
||
}
|
||
}
|
||
|
||
fn flush_exec(&mut self) {
|
||
if self.exec_buffer.is_empty() {
|
||
return;
|
||
}
|
||
let mut appender = self.conn.appender("exec_trace").unwrap();
|
||
for e in &self.exec_buffer {
|
||
appender
|
||
.append_row(params![
|
||
e.address as i64,
|
||
e.cycle as i64,
|
||
e.r3 as i64,
|
||
e.r4 as i64,
|
||
e.lr as i64,
|
||
e.sp as i64,
|
||
])
|
||
.ok();
|
||
}
|
||
appender.flush().ok();
|
||
self.exec_count += self.exec_buffer.len() as u64;
|
||
self.exec_buffer.clear();
|
||
}
|
||
|
||
fn flush_imports(&mut self) {
|
||
if self.import_buffer.is_empty() {
|
||
return;
|
||
}
|
||
let mut appender = self.conn.appender("import_calls").unwrap();
|
||
for e in &self.import_buffer {
|
||
appender
|
||
.append_row(params![
|
||
e.address as i64,
|
||
e.cycle as i64,
|
||
e.module.as_str(),
|
||
e.ordinal as i64,
|
||
e.name.as_str(),
|
||
e.arg_r3 as i64,
|
||
e.arg_r4 as i64,
|
||
e.arg_r5 as i64,
|
||
e.arg_r6 as i64,
|
||
e.return_value as i64,
|
||
])
|
||
.ok();
|
||
}
|
||
appender.flush().ok();
|
||
self.import_count += self.import_buffer.len() as u64;
|
||
self.import_buffer.clear();
|
||
}
|
||
|
||
fn flush_branches(&mut self) {
|
||
if self.branch_buffer.is_empty() {
|
||
return;
|
||
}
|
||
let mut appender = self.conn.appender("branch_trace").unwrap();
|
||
for e in &self.branch_buffer {
|
||
appender
|
||
.append_row(params![
|
||
e.cycle as i64,
|
||
e.source as i64,
|
||
e.target as i64,
|
||
e.kind,
|
||
e.lr as i64,
|
||
])
|
||
.ok();
|
||
}
|
||
appender.flush().ok();
|
||
self.branch_count += self.branch_buffer.len() as u64;
|
||
self.branch_buffer.clear();
|
||
}
|
||
|
||
/// Flush remaining trace buffers and create their indices.
|
||
#[tracing::instrument(skip_all, name = "db.finalize_traces")]
|
||
pub fn finalize_traces(&mut self) -> anyhow::Result<()> {
|
||
self.flush_exec();
|
||
self.flush_imports();
|
||
self.flush_branches();
|
||
|
||
if self.trace_instructions {
|
||
tracing::debug!("creating idx_exec_trace_address");
|
||
self.conn
|
||
.execute_batch("CREATE INDEX idx_exec_trace_address ON exec_trace(address);")?;
|
||
tracing::debug!("creating idx_exec_trace_cycle");
|
||
self.conn
|
||
.execute_batch("CREATE INDEX idx_exec_trace_cycle ON exec_trace(cycle);")?;
|
||
}
|
||
if self.trace_imports {
|
||
tracing::debug!("creating idx_import_calls_name");
|
||
self.conn
|
||
.execute_batch("CREATE INDEX idx_import_calls_name ON import_calls(name);")?;
|
||
tracing::debug!("creating idx_import_calls_cycle");
|
||
self.conn
|
||
.execute_batch("CREATE INDEX idx_import_calls_cycle ON import_calls(cycle);")?;
|
||
}
|
||
if self.trace_branches {
|
||
tracing::debug!("creating idx_branch_trace_source");
|
||
self.conn
|
||
.execute_batch("CREATE INDEX idx_branch_trace_source ON branch_trace(source);")?;
|
||
tracing::debug!("creating idx_branch_trace_target");
|
||
self.conn
|
||
.execute_batch("CREATE INDEX idx_branch_trace_target ON branch_trace(target);")?;
|
||
tracing::debug!("creating idx_branch_trace_kind");
|
||
self.conn
|
||
.execute_batch("CREATE INDEX idx_branch_trace_kind ON branch_trace(kind);")?;
|
||
tracing::debug!("creating idx_branch_trace_cycle");
|
||
self.conn
|
||
.execute_batch("CREATE INDEX idx_branch_trace_cycle ON branch_trace(cycle);")?;
|
||
}
|
||
|
||
metrics::counter!("db.rows", "table" => "exec_trace").increment(self.exec_count);
|
||
metrics::counter!("db.rows", "table" => "import_calls").increment(self.import_count);
|
||
metrics::counter!("db.rows", "table" => "branch_trace").increment(self.branch_count);
|
||
tracing::info!(
|
||
instructions = self.exec_count,
|
||
imports = self.import_count,
|
||
branches = self.branch_count,
|
||
"trace totals"
|
||
);
|
||
Ok(())
|
||
}
|
||
}
|
||
|
||
/// Backwards-compatible wrapper that writes the full base + disasm layers.
|
||
pub fn write_db(
|
||
path: &Path,
|
||
pe: &[u8],
|
||
info: &DisasmInfo,
|
||
func_analysis: &FuncAnalysis,
|
||
labels: &HashMap<u32, String>,
|
||
_import_map: &HashMap<u32, String>,
|
||
xrefs: &XrefMap,
|
||
) -> anyhow::Result<()> {
|
||
let mut w = DbWriter::open_fresh(path)?;
|
||
w.write_base(info)?;
|
||
w.write_disasm(pe, info, func_analysis, labels, xrefs)?;
|
||
Ok(())
|
||
}
|
||
|
||
// ── Helpers ────────────────────────────────────────────────────────────────
|
||
|
||
fn insert_metadata(conn: &Connection, info: &DisasmInfo) -> anyhow::Result<()> {
|
||
let mut stmt = conn.prepare("INSERT OR REPLACE INTO metadata (key, value) VALUES (?, ?)")?;
|
||
let mut put = |k: &str, v: String| -> anyhow::Result<()> {
|
||
stmt.execute(params![k, v])?;
|
||
Ok(())
|
||
};
|
||
|
||
put("image_base", format!("0x{:08X}", info.image_base))?;
|
||
put("entry_point", format!("0x{:08X}", info.entry_point))?;
|
||
if let Some(name) = info.original_pe_name {
|
||
put("original_pe_name", name.to_string())?;
|
||
}
|
||
if let Some(title_id) = info.title_id {
|
||
put("title_id", format!("0x{:08X}", title_id))?;
|
||
}
|
||
if let Some(media_id) = info.media_id {
|
||
put("media_id", format!("0x{:08X}", media_id))?;
|
||
}
|
||
|
||
// Section geometry is useful enough on its own to be worth denormalising:
|
||
// a query that just wants "how big is the code" should not have to join.
|
||
let code_bytes: u64 = info
|
||
.sections
|
||
.iter()
|
||
.filter(|s| s.is_code())
|
||
.map(|s| s.virtual_size as u64)
|
||
.sum();
|
||
put("section_count", info.sections.len().to_string())?;
|
||
put("code_bytes", code_bytes.to_string())?;
|
||
|
||
let Some(header) = info.xex_header else {
|
||
return Ok(());
|
||
};
|
||
|
||
put("xex_module_flags", format!("0x{:08X}", header.module_flags))?;
|
||
put(
|
||
"xex_module_flags_decoded",
|
||
decode_module_flags(header.module_flags),
|
||
)?;
|
||
put("xex_header_count", header.header_count.to_string())?;
|
||
|
||
if let Some(sec) = &header.security_info {
|
||
put("image_size", format!("0x{:08X}", sec.image_size))?;
|
||
put("load_address", format!("0x{:08X}", sec.load_address))?;
|
||
put("image_flags", format!("0x{:08X}", sec.image_flags))?;
|
||
put(
|
||
"page_descriptor_count",
|
||
sec.page_descriptors.len().to_string(),
|
||
)?;
|
||
if sec.export_table_address != 0 {
|
||
put(
|
||
"export_table_address",
|
||
format!("0x{:08X}", sec.export_table_address),
|
||
)?;
|
||
}
|
||
}
|
||
|
||
if let Some(ff) = &header.file_format_info {
|
||
put(
|
||
"encryption_type",
|
||
match ff.encryption_type {
|
||
0 => "none".into(),
|
||
1 => "normal (AES-128-CBC)".into(),
|
||
n => format!("unknown ({n})"),
|
||
},
|
||
)?;
|
||
put(
|
||
"compression_type",
|
||
match ff.compression_type {
|
||
0 => "none".into(),
|
||
1 => "basic (raw + zero-fill blocks)".into(),
|
||
2 => "normal (LZX)".into(),
|
||
n => format!("unknown ({n})"),
|
||
},
|
||
)?;
|
||
if ff.compression_type == 1 {
|
||
put("basic_block_count", ff.basic_blocks.len().to_string())?;
|
||
}
|
||
if ff.compression_type == 2 {
|
||
put(
|
||
"lzx_window_size",
|
||
format!("0x{:08X}", ff.normal_window_size),
|
||
)?;
|
||
}
|
||
}
|
||
|
||
if let Some(exec) = &header.execution_info {
|
||
put("disc_number", exec.disc_number.to_string())?;
|
||
put("disc_count", exec.disc_count.to_string())?;
|
||
}
|
||
|
||
// Import libraries carry the SDK version each module was linked against —
|
||
// the single most useful "what toolchain built this" signal in the header.
|
||
put(
|
||
"import_library_count",
|
||
header.import_libraries.len().to_string(),
|
||
)?;
|
||
for lib in &header.import_libraries {
|
||
put(
|
||
&format!("import_lib.{}.version_min", lib.name),
|
||
format_xex_version(lib.version_min),
|
||
)?;
|
||
put(
|
||
&format!("import_lib.{}.version_cur", lib.name),
|
||
format_xex_version(lib.version_cur),
|
||
)?;
|
||
put(
|
||
&format!("import_lib.{}.imports", lib.name),
|
||
lib.imports.len().to_string(),
|
||
)?;
|
||
}
|
||
|
||
// Any optional header we do not model explicitly is still recorded by key,
|
||
// so nothing in the XEX is silently dropped.
|
||
for oh in &header.optional_headers {
|
||
put(
|
||
&format!("xex_optional_header.0x{:08X}", oh.key),
|
||
format!("0x{:08X}", oh.value),
|
||
)?;
|
||
}
|
||
|
||
Ok(())
|
||
}
|
||
|
||
/// Render a XEX version word (`major.minor.build.qfe`, 4/4/16/8 bits).
|
||
fn format_xex_version(v: u32) -> String {
|
||
let major = (v >> 28) & 0xF;
|
||
let minor = (v >> 24) & 0xF;
|
||
let build = (v >> 8) & 0xFFFF;
|
||
let qfe = v & 0xFF;
|
||
format!("{major}.{minor}.{build}.{qfe}")
|
||
}
|
||
|
||
/// Human-readable form of the XEX2 module flags bit field.
|
||
fn decode_module_flags(flags: u32) -> String {
|
||
const NAMES: &[(u32, &str)] = &[
|
||
(0x0000_0001, "title_module"),
|
||
(0x0000_0002, "exports_to_title"),
|
||
(0x0000_0004, "system_debugger"),
|
||
(0x0000_0008, "dll_module"),
|
||
(0x0000_0010, "module_patch"),
|
||
(0x0000_0020, "patch_full"),
|
||
(0x0000_0040, "patch_delta"),
|
||
(0x0000_0080, "user_mode"),
|
||
];
|
||
let set: Vec<&str> = NAMES
|
||
.iter()
|
||
.filter(|&&(b, _)| flags & b != 0)
|
||
.map(|&(_, n)| n)
|
||
.collect();
|
||
if set.is_empty() {
|
||
"none".to_string()
|
||
} else {
|
||
set.join("|")
|
||
}
|
||
}
|
||
|
||
fn insert_sections(
|
||
conn: &Connection,
|
||
sections: &[sylpheed_xex::pe::PeSection],
|
||
) -> anyhow::Result<()> {
|
||
let mut stmt = conn.prepare(
|
||
"INSERT INTO sections (name, virtual_address, virtual_size, raw_offset, raw_size, flags, is_code)
|
||
VALUES (?, ?, ?, ?, ?, ?, ?)"
|
||
)?;
|
||
for s in sections {
|
||
stmt.execute(params![
|
||
s.name,
|
||
s.virtual_address as i64,
|
||
s.virtual_size as i64,
|
||
s.raw_offset as i64,
|
||
s.raw_size as i64,
|
||
s.flags as i64,
|
||
s.is_code(),
|
||
])?;
|
||
}
|
||
Ok(())
|
||
}
|
||
|
||
fn insert_imports(conn: &Connection, info: &DisasmInfo) -> anyhow::Result<()> {
|
||
let mut stmt = conn.prepare(
|
||
"INSERT INTO imports (library, ordinal, name, record_type, address)
|
||
VALUES (?, ?, ?, ?, ?)",
|
||
)?;
|
||
// `address` is UNIQUE so that `instructions.import_address` can key off
|
||
// it. Two entries sharing an address would be a malformed import table —
|
||
// one slot cannot hold two imports — but a duplicate must not abort the
|
||
// whole database build, so keep the first and say what was dropped.
|
||
let mut seen: HashSet<u32> = HashSet::new();
|
||
let mut duplicates = 0usize;
|
||
for lib in info.import_libraries {
|
||
for imp in &lib.imports {
|
||
if !seen.insert(imp.address) {
|
||
duplicates += 1;
|
||
tracing::warn!(
|
||
library = %lib.name,
|
||
ordinal = imp.ordinal,
|
||
address = format_args!("{:#010x}", imp.address),
|
||
"duplicate import address; keeping the first entry",
|
||
);
|
||
continue;
|
||
}
|
||
let resolved = crate::resolve_ordinal(&lib.name, imp.ordinal);
|
||
stmt.execute(params![
|
||
lib.name,
|
||
imp.ordinal as i64,
|
||
resolved,
|
||
imp.record_type as i64,
|
||
imp.address as i64,
|
||
])?;
|
||
}
|
||
}
|
||
if duplicates > 0 {
|
||
tracing::warn!(duplicates, "import table has entries sharing an address");
|
||
}
|
||
Ok(())
|
||
}
|
||
|
||
fn insert_functions(
|
||
conn: &Connection,
|
||
func_analysis: &FuncAnalysis,
|
||
labels: &HashMap<u32, String>,
|
||
) -> anyhow::Result<()> {
|
||
let mut appender = conn.appender("functions")?;
|
||
for (&addr, fi) in &func_analysis.functions {
|
||
let name = labels
|
||
.get(&addr)
|
||
.cloned()
|
||
.unwrap_or_else(|| format!("sub_{addr:08X}"));
|
||
appender.append_row(params![
|
||
addr as i64,
|
||
name,
|
||
fi.end as i64,
|
||
fi.frame_size as i64,
|
||
fi.saved_gprs as i64,
|
||
fi.is_leaf,
|
||
fi.is_saverestore,
|
||
fi.pdata_validated,
|
||
fi.pdata_length.map(|n| n as i64),
|
||
fi.pdata_prolog_length.map(|n| n as i64),
|
||
fi.has_eh,
|
||
])?;
|
||
}
|
||
appender.flush()?;
|
||
Ok(())
|
||
}
|
||
|
||
fn insert_vtables(
|
||
conn: &Connection,
|
||
vtables: &[crate::vtables::Vtable],
|
||
_pe: &[u8],
|
||
_image_base: u32,
|
||
) -> anyhow::Result<()> {
|
||
if vtables.is_empty() {
|
||
return Ok(());
|
||
}
|
||
let mut stmt = conn.prepare(
|
||
"INSERT INTO vtables
|
||
(address, length, col_address, class_name, rtti_present, base_classes_json)
|
||
VALUES (?, ?, ?, ?, ?, ?)
|
||
ON CONFLICT DO NOTHING",
|
||
)?;
|
||
let mut count = 0u64;
|
||
for v in vtables {
|
||
stmt.execute(params![
|
||
v.address as i64,
|
||
v.length as i64,
|
||
v.col_address.map(|a| a as i64),
|
||
v.class_name.as_str(),
|
||
v.rtti_present,
|
||
v.base_classes_json.as_deref(),
|
||
])?;
|
||
count += 1;
|
||
}
|
||
metrics::counter!("db.rows", "table" => "vtables").increment(count);
|
||
tracing::info!(rows = count, table = "vtables", "bulk insert complete");
|
||
Ok(())
|
||
}
|
||
|
||
fn insert_methods_and_classes(
|
||
conn: &Connection,
|
||
vtables: &[crate::vtables::Vtable],
|
||
labels: &HashMap<u32, String>,
|
||
) -> anyhow::Result<()> {
|
||
if vtables.is_empty() {
|
||
return Ok(());
|
||
}
|
||
|
||
// methods rows — keyed by (vtable_address, slot), which `methods_table`
|
||
// emits at most once each.
|
||
let methods = crate::vtables::methods_table(vtables, labels);
|
||
if !methods.is_empty() {
|
||
let mut appender = conn.appender("methods")?;
|
||
for (vt_addr, slot, fn_addr, mangled, demangled) in &methods {
|
||
appender.append_row(params![
|
||
*vt_addr as i64,
|
||
*slot as i64,
|
||
*fn_addr as i64,
|
||
mangled.as_deref(),
|
||
demangled.as_deref(),
|
||
])?;
|
||
}
|
||
appender.flush()?;
|
||
metrics::counter!("db.rows", "table" => "methods").increment(methods.len() as u64);
|
||
tracing::info!(
|
||
rows = methods.len(),
|
||
table = "methods",
|
||
"bulk insert complete"
|
||
);
|
||
}
|
||
|
||
// classes rows (deduped by class_name, first-detected wins)
|
||
let classes = crate::vtables::classes_table(vtables);
|
||
if !classes.is_empty() {
|
||
let mut stmt = conn.prepare(
|
||
"INSERT INTO classes
|
||
(name, vtable_address, rtti_present, base_classes_json)
|
||
VALUES (?, ?, ?, ?)
|
||
ON CONFLICT DO NOTHING",
|
||
)?;
|
||
for (name, vt_addr, rtti, bases) in &classes {
|
||
stmt.execute(params![
|
||
name.as_str(),
|
||
*vt_addr as i64,
|
||
*rtti,
|
||
bases.as_deref(),
|
||
])?;
|
||
}
|
||
metrics::counter!("db.rows", "table" => "classes").increment(classes.len() as u64);
|
||
tracing::info!(
|
||
rows = classes.len(),
|
||
table = "classes",
|
||
"bulk insert complete"
|
||
);
|
||
}
|
||
|
||
Ok(())
|
||
}
|
||
|
||
fn insert_strings(
|
||
conn: &Connection,
|
||
strings: &[crate::strings::DetectedString],
|
||
) -> anyhow::Result<()> {
|
||
if strings.is_empty() {
|
||
return Ok(());
|
||
}
|
||
// The ascii / shift_jis / utf8 scans all run over the same bytes, so two
|
||
// of them can report a string at the same address. `address` is the
|
||
// primary key and the Appender cannot absorb that the way
|
||
// `ON CONFLICT DO NOTHING` did — keep the first detection per address.
|
||
let mut seen: std::collections::HashSet<u32> = std::collections::HashSet::new();
|
||
let mut appender = conn.appender("strings")?;
|
||
let mut count = 0u64;
|
||
for s in strings {
|
||
if !seen.insert(s.address) {
|
||
continue;
|
||
}
|
||
appender.append_row(params![
|
||
s.address as i64,
|
||
s.encoding,
|
||
s.length as i64,
|
||
s.content.as_str(),
|
||
s.section.as_str(),
|
||
])?;
|
||
count += 1;
|
||
}
|
||
appender.flush()?;
|
||
metrics::counter!("db.rows", "table" => "strings").increment(count);
|
||
tracing::info!(rows = count, table = "strings", "bulk insert complete");
|
||
Ok(())
|
||
}
|
||
|
||
fn insert_eh_records(
|
||
conn: &Connection,
|
||
records: &[crate::eh_scope::EhFuncInfo],
|
||
) -> anyhow::Result<()> {
|
||
if records.is_empty() {
|
||
return Ok(());
|
||
}
|
||
let mut stmt_fi = conn.prepare(
|
||
"INSERT INTO eh_funcinfo
|
||
(address, magic, max_state, p_unwind_map, n_try_blocks,
|
||
p_try_block_map, n_ip_map_entries, p_ip_to_state_map,
|
||
p_es_type_list, eh_flags)
|
||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||
ON CONFLICT DO NOTHING",
|
||
)?;
|
||
let mut n_fi = 0u64;
|
||
let mut kept: Vec<&crate::eh_scope::EhFuncInfo> = Vec::with_capacity(records.len());
|
||
for r in records {
|
||
let inserted = stmt_fi.execute(params![
|
||
r.address as i64,
|
||
r.magic as i64,
|
||
r.max_state as i64,
|
||
r.p_unwind_map as i64,
|
||
r.n_try_blocks as i64,
|
||
r.p_try_block_map as i64,
|
||
r.n_ip_map_entries as i64,
|
||
r.p_ip_to_state_map as i64,
|
||
r.p_es_type_list.map(|p| p as i64),
|
||
r.eh_flags.map(|f| f as i64),
|
||
])?;
|
||
if inserted > 0 {
|
||
n_fi += 1;
|
||
kept.push(r);
|
||
}
|
||
}
|
||
drop(stmt_fi);
|
||
|
||
// The child rows are keyed by (funcinfo_address, index) and each parent
|
||
// survived the ON CONFLICT above, so these cannot collide.
|
||
let mut n_unwind = 0u64;
|
||
{
|
||
let mut appender = conn.appender("eh_unwind_map")?;
|
||
for r in &kept {
|
||
for (i, e) in r.unwind_map.iter().enumerate() {
|
||
appender.append_row(params![
|
||
r.address as i64,
|
||
i as i64,
|
||
e.to_state as i64,
|
||
e.action_pc as i64,
|
||
])?;
|
||
n_unwind += 1;
|
||
}
|
||
}
|
||
appender.flush()?;
|
||
}
|
||
|
||
let mut n_try = 0u64;
|
||
{
|
||
let mut appender = conn.appender("eh_try_blocks")?;
|
||
for r in &kept {
|
||
for (i, t) in r.try_blocks.iter().enumerate() {
|
||
appender.append_row(params![
|
||
r.address as i64,
|
||
i as i64,
|
||
t.try_low as i64,
|
||
t.try_high as i64,
|
||
t.catch_high as i64,
|
||
t.n_catches as i64,
|
||
t.p_handler_array as i64,
|
||
])?;
|
||
n_try += 1;
|
||
}
|
||
}
|
||
appender.flush()?;
|
||
}
|
||
|
||
metrics::counter!("db.rows", "table" => "eh_funcinfo").increment(n_fi);
|
||
metrics::counter!("db.rows", "table" => "eh_unwind_map").increment(n_unwind);
|
||
metrics::counter!("db.rows", "table" => "eh_try_blocks").increment(n_try);
|
||
tracing::info!(
|
||
funcinfo = n_fi,
|
||
unwind = n_unwind,
|
||
try_blocks = n_try,
|
||
"EH scope-table insert complete"
|
||
);
|
||
Ok(())
|
||
}
|
||
|
||
fn insert_typed_ind_dispatch(
|
||
conn: &Connection,
|
||
t: &crate::ind_dispatch_typed::TypedIndirectResult,
|
||
) -> anyhow::Result<()> {
|
||
if !t.dispatches.is_empty() {
|
||
let mut stmt_site = conn.prepare(
|
||
"INSERT INTO indirect_dispatch_sites
|
||
(dispatch_pc, vptr_offset, slot, candidate_count, truncated)
|
||
VALUES (?, ?, ?, ?, ?) ON CONFLICT DO NOTHING",
|
||
)?;
|
||
let mut n_sites = 0u64;
|
||
for d in &t.dispatches {
|
||
stmt_site.execute(params![
|
||
d.dispatch_pc as i64,
|
||
d.vptr_offset as i64,
|
||
d.slot as i64,
|
||
d.total_candidates as i64,
|
||
d.truncated,
|
||
])?;
|
||
n_sites += 1;
|
||
}
|
||
drop(stmt_site);
|
||
|
||
// `indirect_dispatch_candidates` used to be by far the largest table
|
||
// this writer produced — 1.8M rows before unresolved sites stopped
|
||
// materialising their cross product (see
|
||
// `ind_dispatch_typed::analyze`). It still goes through the Appender:
|
||
// the ceiling is configurable and a caller that raises it gets the
|
||
// volume back.
|
||
//
|
||
// The Appender bypasses the SQL layer, which means `ON CONFLICT DO
|
||
// NOTHING` is not available to absorb duplicates and a repeated
|
||
// `(dispatch_pc, vtable_address)` would violate the primary key at
|
||
// flush. Dedupe up front instead.
|
||
let mut appender = conn.appender("indirect_dispatch_candidates")?;
|
||
let mut seen: std::collections::HashSet<(u32, u32)> = std::collections::HashSet::new();
|
||
let mut n_cand = 0u64;
|
||
for d in &t.dispatches {
|
||
for (vt, m) in d.candidate_vtables.iter().zip(d.method_pcs.iter()) {
|
||
if seen.insert((d.dispatch_pc, *vt)) {
|
||
appender.append_row(params![d.dispatch_pc as i64, *vt as i64, *m as i64,])?;
|
||
n_cand += 1;
|
||
}
|
||
}
|
||
}
|
||
appender.flush()?;
|
||
|
||
metrics::counter!("db.rows", "table" => "indirect_dispatch_sites").increment(n_sites);
|
||
metrics::counter!("db.rows", "table" => "indirect_dispatch_candidates").increment(n_cand);
|
||
tracing::info!(
|
||
sites = n_sites,
|
||
candidates = n_cand,
|
||
"typed indirect-dispatch insert complete"
|
||
);
|
||
}
|
||
if !t.vptr_writes.is_empty() {
|
||
let mut stmt = conn.prepare(
|
||
"INSERT INTO vptr_writes
|
||
(writer_pc, vtable_address, vptr_offset, writer_function)
|
||
VALUES (?, ?, ?, ?) ON CONFLICT DO NOTHING",
|
||
)?;
|
||
let mut n = 0u64;
|
||
for w in &t.vptr_writes {
|
||
stmt.execute(params![
|
||
w.writer_pc as i64,
|
||
w.vtable_addr as i64,
|
||
w.vptr_offset as i64,
|
||
w.writer_function as i64,
|
||
])?;
|
||
n += 1;
|
||
}
|
||
metrics::counter!("db.rows", "table" => "vptr_writes").increment(n);
|
||
tracing::info!(rows = n, "vptr_writes insert complete");
|
||
}
|
||
Ok(())
|
||
}
|
||
|
||
/// Write the XDBF package tables. Achievement names are resolved through the
|
||
/// package's own default-language string table (`XSTC`), falling back to
|
||
/// English and then to whatever table exists, so the text columns are populated
|
||
/// even for a title that ships no `XSTC`.
|
||
fn insert_xdbf(conn: &Connection, xdbf: Option<&crate::xdbf::Xdbf>) -> anyhow::Result<()> {
|
||
let Some(x) = xdbf else { return Ok(()) };
|
||
|
||
let mut stmt = conn.prepare(
|
||
"INSERT INTO xdbf_entries (namespace, namespace_name, id, body_offset, size, magic)
|
||
VALUES (?, ?, ?, ?, ?, ?) ON CONFLICT DO NOTHING",
|
||
)?;
|
||
for e in &x.entries {
|
||
let ns_name = match e.namespace {
|
||
1 => "metadata",
|
||
2 => "image",
|
||
3 => "string_table",
|
||
_ => "unknown",
|
||
};
|
||
stmt.execute(params![
|
||
e.namespace as i64,
|
||
ns_name,
|
||
e.id as i64,
|
||
e.offset as i64,
|
||
e.size as i64,
|
||
e.magic.as_deref(),
|
||
])?;
|
||
}
|
||
drop(stmt);
|
||
|
||
let mut stmt = conn.prepare(
|
||
"INSERT INTO xdbf_strings (language, language_name, string_id, value)
|
||
VALUES (?, ?, ?, ?) ON CONFLICT DO NOTHING",
|
||
)?;
|
||
let mut n_strings = 0u64;
|
||
for t in &x.string_tables {
|
||
let name = crate::xdbf::language_name(t.language);
|
||
for (id, v) in &t.strings {
|
||
stmt.execute(params![t.language as i64, name, *id as i64, v.as_str()])?;
|
||
n_strings += 1;
|
||
}
|
||
}
|
||
drop(stmt);
|
||
|
||
// Pick the table used to resolve achievement text.
|
||
let preferred = x.default_language.unwrap_or(1);
|
||
let lookup = x
|
||
.string_tables
|
||
.iter()
|
||
.find(|t| t.language == preferred)
|
||
.or_else(|| x.string_tables.iter().find(|t| t.language == 1))
|
||
.or_else(|| x.string_tables.first());
|
||
let text = |id: u16| -> Option<String> {
|
||
lookup?
|
||
.strings
|
||
.iter()
|
||
.find(|(sid, _)| *sid == id)
|
||
.map(|(_, s)| s.clone())
|
||
};
|
||
|
||
// Images before achievements: `xdbf_achievements.image_id` is a foreign
|
||
// key onto `xdbf_images.id`, so the icon rows must already exist.
|
||
let mut stmt = conn.prepare(
|
||
"INSERT INTO xdbf_images (id, is_title_icon, body_offset, size, format)
|
||
VALUES (?, ?, ?, ?, ?) ON CONFLICT DO NOTHING",
|
||
)?;
|
||
for i in &x.images {
|
||
stmt.execute(params![
|
||
i.id as i64,
|
||
i.id == crate::xdbf::ID_TITLE,
|
||
i.offset as i64,
|
||
i.size as i64,
|
||
i.format,
|
||
])?;
|
||
}
|
||
drop(stmt);
|
||
|
||
let mut stmt = conn.prepare(
|
||
"INSERT INTO xdbf_achievements
|
||
(id, name, unlocked_desc, locked_desc, label_id, description_id,
|
||
unachieved_id, image_id, gamerscore, flags)
|
||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT DO NOTHING",
|
||
)?;
|
||
for a in &x.achievements {
|
||
stmt.execute(params![
|
||
a.id as i64,
|
||
text(a.label_id),
|
||
text(a.description_id),
|
||
text(a.unachieved_id),
|
||
a.label_id as i64,
|
||
a.description_id as i64,
|
||
a.unachieved_id as i64,
|
||
a.image_id as i64,
|
||
a.gamerscore as i64,
|
||
a.flags as i64,
|
||
])?;
|
||
}
|
||
drop(stmt);
|
||
|
||
let mut meta = conn.prepare("INSERT OR REPLACE INTO metadata (key, value) VALUES (?, ?)")?;
|
||
meta.execute(params!["xdbf.entry_count", x.entries.len().to_string()])?;
|
||
if let Some(l) = x.default_language {
|
||
meta.execute(params![
|
||
"xdbf.default_language",
|
||
crate::xdbf::language_name(l)
|
||
])?;
|
||
}
|
||
if let Some(t) = x.title {
|
||
meta.execute(params!["xdbf.title_id", format!("{:#010X}", t.title_id)])?;
|
||
meta.execute(params![
|
||
"xdbf.title_version",
|
||
format!("{}.{}.{}.{}", t.major, t.minor, t.build, t.revision)
|
||
])?;
|
||
}
|
||
// The title's own name lives at the well-known string id 0x8000, in the
|
||
// package's default language.
|
||
if let Some(name) = text(crate::xdbf::ID_TITLE as u16) {
|
||
meta.execute(params!["xdbf.title_name", name])?;
|
||
}
|
||
|
||
metrics::counter!("db.rows", "table" => "xdbf_strings").increment(n_strings);
|
||
tracing::info!(
|
||
entries = x.entries.len(),
|
||
achievements = x.achievements.len(),
|
||
strings = n_strings,
|
||
images = x.images.len(),
|
||
"XDBF tables written",
|
||
);
|
||
Ok(())
|
||
}
|
||
|
||
fn insert_funcptr_arrays(
|
||
conn: &Connection,
|
||
arrays: &[crate::funcptr_arrays::FuncPtrArray],
|
||
) -> anyhow::Result<()> {
|
||
if arrays.is_empty() {
|
||
return Ok(());
|
||
}
|
||
// Parents first (few, and `ON CONFLICT` decides which survive), then the
|
||
// entry rows in one Appender pass — only entries of a parent that was
|
||
// actually inserted.
|
||
let mut stmt_arr = conn.prepare(
|
||
"INSERT INTO function_pointer_arrays (address, length, kind) VALUES (?, ?, ?)
|
||
ON CONFLICT DO NOTHING",
|
||
)?;
|
||
let mut n_arr = 0u64;
|
||
let mut kept: Vec<&crate::funcptr_arrays::FuncPtrArray> = Vec::with_capacity(arrays.len());
|
||
for a in arrays {
|
||
let inserted = stmt_arr.execute(params![a.address as i64, a.length as i64, a.kind,])?;
|
||
if inserted > 0 {
|
||
n_arr += 1;
|
||
kept.push(a);
|
||
}
|
||
}
|
||
drop(stmt_arr);
|
||
|
||
let mut appender = conn.appender("function_pointer_array_entries")?;
|
||
let mut n_ent = 0u64;
|
||
for a in kept {
|
||
for (i, &fn_va) in a.entries.iter().enumerate() {
|
||
appender.append_row(params![a.address as i64, i as i64, fn_va as i64])?;
|
||
n_ent += 1;
|
||
}
|
||
}
|
||
appender.flush()?;
|
||
|
||
metrics::counter!("db.rows", "table" => "function_pointer_arrays").increment(n_arr);
|
||
metrics::counter!("db.rows", "table" => "function_pointer_array_entries").increment(n_ent);
|
||
tracing::info!(
|
||
arrays = n_arr,
|
||
entries = n_ent,
|
||
"function-pointer arrays insert complete"
|
||
);
|
||
Ok(())
|
||
}
|
||
|
||
fn insert_demangled_from_labels(
|
||
conn: &Connection,
|
||
labels: &HashMap<u32, String>,
|
||
import_libraries: &[sylpheed_xex::header::ImportLibrary],
|
||
) -> anyhow::Result<()> {
|
||
let mut stmt = conn.prepare(
|
||
"INSERT INTO demangled_names
|
||
(address, mangled, raw_demangled, namespace_path, class_name,
|
||
method_name, params_signature)
|
||
VALUES (?, ?, ?, ?, ?, ?, ?)",
|
||
)?;
|
||
|
||
let mut count = 0u64;
|
||
|
||
for (&addr, name) in labels {
|
||
// The label table holds raw symbol names (`?...@...`). Imports come
|
||
// wrapped as `__imp_<lib>_<sym>`; strip the `__imp_<lib>_` prefix to
|
||
// recover any mangled inner name (rare for kernel imports but
|
||
// defensive). For now, skip imports entirely — they're handled below
|
||
// via `import_libraries`.
|
||
if name.starts_with("__imp_") {
|
||
continue;
|
||
}
|
||
if let Some(d) = crate::demangle::demangle(name) {
|
||
stmt.execute(params![
|
||
addr as i64,
|
||
d.mangled,
|
||
d.raw_demangled,
|
||
d.namespace_path,
|
||
d.class_name,
|
||
d.method_name,
|
||
d.params_signature,
|
||
])?;
|
||
count += 1;
|
||
}
|
||
}
|
||
|
||
// Defensive: also demangle any import name that happens to be mangled.
|
||
for lib in import_libraries {
|
||
for imp in &lib.imports {
|
||
let resolved = crate::resolve_ordinal(&lib.name, imp.ordinal);
|
||
if let Some(name) = resolved
|
||
&& let Some(d) = crate::demangle::demangle(name)
|
||
{
|
||
stmt.execute(params![
|
||
imp.address as i64,
|
||
d.mangled,
|
||
d.raw_demangled,
|
||
d.namespace_path,
|
||
d.class_name,
|
||
d.method_name,
|
||
d.params_signature,
|
||
])?;
|
||
count += 1;
|
||
}
|
||
}
|
||
}
|
||
|
||
metrics::counter!("db.rows", "table" => "demangled_names").increment(count);
|
||
tracing::info!(
|
||
rows = count,
|
||
table = "demangled_names",
|
||
"demangler complete"
|
||
);
|
||
Ok(())
|
||
}
|
||
|
||
fn insert_pdata_entries(
|
||
conn: &Connection,
|
||
entries: &[sylpheed_xex::pdata::PdataEntry],
|
||
) -> anyhow::Result<()> {
|
||
if entries.is_empty() {
|
||
return Ok(());
|
||
}
|
||
// `parse_pdata` already guarantees strictly ascending, unique
|
||
// `begin_address` values, so the primary key cannot collide.
|
||
let mut appender = conn.appender("pdata_entries")?;
|
||
for e in entries {
|
||
appender.append_row(params![
|
||
e.begin_address as i64,
|
||
e.end_address() as i64,
|
||
e.function_length as i64,
|
||
e.prolog_length as i64,
|
||
e.flags as i64,
|
||
])?;
|
||
}
|
||
appender.flush()?;
|
||
Ok(())
|
||
}
|
||
|
||
fn insert_labels(conn: &Connection, labels: &HashMap<u32, String>) -> anyhow::Result<()> {
|
||
// `labels` is keyed by address, so it cannot contain a duplicate primary
|
||
// key — the Appender is safe here without a dedupe pass.
|
||
let mut appender = conn.appender("labels")?;
|
||
for (&addr, name) in labels {
|
||
let kind = if name.starts_with("sub_") || name == "entry_point" {
|
||
"function"
|
||
} else if name.starts_with("__imp_") {
|
||
"import"
|
||
} else if name.starts_with("__savegprlr_") || name.starts_with("__restgprlr_") {
|
||
"saverestore"
|
||
} else if name.starts_with("loc_") {
|
||
"local"
|
||
} else if name.starts_with("dat_") {
|
||
"data"
|
||
} else {
|
||
"other"
|
||
};
|
||
appender.append_row(params![addr as i64, name, kind])?;
|
||
}
|
||
appender.flush()?;
|
||
Ok(())
|
||
}
|
||
|
||
fn insert_instructions_streaming(
|
||
conn: &Connection,
|
||
pe: &[u8],
|
||
info: &DisasmInfo,
|
||
func_analysis: &FuncAnalysis,
|
||
labels: &HashMap<u32, String>,
|
||
data_words: &std::collections::BTreeSet<u32>,
|
||
import_sites: &crate::imports::ImportSites,
|
||
) -> anyhow::Result<()> {
|
||
let mut appender = conn.appender("instructions")?;
|
||
let mut total: u64 = 0;
|
||
|
||
for section in info.sections {
|
||
if !section.is_code() {
|
||
continue;
|
||
}
|
||
let va_start = info.image_base + section.virtual_address;
|
||
let va_end = info.image_base + section.virtual_address + section.virtual_size;
|
||
let items = crate::disasm::enrich_section(
|
||
pe,
|
||
info.image_base,
|
||
§ion.name,
|
||
va_start,
|
||
va_end,
|
||
func_analysis,
|
||
labels,
|
||
data_words,
|
||
import_sites,
|
||
);
|
||
total += crate::sinks::duckdb::append_instructions(&mut appender, items)?;
|
||
}
|
||
|
||
appender.flush()?;
|
||
metrics::counter!("db.rows", "table" => "instructions").increment(total);
|
||
tracing::info!(rows = total, table = "instructions", "bulk insert complete");
|
||
Ok(())
|
||
}
|
||
|
||
/// Record each import thunk's two loader-rewritten words in `data_in_code`.
|
||
///
|
||
/// `data_in_code` already lists recovered jump tables; import records are the
|
||
/// other kind of non-instruction word sitting in a code section, so a consumer
|
||
/// asking "which bytes of `.text` are not code?" gets a complete answer from
|
||
/// one table instead of having to know about thunk layout.
|
||
fn insert_import_record_extents(
|
||
conn: &Connection,
|
||
import_sites: &crate::imports::ImportSites,
|
||
) -> anyhow::Result<()> {
|
||
let mut heads: Vec<u32> = import_sites.thunk_heads().collect();
|
||
heads.sort_unstable();
|
||
let mut stmt =
|
||
conn.prepare("INSERT INTO data_in_code (address, length, kind) VALUES (?, ?, ?)")?;
|
||
for head in &heads {
|
||
// The two records are contiguous at the head of the thunk; the two
|
||
// instructions that follow them are real code.
|
||
stmt.execute(params![*head as i64, 8i64, "import_record"])?;
|
||
}
|
||
metrics::counter!("db.rows", "table" => "data_in_code").increment(heads.len() as u64);
|
||
tracing::info!(
|
||
rows = heads.len(),
|
||
kind = "import_record",
|
||
"data_in_code write complete"
|
||
);
|
||
Ok(())
|
||
}
|
||
|
||
/// Write the M12 jump-table tables plus the data-in-code extent list.
|
||
fn insert_jump_tables(
|
||
conn: &Connection,
|
||
tables: &[crate::jumptables::JumpTable],
|
||
) -> anyhow::Result<()> {
|
||
let mut t = conn.prepare(
|
||
"INSERT INTO jump_tables
|
||
(bctr_pc, function, table_address, entry_count, table_slots,
|
||
index_map_address, index_map_count, case_bound, kind)
|
||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||
)?;
|
||
for jt in tables {
|
||
t.execute(params![
|
||
jt.bctr_pc as i64,
|
||
jt.function.map(|f| f as i64),
|
||
jt.table_address as i64,
|
||
jt.entry_count as i64,
|
||
jt.table_slots as i64,
|
||
jt.index_map_address.map(|a| a as i64),
|
||
jt.index_map_count.map(|n| n as i64),
|
||
jt.bound.map(|n| n as i64),
|
||
jt.kind,
|
||
])?;
|
||
}
|
||
drop(t);
|
||
|
||
// Keyed by (bctr_pc, case_index); each bctr_pc yields one table.
|
||
let mut e = conn.appender("jump_table_entries")?;
|
||
for jt in tables {
|
||
for (i, &target) in jt.targets.iter().enumerate() {
|
||
e.append_row(params![jt.bctr_pc as i64, i as i64, target as i64])?;
|
||
}
|
||
}
|
||
e.flush()?;
|
||
drop(e);
|
||
|
||
// `data_in_code` is keyed by address, so the per-table extents are merged
|
||
// first — two switches in the same function can share one table.
|
||
let mut kinds: HashMap<u32, &'static str> = HashMap::new();
|
||
for jt in tables {
|
||
kinds.insert(jt.table_address, "jump_table");
|
||
if let Some(a) = jt.index_map_address {
|
||
kinds.insert(a, "jump_index_map");
|
||
}
|
||
}
|
||
let mut d =
|
||
conn.prepare("INSERT INTO data_in_code (address, length, kind) VALUES (?, ?, ?)")?;
|
||
for (addr, len) in crate::jumptables::data_regions(tables) {
|
||
let kind = kinds.get(&addr).copied().unwrap_or("jump_table");
|
||
d.execute(params![addr as i64, len as i64, kind])?;
|
||
}
|
||
|
||
metrics::counter!("db.rows", "table" => "jump_tables").increment(tables.len() as u64);
|
||
tracing::info!(
|
||
rows = tables.len(),
|
||
entries = tables.iter().map(|t| t.targets.len()).sum::<usize>(),
|
||
table = "jump_tables",
|
||
"insert complete",
|
||
);
|
||
Ok(())
|
||
}
|
||
|
||
/// Write the M13 RTTI tables.
|
||
fn insert_rtti(conn: &Connection, rtti: &crate::rtti::RttiResult) -> anyhow::Result<()> {
|
||
let mut td = conn.prepare(
|
||
"INSERT INTO rtti_type_descriptors (address, mangled_name, demangled_name)
|
||
VALUES (?, ?, ?)",
|
||
)?;
|
||
// RTTI descriptors are the only mangled names a stripped retail binary
|
||
// still carries, so they are also the only thing `demangled_names` can be
|
||
// populated from — without this it stays empty on every shipped title.
|
||
let mut dn = conn.prepare(
|
||
"INSERT INTO demangled_names
|
||
(address, mangled, raw_demangled, namespace_path, class_name,
|
||
method_name, params_signature)
|
||
VALUES (?, ?, ?, ?, ?, ?, ?)",
|
||
)?;
|
||
for t in &rtti.type_descriptors {
|
||
td.execute(params![t.address as i64, t.mangled_name, t.demangled_name])?;
|
||
let (ns, cls) = match t.demangled_name.rfind("::") {
|
||
Some(i) => (Some(&t.demangled_name[..i]), &t.demangled_name[i + 2..]),
|
||
None => (None, t.demangled_name.as_str()),
|
||
};
|
||
dn.execute(params![
|
||
t.address as i64,
|
||
t.mangled_name,
|
||
t.demangled_name,
|
||
ns,
|
||
cls,
|
||
Option::<&str>::None,
|
||
Option::<&str>::None,
|
||
])?;
|
||
}
|
||
|
||
let mut col = conn.prepare(
|
||
"INSERT INTO rtti_locators
|
||
(address, subobject_offset, cd_offset, type_descriptor, class_hierarchy, vtable_address)
|
||
VALUES (?, ?, ?, ?, ?, ?)",
|
||
)?;
|
||
for c in &rtti.locators {
|
||
col.execute(params![
|
||
c.address as i64,
|
||
c.offset as i64,
|
||
c.cd_offset as i64,
|
||
c.type_descriptor as i64,
|
||
c.class_hierarchy as i64,
|
||
c.vtable_address.map(|v| v as i64),
|
||
])?;
|
||
}
|
||
|
||
let mut bc = conn.prepare(
|
||
"INSERT INTO rtti_base_classes
|
||
(class_hierarchy, base_index, type_descriptor, name, num_contained_bases,
|
||
mdisp, pdisp, vdisp, attributes)
|
||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||
)?;
|
||
for b in &rtti.base_classes {
|
||
bc.execute(params![
|
||
b.class_hierarchy as i64,
|
||
b.index as i64,
|
||
b.type_descriptor as i64,
|
||
b.name,
|
||
b.num_contained_bases as i64,
|
||
b.mdisp as i64,
|
||
b.pdisp as i64,
|
||
b.vdisp as i64,
|
||
b.attributes as i64,
|
||
])?;
|
||
}
|
||
|
||
tracing::info!(
|
||
type_descriptors = rtti.type_descriptors.len(),
|
||
locators = rtti.locators.len(),
|
||
base_classes = rtti.base_classes.len(),
|
||
"RTTI tables written",
|
||
);
|
||
Ok(())
|
||
}
|
||
|
||
fn insert_xrefs_streaming(
|
||
conn: &Connection,
|
||
xrefs: &XrefMap,
|
||
pe: &[u8],
|
||
image_base: u32,
|
||
func_analysis: &FuncAnalysis,
|
||
labels: &HashMap<u32, String>,
|
||
) -> anyhow::Result<()> {
|
||
let mut appender = conn.appender("xrefs")?;
|
||
let mut count: u64 = 0;
|
||
|
||
for (&target, refs) in xrefs {
|
||
let target_label = labels.get(&target).map(|s| s.as_str());
|
||
|
||
for xref in refs {
|
||
let kind = xref.kind.db_tag();
|
||
|
||
let instruction: Option<String> = {
|
||
let off = xref.source.wrapping_sub(image_base) as usize;
|
||
if off + 4 <= pe.len() {
|
||
let raw = u32::from_be_bytes([pe[off], pe[off + 1], pe[off + 2], pe[off + 3]]);
|
||
let d = sylpheed_ppc::decode(raw, xref.source);
|
||
let t = sylpheed_ppc::disasm::format(&d);
|
||
// Prefer the simplified mnemonic when present (matches what
|
||
// a human reading the .asm file sees for that line).
|
||
Some(t.ext_mnemonic.unwrap_or(t.mnemonic))
|
||
} else {
|
||
None
|
||
}
|
||
};
|
||
|
||
let source_func = func_analysis
|
||
.functions
|
||
.range(..=xref.source)
|
||
.next_back()
|
||
.map(|(&a, _)| a as i64);
|
||
|
||
let source_label = resolve_source_label(xref.source, func_analysis, labels);
|
||
|
||
let addr_mode = xref.addr_mode.map(|m| m.tag());
|
||
appender.append_row(params![
|
||
xref.source as i64,
|
||
target as i64,
|
||
kind,
|
||
addr_mode,
|
||
instruction.as_deref(),
|
||
source_func,
|
||
source_label.as_str(),
|
||
target_label,
|
||
])?;
|
||
|
||
count += 1;
|
||
}
|
||
}
|
||
|
||
appender.flush()?;
|
||
metrics::counter!("db.rows", "table" => "xrefs").increment(count);
|
||
tracing::info!(rows = count, table = "xrefs", "bulk insert complete");
|
||
Ok(())
|
||
}
|