Second coverage batch, data-driven from the histogram. Diff harness validates
every op against the interpreter (MISMATCHES=0 over 47.96M blocks).
Update-form load/stores (24 ops: lbzu..stdux, lfsu..stfdux and their x-forms):
same access as the base form but with EA = gpr[ra] + offset (ra used directly —
update forms are illegal with ra==0) and rA := EA written back (zero-extended)
after the access. New ea_d_update/ea_x_update/write_ea_back helpers; loads write
rD then rA (so rA wins if rd==ra, matching the interpreter). stwu especially is
the standard stack-frame push in every function prologue.
mfspr/mtspr for LR and CTR only (64-bit field copies): the ubiquitous mflr/mtlr
prologue-epilogue pair and mtctr. covered() gates on the compile-time SPR number
so the offset always resolves; XER (packs CA/OV/SO) and the modelled SPRs stay
uncovered. This is the biggest single jump — mflr/mtlr gate almost every
non-leaf function's prologue and epilogue blocks.
Coverage (single-block, diff mode): 80.6% -> 82.8% (update forms) -> 89.6%
(mfspr/mtspr). 7/7 jit unit tests; foreground-validated per the bg-SIGTERM
finding. Carry-setting shifts (sraw*/srad*), rotate-double (rldic*), and XER
mfspr/mtspr remain for later batches; VMX128 is the long pole.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Data-driven from XENIA_JIT_HIST: after register caching the JIT was 73% native;
the frequent uncovered opcodes are VMX128 (the long pole), mtspr/mfspr, update-
form loads/stores, and a batch of cheap pure-register integer ops. This does the
cheap integer batch (each unblocks any block that only lacked it, and — since a
covered block can now chain through where an uncovered op used to break the
chain — the gain compounds: more native execution AND longer superblocks).
Added to covered() + emit_op, all validated against the interpreter by the diff
harness (none are sync_sensitive, so fully covered):
- extsbx/extshx/extswx — sign-extend byte/half/word (extsb/extsh write the
i32-view zero-extended per the 32-bit ABI, CR0 i32; extsw sign-extends into
the full 64, CR0 i64).
- cntlzwx/cntlzdx — count leading zeros (Cranelift clz).
- negx (OE=0) — rD = 0 - rA (ineg), full 64-bit, CR0 on low 32.
- slwx/srwx — word shifts; explicit `sh<32 ? shift : 0` select
because the count is rB[58:63] (6 bits) and Cranelift's ishl/ushr mask to 5.
- sldx/srdx — doubleword shifts, `sh<64 ? shift : 0` (count 7 bits).
New emit_cr0_signed64 for the doubleword-result CR0 (a value with low32==0 but
high bits set is eq in the 32-bit view but not the 64-bit one). The carry-setting
shifts (sraw*/srad*, set XER-CA) and rotate-double (rldic*) are deferred to later
batches.
Validated (foreground per the bg-SIGTERM finding): 7/7 jit unit tests; diff
MISMATCHES=0 over 47.9M blocks; coverage 75.4% -> 80.6% (single-block, diff
mode). Perf (RUST_LOG=warn, -n 3e9 to amortize compiles): chain wall 57952ms ->
54748ms (-5.5% from coverage alone; run_block calls 110M -> 98M = more chaining),
chain -14.3% vs interp (63.9s) this session. -n 2e9 understates it (more blocks
compiled = more compile cost inside the STEP timer at under-amortized -n).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Guest GPRs were reloaded from PpcContext memory on every use and stored back
after every write, forcing memory traffic that held the JIT bodies at mere
interpreter parity. Now each GPR is loaded ONCE per block into a Cranelift SSA
value (kept in a host register by the register allocator); writes update an
in-EmitCtx cache (RegCache: [Option<Value>;32] + dirty bits) without storing;
dirty GPRs are flushed to memory only at the block boundary (cache_flush at the
end of emit_node_body).
Sound because nothing a block calls back into touches guest GPRs: the memory
trampolines take addr/val as args (I pass the cached SSA values), and the
FP-punt runs interpreter FP ops that touch fpr/fpscr/cr only. So the cache never
goes stale mid-block; flushing at the boundary makes gpr[] interpreter-identical
for the runner and the next block. FPR/CR still go through memory (the FP-punt
writes them, which would need flush-around-punt — deferred).
emit_op / ea_d / ea_x / gpr32 / store_gpr32z rewired to cache_read_gpr /
cache_write_gpr; cache_read_gpr copies the Option out before re-borrowing (a
RefCell double-borrow would panic).
Validated: 7/7 jit unit tests; diff MISMATCHES=0 over 47.9M blocks (foreground).
Perf (foreground, RUST_LOG=warn, -n 3e9 to amortize compile cost — -n<=1.5e9 is
compile-dominated and misleads): interp 95.8 MIPS -> jit-no-chain 100.2 MIPS
(bodies now BEAT interp, were parity) -> chain 100.4 MIPS / 57.95s = -15.6% wall
vs interp (was ~-6% at B2), dispatch 18.9%, 110M calls (1.95x merge). Same-batch
-n 2e9: interp 44.8s -> chain 41.0s = -8.6%. Register caching is what makes the
bodies fast enough that chaining's dispatch cut converts to real wall time
(B2's bodies had slowed to ~82 MIPS and cancelled the win).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extends B1's forward-linear superblock into a full same-page CFG: chains
through conditional bcx (BOTH directions) and loop back-edges (a successor
already in the chain jumps to its existing IR block), not just unconditional
bx + fall-through. This roughly doubles the block-merge ratio (run_block calls
145.7M interp -> 116.2M B1 (1.25x) -> 74.7M B2 (1.95x)) and halves the
run_superblock dispatch bucket (30.4% interp -> 26.2% B1 -> 16.2% B2).
Mechanism:
- enumerate_chain is now a two-pass CFG builder: BFS the reachable, chainable
(same-page / covered / non-thunk) blocks deduped by start PC (so a back-edge
becomes a loop, not a new node), capped at MAX_CHAIN_NODES=64; then resolve
each terminator to Succ edges (node indices or Exit). bx/fall -> One; bcx ->
Two{taken,fall} (each side Node or Exit); bclrx/bcctrx/uncovered -> Exit.
- emit_bcx extracted from emit_op's bcx arm: computes the taken predicate ONCE
(decrementing CTR at most once) and returns it; emit_node_body captures it so
the two-way boundary branches on the SAME value (recomputing would double the
CTR decrement). Single-block compile is unchanged (ignores the return).
- No Cranelift Variables/phi needed for the loop CFG: the budget/MMIO state
lives in memory (cycle_count, *mmio_count), so emit_stop reloads it and
compares against entry-sampled constants (cycle_entry/mmio_entry), which the
entry block dominates across back-edges. Budget = (cycle_count - cycle_entry)
>= remaining, correct across loop iterations; a native loop spins until the
budget is spent then hands back — same bound as the runner. MMIO check still
skipped for load/store-free blocks. seal_all_blocks handles the arbitrary CFG.
- Degenerate guard fixed: compile single-block only when the ENTRY has no
chainable successor (Succ::Exit) — a one-node self-loop is NOT degenerate.
Validation: 3 new unit tests (bcx always-taken chains to target; bcx not-taken
falls through; self-loop runs natively until budget then stops — all bit-equal
to the interpreter over the same PCs) -> 7/7 jit tests pass. Diff MISMATCHES=0
over 47.95M blocks (the emit_bcx refactor didn't perturb op bodies).
XENIA_JIT_CHAIN=1 movie plays: tid25 resumes (decode handoff / interleaving
preserved under the deeper chaining).
Perf finding (honest): despite ~2x merge and half the dispatch, wall is ~parity
with B1 (both ~-6% vs interp same-batch). step_block MIPS dropped (~82 vs B1
~102) because the deeper/bigger native superblocks cost more per boundary (B1's
compile-time-constant budget became a runtime cycle_count reload, needed for
loops) and Cranelift's default regalloc spills more in larger functions. The
dispatch savings are real but offset by native-code overhead that grows with
chain length. Converting the extra chaining into wall time now needs
host-register allocation (tighter code) — the clear next lever. Env gate,
diff-off-under-chain, and probe/mem-watch exclusivity unchanged from B1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Increment B1 of block-linking: compile a chain of same-page, fully-covered
blocks into ONE Cranelift function whose internal block-to-block edges are
direct machine jumps, eliminating the per-block return to run_block and the
run_superblock dispatch tax for the chain. Env-gated XENIA_JIT_CHAIN (off by
default), and additionally disabled under the diff harness or with diagnostic
probes / mem-watch armed (chaining runs several blocks natively without the
per-block-entry observation those need, so they are mutually exclusive).
Mechanism (cranelift-jit can't patch finalized code, so no QEMU-style TB
chaining): SUPERBLOCK COMPILATION. compile_chain enumerates the forward-linear
chain from the entry — following static bx targets and fall-throughs, all
same-page (so the entry page_version is a sound cache key), outside the thunk
band, and covered — and emits one node per block plus a shared exit. At every
internal boundary it re-checks the two runtime stop-conditions exactly as the
Rust runner does: an MMIO touch (*mmio_count advanced vs the entry sample) and
the instruction budget (cumulative >= remaining, passed as a runtime param);
either stops the chain with pc/cycle/timebase already correct, and the runner
re-dispatches from the clean boundary. B1 scope is forward-linear: a bx or
fall-through continues; a conditional bcx, dynamic bclrx/bcctrx, off-page /
thunk / uncovered / already-visited successor ends the chain (B2 adds bcx
two-way + loop back-edges). Being *more* conservative (stopping earlier) is
always safe.
Correctness rests on composition: each node body is the same emit_node_body IR
the single-block path emits (already diff-validated bit-exact), so only the
chain glue is new — validated by two new unit tests (multi-block chain vs
interpreter over the same PCs; budget cut at the exact boundary) plus e2e.
Boundary MMIO check is skipped for load/store-free blocks (a pure-ALU block
can't advance mmio_count), dropping a memory load+compare per ALU boundary.
Plumbing: build_block exposed (xenia-cpu); KernelState::thunk_band getter;
MemEnv mmio_count null-safety (points at a static zero when no flat mapping);
CompiledFn gains a remaining-budget param (single-block ignores it); all
compiled entries share the (ctx, mem_env, remaining) ABI.
Validation: diff MISMATCHES=0 over 25.0M blocks (single-block op bodies
unbroken by the emit_node_body refactor); XENIA_JIT_CHAIN=1 movie plays,
tid25 resumes, source-read=12; fair perf (RUST_LOG=warn) interp 44.1s vs
chain 41.3s = −6.4% wall — run_superblock "other" 30.4%->26.2%, run_block
calls 145.7M->116.2M (1.25x block merge), step_block at parity (100.6->102.4
MIPS; the earlier "slower" reading was cranelift IR-logging inside the STEP
timer). Uncommitted probe knobs unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Behavior-neutral prep on top of the inline-load parity milestone:
* JitCache caches mem.fast_mem() (resolved once — the mapping is invariant for
a run) so compiled blocks build their MemEnv without a virtual call per
execution. MemEnv::from_fast(mem, Option<FastMem>) is the non-virtual
constructor. Measured impact is within noise (96.5 -> 96.4 MIPS) but it
removes redundant per-block work.
* FastMem/MemEnv gain mmio_count: a pointer to GuestMemory's monotonic MMIO
access counter. Unused for now; the upcoming superblock-chaining JIT will
sample it across a block boundary to stop chaining on an MMIO touch
(preserving the interpreter's fine-grained MMIO ordering).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Loads no longer call a trampoline on the common path. When memory exposes a flat
mapping and the address is proven non-MMIO and committed, the compiled block
loads directly from `membase + ea` (byte-swapped) with zero calls; otherwise it
falls back to the read trampoline.
Mechanism:
* MemoryAccess::fast_mem() -> Option<FastMem { membase, page_table, mmio_mask,
mmio_value }>. Default None; GuestMemory returns Some (flat 4 GiB mapping,
page-table pointer, MMIO aperture pair). Wrappers that intercept accesses
(the recompiler's speculative OverlayMemory) inherit None, so the fast path
is disabled under the diff harness and their interception is preserved.
* MemEnv carries membase/page_table/mmio_mask/mmio_value; MemEnv::new(mem)
fills them (or nulls when fast_mem() is None).
* emit_inline_load emits a 5-block diamond: membase==0 -> slow (disabled);
(ea & mmio_mask) == mmio_value -> slow (maybe MMIO); page_table[ea>>12]
COMMIT bit (49) clear -> slow (unmapped; the mapped check is mandatory —
unmapped pages are PROT_NONE and a raw load would fault); else fast load
membase+ea, bswap, zero-extend. Wired all integer loads (lbz/lhz/lwz/ld +
x-forms) and FP loads (lfs/lfsx reuse the diamond then bitcast/fpromote;
lfd/lfdx bit-copy). Stores and FP-punt still trampoline.
Validation (three ways, since the diff harness disables the fast path):
* unit test jit_inline_load_matches_interpreter — real GuestMemory, both the
mapped (fast) and unmapped (slow) address bit-match the interpreter.
* e2e XENIA_JIT=1 boot+movie plays, clean exit.
* diff regression: checked 149.2M blocks, MISMATCHES=0 (validates the
unchanged slow/ALU/branch/FP logic).
Measured (2e9 instr, block-exec MIPS): interpreter 98.2 / JIT trampoline-loads
93.8 / JIT inline-loads 96.5 — inline loads recover the trampoline overhead,
bringing the JIT to parity (~1.7% slower, within noise). Beating the interpreter
needs the levers it can't do: block-linking (30% dispatch), host registers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Solve the FP-arithmetic fpscr problem without lowering fpscr into IR. Add a
generic single-instruction shim, xj_interp_op(ctx, env, raw, addr), that decodes
and runs one instruction through the interpreter's execute() on the live
context/memory — bit-exact by construction. This lets a block containing FP math
still be JIT-compiled: the surrounding integer/memory/branch ops run as machine
code and only the FP op calls back into Rust (the technique production JITs use
for complex ops). Validated on a full boot+movie run (movie plays, clean exit):
checked 147.2M blocks, 73.48% native (108.2M, up from 39.82%!), MISMATCHES=0
The fpscr bookkeeping (rounding-mode-dependent rounding, sticky exception bits,
FPRF classification) was too intricate to emit correctly in IR; punting sidesteps
it entirely while still capturing the coverage.
is_fp_punt allowlist (all verified always-Continue, non-branch):
faddx/faddsx, fsubx/fsubsx, fmulx/fmulsx, fdivx/fdivsx, fmaddx/fmaddsx,
fmsubx/fmsubsx, fnmaddx/fnmaddsx, fnmsubx/fnmsubsx, frspx, fsqrtx, fresx,
frsqrtex, fselx, fnabsx, fcmpu, fcmpo.
covered() returns true for them; emit_op emits `call interp(ctx, env, raw,
addr)`. execute() advances pc by 4 (harmless — the block epilogue stamps end_pc
for non-branch blocks; cycle_count is bumped once per block, not by execute).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cover the fpscr-free FP moves. Validated bit-exact via the in-process
differential harness on a full boot+movie run (movie plays, clean exit):
checked 146.2M blocks, 39.82% native (up from 38.78%), MISMATCHES=0
Coverage (all fpscr-free — pure data movement, no rounding flags):
* FP loads: lfs/lfsx (load single, IEEE-widen to the f64 FPR via
read32 -> bitcast F32 -> fpromote F64), lfd/lfdx (pure 64-bit bit copy).
* FP stores: stfs/stfsx (fdemote f64 -> f32 -> bitcast i32 -> write32),
stfd/stfdx (pure 64-bit bit copy). Stores reuse the reservation-kicking
write trampolines.
* FP reg moves (Rc=0 only — the `.` forms update CR1 from fpscr): fmr (bit
copy), fabs (band_imm i64::MAX, clear sign), fneg (bxor_imm i64::MIN, flip
sign) — bit ops that exactly match Rust f64 copy/abs/neg.
Note: the +1% lift is small because the movie's FP-heavy blocks almost always
mix an FP load with FP *arithmetic* (fadds/fmuls/fmadds), which stays
uncovered — it updates fpscr (FPRF/FI/FR, invalid-op flags), not yet lowered.
The moves that landed are the pure data-copy blocks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the memory tier: compiled loads/stores call `extern "C"` trampolines that
dispatch through the `MemoryAccess` trait, so MMIO dispatch, mem-watch,
page_version, and mmio_access_count stay bit-identical to the interpreter.
Validated bit-exact via the in-process differential harness on a full
boot+movie run (movie plays, clean exit):
checked 148.2M blocks, 38.78% native (57.5M, up from 22.02%), MISMATCHES=0
Mechanism:
* 8 trampolines: xj_read8/16/32/64(env, addr) and
xj_write8/16/32/64(ctx, env, addr, val). Registered with the JITBuilder via
symbol(), declared Linkage::Import in Jit::new (FuncIds in TrampIds), and
re-referenced into each compiled function via declare_func_in_func.
* emit_op now takes an EmitCtx { ctxp, memenv, trampoline FuncRefs }.
* Store trampolines replicate the interpreter's pre-store reservation
invalidation (store_reservation_kick) — an ordinary store to a reserved
line must be observed by stwcx peers. They receive the PpcContext pointer so
they can read ctx.reservation_table; the diff clone clears it (None), so
speculation never touches shared reservation state.
Coverage added (all mirroring execute() exactly):
* loads (zero-extended, non-update): lbz/lbzx, lhz/lhzx, lwz/lwzx, ld/ldx.
* stores (non-update): stb/stbx, sth/sthx, stw/stwx, std/stdx.
EA via ea_d (D-form disp) / ea_x (X-form indexed): ra==0 => 0 base, then
truncate to 32 bits. Update (u) forms and algebraic sign-extending loads
(lha/lwa) are not lowered yet.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Grow the covered opcode set from branches-only to the hot integer core.
Validated bit-exact vs the interpreter via the in-process differential harness
on a full boot+movie run (movie plays, clean exit):
checked 147.9M blocks, 22.02% native (32.6M, up from 6.57%), MISMATCHES=0
New coverage (all mirroring interpreter::execute exactly):
* add/sub: addx, subfx (OE=0 only — the overflow path is not lowered yet).
Full 64-bit result; CR0 (when Rc) from the low-32 signed value.
* reg-reg logical, 64-bit-preserving: orx (excluding the 0x7FFFFB78 db16cyc
spin hint, which yields), andx, xorx.
* reg-reg logical, u32-truncating (zeroes the upper 32): norx, nandx, andcx,
orcx — stored zero-extended via store_gpr32z.
* immediate logical: ori/oris/xori/xoris (64-bit, no CR); andi./andis.
(always update CR0).
* rlwinmx: rotate-left-word + mask (mask computed at emit time from the mb/me
immediates); zeroes upper 32; CR0 when Rc.
* compares: cmp/cmpi (signed, 64- or 32-bit per L), cmpl/cmpli (unsigned);
write the crfd() field. Immediates sign- or zero-extended per form.
CR-write emission:
* emit_store_cr(field, lt, gt, eq): stores the four CrField bytes
{lt@0,gt@1,eq@2,so@3}; so = (xer_so != 0).
* emit_cr0_signed32(val32): the Rc-form CR0 update — signed compare of the
32-bit result against zero. Reuses emit_store_cr.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the Cranelift block-JIT into execution and add branch coverage. Both
increments validated bit-exact against the interpreter via the in-process
differential harness on a full boot+movie run (movie plays, clean exit):
wiring only (addi/addis): checked 146.8M blocks, 0.01% native, MISMATCHES=0
+ branches (bx/bcx/bclrx): checked 148.5M blocks, 6.57% native (9.76M), MISMATCHES=0
jit.rs
* JitCache: direct-mapped 64K-slot compiled-block cache keyed (start_pc,
page_version) identically to BlockCache, so self-modifying / DMA'd code
invalidates native code the same way. Caches the None ("uncovered") verdict
so an uncovered block is compile-attempted at most once per (pc,version).
Owns the Jit/JITModule (keeps every CompiledFn valid for its lifetime).
* covered(): addi/addis + bx/bcx/bclrx. bcctrx excluded (indirect target +
dispatch_rec diagnostic hook the native path would skip).
* pc-handling refactor: a branch terminator writes pc itself (writes_pc());
the block epilogue stores end_pc only for straight-line (max-len / page-
boundary) blocks. cycle/timebase still += N (covered ops never fault/yield;
a branch is always the last instruction).
* Branch lowering uses immediate targets — the interpreter's ctx.pc equals the
instruction address at emit time, so bx/bcx relative targets are constants.
emit_branch_taken mirrors the interpreter: optional CTR decrement, ctr_ok =
(ctr as u32 vs 0) inverted by BO3, cond_ok = CR-bit BI byte == BO1 (both BO
sub-cases const-fold), combined with select. bclrx reads lr&!3 before the LK
link overwrites lr.
recompiler.rs
* run_block / diff_step take &mut JitCache. run_block runs the native fn when
get_or_compile returns one (else interpreter fallback); diff_step runs it on
the speculative clone/OverlayMemory so every native block is diff-checked.
* report_jit_summary(): native-vs-interpreted block counts (XENIA_JIT / _DIFF).
main.rs
* WorkerCtx owns a JitCache; run_superblock routing passes it to
diff_step/run_block; report_jit_summary() at clean exit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stage-2 foundation. Adds cranelift-jit/frontend/module/codegen 0.128.4 (the
1.90-compatible line; 0.133 needs Rust 1.94) and a new crates/xenia-cpu/src/jit.rs.
Jit owns a JITModule (and thus all compiled code memory). compile(&DecodedBlock)
lowers a block to native code ONLY if every opcode is covered() — otherwise
returns None and the caller interprets the whole block (coverage grows
opcode-by-opcode). ABI: extern "C" fn(*mut PpcContext, *const MemEnv) -> u32
(StepResult discriminant); guest registers are loaded/stored directly from the
#[repr(C)] PpcContext at offset_of! offsets. A covered block is straight-line and
always runs to completion, so pc advances to end_pc and cycle_count/timebase bump
by the instruction count once — matching the interpreter's per-instruction bump.
Covered set so far: addi, addis. Unit test jit_matches_interpreter_addi_block
compiles a 32x addi block and asserts the JIT's r3/pc/cycle_count/timebase match
the interpreter exactly — proves module setup, offset_of register access, IR
emission, the extern "C" calling convention, and cycle/pc accounting end-to-end.
Not yet wired into run_superblock (needs a compiled-block cache + routing); every
future opcode will be validated against the interpreter via the M0 XENIA_JIT_DIFF
harness before it counts as covered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
