xenia-rs/HANDOFF-iterate-4A-milestone2.md

Handoff — branch iterate-4A/apu-xma-stage1 (Milestone 2: intro-video / XMA audio + RE tooling)

Reverse-engineering Project Sylpheed under this Rust Xbox-360 emulator (xenia-rs), using Wine xenia-canary as the ground-truth oracle. This branch carries Milestone 2 work plus major RE-tooling improvements, on top of the (uncommitted-until-now) Milestone-1 renderer history.

Method: first-divergence vs canary · fix causes not symptoms · NO faking/masking · measure the oracle, never infer · refute before believing · ground every claim in evidence.

---

0. SET UP ON A NEW MACHINE (do this first)

a) FFmpeg system libraries — REQUIRED to build (crate xenia-apu links them via pkg-config)

The XMA audio decoder uses ffmpeg-sys-next (crates/xenia-apu/Cargo.toml: ffmpeg-sys-next = { version = "6.1", default-features = false, features = ["avcodec"] }), which links the system FFmpeg dev libraries. Install them:

sudo apt update
sudo apt install -y libavcodec-dev libavformat-dev libavutil-dev libswresample-dev pkg-config ffmpeg

Verify the toolchain (the XMA path needs the xma1/xma2 decoders — present in distro FFmpeg ≥ ~2015):

pkg-config --modversion libavcodec          # expect 60.x (this branch built against 60.31)
ffmpeg -hide_banner -decoders | grep -iE 'xma1|xma2'   # expect: A....D xma1 / A....D xma2

(Decoder note: distro FFmpeg has no AV_CODEC_ID_XMAFRAMES; we use AV_CODEC_ID_XMA2 — see crates/xenia-apu/src/xma2_codec.rs.) On non-Debian distros install the equivalent -dev packages.

b) The game ISO (gitignored — *.iso)

Not in the repo. Place the Project Sylpheed ISO somewhere and create a sylpheed.iso symlink to it in the repo root (the run/test commands use sylpheed.iso):

ln -s "/path/to/Project Sylpheed - Arc of Deception (USA, Europe) (En,Ja).iso" sylpheed.iso

⚠️ For canary runs, point at the REAL ISO path, not the symlink (Wine can't resolve the symlink).

c) Build — always cap parallelism (a default -j build OOM-crashed a 15 GB box)

export CARGO_BUILD_JOBS=4        # NEVER default -j12; check `free -h` first, drop to -j2 if <4GB free
cargo build --release

d) Regenerate the static-analysis DB sylpheed.db (gitignored — *.db, ~586 MB, ~1h35m)

Used by the RE/analysis queries (NOT needed to run the emulator). Rebuild from the ISO:

cargo run --release -- dis "/path/to/<the ISO>" --db sylpheed.db
# analysis passes run in <1s; the ~1h35m is DuckDB persisting ~1.8M dispatch rows. Be patient.

This branch's analyzer fix (see §3) makes the regenerated DB include the previously-missing XMV engine vtables (0x8200a1e8/0x8200a908). A local pre-fix backup may exist as sylpheed.db.bak-pre-vtablefix (gitignored, not pushed).

---

1. WHAT'S ON THIS BRANCH (all in this one commit, on top of acb29db = iterate-3AL)

Milestone-1 renderer history (publisher/dev splash renders) is in the ancestry (iterate-2x → 3M → 3O → 3AL); pushing this branch carries it. Milestone 2 + tooling added here:

✅ XMA AUDIO path — BUILT, WORKING, deterministic, tested

• crates/xenia-apu/src/xma.rs — register-mapped XMA context system (MMIO 0x7FEA0000, 320×64B context array, Kick/Lock/Clear decode). xma_decode.rs + xma2_codec.rs — the real FFmpeg xma2 decoder (XMA_CONTEXT_DATA bitfields, BitStream packet parse, planar-f32→S16BE PCM). Decode runs synchronously on the CPU thread (deterministic, no host thread). Wired via KernelState.xma (state.rs), exports (exports.rs), xaudio.rs (XAudioSubmitRenderDriverFrame made faithful), main.rs (MMIO install + per-round pump).
• Audio-worker scheduler fix (main.rs LR_HALT restore + scheduler.rs): the XAudio render callback worker was wrongly exited after ~2 deliveries → fixed → the guest now drives XMA decode.
• Verified: real PCM out; golden sylpheed_n50m re-baselined (crates/xenia-app/tests/golden/) and PASSES; milestone-1 splash intact; apu/cpu/kernel tests pass.

🛠️ RE TOOLING (this branch's lasting wins)

• Runtime dispatch-recorder crates/xenia-cpu/src/dispatch_rec.rs — records (call-site → target, r3, lr) for every indirect (bcctr-family) call. Off by default; enable with XENIA_DISPATCH_REC=1, optional filters XENIA_DISPATCH_REC_TARGETS=<hex,…> / _SITES=<hex,…>, dumps to XENIA_DISPATCH_REC_OUT (default /tmp/dispatch_rec.txt). Deterministic, observe-only.
• Repaired static analyzer crates/xenia-analysis/src/vtables.rs — the vtable extractor silently fragmented vtables with non-function head slots (missed the XMV engine vtable entirely → blocked ~6 investigations). Fixed via vptr-write-anchoring (find addis/addi → stw rX,0(rThis) constant-vptr installs; read the fnptr run from each anchor). Result on rebuild: vtables 722→1150, dispatch candidates 688K→1.83M, engine fully typed. (Requires the §0d DB rebuild to take effect.)
• Probe Heisenbug FIXED (main.rs run_superblock) — --audit-pc-probe-hex / --mem-watch used to disable superblock chaining, which changed thread scheduling and starved the movie subsystem so the probes couldn't observe it. Now probes fire inside the chain loop → scheduling is identical armed-vs-unarmed (verified byte-identical golden) → the probe suite is finally usable on the movie subsystem. Also fixed a --quiet bug that swallowed armed --trace-handles/--dump-addr reports.

---

2. CURRENT STATE & WHERE TO CONTINUE (the video still doesn't play)

Audio works; the intro VIDEO doesn't play yet. Root, runtime-pinned: a 2000ms readiness timeout (sub_821B66B8) abandons because the XMV engine (0x40d101c0, runtime vtable 0x8200a1e8) never primes — engine begin-playback sub_825076F0 (slot 21) is never dispatched (0×), so the per-frame full-start always takes its skip branch and the playback clock never starts.

• Classification: (B) guest-side state machine. The gate fields are the engine's correct reset defaults → there is NO honest our-side fix at the gate (forcing them = masking, forbidden). The defect is upstream: the guest SM reaches "create decoder (success)" but never issues begin-playback.
• Latest narrowing (evidence, fixed probes): ARM2-setup sub_821B55D8 runs once, create-decoder sub_824F8398 succeeds, and ARM2 then calls engine-setup wrappers sub_824F7778 / sub_824F7630 / sub_824F7558 / sub_824F7538 / sub_824FCB68 (on [movie+104]=engine) — the begin-playback dispatch is gated inside one of these. Tracing them (now possible with the fixed probes) for the begin-playback gate + why ours never satisfies it is the next step. The likely ultimate unlock is measuring canary (same XEX reaches begin-playback) to find the upstream state/signal we don't produce.

Full, evidence-grounded detail (engine/vtable/slot map, the eliminations, the investigation arc, the method lessons) lives in the agent-memory grounding file referenced in the project memory index (milestone2_xma_grounding). Key anchors: engine 0x40d101c0 vtable 0x8200a1e8 — PUMP slot19 sub_825078D8, begin-playback slot21 sub_825076F0, submit slot27 sub_82505C08, full-start slot40 sub_825061E0; movie host 0x40bb0440 (engine at [host+104]); SM ARM1 sub_821B4C98 → ARM2 sub_821B55D8 → ARM3 sub_821B5FB8 → poll sub_821B66B8.

Useful commands

# Headless run to the video state (~30-40s, ~1B instr); add diagnostic flags as needed:
./target/release/xenia-rs exec sylpheed.iso -n 6000000000 --quiet
# Non-perturbing PC probes (now usable on the movie subsystem):
RUST_LOG=warn,xenia_apu=info XENIA_AUDIT_PC_PROBE=0x825078d8,0x82505c08 \
  ./target/release/xenia-rs exec sylpheed.iso -n 6000000000 --quiet
# Dispatch recorder (filtered):
XENIA_DISPATCH_REC=1 XENIA_DISPATCH_REC_TARGETS=0x825076f0,0x82505c08 \
  ./target/release/xenia-rs exec sylpheed.iso -n 6000000000 --quiet
# Golden / determinism check:
CARGO_BUILD_JOBS=4 cargo test -p xenia-app --release --test sylpheed_oracles -- --ignored sylpheed_n50m
# Visual (watch the splash; ASK a human to watch — never self-screenshot):
./target/release/xenia-rs exec sylpheed.iso --ui

⚠️ Probe/run discipline: kill background runs by pid or pkill -x xenia-rs (NEVER pkill -f, it self-matches the launcher). Runs are deterministic (instruction-count clock).

🤖 Generated with Claude Code