23189b95af
Milestone-2 (intro video dat/movie/ADV.wmv) audio path + major RE tooling. XMA AUDIO (built, working, deterministic, tested): - APU MMIO 0x7FEA0000 + 320x64B register-mapped context array; real XMACreateContext/Release (xma.rs); real FFmpeg xma2 decoder XMA_CONTEXT_DATA->S16BE PCM (xma_decode.rs, xma2_codec.rs, ffmpeg-sys-next). Decode runs synchronously on the CPU thread (deterministic, no host thread). - Audio-worker scheduler fix (main.rs LR_HALT restore + scheduler.rs): the XAudio render-callback worker was wrongly exited after ~2 deliveries; now survives -> guest drives XMA decode (70 kicks). - XAudioSubmitRenderDriverFrame made faithful. Golden sylpheed_n50m re-baselined; tests pass. RE TOOLING: - Runtime indirect-dispatch recorder (dispatch_rec.rs): records (call-site->target, r3, lr); env-gated XENIA_DISPATCH_REC, filters XENIA_DISPATCH_REC_TARGETS/_SITES; deterministic, observe-only. - Repaired static analyzer (vtables.rs): vtable extraction silently fragmented vtables with non-function head slots (missed the XMV engine vtable). Fixed via vptr-write-anchoring -> engine fully typed (vtables 722->1150 on rebuild). - Fixed probe HEISENBUG (main.rs run_superblock): --audit-pc-probe-hex/--mem-watch no longer disable superblock chaining; probes fire inside the chain loop -> scheduling identical armed-vs-unarmed, movie subsystem now observable. Fixed a --quiet bug swallowing armed trace reports. VIDEO still doesn't play (B, guest-side): the XMV engine never issues begin-playback (sub_825076F0, vtable 0x8200a1e8 slot21) -> never primes -> 2000ms timeout. Narrowed to the ARM2 engine-setup wrappers; no honest our-side gate-fix (masking forbidden). See HANDOFF-iterate-4A-milestone2.md for new-machine setup (incl. the FFmpeg apt deps + sylpheed.db regeneration) and continuation pointers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
134 lines
8.4 KiB
Markdown
134 lines
8.4 KiB
Markdown
# Handoff — branch `iterate-4A/apu-xma-stage1` (Milestone 2: intro-video / XMA audio + RE tooling)
|
||
|
||
Reverse-engineering Project Sylpheed under this Rust Xbox-360 emulator (`xenia-rs`), using Wine
|
||
xenia-canary as the ground-truth oracle. This branch carries **Milestone 2** work plus major
|
||
RE-tooling improvements, on top of the (uncommitted-until-now) Milestone-1 renderer history.
|
||
|
||
> Method: first-divergence vs canary · fix causes not symptoms · NO faking/masking · measure the
|
||
> oracle, never infer · refute before believing · ground every claim in evidence.
|
||
|
||
---
|
||
|
||
## 0. SET UP ON A NEW MACHINE (do this first)
|
||
|
||
### a) FFmpeg system libraries — **REQUIRED to build** (crate `xenia-apu` links them via pkg-config)
|
||
The XMA audio decoder uses `ffmpeg-sys-next` (`crates/xenia-apu/Cargo.toml`:
|
||
`ffmpeg-sys-next = { version = "6.1", default-features = false, features = ["avcodec"] }`),
|
||
which links the **system** FFmpeg dev libraries. Install them:
|
||
|
||
```bash
|
||
sudo apt update
|
||
sudo apt install -y libavcodec-dev libavformat-dev libavutil-dev libswresample-dev pkg-config ffmpeg
|
||
```
|
||
|
||
Verify the toolchain (the XMA path needs the `xma1`/`xma2` decoders — present in distro FFmpeg ≥ ~2015):
|
||
```bash
|
||
pkg-config --modversion libavcodec # expect 60.x (this branch built against 60.31)
|
||
ffmpeg -hide_banner -decoders | grep -iE 'xma1|xma2' # expect: A....D xma1 / A....D xma2
|
||
```
|
||
(Decoder note: distro FFmpeg has **no** `AV_CODEC_ID_XMAFRAMES`; we use `AV_CODEC_ID_XMA2` — see
|
||
`crates/xenia-apu/src/xma2_codec.rs`.) On non-Debian distros install the equivalent `-dev` packages.
|
||
|
||
### b) The game ISO (gitignored — `*.iso`)
|
||
Not in the repo. Place the Project Sylpheed ISO somewhere and create a `sylpheed.iso` symlink to it
|
||
in the repo root (the run/test commands use `sylpheed.iso`):
|
||
```bash
|
||
ln -s "/path/to/Project Sylpheed - Arc of Deception (USA, Europe) (En,Ja).iso" sylpheed.iso
|
||
```
|
||
⚠️ For **canary** runs, point at the REAL ISO path, not the symlink (Wine can't resolve the symlink).
|
||
|
||
### c) Build — **always cap parallelism** (a default `-j` build OOM-crashed a 15 GB box)
|
||
```bash
|
||
export CARGO_BUILD_JOBS=4 # NEVER default -j12; check `free -h` first, drop to -j2 if <4GB free
|
||
cargo build --release
|
||
```
|
||
|
||
### d) Regenerate the static-analysis DB `sylpheed.db` (gitignored — `*.db`, ~586 MB, ~1h35m)
|
||
Used by the RE/analysis queries (NOT needed to run the emulator). Rebuild from the ISO:
|
||
```bash
|
||
cargo run --release -- dis "/path/to/<the ISO>" --db sylpheed.db
|
||
# analysis passes run in <1s; the ~1h35m is DuckDB persisting ~1.8M dispatch rows. Be patient.
|
||
```
|
||
This branch's analyzer fix (see §3) makes the regenerated DB include the previously-missing XMV
|
||
engine vtables (`0x8200a1e8`/`0x8200a908`). A local pre-fix backup may exist as
|
||
`sylpheed.db.bak-pre-vtablefix` (gitignored, not pushed).
|
||
|
||
---
|
||
|
||
## 1. WHAT'S ON THIS BRANCH (all in this one commit, on top of `acb29db` = iterate-3AL)
|
||
**Milestone-1 renderer history** (publisher/dev splash renders) is in the ancestry (iterate-2x → 3M →
|
||
3O → 3AL); pushing this branch carries it. **Milestone 2** + tooling added here:
|
||
|
||
### ✅ XMA AUDIO path — BUILT, WORKING, deterministic, tested
|
||
- `crates/xenia-apu/src/xma.rs` — register-mapped XMA context system (MMIO `0x7FEA0000`, 320×64B
|
||
context array, Kick/Lock/Clear decode). `xma_decode.rs` + `xma2_codec.rs` — the real FFmpeg
|
||
`xma2` decoder (XMA_CONTEXT_DATA bitfields, BitStream packet parse, planar-f32→S16BE PCM).
|
||
Decode runs synchronously on the CPU thread (deterministic, no host thread). Wired via
|
||
`KernelState.xma` (`state.rs`), exports (`exports.rs`), `xaudio.rs` (`XAudioSubmitRenderDriverFrame`
|
||
made faithful), `main.rs` (MMIO install + per-round pump).
|
||
- **Audio-worker scheduler fix** (`main.rs` LR_HALT restore + `scheduler.rs`): the XAudio render
|
||
callback worker was wrongly exited after ~2 deliveries → fixed → the guest now drives XMA decode.
|
||
- Verified: real PCM out; golden `sylpheed_n50m` **re-baselined** (`crates/xenia-app/tests/golden/`)
|
||
and PASSES; milestone-1 splash intact; apu/cpu/kernel tests pass.
|
||
|
||
### 🛠️ RE TOOLING (this branch's lasting wins)
|
||
- **Runtime dispatch-recorder** `crates/xenia-cpu/src/dispatch_rec.rs` — records `(call-site → target,
|
||
r3, lr)` for every indirect (`bcctr`-family) call. Off by default; enable with `XENIA_DISPATCH_REC=1`,
|
||
optional filters `XENIA_DISPATCH_REC_TARGETS=<hex,…>` / `_SITES=<hex,…>`, dumps to
|
||
`XENIA_DISPATCH_REC_OUT` (default `/tmp/dispatch_rec.txt`). Deterministic, observe-only.
|
||
- **Repaired static analyzer** `crates/xenia-analysis/src/vtables.rs` — the vtable extractor silently
|
||
**fragmented vtables with non-function head slots** (missed the XMV engine vtable entirely →
|
||
blocked ~6 investigations). Fixed via **vptr-write-anchoring** (find `addis/addi → stw rX,0(rThis)`
|
||
constant-vptr installs; read the fnptr run from each anchor). Result on rebuild: vtables 722→1150,
|
||
dispatch candidates 688K→1.83M, engine fully typed. (Requires the §0d DB rebuild to take effect.)
|
||
- **Probe Heisenbug FIXED** (`main.rs run_superblock`) — `--audit-pc-probe-hex` / `--mem-watch` used to
|
||
**disable superblock chaining**, which changed thread scheduling and *starved the movie subsystem*
|
||
so the probes couldn't observe it. Now probes fire *inside* the chain loop → scheduling is identical
|
||
armed-vs-unarmed (verified byte-identical golden) → the probe suite is finally usable on the movie
|
||
subsystem. Also fixed a `--quiet` bug that swallowed armed `--trace-handles`/`--dump-addr` reports.
|
||
|
||
---
|
||
|
||
## 2. CURRENT STATE & WHERE TO CONTINUE (the video still doesn't play)
|
||
**Audio works; the intro VIDEO doesn't play yet.** Root, runtime-pinned: a 2000ms readiness timeout
|
||
(`sub_821B66B8`) abandons because the XMV engine (`0x40d101c0`, runtime vtable `0x8200a1e8`) never
|
||
**primes** — engine begin-playback `sub_825076F0` (slot 21) is **never dispatched** (0×), so the
|
||
per-frame full-start always takes its skip branch and the playback clock never starts.
|
||
- **Classification: (B) guest-side state machine.** The gate fields are the engine's *correct* reset
|
||
defaults → there is **NO honest our-side fix at the gate** (forcing them = masking, forbidden). The
|
||
defect is upstream: the guest SM reaches "create decoder (success)" but never issues begin-playback.
|
||
- **Latest narrowing (evidence, fixed probes):** ARM2-setup `sub_821B55D8` runs once, create-decoder
|
||
`sub_824F8398` succeeds, and ARM2 then calls engine-setup wrappers
|
||
**`sub_824F7778` / `sub_824F7630` / `sub_824F7558` / `sub_824F7538` / `sub_824FCB68`** (on
|
||
`[movie+104]`=engine) — the begin-playback dispatch is gated **inside one of these**. Tracing them
|
||
(now possible with the fixed probes) for the begin-playback gate + why ours never satisfies it is
|
||
**the next step**. The likely ultimate unlock is **measuring canary** (same XEX reaches begin-playback)
|
||
to find the upstream state/signal we don't produce.
|
||
|
||
Full, evidence-grounded detail (engine/vtable/slot map, the eliminations, the investigation arc, the
|
||
method lessons) lives in the agent-memory grounding file referenced in the project memory index
|
||
(`milestone2_xma_grounding`). Key anchors: engine `0x40d101c0` vtable `0x8200a1e8` — PUMP slot19
|
||
`sub_825078D8`, begin-playback slot21 `sub_825076F0`, submit slot27 `sub_82505C08`, full-start slot40
|
||
`sub_825061E0`; movie host `0x40bb0440` (engine at `[host+104]`); SM ARM1 `sub_821B4C98` → ARM2
|
||
`sub_821B55D8` → ARM3 `sub_821B5FB8` → poll `sub_821B66B8`.
|
||
|
||
### Useful commands
|
||
```bash
|
||
# Headless run to the video state (~30-40s, ~1B instr); add diagnostic flags as needed:
|
||
./target/release/xenia-rs exec sylpheed.iso -n 6000000000 --quiet
|
||
# Non-perturbing PC probes (now usable on the movie subsystem):
|
||
RUST_LOG=warn,xenia_apu=info XENIA_AUDIT_PC_PROBE=0x825078d8,0x82505c08 \
|
||
./target/release/xenia-rs exec sylpheed.iso -n 6000000000 --quiet
|
||
# Dispatch recorder (filtered):
|
||
XENIA_DISPATCH_REC=1 XENIA_DISPATCH_REC_TARGETS=0x825076f0,0x82505c08 \
|
||
./target/release/xenia-rs exec sylpheed.iso -n 6000000000 --quiet
|
||
# Golden / determinism check:
|
||
CARGO_BUILD_JOBS=4 cargo test -p xenia-app --release --test sylpheed_oracles -- --ignored sylpheed_n50m
|
||
# Visual (watch the splash; ASK a human to watch — never self-screenshot):
|
||
./target/release/xenia-rs exec sylpheed.iso --ui
|
||
```
|
||
⚠️ Probe/run discipline: kill background runs by pid or `pkill -x xenia-rs` (NEVER `pkill -f`, it
|
||
self-matches the launcher). Runs are deterministic (instruction-count clock).
|
||
|
||
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|