diff --git a/docs/agents/CONSOLIDATION.md b/docs/agents/CONSOLIDATION.md new file mode 100644 index 00000000..cb9f02af --- /dev/null +++ b/docs/agents/CONSOLIDATION.md @@ -0,0 +1,461 @@ +# Consolidating six repositories into two + +**Set by the human, 2026-09-13.** The end state is **two** repositories: + +| repo | holds | +|---|---| +| **`Sylpheed`** | everything β€” the port, the RE corpus, every tool | +| **`Xenia-Canary`** (our fork) | the emulator and oracle, with our instrumentation | + +Everything else is retired. This page is the ordered **how**, with a gate after +each phase, and a ledger of what was checked rather than assumed. + +⚠️ **Most paths on this page do not resolve in this repository, on purpose.** +They name files in `xenia-rs`, `xex2tractor`, `xenia-canary` and the project +root β€” the things being moved *in*. A citation checker pointed at +`docs/agents/` will flag every one of them until the phases below are done, and +that is the correct reading: **each unresolved path here is work outstanding.** + +πŸ“Œ Two decisions from the human frame the whole plan: + +> *"The emulator is dead and can be dropped altogether. But the DuckDB generator +> should still be a useful tool… We probably can move it into a standalone tool +> under Sylpheed."* + +> *"xex2tractor was an early attempt and is long dead too… you may want to scan +> and compare the tools for anything that is in xex2tractor only."* + +--- + +## Where things stand + +| repo | server | local tree | tracked | disposition | +|---|---|---|---|---| +| **Sylpheed** | 598 MB | 33 GB | 131 MB / 1470 files | the target | +| **Xenia-Canary** | 71 MB | 20 GB | β€” | the keeper β€” ⚠️ **dirty** | +| Syplheed-Reborn | 103 MB | 1.4 GB | 100 MB / 787 files | βœ… absorbed β†’ archive | +| Sylpheed-Godot | 0.4 MB | 99 MB | 0.3 MB / 39 files | βœ… absorbed β†’ archive | +| xenia-rs | 9 MB | 1.2 GB | 61 MB | harvest the DB tool β†’ archive | +| xex2tractor | 0.3 MB | *no clone* | β€” | harvest four things β†’ archive | + +`Sylpheed` has **two root commits** β€” `feat: initialise workspace` (2026-03-25, +Reborn's) and `scaffold the Godot port` (2026-08-28). Both histories were +grafted in, not copied, which is why the absorption checks below pass. + +## The ledger β€” what was verified, and how + +Nothing here is inference. Each row is a command that was run on 2026-09-13. + +| claim | method | result | +|---|---|---| +| Reborn fully absorbed | all **21 remote + 8 local** branch tips vs `Sylpheed/main` | every one an ancestor | +| Sylpheed-Godot absorbed | both branch tips, + the root-commit graft | every one an ancestor | +| `agent-backups/` redundant | `git bundle list-heads`, 23 refs across 4 bundles | all present in Sylpheed | +| 3 unpushed `pi/*` branches redundant | `git patch-id --stable`, per commit | all 4 patches on `main` | +| nothing lives outside git | issues / PRs / releases / wikis, all six repos | **0** everywhere but Sylpheed | +| the DB tool is separable | `git grep "use xenia_cpu::"` in `xenia-analysis` | **three symbols** | + +### πŸ”΄ What the ledger found that a naive plan would have destroyed + +1. **The F1 instrument exists in one place, uncommitted.** `xenia-canary` has two + dirty files β€” `hid/file/file_input_driver.h` (`--pad_file_repeat`) and + `app/xenia_main.cc` β€” plus **4 unpushed commits** on `sylpheed-re` and a + branch, `audit-handle-lifecycle-probes`, **contained in no remote at all**. + That patch is the instrument behind `REPEAT_DELAY`/`REPEAT_INTERVAL`, now + shipped in the port. Its own finding page says *"not yet upstreamed."* +2. **xenia-rs has 4 unpushed branches** β€” `iterate-4B/ui-perf`, `4C/jit`, + `4D/parallel`, `4E/native-threads`. Dropping the emulator is a decision; + dropping work that was never backed up is a different act. +3. **The tool is not on `master`.** `xenia-rs/master` last moved **2026-06-15**; + the DuckDB and disassembler work is on `origin/iterate-4A/apu-xma-stage1` + (**2026-09-10**, +22 commits). Extract from the default branch and you get a + three-month-old tool. + +--- + +## Phase 0 β€” Secure the single-copy work βœ… **DONE** + +**Before touching anything else.** One `git checkout` in `xenia-canary` ends the +reproducibility of a measurement that is already live in the port. + +1. Commit `file_input_driver.h` + `xenia_main.cc` on `sylpheed-re`. +2. Push `sylpheed-re` (4 + 1 commits) and `audit-handle-lifecycle-probes`. +3. ~~Resolve the `third_party/DirectXShaderCompiler` submodule drift.~~ + βœ… **There is none.** `git submodule status` shows no `+`: the checked-out + commit matches the recorded one. The ` m` in `git status` is *nested* + submodule dirt β€” DXC's own `external/SPIRV-Headers` and `SPIRV-Tools` β€” which + is upstream's business and must not be committed here. + +**Gate β€” and the first version of it was wrong.** It compared each local branch +to a remote **by SHA**, and reported seven safe branches as unpushed: a branch +that is an *ancestor* of a pushed branch is already preserved. A check that goes +red for something unactionable is the failure this repo keeps naming, so the +gate tests what it means β€” **reachability**: + +```bash +git for-each-ref --format='%(refname:short)' refs/heads | while read b; do + s=$(git rev-parse "$b"); found="" + for r in $(git for-each-ref --format='%(refname:short)' refs/remotes); do + git merge-base --is-ancestor "$s" "$r" 2>/dev/null && { found=$r; break; } + done + [ -n "$found" ] || echo "πŸ”΄ $b NOT REACHABLE from any remote" +done +``` + +βœ… **PASSED 2026-09-13** β€” `8e63a9542` commits the instrument; `sylpheed-re` and +`audit-handle-lifecycle-probes` pushed; all **14** local branches reachable. + +## Phase 1 β€” Make the archive honest βœ… **DONE** + +Push xenia-rs `iterate-4B/ui-perf`, `4C/jit`, `4D/parallel`, `4E/native-threads`. + +They are being retired, not rescued. But a retired branch that was never pushed +is not retired β€” it is deleted, quietly, by someone who thought it was safe. + +**Gate:** the same loop, run in every repo, prints nothing. + +βœ… **PASSED 2026-09-13** β€” all four pushed (32 / 52 / 56 / 59 commits ahead of +`master`). The loop is clean in all five repos **except** `pi/clippy`, +`pi/clippy-clean` and `pi/reauth3` in Sylpheed, which are unreachable by +ancestry and redundant by *patch-id* β€” the ledger's reason, and Phase 7's +disposal. The gate is right to flag them; they are the one case where the +answer lives outside it. + +## Phase 2 β€” Harvest xex2tractor, then retire it βœ… **DONE (PR #30)** + +Four things exist only there. + +| take | why | +|---|---| +| **XEX2 devkit + XEX1 retail master keys** | `xenia-xex` hardcodes retail only; devkit is present but `#[allow(dead_code)]` and XEX1 is absent. xex2tractor tries all three in a validating loop. | +| ~~**`extract -r`**~~ | πŸ”΄ **RUN, AND THE INFERENCE WAS WRONG** β€” see the gate below. Optional, and dangerous if it is ever mistaken for the `.pe`. | +| **`doc/xex2_format.md`** (39 KB) and **`doc/xbox360_exports.json`** (938 KB, 2,913 exports across xboxkrnl / xam / xbdm) | byte-identical to untracked loose copies in the project root. Tracked by **no repo**. | +| **`LICENSE`** (MIT) | Sylpheed has none. | + +Leave everything else: `xenia-xex` is ahead on `resources.rs` (XDBF/XACH), +`pdata.rs`, `tls.rs`, basic zero-fill compression, and the analysis layer. + +**Gate β€” βœ… RUN 2026-09-13, and it corrected this page in two directions.** + +``` +extract (xex2tractor, no -r) βœ… IDENTICAL to the project .pe +extract -r (xex2tractor) ❌ differs β€” 1,903 bytes over 273 runs +extract (xenia-rs, iterate-4A) βœ… IDENTICAL β€” and .xex.json identical too +``` + +**Right:** the `.pe` *is* an xex2tractor extract. Hash-for-hash. + +**Wrong:** it was made **without** `-r`. The project's `.pe` is the plain +decrypted-and-decompressed image, imports **unresolved** β€” and `-r`'s 1,903 +differing bytes sit in the import thunk region from `VA 0x82000600`. + +πŸ”΄ **So the dependency this gate existed to protect does not exist.** This page +claimed *"drop this and the file every `.pe`-offset citation depends on can +never be rebuilt."* It can: **`xenia-rs extract`, the tool Phase 3 keeps, +reproduces it byte-for-byte** β€” along with the loose `.xex.json`, which turns out +to have come from xenia-rs, not xex2tractor. + +⚠️ **And `-r` must never become the `.pe`.** It writes into the image, so a `.pe` +built with it would silently invalidate every byte-offset citation in +`docs/re/`. If resolved-import disassembly is ever wanted, it is a **separate +artifact under a different name**, never a replacement. + +That leaves xex2tractor's unique list at three items, none urgent β€” the two +master keys, the two doc assets, and the licence. **The doc assets are now the +only reason to visit that repository.** + +πŸ“Œ Learned in passing: `iterate-4A` **builds clean** (`xenia-app`, one unused +variable warning), so Phase 3's source is not bit-rotted β€” and it needs +`libavutil-dev` and friends via `ffmpeg-sys-next`, dragged in by the emulator's +XMA audio. The standalone tool sheds a C library the CI image did not have. + +## Phase 3 β€” Lift the DB tool out of the emulator βœ… **DONE (PR #32)** + +From `origin/iterate-4A/apu-xma-stage1`, **never** from `master`. + +``` +xenia-analysis 10,658 db, vtables, jumptables, func, xref, rtti, + strings, xdbf, static_init, demangle, sql_views +disasm slice of cpu 3,681 disasm.rs + decoder.rs + opcode.rs (of 21,813) +xenia-xex 2,018 header, loader, lzx, pe, pdata, tls, resources +xenia-vfs 444 so it reads an ISO, not just a loose .xex +xenia-types 361 + ─────── + β‰ˆ17,200 of ~71,000 +``` + +The coupling is exactly three symbols β€” `decoder::decode`, `disasm::DisasmItem`, +`disasm::format`. Interpreter, JIT, scheduler, VMX, block cache, reservations: +none of it is reachable from the generator. + +`xenia-xex` declares `xenia-memory` and `xenia-types` as dependencies and has +**zero `use`** of either. Probably droppable β€” confirm by compiling, do not +assume. + +Travelling with it: `zq.py`, `tests/db_schema_golden.rs`, +`tests/disasm_goldens.rs`, `examples/decode_table_check.rs`. + +**Two corrections to make during the move, not after:** + +* the CLI help says **"SQLite database"** in two places. It is **DuckDB** + (`duckdb = { workspace = true }`). +* `zq.py`'s documented escape hatch β€” *"the engine vtable / rdata is NOT in the + DB … read it from guest memory with `xenia-rs exec --dump-addr`"* β€” **dies with + the emulator.** Replace it with Canary's dump or a direct read of the `.pe` at + `VA βˆ’ 0x82000000`, and say which, in the file. + +**Gate β€” βœ… PASSED 2026-09-13.** Schema a **superset** (22 new tables, none +lost). `instructions` 1,865,751, `pdata_entries`, `imports`, `sections` and all +three `eh_*` identical to the row; most others slightly higher. Two tables drop +~1.8 M and it is not a loss β€” the xref delta is **entirely `ind_call`** +(everything else nets **+1,512**), and the new schema records why: **6,556 sites +`truncated=true` carrying 1,801,075 candidates** against 427 untruncated +producing 4,199 rows. The old database materialised a cross product. + +## Phase 4 β€” Prune the captures, and give `docs/re/` the check `docs/port/` has βœ… **DONE (PR #33)** + +**`docs/re/captures/` is 257 files / 117.6 MB, and it is not junk.** Measured: + +``` +193 (81.2 MB) cited by a docs/re page, or opened by a tool + 64 (36.5 MB) referenced by nothing +``` + +πŸ”΄ **Two are test fixtures and `cargo test` opens them:** + +``` +crates/sylpheed-formats/tests/ui_paint_order_disc.rs β†’ captures/title-screen-oracle.png +crates/sylpheed-formats/tests/unit_layout_disc.rs β†’ captures/stage02-live-unit-definitions-deep.txt +``` + +Sixteen more tools under `tools/port/` and `tools/re-capture/` read from there. + +**Do:** + +1. βœ… **DONE β€” `tools/re/check-capture-citations`**, the mirror of + `tools/port/check-citations`, which has only ever checked `docs/port/*.md`. + It reports two classes β€” a capture nothing cites, and a page citing a capture + that was never committed. It needs a `--selftest` that plants one of each, or + it is not a check. +2. βœ… **DONE β€” and there was ~~**10**~~ ONE.** Eight of the ten were *directory* + references that resolve; one was sentence punctuation in the regex. The real + one, `live-submenu-unidentified.png`, had been **committed and then deleted** + on 2026-08-30 while the page kept citing it and kept making the claim it + backs. Restored, not de-linked. +3. βœ… **DONE β€” 46, not 64.** Two are opened by *filename* from + `screen_match.py`, and sixteen sit inside directories that pages cite as + directories. 30.7 MB out of the checkout. + +⚠️ **An orphan is not automatically waste.** It may be evidence a page *should* +have cited, and the page is the thing that is wrong. Fix the citations before +deleting, or step 3 silently ratifies every omission in step 2. + +πŸ“Œ **Going forward the class that matters is not captures-vs-code:** + +| | | +|---|---| +| **evidence** β€” a screenshot a finding points at | belongs beside the finding, as the convention says | +| **run output** β€” `.jsonl`, `.log`, `.csv` | the same class as xenia-rs's `audit-runs/`, which this plan drops | + +The two `.jsonl` mission-state dumps alone are 4.3 MB, and one is an orphan. + +## Phase 5 β€” Verify the invariant, then archive ⬜ **NOT STARTED** + +For each of `Syplheed-Reborn`, `Sylpheed-Godot`, `xenia-rs`, `xex2tractor`: +re-run the containment check **at that moment** β€” not from this page's ledger, +which will be days old β€” then archive read-only on Gitea, and only then delete +the local clone. + +**Archive before delete, and leave them archived.** The check proves the commits +are reachable. It does not prove nobody will want the repository boundary back. + +### ⚠️ The fork: history rewrite, or not + +Dropping files in Phase 4 **reclaims no space**. The 118 MB is in history β€” 512 +distinct blobs, 120.5 MB β€” and `git rm` leaves every byte in `.git` (590 MB +local, 598 MB on the server). Only a rewrite reclaims it, and it has a hard +blocker: + +```toml +# crates/sylpheed-export/Cargo.toml:84 +sylpheed-formats = { git = "…/Sylpheed.git", tag = "formats-pin-2026-09-01" } +``` + +**The build depends on this repo by tag**, pinned in `Cargo.lock` to `#1cd5b8b1`. +A rewrite invalidates all 8 `formats-pin-*` tags, every commit SHA, PR #23's +merge, both agents' clones, and needs a force-push through branch protection. + +* **Don't rewrite** (recommended): 118 MB on a host with 154 GB free, where + `Sylpheed/target` alone is 32 GB. The cost is not worth the blast radius. +* **Do rewrite:** then it happens **here, in Phase 5** β€” while tags are being + re-cut and agents re-cloned anyway, so the disruption is paid once instead of + twice. Re-point the `formats-pin` dependency **first**. + +This is a decision, not a default. Whichever is chosen, record it here. + +## Phase 6 β€” Adopt the orphan assets βœ… **DONE (PR #34)** + +Tracked by nobody today, and tools or references by the two-repo rule: + +| asset | β†’ | +|---|---| +| `ppc-manual/` β€” 397 files, **including `generator/generate_manual.py`** | Sylpheed | +| `docs/ppc_instructions.{json,md}`, `docs/xbox360_exports.*`, `docs/xex2_format.md` | Sylpheed `docs/` | +| `docs/CROSS_BUILD_SETUP.md` (58 KB) | **Canary** β€” it builds Canary | +| `run-canary.sh` | Sylpheed `tools/` | +| `ai-agent-*.md` Γ—4 (56 KB) | Sylpheed `docs/`, or discard β€” human's call | +| `scratch/xbg7/extract.py` β€” clean-room XBG7 scanner | superseded by `sylpheed_formats::mesh`; keep as history or drop | +| `Sylpheed-try/…/examples/_voice_span.rs` β€” 46-line untracked probe | keep or drop | + +## Phase 7 β€” Intended drops ⬜ **NOT STARTED** + +Stated so that nothing is dropped by omission. + +| drop | size | why it is safe | +|---|---|---| +| `agent-backups/` | 687 MB | all 23 refs verified present in Sylpheed | +| `stock-oracle/` | 34 MB | two Canary **binaries**, rebuildable from source | +| root `canary_*.log` / `.stdout` / `.stderr` | ~15 MB | run output | +| `vkd3d-proton.cache` | 15 KB | runtime cache | +| xenia-rs `audit-runs/` | 57 MB, 41 files | already `.gitignore`d; `iterate-4A` deleted 40 of 41 | +| local `pi/clippy`, `pi/clippy-clean`, `pi/reauth3` | β€” | all 4 patches on `main`, by patch-id | +| `Sylpheed/target/` | **32 GB** | build output; the host is at 83 % | +| `texcompare/` (14 MB), `ship_render/` (64 KB) | β€” | tool output β€” **confirm no `docs/re/` page cites them first** | +| **the xenia-rs emulator** | ~54k LOC | the human's decision: superseded by Canary | + +--- + +# ▢️ STATUS 2026-09-13 β€” paused here, for the other machine to finish + +**Six of eight phases are done and pushed.** Everything below is committed; +nothing is left in a working tree. Five pull requests are open and **none has +been merged** β€” merging is the human's, and #32 contains #30 and #31. + +| phase | state | where | +|---|---|---| +| **0** secure single-copy work | βœ… done | Canary `8e63a9542`, `sylpheed-re` + `audit-handle-lifecycle-probes` pushed | +| **1** make the archive honest | βœ… done | xenia-rs `iterate-4B/4C/4D/4E` pushed (32/52/56/59 commits) | +| **2** harvest xex2tractor | βœ… done | **PR #30** `harvest/xex2tractor-assets` | +| **3** lift the DB tool | βœ… done | **PR #32** `feat/xexdb-tool` | +| **4** captures + the missing check | βœ… done | **PR #33** `fix/capture-citations` | +| **6** adopt the orphan assets | βœ… done | **PR #34** `chore/adopt-orphan-assets`, + Canary `99cc72662` | +| β€” containers and agents | βœ… done | **PR #31** `docs/containers-setup` | +| **5** archive the four repos | ⬜ **open β€” needs the human** | | +| **7** intended drops | ⬜ **open β€” partly needs the human** | | + +## πŸ”΄ Four claims on this page were wrong. Read these before trusting the rest. + +1. **"10 dangling citations" was ONE.** Eight were *directory* references, which + resolve and are simply absent from `git ls-tree`; one was a path at the end of + a sentence with the full stop pulled into the match. The figure is corrected + in Phase 4 below and in PR #33. +2. **`extract -r` is not how the `.pe` was made**, and nothing depends on it β€” + Phase 2's gate, run. +3. **The Phase 0 gate compared branches by SHA** and called seven safe branches + unpushed. It tests reachability now. +4. **There is no DXC submodule drift.** The ` m` is *nested* submodule dirt. + +## What the other machine should do next + +### Before anything: merge or decide on the open PRs + +``` +#30 harvest/xex2tractor-assets reference files + LICENSE +#31 docs/containers-setup container and agent setup +#32 feat/xexdb-tool the DuckDB tool ⚠️ CONTAINS #30 and #31 +#33 fix/capture-citations the docs/re check + prune +#34 chore/adopt-orphan-assets PPC manual + canary launcher +``` + +⚠️ **#32 is stacked.** Merge #30 and #31 first, or merge #32 and close them β€” +its `build.rs` needs #30's export table, so they cannot be independent. + +⏳ **One result outstanding:** the workspace `cargo test` count for #32. `check`, +`clippy`, `fmt` and `wasm` are all 0; the test leg was interrupted twice by +branch switches and never reported cleanly. Run it and post the number: + +```bash +docker/ci/run cargo test --workspace +``` + +### Phase 5 β€” archive, and the history fork + +**Do not trust this page's ledger when you get there.** Re-run the containment +check *at that moment* β€” days will have passed and branches may have moved: + +```bash +for repo in Syplheed-Reborn Sylpheed-Godot xenia-rs xex2tractor; do + # every ref of $repo must be an ancestor of a Sylpheed ref +done +``` + +Then **archive read-only on Gitea**. πŸ”΄ **Do not delete any repository.** Archiving +is reversible; deletion is not, and it is the human's call, separately, later. +Nothing outside git needs migrating β€” issues, PRs, releases and wikis are **zero** +in all four (verified). + +The **history-rewrite fork** is still open and still recommended AGAINST: the +118 MB is in history, and `crates/sylpheed-export/Cargo.toml` depends on this +repo **by tag** (`formats-pin-2026-09-01`, pinned in `Cargo.lock`). A rewrite +invalidates all 8 tags, every SHA, PR #23's merge and both agents' clones. + +### Phase 7 β€” the drops + +All verified, none done β€” they are deletions on the human's disk: + +``` +agent-backups/ 687 MB 23 refs verified present in Sylpheed +Sylpheed/target/ 32 GB build output; the host is at 83 % +stock-oracle/ 34 MB two Canary binaries, rebuildable +root canary_*.log 15 MB run output +xenia-rs audit-runs/ 57 MB already .gitignore'd; iterate-4A deleted 40 of 41 +pi/clippy, pi/clippy-clean, pi/reauth3 all 4 patches on main, by patch-id +texcompare/ (14 MB), ship_render/ confirm no docs/re/ page cites them first +``` + +### Still the human's, untouched + +`ai-agent-*.md` Γ—4 (56 KB), `scratch/xbg7/extract.py`, `_voice_span.rs`, and the +**retention question**: all six repos are `private=False`, and `docs/re/captures/` +is ~87 MB of game screenshots in a public repository. + +## What each finished phase actually produced + +* **Phase 2** β€” `docs/reference/{xex2-format.md, xbox360-exports.*, ppc-instructions.*}` + and an MIT `LICENSE`; this repo had none. The first two are byte-identical to + xex2tractor's, verified with `cmp`. +* **Phase 3** β€” `sylpheed-xex` + `sylpheed-ppc` + `sylpheed-xexdb`, β‰ˆ17,200 lines + of ~71,000, coupled to the emulator by **three symbols**. The gate: a + regenerated database whose schema is a superset, with `instructions`, + `pdata_entries`, `imports`, `sections` and `eh_*` identical to the row. The two + βˆ’1.8 M deltas are the `max_indirect_candidates` ceiling, recorded by the tool + as **6,556 truncated sites standing for 1,801,075 candidates** β€” checked, not + assumed: the xref delta is entirely `ind_call`. `build.rs` no longer reads a + sibling checkout, and a missing export table now **fails** instead of silently + producing a database with no import names. +* **Phase 4** β€” `tools/re/check-capture-citations`, 212 captures all cited, 46 + orphans dropped. It caught its own damage twice: two "orphans" are opened by + *filename* from `screen_match.py`, and the first prune emptied two directories + that pages cite *as directories*. +* **Phase 6** β€” `tools/ppc-manual/` (393 files) and a rewritten + `tools/run-canary.sh`; the old one pointed at a **symlink** Wine cannot resolve + and hardcoded one machine's paths. The full `CROSS_BUILD_SETUP.md` went to + Canary, where the tracked copy was a 6.8 KB stub of a 59 KB guide. + +## The method, if you only read one thing + +Every claim on this page that was *reasoned* turned out wrong, and every one that +was *run* held. The gates are the point: Phase 2's gate refuted its own premise, +Phase 3's proved the two alarming row deltas were an improvement, and Phase 4's +check caught a deletion I had just made. **Run them again rather than trusting +the results recorded here.** + +## Still open, for the human + +* **Retention.** The `.iso` (7.6 GB), `.pe`, `.xex.json` and `sylph_extract/` + (6.2 GB) stay on disk and out of git. But `docs/re/captures/` is 118 MB of + game screenshots **inside a public repository**, about to be joined by a + 938 KB export database, and **all six repos are `private=False`**. Worth + choosing rather than inheriting. +* **The history fork** in Phase 5. +* **`ai-agent-*.md`, `scratch/`, `_voice_span.rs`** β€” keep or drop.