Sylpheed/docs/agents/CONSOLIDATION.md

# Consolidating six repositories into two

**Set by the human, 2026-09-13.** The end state is **two** repositories:

| repo | holds |
|---|---|
| **`Sylpheed`** | everything — the port, the RE corpus, every tool |
| **`Xenia-Canary`** (our fork) | the emulator and oracle, with our instrumentation |

Everything else is retired. This page is the ordered **how**, with a gate after
each phase, and a ledger of what was checked rather than assumed.

⚠️ **Most paths on this page do not resolve in this repository, on purpose.**
They name files in `xenia-rs`, `xex2tractor`, `xenia-canary` and the project
root — the things being moved *in*. A citation checker pointed at
`docs/agents/` will flag every one of them until the phases below are done, and
that is the correct reading: **each unresolved path here is work outstanding.**

📌 Two decisions from the human frame the whole plan:

> *"The emulator is dead and can be dropped altogether. But the DuckDB generator
> should still be a useful tool… We probably can move it into a standalone tool
> under Sylpheed."*

> *"xex2tractor was an early attempt and is long dead too… you may want to scan
> and compare the tools for anything that is in xex2tractor only."*

---

## Where things stand

| repo | server | local tree | tracked | disposition |
|---|---|---|---|---|
| **Sylpheed** | 598 MB | 33 GB | 131 MB / 1470 files | the target |
| **Xenia-Canary** | 71 MB | 20 GB | — | the keeper — ⚠️ **dirty** |
| Syplheed-Reborn | 103 MB | 1.4 GB | 100 MB / 787 files | ✅ absorbed → archive |
| Sylpheed-Godot | 0.4 MB | 99 MB | 0.3 MB / 39 files | ✅ absorbed → archive |
| xenia-rs | 9 MB | 1.2 GB | 61 MB | harvest the DB tool → archive |
| xex2tractor | 0.3 MB | *no clone* | — | harvest four things → archive |

`Sylpheed` has **two root commits** — `feat: initialise workspace` (2026-03-25,
Reborn's) and `scaffold the Godot port` (2026-08-28). Both histories were
grafted in, not copied, which is why the absorption checks below pass.

## The ledger — what was verified, and how

Nothing here is inference. Each row is a command that was run on 2026-09-13.

| claim | method | result |
|---|---|---|
| Reborn fully absorbed | all **21 remote + 8 local** branch tips vs `Sylpheed/main` | every one an ancestor |
| Sylpheed-Godot absorbed | both branch tips, + the root-commit graft | every one an ancestor |
| `agent-backups/` redundant | `git bundle list-heads`, 23 refs across 4 bundles | all present in Sylpheed |
| 3 unpushed `pi/*` branches redundant | `git patch-id --stable`, per commit | all 4 patches on `main` |
| nothing lives outside git | issues / PRs / releases / wikis, all six repos | **0** everywhere but Sylpheed |
| the DB tool is separable | `git grep "use xenia_cpu::"` in `xenia-analysis` | **three symbols** |

### 🔴 What the ledger found that a naive plan would have destroyed

1. **The F1 instrument exists in one place, uncommitted.** `xenia-canary` has two
   dirty files — `hid/file/file_input_driver.h` (`--pad_file_repeat`) and
   `app/xenia_main.cc` — plus **4 unpushed commits** on `sylpheed-re` and a
   branch, `audit-handle-lifecycle-probes`, **contained in no remote at all**.
   That patch is the instrument behind `REPEAT_DELAY`/`REPEAT_INTERVAL`, now
   shipped in the port. Its own finding page says *"not yet upstreamed."*
2. **xenia-rs has 4 unpushed branches** — `iterate-4B/ui-perf`, `4C/jit`,
   `4D/parallel`, `4E/native-threads`. Dropping the emulator is a decision;
   dropping work that was never backed up is a different act.
3. **The tool is not on `master`.** `xenia-rs/master` last moved **2026-06-15**;
   the DuckDB and disassembler work is on `origin/iterate-4A/apu-xma-stage1`
   (**2026-09-10**, +22 commits). Extract from the default branch and you get a
   three-month-old tool.

---

## Phase 0 — Secure the single-copy work

**Before touching anything else.** One `git checkout` in `xenia-canary` ends the
reproducibility of a measurement that is already live in the port.

1. Commit `file_input_driver.h` + `xenia_main.cc` on `sylpheed-re`.
2. Push `sylpheed-re` (4 + 1 commits) and `audit-handle-lifecycle-probes`.
3. ~~Resolve the `third_party/DirectXShaderCompiler` submodule drift.~~
   ✅ **There is none.** `git submodule status` shows no `+`: the checked-out
   commit matches the recorded one. The ` m` in `git status` is *nested*
   submodule dirt — DXC's own `external/SPIRV-Headers` and `SPIRV-Tools` — which
   is upstream's business and must not be committed here.

**Gate — and the first version of it was wrong.** It compared each local branch
to a remote **by SHA**, and reported seven safe branches as unpushed: a branch
that is an *ancestor* of a pushed branch is already preserved. A check that goes
red for something unactionable is the failure this repo keeps naming, so the
gate tests what it means — **reachability**:

```bash
git for-each-ref --format='%(refname:short)' refs/heads | while read b; do
  s=$(git rev-parse "$b"); found=""
  for r in $(git for-each-ref --format='%(refname:short)' refs/remotes); do
    git merge-base --is-ancestor "$s" "$r" 2>/dev/null && { found=$r; break; }
  done
  [ -n "$found" ] || echo "🔴 $b NOT REACHABLE from any remote"
done
```

✅ **PASSED 2026-09-13** — `8e63a9542` commits the instrument; `sylpheed-re` and
`audit-handle-lifecycle-probes` pushed; all **14** local branches reachable.

## Phase 1 — Make the archive honest

Push xenia-rs `iterate-4B/ui-perf`, `4C/jit`, `4D/parallel`, `4E/native-threads`.

They are being retired, not rescued. But a retired branch that was never pushed
is not retired — it is deleted, quietly, by someone who thought it was safe.

**Gate:** the same loop, run in every repo, prints nothing.

✅ **PASSED 2026-09-13** — all four pushed (32 / 52 / 56 / 59 commits ahead of
`master`). The loop is clean in all five repos **except** `pi/clippy`,
`pi/clippy-clean` and `pi/reauth3` in Sylpheed, which are unreachable by
ancestry and redundant by *patch-id* — the ledger's reason, and Phase 7's
disposal. The gate is right to flag them; they are the one case where the
answer lives outside it.

## Phase 2 — Harvest xex2tractor, then retire it

Four things exist only there.

| take | why |
|---|---|
| **XEX2 devkit + XEX1 retail master keys** | `xenia-xex` hardcodes retail only; devkit is present but `#[allow(dead_code)]` and XEX1 is absent. xex2tractor tries all three in a validating loop. |
| ~~**`extract -r`**~~ | 🔴 **RUN, AND THE INFERENCE WAS WRONG** — see the gate below. Optional, and dangerous if it is ever mistaken for the `.pe`. |
| **`doc/xex2_format.md`** (39 KB) and **`doc/xbox360_exports.json`** (938 KB, 2,913 exports across xboxkrnl / xam / xbdm) | byte-identical to untracked loose copies in the project root. Tracked by **no repo**. |
| **`LICENSE`** (MIT) | Sylpheed has none. |

Leave everything else: `xenia-xex` is ahead on `resources.rs` (XDBF/XACH),
`pdata.rs`, `tls.rs`, basic zero-fill compression, and the analysis layer.

**Gate — ✅ RUN 2026-09-13, and it corrected this page in two directions.**

```
extract      (xex2tractor, no -r)    ✅ IDENTICAL to the project .pe
extract -r   (xex2tractor)           ❌ differs — 1,903 bytes over 273 runs
extract      (xenia-rs, iterate-4A)  ✅ IDENTICAL — and .xex.json identical too
```

**Right:** the `.pe` *is* an xex2tractor extract. Hash-for-hash.

**Wrong:** it was made **without** `-r`. The project's `.pe` is the plain
decrypted-and-decompressed image, imports **unresolved** — and `-r`'s 1,903
differing bytes sit in the import thunk region from `VA 0x82000600`.

🔴 **So the dependency this gate existed to protect does not exist.** This page
claimed *"drop this and the file every `.pe`-offset citation depends on can
never be rebuilt."* It can: **`xenia-rs extract`, the tool Phase 3 keeps,
reproduces it byte-for-byte** — along with the loose `.xex.json`, which turns out
to have come from xenia-rs, not xex2tractor.

⚠️ **And `-r` must never become the `.pe`.** It writes into the image, so a `.pe`
built with it would silently invalidate every byte-offset citation in
`docs/re/`. If resolved-import disassembly is ever wanted, it is a **separate
artifact under a different name**, never a replacement.

That leaves xex2tractor's unique list at three items, none urgent — the two
master keys, the two doc assets, and the licence. **The doc assets are now the
only reason to visit that repository.**

📌 Learned in passing: `iterate-4A` **builds clean** (`xenia-app`, one unused
variable warning), so Phase 3's source is not bit-rotted — and it needs
`libavutil-dev` and friends via `ffmpeg-sys-next`, dragged in by the emulator's
XMA audio. The standalone tool sheds a C library the CI image did not have.

## Phase 3 — Lift the DB tool out of the emulator

From `origin/iterate-4A/apu-xma-stage1`, **never** from `master`.

```
xenia-analysis          10,658   db, vtables, jumptables, func, xref, rtti,
                                 strings, xdbf, static_init, demangle, sql_views
disasm slice of cpu      3,681   disasm.rs + decoder.rs + opcode.rs  (of 21,813)
xenia-xex                2,018   header, loader, lzx, pe, pdata, tls, resources
xenia-vfs                  444   so it reads an ISO, not just a loose .xex
xenia-types                361
                       ───────
                       ≈17,200   of ~71,000
```

The coupling is exactly three symbols — `decoder::decode`, `disasm::DisasmItem`,
`disasm::format`. Interpreter, JIT, scheduler, VMX, block cache, reservations:
none of it is reachable from the generator.

`xenia-xex` declares `xenia-memory` and `xenia-types` as dependencies and has
**zero `use`** of either. Probably droppable — confirm by compiling, do not
assume.

Travelling with it: `zq.py`, `tests/db_schema_golden.rs`,
`tests/disasm_goldens.rs`, `examples/decode_table_check.rs`.

**Two corrections to make during the move, not after:**

* the CLI help says **"SQLite database"** in two places. It is **DuckDB**
  (`duckdb = { workspace = true }`).
* `zq.py`'s documented escape hatch — *"the engine vtable / rdata is NOT in the
  DB … read it from guest memory with `xenia-rs exec --dump-addr`"* — **dies with
  the emulator.** Replace it with Canary's dump or a direct read of the `.pe` at
  `VA − 0x82000000`, and say which, in the file.

**Gate:** regenerate `sylpheed.db` from the ISO; diff schema and row counts
against the existing 586 MB database. `db_schema_golden` and `disasm_goldens`
green.

## Phase 4 — Prune the captures, and give `docs/re/` the check `docs/port/` has

**`docs/re/captures/` is 257 files / 117.6 MB, and it is not junk.** Measured:

```
193  (81.2 MB)  cited by a docs/re page, or opened by a tool
 64  (36.5 MB)  referenced by nothing
```

🔴 **Two are test fixtures and `cargo test` opens them:**

```
crates/sylpheed-formats/tests/ui_paint_order_disc.rs  → captures/title-screen-oracle.png
crates/sylpheed-formats/tests/unit_layout_disc.rs     → captures/stage02-live-unit-definitions-deep.txt
```

Sixteen more tools under `tools/port/` and `tools/re-capture/` read from there.

**Do:**

1. **Build `tools/re/check-capture-citations`**, the mirror of
   `tools/port/check-citations`, which has only ever checked `docs/port/*.md`.
   It reports two classes — a capture nothing cites, and a page citing a capture
   that was never committed. It needs a `--selftest` that plants one of each, or
   it is not a check.
2. **Fix the dangling citations first.** There are already **10**, including
   whole directories (`focus-ring/`, `hud-runtime/`, `title-builds/`). The
   corpus is already lossy in that direction and nothing was watching.
3. **Then drop the 64 orphans.**

⚠️ **An orphan is not automatically waste.** It may be evidence a page *should*
have cited, and the page is the thing that is wrong. Fix the citations before
deleting, or step 3 silently ratifies every omission in step 2.

📌 **Going forward the class that matters is not captures-vs-code:**

| | |
|---|---|
| **evidence** — a screenshot a finding points at | belongs beside the finding, as the convention says |
| **run output** — `.jsonl`, `.log`, `.csv` | the same class as xenia-rs's `audit-runs/`, which this plan drops |

The two `.jsonl` mission-state dumps alone are 4.3 MB, and one is an orphan.

## Phase 5 — Verify the invariant, then archive

For each of `Syplheed-Reborn`, `Sylpheed-Godot`, `xenia-rs`, `xex2tractor`:
re-run the containment check **at that moment** — not from this page's ledger,
which will be days old — then archive read-only on Gitea, and only then delete
the local clone.

**Archive before delete, and leave them archived.** The check proves the commits
are reachable. It does not prove nobody will want the repository boundary back.

### ⚠️ The fork: history rewrite, or not

Dropping files in Phase 4 **reclaims no space**. The 118 MB is in history — 512
distinct blobs, 120.5 MB — and `git rm` leaves every byte in `.git` (590 MB
local, 598 MB on the server). Only a rewrite reclaims it, and it has a hard
blocker:

```toml
# crates/sylpheed-export/Cargo.toml:84
sylpheed-formats = { git = "…/Sylpheed.git", tag = "formats-pin-2026-09-01" }
```

**The build depends on this repo by tag**, pinned in `Cargo.lock` to `#1cd5b8b1`.
A rewrite invalidates all 8 `formats-pin-*` tags, every commit SHA, PR #23's
merge, both agents' clones, and needs a force-push through branch protection.

* **Don't rewrite** (recommended): 118 MB on a host with 154 GB free, where
  `Sylpheed/target` alone is 32 GB. The cost is not worth the blast radius.
* **Do rewrite:** then it happens **here, in Phase 5** — while tags are being
  re-cut and agents re-cloned anyway, so the disruption is paid once instead of
  twice. Re-point the `formats-pin` dependency **first**.

This is a decision, not a default. Whichever is chosen, record it here.

## Phase 6 — Adopt the orphan assets

Tracked by nobody today, and tools or references by the two-repo rule:

| asset | → |
|---|---|
| `ppc-manual/` — 397 files, **including `generator/generate_manual.py`** | Sylpheed |
| `docs/ppc_instructions.{json,md}`, `docs/xbox360_exports.*`, `docs/xex2_format.md` | Sylpheed `docs/` |
| `docs/CROSS_BUILD_SETUP.md` (58 KB) | **Canary** — it builds Canary |
| `run-canary.sh` | Sylpheed `tools/` |
| `ai-agent-*.md` ×4 (56 KB) | Sylpheed `docs/`, or discard — human's call |
| `scratch/xbg7/extract.py` — clean-room XBG7 scanner | superseded by `sylpheed_formats::mesh`; keep as history or drop |
| `Sylpheed-try/…/examples/_voice_span.rs` — 46-line untracked probe | keep or drop |

## Phase 7 — Intended drops

Stated so that nothing is dropped by omission.

| drop | size | why it is safe |
|---|---|---|
| `agent-backups/` | 687 MB | all 23 refs verified present in Sylpheed |
| `stock-oracle/` | 34 MB | two Canary **binaries**, rebuildable from source |
| root `canary_*.log` / `.stdout` / `.stderr` | ~15 MB | run output |
| `vkd3d-proton.cache` | 15 KB | runtime cache |
| xenia-rs `audit-runs/` | 57 MB, 41 files | already `.gitignore`d; `iterate-4A` deleted 40 of 41 |
| local `pi/clippy`, `pi/clippy-clean`, `pi/reauth3` | — | all 4 patches on `main`, by patch-id |
| `Sylpheed/target/` | **32 GB** | build output; the host is at 83 % |
| `texcompare/` (14 MB), `ship_render/` (64 KB) | — | tool output — **confirm no `docs/re/` page cites them first** |
| **the xenia-rs emulator** | ~54k LOC | the human's decision: superseded by Canary |

## Still open, for the human

* **Retention.** The `.iso` (7.6 GB), `.pe`, `.xex.json` and `sylph_extract/`
  (6.2 GB) stay on disk and out of git. But `docs/re/captures/` is 118 MB of
  game screenshots **inside a public repository**, about to be joined by a
  938 KB export database, and **all six repos are `private=False`**. Worth
  choosing rather than inheriting.
* **The history fork** in Phase 5.
* **`ai-agent-*.md`, `scratch/`, `_voice_span.rs`** — keep or drop.