Files
Syplheed-Reborn/docs/re/challenge-mission-gate.md
Claude (auto-RE) 023bb71cfd re(challenge): the cleared-stage mask is CONFIRMED on the running game
Booted the title and read the two gate words live:

    0x828F40C0 = 0x00000002     word A
    0x828F4814 = 0x00000000     word B

Word A = 2 = bit 1. The profile's save is Stage 02 "At Standby" -- stage 01
cleared -- so the mask is exactly one bit, at the index of the one cleared
stage, 1-BASED. Reproduced across two cold boots. That confirms against a known
progress state, on the real game:

  - the singleton is the static object at 0x828F4070, as derived statically;
  - word A is a cleared-stage bitmask (not achievements, not a stage number);
  - bit index = stage id, 1-based, so TimeAttack's REQUIREMENT 16 means "clear
    stage 16" -- the last story mission;
  - word B is the challenge half and is 0 on a story-only profile.

New tools: gpoke.py (live guest-memory WRITE, companion to gmem.py, prints
before/after for every word), pad.py (drives the new --hid=file pad; replaces
vgamepad, which leaked to the host through /dev/uinput), challenge_probe.sh
(one blocking session: boot, wait for title, drive in, poke, screenshot).

Poking both words did NOT surface a challenge entry in EXTRAS -- and that menu
was built 26 s after the poke, so it is not staleness. Entering MISSION SELECT
then failed, but the log names the real cause and it is not the gate:
MmAllocatePhysicalMemoryEx could not satisfy a 128 MB request (parent free
30633/131072 pages), the guest threw a C++ exception, and Xenia surfaced its
generic "Disc Read Error". It is preceded by "BaseHeap::Release failed because
address is not a region start" -- a failed release leaking the range. Recorded
as an emulator heap problem, with the control run (same navigation, no poke)
named as the next step.
2026-08-13 20:28:19 +00:00

387 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Challenge / EX missions — the stage set, the GamePart graph, and the kind field
**Status:** ✅ for the static structure (stage set, GamePart ids, the config-section
switch) and for the **unlock mechanism** — a bit test against a *cleared-stage* mask,
now **read live off the running game and matching the profile's progress exactly**
(§5.6). 🟡 for the stage-field crash mechanism, for word B's writer, and for which
mission consumes which bit; ❔ how the challenge menu is actually entered.
**Method:** static analysis (`.pe` strings/pointers + DuckDB disassembly + disc
records), then confirmed on the running title via the container-safe
[`--hid=file` pad](../../../xenia-canary-native/src/xenia/hid/file/file_input_driver.h)
and a live guest-memory read.
**Evidence:** [`captures/challenge-map.txt`](captures/challenge-map.txt),
[`captures/challenge-screen-config.txt`](captures/challenge-screen-config.txt),
[`structures/achievements.md`](structures/achievements.md);
`examples/challenge_map.rs`, `examples/challenge_screen.rs`.
## Why this was worth doing
The Route-B unit harvest is complete for the **story** campaign (68 of 110 units,
7 115 defaulted-on-disc values) and stalled there: the remaining units are `*_EX4` /
`*_EX5` / `*EX` variants that only the challenge missions field. The save's stage
field addresses those stage records — but setting it to **27** boots to the title and
then the emulator **exits during the load**, while 116 all load normally. This note
answers *what the challenge missions are on disc* and *why the stage field alone is
not enough*, so the next runtime attempt is targeted rather than another probe sweep.
## 1. The disc has exactly 29 stage records, in three families ✅
`StageResource` (schema `0x3c9ae32e`) in `dat/GP_MAIN_GAME_E.pak`:
| ids | count | `BackGroundID` | reading |
|---|---|---|---|
| `S01``S16` | 16 | Lebendorf … PD (real locations) | the **story** campaign |
| `S18``S23` | 6 | `Original` (all six) | the **tutorial** missions |
| `S24``S29` | 6 | Anastasis, Hargenteen ×2, Planet_Lebendorf, Lebendorf, Earth | the **challenge** missions |
| `Test` | 1 | Earth | developer stage |
`S17` does not exist. This **matches `weapon.tbl`'s sprite-key set exactly**
`stage01…16` + `tutorial01…06` + `challenge01…06` (recorded in
[static screen config](#)) — which is an independent confirmation of the three-way
split: 16 + 6 + 6 + Test = 29.
The `Original` six carry ~43 tokens each while the story and challenge records carry
5159, consistent with tutorials having no squadron/route/formation tables.
## 2. `GP_CHALLENGE.pak` is a screen, not mission data ✅
151 entries, **0 IDXD objects** — sprites and layout only. So challenge missions are
*not* a separate data set: they are ordinary `StageResource` records in
`GP_MAIN_GAME_E.pak`, reached through a different menu. That is why the EX unit
rosters show up in the same `EnumUnit_S<NN>` tables the story stages use.
## 3. The GamePart id table — the game's whole screen graph ✅
`.rdata` pointer array at **`0x820A1630`**, 29 entries, index = GamePart id:
```
0 GP_TITLE 10 GP_BUNK 20 GP_STAGE_CLEAR
1 GP_ADVERTISE_DEMO 11 GP_READY_ROOM 21 GP_MISSION_LOG
2 GP_SELECT_STORAGE 12 GP_HANGAR 22 GP_GAMEOVER
3 GP_LOAD 13 GP_ARSENAL 23 GP_DEBRIEFING
4 GP_SAVE 14 GP_PILOT_LOG 24 GP_DIALOG
5 GP_EXTRAS 15 GP_SYSTEM 25 GP_TUTORIAL
6 GP_MOVIE_THEATER 16 GP_DEMO *26 GP_CHALLENGE
7 GP_MISSION_SELECT 17 GP_MAIN_GAME 27 GP_LEADERBOARD
8 GP_OPTIONS 18 GP_SELECTOR 28 GP_TEST
9 GP_MOVIE 19 GP_PAUSE_MENU
```
The indices are **not inferred from position** — the image carries the factory
registration text, e.g.
`silph::GamePartTask::RegisterToFactory<26, class silph::GamePart_ChallengeMission>::RegisterToFactory is failed!`
and `<10, … GamePart_Bunk>`, both of which agree with this table. So `GP_CHALLENGE`
is GamePart **26** and `GP_TUTORIAL` is **25**.
## 4. A mission-**kind** field selects the stage config section ✅
Immediately before the array above, `0x820A1600``0x820A162C` holds the stage-config
key list: `BASE_INFO`, `StageResource`, `Background`, `PlayerUnit`,
`EQUIIP_LIMITATION` *(sic)*, `MODEL_PATH`, `NEW_ITEM`, `LOADING`, `CHALLENGE`,
`EXTRA`, `FILE`.
The stage loader picks **one of the last three** by a field at **`object + 144`**, and
the same three-way switch appears at two independent sites (`0x82184df0` inside the
all-keys config reader, and `0x82185ed0` in `sub_82185E80`):
```asm
lwz r11, 144(r30)
cmpi r11, 3 ; == 3 -> "EXTRA" (0x820A2160)
beq extra
addi r11, r11, -5
cmpli r11, 1 ; == 5 or 6 -> "CHALLENGE" (0x820A2154)
bgt file ; otherwise -> "FILE" (0x820A2168)
```
Two further sites (`0x82186a60`, `0x82186cb8`) classify the same field as
`{3, 5, 6}` versus everything else — i.e. **EXTRA and CHALLENGE together are "not an
ordinary story load"**. The field is initialised to **0** in the constructor
(`sub_821783D8`, `0x8217851c`, alongside `+148 = 0` — the stage index — and
`+132 = 2`, `+136/+140 = -1`), and every write to it inside this class
(`0x82184b10`, `0x82185d48`, `0x82186ab4`) only *clears* it. Nothing in the image
stores an immediate 3/5/6 into it, so the kind is **supplied from outside the class**
— by whichever GamePart launches the mission — not derived from the stage number.
### 4.1 Why the stage-field probe crashes 🟡
That gives a mechanism for the observed failure: patching the save's stage field to
27 selects the `S27` **record** while the kind stays **0**, so the loader reads the
`FILE` section for a stage whose config lives under `CHALLENGE`. A missing section
then propagates into the load, and the title exits. It fits the evidence (116 fine,
every 2429 the same failure, `/dev/shm` empty and disk free, so not the resource
trap) but it is **not proven** — proving it needs a run.
**Cheap untried discriminator, no new tooling:** patch the stage field to **1823**
(the tutorials). If those also die, the failure tracks "record outside the story
range", and the kind field is the likely gate. It has never been tested — only
116 and 2429 were.
## 5. The challenge screen, its six missions, and the unlock gate ✅
The factory registration sites (`0x8280C000``0x8280F800`) give **id → creator** for
22 of the 24 registered parts, and the creators bound each class's code block. So
`GamePart_ChallengeMission`'s methods are **`0x82187E60``0x8218CF10`** (between
`GP_BUNK`'s creator `0x82187E38` and its own `0x8218CF10`). Resolving every string
that range references gives the screen's whole config schema:
```
MISSIONS · MISSION_ID · RECORD_TYPE · REQUIREMENT · REQUIREMENT_DESC
NORMAL_BUTTON · GRAY_BUTTON · THUMBNAIL · STAGE_DESC · TEXT_STAGE
TEXT_RECORD · NEW_STAGE · "Always" · "Time" · BASE_INFO
```
**The record itself is on disc**, in `tables.pak` (schema `54a10697`, one copy per
language — English is entry #64), *not* in `GP_CHALLENGE.pak`. It names six missions:
| slot | `MISSION_ID` | buttons / thumbnail | record |
|---|---|---|---|
| 1 | `TimeAttack` | `Button_TimeAttack{,_Gray}`, `Thumbnail_TimeAttack` | `Time` |
| 2 | `ScoreAttack` | `Button_ScoreAttack{,_Gray}` | `Points` |
| 36 | `Extra01``Extra04` | `Button_Extra0N{,_Gray}` | — |
`GRAY_BUTTON` is the locked art, `NORMAL_BUTTON` the unlocked art — so the screen
always shows all six and greys out what you have not earned. Full dump:
[`captures/challenge-screen-config.txt`](captures/challenge-screen-config.txt).
### 5.1 The gate is a bit test against a progress bitfield ✅
`0x82189870` builds the list, and `0x82189970``0x821899D8` is the availability
decision for each mission:
```asm
lookup REQUIREMENT in the mission record ; bl 0x82448C50
absent -> AVAILABLE
== "Always" -> AVAILABLE ; strcmp, bl 0x825EDD20
else n = atoi(v) ; bl 0x825EDCD0
n == 0 -> LOCKED
n < 24 -> bit n of word A
n >= 24 -> bit (n-24) of word B
bit set ? AVAILABLE : LOCKED
```
Word A and word B are read from a **singleton** (`0x821707C0`; the object pointer
lives at the global `0x828F48B0`, with a construct-once flag at `0x828F48BC`) at
offsets **`+80`** and **`+1956`**. So challenge
availability is one bit in a progress bitfield, and `REQUIREMENT` is that bit's index
**not** a stage number, a score, or a difficulty.
### 5.2 The bit space is CLEARED STAGES ✅
Word A has exactly **one** writer in the image, and finding it settles the question.
Scanning all 22 callers of the `+80` struct copier (`0x82175110`) for one that stores
to the copy's **word 0** gives a single hit, `0x821C1820`, inside
**`GamePart_StageClear`** (`0x821C09D8``0x821C29F0`):
```asm
if (this+1004 & 0x20000) skip ; already recorded
this+80 = 1
copy local = singleton->progress ; bl 0x82175110, src = obj+80
x = this+84
local.word0 |= 1 << x ; slw r10, r26, r10
if changed: singleton->set(local) ; bl 0x8216FF70 (assign + async persist)
```
**`this+84` is the stage number**, and two independent uses prove it:
- `0x821C1760` indexes a **20-byte record array** with it — `this + (x+7)*20`, whose
records are written as three words plus an 8-byte timestamp, i.e. the shape of the
savegame's 16×20-byte `SHAB` table;
- `0x821C1EEC` and `0x821C1924` pass it as the **index into the config key `STAGE`**
(`0x820A2540`) — the debriefing config's `px_deb_stage01…16` sprite list.
So **word A is a "stage cleared" bitmask**, and a challenge mission's `REQUIREMENT n`
means **"stage `n` has been cleared"**. The `< 24` / `>= 24` split then lines up with
the disc's own stage numbering from §1: story `1``16` and tutorial `18``23` sit in
word A, and the challenge stages `24``29` are exactly word B's bits `0``5`.
> ⚠️ **Third revision of this claim — the first two were wrong, and how they went
> wrong is worth keeping.** An earlier pass read the split at 24 as "the game's 24
> achievements" because `ACHIEVEMENTS_REQUIREMENTS` has exactly 24 entries. That is a
> **coincidence**: the achievement count and the first challenge stage id are both 24
> for unrelated reasons. The achievement work itself stands — the `XACH` table, the
> 1000G self-check, and the `XACHIEVEMENT_DETAILS`/XAM enumeration are all solid, and
> are documented in [structures/achievements.md](structures/achievements.md) — it just
> **does not gate the challenge missions**. The lesson: a numeric coincidence is not a
> join; find the writer.
### 5.3 What the six requirement values are ✅/🟡
The record's numeric tokens are `16`, `25`, `26`, `27`, `29`, with `24`/`28` already
in the pool earlier (they double as font metrics) and so **deduped away** if used.
IDXD dedup makes positional key/value pairing unsound in general — but `16` sits
*immediately* before `REQUIREMENT` (tokens 96 → 97), the documented value-before-key
adjacency, so the first mission's value is well-supported.
**`TimeAttack` requires stage 16 cleared** — the final story mission. That is the
natural gate, and it is what the *first* reading of these numbers suggested before the
achievement detour talked me out of it.
The other five values are `≥ 24`, i.e. **challenge stages 2429**: the challenge
missions chain off each other. 🟡 on the exact pairing (which mission needs which),
which needs the record's binary index section rather than the string pool.
🟡 **Word B has no known writer.** `GamePart_StageClear` sets `1 << x` into word A
unconditionally, which for a challenge stage (`x ≥ 24`) would land on word A bits
2429, *not* word B — so clearing a challenge mission must be recorded by a different
path, presumably one that also stores its `Time`/`Points` record. Not yet found.
**Negative worth keeping:** the requirement *text* (`TimeAttackRequirement`,
`Extra01Requirement`, …) is **not** in `GP_CHALLENGE.pak` — building a `TextIndex`
over it yields **0 entries** and its only wordy payload is embedded font copyright.
The config's `PATH` is `dat\GP_CHALLENGE.pak+eng\`, a per-language branch, so those
keys resolve through a naming scheme the current loader does not reproduce. Reading
them would say in plain English what each mission asks for — worth one more attempt
via `hash::TOC_NAME_SCHEMES`.
### 5.4 Where the mask lives, and one lead refuted
### 5.5 Both gate words are one record — and it is **not** the savegame ✅
`0x82175110` copies the record at singleton `+80` in full: two words, an 8-byte pair, a
184-byte `memcpy`, 8 words from `+200`, then **816 bytes at `+232`, 816 more at
`+1048`**, a sub-object at `+1864`, and a final word at **`+1876`**. So the record runs
`+0 … ~+1880`, i.e. singleton `+80 … +1960` — which means
- **word A** is record `+0` (singleton `+80`), and
- **word B is record `+1876`** (singleton `+1956`).
That answers §5.3's "word B has no known writer": there is no separate `stw` because
the whole record is copied out, modified and assigned back as a unit
(`0x8216FF70` → compare `0x822C3708`, assign `0x82170650`, then spawn a worker
`0x821700A8` under a lock which retries a commit `0x822C33B8` up to five times).
`GamePart_Debriefing` uses the same get/modify/set to accumulate saturating career
counters at `+200``+224` (the last 64-bit) — the quantities the achievement
requirements test.
**And the record is not what `savedata` holds.** Two independent checks:
- **Size.** Every real save on disk is a **276-byte** container deflating to a
**545-byte** payload. The progress record is **~1 880 bytes**. It does not fit.
- **Files.** After many sessions the game's content tree holds only
`game0N/savedata` + `game0N/__thumbnail.png` per slot and the three
`Headers/*.header` — no second data file anywhere.
Also checked and negative: the savegame object is `*(*(this+4)) + 304`, and the
singleton's holder address (`0x828F48B0`) is referenced **nowhere but inside the
accessor itself**, so `this+4` is a different holder — the save block is not a window
into this record.
⚠️ **So hand-editing a save cannot unlock the challenge missions.** That kills the
operational hope this section previously carried; the earlier savegame-editing win
does not extend here.
### 5.6 CONFIRMED ON THE RUNNING GAME ✅
Read live from a booted title (`tools/re-capture/gpoke.py r32`):
```
0x828F40C0 = 0x00000002 word A
0x828F4814 = 0x00000000 word B
```
**Word A = 2 = bit 1 set.** The profile's save is *Stage 02, "At Standby"* — i.e.
**stage 01 cleared** — so the mask is exactly one bit, at the index of the one
cleared stage, **1-based**. Reproduced on two separate cold boots. That confirms, on
the running game and against a known progress state:
- the singleton really is the static object at `0x828F4070`;
- word A is a **cleared-stage bitmask**, not achievements, not a stage number;
- bit index = **stage id, 1-based** — so `TimeAttack`'s `REQUIREMENT 16` is
"clear stage 16", the last story mission;
- word B is the challenge half and is `0` on a story-only profile, as expected.
Both words were then poked (`0xFFFFFFFF` / `0x3F`) and read back OK.
**What the poke did not do (yet):** `EXTRAS` still shows only `MISSION SELECT /
MOVIE THEATER / BACK` — no challenge entry — although the menu was built 26 s
*after* the poke, so this is not staleness. Entering `MISSION SELECT` then failed,
and the log gives the real reason: **`MmAllocatePhysicalMemoryEx` could not satisfy
a 128 MB request** (`parent free 30633/131072 pages`), the guest threw a C++
exception, and Xenia surfaced it as its generic *"Disc Read Error"* dialog. It is
preceded by `BaseHeap::Release failed because address is not a region start`, a
failed release that leaks the range. So that is an emulator/heap problem on the way
into the screen, **not** evidence about the gate. Open: repeat without the poke to
see whether `MISSION SELECT` fails the same way regardless.
### 5.7 What *would* work — static addresses for a live write
The singleton is a **static object at `0x828F4070`** (`0x8216F650`:
`addis 0x828F` + `addi …, 16496`), with the holder at `0x828F48B0` pointing at it. So
the two gate words are at fixed guest addresses, no scanning required:
| word | guest VA | meaning |
|---|---|---|
| A | **`0x828F40C0`** | cleared stages, bit = stage id (`< 24`) |
| B | **`0x828F4814`** | cleared stages, bit = stage id 24 (challenge 2429) |
Canary maps guest RAM into `/dev/shm`, which the project already reads live
([`tools/re-capture/gmem.py`](../../tools/re-capture/gmem.py)). Writing
`0xFFFF` into word A and `0x3F` into word B while the title sits on a menu should
open all six challenge missions **without playing the campaign** — and that is the
route to the last 42 `*_EX4`/`*_EX5` units. Untested: it needs a run, and the run
needs gamepad input to reach the menu.
**The obvious lead is dead.** The three `stw`s to `+1956` in `0x822AF278` /
`sub_822C8748` looked promising because they sit in the save serializer's code region
— they are on a **different object**. That one is fetched through `0x822CEB30`, has a
`+2652` flag the code checks first, and stores **string pointers** at `+1956`/`+2024`
(built by `0x822D35F8`); a pointer `AND`ed with `1 << n` would be meaningless as a
gate. So **nothing in the image stores to this singleton's `+1956` field-wise**, which
means it is filled by a bulk copy or by a path not yet found.
Two candidates remain and they need different levers: the **save** (hand-write it) or
the **Xbox profile** (the emulator's profile data). Note that XEX imports are resolved
**by ordinal**, so the absence of `XamUser*` name strings in the `.pe` is not evidence
against the profile route. `+80` at least is handled as a small struct by address
(`addi r4, obj, 80` → copy helper `0x82175110`, written back via `0x8216FF70`), which
is what a serialised value object looks like.
## 6. The unlock condition, from the strings ❔
Three strings say challenge missions are *announced*, not menu-browsed:
- `AVSCRIPT_COMMAND_ATTAINMENT_CHALLENGE_MISSION_CARGO_SCORE` — a mission-script
command that grants challenge-mission attainment from a **cargo score**;
- `DLG_CHALLENGE_MISSION_AVAILABLE` — the "a challenge mission is now available"
dialog;
- `DLG_GO_CHALLENGE_MISSION_MENU` — the prompt that takes you to GamePart 26.
So the gate is plausibly an in-mission achievement, not a title-menu state — which is
consistent with `Game Status = GAME_CLEAR` unlocking nothing (measured, refuted).
Neither dialog key is reachable by xref: they are selected **by index** through a
config lookup, exactly like the save screen's sprite keys, so there is nothing to
grep back to. Naming the flag needs the challenge-availability check disassembled
from the screen side.
## 7. What this makes actionable
`roster_target` against the harvested CSV, with the stage families now named:
| stage | family | roster units not yet read |
|---|---|---|
| `S27` | challenge | `f003_ArrowHead_EX4`, `e107_AAFrigate_EX4`, `e010_Attacker_S_EX4`, `e011_Attacker_B_EX4`, `e007_Turret_EX4`, `e008_TurretPlus_EX4` |
| `S28` | challenge | `f004_DeltaSaber_A_Player`, `f001_DeltaSaber_T_EX5(_el)`, `f003_ArrowHead_EX5`, `f104_Battleship_EX5`, `f105_Cruiser_EX5` |
| `S25` | challenge | `e101_SDBattleshipEX`, `e102_BattleshipEX`, `e105_CruiserEX`, `e108_ASFrigateEX` |
| `S29` | challenge | `e101_SDBattleshipEX`, `e102_BattleshipEX`, `e105_CruiserEX` |
| `S24` | challenge | `e105_CruiserEX`, `e106_DestroyerEX` |
| `S10` | **story** | `e005_ADAN_ElanTypeQ_Margras` |
⚠️ **The `S10` row contradicts "the story campaign is complete."** One story stage
still fields a unit that has never been read. Either `S10` was never flown (it is the
smallest stage container on the disc — 7 XBG7 resources — so it may be a cutscene
stage that is not flyable), or the completeness claim was one unit optimistic. Flagged
rather than resolved: it costs one ordinary story run to settle.
Remaining coverage: 42 units missing, of which the rosters above account for ~23. The
rest are not in any stage roster table and need a different lever.
## Reproduce
```bash
cargo run --release -q -p sylpheed-formats --example challenge_map -- <disc-root>
cargo run --release -q -p sylpheed-formats --example challenge_screen -- <disc-root>
python3 xenia-rs/zq.py dis 0x82184df0 0x82184e30 # the three-way section switch
python3 xenia-rs/zq.py dis 0x82185ed0 0x82185f10 # the same switch, second site
python3 xenia-rs/zq.py dis 0x82189860 0x821899f0 # the challenge availability gate
```