Syplheed-Reborn/docs/re/challenge-mission-gate.md

Challenge / EX missions — the stage set, the GamePart graph, and the kind field

Status: ✅ for the static structure (stage set, GamePart ids, the config-section switch) and for the unlock mechanism — a bit test against a cleared-stage mask, now read live off the running game and matching the profile's progress exactly (§5.6). 🟡 for the stage-field crash mechanism, for word B's writer, and for which mission consumes which bit; ❔ how the challenge menu is actually entered. Method: static analysis (.pe strings/pointers + DuckDB disassembly + disc records), then confirmed on the running title via the container-safe --hid=file pad and a live guest-memory read. Evidence: captures/challenge-map.txt, captures/challenge-screen-config.txt, structures/achievements.md; examples/challenge_map.rs, examples/challenge_screen.rs.

Why this was worth doing

The Route-B unit harvest is complete for the story campaign (68 of 110 units, 7 115 defaulted-on-disc values) and stalled there: the remaining units are *_EX4 / *_EX5 / *EX variants that only the challenge missions field. The save's stage field addresses those stage records — but setting it to 27 boots to the title and then the emulator exits during the load, while 1–16 all load normally. This note answers what the challenge missions are on disc and why the stage field alone is not enough, so the next runtime attempt is targeted rather than another probe sweep.

1. The disc has exactly 29 stage records, in three families ✅

StageResource (schema 0x3c9ae32e) in dat/GP_MAIN_GAME_E.pak:

ids	count	BackGroundID	reading
S01–S16	16	Lebendorf … PD (real locations)	the story campaign
S18–S23	6	Original (all six)	the tutorial missions
S24–S29	6	Anastasis, Hargenteen ×2, Planet_Lebendorf, Lebendorf, Earth	the challenge missions
Test	1	Earth	developer stage

S17 does not exist. This matches weapon.tbl's sprite-key set exactly — stage01…16 + tutorial01…06 + challenge01…06 (recorded in static screen config) — which is an independent confirmation of the three-way split: 16 + 6 + 6 + Test = 29.

The Original six carry ~43 tokens each while the story and challenge records carry 51–59, consistent with tutorials having no squadron/route/formation tables.

2. GP_CHALLENGE.pak is a screen, not mission data ✅

151 entries, 0 IDXD objects — sprites and layout only. So challenge missions are not a separate data set: they are ordinary StageResource records in GP_MAIN_GAME_E.pak, reached through a different menu. That is why the EX unit rosters show up in the same EnumUnit_S<NN> tables the story stages use.

3. The GamePart id table — the game's whole screen graph ✅

.rdata pointer array at 0x820A1630, 29 entries, index = GamePart id:

 0 GP_TITLE            10 GP_BUNK             20 GP_STAGE_CLEAR
 1 GP_ADVERTISE_DEMO   11 GP_READY_ROOM       21 GP_MISSION_LOG
 2 GP_SELECT_STORAGE   12 GP_HANGAR           22 GP_GAMEOVER
 3 GP_LOAD             13 GP_ARSENAL          23 GP_DEBRIEFING
 4 GP_SAVE             14 GP_PILOT_LOG        24 GP_DIALOG
 5 GP_EXTRAS           15 GP_SYSTEM           25 GP_TUTORIAL
 6 GP_MOVIE_THEATER    16 GP_DEMO            *26 GP_CHALLENGE
 7 GP_MISSION_SELECT   17 GP_MAIN_GAME        27 GP_LEADERBOARD
 8 GP_OPTIONS          18 GP_SELECTOR         28 GP_TEST
 9 GP_MOVIE            19 GP_PAUSE_MENU

The indices are not inferred from position — the image carries the factory registration text, e.g. silph::GamePartTask::RegisterToFactory<26, class silph::GamePart_ChallengeMission>::RegisterToFactory is failed! and <10, … GamePart_Bunk>, both of which agree with this table. So GP_CHALLENGE is GamePart 26 and GP_TUTORIAL is 25.

4. A mission-kind field selects the stage config section ✅

Immediately before the array above, 0x820A1600–0x820A162C holds the stage-config key list: BASE_INFO, StageResource, Background, PlayerUnit, EQUIIP_LIMITATION (sic), MODEL_PATH, NEW_ITEM, LOADING, CHALLENGE, EXTRA, FILE.

The stage loader picks one of the last three by a field at object + 144, and the same three-way switch appears at two independent sites (0x82184df0 inside the all-keys config reader, and 0x82185ed0 in sub_82185E80):

lwz  r11, 144(r30)
cmpi r11, 3          ; == 3          -> "EXTRA"     (0x820A2160)
beq  extra
addi r11, r11, -5
cmpli r11, 1         ; == 5 or 6     -> "CHALLENGE" (0x820A2154)
bgt  file            ; otherwise     -> "FILE"      (0x820A2168)

Two further sites (0x82186a60, 0x82186cb8) classify the same field as {3, 5, 6} versus everything else — i.e. EXTRA and CHALLENGE together are "not an ordinary story load". Every write to +144 inside this class (0x82184b10, 0x82185d48, 0x82186ab4) only clears it, and nothing in the image stores an immediate 3/5/6 into it, so the kind is supplied from outside the class — by whichever GamePart launches the mission — not derived from the stage number.

⚠️ Withdrawn: this section used to add "the field is initialised to 0 in the constructor sub_821783D8, alongside +148 = 0, +132 = 2, +136/+140 = -1". That linked two code regions on nothing more than both touching +144/+148, and a snapshot refutes it: sub_821783D8 initialises the static at 0x828F3EC0 (its first act is InitializeCriticalSection(obj, 256)), and in an in-flight snapshot that object's +144 holds 0x000B1C8B and +592 a float — not a kind and not flags. So the object owning the kind field is unidentified. Scanning the snapshot for it (kind ∈ {0,3,5,6} at +144, stage at +148, pointers at +0/+52/+604) returns only matches inside the executable's own static data — 10 at +148 is far too common to discriminate. The switch itself stands: it is direct disassembly at two sites. Only the constructor attribution was wrong.

Confirmed on screen instead: launching a story stage from MISSION SELECT reaches a READY ROOM carrying an "EXTRA" watermark (capture) — the EXTRA config section, i.e. kind 3, visible in the UI. That is independent evidence the kind field means what §4 says, even though its owning object is not pinned.

4.1 Why the stage-field probe crashes 🟡

That gives a mechanism for the observed failure: patching the save's stage field to 27 selects the S27 record while the kind stays 0, so the loader reads the FILE section for a stage whose config lives under CHALLENGE. A missing section then propagates into the load, and the title exits. It fits the evidence (1–16 fine, every 24–29 the same failure, /dev/shm empty and disk free, so not the resource trap) but it is not proven — proving it needs a run.

Cheap untried discriminator, no new tooling: patch the stage field to 18–23 (the tutorials). If those also die, the failure tracks "record outside the story range", and the kind field is the likely gate. It has never been tested — only 1–16 and 24–29 were.

5. The challenge screen, its six missions, and the unlock gate ✅

The factory registration sites (0x8280C000–0x8280F800) give id → creator for 22 of the 24 registered parts, and the creators bound each class's code block. So GamePart_ChallengeMission's methods are 0x82187E60–0x8218CF10 (between GP_BUNK's creator 0x82187E38 and its own 0x8218CF10). Resolving every string that range references gives the screen's whole config schema:

MISSIONS · MISSION_ID · RECORD_TYPE · REQUIREMENT · REQUIREMENT_DESC
NORMAL_BUTTON · GRAY_BUTTON · THUMBNAIL · STAGE_DESC · TEXT_STAGE
TEXT_RECORD · NEW_STAGE · "Always" · "Time" · BASE_INFO

The record itself is on disc, in tables.pak (schema 54a10697, one copy per language — English is entry #64), not in GP_CHALLENGE.pak. It names six missions:

slot	MISSION_ID	buttons / thumbnail	record
1	TimeAttack	Button_TimeAttack{,_Gray}, Thumbnail_TimeAttack	Time
2	ScoreAttack	Button_ScoreAttack{,_Gray}	Points
3–6	Extra01…Extra04	Button_Extra0N{,_Gray}	—

GRAY_BUTTON is the locked art, NORMAL_BUTTON the unlocked art — so the screen always shows all six and greys out what you have not earned. Full dump: captures/challenge-screen-config.txt.

5.1 The gate is a bit test against a progress bitfield ✅

0x82189870 builds the list, and 0x82189970–0x821899D8 is the availability decision for each mission:

lookup REQUIREMENT in the mission record        ; bl 0x82448C50
  absent            -> AVAILABLE
  == "Always"       -> AVAILABLE                ; strcmp, bl 0x825EDD20
  else n = atoi(v)                              ; bl 0x825EDCD0
     n == 0         -> LOCKED
     n <  24        -> bit n      of word A
     n >= 24        -> bit (n-24) of word B
     bit set ? AVAILABLE : LOCKED

Word A and word B are read from a singleton (0x821707C0; the object pointer lives at the global 0x828F48B0, with a construct-once flag at 0x828F48BC) at offsets +80 and +1956. So challenge availability is one bit in a progress bitfield, and REQUIREMENT is that bit's index — not a stage number, a score, or a difficulty.

5.2 The bit space is CLEARED STAGES ✅

Word A has exactly one writer in the image, and finding it settles the question. Scanning all 22 callers of the +80 struct copier (0x82175110) for one that stores to the copy's word 0 gives a single hit, 0x821C1820, inside GamePart_StageClear (0x821C09D8–0x821C29F0):

if (this+1004 & 0x20000) skip            ; already recorded
this+80 = 1
copy   local = singleton->progress        ; bl 0x82175110, src = obj+80
x      = this+84
local.word0 |= 1 << x                     ; slw r10, r26, r10
if changed: singleton->set(local)         ; bl 0x8216FF70 (assign + async persist)

this+84 is the stage number, and two independent uses prove it:

• 0x821C1760 indexes a 20-byte record array with it — this + (x+7)*20, whose records are written as three words plus an 8-byte timestamp, i.e. the shape of the savegame's 16×20-byte SHAB table;
• 0x821C1EEC and 0x821C1924 pass it as the index into the config key STAGE (0x820A2540) — the debriefing config's px_deb_stage01…16 sprite list.

So word A is a "stage cleared" bitmask, and a challenge mission's REQUIREMENT n means "stage n has been cleared". The < 24 / >= 24 split then lines up with the disc's own stage numbering from §1: story 1–16 and tutorial 18–23 sit in word A, and the challenge stages 24–29 are exactly word B's bits 0–5.

⚠️ Third revision of this claim — the first two were wrong, and how they went wrong is worth keeping. An earlier pass read the split at 24 as "the game's 24 achievements" because ACHIEVEMENTS_REQUIREMENTS has exactly 24 entries. That is a coincidence: the achievement count and the first challenge stage id are both 24 for unrelated reasons. The achievement work itself stands — the XACH table, the 1000G self-check, and the XACHIEVEMENT_DETAILS/XAM enumeration are all solid, and are documented in structures/achievements.md — it just does not gate the challenge missions. The lesson: a numeric coincidence is not a join; find the writer.

5.3 What the six requirement values are ✅/🟡

The record's numeric tokens are 16, 25, 26, 27, 29, with 24/28 already in the pool earlier (they double as font metrics) and so deduped away if used. IDXD dedup makes positional key/value pairing unsound in general — but 16 sits immediately before REQUIREMENT (tokens 96 → 97), the documented value-before-key adjacency, so the first mission's value is well-supported.

TimeAttack requires stage 16 cleared — the final story mission. That is the natural gate, and it is what the first reading of these numbers suggested before the achievement detour talked me out of it.

The other five values are ≥ 24, i.e. challenge stages 24–29: the challenge missions chain off each other. 🟡 on the exact pairing (which mission needs which), which needs the record's binary index section rather than the string pool.

🟡 Word B has no known writer. GamePart_StageClear sets 1 << x into word A unconditionally, which for a challenge stage (x ≥ 24) would land on word A bits 24–29, not word B — so clearing a challenge mission must be recorded by a different path, presumably one that also stores its Time/Points record. Not yet found.

Negative worth keeping: the requirement text (TimeAttackRequirement, Extra01Requirement, …) is not in GP_CHALLENGE.pak — building a TextIndex over it yields 0 entries and its only wordy payload is embedded font copyright. The config's PATH is dat\GP_CHALLENGE.pak+eng\, a per-language branch, so those keys resolve through a naming scheme the current loader does not reproduce. Reading them would say in plain English what each mission asks for — worth one more attempt via hash::TOC_NAME_SCHEMES.

5.4 Where the mask lives, and one lead refuted

5.5 Both gate words are one record — and it is not the savegame ✅

0x82175110 copies the record at singleton +80 in full: two words, an 8-byte pair, a 184-byte memcpy, 8 words from +200, then 816 bytes at +232, 816 more at +1048, a sub-object at +1864, and a final word at +1876. So the record runs +0 … ~+1880, i.e. singleton +80 … +1960 — which means

• word A is record +0 (singleton +80), and
• word B is record +1876 (singleton +1956).

That answers §5.3's "word B has no known writer": there is no separate stw because the whole record is copied out, modified and assigned back as a unit (0x8216FF70 → compare 0x822C3708, assign 0x82170650, then spawn a worker 0x821700A8 under a lock which retries a commit 0x822C33B8 up to five times). GamePart_Debriefing uses the same get/modify/set to accumulate saturating career counters at +200…+224 (the last 64-bit) — the quantities the achievement requirements test.

And the record is not what savedata holds. Two independent checks:

• Size. Every real save on disk is a 276-byte container deflating to a 545-byte payload. The progress record is ~1 880 bytes. It does not fit.
• Files. After many sessions the game's content tree holds only game0N/savedata + game0N/__thumbnail.png per slot and the three Headers/*.header — no second data file anywhere.

Also checked and negative: the savegame object is *(*(this+4)) + 304, and the singleton's holder address (0x828F48B0) is referenced nowhere but inside the accessor itself, so this+4 is a different holder — the save block is not a window into this record.

⚠️ So hand-editing a save cannot unlock the challenge missions. That kills the operational hope this section previously carried; the earlier savegame-editing win does not extend here.

5.6 CONFIRMED ON THE RUNNING GAME ✅

Read live from a booted title (tools/re-capture/gpoke.py r32):

0x828F40C0 = 0x00000002      word A
0x828F4814 = 0x00000000      word B

Word A = 2 = bit 1 set. The profile's save is Stage 02, "At Standby" — i.e. stage 01 cleared — so the mask is exactly one bit, at the index of the one cleared stage, 1-based. Reproduced on two separate cold boots. That confirms, on the running game and against a known progress state:

• the singleton really is the static object at 0x828F4070;
• word A is a cleared-stage bitmask, not achievements, not a stage number;
• bit index = stage id, 1-based — so TimeAttack's REQUIREMENT 16 is "clear stage 16", the last story mission;
• word B is the challenge half and is 0 on a story-only profile, as expected.

Both words were then poked (0xFFFFFFFF / 0x3F) and read back OK.

MISSION SELECT shows the mask directly (capture). With word A = 2 the screen lists Stage01 selectable, with a High Score and Best Time, and Stage02–Stage08 greyed out. One cleared stage, one selectable entry, at the bit index that names it — the semantics are visible on screen, not inferred.

And the mask drives that screen. Two runs, identical navigation, fresh boot each:

run	word A	MISSION SELECT
control	0x00000002 (untouched)	opens; Stage01 selectable, rest greyed
poked	0xFFFFFFFF	MmAllocatePhysicalMemoryEx fails on a 128 MB request, guest throws, Xenia shows "Disc Read Error"

So the earlier failure was caused by the poke, and by a careless one: 0xFFFFFFFF claims stages that do not exist (0, 17, 24–31 in word A). Poking only real story ids (0x0001FFFE = stages 1–16) does not blow the heap. That the list screen changes behaviour with the mask is itself confirmation that word A feeds it.

Poking real stage ids unlocks the story campaign in the menu ✅. With word A = 0x0001FFFE (stages 1–16) every entry Stage01…Stage16 is selectable (capture) where the control had only Stage01; Stage16 reads "Lonely Blue Planet — NO RECORD". So any story stage can now be launched from the menu, with no save editing, by poking one word.

But MISSION SELECT is story-only ❌. With word B = 0x3F (challenge stages 24–29 marked cleared) the list still saturates at Stage16 (capture) — the cursor stops there and further presses do nothing. That matches the data: the debriefing config declares exactly px_deb_stage01…16, so the list length is capped by the disc, not by the mask. The challenge missions cannot be reached through this screen, and word B does not feed it.

What the poke did not do (yet): EXTRAS still shows only MISSION SELECT / MOVIE THEATER / BACK — no challenge entry — although the menu was built 26 s after the poke, so this is not staleness. Entering MISSION SELECT then failed, and the log gives the real reason: MmAllocatePhysicalMemoryEx could not satisfy a 128 MB request (parent free 30633/131072 pages), the guest threw a C++ exception, and Xenia surfaced it as its generic "Disc Read Error" dialog. It is preceded by BaseHeap::Release failed because address is not a region start, a failed release that leaks the range. So that is an emulator/heap problem on the way into the screen, not evidence about the gate. Open: repeat without the poke to see whether MISSION SELECT fails the same way regardless.

5.7 What would work — static addresses for a live write

The singleton is a static object at 0x828F4070 (0x8216F650: addis 0x828F + addi …, 16496), with the holder at 0x828F48B0 pointing at it. So the two gate words are at fixed guest addresses, no scanning required:

word	guest VA	meaning
A	0x828F40C0	cleared stages, bit = stage id (< 24)
B	0x828F4814	cleared stages, bit = stage id − 24 (challenge 24–29)

Canary maps guest RAM into /dev/shm, which the project already reads live (tools/re-capture/gmem.py). Writing 0xFFFF into word A and 0x3F into word B while the title sits on a menu should open all six challenge missions without playing the campaign — and that is the route to the last 42 *_EX4/*_EX5 units. Untested: it needs a run, and the run needs gamepad input to reach the menu.

The obvious lead is dead. The three stws to +1956 in 0x822AF278 / sub_822C8748 looked promising because they sit in the save serializer's code region — they are on a different object. That one is fetched through 0x822CEB30, has a +2652 flag the code checks first, and stores string pointers at +1956/+2024 (built by 0x822D35F8); a pointer ANDed with 1 << n would be meaningless as a gate. So nothing in the image stores to this singleton's +1956 field-wise, which means it is filled by a bulk copy or by a path not yet found.

Two candidates remain and they need different levers: the save (hand-write it) or the Xbox profile (the emulator's profile data). Note that XEX imports are resolved by ordinal, so the absence of XamUser* name strings in the .pe is not evidence against the profile route. +80 at least is handled as a small struct by address (addi r4, obj, 80 → copy helper 0x82175110, written back via 0x8216FF70), which is what a serialised value object looks like.

5.8 The part id is never persisted — differential search ✅ (negative)

sub_821749C0 creates a part from slot+12, and no literal 26 exists anywhere, so the id is computed. It can still be found by differential search, because the id is known at each screen from §3: GP_EXTRAS = 5, GP_MISSION_SELECT = 7. Snapshot both screens and intersect (tools/re-capture/diff_words.py):

scanned 171 MB of allocated guest memory
addresses reading 5 on EXTRAS and 7 on MISSION SELECT: 4
   0x708FFBEC  0x708FFCBC  0x708FFDAC  0x708FFE20

All four are guest stack (thread stacks sit at 0x709… in the same run's log). So the requested part id exists only as a stack argument in flight — there is no persistent field holding it, which is consistent with finding no literal store and means there is nothing stable to poke. Forcing a transition to GamePart 26 needs the caller's context, i.e. an emulator-side hook rather than a memory write.

Every memory-and-menu route to the challenge missions is now closed (§5.4, §5.6, §5.7, this section). What remains is the genuine in-game unlock — an in-mission attainment, per AVSCRIPT_COMMAND_ATTAINMENT_CHALLENGE_MISSION_CARGO_SCORE — or a Canary patch that forces the transition in emulator code.

6. The unlock condition, from the strings ❔

Three strings say challenge missions are announced, not menu-browsed:

• AVSCRIPT_COMMAND_ATTAINMENT_CHALLENGE_MISSION_CARGO_SCORE — a mission-script command that grants challenge-mission attainment from a cargo score;
• DLG_CHALLENGE_MISSION_AVAILABLE — the "a challenge mission is now available" dialog;
• DLG_GO_CHALLENGE_MISSION_MENU — the prompt that takes you to GamePart 26.

So the gate is plausibly an in-mission achievement, not a title-menu state — which is consistent with Game Status = GAME_CLEAR unlocking nothing (measured, refuted). Neither dialog key is reachable by xref: they are selected by index through a config lookup, exactly like the save screen's sprite keys, so there is nothing to grep back to. Naming the flag needs the challenge-availability check disassembled from the screen side.

7. What this makes actionable

roster_target against the harvested CSV, with the stage families now named:

stage	family	roster units not yet read
S27	challenge	f003_ArrowHead_EX4, e107_AAFrigate_EX4, e010_Attacker_S_EX4, e011_Attacker_B_EX4, e007_Turret_EX4, e008_TurretPlus_EX4
S28	challenge	f004_DeltaSaber_A_Player, f001_DeltaSaber_T_EX5(_el), f003_ArrowHead_EX5, f104_Battleship_EX5, f105_Cruiser_EX5
S25	challenge	e101_SDBattleshipEX, e102_BattleshipEX, e105_CruiserEX, e108_ASFrigateEX
S29	challenge	e101_SDBattleshipEX, e102_BattleshipEX, e105_CruiserEX
S24	challenge	e105_CruiserEX, e106_DestroyerEX
S10	story	e005_ADAN_ElanTypeQ_Margras

⚠️ The S10 row contradicts "the story campaign is complete." One story stage still fields a unit that has never been read. Either S10 was never flown (it is the smallest stage container on the disc — 7 XBG7 resources — so it may be a cutscene stage that is not flyable), or the completeness claim was one unit optimistic. Flagged rather than resolved: it costs one ordinary story run to settle.

Remaining coverage: 42 units missing, of which the rosters above account for ~23. The rest are not in any stage roster table and need a different lever.

Reproduce

cargo run --release -q -p sylpheed-formats --example challenge_map    -- <disc-root>
cargo run --release -q -p sylpheed-formats --example challenge_screen -- <disc-root>
python3 xenia-rs/zq.py dis 0x82184df0 0x82184e30      # the three-way section switch
python3 xenia-rs/zq.py dis 0x82185ed0 0x82185f10      # the same switch, second site
python3 xenia-rs/zq.py dis 0x82189860 0x821899f0      # the challenge availability gate