re: decode the UnitGroup squadron roster, and correct yesterday's container reading
stage\UnitGroup_S<NN>.tbl is now fully readable. A squadron record is Count
member tuples -- (unit model, message set, n, identity/nameplate) -- followed by
five named fields: Count, SideID, AIID, FormationID, DisableInterval. The
property entries carry their own field names inline, so the tag hash never has
to be inverted.
Two independent self-checks validate it corpus-wide, 1160/1160 each across all
28 stage tables on the disc:
- the length identity len(fields) == Count * 4 + 5, which is what pins the
member-tuple width at 4 and the named-field count at 5;
- agreement with the file's own Enumerate_Squadrons roster, which maps record
key to squadron id independently of the per-record string offset.
Adds tools/re-capture/unitgroup.py (pure static, runs no emulator) with a
--all --check self-check mode, and commits the Stage 02 dump as evidence.
Corrections to the container layout written yesterday, all three wrong:
- the 20-byte "(tag, 0, 0, count, size) section header" does not exist. It
was the file's last 16-byte record followed by a plain npool word. The
corrected layout is uniform across all 28 files; the old one failed on 9.
- squadron ids do not use a separate string base. Every offset in the file is
relative to the one string pool. The earlier "109 of 111" score was an
artefact of the uniform 7-byte id stride and had silently shifted every
name by three entries, which is why 17 TC*-named squadrons came out as
SideID=ADAN. The roster record refuted it outright.
- the roster is not always the last record; 9 stages put it elsewhere, so it
is found by its missing Count.
Refuted and kept: the 4-byte record key is not the squadron id's name hash
(0 of 112).
Not settled: what the key encodes, the member tuple's third field n, and where
the arrival interval values live. DisableInterval is only a per-squadron flag
(Yes for 31 of 1160); the durations, triggers and arrival positions are not in
this file. Formation_*.tbl and EnumSquadron_Test.tbl are next.
This commit is contained in:
Sylpheed RE agent
@@ -97,26 +97,19 @@ scheduler uses: squadrons are gated on an *interval*, and individual records can
|
||||
opt out of it. That is consistent with the observed behaviour that a mission's
|
||||
live enemy count rises and falls rather than being fixed.
|
||||
|
||||
## 🟡 Container layout
|
||||
## ✅ Container layout — see [unit-group-table.md](unit-group-table.md)
|
||||
|
||||
The file is a multi-section IDXD container, and `pak dump`'s current labels for
|
||||
it are wrong — it prints the first section's key as "schema".
|
||||
**Superseded.** An earlier draft of this section described a multi-section
|
||||
container with a 20-byte `(tag, 0, 0, count, size)` header at `0x708`, 112
|
||||
records, and a separate squadron-id string base. That reading was wrong on all
|
||||
three points: the "header" was the file's last 16-byte record followed by a
|
||||
plain `npool` word, and it failed to parse 9 of the 28 stage tables.
|
||||
|
||||
```
|
||||
0x0000 "IDXD"
|
||||
0x0004 u32 be = 0x70 = 112 record count
|
||||
0x0008 112 x 16-byte entries: u32 key, u32 a, u32 b, u32 c
|
||||
0x0708 next section: u32 tag (0x c5c9f110), 0,0, u32 count=0x6f, u32 size=0x46a,
|
||||
then the SAME 112 keys again with a narrower payload
|
||||
...
|
||||
0x3c1d 112 squadron-ID strings: TCN001.. ADN101.. ADT102.. TCS151.. ADS151..
|
||||
(prefixes ADN, ADS, ADT, TCN, TCS, TCT)
|
||||
0x3f26 field-name / value string pool (the vocabulary above)
|
||||
```
|
||||
|
||||
The 112×16 entry array was confirmed by boundary, not assumed: keys are strictly
|
||||
increasing for exactly 112 entries and the run breaks at `0x708`, where the next
|
||||
section header sits.
|
||||
The corrected, uniform layout — header, 16-byte records, a 12-byte property
|
||||
pool, a string-pool size trailer that checks out against the file length exactly,
|
||||
and every offset relative to one string base — is documented in
|
||||
[unit-group-table.md](unit-group-table.md), together with the two corpus-wide
|
||||
self-checks that validate it (1160/1160 on both).
|
||||
|
||||
### ❌ Refuted: the record key is not the squadron-ID name hash
|
||||
|
||||
@@ -128,10 +121,9 @@ than a hash, but that is untested.
|
||||
|
||||
## What is still open
|
||||
|
||||
* The per-record payload: which of the three words per entry selects the
|
||||
formation, the AI, the side, the count.
|
||||
* The meaning of the record key.
|
||||
* **The interval schedule itself.** `DisableInterval` names the mechanism but
|
||||
the interval *values* have not been located — they may be in the second
|
||||
section, in `Formation_*.tbl`, or in `EnumSquadron_Test.tbl`.
|
||||
* S17–S23 stage records.
|
||||
* **The interval schedule itself.** `DisableInterval` names the mechanism and 31
|
||||
squadrons set it, but the interval *values* have not been located — next
|
||||
candidates are `Formation_*.tbl` and `EnumSquadron_Test.tbl`.
|
||||
* S17–S23 stage records. S17 also has no `UnitGroup_S17.tbl`, so its whole
|
||||
stage is absent from `GP_MAIN_GAME_E.pak` rather than merely unnamed.
|
||||
|
||||
122
docs/re/structures/unit-group-table.md
Normal file
122
docs/re/structures/unit-group-table.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# `stage\UnitGroup_S<NN>.tbl` — the per-stage squadron roster
|
||||
|
||||
Status: ✅ container format and field semantics, validated across all 28 stage
|
||||
tables present on the disc; 🟡 one member field; ❔ the record key and the
|
||||
arrival-interval *values*.
|
||||
|
||||
Reached from the per-stage definition record's `EnumerateSquadron` field — see
|
||||
[stage-definition-table.md](stage-definition-table.md). Tool:
|
||||
`tools/re-capture/unitgroup.py` (pure static; runs no emulator).
|
||||
|
||||
```
|
||||
python3 tools/re-capture/unitgroup.py S02 # full roster
|
||||
python3 tools/re-capture/unitgroup.py --all --check # self-check every stage
|
||||
```
|
||||
|
||||
A committed dump of Stage 02 is at [`../data/unitgroup-s02.txt`](../data/unitgroup-s02.txt).
|
||||
|
||||
## ✅ Container
|
||||
|
||||
```
|
||||
0x00 "IDXD"
|
||||
0x04 u32 nrec
|
||||
0x08 nrec x 16 record: (key, squadron_off, field_lo, field_hi)
|
||||
u32 npool
|
||||
npool x 12 property entry: (tag, name_off | 0xffffffff, value_off)
|
||||
u32 strsize -- and STR + strsize == filesize, exactly
|
||||
strsize string pool; [0] = "Enumerate_Squadrons", [0x14] = ""
|
||||
```
|
||||
|
||||
**Every offset in the file is relative to `STR`**, the string-pool base. A
|
||||
record's fields are pool entries `[field_lo, field_hi)`. An entry with
|
||||
`name_off == 0xffffffff` is *positional*; otherwise the entry carries its own
|
||||
field name inline, so the table is self-describing and the tag hash never has to
|
||||
be inverted.
|
||||
|
||||
Exactly one record per file is not a squadron: it is the **`Enumerate_Squadrons`
|
||||
roster**, whose entries map `key -> squadron id` for every other record. It is
|
||||
usually the last record but not always — in S05, S06, S07, S08, S09, S11, S13,
|
||||
S28 and S29 it sits elsewhere, so find it by its missing `Count`, not by
|
||||
position.
|
||||
|
||||
## ✅ A squadron record
|
||||
|
||||
Five named fields, always present and always these five:
|
||||
|
||||
| field | values |
|
||||
|---|---|
|
||||
| `Count` | 1 (×1082), 2 (×50), 3 (×8), 4 (×20) |
|
||||
| `SideID` | `ADAN` (745), `TCAF` (294), `Neutral` (121) |
|
||||
| `AIID` | 31 distinct, e.g. `AI_ADAN_CraftSquadron_Veteran`, `AI_TCAF_BirdFlight`, `AI_ADAN_Fleet`, `AI_Structure`, `AI_TraceRoute` |
|
||||
| `FormationID` | e.g. `Formation_4_Bird`, `Formation_ADAN_Turret07_30`, `Formation_1_only` |
|
||||
| `DisableInterval` | `No` (1129), **`Yes` (31)** |
|
||||
|
||||
…preceded by `Count` **member tuples** of four positional entries:
|
||||
|
||||
```
|
||||
(unit model, message set, n, identity)
|
||||
```
|
||||
|
||||
* unit model — 122 distinct, and they are our XBG7 mesh names:
|
||||
`UN_f001_TCAF_DeltaSaber_T_Player`, `UN_e010_ADAN_Attacker_S`,
|
||||
`UN_e007_ADAN_Turret`, `UN_bf001_TCAF_SchlosBase`.
|
||||
* message set — the squadron's radio chatter, e.g. `MessageSet_Ellen`,
|
||||
`MessageSet_ADAN_plA`.
|
||||
* `n` — 🟡 **unidentified**; integer 1…30, dominated by 1 (735) and 9 (197).
|
||||
It is *not* the `_NN` suffix of `FormationID` (`Formation_ADAN_Turret07_30`
|
||||
pairs with `n = 9`). Plausibly a spawn or wing size, untested.
|
||||
* identity — 63 distinct: named pilots (`ELLEN`, `SANDRA`, `RAYMOND`, `YOJI`),
|
||||
ship nameplates (`NP_Charon`, `NP_Amalthea`, `NP_Olympus`), carrier tags
|
||||
(`Carrier_ADAN01`), cargo tags (`CARGO_1`), or empty.
|
||||
|
||||
Worked example — Stage 02, squadron `TCN004`, the player's own flight:
|
||||
|
||||
```
|
||||
TCN004 TCAF Count=2 DisableInterval=No Formation_2_Rhino1 AI_TCAF_RhinoFlight
|
||||
unit=UN_f001_TCAF_DeltaSaber_T_Player msg=MessageSet_Katana n=1 id=Character_Player_Test
|
||||
unit=UN_f001_TCAF_DeltaSaber_T msg=MessageSet_Ellen n=1 id=ELLEN
|
||||
```
|
||||
|
||||
## ✅ Two independent self-checks, both corpus-wide
|
||||
|
||||
1. **`len(fields) == Count * 4 + 5`** — holds for **1160 of 1160** squadrons
|
||||
across all 28 stage tables. This is what pins the member-tuple width at 4 and
|
||||
the named-field count at 5; it was not assumed.
|
||||
2. **Roster agreement** — the `Enumerate_Squadrons` record's `key -> id` mapping
|
||||
agrees with the name each record resolves independently through its own
|
||||
`squadron_off`, for **1160 of 1160**.
|
||||
|
||||
S17 has no `stage\UnitGroup_S17.tbl` in `GP_MAIN_GAME_E.pak`, matching the
|
||||
missing S17 stage record.
|
||||
|
||||
## ❌ Refuted along the way
|
||||
|
||||
* **The record key is not the squadron id's name hash.**
|
||||
`name_hash("TCN001") = 0xd639f1a4`, but the keys run
|
||||
`0x659aff47, 0x659b0046, 0x669b0047, …` — 0 of 112 match. What the key encodes
|
||||
is still ❔. Its byte structure (`b0` ramping, `b2` taking small signed values)
|
||||
looks like a packed tuple, untested.
|
||||
* **Squadron ids do not use a separate string base.** An earlier reading here
|
||||
put them at `STR + 0x15` and scored "109 of 111", which looked like a near-fit
|
||||
but was an artefact of the uniform 7-byte id stride — it silently shifted
|
||||
every name by three entries (record 0 read as `ADN104` instead of `ADN101`,
|
||||
and 17 `TC*`-named squadrons came out with `SideID = ADAN`). The roster record
|
||||
refuted it outright, and with the correct base — plain `STR` — agreement is
|
||||
111/111. The lesson is that a 98 %-looking score on a self-consistent stride
|
||||
is not evidence; the roster was an independent oracle and should have been
|
||||
consulted first.
|
||||
* **The 20-byte "section header"** described in an earlier draft of
|
||||
[stage-definition-table.md](stage-definition-table.md) does not exist. What
|
||||
looked like `(tag, 0, 0, count, size)` was the file's last 16-byte record
|
||||
followed by the `npool` word. The corrected layout above is uniform across all
|
||||
28 files, which the earlier reading was not — it failed on 9 of them.
|
||||
|
||||
## What is still open
|
||||
|
||||
* The meaning of the 4-byte record key.
|
||||
* The member tuple's third field `n`.
|
||||
* **The interval values.** `DisableInterval` is a per-squadron *flag* and 31
|
||||
squadrons set it, but the interval durations, spawn triggers and arrival
|
||||
positions are not in this file. The next places to look are
|
||||
`Formation_*.tbl` (the formation geometry and possibly its timing) and
|
||||
`EnumSquadron_Test.tbl`, both named by the stage record.
|
||||
Reference in New Issue
Block a user