Files
Sylpheed/tools/re-capture/isl.py
Sylpheed RE agent 142b8d7200 re: close the ISL operand chain -- built-in result is special[0], via a vtable
Answers what the previous commit left open: naming the branches did not give a
clear condition, because that needs the operand chain feeding each compare.

First, a correction to my own work.  isl-bytecode.md -- which OWNS the opcode table
-- already named ops 21-24 push.i/push.f/pop.i/pop.f.  isl-branches.md, which I
wrote last iteration, said op21 and op23 were unread.  The stale file was mine.
Verified from the thunks rather than accepted: 21 pushes [phase+168] onto the deque
at phase+44, 22 pushes [phase+184] onto phase+64, and the 23/24 handlers touch only
r3+168 and r3+184.  So pop.i lands in special[1].

New: the 147-entry built-in table is a thin DISPATCH LAYER, not implementations.
Each stub resolves the local[] argument base and tail-calls a fixed ScriptPhase
vtable slot.  112 of 147 dispatch that way; 17 write [phase+164] inline; 0 write
+184.  Every named predicate is in the vtable group -- unit_state 184, unit_alive
188, hp_pct_test 64, dist_lt 56, is_engaged 252, timer_elapsed 372 -- which is the
control that the split separates engine queries from script bookkeeping.

The vtable is 0x820A84BC, derived from a known implementation rather than a stride:
MARK_LAST_PHASE is documented as [phase+300]=2; the function 0x8226B498 is exactly
that stub; it appears as a data word at exactly one address, 0x820A8570; built-in
39 uses slot 180.  The check NOT used in the derivation: built-in 40 mark_not_last
uses slot 176, and slot 176 holds the [phase+300]=1 stub.  Predicted and confirmed.
The db's own vptr_writes independently lists 0x820A84BC, written at 0x82261B80.

unit_state = slot 184 = 0x8226ADF0, which indexes [phase+324] by local[4] and writes
its answer to [phase+164] = special[0] at both exits.  The phase-3 poll loop now
reads end to end: unit_state(ADT308) -> special[0]; pop.i -> special[1]; cmp.i; beq.

isl.py names ops 21-24; the calls artefact regenerates with NO diff.

Left open and said so: the other 111 vtable slots, which comparand each site pushes,
the 35 non-vtable built-ins, and the vtable's length.
2026-08-27 05:21:10 +00:00

434 lines
20 KiB
Python
Executable File

#!/usr/bin/env python3
"""Disassemble the ISL script bytecode inside a `Stage\\StageNN.ssb`.
The VM is `ScriptPhase::Update` (`sub_82263408`). Everything below is read off
the dispatcher and its 25 handlers, not guessed:
0x822635D4 lwz r11,0(r31) ; instruction = one big-endian u32
0x822635D8 clrlwi r4,r11,24 ; OPCODE = the LOW byte (= byte[3])
0x822635DC cmplwi 0x18 ; 25 opcodes
0x822635FC jump table (25 absolute VAs)
Each handler advances the pc by `lbz r11,2(r31); add r31,r11,r31`, so
**byte[2] is the instruction length in bytes**, and bytes [0]/[1] are operand
kind selectors passed to the operand resolvers as `r4`.
op 0 `lbz 0` + word@+8 -> resolve ; `lbz 1` + word@+4 -> lvalue ; stw
(integer assignment; resolvers 0x82271D40 / 0x82272030)
op 1 same shape with fmr/stfd (float assignment; 0x82271F10/0x82272120)
op 12 JUMP: r31 = [phase+232] + word@+4
-> jump operands are relative to `[phase+232]`, which is **PER PHASE**,
not the file's 0x24. The phase initialiser sub_82270DF8 writes it
as 0x24 + the phase's entry from the mission-level stream, whose
three `0x1883` records carry 0xC0 / 0x14A84 / 0x24B28 for Stage 02
-> bases 0xE4 / 0x14AA8 / 0x24B4C.
MEASURED: with 0xE4, 525 of 525 phase-1 branch targets land on an
instruction boundary; with 0x24, only 188. Using 0x24 for every
phase -- which this tool did -- gives wrong targets in phases 2
and 3, and mostly-wrong ones in phase 1.
op 19 CALL BUILT-IN: `sub_82272220` reads the id from **word@+4**
(`lwz r11,4(r28); cmplwi 0x92` -> 147 built-ins, table 0x8227226C)
and word@+8 into [phase+200].
op 20 sets r29=1 and takes the suspend path -> yield/return.
Handler return codes drive the outer loop: 0 = continue, 1 = suspend,
2/3 = other exits (`0x82263828`).
Instruction layout, confirmed by the decode reading cleanly from the code base
and by every routine ending on a `ret`:
byte[3] opcode | byte[2] length | byte[1],byte[0] operand kinds
following words: operands (12 bytes is the common `call` form)
**Operand kinds** (resolver table `0x82271D74`, 4 entries):
0 global[i] lis 0x828E / bl 82454A40 / lwzx -- indexed global array
1 immediate mr r3,r31 -- the operand word itself
2 special[i] [phase+164] if i==0 else [phase+168]
3 local[i] addi r3,r3,20 / lwzx -- [phase+20 + i]
so the recurring pair
set.i k=01,02 <A> <V> special[A] = V (immediate -> special)
set.i k=02,03 <B> <0> local[B] = special[0]
is **argument staging**: values land in `local[]` slots 0,4,8,0xC… and the next
`call` consumes them. That is why a built-in's arguments are not in its own
instruction.
A `call` carries the built-in id in word@+4 and a monotonically increasing
STATEMENT ID in word@+8 (0x245, 0x248, 0x24A, ... across a routine) -- the value
`sub_82272220` stores to `[phase+200]`, i.e. a source-position counter.
Usage: isl.py <file.ssb> <offset> [count] offsets are FILE offsets
isl.py <file.ssb> --entry <off> follow from a code-base offset
isl.py <file.ssb> --calls every built-in call site + histogram
isl.py <file.ssb> --to <target> [n] resync and disassemble INTO target
"""
import struct
import sys
CODE_BASE_FIELD = 0x08 # .ssb header: code offset (0x24 in every file)
# opcode -> (mnemonic, handler VA) from the jump table
KIND = {0: 'global', 1: 'imm', 2: 'special', 3: 'local'}
# op10 `cmp.i` resolves its two operands as
# LHS = resolve(kind byte[1], word@+4) `lbz r4,1(pc)` + `lwz r5,4(pc)`
# RHS = resolve(kind byte[0], word@+8) `lbz r4,0(pc)` + `lwz r5,8(pc)`
# and issues a SIGNED `cmp cr6, 0, LHS, RHS`. op11 `cmp.f` is the same shape
# through the float resolvers with `fcmpu`. So a listing line
# cmp.i k=01,02 00000000 00000002
# reads LHS = special[0], RHS = imm 2 -- "compare special[0] with 2".
# Built-in names, from the 147-entry table at 0x8227226C. Only the ones whose
# handler was actually read are named; the rest print as a bare id rather than a
# guess. See docs/re/structures/isl-builtins.md.
BUILTIN = {
1: 'start_coroutine', 2: 'deploy_squadron', 3: 'move_order', 4: 'wait_s',
5: 'kill_coroutine', 6: 'END_PHASE', 8: 'set_flag', 9: 'read_freg',
10: 'random', 11: 'end_coroutine', 12: 'activate_unit',
13: 'play_se', 14: 'play_bgm',
15: 'set_group_speed',
17: 'wait_frames', 18: 'dist_lt', 20: 'hp_pct_test', 24: 'squad_survival_pct',
26: 'damage_unit', 30: 'objective_marker', 31: 'objective_marker_at_route',
33: 'global_counter0', 34: 'global_counter1', 36: 'screen_fade',
39: 'MARK_LAST_PHASE', 40: 'mark_not_last', 43: 'play_voice',
45: 'play_voice_vol', 46: 'squadron_trace', 47: 'squadron_attack',
48: 'squadron_escort', 52: 'play_stream', 53: 'sound_busy', 54: 'stop_sound',
56: 'unit_relation', 59: 'fade_sound', 62: 'FORCE_END_PHASE', 64: 'request_script_message',
69: 'unit_state', 70: 'unit_alive', 72: 'group_ratio_pct', 73: 'timer_start',
74: 'timer_limit',
# ❌ 88 'camera_at' and 90 'camera_at_route' WITHDRAWN. 88 has ZERO call
# sites in all 28 stages, so its name was never testable. 90 has exactly 8,
# all in Stage 02 phase 3 (the cruise-missile act), and its first operand
# resolves to symbol-table-1 type 7 -- `eff_n0071`, an EFFECT name -- in
# 8/8, with a per-missile `Route_ADT30N_p3M` at slot 20. Whatever it does,
# it is not aimed at a camera. Left unnamed rather than renamed on a guess.
77: 'banner_mission_start', 78: 'banner_mission_complete',
81: 'banner_objective_update', 82: 'banner_mission_failed',
135: 'banner_mission_restart',
93: 'clear_flag', 94: 'is_engaged', 95: 'unit_hp_pct', 100: 'reset_phase_threads',
102: 'prompt_yes_no', 108: 'deploy_squadron_ex',
109: 'set_unit_flags', 115: 'named_event',
120: 'wait_cmds_drained', 123: 'timer_resume', 124: 'timer_stop',
125: 'timer_reset', 126: 'timer_elapsed', 127: 'timer_set',
132: 'player_gauge0_test', 133: 'player_gauge1_test', 134: 'player_byte',
137: 'wait_units_ready', 139: 'fade_to_black_end', 142: 'deploy_and_wait',
143: 'deploy_and_wait2', 145: 'random_rand',
}
OPS = {
0: 'set.i', 1: 'set.f',
2: 'cmp.a', 4: 'cmp.a', 6: 'cmp.a', 8: 'cmp.a',
3: 'cmp.b', 5: 'cmp.b', 7: 'cmp.b', 9: 'cmp.b',
# 10-18: a condition-code architecture, read off the handlers. op10/op11
# COMPARE and write three bits into a bitset at `phase+24`
# bit 0 = EQ bit 1 = GT bit 2 = LT
# and 13-18 branch on those bits to `[phase+232] + word@+4` -- the same
# phase-relative target form as the unconditional op12. All six relations
# are present, which is itself the check that the reading is right.
10: 'cmp.i', 11: 'cmp.f', 12: 'jmp',
13: 'beq', 14: 'bne', 15: 'blt', 16: 'ble', 17: 'bgt', 18: 'bge',
19: 'call', 20: 'ret',
# 21-24 name the deque ops -- see structures/isl-bytecode.md, verified from
# the thunks: 21 pushes [phase+168] onto the deque at phase+44, 22 pushes
# [phase+184] onto phase+64, and 23/24 pop back into +168 / +184.
21: 'push.i', 22: 'push.f', 23: 'pop.i', 24: 'pop.f',
}
def load(path):
return open(path, 'rb').read()
def symbols(b, which):
"""Parse a .ssb symbol table -> {index: (type, name)}.
Built-in argument blobs carry INDICES into these: fields that index
`[phase+244]` are symtab-1 (routes, messages, subobjectives) and fields that
index `[phase+324]` are symtab-2 (the unit ids). Resolving them is what turns
`unit_state(0x2b)` into `unit_state(ADN201)`.
"""
off = struct.unpack_from('>I', b, 0x0C if which == 1 else 0x10)[0]
cnt = struct.unpack_from('>I', b, off)[0]
base = off + 4
out = {}
for i in range(cnt):
o = struct.unpack_from('>I', b, base + 4 * i)[0]
if o == 0:
continue
rp = base + o
typ = struct.unpack_from('>I', b, rp)[0]
e = b.index(b'\0', rp + 4)
out[i] = (typ, b[rp + 4:e].decode('latin-1'))
return out
# Built-ins whose operand blob carries a symbol-table-2 (unit) index, by slot.
#
# Derived from the DATA, not from reading 147 handlers: across all 28 stages a
# slot qualifies only if every observed value is a valid symtab-2 index, it takes
# >=15 distinct values, and its maximum reaches most of the table (symtab-2 tops
# out at 122 entries, so a non-index slot overruns). That last clause is what
# makes the test discriminating -- plain range-checking cannot separate an index
# from a bool, because every small integer is "in range".
#
# It also refutes one tempting entry: `set_flag`'s slot 0 passes the range and
# spread tests but its maximum EXCEEDS the table (flag indices run 0..31 against
# tables as small as 40), so it is excluded. Slots are only listed here when the
# ratio stayed below 1.0.
UNIT_ARG = {2, 3, 7, 12, 15, 16, 18, 19, 20, 24, 25, 26, 28, 29, 30, 47, 48,
56, 57, 58, 63, 69, 70, 79, 91, 92, 95, 105, 108, 128, 143}
UNIT_ARG2 = {2, 18, 47, 48, 56, 79, 95, 128} # a SECOND unit index at blob[12]
UNIT_ARG3 = {128} # and a third at blob[20]
UNIT_SLOTS = {4: UNIT_ARG, 12: UNIT_ARG2, 20: UNIT_ARG3}
# WHY the unit indices sit at 4/12/20 and never at 0/8/16: a **symbol operand is
# a two-word pair** -- a tag word holding the constant 1, then the index. The
# tag is not data, so printing it puts a meaningless leading `0x1` in front of
# every unit predicate.
#
# Measured over all 28 stages:
# * slot 0 is the integer 1 in 19899 / 19899 calls whose slot 4 is a unit;
# * slot 8 is tag-shaped in 100% of calls for every built-in taking a second
# unit, and slot 16 is the constant 1 in 152/152 for built-in 128, the only
# one taking a third;
# * 24 built-ins have a slot 0 that is NOT the constant -- and every one of
# them takes no symbol at slot 4 (`start_coroutine` a code offset, `wait_s`
# a double, `set_flag` an index).
#
# The tag does NOT generalise to "every even slot is a tag": slot 8 is a bare
# double for built-ins 4, 20, 24, 26, 28, 29, 90, 106 and 127, and built-in 75
# carries five bare symbol indices at 0/4/8/12/16 with no tags at all. Each
# built-in has a fixed signature and is 100% consistent with itself; none mixes.
TAG_SLOTS = {slot - 4 for slot in UNIT_SLOTS}
# Symbol table 1 holds three types, and its slots were measured the same way as
# the unit slots (every observed value resolves, >=5 distinct values, and the
# resolved type is pure):
# type 1 (1362 entries) `Route_*` names
# type 6 (2247) message / objective names
# type 7 (81) `eff_*` effect names
SYM1_SLOTS = {
0: {64, 75, 115}, # 64 & 75 type 6; 115 type 7
4: {75, 136},
8: {75},
12: {2, 3, 7, 16, 19, 25, 75, 108, 143},
16: {75},
20: {90},
24: {48},
28: {128},
}
# Deliberately NOT listed: built-ins 24@4, 46@12 and 114@4 resolve 100% but mix
# type 6 and type 1, so the slot's meaning is not one thing. Recorded rather
# than guessed at.
# Symbol table 2 holds TWO entity types: type 2 (1160 entries disc-wide) and
# type 8 (249). They are not interchangeable -- built-ins 95 and 128 take a
# type-2 unit at slot 4 and, at slot 12, an operand that is type 8 in 100% of
# its 90 and 152 call sites respectively. What distinguishes the two classes is
# not yet established.
def linear_offsets(b, start=None):
"""Every instruction offset, decoding linearly from the first phase base.
MEASURED over all 28 stages: this reaches 25705/25705 of the call sites
`call_sites()` finds by scanning the encoding, and every file decodes clean
to `code_end` with no desync. The instruction stream is therefore FLAT --
reaching a call site needs no control-flow reconstruction at all.
"""
end = struct.unpack_from('>I', b, 0x0C)[0] # symtab1 = end of code
if start is None:
start = phase_bases(b)[0]
out = []
off = start
while off + 4 <= end:
out.append(off)
ln = (struct.unpack_from('>I', b, off)[0] >> 8) & 0xFF
if ln == 0 or ln % 2:
break
off += ln
return out
def dis(b, off, count=40, code_base=0x24, args=True, sym2=None, sym1=None,
stop_at_ret=True):
"""`stop_at_ret` preserves the ORIGINAL behaviour and is wrong for reading.
op 20 is `ret`, but in a coroutine VM that is a YIELD: the thread suspends
and later resumes at the following instruction, so code continues after it.
Stopping there reaches only 133 of Stage 02's 2846 call sites (4.7%);
continuing reaches all 2846. Pass `stop_at_ret=False` to read a listing.
"""
out = []
staged = {} # local[] slot -> last value staged into it
pending = None # value most recently put in special[0]
for _ in range(count):
if off + 4 > len(b):
break
w = struct.unpack_from('>I', b, off)[0]
op = w & 0xFF
ln = (w >> 8) & 0xFF
k1 = (w >> 24) & 0xFF
k0 = (w >> 16) & 0xFF
name = OPS.get(op, 'op%d?' % op)
words = []
n = max(ln, 4)
for i in range(4, n, 4):
if off + i + 4 <= len(b):
words.append(struct.unpack_from('>I', b, off + i)[0])
extra = ''
if op in (0, 1) and len(words) >= 2:
# op 0/1: lvalue = (kind byte[1], word@+4); rvalue = (kind byte[0], word@+8)
rv = words[1]
extra = ' %s[%d] = %s%s' % (
KIND.get(k0, '?%d' % k0), words[0],
KIND.get(k1, '?%d' % k1),
('' if k1 == 1 else '[%s]' % rv) if True else '')
if k1 == 1:
if op == 1:
lo = words[2] if len(words) > 2 else 0
extra += ' %.6g' % struct.unpack(
'>d', struct.pack('>II', words[1], lo))[0]
else:
extra += ' 0x%X' % words[1]
# track the staging pattern so a call can show its arguments
if op in (0, 1) and len(words) >= 2:
if k0 == 2 and k1 == 1:
if op == 1:
# op 1 stores with stfd, so an immediate float operand is a
# DOUBLE carried as two words -- reading only the high word
# as a float gives 2.125 where the script means 3.0.
lo = words[2] if len(words) > 2 else 0
pending = '%.6g' % struct.unpack(
'>d', struct.pack('>II', words[1], lo))[0]
else:
pending = words[1]
elif k0 == 3 and k1 == 2 and pending is not None:
staged[words[0]] = pending
elif k0 == 3 and k1 == 1:
# local[i] = immediate, DIRECTLY -- the common form. Missing this
# made every unit predicate print with no arguments at all.
if op == 1:
lo = words[2] if len(words) > 2 else 0
staged[words[0]] = '%.6g' % struct.unpack(
'>d', struct.pack('>II', words[1], lo))[0]
else:
staged[words[0]] = words[1]
if op == 19 and words:
extra = ' %s' % BUILTIN.get(words[0], 'builtin%d' % words[0])
if args and staged:
parts = []
# The tag word in front of a symbol operand is not an argument.
tags = {slot - 4 for slot, ids in UNIT_SLOTS.items()
if words[0] in ids and slot in staged}
for slot, v in sorted(staged.items()):
if slot in tags and v == 1:
continue
txt = ('0x%X' % v) if isinstance(v, int) else v
# Resolve only a slot that is declared an index AND whose
# value really is one -- a resolver that invents a name for
# a non-index is worse than one that prints the raw number.
if (sym2 and words[0] in UNIT_SLOTS.get(slot, ())
and isinstance(v, int) and v in sym2):
txt = sym2[v][1]
elif (sym1 and words[0] in SYM1_SLOTS.get(slot, ())
and isinstance(v, int) and v in sym1):
txt = sym1[v][1]
parts.append(txt)
extra += '(' + ', '.join(parts) + ')'
staged = {}
elif op == 12 and words:
extra = ' -> code+0x%X (file 0x%X)' % (words[0], code_base + words[0])
out.append('%06X: %08X %-6s len=%-3d k=%02x,%02x %s%s' % (
off, w, name, ln, k1, k0,
' '.join('%08X' % x for x in words), extra))
if ln == 0:
out.append(' (length 0 -- stopping)')
break
off += ln
if op == 20 and stop_at_ret:
break
return out
def call_sites(b):
"""Every `call` in the code region. Scans on the encoding, not by decoding,
so a bad length somewhere cannot hide the rest of the file."""
code_end = struct.unpack_from('>I', b, 0x0C)[0] # symtab1 = end of code
out = []
off = struct.unpack_from('>I', b, CODE_BASE_FIELD)[0]
while off + 12 <= code_end:
w = struct.unpack_from('>I', b, off)[0]
if (w & 0xFF) == 0x13 and ((w >> 8) & 0xFF) == 12 and (w >> 16) == 0:
bid = struct.unpack_from('>I', b, off + 4)[0]
if bid <= 0x92:
out.append((off, bid, struct.unpack_from('>I', b, off + 8)[0]))
off += 4
return out
def resync(b, target, back=400):
"""Find a start from which linear decode lands exactly on `target`.
Instructions are variable-length, so you cannot simply walk backwards; but a
wrong start almost always desynchronises into an invalid length, so trying
every 4-byte start in a window and keeping the one that hits the target
exactly is reliable in practice.
"""
for start in range(max(0, target - back), target, 4):
off = start
for _ in range(300):
if off >= target or off + 4 > len(b):
break
ln = (struct.unpack_from('>I', b, off)[0] >> 8) & 0xFF
if ln == 0 or ln % 2:
off = -1
break
off += ln
if off == target:
return start
return None
def phase_bases(b):
"""The per-phase code bases, from the mission-level stream's 0x1883 records."""
out = []
off = struct.unpack_from('>I', b, CODE_BASE_FIELD)[0]
for o in range(0x24, 0x100, 4):
if struct.unpack_from('>I', b, o)[0] == 0x1883:
out.append(off + struct.unpack_from('>I', b, o + 4)[0])
return out
if __name__ == '__main__':
b = load(sys.argv[1])
if sys.argv[2:3] == ['--calls']:
import collections
cs = call_sites(b)
h = collections.Counter(bid for _, bid, _ in cs)
print('%d call sites, %d distinct built-ins' % (len(cs), len(h)))
for bid, n in h.most_common():
print(' builtin %-4d %5d site(s)' % (bid, n))
sys.exit(0)
if sys.argv[2:3] == ['--to']:
t = int(sys.argv[3], 0)
st = resync(b, t)
if st is None:
print('could not resync into 0x%X' % t); sys.exit(1)
print('resync from 0x%X' % st)
print('\n'.join(dis(b, st, int(sys.argv[4], 0) if len(sys.argv) > 4 else 40,
sym2=symbols(b, 2))))
sys.exit(0)
code_base = struct.unpack_from('>I', b, CODE_BASE_FIELD)[0]
a = sys.argv[2]
if a == '--entry':
off = code_base + int(sys.argv[3], 0)
else:
off = int(a, 0)
cnt = int(sys.argv[4], 0) if len(sys.argv) > 4 else 40
print('code base 0x%X, disassembling from 0x%X' % (code_base, off))
print('\n'.join(dis(b, off, cnt, code_base)))