Syplheed-Reborn/docs/re/autopilot-memory-driven.md

Memory-driven autopilot — build log and current state

Status: 🟢 IT FLIES, KILLS AND SURVIVES — but it loses the mission anyway. Updated 2026-07-30. pilot.py flew Stage 02 for 300 s with the hull untouched at 1500/1500 and scored the first confirmed autopilot kill (YOU KILLED WARPLANES 0001 on the HUD, screenshots shots/pilot1-*.png); the scene's hostile count fell from 134 to 111 over the run. The day before, every run was dead inside 35 s. What is still missing is the end of a mission: the objective counter (REMAINING OB) rises as new waves spawn, and nothing yet tracks which targets actually close it out — and the second run proved the point the hard way: GAME OVER with the hull at 1500/1500, because Stage 02 is an escort and the ACROPOLIS was sunk while the pilot chased fighters two kilometres away.

2026-07-30 — the numbers survival needs

Three things the loop was missing were measured this session, each by consequence rather than by reading a field and hoping.

Hull is position + 0x154 ✅ CONFIRMED

The unit definition already had HP solved at +0x054 (unit-struct-runtime); the Delta Saber's is 1500. A craft that has taken no damage must therefore contain that number, which turns "find the HP field" into a two-float lookup rather than a value scan (own_state.py). It appears once in the entity object, at pos+0x154, and the trace across a death settles it (ctrl_probe.py capture, binq.py trace):

  t      phase        hull
  0.00   base         1500.00      <- == definition HP
 23.25   rest_A       1380.00      <- first hit, -120
 26.68 … 27.18  B     1320 … 930   <- seven hits in 0.5 s
 31.93   X            150.00
 35.21   Y            -30.00       <- goes negative
 35.26   Y           -180.00       -> GAME OVER on screen

Damage arrives in 30/60/90-point steps and the field goes negative at death, so it is the raw hull counter, not a clamped display value. 1500 hull lost in 12 s of sitting in a turret's line of fire is the whole reason every earlier run died.

Shield is position + 0x430 🟡 PROBABLE — not yet confirmed live

Same anchor trick: the definition's shield MaxValue is 400 and ChargeSpeed 25, and the entity object holds 400.0 at +0x430, +0x434 and +0x438, with 25.0 at +0x448. Which of the three is the current value is unproven — the capture that spanned the death used a ±0x400 window and cropped them out. ctrl_probe.py now samples ±0x800.

RT accelerates, LT brakes, and the throttle is a setting ✅ CONFIRMED

ctrl_probe.py holds each input in turn and measures the craft's own speed as displacement per second from the position triple, so no speed field is needed. Distance flown / phase duration, one 3 s hold each, sticks neutral:

phase	speed (units/s)		phase	speed (units/s)
base (no input)	488		A	287
RT	1510		B	139
rest after RT	1056		X	125
LT	174		Y	(dying)
rest after LT	428		LB / LS / RS / RY / RX / dpad	no effect

RT triples the speed, LT cuts it to a third, and the braked state persists: after the LT phase the craft sat at 125–140 units/s with the sticks and triggers neutral for the remaining 40 s, and nothing but RT brought it back. So these are a throttle setting, not a momentary boost — which also means a control loop must send only the changes.

This corrects the earlier note in this file ("RT is not the throttle, and no button tested is"). That conclusion came from findspeed.py, which assumed the control and went looking for a field that rose; measuring the speed directly reverses it.

Two method corrections

• Pick the attitude block by the flight path, not by address order. The player object contains 20 orthonormal 3×3 blocks (identity frames, bone or camera frames), and pos-0x70 and pos-0x30 hold the same matrix. Taking found[0] wrote a config with rot_delta = -0x764 and a nonsense forward axis; entities2.py self now scores every block against the measured direction of travel and picks the best (cos = +1.000, row 2, sign +1).
• The entity-heap scan has to be numpy. A per-word Python loop over the 16 MB entity region costs seconds per scan, which is the whole budget of a 10 Hz control loop; np.isin over a >u4 view is milliseconds.

Stage 02 as an autopilot testbed (from the in-flight HUD)

OBJECTIVE: shoot down all invading enemy fighters while watching out for attacks on the ACROPOLIS · DEFEAT: your fighter is shot down, or the ACROPOLIS is sunk · HINT: you can resupply at the ACROPOLIS. The HUD shows REMAINING OB 004 — only four objective targets — so this mission is winnable by an autopilot that survives. It also shows separate SHIELD and ARMOR bars (matching a 400-point shield over 1500 hull), A/B 7,635 afterburner, and NOSE BM 06000 / MAIN MPM 00300 ammo.

What the loop did before that, observed

[ 82.5] tgt=e007_ADAN_Turret  d=3384  yaw= -7.3 pit=+14.6 stick=(-0.13,-0.34) fire=0
[ 85.7] tgt=e007_ADAN_Turret  d=2797  yaw= +1.9 pit=+27.2 stick=(+0.20,-0.59) fire=0
[117.3] tgt=e007_ADAN_Turret  d=4211  yaw= +5.7 pit= +9.0 stick=(+0.25,-0.40) fire=1

Distance closes monotonically, yaw error is driven from −8° to ~0, and once both errors are inside the firing cone it holds RB and the ammo counter falls. A rescan reports the scene as e.g. 136 entities {'TCAF': 16, 'ADAN': 120}.

The chain that made it work — each link checked, not assumed:

1. Entity typing. A live entity's definition pointer sits at position + 0x130. One heap scan then yields every craft with its unit type, which is what separates 20-odd real combatants from ~30 000 moving particles. Verified by the result being coherent: wingmen, enemy turrets and attackers, friendly capital ships, and exactly one …_Player.
2. Orientation. A 3×3 rotation at position − 0x70, stored with a 16-byte row stride (a 4×4 transform whose translation row is the position). An earlier search for nine contiguous floats structurally could not find this, which is why the first pass concluded "no transform". The binding is confirmed independently: its row 2 matches the craft's measured direction of travel with cos = +1.000.
3. The fire button is RB — established by consequence, not by guessing: of RB/LB/A/B/X/Y/RT/LT, pressing RB is the only one that makes the nose-ammo counter in RAM fall (5958 → 5940). (This entry also claimed RT is not the throttle — wrong, see the 2026-07-30 measurement above.)
4. Control. PD on the aiming error with the derivative taken from the craft's own body angular velocity (from two consecutive rotation matrices), and target selection weighted by off-boresight angle (score = d·(1 + 3·(θ/π)²)) rather than pure nearest — closing on a target 90° off the nose only raises the bearing rate, which is what held the first run outside its firing cone at a steady ~27° pitch error.

Goal

Fly and fight a mission by reading the game's own world state out of guest RAM and driving the pad from it — the RE payoff being an oracle for the reimplementation's flight model and AI, and a way to reach missions the save cannot otherwise reach (unit coverage for unit-struct-runtime is capped at 21/110 because definitions load per stage).

What works (verified)

Piece	Tool	Evidence
Live guest-RAM reads at loop rate	gworld.py	/dev/shm file opened once, pread per tick; a whole-RAM scan is ~6 s, a targeted read is microseconds
Whole-RAM float scanning	numpy over SEEK_DATA extents	1 270 orthonormal 3×3 blocks located in 6.2 s
Entity enumeration by unit type	gworld.py entities	116 live instances in Stage 02, typed by unit ID, incl. exactly one …_Player
Pad control at loop rate	flight_probe.Pad	writes command lines straight into the vgamepad FIFO; the vgamepad CLI spawns a process per command and its tap/hold sleep inside the server, so neither is usable in a control loop
Unattended mission entry	launch_mission.sh	title → LOAD GAME → slot 01 → READY ROOM → TAKE OFF → in flight, repeatable
Hangar loadout	launch_mission.sh --hangar	the "Recommended" control is AUTO SELECT — "Mount most suitable weapons", already the default cursor position. At 5 % progress it is a no-op: only two weapons are developed, and they are already mounted
Finding moving objects	findplayer.py	position triples recovered from motion alone — straight-line, constant-speed filter over K whole-RAM samples

What is NOT solved

Survival, and therefore mission completion. The loop has no evasion, no shield/armour awareness and no throttle control, so it flies a straight pursuit into defended space and is eventually shot down — every long run so far has ended in GAME OVER. Completing a mission needs, at least: reading own shield/armour, breaking off when hit, and prioritising the mission's actual objective targets over the nearest turret.

Superseded (kept because the reasoning still matters)

The notes below were written before the chain above worked. They remain true as statements about 0x820af030, which is not the live entity —

1. The 0x820af030 class is not the live entity. It has one object per spawned thing and carries the unit-ID string, so it looked like the entity list — but over a 29 s in-flight capture all 384 words of it are constant (whatchanges.py). It is a static spawn record. The earlier claim in unit-struct-runtime.md that this class is "the spawned entity instance — live state" is wrong in the second half: it is per-spawn, but it is not live state. The parts of that document that depend on the definition class 0x820af844 are unaffected.
2. No transform in or one hop from that object: no orthonormal 3×3, and no unit quaternion, within 0x2000 of it or behind any of its 121 pointers.
3. Input correlation finds a self-object, but not obviously the craft. Holding hard-left then hard-right yaw and looking for a position whose turn axis reverses (findself.py, selfstate.py) gives clean hits (cos ≈ −0.99), but they cluster at 0x40009xxx in what looks like an 8-corner box with ±45 000 coordinates — a camera/skybox volume that follows the player, not the craft. Its speed (≈359/s) is suspiciously close to the HUD's 350, which supports "follows the player" but is not proof of identity.
4. Entity typing is unavailable: no definition pointer within ±0x800 of the self-position, so the trick of learning one object's layout and applying it to all the others has nothing to anchor on. Without typing, the 33 418 moving triples in a firefight cannot be separated into enemies, friendlies and bullets, so there is nothing to aim at.

Dead ends, recorded so they are not re-run

• Speed-scan for the player object (findspeed.py, the classic two-state value scan: coast → boost → coast). Sound method, but RT is not the throttle — 5 864 floats matched the cruise speed and none rose. The control actually bound to acceleration was never established, and the run that would have established it ended in GAME OVER.
• Comparing orientation matrices 2 s apart. At a real turn rate that is far outside the small-angle regime, so the skew part of A·Bᵀ is not the rotation vector and the "angular velocity" comes out as ~30 000. Sample incrementally (6 Hz) and re-check orthonormality on every read — blocks found by a scan get overwritten between the scan and the read.

The second run lost the mission without being hit (2026-07-30)

A second 240 s flight, with the two fixes above, ended on the GAME OVER screen — while the hull read 1500/1500 on the last live tick. Nothing shot us down. The other defeat condition fired: the ACROPOLIS is sunk. The HUD had been showing a red WARNING banner for a while, and the pilot spent the whole run pursuing an e010_ADAN_Attacker_S two kilometres away.

So surviving is necessary and not sufficient, and "nearest hostile fighter" is the wrong objective function for this stage. The mission is an escort. What follows:

• Prioritise hostiles by their distance to the protected asset, not to us. The attackers worth killing are the ones closing on the ACROPOLIS.
• The protected asset's health is readable with the same anchor as ours — hull at position + 0x154, its maximum being its own definition's HP. That gives a live "are we winning" signal for the escort, and it should drive the target choice directly.
• A frozen tail in the log (identical position, speed and target for the last five seconds) is what mission-end looks like from the outside, not an emulator wedge. Worth knowing before diagnosing the wrong thing.
• Practical: do not pipe a long run's log through tail — that discards everything but the end, and the interesting part of this run is gone.

After survival, the blocker is lethality (2026-07-30)

The 300 s run took no damage at all and killed one warplane, spending ~800 rounds of nose ammo (06000 → 05193) to do it, while REMAINING OB rose from 004 to 011 as fresh waves spawned. So attrition at this rate never finishes the mission, and the ranking of open problems has changed:

1. Hit rate. It opens fire at 2–5 km with a 9° cone and a crude lead (p + v·d/speed, no projectile speed). The Shell records in weapon-struct-runtime carry the real projectile speed and MaximumRange per weapon — the lead and the firing range should come from those, not from constants.
2. Which targets count. REMAINING OB is the mission's own objective counter and it is on screen, so it is in RAM; finding it turns "shoot whatever is nearest" into "shoot what closes the mission". Objective-marked entities also draw an OB badge in the HUD, so the flag is likely a word in the entity object.
3. Confirming the shield word — needs a run that actually takes damage; the pilot is now good enough at avoiding that to make it awkward, so drive straight at a turret on purpose with --dry steering disabled.
4. Does the ACROPOLIS repair? RETIRE mode has never triggered (the hull never fell), so the resupply hint is still untested.

The next step that unblocks the most (superseded — kept for the reasoning)

Update 2026-07-30: this is no longer the blocker. Entity typing via the definition pointer already solved target selection, so the game's own target pointer is now a convenience rather than a prerequisite. It would still be the cheapest route to problem 2 above (objective targets), because whatever the HUD locks on to is what the game itself considers a target.

Find the game's own target pointer instead of typing entities ourselves. The HUD has a lock-on system (a TARGET marker and a target-cycle button), so a global almost certainly holds a pointer to the currently-targeted entity. Reading that gives an enemy's live object address directly — which yields both target selection and the entity layout (position offset within it), i.e. it collapses problems 3 and 4 into one. It is also cheap to find: cycle the target with the pad and watch which pointer-shaped global changes in step.

Operational notes

• An unattended craft dies — the ship flies straight into a firefight, and two long scans were invalidated by a GAME OVER mid-run. Any scan longer than ~30 s needs either a survivable holding pattern or a fresh mission.
• Xvfb and the emulator die on their own every few minutes here, cleanly (exit 0), cause unidentified. Everything that must not be interrupted is run as one background task that starts the display, the emulator, the navigation and the measurement together, so nothing has to survive between tool calls.
• pgrep cannot be used for liveness in this container: PID 1 is sleep infinity and never reaps, so dead processes linger as <defunct> and still match by name. Use ps -o stat= and skip Z, or xdpyinfo for X.
• numpy is not installed system-wide; pip install --break-system-packages numpy puts it in /sylph-home/.local, which is only on sys.path when HOME=/sylph-home. Scripts run with HOME=/sylph-home/re need PYTHONPATH=/sylph-home/.local/lib/python3.12/site-packages.

Files

pilot.py (the survival loop) · ctrl_probe.py (input → speed calibration, plus a per-tick window of the player object) · binq.py (query that capture) · own_state.py (definition-anchored hull/shield lookup) · fly_session.sh (boot → mission → bind → fly, one task) · wait_flight.sh (wait for the real HUD instead of a fixed sleep) · navigator.py (drift-aware steering + CPA avoidance, reused by the pilot) · gworld.py (live reader + entity list) · flight_probe.py (scripted inputs + sampling, and the Pad FIFO client) · flight_analyze.py · whatchanges.py (encoding-agnostic "which words are live") · findplayer.py · findself.py · findrot_global.py · findspeed.py · liveents.py · selfstate.py (the whole chain → JSON) · autopilot2.py (PD controller; untested — it has never had a valid config to run against) · launch_mission.sh.