From b9cb0135e30d9113ef4e4628a22a0145da415e61 Mon Sep 17 00:00:00 2001 From: Sylpheed RE agent Date: Mon, 24 Aug 2026 13:13:43 +0000 Subject: [PATCH] =?UTF-8?q?re:=20sites=20are=20entities=201:1=20=E2=80=94?= =?UTF-8?q?=20withdraw=20"n=20is=20craft-per-member"?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The suspected confound turned out not to exist. Gaps between consecutive same-unit definition-pointer sites are all >= 0x1000, with 274 of them exactly 0x1000, so entities are page-spaced and there are no near-adjacent pairs to merge. Clustering at any threshold below 0x1000 gives ratio 1.00 for every unit type, and hull is plausible on 298 of 298 clustered bases at delta 0x130. The player shows two objects because there are two, not because one holds two pointers. That removes the excuse the previous iteration had used to keep the reading alive, and the reading does not survive: sum(n) fits the turret row well (216 against 214, with kills already recorded), but DeltaSaber_T, Player and Acropolis all come out at exactly twice their sum(n). An undershoot can be blamed on phases 2-3 not having started; an overshoot cannot. n goes back to ❔ and the previous 🟡 is withdrawn. All the turret row establishes is that a roster member expands into many craft, not that n is the factor. Formation slot count was tested as the alternative and rejected outright: 630 turret slots against 214 live. Side result worth keeping: a FormationSet record's FrameCount is its slot count, and the name suffix usually agrees -- Turret07_30 -> 30, ArrowHead03_64 -> 64, 4_Bird -> 4 -- with one exception, AttackerS03_12 having 14 slots, so the suffix is a label and not a guarantee. Also recorded: the 298 live entities are not the 116 roster records. Both structures exist at once, and the rule mapping one onto the other is the real open question. Probe caveat noted in the doc: entities2.moving() found no movers this run, so the delta spectrum was empty and the clustering threshold was a fallback rather than a measurement. It does not change the conclusion, since every gap exceeds any threshold below 0x1000. --- docs/re/BACKLOG.md | 12 ++-- docs/re/mission-liveness-probe.md | 70 +++++++++++++++++++++ tools/re-capture/dedupe_probe.py | 98 ++++++++++++++++++++++++++++++ tools/re-capture/dedupe_session.sh | 16 +++++ 4 files changed, 192 insertions(+), 4 deletions(-) create mode 100755 tools/re-capture/dedupe_probe.py create mode 100755 tools/re-capture/dedupe_session.sh diff --git a/docs/re/BACKLOG.md b/docs/re/BACKLOG.md index cc42d006..39b15984 100644 --- a/docs/re/BACKLOG.md +++ b/docs/re/BACKLOG.md @@ -326,10 +326,14 @@ search cannot find a *schedule*. (monotone 298→280). The hunting pilot **does** kill (hull crossing at t=57s). 🔴 "No births in 164 s" does NOT separate the wave models — everything is pre-allocated, so an arrival must be a STATE CHANGE, not an allocation. - 🟡 Turret `sum(n)`=216 vs 214 sites found (vs `Count`=21) suggests the mystery - member field `n` is the craft count per member — NOT promoted: the probe counts - definition-pointer *sites*, and the player (1 member → 2 sites) proves some - entities hold several. **Next: collapse sites into entities, then re-test.** + 🔴 **`n` = craft-per-member is WITHDRAWN** (re-tested 2026-08-24): sites really + are entities 1:1 (all same-unit gaps ≥0x1000, hull plausible 298/298), so the + confound was not the explanation — and with it gone, `DeltaSaber_T`, `Player` + and `Acropolis` all come out at exactly **2×** `sum(n)`. An undershoot is + explainable by phases; an overshoot is not. `n` back to ❔. Formation slot count + also rejected (630 turret slots vs 214 live). ✅ Side result: `FormationSet` + `FrameCount` = slot count. ❔ **Open: the 116 roster records and the 298 live + entities are different structures; the expansion rule between them is unknown.** * 🟡 **COMPETING MODEL (user, 2026-08-24): waves are event-gated, not clock-driven** — released by kills/objectives rather than at a fixed time, with the route's start time being a post-trigger delay. Fits the evidence better: diff --git a/docs/re/mission-liveness-probe.md b/docs/re/mission-liveness-probe.md index 252181c8..11309519 100644 --- a/docs/re/mission-liveness-probe.md +++ b/docs/re/mission-liveness-probe.md @@ -84,3 +84,73 @@ implied entity base and take distinct bases. Then the table above becomes a real test rather than a suggestive one. The same de-duplication is a precondition for using this probe to watch an arrival, since an arrival is now known to be a state change rather than an allocation. + +--- + +# Sites really are entities — and `n` does not survive the fix (2026-08-24) + +## ✅ The multiplicity confound is refuted + +The worry was that some entity types hold several pointers to their definition, +so "sites" over-counted entities. Measured, it does not happen: + +* **Gaps between consecutive same-unit sites are all ≥ `0x1000`**, and 274 of + them are *exactly* `0x1000`. Entities are page-spaced; there are no pairs + sitting a few bytes apart that a clustering pass could merge. +* Clustering at any threshold below `0x1000` therefore gives **ratio 1.00 for + every unit type** — 298 sites, 298 entities. +* **Hull is plausible (`0 < h ≤ 200000`) on 298 of 298** clustered bases at + delta `0x130`. These are real objects, not stray pointer words. + +So the previous run's counts were right after all, and the reason the player +shows two is that there genuinely are two such objects — not two pointers to one. + +*(The delta spectrum came back empty this run: `entities2.moving()` found no +movers in its sample window, so `find_delta` had nothing to vote on. It does not +affect the conclusion — every gap is ≥ `0x1000`, so any threshold below that +gives the same clustering — but the threshold was a fallback, not a measurement, +and that is worth knowing if this probe is reused.)* + +## 🔴 With the confound gone, the `n` = craft-per-member reading fails + +| unit | members | `sum(n)` | formation slots | **live entities** | +|---|---|---|---|---| +| `UN_e007_ADAN_Turret` | 21 | 216 | 630 | **214** | +| `UN_e010_ADAN_Attacker_S` | 9 | 45 | 126 | **32** | +| `UN_f001_TCAF_DeltaSaber_T` | 7 | 7 | 23 | **14** | +| `UN_e106_ADAN_Destroyer` | 19 | 19 | 23 | **14** | +| `UN_f106_TCAF_Destroyer` | 14 | 16 | 16 | **10** | +| `UN_e105_ADAN_Cruiser` | 7 | 7 | 7 | **6** | +| `UN_f105_TCAF_Cruiser` | 11 | 11 | 11 | **4** | +| `UN_f001_..._Player` | 1 | 1 | 3 | **2** | +| `UN_f101_TCAF_Acropolis` | 1 | 1 | 6 | **2** | + +`sum(n)` fits the turret row well (216 against 214, with kills already recorded +before the sample). It does **not** generalise: `DeltaSaber_T`, `Player` and +`Acropolis` all come out at exactly **twice** their `sum(n)`, and an undershoot +can be blamed on phases 2–3 not having started but an *overshoot* cannot. + +Two different multipliers are at work and only one of them is explained. **`n` +goes back to ❔** — the 🟡 in the previous section is withdrawn. What the turret +row shows is only that a roster member expands into many craft; it does not show +that `n` is the expansion factor. + +## 🔴 Formation slot count is not the expansion factor either + +Tested because the formation names carry a size suffix. `FormationSet_S02.tbl` +gives 630 turret slots against 214 live — an overshoot of 3×. Rejected. + +## ✅ Side result: `FormationSet` `FrameCount` is the slot count + +The frame count of a formation record is the number of slots, and the name +suffix usually agrees: `Formation_ADAN_Turret07_30` → 30, +`Formation_TCAF_ArrowHead03_64` → 64, `Formation_4_Bird` → 4. One exception +found: `Formation_ADAN_AttackerS03_12` has 14 slots, not 12, so the suffix is a +label rather than a guarantee. + +## Also worth recording + +The live entity population (**298**) is not the 116 `0x820af030` roster records. +Both structures exist simultaneously: 116 records, one per `UnitGroup` member, +and 298 heap objects, one per actual craft. Whatever maps one onto the other is +the expansion rule, and it is still unknown. diff --git a/tools/re-capture/dedupe_probe.py b/tools/re-capture/dedupe_probe.py new file mode 100755 index 00000000..6ab6e2d1 --- /dev/null +++ b/tools/re-capture/dedupe_probe.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +"""Collapse definition-pointer SITES into distinct entities, then re-test `n`. + +liveness_probe.py counts every aligned word equal to a unit-definition VA. Some +entity types hold more than one such pointer -- the player is one roster member +and yields two sites -- so its counts are sites, not entities, and the +`sum(n)` == entities test could not be believed (mission-liveness-probe.md). + +This measures the multiplicity instead of assuming it: + 1. the delta spectrum -- how many distinct (entity -> def pointer) offsets exist; + 2. the gap distribution between consecutive sites of the SAME unit type, which + shows whether sites cluster in pairs; + 3. a clustering threshold derived from that gap distribution, not guessed; + 4. hull sanity: a real entity's f32 at +0x154 should be a plausible HP. +""" +import os, sys, collections, struct +sys.path.insert(0, __file__.rsplit('/', 1)[0]) +import gmem, gworld, entities2 + +HULL_OFF = 0x154 + +def sites(fd, defs): + lo, hi = gmem.va_to_off(entities2.ENT_VA_LO), gmem.va_to_off(entities2.ENT_VA_HI) + out, pos = [], lo + while pos < hi: + n = min(1 << 24, hi - pos) + blob = os.pread(fd, n, pos) + for k in range(0, len(blob) - 3, 4): + nm = defs.get(blob[k:k+4]) + if nm: out.append((pos + k, nm)) + pos += n + return out + +def f32(fd, off): + try: return struct.unpack('>f', os.pread(fd, 4, off))[0] + except Exception: return None + +def main(): + w = gworld.World(); fd = w.fd + defs = entities2.definitions(w) + print('definitions: %d' % len(defs)) + if not defs: print('NOT IN A MISSION'); return 2 + S = sites(fd, defs) + print('sites: %d' % len(S)) + print('by unit:', collections.Counter(n for _, n in S).most_common(8)) + + # (1) delta spectrum, measured on movers + movers = entities2.moving(fd, w.size) + votes = entities2.find_delta(fd, defs, movers) + print('\n--- delta spectrum (entity -> def pointer offsets) ---') + for d, c in votes.most_common(8): + print(' %+#08x seen %d' % (d, c)) + + # (2) gaps between consecutive sites of the same unit + bygroup = collections.defaultdict(list) + for off, nm in S: bygroup[nm].append(off) + gaps = collections.Counter() + for nm, offs in bygroup.items(): + offs.sort() + for i in range(1, len(offs)): gaps[offs[i] - offs[i-1]] += 1 + print('\n--- gaps between same-unit sites (smallest 10) ---') + for g, c in sorted(gaps.items())[:10]: print(' %#08x (%6d) x%d' % (g, g, c)) + + # (3) cluster with a threshold taken from the spectrum, not guessed + span = max(votes.most_common(4), key=lambda kv: kv[0])[0] - min( + d for d, _ in votes.most_common(4)) if len(votes) > 1 else 0 + thresh = max(0x40, abs(span) + 4) + print('\nclustering threshold from delta spread: %#x' % thresh) + ents = collections.Counter() + for nm, offs in bygroup.items(): + offs.sort(); last = None + for o in offs: + if last is None or o - last > thresh: ents[nm] += 1 + last = o + print('\n--- sites vs clustered entities ---') + sc = collections.Counter(n for _, n in S) + print('%-34s %7s %9s %6s' % ('unit', 'sites', 'entities', 'ratio')) + for nm, c in sc.most_common(): + e = ents[nm] + print('%-34s %7d %9d %6.2f' % (nm, c, e, c / e if e else 0)) + print('\ntotal sites=%d entities=%d' % (len(S), sum(ents.values()))) + + # (4) hull sanity on clustered bases for the top delta + d0 = votes.most_common(1)[0][0] if votes else 0x130 + ok = bad = 0 + for nm, offs in bygroup.items(): + offs.sort(); last = None + for o in offs: + if last is None or o - last > thresh: + h = f32(fd, o - d0 + HULL_OFF) + if h is not None and 0 < h <= 200000: ok += 1 + else: bad += 1 + last = o + print('hull plausible on %d clustered bases, implausible on %d (delta %#x)' % (ok, bad, d0)) + return 0 + +if __name__ == '__main__': + sys.exit(main()) diff --git a/tools/re-capture/dedupe_session.sh b/tools/re-capture/dedupe_session.sh new file mode 100755 index 00000000..21d54bed --- /dev/null +++ b/tools/re-capture/dedupe_session.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash +set -u +export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy DISPLAY=:98 +export PYTHONPATH=/sylph-home/.local/lib/python3.12/site-packages +SD="$(cd "$(dirname "$0")" && pwd)" +SECS="${1:-180}"; EVERY="${2:-10}"; HUNT="${3:-1}" +CFG=/tmp/nav-live.json +"$SD/launch_mission.sh" fly || { echo "BOOT FAILED"; exit 1; } +if python3 "$SD/entities2.py" self 0x130 "$CFG" >/dev/null 2>&1; then + SYLPH_HUNT="$HUNT" SYLPH_KILL_TURRETS=1 nohup python3 "$SD/pilot.py" "$CFG" "$SECS" \ + /tmp/live-pilot.log 2>&1 & + PILOT=$!; echo "--- pilot (SYLPH_HUNT=$HUNT)" +else PILOT=""; echo "--- BIND FAILED, no pilot"; fi +python3 "$SD/dedupe_probe.py" ; rc=$? +[ -n "$PILOT" ] && kill "$PILOT" 2>/dev/null +echo "LIVENESS DONE rc=$rc"