aea529e7dc
ambiguous, the sampling was Continuing last iteration's amber rather than starting something new. The fix was already named there: stop using 5-second screenshots and record the display. Recorded with ffmpeg x11grab at 10 fps and matched every 0.5 s. Across the 25 consecutive samples from capture 5.0 s to 17.5 s the S00A playhead is strictly non-decreasing -- 1.0 through 11.0 s, advancing at essentially real time -- with scores at or above 0.96 and four of them at 0.999 or 1.000 against a runner-up in the 0.78-0.83 range. S00A is the top match on 23 of the 27 frames carrying signal. So MS00A -> S00A.wmv is decoded from the manifest AND measured off the game, and the intro begins about 4.5 s after A on the save slot. The previous attempt's failure is worth its own METHOD line, because it did not look like a sampling problem: it looked like weak evidence for the wrong film. Sparse sampling produced contrast-23 frames, a playhead that would not join up, and one frame preferring ADV. Sampling does not weaken a signal gracefully; it turns it into noise shaped like a different answer. One aside recorded and not chased: in the S00A 5.5-10 s window, ADV also scores 0.97-0.99 at its OWN monotone playhead of 33.5-37.5 s. Two films matching strongly with two consistent playheads is not noise -- it reads as the boot movie being a trailer cut from the story cutscenes, which also explains why the sparse run kept flipping between them.
206 lines
12 KiB
Markdown
206 lines
12 KiB
Markdown
# Method traps already paid for
|
||
|
||
Each line cost an iteration at least once. They are general — they are not about
|
||
Sylpheed, they are about how this kind of measurement goes wrong.
|
||
|
||
Like [`REFUTED.md`](REFUTED.md), this list had been living in the autonomous
|
||
agent's loop prompt, i.e. nowhere durable. See [`README.md`](README.md) for the
|
||
✅/🟡/❔ confidence convention itself.
|
||
|
||
## Controls
|
||
|
||
* **Every result needs a control. A control that fails kills the instrument.**
|
||
* **Run the known-positive through a new filter FIRST.** Three filters have been
|
||
killed by their own control. When one fails, **read the known-good's
|
||
disassembly** before assuming a shape.
|
||
* **A measured negative is a result** — but a negative is only as strong as the
|
||
route you ran, so **state its reach**.
|
||
* **A null result needs its cause shown to have happened.**
|
||
* **A result with NO unknowns is suspicious.**
|
||
* **Census the whole set; always run the other population as the control.**
|
||
**Zero partials is stronger than a majority.**
|
||
* **A 2×2 partition is the sharpest general tool** — both off-diagonals empty is
|
||
a law.
|
||
* **Re-derive a doc's own numbers as the control.**
|
||
|
||
## Inference
|
||
|
||
* **Never conclude from ONE sample.**
|
||
* **A law proved on one population is a hypothesis on the next.**
|
||
* **Finding one exception does not imply a family.**
|
||
* **Consistency is not proof. A suggestive coincidence is a coincidence until
|
||
measured.** **An analogy is not a measurement.**
|
||
* **Same layout ≠ same instance.** **Same record-name set ≠ same object.**
|
||
* **A marker is only proven by what it leaves out.**
|
||
* **A high-confidence SCORE is not a high-confidence MECHANISM.**
|
||
* **Knowing HOW MANY is not knowing WHICH.**
|
||
* **Round numbers matching is weak evidence — unless you read the constant.**
|
||
* **My own last-turn result is a hypothesis too.**
|
||
* **A global partition can understate a per-owner one.**
|
||
* **A residual is measured against a population — name it.**
|
||
|
||
## Searching and tooling
|
||
|
||
* **A search that returns thousands has no power; state the reach.**
|
||
* **A substring match is not a hit.** **A regex miss looks like a null result —
|
||
print one raw sample before believing a zero.**
|
||
* **A derived table can be a cross product — measure its shape first.**
|
||
* **After refuting an instrument, sweep everything that depended on it.**
|
||
* **Before measuring how wrong a tool is, read what the tool actually does.**
|
||
* **The instrument must pass its own control.**
|
||
* **Classify a bulk before mining it. The residual is the prize.**
|
||
* **Rank by similarity — the cliff is the finding.** But **read the values
|
||
before trusting the rank.**
|
||
* **Grep the nouns before designing the experiment — and believe it.**
|
||
* **Grep gives you a file list — read *every* file on it.**
|
||
* **The answer is often already in the doc that owns the subject — read it end
|
||
to end.** A 🟡 often names its own route.
|
||
* **Before re-trying a blocked idea, check whether the blocker's own doc already
|
||
tried it.**
|
||
* **Ship a regenerator with every artefact.** An artefact that moves by a pure
|
||
reorder is a tool bug.
|
||
* **Never print per-entry lines from a disc-wide sweep — aggregate.**
|
||
|
||
## Reading the data
|
||
|
||
* **Read what a loader NAMES, not where it stores.**
|
||
* **A field the disc never values still gets named by the loader.**
|
||
* **An indexed read beats a deduped-pool adjacency read.**
|
||
* **Check the whole string set, not the one matching word.**
|
||
* **A dict keyed by record name across a multi-entry pak is a lie.**
|
||
* **A set-difference over names hides reuse — join per USER.**
|
||
* **A self-index names records, not files.**
|
||
* **Case-insensitive hashing means two spellings can be one entry.**
|
||
* **An "unresolved" name may be the wrong kind, namespace or prefix — or part of
|
||
a cut asset.**
|
||
* **A garbled value may be a real string in another encoding.**
|
||
* **Two of my own counts disagreeing is a grammar clue.**
|
||
* **A game's own typo is a join key.**
|
||
* **A bias constant in the code is a join key.**
|
||
* **A prefix trap: enumerate maximal `[A-Za-z0-9_]` runs, not `startswith`.**
|
||
* **Re-deriving a format is not a finding — asking whether its values *resolve*
|
||
is.**
|
||
|
||
## Mechanics that have bitten
|
||
|
||
* **Never hand-convert a decimal VA — print `hex()`.**
|
||
* **`grep -c` counts LINES** — use `grep -o | wc -l`.
|
||
* **`Counter.most_common()` tie-breaks by insertion order — use `sorted()`.**
|
||
* **Raw grep cannot see inside compressed pak entries.**
|
||
* **Commit messages go in a file** (`git commit -F`); a literal `|` in a table
|
||
cell needs escaping; `git log --all -- <path>` can hang.
|
||
|
||
## Runtime / emulator
|
||
|
||
* **Look at the PNG** — and check its dimensions.
|
||
* **"Animating" is not "still in a mission".**
|
||
* **Dedup entity enumerations by position value.**
|
||
* **Do not diagnose timing or liveness under gdb.** `ps %cpu` is cumulative.
|
||
* **Classify screens by whole-image statistics, not named pixels** — a named
|
||
pixel is only valid while the image sits at a known place, and nothing errors
|
||
when it moves.
|
||
* **Do not poll faster than the guest updates** — it manufactures a clean curve
|
||
out of noise.
|
||
* **A probe that never performs the action will "prove" the action does not
|
||
exist.**
|
||
* **The container's Canary binary can be older than the Canary source tree, and
|
||
the failure mode is a hang, not an error.** After a merge into `sylpheed-re`
|
||
the prebuilt `xenia_canary` had no `log_ui_draws`, no `mem_watch`, no
|
||
`create_profile_if_none` — and an unknown cvar makes xenia open an SDL message
|
||
box before logging is up, which headless is an unexplained freeze. Check before
|
||
trusting a harness flag: `nm -C <binary> | grep cvars::<flag>`, and
|
||
`build-canary Release` if it is missing.
|
||
* **A capture armed *at* a screen only ever sees the steady state.** Anything
|
||
about how a screen is built or animated has to be armed *before* it exists.
|
||
Re-arming every few seconds and keeping every log tiles the approach: each F10
|
||
opens a new numbered file and closes the previous one complete.
|
||
* **Measure animation in submitted frames, not in seconds.** `VdSwap` counts are
|
||
the guest's own frames, so an emulator at 80 % of real time does not move them;
|
||
a stopwatch reading does, silently and by an unknown factor.
|
||
* **~~This game's menus drop d-pad presses shorter than ~0.3 s.~~ WITHDRAWN
|
||
2026-08-28 — the menu WRAPS, and I had not measured that.** The claim came from
|
||
reading a cursor that ended up "one item short"; once wrap-around at both ends
|
||
was measured ([`menu-navigation-semantics.md`](menu-navigation-semantics.md)),
|
||
every one of those press counts is exactly right — four presses at 0.12 s moved
|
||
four steps *through the bottom*, which lands one above where a non-wrapping
|
||
menu would put it. **No press was ever dropped.** The real lesson is the
|
||
general one: *a step count is only readable once you know the topology*, and I
|
||
invented a hardware-flakiness story rather than testing the ends of the list.
|
||
Still true and worth keeping: screenshot after every step and read the cursor,
|
||
rather than trusting arithmetic over the press count.
|
||
* **A screenshot taken right after a transition can catch a screen mid-fade.**
|
||
A grab 2.5 s after Ⓑ returned the game to the title showed the title art with
|
||
no `PRESS Ⓐ BUTTON` plate; one second later the plate was there. That very
|
||
nearly went into the corpus as "the returned title has no plate". Sample a
|
||
changing screen several times before writing down what it does *not* contain.
|
||
* **Do not identify a menu cursor by label brightness.** The obvious oracle —
|
||
"the focused label is the brightest row" — fails on this game's menus, because
|
||
the background art is brighter behind some rows than the highlight is. It
|
||
confidently named the wrong item on a frame whose ring was plainly elsewhere.
|
||
Detect the **focus ring** in the gutter left of the labels instead
|
||
(`tools/re-capture/menu_focus.py`, 254 vs <82 — no threshold tuning needed),
|
||
and look at the PNG before believing either.
|
||
* **`screenshot` samples at 0.5 Hz — it cannot time an animation.** Measured:
|
||
~2 s per grab (an `import` of the root plus an ImageMagick crop). A 0.4 s fade
|
||
falls entirely between two samples, which is why a 40-frame burst across a
|
||
screen change looked like an instant cut. For anything timed, record the
|
||
display instead: `ffmpeg -f x11grab -framerate 30 -video_size <W>x<H> -i
|
||
:98+<X>,<Y> -t <secs>`, then read per-frame statistics off the file. Take the
|
||
geometry from `xwininfo -root -tree`, the same way `bin/screenshot` does.
|
||
* **A screen's brightness curve is not its fade quad.** The incoming screen's own
|
||
elements animate in *after* the transition quad has cleared, so mean luminance
|
||
keeps rising long after the fade is over — 1.47 s against a declared 0.97 s on
|
||
one screen. Time the fade from where the frame is *pure black*, and take the
|
||
ramp itself from the keyframes.
|
||
* **A `sound.pak` entry is not a bank.** Entry windows straddle bank boundaries
|
||
(the leading-region effect), so a naive "walk the RIFFs in this entry" gives
|
||
the tail of the previous bank, then one of your own, then the *next* bank's
|
||
header. It looks like a bank with the wrong number of waves. The tell is a
|
||
`BANK id=` that is not the entry's own number, and the fix is the `seek` packet
|
||
counts — they pin each wave's join exactly, so a straddle can be realigned
|
||
rather than thrown away. 4 of 32 BGM banks look like counterexamples until this
|
||
is applied.
|
||
* **Do not time a video whose start you did not see.** An attract cycle sampled
|
||
from partway through measured "≈85 s", which was used to argue it was not the
|
||
137 s `ADV.wmv`. It was `ADV.wmv`; the sampling began 39 s in. Identify a video
|
||
by **content** — match frames against the candidates and check the matched
|
||
playhead advances monotonically — before reasoning about its duration.
|
||
* **A near-black frame has no signature.** In frame-matching, low-contrast frames
|
||
drive every correlation toward zero and the ranking becomes noise; one control
|
||
frame scored 0.000 and another tied the wrong movie at 0.98. Read a dark
|
||
frame's runner-up as *no evidence*, not as evidence for the runner-up.
|
||
* **A screen that never changes is not necessarily hung — it may be a menu.** A
|
||
90-second screenshot loop that presses nothing will report a perfectly healthy
|
||
`DIFFICULTY` menu as a standing hang, and that reading survived in this corpus
|
||
for months. Before calling a static screen a hang, *look at it*, and press
|
||
something.
|
||
* **Never run `ps -ef` in this container.** Each of the three long-lived
|
||
processes carries the whole loop prompt as its argv, so the listing is tens of
|
||
kilobytes of the prompt you already have. Use `pgrep -x <name>` or
|
||
`ps -o pid=,stat= -C <name>`.
|
||
* **A file extension is not a format.** `sound.pak` is full of `.slb` banks and
|
||
names a `.xgs` project, and neither is an XACT container: the whole 1.08 GB
|
||
archive contains zero `XGSF`, `SDBK` or `WBND` magic, and the executable has no
|
||
XACT string at all. The names came from the authoring tool and survived into the
|
||
shipped data. Check for the magic before committing to a parser for the format
|
||
the extension advertises.
|
||
* **A log-polling probe must compare a COUNT, not test for presence.** Timing an
|
||
operation by waiting for its "done" line in a log gives an instant false
|
||
positive if any earlier run left one there. This produced a confident
|
||
"39.5 fps" that was really 0.1 s of matching a stale line — and it disagreed
|
||
with the true 28.5 fps by enough to have flipped a conclusion. Snapshot
|
||
`log.count(marker)` before starting and wait for it to *increase*.
|
||
* **This container kills long emulator runs.** Twice in one session the
|
||
`xenia_canary` process vanished with **no crash line in its own log** — once
|
||
~50 s into a boot, once ~145 s in at 1.15 GB RSS — while the box sat at
|
||
~1 GB free with swap exhausted. Budget dynamic experiments to fit in ~2 minutes
|
||
of guest time, take the measurement early, and never read a disappearance as a
|
||
guest crash without checking the log tail for one.
|
||
* **When a frame-match will not resolve, sample denser before doubting the
|
||
hypothesis.** Identifying a movie from 5-second screenshots gave contrast-23
|
||
frames, a playhead that would not join up, and a runner-up winning one frame —
|
||
it looked like weak evidence for the wrong film. The same question sampled every
|
||
0.5 s (`ffmpeg x11grab` at 10 fps) answered at **0.999 with a strictly monotone
|
||
playhead**. Sparse sampling does not weaken a signal gracefully; it turns it
|
||
into noise that looks like a different answer.
|