re: the ISL stream is flat -- refute the "needs coroutine entry points" blocker

Two files (isl_report.py's docstring and structures/isl-builtins.md) recorded the
same blocker on a faithful per-phase condition listing: that it needs the coroutine
entry points from start_coroutine's operand.  Measured against isl.call_sites(),
which enumerates by scanning the encoding rather than by decoding and so is an
independent denominator:

  linear + jumps, stopping at ret (what the tool did)   133 / 2846 =   4.7%
  linear + jumps, continuing past ret                  2275 / 2846 =  79.9%
  ... + following start_coroutine (the recorded fix)   2355 / 2846 =  82.7%
  plain linear decode, no control flow at all          2846 / 2846 = 100.0%

Following the coroutine entries buys 2.8 points.  Disc-wide, a plain linear decode
from the first phase base reaches 25705/25705 call sites over all 28 stages, and
28/28 decode clean to code_end with no desync.

The real bug was isl.dis ending on `if op == 20: break`.  Op 20 is `ret`, but this
is a coroutine VM -- the thread suspends and resumes at the FOLLOWING instruction,
so code continues past it.  dis() now takes stop_at_ret (default True, preserving
the old output: data/isl-stage02.txt regenerates byte-identical) and
isl.linear_offsets() is the correct walk.

By-product, kept with its control: start_coroutine's target is staged slot 0 --
73/83 phase-1 sites land on a valid instruction, against a 38.7% chance rate for an
arbitrary 4-aligned offset.

New artefact data/isl-stage02-phase-ends.txt with a committed generator
(isl_report.py phase-ends).  It shows END_PHASE's call site is the WRONG place to
read a clear condition: all 12 Stage-02 sites sit in one stereotyped outro.  Not
settled, and stated as such: op10/op13/op14/op21/op23 are unread handlers, so the
condition in the poll loop upstream cannot be named yet.

tools/re-capture/isl.py

@@ -213,7 +213,37 @@ SYM1_SLOTS = {
 # not yet established.
 
 
-def dis(b, off, count=40, code_base=0x24, args=True, sym2=None, sym1=None):
+def linear_offsets(b, start=None):
+    """Every instruction offset, decoding linearly from the first phase base.
+
+    MEASURED over all 28 stages: this reaches 25705/25705 of the call sites
+    `call_sites()` finds by scanning the encoding, and every file decodes clean
+    to `code_end` with no desync. The instruction stream is therefore FLAT --
+    reaching a call site needs no control-flow reconstruction at all.
+    """
+    end = struct.unpack_from('>I', b, 0x0C)[0]          # symtab1 = end of code
+    if start is None:
+        start = phase_bases(b)[0]
+    out = []
+    off = start
+    while off + 4 <= end:
+        out.append(off)
+        ln = (struct.unpack_from('>I', b, off)[0] >> 8) & 0xFF
+        if ln == 0 or ln % 2:
+            break
+        off += ln
+    return out
+
+
+def dis(b, off, count=40, code_base=0x24, args=True, sym2=None, sym1=None,
+        stop_at_ret=True):
+    """`stop_at_ret` preserves the ORIGINAL behaviour and is wrong for reading.
+
+    op 20 is `ret`, but in a coroutine VM that is a YIELD: the thread suspends
+    and later resumes at the following instruction, so code continues after it.
+    Stopping there reaches only 133 of Stage 02's 2846 call sites (4.7%);
+    continuing reaches all 2846. Pass `stop_at_ret=False` to read a listing.
+    """
     out = []
     staged = {}          # local[] slot -> last value staged into it
     pending = None       # value most recently put in special[0]
@@ -301,7 +331,7 @@ def dis(b, off, count=40, code_base=0x24, args=True, sym2=None, sym1=None):
             out.append('       (length 0 -- stopping)')
             break
         off += ln
-        if op == 20:
+        if op == 20 and stop_at_ret:
             break
     return out
 

tools/re-capture/isl_report.py

@@ -8,12 +8,13 @@ twice: once when operand staging was fixed (calls printed with too few
 arguments) and once when three built-in names were corrected. Keeping the
 generator in the tree is the point of this file.
 
-`data/isl-stage02-conditions.txt` still has no generator here. A first attempt
-is not committed because it printed most sites as bare `builtinN`: neither
-`isl.resync` (it gives up far from a valid start) nor a naive linear decode from
-the phase base reaches every call site, so producing that listing faithfully
-needs the coroutine entry points, which `start_coroutine`'s operand carries and
-this tool does not yet follow.
+    isl_report.py <StageNN.ssb> phase-ends -> every END_PHASE with its context
+
+The "needs the coroutine entry points" blocker recorded here is REFUTED: the
+instruction stream is FLAT and `isl.linear_offsets` reaches 25705/25705 call
+sites across all 28 stages. What broke the naive linear decode was `isl.dis`
+stopping at op 20 (`ret`), which in a coroutine VM is a yield, not an end of
+code -- see `docs/re/isl-stream-is-flat.md`.
 """
 import collections
 import sys
@@ -61,11 +62,42 @@ def emit_calls(b, path):
         print(line)
 
 
+def emit_phase_ends(b, path):
+    """Every phase-ending call with the instructions that lead into it."""
+    offs = isl.linear_offsets(b)
+    idx = {o: k for k, o in enumerate(offs)}
+    bases = isl.phase_bases(b)
+    s1, s2 = isl.symbols(b, 1), isl.symbols(b, 2)
+    ends = [(o, bid) for o, bid, _ in isl.call_sites(b) if bid in (6, 62)]
+    print('# %s — where each phase ends' % path)
+    print()
+    print('Generated by `tools/re-capture/isl_report.py phase-ends`.')
+    print('Decoded with `stop_at_ret=False`; a `ret` is a coroutine YIELD, so')
+    print('the listing continues past it.')
+    print()
+    print('phase code bases: ' + ' '.join('0x%x' % x for x in bases))
+    print('%d phase-ending call(s): %d END_PHASE, %d FORCE_END_PHASE'
+          % (len(ends), sum(1 for _, b2 in ends if b2 == 6),
+             sum(1 for _, b2 in ends if b2 == 62)))
+    for o, bid in ends:
+        ph = sum(1 for x in bases if x <= o)
+        k = idx.get(o)
+        if k is None:
+            print('\n## 0x%X builtin %d — NOT on the linear stream' % (o, bid))
+            continue
+        print('\n## phase %d — builtin %d (%s) at 0x%X'
+              % (ph, bid, isl.BUILTIN.get(bid, '?'), o))
+        for line in isl.dis(b, offs[max(0, k - 16)], 22,
+                            code_base=bases[ph - 1], sym2=s2, sym1=s1,
+                            stop_at_ret=False):
+            print('    ' + line)
+
+
 def main():
     path = sys.argv[1]
     b = isl.load(path)
     name = path.replace('\\', '/').split('/')[-1]
-    {'calls': emit_calls}[sys.argv[2]](b, name)
+    {'calls': emit_calls, 'phase-ends': emit_phase_ends}[sys.argv[2]](b, name)
 
 
 if __name__ == '__main__':