1a82ade187
Closes the reach caveat the `opt ` name fix left behind: 60 of 18 002 RATC children carry no `opt ` block, and it was not established whether they lack one or sit past our 128-byte window. Neither. They are not children. `examples/ratc_optless_children.rs` re-runs `ratc::parse`'s own guards over the disc and reports which one fired: all 60 are "tag beyond the window", none is rejected by length, gap or charset, none is child #0, and all 60 live in six bundles of one archive. Within a bundle the distances back to the nearest tag are an exact arithmetic progression, step 60 600 -- ten different records finding the SAME tag, because there is only one. Reading a bundle directly: children 1..10 are equal-size T8aD blocks under a single `opt ` name, `pb_f15_eg_anm.tan`. `.tan` is a FRAME SEQUENCE. One block declares the resource; its payload is a run of T8aD frames. Disc-wide, over all 18 718 `opt ` names in all 33 paks: a RATC bundle names exactly six kinds of resource -- `.t32` 14 756, `.rat` 3 311, `.prm` 367, `.tbm` 224, `.sbo` 54, `.tan` 6. Six `.tan`, ten frames each = 60, the entire population with nothing left over. The negative is closed, not narrowed. Consequence recorded but deliberately not fixed: `ratc::parse` over-reports there, listing a `.tan`'s frames as anonymous children. Nothing in the menu milestone reads a `.tan` -- it occurs only in GP_READY_ROOM, which S1 ruled out -- so no screen the port draws changes. Also a METHOD entry for this container OOM-killing `slb_leading_segment_disc` under default test parallelism (SIGKILL, no assertion; 8/8 pass with --test-threads=1).
76 lines
3.1 KiB
Python
76 lines
3.1 KiB
Python
#!/usr/bin/env python3
|
|
"""Every `opt ` name in every RATC bundle on the disc, by extension -- and the
|
|
`.tan` frame sequences among them.
|
|
|
|
Written to close the reach caveat in docs/re/structures/ratc-child-names.md:
|
|
60 of 18 002 RATC children carry no `opt ` block of their own. They are not
|
|
children. They are the ten frames of the disc's only `.tan` resource, and one
|
|
`opt ` block names the whole run.
|
|
|
|
python3 tools/re-capture/ratc_opt_name_census.py
|
|
|
|
Reads $SYLPHEED_DISC/dat/*.pak directly (IPFB TOC + Z1/zlib entries), so it does
|
|
not depend on the Rust parser it is checking. Takes a few minutes.
|
|
"""
|
|
|
|
import struct, zlib, glob, os, collections, bisect
|
|
MAG = (b"T8aD", b"RATC", b"ttcf", b"\x89PNG")
|
|
ext = collections.Counter()
|
|
tan_sites = []
|
|
opt_total = 0
|
|
DISC = os.environ.get("SYLPHEED_DISC", "/work/sylph_extract")
|
|
for pakpath in sorted(glob.glob(f"{DISC}/dat/*.pak")):
|
|
base = pakpath[:-4]
|
|
pak = open(pakpath, "rb").read()
|
|
if pak[:4] != b"IPFB": continue
|
|
n = struct.unpack_from(">I", pak, 4)[0]
|
|
toc = [struct.unpack_from(">III", pak, 0x10 + 12*i) for i in range(n)]
|
|
segs = sorted(glob.glob(base + ".p[0-9][0-9]"))
|
|
if not segs: continue
|
|
data = b"".join(open(s, "rb").read() for s in segs)
|
|
for ei, (h, off, cs) in enumerate(toc):
|
|
raw = data[off:off+cs]
|
|
try:
|
|
b = zlib.decompress(raw[10:]) if raw[:2] == b"Z1" else raw
|
|
except Exception:
|
|
continue
|
|
if b[:4] != b"RATC": continue
|
|
names = []
|
|
p = b.find(b"opt ")
|
|
while p >= 0:
|
|
ln = struct.unpack_from(">I", b, p+4)[0] if p+8 <= len(b) else 0
|
|
if 0 < ln <= 64 and p+8+ln <= len(b):
|
|
nm = b[p+8:p+8+ln].decode('latin1', 'replace')
|
|
if nm and all(32 < ord(c) < 127 for c in nm):
|
|
names.append((p, nm)); opt_total += 1
|
|
ext[os.path.splitext(nm)[1].lower()] += 1
|
|
p = b.find(b"opt ", p+4)
|
|
offs, i = [], 4
|
|
while i + 4 <= len(b):
|
|
if b[i:i+4] in MAG:
|
|
offs.append(i); i += 4
|
|
else: i += 1
|
|
if not names or not offs: continue
|
|
npos = [p for p, _ in names]
|
|
# each child -> index of the nearest preceding opt
|
|
owner = collections.defaultdict(list)
|
|
for k, o in enumerate(offs):
|
|
j = bisect.bisect_left(npos, o) - 1
|
|
if j >= 0: owner[j].append(k)
|
|
for j, (p, nm) in enumerate(names):
|
|
if not nm.lower().endswith(".tan"): continue
|
|
ks = owner.get(j, [])
|
|
if not ks: continue
|
|
sizes = sorted({(offs[k+1] if k+1 < len(offs) else len(b)) - offs[k] for k in ks})
|
|
tan_sites.append((os.path.basename(pakpath), ei, nm, len(ks), sizes))
|
|
print(f"`opt ` blocks disc-wide: {opt_total}")
|
|
print("\nby extension:")
|
|
for e, c in ext.most_common(25):
|
|
print(f" {e or '(none)':10} x{c}")
|
|
print(f"\n.tan resources with >=1 child: {len(tan_sites)}")
|
|
seen = collections.Counter()
|
|
for t in tan_sites:
|
|
seen[(t[0], t[2], t[3], tuple(t[4]))] += 1
|
|
for (pk, nm, fr, sz), c in sorted(seen.items()):
|
|
print(f" {pk:24} {nm:30} frames={fr:3} sizes={list(sz)} x{c} bundles")
|