diff --git a/docs/re/BACKLOG.md b/docs/re/BACKLOG.md index 3536d12..d77bf2d 100644 --- a/docs/re/BACKLOG.md +++ b/docs/re/BACKLOG.md @@ -6707,3 +6707,31 @@ cluster on `j` (22) and `t` (14); `ptc_pack` holds just one `j` name, so 🟡 **Not closed**: 66 of 137 bound effects still have no located home, `eff_n0071` included. ❔ Next: the `eff_j###` family and the four-digit series — not another disc-wide grep. + +--- + +## ✅❌ 2026-08-28 — the effect→package map (103 of 137), and a substring correction + +Item (a). Enumerating `eff_*` **per `.xpr`** across all 166 packages and matching +bound names **exactly** locates **103 of the 137**, up from 71. Only 36 packages +carry an effect name; two dominate: **`Base.xpr` 53** and **`ptc_pack.xpr` 46** +(then `Stage_S28.xpr` 2 and five `rou_f001_wep_NN.xpr` at 1 each). + +🔑 **Two shared effect libraries, not one** — and `ptc_pack.xpr` is the only +`*_pack` bundle on the disc. By digit-width: **3-digit 80 of 110, 4-digit 23 of +27**, so the previous iteration's split survives and sharpens — the four-digit +series lives outside `ptc_pack`, in `Base.xpr`. + +❌ **Correction to the previous entry**: `eff_f0002` and `eff_f0002_barn` do +**not** ship. `Base.xpr` contains **`eff_f0002_barnhaze`**, and both substring +greps matched inside it. The corpus's paid-for prefix lesson arriving from the +other direction — this time the *bound* name was a prefix of the *stored* one. + +🟡 **34 remain**, dominated by **`eff_l###` (17)**, then `h` 4, `s` 4, `j` 2, +`m` 2, `t` 1, and four four-digit names including `eff_n0071`. ⚠️ Scope note: the +`j` 22 / `t` 14 clustering reported last time was the residual against +`ptc_pack` alone; against all packages `l` is what is left. **Say which +population a residual is measured against.** + +New artefact: `tools/re-capture/effect_homes.py` → `docs/re/data/effect-homes.txt` +(lists all 34 by name). diff --git a/docs/re/data/effect-homes.txt b/docs/re/data/effect-homes.txt new file mode 100644 index 0000000..06c5c98 --- /dev/null +++ b/docs/re/data/effect-homes.txt @@ -0,0 +1,53 @@ +effect names bound by a datasheet field : 137 +resolved to an .xpr package (exact name): 103 +unlocated : 34 + +resolved, by package: + Base.xpr 53 + ptc_pack.xpr 46 + Stage_S28.xpr 2 + rou_f001_wep_28.xpr 1 + rou_f001_wep_27.xpr 1 + rou_f001_wep_12.xpr 1 + rou_f001_wep_82.xpr 1 + rou_f001_wep_59.xpr 1 + +by digit-width: + 3-digit: 80 resolved of 110 + 4-digit: 23 resolved of 27 + +the 34 unlocated, in full: + eff_e0044 + eff_f0002 + eff_f0002_barn + eff_h000 + eff_h006 + eff_h308 + eff_h310 + eff_j002_e01 + eff_j002_e02 + eff_l002_f + eff_l010_e + eff_l011_e + eff_l101_e + eff_l101_f + eff_l102_e + eff_l102_f + eff_l104_e + eff_l104_f + eff_l105_e + eff_l105_f + eff_l106_e + eff_l106_f + eff_l107_e + eff_l108_e + eff_l201_e + eff_l201_f + eff_m010_wep_85 + eff_m011_wep_85 + eff_n0071 + eff_s100 + eff_s901_e01_B + eff_s901_e02_D + eff_s901_e03_D + eff_t130 diff --git a/docs/re/structures/unit-substructure-records.md b/docs/re/structures/unit-substructure-records.md index 8657844..170aea6 100644 --- a/docs/re/structures/unit-substructure-records.md +++ b/docs/re/structures/unit-substructure-records.md @@ -308,3 +308,56 @@ So `eff_j###` is a third grouping that is almost entirely elsewhere. no located home, `eff_n0071` among them. But the route is now a real one with a number on it, and the next step is the `eff_j###` family and the four-digit series rather than another disc-wide grep. + +### ✅ The effect→package map — 103 of 137 located, and a correction + +Enumerating `eff_*` names **per `.xpr`** across all 166 packages and matching the +bound names **exactly** gives a real home for **103 of the 137**, up from the 71 +`ptc_pack` alone accounted for. Only **36** packages carry an effect name at +all, and two dominate: + +| package | bound effects it holds | +|---|---| +| `Base.xpr` | **53** | +| `ptc_pack.xpr` | **46** | +| `Stage_S28.xpr` | 2 | +| five `rou_f001_wep_NN.xpr` | 1 each | + +🔑 **There are two shared effect libraries, not one.** `ptc_pack.xpr` is the +particle library the previous iteration found; **`Base.xpr` is the larger home** +for bound effects. `ptc_pack.xpr` is also the *only* `*_pack` library on the +disc — no other shared bundle exists. + +By digit-width: **3-digit 80 resolved of 110, 4-digit 23 of 27.** So the +previous iteration's reading survives and sharpens — the four-digit series +really does live outside `ptc_pack` (that zero stands), and now we can say where: +`Base.xpr`. + +### ❌ Correction: `eff_f0002` does **not** ship — that was a substring hit + +The previous iteration reported `eff_f0002` and `eff_f0002_barn` as present in +`Base.xpr`. **Both were substring artefacts.** What the file actually contains +is **`eff_f0002_barnhaze`** — one longer resource name that `grep -l eff_f0002` +and `grep -l eff_f0002_barn` both match inside. Neither bound name is there. + +This is the corpus's own paid-for lesson arriving from the other direction: a +prefix hit is not an exact hit (cf. `rot_n001` vs `rot_n001_break`), and here it +was the *bound* name that was a prefix of the *stored* one. The map above is +exact-keyed, so it does not have this failure; the earlier positive is withdrawn. + +### 🟡 The 34 that remain + +The unlocated residual is dominated by a family the previous iteration did not +single out — **`eff_l###` with 17 of the 34** — then `eff_h###` 4, `eff_s###` 4, +`eff_j###` 2, `eff_m###` 2, `eff_t###` 1, and four four-digit names +(`eff_e0044`, `eff_f0002`, `eff_f0002_barn`, `eff_n0071`). + +⚠️ Note the scope shift: the `j` 22 / `t` 14 clustering reported last time was +the residual **against `ptc_pack` alone**; against *all* packages those families +are largely accounted for and `l` is what is left. Both numbers are right for +their own population — which is the reason to say which population a residual +is measured against. + +Regenerate with `tools/re-capture/effect_homes.py` → +[`docs/re/data/effect-homes.txt`](../data/effect-homes.txt), which lists all 34 +by name. diff --git a/tools/re-capture/effect_homes.py b/tools/re-capture/effect_homes.py new file mode 100644 index 0000000..5ad95bf --- /dev/null +++ b/tools/re-capture/effect_homes.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python3 +"""Which .xpr package holds each effect the datasheets bind. + +Effect names are matched EXACTLY, not as substrings: Base.xpr contains +eff_f0002_barnhaze, so a substring test for "eff_f0002" reports a false hit for +a resource that is not there. Names are pulled as maximal [A-Za-z0-9_] runs +containing "eff_", which yields bare resource names. + +Regenerates docs/re/data/effect-homes.txt. +""" +import glob, os, re, subprocess, sys, collections +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from unit_substructures import pak_entries +import unitgroup as U + +XPR = '/work/sylph_extract/hidden/resource3d' +FX = re.compile(r'(FxModel|EffectName|Effect_|ShellModel|CoverModel)') + +def width(n): + m = re.match(r'eff_[a-z](\d+)', n) + return len(m.group(1)) if m else 0 + +def main(): + if not os.path.isdir(XPR): + print('disc not mounted; nothing to do'); return + home = collections.defaultdict(set) + for path in sorted(glob.glob(os.path.join(XPR, '*.xpr'))): + out = subprocess.run(['grep', '-o', '-a', '[A-Za-z0-9_]*eff_[A-Za-z0-9_]*', path], + capture_output=True, text=True, errors='replace').stdout + for name in set(out.split()): + home[name].add(os.path.basename(path)) + + bound = set() + for pk in sorted(glob.glob('/work/sylph_extract/**/*.pak', recursive=True)): + for _h, b in pak_entries(pk): + if b[:4] != b'IDXD': + continue + try: + recs = U.parse(b) + except Exception: + continue + for r in recs: + for _t, f, v in r['fields']: + if f and FX.search(f) and isinstance(v, str) and v.startswith('eff_'): + bound.add(v) + + res = sorted(n for n in bound if n in home) + un = sorted(n for n in bound if n not in home) + print('effect names bound by a datasheet field : %d' % len(bound)) + print('resolved to an .xpr package (exact name): %d' % len(res)) + print('unlocated : %d' % len(un)) + print() + per = collections.Counter() + for n in res: + for f in home[n]: + per[f] += 1 + print('resolved, by package:') + for f, k in per.most_common(): + print(' %-26s %d' % (f, k)) + print() + print('by digit-width:') + for k in (3, 4, 0): + s = [n for n in bound if width(n) == k] + if s: + print(' %d-digit: %d resolved of %d' % (k, len([n for n in s if n in home]), len(s))) + print() + print('the %d unlocated, in full:' % len(un)) + for n in un: + print(' %s' % n) + +if __name__ == '__main__': + main()