re: the effect->package map, 103 of 137 - and eff_f0002 was a substring artefact
Enumerating eff_* names per .xpr across all 166 packages and matching the bound names EXACTLY gives a real home for 103 of the 137, up from the 71 ptc_pack alone accounted for. Only 36 packages carry an effect name at all, and two dominate: Base.xpr 53 ptc_pack.xpr 46 Stage_S28.xpr 2 five rou_f001_wep_NN.xpr 1 each So there are TWO shared effect libraries, not one - and ptc_pack.xpr is the only *_pack bundle on the disc, so no third shared library is hiding. By digit-width: 3-digit 80 resolved of 110, 4-digit 23 of 27. The previous iteration's split survives and sharpens - the four-digit series really does live outside ptc_pack (that zero stands), and now we can say where: Base.xpr. Correction to the previous commit. It reported eff_f0002 and eff_f0002_barn as present in Base.xpr. Both were SUBSTRING artefacts: what the file actually holds is eff_f0002_barnhaze, one longer resource name that grep -l eff_f0002 and grep -l eff_f0002_barn each match inside. Neither bound name is there. This is the corpus's own paid-for prefix lesson arriving from the other direction - last time it was rot_n001 vs rot_n001_break with the stored name longer; here the BOUND name was the prefix. The new map is exact-keyed and does not have this failure mode, so the earlier positive is withdrawn. 34 names remain unlocated, dominated by a family the last pass did not single out: eff_l### with 17 of the 34, then h 4, s 4, j 2, m 2, t 1, and four four-digit names - eff_e0044, eff_f0002, eff_f0002_barn, eff_n0071. Scope note worth keeping: the j 22 / t 14 clustering reported last time was the residual against ptc_pack ALONE; against all packages those families are largely accounted for and l is what is left. Both numbers are right for their own population, which is exactly why a residual has to say what it was measured against. New artefact with its regenerator: tools/re-capture/effect_homes.py -> docs/re/data/effect-homes.txt, which lists all 34 by name. All sixteen existing artefacts byte-identical.
This commit is contained in:
72
tools/re-capture/effect_homes.py
Normal file
72
tools/re-capture/effect_homes.py
Normal file
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Which .xpr package holds each effect the datasheets bind.
|
||||
|
||||
Effect names are matched EXACTLY, not as substrings: Base.xpr contains
|
||||
eff_f0002_barnhaze, so a substring test for "eff_f0002" reports a false hit for
|
||||
a resource that is not there. Names are pulled as maximal [A-Za-z0-9_] runs
|
||||
containing "eff_", which yields bare resource names.
|
||||
|
||||
Regenerates docs/re/data/effect-homes.txt.
|
||||
"""
|
||||
import glob, os, re, subprocess, sys, collections
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from unit_substructures import pak_entries
|
||||
import unitgroup as U
|
||||
|
||||
XPR = '/work/sylph_extract/hidden/resource3d'
|
||||
FX = re.compile(r'(FxModel|EffectName|Effect_|ShellModel|CoverModel)')
|
||||
|
||||
def width(n):
|
||||
m = re.match(r'eff_[a-z](\d+)', n)
|
||||
return len(m.group(1)) if m else 0
|
||||
|
||||
def main():
|
||||
if not os.path.isdir(XPR):
|
||||
print('disc not mounted; nothing to do'); return
|
||||
home = collections.defaultdict(set)
|
||||
for path in sorted(glob.glob(os.path.join(XPR, '*.xpr'))):
|
||||
out = subprocess.run(['grep', '-o', '-a', '[A-Za-z0-9_]*eff_[A-Za-z0-9_]*', path],
|
||||
capture_output=True, text=True, errors='replace').stdout
|
||||
for name in set(out.split()):
|
||||
home[name].add(os.path.basename(path))
|
||||
|
||||
bound = set()
|
||||
for pk in sorted(glob.glob('/work/sylph_extract/**/*.pak', recursive=True)):
|
||||
for _h, b in pak_entries(pk):
|
||||
if b[:4] != b'IDXD':
|
||||
continue
|
||||
try:
|
||||
recs = U.parse(b)
|
||||
except Exception:
|
||||
continue
|
||||
for r in recs:
|
||||
for _t, f, v in r['fields']:
|
||||
if f and FX.search(f) and isinstance(v, str) and v.startswith('eff_'):
|
||||
bound.add(v)
|
||||
|
||||
res = sorted(n for n in bound if n in home)
|
||||
un = sorted(n for n in bound if n not in home)
|
||||
print('effect names bound by a datasheet field : %d' % len(bound))
|
||||
print('resolved to an .xpr package (exact name): %d' % len(res))
|
||||
print('unlocated : %d' % len(un))
|
||||
print()
|
||||
per = collections.Counter()
|
||||
for n in res:
|
||||
for f in home[n]:
|
||||
per[f] += 1
|
||||
print('resolved, by package:')
|
||||
for f, k in per.most_common():
|
||||
print(' %-26s %d' % (f, k))
|
||||
print()
|
||||
print('by digit-width:')
|
||||
for k in (3, 4, 0):
|
||||
s = [n for n in bound if width(n) == k]
|
||||
if s:
|
||||
print(' %d-digit: %d resolved of %d' % (k, len([n for n in s if n in home]), len(s)))
|
||||
print()
|
||||
print('the %d unlocated, in full:' % len(un))
|
||||
for n in un:
|
||||
print(' %s' % n)
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user