List and edit in-memory sounds in soundbanks
This commit is contained in:
11
CLAUDE.md
11
CLAUDE.md
@@ -4,13 +4,16 @@ Batman: Arkham City (PC) audio toolkit + editor. Read `NOTES.md` first (status,
|
||||
|
||||
- Layout: `ackit/` library + CLI (`python -m ackit -h`), `app/` PySide6 editor, `tests/` pytest, `data/catalog/`
|
||||
CSVs (+ user's original lists in `data/catalog/source/`, never edit them), `docs/`, `tools/` build scripts.
|
||||
- Dev env: `.venv` (Python 3.14, `pip install -e .[app,dev]`). Run tests with `.venv/Scripts/python -m pytest -q`.
|
||||
- Dev env: `.venv` (Python 3.14, `pip install -e .[app,dev,verify]`). Run tests with `.venv/Scripts/python -m pytest -q`.
|
||||
- Writing game files goes through `ackit.workspace.Workspace` only (originals store, atomic writes, conflict checks).
|
||||
Speech/ambience: in place, never larger than the original WEM; music packs: any size (`docs/formats/wad.md`).
|
||||
In-memory sounds (archive BANK, loose `.bnk` + `.upk` packages): in place into every copy; a bank file is always
|
||||
rebuilt from the store's original copy + the applied replacements (`docs/formats/banks.md`, `upk.md`).
|
||||
- Never use Steam "verify integrity". Keep game files original after tests (`ackit status`, `ackit revert --all`).
|
||||
- Real-game checks: `ackit verify` (offline), `ackit verify --ingame` (Windows; needs Steam running and logged in,
|
||||
WinDbg; ~1.5 min; reverts and checks byte-identity itself).
|
||||
- Rebuild catalogs only from original game files: `python tools/build_catalog.py`.
|
||||
- Real-game checks: `ackit verify` (offline), `ackit verify --catalog [--quick]` (catalog vs game files),
|
||||
`ackit verify --ingame [--banks]` (Windows; needs Steam running and logged in, WinDbg; ~1.5 min; reverts and
|
||||
checks byte-identity itself).
|
||||
- Rebuild catalogs only from original game files: `python tools/build_catalog.py` (~5 min, scans all packages).
|
||||
- Shell: git at `C:\Program Files\Git\cmd` (prepend to PATH). Write Python with the Write/Edit tools; avoid
|
||||
PowerShell here-strings and heredoc string surgery with backslash escapes (they produced NUL bytes before).
|
||||
- The speaker loopback can contain other apps' audio (voice chat): never save raw captures.
|
||||
|
||||
20
NOTES.md
20
NOTES.md
@@ -20,15 +20,18 @@
|
||||
`ackit/bnk.py` (bank parsing: DIDX/DATA/HIRC sources). Docs: formats/upk.md, banks.md. Findings: pure-Python
|
||||
LZO ~47 MB/s (== decompress.exe on every block); 15,800 media IDs in 96,011 copies, 396 are prefetch heads
|
||||
of WAD sounds (2,101 copies) -> 15,404 in-memory sounds; copies identical except 1C898731.
|
||||
2. [ ] Catalog of in-memory sounds (archive `BANK`): build from loose banks + packages, location index,
|
||||
names/categories, prefetch handling; `ackit verify --catalog` cross-check (missing/extra/duplicate/
|
||||
character/category/language) + test.
|
||||
3. [ ] Writing in-memory sounds through `Workspace` (in place, <= original size; every copy in every
|
||||
package; byte-identical revert), CLI + editor; real-game e2e.
|
||||
2. [x] Catalog archive `BANK` (15,404 sounds, 6,416 named by events) + `BANK_locations.csv.gz` (93,910 copies,
|
||||
2,101 prefetch heads of 396 speech lines); `ackit verify --catalog` (full: catalog OK; fixed 2 duplicate
|
||||
names per language: variation shown as NAME#i); categories for bank prefixes.
|
||||
3. [x] Workspace writes BANK sounds into every copy and keeps speech-line prefetch heads in sync (bank files
|
||||
rebuilt from a mandatory full original copy); editor shows bank + copies. `ackit verify` PASS (300/300
|
||||
BANK WEMs rebuilt identically); `ackit verify --ingame --banks` PASS (loaded, no crash, revert identical;
|
||||
audible check not possible without input: findings 18).
|
||||
4. [ ] Subtitles + menu text: find/document formats, `ackit` parse/write with backup/revert.
|
||||
5. [ ] Editor: Subtitles section linked to audio, separate Menu text section, revert, mod export/import.
|
||||
6. [ ] Final e2e (audio + text), packaged build, REPORT.md.
|
||||
- OPEN: channel order for >2 channels unverified; speech heard in-game (needs gameplay).
|
||||
- OPEN: channel order for >2 channels unverified; speech and in-memory sounds *heard* in-game (need gameplay;
|
||||
posted keys do not drive the menu).
|
||||
|
||||
## Environment facts
|
||||
- Game: Steam AppID 200260 "Batman: Arkham City GOTY", `C:\Program Files (x86)\Steam\steamapps\common\Batman Arkham
|
||||
@@ -97,3 +100,8 @@
|
||||
soundbanks are missing (docs/formats/banks.md). Packages were unpacked with Gildor's decompress.exe from
|
||||
`%USERPROFILE%\Desktop\ArkhamCityModding\Audio\decompress\` (outside the repo). User: document now, implement
|
||||
editing later.
|
||||
- 2026-10-09 (mission 3): BANK = one row per media ID, written into every copy in place (assumption: the game
|
||||
may load any copy; a partial write would be inconsistent). Package writes store changed blocks as LZO literal
|
||||
runs (no compressor needed). Bank-file conflicts use the whole-file SHA-1 (the cheap guid+chunk-table
|
||||
fingerprint missed same-size edits in tests). Name collisions in the WAD catalogs (2 per language) are shown as
|
||||
NAME#i. `soundcard` comes from the `verify` extra (was missing from the venv).
|
||||
|
||||
@@ -4,9 +4,9 @@ Replace any voice line, music track or ambience in **Batman: Arkham City (PC, St
|
||||
by name, from a file or a recording, with one-click revert. No Wwise, no game patch, no external programs.
|
||||
|
||||
- **Unheard City** (`app/`, [user guide](docs/user-guide.md)): the desktop editor for Windows and Linux. Browse and
|
||||
search the ~11,500 sounds of the game's audio archives, filter by category and character, listen, replace or
|
||||
record, apply and revert. Most short effects (footsteps, foley, weapons, UI) live inside soundbanks and are
|
||||
not listed yet ([docs/formats/banks.md](docs/formats/banks.md)).
|
||||
search all ~26,900 sounds of the game (the voice lines of one language, music, ambience, and ~15,400 effects
|
||||
stored inside soundbanks: footsteps, foley, combat, weapons, UI), filter by category and character, listen,
|
||||
replace or record, apply and revert.
|
||||
- **`ackit`** (`ackit/`): the Python library and command-line tool behind it.
|
||||
|
||||
How it works in one paragraph: the game's Wwise 2011.1 decoder only knows the standard Vorbis codebooks, so ackit
|
||||
@@ -23,7 +23,7 @@ archives atomically, keeping the original bytes in a store for exact reverts. De
|
||||
- **Toolkit (developers):** Python 3.10+:
|
||||
```
|
||||
python -m venv .venv && .venv/Scripts/activate # Linux: source .venv/bin/activate
|
||||
pip install -e .[app,dev] # app = editor, dev = tests/packaging
|
||||
pip install -e .[app,dev,verify] # app = editor, dev = tests/packaging, verify = in-game test
|
||||
python -m app.main # run the editor from source
|
||||
```
|
||||
|
||||
|
||||
362
ackit/banks.py
Normal file
362
ackit/banks.py
Normal file
@@ -0,0 +1,362 @@
|
||||
"""In-memory sounds: WEMs held inside soundbanks, i.e. the loose SFX\\*.bnk files and the banks embedded
|
||||
in the .upk packages (docs/formats/banks.md, upk.md). Catalog archive `BANK`, target = media ID.
|
||||
|
||||
A media ID can be stored many times (the same bank copied into several packages, or several banks
|
||||
holding the same media). The location index (data/catalog/BANK_locations.csv.gz) lists every copy:
|
||||
media, file (relative to CookedPCConsole, '/'-separated), offset (in the *uncompressed* file), size, kind
|
||||
kind 'm' = an in-memory sound, 'p' = the in-memory head of a streamed WAD sound (prefetch); for 'p' rows
|
||||
`sound` is the WAD sound's key (e.g. ITA:2B14E472:3): speech lines have per-language media IDs and heads.
|
||||
Replacements are written in place into every copy (<= original size, zero-padded), so bank and package
|
||||
layouts never change.
|
||||
"""
|
||||
import csv
|
||||
import gzip
|
||||
import hashlib
|
||||
import io
|
||||
import os
|
||||
import struct
|
||||
from collections import defaultdict, namedtuple
|
||||
|
||||
from .bnk import Bank, BankError, find_banks, fnv1, PREFETCH
|
||||
from .upk import Package, PackageError, is_package
|
||||
|
||||
ARCHIVE = "BANK"
|
||||
LOCATIONS = "BANK_locations.csv.gz"
|
||||
SOUND, MEMORY, PREFETCH_HEAD = 2, "m", "p"
|
||||
EVENT, ACTION = 4, 3
|
||||
CONTAINERS = (5, 6, 7, 9) # random/sequence, switch, actor-mixer, layer
|
||||
|
||||
Location = namedtuple("Location", "media file offset size kind sound", defaults=("",))
|
||||
|
||||
|
||||
# --- files ----------------------------------------------------------------------------------------
|
||||
def bank_files(cooked):
|
||||
"""Relative paths ('/'-separated) of every loose regular bank and every package."""
|
||||
out = []
|
||||
sfx = os.path.join(cooked, "SFX")
|
||||
if os.path.isdir(sfx):
|
||||
for f in sorted(os.listdir(sfx)):
|
||||
if f.lower().endswith(".bnk"):
|
||||
with open(os.path.join(sfx, f), "rb") as fh:
|
||||
if fh.read(4) == b"BKHD":
|
||||
out.append("SFX/" + f)
|
||||
for f in sorted(os.listdir(cooked)):
|
||||
if f.lower().endswith(".upk") and is_package(os.path.join(cooked, f)):
|
||||
out.append(f)
|
||||
return out
|
||||
|
||||
|
||||
class _Plain:
|
||||
"""Uniform read access for a loose bank (offsets = file offsets)."""
|
||||
|
||||
def __init__(self, path):
|
||||
self.path = path
|
||||
|
||||
def read(self, off, n):
|
||||
with open(self.path, "rb") as f:
|
||||
f.seek(off)
|
||||
return f.read(n)
|
||||
|
||||
def read_all(self):
|
||||
return 0, open(self.path, "rb").read()
|
||||
|
||||
def fingerprint(self):
|
||||
return file_fingerprint(self.path)
|
||||
|
||||
def patched(self, edits):
|
||||
out = bytearray(open(self.path, "rb").read())
|
||||
for off, data in edits:
|
||||
out[off:off + len(data)] = data
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def file_fingerprint(path):
|
||||
"""Identity of a loose bank (small files: content hash)."""
|
||||
return hashlib.sha1(open(path, "rb").read()).hexdigest()[:16]
|
||||
|
||||
|
||||
_open_cache = {}
|
||||
|
||||
|
||||
def open_file(path):
|
||||
"""A reader for a bank file or package (cached while the file is unchanged)."""
|
||||
st = os.stat(path)
|
||||
key = (os.path.normcase(os.path.abspath(path)), st.st_size, st.st_mtime_ns)
|
||||
r = _open_cache.get(key)
|
||||
if r is None:
|
||||
r = Package(path) if path.lower().endswith(".upk") else _Plain(path)
|
||||
if len(_open_cache) > 64:
|
||||
_open_cache.clear()
|
||||
_open_cache[key] = r
|
||||
return r
|
||||
|
||||
|
||||
def file_path(cooked, rel):
|
||||
return os.path.join(cooked, *rel.split("/"))
|
||||
|
||||
|
||||
def riff_trim(b):
|
||||
"""A WEM slot without the zero padding after a smaller replacement (RIFF length)."""
|
||||
if len(b) >= 8 and b[:4] == b"RIFF":
|
||||
n = 8 + struct.unpack_from("<I", b, 4)[0]
|
||||
if n <= len(b):
|
||||
return b[:n]
|
||||
return b
|
||||
|
||||
|
||||
# --- HIRC graph (for names) ---------------------------------------------------------------------
|
||||
def _hirc_objects(buf, bank):
|
||||
"""(type, id, payload memoryview) of every HIRC object of a bank."""
|
||||
if "HIRC" not in bank.chunks:
|
||||
return
|
||||
o, n = bank.chunks["HIRC"]
|
||||
p = bank.at + o
|
||||
end = p + n
|
||||
count = struct.unpack_from("<I", buf, p)[0]
|
||||
p += 4
|
||||
for _ in range(count):
|
||||
if p + 9 > end:
|
||||
return
|
||||
t = buf[p]
|
||||
ln = struct.unpack_from("<I", buf, p + 1)[0]
|
||||
yield t, struct.unpack_from("<I", buf, p + 5)[0], bytes(buf[p + 5:p + 5 + ln])
|
||||
p += 5 + ln
|
||||
|
||||
|
||||
def _children(payload, known):
|
||||
"""A container's child list: the first `u32 count, count x u32 id` whose ids are all known objects."""
|
||||
n = len(payload)
|
||||
for p in range(4, n - 7):
|
||||
c = struct.unpack_from("<I", payload, p)[0]
|
||||
if 0 < c <= 4096 and p + 4 + 4 * c <= n:
|
||||
ids = struct.unpack_from("<%dI" % c, payload, p + 4)
|
||||
if all(i in known for i in ids):
|
||||
return ids
|
||||
return ()
|
||||
|
||||
|
||||
# --- scan ---------------------------------------------------------------------------------------
|
||||
class Scan:
|
||||
"""Everything found in the game's banks (built by `scan`)."""
|
||||
|
||||
def __init__(self):
|
||||
self.locations = [] # [Location]
|
||||
self.files = {} # rel -> fingerprint
|
||||
self.media = {} # id -> {"size", "sha1": set, "head": bytes, "banks": set}
|
||||
self.bank_names = {} # bank id -> name
|
||||
self.names = set() # identifier-like names seen in packages (for event names)
|
||||
self.objects = {} # HIRC id -> type
|
||||
self.containers = {} # id -> payload
|
||||
self.events = {} # id -> [action ids]
|
||||
self.actions = {} # id -> target id
|
||||
self.sound_media = defaultdict(set) # sound object id -> media ids
|
||||
self.prefetch = set() # media ids whose in-memory copy is a prefetch head
|
||||
self.packs = defaultdict(dict) # WAD entry id -> {media id: (offset in entry, size)} (BKPK tables)
|
||||
self.issues = [] # (kind, detail)
|
||||
|
||||
|
||||
def _package_names(pkg, start, data):
|
||||
"""Name table of an uncompressed package."""
|
||||
out = []
|
||||
o = pkg.name_offset - start
|
||||
for _ in range(pkg.name_count):
|
||||
n = struct.unpack_from("<i", data, o)[0]
|
||||
o += 4
|
||||
if n > 0:
|
||||
out.append(bytes(data[o:o + n - 1]).decode("latin-1"))
|
||||
o += n
|
||||
else:
|
||||
out.append(bytes(data[o:o - 2 * n - 2]).decode("utf-16-le", "replace"))
|
||||
o += -2 * n
|
||||
o += 8
|
||||
return out
|
||||
|
||||
|
||||
def _bkpk_tables(data, packs):
|
||||
"""BKPK tables (docs/formats/wad.md) embedded in a package: WAD entry -> media offsets/sizes."""
|
||||
import re
|
||||
for m in re.finditer(b"BKPK", data):
|
||||
p = m.start()
|
||||
if p + 0x48 > len(data) or struct.unpack_from("<I", data, p + 0x1C)[0] != 0x48:
|
||||
continue
|
||||
entry, n = struct.unpack_from("<I", data, p + 8)[0], struct.unpack_from("<I", data, p + 0x10)[0]
|
||||
if n > 100000 or p + 0x48 + 12 * n > len(data):
|
||||
continue
|
||||
for i in range(n):
|
||||
mid, size, off = struct.unpack_from("<III", data, p + 0x48 + 12 * i)
|
||||
packs[entry][mid] = (off, size)
|
||||
|
||||
|
||||
def scan(cooked, progress=None, cooked_banks=None, files=None):
|
||||
"""Scan every loose bank and package of the game (minutes), or only `files`. Returns a `Scan`."""
|
||||
sc = Scan()
|
||||
files = bank_files(cooked) if files is None else files
|
||||
if cooked_banks is None:
|
||||
p = os.path.join(cooked, "CookedBanks.txt")
|
||||
cooked_banks = [l.strip() for l in open(p, encoding="latin-1")] if os.path.exists(p) else []
|
||||
sc.bank_names.update({fnv1(n): n for n in cooked_banks if n})
|
||||
for k, rel in enumerate(files):
|
||||
if progress and k % 50 == 0:
|
||||
progress(k, len(files))
|
||||
path = file_path(cooked, rel)
|
||||
try:
|
||||
r = open_file(path)
|
||||
start, data = r.read_all()
|
||||
except (PackageError, OSError) as e:
|
||||
sc.issues.append(("unreadable", "%s: %s" % (rel, e)))
|
||||
continue
|
||||
if isinstance(r, Package):
|
||||
sc.names.update(_package_names(r, start, data))
|
||||
_bkpk_tables(data, sc.packs)
|
||||
banks = list(find_banks(data))
|
||||
else:
|
||||
try:
|
||||
banks = [Bank(data, 0)]
|
||||
except BankError:
|
||||
continue
|
||||
if banks:
|
||||
sc.files[rel] = r.fingerprint()
|
||||
for bk in banks:
|
||||
sc.bank_names.update(bk.names())
|
||||
media = {m.id: m for m in bk.media()}
|
||||
prefetched = set()
|
||||
for s in bk.sources():
|
||||
if s.stream == PREFETCH and s.media in media:
|
||||
prefetched.add(s.media)
|
||||
for m in media.values():
|
||||
w = bytes(data[bk.at + m.off:bk.at + m.off + m.size])
|
||||
kind = PREFETCH_HEAD if m.id in prefetched else MEMORY
|
||||
if kind == PREFETCH_HEAD:
|
||||
sc.prefetch.add(m.id)
|
||||
e = sc.media.setdefault(m.id, {"size": m.size, "sha1": set(), "head": w[:0x60], "banks": set(),
|
||||
"kinds": set()})
|
||||
e["sha1"].add(hashlib.sha1(w).hexdigest()[:16])
|
||||
e["banks"].add(bk.id)
|
||||
e["kinds"].add(kind)
|
||||
sc.locations.append(Location(m.id, rel, start + bk.at + m.off, m.size, kind))
|
||||
for t, oid, pl in _hirc_objects(data, bk):
|
||||
sc.objects[oid] = t
|
||||
if t == EVENT and len(pl) >= 8:
|
||||
n = struct.unpack_from("<I", pl, 4)[0]
|
||||
if len(pl) >= 8 + 4 * n:
|
||||
sc.events[oid] = list(struct.unpack_from("<%dI" % n, pl, 8))
|
||||
elif t == ACTION and len(pl) >= 12:
|
||||
sc.actions[oid] = struct.unpack_from("<I", pl, 8)[0]
|
||||
elif t in CONTAINERS:
|
||||
sc.containers[oid] = pl
|
||||
for s in bk.sources():
|
||||
if s.stream != PREFETCH:
|
||||
sc.sound_media[s.object].add(s.media)
|
||||
if progress:
|
||||
progress(len(files), len(files))
|
||||
return sc
|
||||
|
||||
|
||||
# --- names --------------------------------------------------------------------------------------
|
||||
def event_media(sc):
|
||||
"""{event name: [media ids in traversal order]} for every event whose name is known."""
|
||||
known = set(sc.objects)
|
||||
kids = {cid: _children(pl, known) for cid, pl in sc.containers.items()}
|
||||
by_hash = {}
|
||||
for n in sc.names:
|
||||
if n and len(n) < 128:
|
||||
by_hash.setdefault(fnv1(n), n)
|
||||
out = {}
|
||||
for eid, acts in sc.events.items():
|
||||
name = by_hash.get(eid)
|
||||
if not name:
|
||||
continue
|
||||
seen, order, stack = set(), [], [sc.actions.get(a) for a in acts][::-1]
|
||||
while stack:
|
||||
o = stack.pop()
|
||||
if o is None or o in seen:
|
||||
continue
|
||||
seen.add(o)
|
||||
if o in sc.sound_media:
|
||||
for m in sorted(sc.sound_media[o]):
|
||||
if m not in order:
|
||||
order.append(m)
|
||||
stack.extend(list(kids.get(o, ()))[::-1])
|
||||
if order:
|
||||
out[name] = order
|
||||
return out
|
||||
|
||||
|
||||
def media_names(sc):
|
||||
"""{media id: (display name, primary bank name)}: named after the most specific 'Play' event that
|
||||
reaches the media, otherwise after its bank (<Bank>_<n>)."""
|
||||
ev = event_media(sc)
|
||||
best = {}
|
||||
for name, ms in ev.items():
|
||||
rank = (0 if name.lower().startswith("play") else 1, len(ms), name.lower())
|
||||
for i, m in enumerate(ms):
|
||||
if m in sc.media and (m not in best or rank < best[m][0]):
|
||||
best[m] = (rank, name, i, len(ms))
|
||||
out = {}
|
||||
taken = set()
|
||||
for m, (_, name, i, n) in sorted(best.items(), key=lambda x: (x[1][1].lower(), x[1][2])):
|
||||
nm = name if n == 1 else "%s_%d" % (name, i + 1)
|
||||
if nm.lower() in taken:
|
||||
nm = "%s_%08X" % (nm, m)
|
||||
taken.add(nm.lower())
|
||||
out[m] = nm
|
||||
# the rest: by bank, numbered in media id order
|
||||
per_bank = defaultdict(list)
|
||||
for m, e in sc.media.items():
|
||||
bank = sorted(sc.bank_names.get(b, "%08X" % b) for b in e["banks"])[0]
|
||||
per_bank[bank].append(m)
|
||||
if m in out:
|
||||
out[m] = (out[m], bank)
|
||||
for bank, ms in per_bank.items():
|
||||
rest = sorted(m for m in ms if m not in out)
|
||||
for i, m in enumerate(rest):
|
||||
nm = "%s_%d" % (bank, i + 1)
|
||||
if nm.lower() in taken:
|
||||
nm = "%s_%08X" % (bank, m)
|
||||
taken.add(nm.lower())
|
||||
out[m] = (nm, bank)
|
||||
return out
|
||||
|
||||
|
||||
# --- location index -------------------------------------------------------------------------------
|
||||
def write_locations(path, locs):
|
||||
raw = io.StringIO()
|
||||
w = csv.writer(raw, lineterminator="\n")
|
||||
w.writerow(["media", "file", "offset", "size", "kind", "sound"])
|
||||
for l in sorted(locs, key=lambda l: (l.media, l.file, l.offset)):
|
||||
w.writerow(["%08X" % l.media, l.file, "%X" % l.offset, l.size, l.kind, l.sound])
|
||||
data = raw.getvalue().encode("utf-8")
|
||||
with open(path + ".tmp", "wb") as f:
|
||||
# mtime=0: identical content gives identical bytes (no spurious git changes)
|
||||
with gzip.GzipFile(fileobj=f, mode="wb", mtime=0) as gz:
|
||||
gz.write(data)
|
||||
os.replace(path + ".tmp", path)
|
||||
|
||||
|
||||
def read_locations(path):
|
||||
"""{media id: [Location]}"""
|
||||
out = defaultdict(list)
|
||||
with gzip.open(path, "rt", encoding="utf-8", newline="") as f:
|
||||
for r in csv.DictReader(f):
|
||||
m = int(r["media"], 16)
|
||||
out[m].append(Location(m, r["file"], int(r["offset"], 16), int(r["size"]), r["kind"], r["sound"]))
|
||||
return dict(out)
|
||||
|
||||
|
||||
def game_fingerprint(cooked):
|
||||
"""Cheap identity of the game's bank set: CookedBanks.txt + names/sizes of the loose banks.
|
||||
(Each file is checked again before it is written: `file_fingerprint` / Package.fingerprint.)"""
|
||||
h = hashlib.sha1()
|
||||
p = os.path.join(cooked, "CookedBanks.txt")
|
||||
if os.path.exists(p):
|
||||
h.update(open(p, "rb").read())
|
||||
sfx = os.path.join(cooked, "SFX")
|
||||
if os.path.isdir(sfx):
|
||||
for f in sorted(os.listdir(sfx)):
|
||||
if f.lower().endswith(".bnk"):
|
||||
h.update(("%s:%d;" % (f.upper(), os.path.getsize(os.path.join(sfx, f)))).encode())
|
||||
return h.hexdigest()[:16]
|
||||
|
||||
|
||||
def has_banks(cooked):
|
||||
return os.path.exists(os.path.join(cooked, "CookedBanks.txt"))
|
||||
124
ackit/catalog.py
124
ackit/catalog.py
@@ -9,6 +9,7 @@ Names follow the community hash list (ackit/data/hashlist.txt): an entry with on
|
||||
(alias NAME_1), entries with n variations are NAME_1..NAME_n (alias NAME#i); stream-pack media are
|
||||
named by their ID or <Pack>_<ID>.
|
||||
"""
|
||||
import collections
|
||||
import csv
|
||||
import difflib
|
||||
import json
|
||||
@@ -17,16 +18,16 @@ import re
|
||||
import struct
|
||||
from dataclasses import dataclass, asdict
|
||||
|
||||
from . import classify
|
||||
from . import banks, classify
|
||||
from .game import wad_fingerprint
|
||||
from .inject import pack_bnk_path
|
||||
from .pack import Pack
|
||||
from .paths import catalog_dir, package_data, user_dir
|
||||
from .wad import Wad
|
||||
|
||||
FIELDS = ["archive", "target", "name", "aliases", "group", "category", "character",
|
||||
FIELDS = ["archive", "target", "name", "aliases", "group", "category", "bank", "character",
|
||||
"characters_inferred", "seconds", "rate", "channels", "size", "sha1", "notes"]
|
||||
SCAN_VERSION = 1
|
||||
SCAN_VERSION = 2
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -37,6 +38,7 @@ class Sound:
|
||||
aliases: tuple = ()
|
||||
group: str = "Unknown"
|
||||
category: str = "Unknown"
|
||||
bank: str = "" # BANK sounds: the soundbank holding it (name); WAD sounds: ''
|
||||
character: str = "" # verified speaker(s), '; '-separated (curated catalog)
|
||||
characters_inferred: str = "" # characters/roles mentioned by the name (hint)
|
||||
seconds: float = 0.0
|
||||
@@ -96,7 +98,7 @@ def read_csv(path):
|
||||
for r in csv.DictReader(open(path, encoding="utf-8", newline="")):
|
||||
out.append(Sound(archive=r["archive"], target=r["target"], name=r["name"],
|
||||
aliases=tuple(a for a in r["aliases"].split("|") if a), group=r["group"],
|
||||
category=r["category"], character=r["character"],
|
||||
category=r["category"], bank=r.get("bank") or "", character=r["character"],
|
||||
characters_inferred=r["characters_inferred"], seconds=float(r["seconds"] or 0),
|
||||
rate=int(r["rate"] or 0), channels=int(r["channels"] or 0), size=int(r["size"] or 0),
|
||||
sha1=r["sha1"], notes=r["notes"]))
|
||||
@@ -185,12 +187,96 @@ def scan_archive(archive, wad_path, characters=None, progress=None):
|
||||
s.seconds, s.rate, s.channels, s.size = secs, rate, ch, ln
|
||||
s.group, s.category = classify.classify(s.name)
|
||||
out.append(s)
|
||||
# a variation named NAME_i can collide with an entry whose own name is NAME_i: the variation is
|
||||
# shown as NAME#i and keeps NAME_i as alias (both stay findable, display names stay unique)
|
||||
taken = collections.Counter(s.name.lower() for s in out)
|
||||
for s in out:
|
||||
if taken[s.name.lower()] > 1 and ":" in s.target:
|
||||
s.name, s.aliases = s.aliases[0], (s.name,)
|
||||
for s in out:
|
||||
s.characters_inferred = "; ".join(classify.infer_characters(s.name))
|
||||
s.character = next((characters[a.lower()] for a in s.names() if a.lower() in characters), "")
|
||||
return out
|
||||
|
||||
|
||||
_CAMEL = re.compile(r"(?<=[a-z])(?=[A-Z])")
|
||||
|
||||
|
||||
def bank_sounds(sc, characters=None):
|
||||
"""[Sound] of the in-memory sounds found by `banks.scan` (prefetch heads belong to WAD sounds)."""
|
||||
characters = characters if characters is not None else load_characters()
|
||||
names = banks.media_names(sc)
|
||||
first = {}
|
||||
for loc in sorted(sc.locations, key=lambda l: (l.media, l.file, l.offset)):
|
||||
first.setdefault(loc.media, loc)
|
||||
out = []
|
||||
for m, e in sc.media.items():
|
||||
if banks.PREFETCH_HEAD in e["kinds"]:
|
||||
continue
|
||||
name, bank = names[m]
|
||||
rate, ch, secs = _hdr(e["head"])
|
||||
s = Sound(banks.ARCHIVE, "%08X" % m, name, ("%s/%08X" % (bank, m),), bank=bank,
|
||||
seconds=secs, rate=rate, channels=ch,
|
||||
size=min(l.size for l in sc.locations if l.media == m) if len(e["sha1"]) > 1 else e["size"],
|
||||
sha1=next(iter(e["sha1"])) if len(e["sha1"]) == 1 else "")
|
||||
s.group, s.category = classify.classify(name, bank=bank)
|
||||
s.characters_inferred = "; ".join(classify.infer_characters(_CAMEL.sub("_", name + "_" + bank)))
|
||||
s.character = next((characters[a.lower()] for a in s.names() if a.lower() in characters), "")
|
||||
out.append(s)
|
||||
out.sort(key=lambda s: (s.bank.lower(), s.name.lower()))
|
||||
return out
|
||||
|
||||
|
||||
def head_owners(game, sc, archives=None):
|
||||
"""{prefetch media id: WAD sound key} -- the speech line (one language) each prefetch head belongs to,
|
||||
found through the per-language BKPK tables of the packages (docs/formats/banks.md)."""
|
||||
want = {m for m, e in sc.media.items() if banks.PREFETCH_HEAD in e["kinds"]}
|
||||
out = {}
|
||||
for a in archives or [x for x in game.archives if x != "SFX"]:
|
||||
wad = Wad(game.wad(a))
|
||||
by_hash = {e.hash: e for e in wad.entries}
|
||||
with open(game.wad(a), "rb") as f:
|
||||
for ent, table in sc.packs.items():
|
||||
mine = [m for m in table if m in want and m not in out]
|
||||
e = by_hash.get(ent)
|
||||
if not mine or not e:
|
||||
continue
|
||||
wems = _walk_entry(f, e.offset, e.size)
|
||||
for m in mine:
|
||||
off, size = table[m]
|
||||
for i, (o, ln, _) in enumerate(wems):
|
||||
if (o, ln) == (off, size):
|
||||
out[m] = "%s:%08X" % (a, ent) + (":%d" % i if len(wems) > 1 else "")
|
||||
return out
|
||||
|
||||
|
||||
def scan_banks(game, progress=None, characters=None):
|
||||
"""Full scan of the game's banks -> ([Sound], [Location] with WAD keys on prefetch heads, Scan)."""
|
||||
sc = banks.scan(game.cooked, progress)
|
||||
owners = head_owners(game, sc)
|
||||
locs = [l._replace(sound=owners.get(l.media, "")) if l.kind == banks.PREFETCH_HEAD else l for l in sc.locations]
|
||||
return bank_sounds(sc, characters), locs, sc
|
||||
|
||||
|
||||
def load_banks(game, progress=None):
|
||||
"""([Sound], {media: [Location]}) of the in-memory sounds: shipped catalog when the game's bank set
|
||||
matches it, else a cached or fresh scan (several minutes)."""
|
||||
fp = banks.game_fingerprint(game.cooked)
|
||||
shipped = _shipped_index().get(banks.ARCHIVE, {})
|
||||
csv_path = os.path.join(catalog_dir(), "%s.csv" % banks.ARCHIVE)
|
||||
loc_path = os.path.join(catalog_dir(), banks.LOCATIONS)
|
||||
if shipped.get("fingerprint") == fp and os.path.exists(csv_path) and os.path.exists(loc_path):
|
||||
return read_csv(csv_path), banks.read_locations(loc_path)
|
||||
cache = os.path.join(user_dir("cache"), "scan_%s_%s_v%d" % (banks.ARCHIVE, fp, SCAN_VERSION))
|
||||
if not (os.path.exists(cache + ".csv") and os.path.exists(cache + ".locations.csv.gz")):
|
||||
sounds, locs, _ = scan_banks(game, progress)
|
||||
for s in sounds:
|
||||
s.sha1 = "" # unknown build: no trusted original checksums
|
||||
banks.write_locations(cache + ".locations.csv.gz", locs)
|
||||
write_csv(cache + ".csv", sounds)
|
||||
return read_csv(cache + ".csv"), banks.read_locations(cache + ".locations.csv.gz")
|
||||
|
||||
|
||||
# --- catalog -------------------------------------------------------------------------------
|
||||
def _shipped_index():
|
||||
p = os.path.join(catalog_dir(), "index.json")
|
||||
@@ -200,6 +286,14 @@ def _shipped_index():
|
||||
return {}
|
||||
|
||||
|
||||
def is_shipped(game, archive):
|
||||
"""True if the game's archive matches the shipped catalog (not a scan of another build)."""
|
||||
shipped = _shipped_index().get(archive, {})
|
||||
if archive == banks.ARCHIVE:
|
||||
return shipped.get("fingerprint") == banks.game_fingerprint(game.cooked)
|
||||
return shipped.get("fingerprint") == wad_fingerprint(game.wad(archive))
|
||||
|
||||
|
||||
def load_archive(game, archive, progress=None):
|
||||
"""[Sound] for one archive of a game (shipped CSV, user-cache scan, or fresh scan)."""
|
||||
path = game.wad(archive)
|
||||
@@ -217,15 +311,31 @@ def load_archive(game, archive, progress=None):
|
||||
return sounds
|
||||
|
||||
|
||||
def available_archives(game):
|
||||
"""WAD archives of the game plus BANK when it has soundbanks."""
|
||||
return list(game.archives) + ([banks.ARCHIVE] if banks.has_banks(game.cooked) else [])
|
||||
|
||||
|
||||
class Catalog:
|
||||
"""All sounds of the selected archives (default: the game's speech language + SFX)."""
|
||||
"""All sounds of the selected archives (default: the game's speech language, SFX and BANK)."""
|
||||
|
||||
def __init__(self, game, archives=None, progress=None):
|
||||
self.game = game
|
||||
self.archives = archives or [a for a in (game.language(), "SFX") if a in game.archives]
|
||||
avail = available_archives(game)
|
||||
self.archives = archives or [a for a in (game.language(), "SFX", banks.ARCHIVE) if a in avail]
|
||||
self.sounds = []
|
||||
self.locations = {} # BANK media id -> [Location]
|
||||
self.heads = {} # WAD sound key -> [Location] of its prefetch heads
|
||||
bank_sounds_ = []
|
||||
if banks.ARCHIVE in avail:
|
||||
# always loaded: speech lines may have prefetch heads that must be written with them
|
||||
bank_sounds_, self.locations = load_banks(game, progress)
|
||||
for ls in self.locations.values():
|
||||
for l in ls:
|
||||
if l.kind == banks.PREFETCH_HEAD and l.sound:
|
||||
self.heads.setdefault(l.sound, []).append(l)
|
||||
for a in self.archives:
|
||||
self.sounds += load_archive(game, a, progress)
|
||||
self.sounds += bank_sounds_ if a == banks.ARCHIVE else load_archive(game, a, progress)
|
||||
self.by_key = {s.key: s for s in self.sounds}
|
||||
self._by_name = {}
|
||||
for s in self.sounds:
|
||||
|
||||
@@ -28,17 +28,26 @@ _RULES = [
|
||||
(r"^SPEECH_CHAR_", "Speech", "Character lines"),
|
||||
(r"^SPEECH_(CH|SEQUENCE|STEEL|MUSEUM|CHURCH|COURT|UNDER|GCPD|ZP1)_", "Speech", "Story missions"),
|
||||
(r"^SPEECH_(GAME|MAP|AUGMENTED|FORENSICS|TEMP)_", "Speech", "Gameplay & hints"),
|
||||
(r"^NPC_", "Ambience", "Crowds"),
|
||||
(r"^NPC_.*CROWD", "Ambience", "Crowds"),
|
||||
(r"^SPEECH_", "Speech", "Other speech"),
|
||||
(r"^MUSIC_PDLC", "Music", "Music (DLC)"),
|
||||
(r"^MUSIC", "Music", "Music"),
|
||||
(r"^(MUSIC|MUS_)", "Music", "Music"),
|
||||
(r"^(AMB|PSA)_", "Ambience", "Ambience"),
|
||||
(r"^PSO_", "Ambience", "Ambient emitters"),
|
||||
(r"^WND_", "Ambience", "Wind"),
|
||||
(r"^WTR_", "Ambience", "Weather"),
|
||||
(r"^P?VOL_", "Ambience", "Room tone"),
|
||||
(r"^SEQ_", "SFX", "Scripted sequences"),
|
||||
(r"FOOTSTEP", "SFX", "Footsteps"),
|
||||
(r"^(CHR|NPC)_.*(COMBAT|BONEBREAK|BODYFALL)", "SFX", "Combat"),
|
||||
(r"^(SEQ|MTN)_", "SFX", "Scripted sequences"),
|
||||
(r"^OBJ_", "SFX", "Objects"),
|
||||
(r"^(PX|PHY)_", "SFX", "Physics"),
|
||||
(r"^WPN_", "SFX", "Weapons"),
|
||||
(r"^CHR_", "SFX", "Character foley"),
|
||||
(r"^DVC_", "SFX", "Gadgets"),
|
||||
(r"^(GUI|SBJ)_", "SFX", "Interface"),
|
||||
(r"^VFX_", "SFX", "Visual effects"),
|
||||
(r"^NPC_", "SFX", "Enemies & NPCs"),
|
||||
(r"^(CHR|GEN)_", "SFX", "Character foley"),
|
||||
(r"^GLOBAL", "SFX", "Global"),
|
||||
]
|
||||
_RULES = [(re.compile(p), g, c) for p, g, c in _RULES]
|
||||
@@ -46,9 +55,10 @@ _RULES = [(re.compile(p), g, c) for p, g, c in _RULES]
|
||||
GROUPS = ("Speech", "Music", "Ambience", "SFX", "Unknown")
|
||||
|
||||
|
||||
def classify(name: str, pack: str = None):
|
||||
"""-> (group, category). `pack` = name of the stream pack a media belongs to, if any."""
|
||||
n = name.upper()
|
||||
def classify(name: str, pack: str = None, bank: str = None):
|
||||
"""-> (group, category). `pack` = name of the stream pack a media belongs to, if any; `bank` = name of
|
||||
the soundbank holding an in-memory sound (its prefix is more reliable than event names)."""
|
||||
n = (bank or name).upper()
|
||||
if pack:
|
||||
p = pack.upper()
|
||||
if p.startswith("MUSIC"):
|
||||
|
||||
37
ackit/cli.py
37
ackit/cli.py
@@ -19,8 +19,7 @@ from . import audio, settings
|
||||
from .game import Game, GameError, find_installs, default_game_dir
|
||||
|
||||
|
||||
def _workspace(a, archives=None):
|
||||
from .workspace import Workspace
|
||||
def _game(a):
|
||||
st = settings.load()
|
||||
gd = a.game or default_game_dir(st)
|
||||
if not gd:
|
||||
@@ -29,9 +28,15 @@ def _workspace(a, archives=None):
|
||||
if a.game:
|
||||
st["game_dir"] = os.path.abspath(a.game)
|
||||
settings.save(st)
|
||||
game = Game(gd)
|
||||
return Game(gd)
|
||||
|
||||
|
||||
def _workspace(a, archives=None):
|
||||
from .workspace import Workspace
|
||||
st = settings.load()
|
||||
game = _game(a)
|
||||
if archives is None and getattr(a, "lang", None):
|
||||
archives = [a.lang, "SFX"]
|
||||
archives = [a.lang, "SFX", "BANK"]
|
||||
return Workspace(game, a.store or st.get("store_root"), archives,
|
||||
full_backup=not getattr(a, "no_full_backup", False) and st.get("full_backup", True),
|
||||
progress=lambda n, t: None)
|
||||
@@ -242,13 +247,25 @@ def cmd_game(a):
|
||||
|
||||
|
||||
def cmd_verify(a):
|
||||
if a.catalog:
|
||||
from .verify import catalogcheck
|
||||
rep = catalogcheck.run(_game(a), [x.upper() for x in a.archive] or None, quick=a.quick)
|
||||
for (arch, kind), n in sorted(rep.counts().items()):
|
||||
print("%-14s %-10s %d" % (arch, kind, n))
|
||||
for arch, kind, key, detail in rep.issues[:a.show]:
|
||||
print(" %s %s %s %s" % (arch, kind, key, detail))
|
||||
print("checked: %s" % ", ".join("%s %d" % kv for kv in sorted(rep.checked.items())))
|
||||
print("catalog OK" if rep.ok else "%d issue(s)" % len(rep.issues))
|
||||
sys.exit(0 if rep.ok else 1)
|
||||
from .verify import selfcheck
|
||||
ws = _workspace(a)
|
||||
if not a.ingame:
|
||||
ok = selfcheck.run(ws, log=print)
|
||||
sys.exit(0 if ok else 1)
|
||||
from .verify import ingame
|
||||
if a.baseline:
|
||||
if a.banks:
|
||||
rep = ingame.run_banks(ws, a.sound or None)
|
||||
elif a.baseline:
|
||||
rep = ingame.run(ws, a.target, mode="baseline", seconds=a.seconds)
|
||||
elif a.installed:
|
||||
rep = ingame.run(ws, a.target, mode="installed", seconds=a.seconds)
|
||||
@@ -310,10 +327,18 @@ def build_parser():
|
||||
s = add("import-mod", cmd_import_mod, "install a .acmod package", (enc,))
|
||||
s.add_argument("file")
|
||||
add("game", cmd_game, "show the detected game folder, archives, language and store")
|
||||
s = add("verify", cmd_verify, "self-check (offline), or test in the real game with --ingame (Windows)")
|
||||
s = add("verify", cmd_verify, "self-check (offline), catalog cross-check (--catalog), or test in the real "
|
||||
"game with --ingame (Windows)")
|
||||
s.add_argument("--catalog", action="store_true", help="cross-check the catalog against the game files")
|
||||
s.add_argument("--archive", action="append", default=[], help="with --catalog: only this archive (repeatable)")
|
||||
s.add_argument("--quick", action="store_true", help="with --catalog: loose banks + a sample of packages")
|
||||
s.add_argument("--show", type=int, default=30, help="with --catalog: issues to list (default 30)")
|
||||
s.add_argument("--ingame", action="store_true"); s.add_argument("--target", default="SFX:EDF036D6/393BC53A")
|
||||
s.add_argument("--audio"); s.add_argument("--baseline", action="store_true")
|
||||
s.add_argument("--installed", action="store_true"); s.add_argument("--seconds", type=float, default=40)
|
||||
s.add_argument("--banks", action="store_true",
|
||||
help="with --ingame: test in-memory sounds (default: the front-end banks, or --sound NAME ...)")
|
||||
s.add_argument("--sound", action="append", default=[], help="with --banks: a sound to test (repeatable)")
|
||||
# advanced
|
||||
s = add("encode", cmd_encode, "[advanced] audio -> .wem file", (enc,))
|
||||
s.add_argument("audio"); s.add_argument("out"); s.add_argument("--like", metavar="NAME")
|
||||
|
||||
159
ackit/verify/catalogcheck.py
Normal file
159
ackit/verify/catalogcheck.py
Normal file
@@ -0,0 +1,159 @@
|
||||
"""Cross-check the catalog against the installed game (`ackit verify --catalog`).
|
||||
|
||||
For every archive the catalog in use (shipped CSV or the user-cache scan) is compared with a fresh scan:
|
||||
missing a sound of the game that is not in the catalog
|
||||
extra a catalog row that is not in the game
|
||||
duplicate a target or a name used twice
|
||||
size/sha1 the catalog's original size or checksum differs from the game (game modified or other build)
|
||||
category the row's group/category is not what the classification rules give for its name
|
||||
character the verified speaker differs from data/catalog/characters.csv
|
||||
language the archive code does not match the WAD's language folder
|
||||
copies (BANK) stored copies or prefetch heads differ from the location index
|
||||
Run it on unmodified game files (`ackit status`), otherwise modified sounds show up as size/sha1 issues.
|
||||
"""
|
||||
import collections
|
||||
import hashlib
|
||||
import os
|
||||
import random
|
||||
|
||||
from .. import banks, classify
|
||||
from ..catalog import (Catalog, available_archives, bank_sounds, hash_names, head_owners, is_shipped, load_characters,
|
||||
scan_archive)
|
||||
from ..inject import read_target
|
||||
from ..wad import Wad
|
||||
|
||||
FOLDER_LANG = {"ENGLISH(US)": "INT", "ITALIAN": "ITA", "FRENCH(FRANCE)": "FRA", "GERMAN": "DEU",
|
||||
"SPANISH(SPAIN)": "ESN"}
|
||||
|
||||
|
||||
class Report:
|
||||
def __init__(self):
|
||||
self.checked = collections.Counter()
|
||||
self.issues = [] # (archive, kind, key, detail)
|
||||
|
||||
def add(self, archive, kind, key, detail=""):
|
||||
self.issues.append((archive, kind, key, detail))
|
||||
|
||||
def counts(self):
|
||||
return collections.Counter((a, k) for a, k, _, _ in self.issues)
|
||||
|
||||
@property
|
||||
def ok(self):
|
||||
return not self.issues
|
||||
|
||||
|
||||
def _expected_category(s):
|
||||
if s.archive == banks.ARCHIVE:
|
||||
return classify.classify(s.name, bank=s.bank)
|
||||
pack = None
|
||||
if s.is_pack_media:
|
||||
pack = hash_names().get(int(s.target.split("/")[0], 16), s.target.split("/")[0])
|
||||
return classify.classify(s.name, pack=pack)
|
||||
|
||||
|
||||
def _check_rows(rep, archive, rows, characters):
|
||||
seen_t, seen_n = set(), {}
|
||||
for s in rows:
|
||||
rep.checked[archive] += 1
|
||||
if s.archive != archive:
|
||||
rep.add(archive, "language", s.key, "row says archive %s" % s.archive)
|
||||
if s.target in seen_t:
|
||||
rep.add(archive, "duplicate", s.key, "target")
|
||||
seen_t.add(s.target)
|
||||
if s.name.lower() in seen_n:
|
||||
rep.add(archive, "duplicate", s.key, "name %s (also %s)" % (s.name, seen_n[s.name.lower()]))
|
||||
seen_n[s.name.lower()] = s.key
|
||||
g, c = _expected_category(s)
|
||||
if (s.group, s.category) != (g, c):
|
||||
rep.add(archive, "category", s.key, "%s/%s, rules give %s/%s" % (s.group, s.category, g, c))
|
||||
want = next((characters[a.lower()] for a in s.names() if a.lower() in characters), "")
|
||||
if s.character != want:
|
||||
rep.add(archive, "character", s.key, "%r, curated list says %r" % (s.character, want))
|
||||
|
||||
|
||||
def check_wad(rep, game, archive, rows, characters):
|
||||
path = game.wad(archive)
|
||||
if archive != "SFX":
|
||||
folder = os.path.basename(os.path.dirname(path)).upper()
|
||||
if FOLDER_LANG.get(folder, archive) != archive:
|
||||
rep.add(archive, "language", archive, "WAD in folder %s" % folder)
|
||||
_check_rows(rep, archive, rows, characters)
|
||||
fresh = {s.target: s for s in scan_archive(archive, path, characters=characters)}
|
||||
cat = {s.target: s for s in rows}
|
||||
for t in sorted(set(fresh) - set(cat)):
|
||||
rep.add(archive, "missing", "%s:%s" % (archive, t), fresh[t].name)
|
||||
for t in sorted(set(cat) - set(fresh)):
|
||||
rep.add(archive, "extra", "%s:%s" % (archive, t), cat[t].name)
|
||||
wad = Wad(path)
|
||||
for t in sorted(set(cat) & set(fresh)):
|
||||
s = cat[t]
|
||||
if s.size and s.size != fresh[t].size and not s.is_pack_media:
|
||||
rep.add(archive, "size", s.key, "catalog %d, game %d" % (s.size, fresh[t].size))
|
||||
if s.sha1:
|
||||
h = hashlib.sha1(read_target(path, t, wad)).hexdigest()[:16]
|
||||
if h != s.sha1:
|
||||
rep.add(archive, "sha1", s.key, "game file differs from the catalog's original")
|
||||
|
||||
|
||||
def check_banks(rep, game, rows, locations, characters, quick=False, sample=40, progress=None):
|
||||
_check_rows(rep, banks.ARCHIVE, rows, characters)
|
||||
files = banks.bank_files(game.cooked)
|
||||
if quick:
|
||||
pk = [f for f in files if f.lower().endswith(".upk")]
|
||||
rnd = random.Random(1)
|
||||
files = [f for f in files if f.startswith("SFX/")] + sorted(rnd.sample(pk, min(sample, len(pk))))
|
||||
sc = banks.scan(game.cooked, progress, files=files)
|
||||
owners = head_owners(game, sc)
|
||||
index = collections.defaultdict(set)
|
||||
for m, ls in locations.items():
|
||||
for l in ls:
|
||||
if l.file in set(files):
|
||||
index[(l.media, l.kind)].add((l.file, l.offset, l.size, l.sound if l.kind == banks.PREFETCH_HEAD else ""))
|
||||
found = collections.defaultdict(set)
|
||||
for l in sc.locations:
|
||||
found[(l.media, l.kind)].add((l.file, l.offset, l.size, owners.get(l.media, "") if l.kind == banks.PREFETCH_HEAD else ""))
|
||||
for k in sorted(set(found) | set(index)):
|
||||
if found.get(k, set()) != index.get(k, set()):
|
||||
rep.add(banks.ARCHIVE, "copies", "%08X" % k[0], "%s: index %d, game %d"
|
||||
% ("head" if k[1] == banks.PREFETCH_HEAD else "copies", len(index.get(k, ())), len(found.get(k, ()))))
|
||||
cat = {s.target: s for s in rows}
|
||||
fresh = {s.target: s for s in bank_sounds(sc, characters)}
|
||||
for t in sorted(set(fresh) - set(cat)):
|
||||
rep.add(banks.ARCHIVE, "missing", "BANK:" + t, fresh[t].bank)
|
||||
if not quick:
|
||||
for t in sorted(set(cat) - set(fresh)):
|
||||
rep.add(banks.ARCHIVE, "extra", "BANK:" + t, cat[t].name)
|
||||
for t in sorted(set(cat) & set(fresh)):
|
||||
s, f = cat[t], fresh[t]
|
||||
if s.sha1 and f.sha1 and s.sha1 != f.sha1:
|
||||
rep.add(banks.ARCHIVE, "sha1", s.key, "game file differs from the catalog's original")
|
||||
if f.size < s.size:
|
||||
rep.add(banks.ARCHIVE, "size", s.key, "catalog %d, game %d" % (s.size, f.size))
|
||||
for m, ls in locations.items():
|
||||
for l in ls:
|
||||
if l.kind == banks.PREFETCH_HEAD and not l.sound:
|
||||
rep.add(banks.ARCHIVE, "copies", "%08X" % m, "prefetch head without a speech line")
|
||||
|
||||
|
||||
def run(game, archives=None, quick=False, log=print, progress=None):
|
||||
"""Check the given archives (default: all). Returns a Report."""
|
||||
rep = Report()
|
||||
characters = load_characters()
|
||||
archives = archives or available_archives(game)
|
||||
cat = Catalog(game, archives, progress)
|
||||
by_arch = collections.defaultdict(list)
|
||||
for s in cat.sounds:
|
||||
by_arch[s.archive].append(s)
|
||||
for a in archives:
|
||||
log("checking %s ..." % a)
|
||||
if a == banks.ARCHIVE:
|
||||
check_banks(rep, game, by_arch[a], cat.locations, characters, quick=quick, progress=progress)
|
||||
else:
|
||||
check_wad(rep, game, a, by_arch[a], characters)
|
||||
known = {n.lower() for s in cat.sounds for n in s.names()}
|
||||
# speaker names are the same in every language; only meaningful for the build the lists were made for
|
||||
if any(a in game.languages and is_shipped(game, a) for a in archives):
|
||||
for n in sorted(characters):
|
||||
if n not in known:
|
||||
rep.add("characters.csv", "extra", n, "no sound with this name")
|
||||
return rep
|
||||
@@ -61,6 +61,43 @@ def read_mem(pid, addr, n):
|
||||
k32.CloseHandle(h)
|
||||
|
||||
|
||||
class _MBI(ctypes.Structure):
|
||||
_fields_ = [("BaseAddress", ctypes.c_void_p), ("AllocationBase", ctypes.c_void_p),
|
||||
("AllocationProtect", wt.DWORD), ("RegionSize", ctypes.c_size_t), ("State", wt.DWORD),
|
||||
("Protect", wt.DWORD), ("Type", wt.DWORD)]
|
||||
|
||||
|
||||
k32.VirtualQueryEx.argtypes = [wt.HANDLE, ctypes.c_void_p, ctypes.POINTER(_MBI), ctypes.c_size_t]
|
||||
k32.VirtualQueryEx.restype = ctypes.c_size_t
|
||||
MEM_COMMIT, PAGE_NOACCESS, PAGE_GUARD = 0x1000, 0x01, 0x100
|
||||
|
||||
|
||||
def search_memory(pid, needles, limit=1 << 32):
|
||||
"""Which of `needles` (bytes) occur anywhere in the committed memory of the process -> set."""
|
||||
h = k32.OpenProcess(PROCESS_VM_READ | PROCESS_QUERY_INFORMATION, False, pid)
|
||||
if not h:
|
||||
return set()
|
||||
found = set()
|
||||
try:
|
||||
addr, mbi = 0, _MBI()
|
||||
while addr < limit and len(found) < len(needles):
|
||||
if not k32.VirtualQueryEx(h, ctypes.c_void_p(addr), ctypes.byref(mbi), ctypes.sizeof(mbi)):
|
||||
break
|
||||
size = mbi.RegionSize
|
||||
if mbi.State == MEM_COMMIT and not mbi.Protect & (PAGE_NOACCESS | PAGE_GUARD):
|
||||
buf = ctypes.create_string_buffer(size)
|
||||
got = ctypes.c_size_t()
|
||||
if k32.ReadProcessMemory(h, ctypes.c_void_p(addr), buf, size, ctypes.byref(got)):
|
||||
data = buf.raw[:got.value]
|
||||
for n in needles:
|
||||
if n not in found and n in data:
|
||||
found.add(n)
|
||||
addr = (mbi.BaseAddress or 0) + size
|
||||
finally:
|
||||
k32.CloseHandle(h)
|
||||
return found
|
||||
|
||||
|
||||
u32 = ctypes.WinDLL("user32", use_last_error=True)
|
||||
WM_KEYDOWN, WM_KEYUP = 0x0100, 0x0101
|
||||
VK = dict(enter=0x0D, space=0x20, esc=0x1B, up=0x26, down=0x28, left=0x25, right=0x27, e=0x45)
|
||||
|
||||
@@ -119,3 +119,92 @@ def run(ws, key=DEFAULT_TARGET, wem=None, mode="modded", seconds=40.0, threshold
|
||||
report["result"] = "PASS" if ok else "FAIL"
|
||||
json.dump(report, open(logp.replace(".log", ".json"), "w"), indent=1)
|
||||
return report
|
||||
|
||||
|
||||
# --- in-memory sounds (soundbanks inside the packages) ------------------------------------------
|
||||
PROBE_RATE, PROBE_Q = 28000, 4 # a codebook setup the game's own WEMs do not use (E10)
|
||||
|
||||
|
||||
def bank_probes(ws, sounds):
|
||||
"""[(sound, probe WEM)]: the same probe format for every sound, shortened to fit its slot."""
|
||||
from ..encoder import encode
|
||||
x, sr = probe_signal(10.0)
|
||||
out = []
|
||||
for s in sounds:
|
||||
lim = ws.limit(s)
|
||||
secs = min(10.0, max(0.4, s.seconds or 1.0))
|
||||
while secs >= 0.1:
|
||||
wem, _ = encode(x[:int(secs * sr)], sr, PROBE_RATE, 1, PROBE_Q)
|
||||
if len(wem) <= lim:
|
||||
out.append((s, wem))
|
||||
break
|
||||
secs *= 0.7
|
||||
return out
|
||||
|
||||
|
||||
def run_banks(ws, keys=None, seconds=45.0, logdir=None, log=print):
|
||||
"""Replace in-memory sounds (default: every sound stored in the Frontend*/Startup packages) with probes, start the game,
|
||||
then check: no crash, the probe data is in the game's memory (the modified packages were loaded), and
|
||||
after the revert every touched file is byte-identical. Probes decoded (codec setup log) and heard
|
||||
(loopback correlation) are reported too, but no in-memory sound plays on the title screen without input,
|
||||
so they are not required. Returns a report dict (report['result'] PASS/FAIL)."""
|
||||
from .debugger import Run, kill_game, search_memory
|
||||
from .loopback import LoopbackRecorder, detect_chunks
|
||||
from ..wem import Wem
|
||||
if keys:
|
||||
sounds = [ws.catalog.resolve(k) for k in keys]
|
||||
else: # everything stored in the packages loaded for the title screen (rain ambience plays without input)
|
||||
sounds = [s for s in ws.catalog.sounds if s.archive == "BANK" and any(
|
||||
l.file.lower().startswith(("frontend", "startup")) for l in ws.locations(s))]
|
||||
probes = bank_probes(ws, sounds)
|
||||
files = sorted({f for s, _ in probes for f in ws.files(s)})
|
||||
logdir = logdir or os.path.join(os.getcwd(), "logs")
|
||||
os.makedirs(logdir, exist_ok=True)
|
||||
logp = os.path.join(logdir, "ingame_banks_%s.log" % time.strftime("%Y%m%d_%H%M%S"))
|
||||
report = dict(mode="banks", sounds=len(sounds), probes=len(probes), files=len(files))
|
||||
kill_game()
|
||||
before = {f: _sha1_file(f) for f in files}
|
||||
hashes = {"%08x" % Wem.parse(w).vorb["codebook_hash"] for _, w in probes}
|
||||
needles = set()
|
||||
for _, w in probes:
|
||||
if len(w) > 400:
|
||||
needles.add(w[len(w) // 2:len(w) // 2 + 32])
|
||||
report.update(probe_hashes=sorted(hashes), needles=len(needles))
|
||||
ws.apply([(s, w, {"source": "ingame-test"}) for s, w in probes], progress=log)
|
||||
from .debugger import ADDR
|
||||
# also log every setup parse (in-memory sources may not go through the stream setup cache)
|
||||
extra = 'bp %#x ".printf \\"ACKIT_PARSE2 hash=%%08x\\\\n\\", poi(poi(@esp+4)+0x9c); gc"' % ADDR["parse_setup"]
|
||||
runner = Run(ws.game.exe, logp, script_extra=extra)
|
||||
found = set()
|
||||
rec = LoopbackRecorder(seconds + 10)
|
||||
rec.start()
|
||||
try:
|
||||
report.update(runner.start())
|
||||
pid = report["pid"]
|
||||
while time.time() - runner.t0 < seconds and runner.alive():
|
||||
time.sleep(0.5)
|
||||
if re.search(r"^ACKIT_CRASH", runner.log(), re.M):
|
||||
break
|
||||
if runner.alive():
|
||||
found = search_memory(pid, needles)
|
||||
report["alive_at_end"] = runner.alive()
|
||||
finally:
|
||||
runner.stop()
|
||||
rec.stop_flag = True
|
||||
rec.join(5)
|
||||
ws.revert([s for s, _ in probes], progress=log)
|
||||
report["reverted_byte_identical"] = all(_sha1_file(f) == h for f, h in before.items())
|
||||
text = runner.log()
|
||||
report["crash"] = bool(re.search(r"^ACKIT_CRASH", text, re.M))
|
||||
ref = probe_signal(0.35)[0][:, 0]
|
||||
report["capture_error"] = rec.error
|
||||
report["ncc_probe"] = round(detect_chunks(rec.audio(), ref)[0], 3)
|
||||
seen = re.findall(r"ACKIT_(?:SETUP|PARSE2) hash=([0-9a-f]+)", text)
|
||||
report["probe_streams_decoded"] = sum(1 for h in seen if h in hashes)
|
||||
report["setups_logged"] = len(seen)
|
||||
report["probes_in_memory"] = len(found)
|
||||
ok = (not report["crash"] and report["alive_at_end"] and report["probes_in_memory"] > 0
|
||||
and report["reverted_byte_identical"])
|
||||
report["result"] = "PASS" if ok else "FAIL"
|
||||
json.dump(report, open(logp.replace(".log", ".json"), "w"), indent=1)
|
||||
return report
|
||||
|
||||
@@ -66,11 +66,23 @@ def run(ws, sample=300, log=print):
|
||||
break
|
||||
log("%-4s layout %s, rebuilt %d/%d sample WEMs byte-identical" % (arch, "OK" if not probs else probs[:2], good, n))
|
||||
ok &= not probs and good >= n - 2 # 4 looping music tracks are known exceptions
|
||||
bank = [s for s in ws.catalog.sounds if s.archive == "BANK"]
|
||||
if bank:
|
||||
n = good = 0
|
||||
for s in random.Random(1).sample(bank, min(len(bank), sample)):
|
||||
n += 1
|
||||
try:
|
||||
good += rebuild_identical(ws.current_bytes(s))
|
||||
except Exception:
|
||||
pass
|
||||
log("BANK in-memory sounds: rebuilt %d/%d sample WEMs byte-identical" % (good, n))
|
||||
ok &= good == n
|
||||
# 3
|
||||
t = np.arange(48000 * 2) / 48000.0
|
||||
sig = (0.3 * np.sin(2 * np.pi * 330 * t) * np.sin(2 * np.pi * 1.5 * t)).astype(np.float32)[:, None]
|
||||
for s in [x for x in ws.catalog.sounds if x.group == "Speech"][:1] + \
|
||||
[x for x in ws.catalog.sounds if x.group == "Music"][:1]:
|
||||
[x for x in ws.catalog.sounds if x.group == "Music"][:1] + \
|
||||
[x for x in ws.catalog.sounds if x.archive == "BANK" and x.seconds >= 1.5][:1]:
|
||||
wem, info = ws.prepare(s, sig, 48000)
|
||||
y, sr = audio.decode_wem(wem)
|
||||
ref = audio.to_channels(audio.resample(sig, 48000, sr), y.shape[1])
|
||||
|
||||
@@ -6,15 +6,20 @@ Safety rules (docs/decisions.md):
|
||||
* each WAD is rewritten atomically; several sounds of one WAD are written in one pass;
|
||||
* if a sound in the game is neither its original nor what we applied (Steam update, "verify
|
||||
integrity", another tool), we refuse to write it unless forced, and report it.
|
||||
In-memory sounds (archive BANK) and the prefetch heads of speech lines live in loose banks and .upk
|
||||
packages (ackit/banks.py). Before the first change to such a file a verified full copy is kept, and the
|
||||
file is always rebuilt as "original copy + every replacement currently applied", so reverting the last
|
||||
change restores it byte-for-byte and any later operation repairs an interrupted one.
|
||||
"""
|
||||
import hashlib
|
||||
import os
|
||||
from collections import defaultdict
|
||||
|
||||
from . import audio
|
||||
from . import audio, banks
|
||||
from .catalog import Catalog
|
||||
from .encoder import encode_fit
|
||||
from .game import Game, GameError
|
||||
from .inject import inject, read_target, parse_target, pack_bnk_path
|
||||
from .inject import inject, read_target, parse_target, pack_bnk_path, SizeLimitError
|
||||
from .originals import OriginalsStore, sha256, sha1_16
|
||||
from .pack import Pack
|
||||
from .paths import user_dir
|
||||
@@ -54,7 +59,35 @@ class Workspace:
|
||||
def wad_path(self, s):
|
||||
return self.game.wad(s.archive)
|
||||
|
||||
def bank_path(self, rel):
|
||||
return banks.file_path(self.game.cooked, rel)
|
||||
|
||||
def locations(self, s_or_key):
|
||||
"""Bank copies written together with a sound: every copy of a BANK sound, or the prefetch heads
|
||||
of a WAD speech line (usually none)."""
|
||||
key = s_or_key if isinstance(s_or_key, str) else s_or_key.key
|
||||
archive, _, target = key.partition(":")
|
||||
if archive == banks.ARCHIVE:
|
||||
return self.catalog.locations.get(int(target, 16), [])
|
||||
return self.catalog.heads.get(key, [])
|
||||
|
||||
def files(self, s):
|
||||
"""Absolute paths of the game files that writing `s` changes."""
|
||||
out = []
|
||||
if s.archive != banks.ARCHIVE:
|
||||
out.append(self.wad_path(s))
|
||||
if s.is_pack_media:
|
||||
out.append(pack_bnk_path(self.wad_path(s), parse_target(s.target)[0]))
|
||||
out += sorted({self.bank_path(l.file) for l in self.locations(s)})
|
||||
return out
|
||||
|
||||
def current_bytes(self, s):
|
||||
if s.archive == banks.ARCHIVE:
|
||||
locs = self.locations(s)
|
||||
if not locs:
|
||||
raise GameError("no copy of %s was found in the game's banks" % s.name)
|
||||
l = locs[0]
|
||||
return banks.riff_trim(banks.open_file(self.bank_path(l.file)).read(l.offset, l.size))
|
||||
return read_target(self.wad_path(s), s.target)
|
||||
|
||||
def original_bytes(self, s):
|
||||
@@ -81,6 +114,9 @@ class Workspace:
|
||||
"""Max size in bytes for a replacement (None = any size)."""
|
||||
if s.is_pack_media:
|
||||
return None
|
||||
if s.archive == banks.ARCHIVE:
|
||||
locs = self.locations(s)
|
||||
return min(l.size for l in locs) if locs else s.size
|
||||
e = self.store.get(s.key)
|
||||
if e:
|
||||
return e["orig_size"]
|
||||
@@ -115,6 +151,11 @@ class Workspace:
|
||||
|
||||
def _write(self, archive, items, progress=None):
|
||||
"""items: [(sound, new_bytes, meta, is_revert)] all in one archive."""
|
||||
if archive == banks.ARCHIVE:
|
||||
return self._write_banks(items, progress)
|
||||
heads = sorted({l.file for s, _, _, _ in items for l in self.locations(s)})
|
||||
for rel in heads:
|
||||
self._keep_original_file(rel)
|
||||
wad = self.game.wad(archive)
|
||||
self.game.check_writable(wad)
|
||||
packs = {parse_target(s.target)[0] for s, _, _, _ in items if s.is_pack_media}
|
||||
@@ -140,13 +181,109 @@ class Workspace:
|
||||
else:
|
||||
self.store.record_applied(s.key, data, meta)
|
||||
self.store.save()
|
||||
self._sync_bank_files(heads, progress)
|
||||
|
||||
# bank files (in-memory sounds, prefetch heads) ----------------------------------------------
|
||||
def _bank_files(self):
|
||||
return self.store.data.setdefault("bank_files", {})
|
||||
|
||||
def _fingerprint(self, path):
|
||||
"""Content hash of a bank file (whole file: any change by another tool is detected)."""
|
||||
h = hashlib.sha1()
|
||||
with open(path, "rb") as f:
|
||||
for c in iter(lambda: f.read(1 << 22), b""):
|
||||
h.update(c)
|
||||
return h.hexdigest()
|
||||
|
||||
def _keep_original_file(self, rel):
|
||||
"""Before the first change to a bank file: verified full copy + its fingerprint."""
|
||||
rec = self._bank_files().get(rel)
|
||||
if rec:
|
||||
return rec
|
||||
path = self.bank_path(rel)
|
||||
self.game.check_writable(path)
|
||||
self.store.full_backup(path, self.game.rel(path))
|
||||
rec = self._bank_files()[rel] = {"orig_fp": self._fingerprint(path), "written_fp": None}
|
||||
self.store.save()
|
||||
return rec
|
||||
|
||||
def changed_files(self, sounds):
|
||||
"""Bank files of `sounds` changed since we last wrote them (Steam update/verify, other tool)."""
|
||||
out = []
|
||||
for rel in sorted({l.file for s in sounds for l in self.locations(s)}):
|
||||
rec = self._bank_files().get(rel)
|
||||
if rec and self._fingerprint(self.bank_path(rel)) not in (rec["orig_fp"], rec["written_fp"]):
|
||||
out.append(rel)
|
||||
return out
|
||||
|
||||
def _edits(self, rel):
|
||||
"""[(offset, bytes)] that the replacements applied now need in bank file `rel`."""
|
||||
out = []
|
||||
for key, e in self.store.entries.items():
|
||||
if not e.get("applied"):
|
||||
continue
|
||||
locs = [l for l in self.locations(key) if l.file == rel]
|
||||
if not locs:
|
||||
continue
|
||||
data = self.store.get_blob(e["applied"])
|
||||
for l in locs:
|
||||
if l.kind == banks.MEMORY:
|
||||
payload = data + bytes(l.size - len(data))
|
||||
else: # head of a WAD line, which is stored zero-padded to its original size
|
||||
full = data + bytes(max(0, e["orig_size"] - len(data)))
|
||||
payload = (full + bytes(l.size))[:l.size]
|
||||
out.append((l.offset, payload))
|
||||
return out
|
||||
|
||||
def _sync_bank_files(self, rels, progress=None):
|
||||
"""Rebuild each bank file from its original copy and the replacements applied now."""
|
||||
for rel in rels:
|
||||
path = self.bank_path(rel)
|
||||
rec = self._keep_original_file(rel)
|
||||
orig = self.store.full_copy(self.game.rel(path))
|
||||
if orig is None:
|
||||
raise GameError("the original copy of %s is missing from the store" % rel)
|
||||
if progress:
|
||||
progress("writing %s" % rel)
|
||||
edits = self._edits(rel)
|
||||
data = banks.open_file(orig).patched(edits) if edits else open(orig, "rb").read()
|
||||
self.game.check_writable(path)
|
||||
tmp = path + ".ackit-tmp"
|
||||
with open(tmp, "wb") as f:
|
||||
f.write(data)
|
||||
os.replace(tmp, path)
|
||||
rec["written_fp"] = self._fingerprint(path) if edits else None
|
||||
self.store.save()
|
||||
|
||||
def _write_banks(self, items, progress=None):
|
||||
for s, data, meta, is_revert in items:
|
||||
locs = self.locations(s)
|
||||
if not locs:
|
||||
raise GameError("no copy of %s was found in the game's banks" % s.name)
|
||||
if not is_revert and len(data) > min(l.size for l in locs):
|
||||
raise SizeLimitError(s.target, min(l.size for l in locs), len(data))
|
||||
files = sorted({l.file for s, _, _, _ in items for l in self.locations(s)})
|
||||
for rel in files:
|
||||
self._keep_original_file(rel)
|
||||
for s, data, meta, is_revert in items:
|
||||
if is_revert:
|
||||
self.store.forget(s.key)
|
||||
else:
|
||||
self.store.remember_original(s.key, s.archive, s.target, self.locations(s)[0].file,
|
||||
self.current_bytes(s))
|
||||
self.store.record_applied(s.key, data, meta)
|
||||
self.store.save() # the store says what the files must contain before they change
|
||||
self._sync_bank_files(files, progress)
|
||||
|
||||
def apply(self, changes, force=False, progress=None):
|
||||
"""changes: [(sound, wem_bytes, meta dict)]. One atomic rewrite per WAD."""
|
||||
"""changes: [(sound, wem_bytes, meta dict)]. One atomic rewrite per WAD / bank file."""
|
||||
conflicts = []
|
||||
for s, _, _ in changes:
|
||||
if self.state(s) in (CHANGED, FOREIGN):
|
||||
conflicts.append(s)
|
||||
changed = set(self.changed_files([s for s, _, _ in changes]))
|
||||
conflicts += [s for s, _, _ in changes
|
||||
if s not in conflicts and {l.file for l in self.locations(s)} & changed]
|
||||
if conflicts and not force:
|
||||
raise ConflictError(conflicts)
|
||||
by_arch = defaultdict(list)
|
||||
@@ -191,6 +328,8 @@ class Workspace:
|
||||
shutil.copy2(src, tmp)
|
||||
os.replace(tmp, dst)
|
||||
out.append(rel)
|
||||
for rec in self._bank_files().values():
|
||||
rec["written_fp"] = None
|
||||
self.store.data["entries"] = {}
|
||||
self.store.save()
|
||||
self.store.gc()
|
||||
|
||||
@@ -260,10 +260,14 @@ class EditorPanel(QWidget):
|
||||
self.title.setText(s.name)
|
||||
who = s.character or (("mentions " + s.characters_inferred) if s.characters_inferred else "—")
|
||||
lim = self.ctx.ws.limit(s)
|
||||
self.meta.setText("%s / %s • %s • %.2f s, %d Hz, %s • max size: %s • %s"
|
||||
where = ""
|
||||
if s.bank:
|
||||
n = len(self.ctx.ws.locations(s))
|
||||
where = " • bank %s (%d %s)" % (s.bank, n, "copy" if n == 1 else "copies")
|
||||
self.meta.setText("%s / %s • %s • %.2f s, %d Hz, %s • max size: %s • %s%s"
|
||||
% (s.group, s.category, who, s.seconds, s.rate,
|
||||
{1: "mono", 2: "stereo"}.get(s.channels, "%d ch" % s.channels),
|
||||
"any" if lim is None else "%.1f KB" % (lim / 1024.0), s.key))
|
||||
"any" if lim is None else "%.1f KB" % (lim / 1024.0), s.key, where))
|
||||
key = s.key
|
||||
ws = self.ctx.ws
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@ from PySide6.QtWidgets import (QAbstractItemView, QCheckBox, QComboBox, QDialog,
|
||||
QPushButton, QScrollArea, QSplitter, QTableView, QToolBar, QVBoxLayout, QWidget)
|
||||
|
||||
from ackit import settings as settings_mod, workspace as W
|
||||
from ackit.catalog import available_archives
|
||||
from ackit.game import Game, GameError, default_game_dir
|
||||
from ackit.modpack import export_mod, match_mod, prepare_import
|
||||
from . import icons, theme
|
||||
@@ -257,8 +258,9 @@ class MainWindow(QMainWindow):
|
||||
full = self.settings.get("full_backup", True)
|
||||
|
||||
def work(job):
|
||||
return W.Workspace(game, store, [lang, "SFX"] if "SFX" in game.archives else [lang], full_backup=full,
|
||||
progress=lambda n, t: job.report("scanning %s: %d / %d" % (lang, n, t)))
|
||||
avail = available_archives(game)
|
||||
return W.Workspace(game, store, [a for a in (lang, "SFX", "BANK") if a in avail], full_backup=full,
|
||||
progress=lambda n, t: job.report("scanning game audio: %d / %d" % (n, t)))
|
||||
|
||||
def done(ws):
|
||||
self.loading = False
|
||||
|
||||
15405
data/catalog/BANK.csv
Normal file
15405
data/catalog/BANK.csv
Normal file
File diff suppressed because it is too large
Load Diff
BIN
data/catalog/BANK_locations.csv.gz
Normal file
BIN
data/catalog/BANK_locations.csv.gz
Normal file
Binary file not shown.
21634
data/catalog/DEU.csv
21634
data/catalog/DEU.csv
File diff suppressed because it is too large
Load Diff
21634
data/catalog/ESN.csv
21634
data/catalog/ESN.csv
File diff suppressed because it is too large
Load Diff
21634
data/catalog/FRA.csv
21634
data/catalog/FRA.csv
File diff suppressed because it is too large
Load Diff
21634
data/catalog/INT.csv
21634
data/catalog/INT.csv
File diff suppressed because it is too large
Load Diff
21634
data/catalog/ITA.csv
21634
data/catalog/ITA.csv
File diff suppressed because it is too large
Load Diff
@@ -2,9 +2,10 @@
|
||||
|
||||
| file | content |
|
||||
|------|---------|
|
||||
| `<ARCHIVE>.csv` | every sound of one audio archive: `ITA`, `INT`, `FRA`, `DEU`, `ESN` (speech, one per language) and `SFX` (music, ambience, effects of the SFX WAD; sounds stored inside soundbanks are not listed, see `docs/formats/banks.md`) |
|
||||
| `<ARCHIVE>.csv` | every sound of one audio archive: `ITA`, `INT`, `FRA`, `DEU`, `ESN` (speech WADs, one per language), `SFX` (music, ambience and effects streamed from the SFX WAD) and `BANK` (in-memory sounds stored inside soundbanks: footsteps, foley, combat, weapons, UI, efforts; `docs/formats/banks.md`) |
|
||||
| `BANK_locations.csv.gz` | every stored copy of the `BANK` sounds, and the prefetch heads of speech lines (below) |
|
||||
| `characters.csv` | `name,character`: verified speaker per sound name (language independent) |
|
||||
| `index.json` | fingerprint of the game WAD each CSV was built from (`ackit/game.py: wad_fingerprint`) and counts |
|
||||
| `index.json` | per archive: fingerprint of the game files the CSV was built from (`ackit/game.py: wad_fingerprint`, `ackit/banks.py: game_fingerprint`) and counts |
|
||||
| `source/<lang>/*.txt` | **the curated original lists** (one file per character, one wemscan-style file name per line, e.g. `Speech_TAPE_JOKER_1_4.wem`). `Files.txt` = all files, `Stats.txt` = counts, `Checkpoint.txt` = progress marker. Never edited by tools. |
|
||||
|
||||
All CSVs are UTF-8 with a header row. They contain only names and metadata (no game audio).
|
||||
@@ -12,26 +13,42 @@ All CSVs are UTF-8 with a header row. They contain only names and metadata (no g
|
||||
## `<ARCHIVE>.csv` columns
|
||||
| column | meaning |
|
||||
|--------|---------|
|
||||
| `archive` | `ITA`, `INT`, ... or `SFX` |
|
||||
| `target` | location inside the archive: `HASH` (entry with one WEM), `HASH:i` (i-th variation, 0-based), `PACK/MEDIA` (music stream pack) |
|
||||
| `name` | display name (`NAME`, `NAME_1..n` for variations, see `ackit/catalog.py`) |
|
||||
| `aliases` | other accepted names, `\|`-separated (`NAME_1` for single entries, `NAME#i` for variations) |
|
||||
| `archive` | `ITA`, `INT`, ..., `SFX` or `BANK` |
|
||||
| `target` | location: `HASH` (WAD entry with one WEM), `HASH:i` (i-th variation, 0-based), `PACK/MEDIA` (music stream pack), `MEDIA` (BANK: Wwise media ID) |
|
||||
| `name` | display name: `NAME`, `NAME_1..n` for variations (`NAME#i` when `NAME_i` is already another entry's own name); BANK: the Wwise event that plays it (`Play_..._i`) or `<Bank>_<n>` |
|
||||
| `aliases` | other accepted names, `\|`-separated (`NAME_1` for single entries, `NAME#i` for variations, `<Bank>/<MEDIA>` for BANK) |
|
||||
| `group` | Speech, Music, Ambience, SFX, Unknown (`docs/categories.md`) |
|
||||
| `category` | finer category (`docs/categories.md`) |
|
||||
| `bank` | BANK: the soundbank holding the sound (first by name if several do); empty for WAD sounds |
|
||||
| `character` | verified speaker(s) from `characters.csv`, `; `-separated, empty if unknown |
|
||||
| `characters_inferred` | characters/roles mentioned by the name (hint only) |
|
||||
| `characters_inferred` | characters/roles mentioned by the name (and bank) (hint only) |
|
||||
| `seconds`, `rate`, `channels` | of the original sound |
|
||||
| `size` | size of the original WEM in bytes = maximum size of a replacement (except music packs: any size) |
|
||||
| `sha1` | first 16 hex digits of the original WEM's SHA-1 (detects files changed outside the tool) |
|
||||
| `sha1` | first 16 hex digits of the original WEM's SHA-1 (detects files changed outside the tool; empty when the copies of a BANK sound differ) |
|
||||
| `notes` | free text (empty; for future curation, e.g. transcripts) |
|
||||
|
||||
The editor uses these files directly when the installed WAD matches `index.json`; for another game build it
|
||||
scans the WAD itself and applies the same rules (no checksums then).
|
||||
## `BANK_locations.csv.gz` columns (gzip-compressed CSV)
|
||||
| column | meaning |
|
||||
|--------|---------|
|
||||
| `media` | Wwise media ID (hex) |
|
||||
| `file` | file under `BmGame/CookedPCConsole`, `/`-separated (`SFX/xxxxxxxx.bnk` or `Name.upk`) |
|
||||
| `offset` | position of the WEM in the *uncompressed* file (hex; `docs/formats/upk.md`) |
|
||||
| `size` | bytes reserved for it there |
|
||||
| `kind` | `m` = copy of a BANK sound, `p` = prefetch head of a speech line |
|
||||
| `sound` | `p` rows: key of that speech line (e.g. `ITA:2B14E472:3`); each language has its own lines and heads |
|
||||
|
||||
A replacement is written into every `m` copy of a BANK sound; replacing a speech line also rewrites its `p`
|
||||
heads (`docs/formats/banks.md`).
|
||||
|
||||
The editor uses these files directly when the installed game matches `index.json`; for another game build it
|
||||
scans the game itself and applies the same rules (no checksums then; the bank scan takes a few minutes).
|
||||
`ackit verify --catalog` cross-checks the catalog in use against the game files (`--quick` samples packages).
|
||||
|
||||
## Rebuilding / adding a language
|
||||
1. Make sure the game files are original (`ackit status`; `ackit revert --all`).
|
||||
2. Optional: add listening-based lists for another language in `source/<lang>/<Character>.txt` (same format;
|
||||
names are the same in every language, so the Italian lists already label all languages).
|
||||
3. Run `python tools/build_catalog.py` - it scans every archive of the installed game (all languages present)
|
||||
and rewrites the CSVs, `characters.csv` and `index.json`.
|
||||
3. Run `python tools/build_catalog.py` - it scans every archive and all soundbanks of the installed game (all
|
||||
languages present; ~5 minutes) and rewrites the CSVs, the location index, `characters.csv` and `index.json`.
|
||||
4. Run `ackit verify --catalog` (no issues expected).
|
||||
New character labels need a display name in `ackit/classify.py: USER_LABELS` (otherwise the file name is used).
|
||||
|
||||
1304
data/catalog/SFX.csv
1304
data/catalog/SFX.csv
File diff suppressed because it is too large
Load Diff
@@ -1381,6 +1381,7 @@ Speech_GCPD_Joker_laugh_2,Joker
|
||||
Speech_GCPD_Radio_intro_1,Penguin
|
||||
Speech_GCPD_Radio_intro_2,Penguin
|
||||
Speech_GCPD_Radio_intro_3,Penguin
|
||||
Speech_INTERROGATION_Cat_01#3,Catwoman
|
||||
Speech_INTERROGATION_Cat_01_2,Catwoman
|
||||
Speech_INTERROGATION_Cat_01_3,Catwoman
|
||||
Speech_INTERROGATION_CH5_MAYOR_01_4,Batman
|
||||
@@ -1994,6 +1995,7 @@ Speech_RIDDLER_Enigma_Control_box_goto_church_3,Riddler
|
||||
Speech_RIDDLER_Enigma_Control_box_too_soon_1,Riddler
|
||||
Speech_RIDDLER_Enigma_Control_box_too_soon_2,Riddler
|
||||
Speech_RIDDLER_Enigma_Control_box_too_soon_3,Riddler
|
||||
Speech_RIDDLER_Enigma_Fail#1,Riddler
|
||||
Speech_RIDDLER_Enigma_Fail_1,Riddler
|
||||
Speech_RIDDLER_Enigma_Fail_10,Riddler
|
||||
Speech_RIDDLER_Enigma_Fail_11,Riddler
|
||||
|
||||
|
@@ -61,5 +61,25 @@
|
||||
},
|
||||
"with_character": 0
|
||||
},
|
||||
"BANK": {
|
||||
"fingerprint": "3f9766af425b8eef",
|
||||
"sounds": 15404,
|
||||
"groups": {
|
||||
"Ambience": 1015,
|
||||
"SFX": 6912,
|
||||
"Speech": 7471,
|
||||
"Music": 2,
|
||||
"Unknown": 4
|
||||
},
|
||||
"copies": 93910,
|
||||
"files": 1499,
|
||||
"named_by_event": 6416,
|
||||
"prefetch_heads": {
|
||||
"lines": 396,
|
||||
"copies": 2101,
|
||||
"unmapped": 0
|
||||
},
|
||||
"issues": []
|
||||
},
|
||||
"_info": "unmodified game files"
|
||||
}
|
||||
@@ -25,19 +25,31 @@ list; unnamed IDs show as hex). First matching rule wins:
|
||||
| `Speech_CHAR_` | Speech | Character lines |
|
||||
| `Speech_CH_`, `Speech_SEQUENCE_`, `Speech_STEEL_`, `_MUSEUM_`, `_CHURCH_`, `_COURT_`, `_UNDER_`, `_GCPD_`, `_ZP1_` | Speech | Story missions |
|
||||
| `Speech_GAME_`, `Speech_MAP_`, `Speech_AUGMENTED_`, `Speech_FORENSICS_`, `Speech_TEMP_` | Speech | Gameplay & hints |
|
||||
| `NPC_` | Ambience | Crowds |
|
||||
| `NPC_...Crowd...` | Ambience | Crowds |
|
||||
| other `Speech_` | Speech | Other speech |
|
||||
| media of the `Music` / `Music_PDLC` stream packs, `Music*` | Music | Music / Music (DLC) |
|
||||
| media of the `Music` / `Music_PDLC` stream packs, `Music*`, `MUS_` | Music | Music / Music (DLC) |
|
||||
| `AMB_`, `PSA_` | Ambience | Ambience |
|
||||
| `PSO_` (flies, drips, fans, lights) | Ambience | Ambient emitters |
|
||||
| `WND_` | Ambience | Wind |
|
||||
| `WTR_` | Ambience | Weather |
|
||||
| `VOL_`, `PVOL_` | Ambience | Room tone |
|
||||
| `SEQ_` | SFX | Scripted sequences |
|
||||
| anything containing `Footstep` | SFX | Footsteps |
|
||||
| `CHR_`/`NPC_` containing `Combat`, `BoneBreak`, `Bodyfall` | SFX | Combat |
|
||||
| `SEQ_`, `MTN_` | SFX | Scripted sequences |
|
||||
| `OBJ_` | SFX | Objects |
|
||||
| `PX_`, `PHY_` | SFX | Physics |
|
||||
| `WPN_` | SFX | Weapons |
|
||||
| `CHR_` (in SFX) | SFX | Character foley |
|
||||
| `DVC_` | SFX | Gadgets |
|
||||
| `GUI_`, `SBJ_` | SFX | Interface |
|
||||
| `VFX_` | SFX | Visual effects |
|
||||
| other `NPC_` | SFX | Enemies & NPCs |
|
||||
| `CHR_`, `GEN_` | SFX | Character foley |
|
||||
| `Global` | SFX | Global |
|
||||
| anything else (e.g. unnamed hex IDs) | Unknown | Unknown |
|
||||
|
||||
BANK sounds (in-memory, `docs/formats/banks.md`) are classified by the name of their **soundbank** (e.g.
|
||||
`CHR_CatwomanFoleySuit`, `Speech_EMOTE_Batman`), which is more systematic than their event names.
|
||||
|
||||
Counts in the ITA catalog: Story missions 1993, Predator chatter 1796, Character lines 1537, Open-world chatter
|
||||
1295, Side missions 808, Combat barks 801, Efforts & emotes 616, Interview tapes 514, Riddler 360, Radio & TV 226,
|
||||
Gameplay & hints 199, DLC 167, Boss fights 150, Cinematics 150, Reactions 108, Interrogations 95. SFX archive:
|
||||
|
||||
@@ -55,8 +55,29 @@ loopback); it is a developer tool and not part of the editor.
|
||||
Speech/ambience variations are addressed by offset+size from BKPK tables inside compressed `.upk` packages,
|
||||
which ackit does not rewrite. Replacements are written in place and must not exceed the original WEM's size;
|
||||
the encoder lowers quality, then sample rate, to fit, and refuses otherwise. Music stream packs have a loose
|
||||
index (`SFX\*.bnk`) that ackit rewrites, so music can be any length. (Rewriting `.upk` packages would lift the
|
||||
limit but means LZO recompression of large engine packages: out of scope, high risk.)
|
||||
index (`SFX\*.bnk`) that ackit rewrites, so music can be any length. In-memory sounds follow the same rule
|
||||
(below). Lifting the limit would mean re-laying out the uncompressed packages (export offsets, BKPK/DIDX
|
||||
tables): high risk for little gain, not done.
|
||||
|
||||
## 4b. In-memory sounds (soundbanks in loose `.bnk` files and `.upk` packages)
|
||||
- **Model:** one catalog row per Wwise media ID (archive `BANK`, target = media ID), not per bank: the same
|
||||
media is stored in up to dozens of bank copies (banks are cooked into every package that uses them), and the
|
||||
game may load any of them. A replacement is written into **every** copy (`BANK_locations.csv.gz`).
|
||||
- **In place:** each copy keeps its size (zero-padded, like WAD speech); DIDX/HIRC offsets and the package
|
||||
layout never change, so nothing else in the engine data has to be rewritten.
|
||||
- **Packages:** ackit reads the LZO packages itself (pure Python, ~47 MB/s, block-level access) and writes
|
||||
only the changed 128 KiB blocks, stored as a single LZO literal run (valid LZO1X without matches), then
|
||||
fixes the chunk table. No compressor and no external tool are needed; untouched blocks stay byte-identical.
|
||||
- **Revert:** before the first change to a bank file a verified full copy goes to the store (mandatory, not
|
||||
optional like the WAD copies: the file is rebuilt from it). The file content is always "original copy +
|
||||
replacements applied now", so reverting the last one restores the original bytes exactly, and an interrupted
|
||||
operation is repaired by the next one. Conflicts: the whole-file SHA-1 must be the original's or the last one
|
||||
ackit wrote.
|
||||
- **Prefetch heads:** 396 speech lines (about 79 per language) have their first bytes copied into banks; a
|
||||
replacement of such a line rewrites those heads too, otherwise the start would play the old line.
|
||||
- **Names:** the hash list names none of these media. Names come from Wwise events whose names are in the
|
||||
packages' name tables (FNV-1 of the name = event ID), following event -> action -> containers -> sounds
|
||||
(6,416 sounds); the rest are numbered per bank. Categories use the bank name.
|
||||
|
||||
## 5. Catalog
|
||||
- Shipped CSVs per archive (names, categories, characters, original size + checksum) make loading instant and
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
|
||||
Most short sound effects (footsteps, foley, combat, weapons, objects, UI, grunts) are **not in the WADs**: they
|
||||
are WEMs stored inside Wwise soundbanks, either loose in `SFX\*.bnk` or embedded in the UE3 packages (`.upk`).
|
||||
The catalog and the editor only cover the WADs (`wad.md`), so none of these sounds are listed or editable yet.
|
||||
Survey of the installed game (buildid 3718300) on 2026-10-09, read-only.
|
||||
ackit lists them as the catalog archive `BANK` and writes them (see the end of this page).
|
||||
Survey of the installed game (buildid 3718300), 2026-10-09.
|
||||
|
||||
## Inventory
|
||||
|
||||
@@ -60,11 +60,12 @@ WAD sounds have such heads (2,101 copies); replacing one of them must rewrite th
|
||||
- Largest: `CHR_RobinFoleyCape` 300 WEMs, `CHR_BatmanFoleyCape` 282, `CHR_Clayface` 263,
|
||||
`CHR_NightwingFoley` 242, `CHR_CatwomanFoleySuit` 225, `CHR_MrFreeze` 176, `CHR_Combat` 166.
|
||||
|
||||
## Consequences for editing (not implemented)
|
||||
- **Loose banks**: a replacement means rebuilding `DATA`/`DIDX` and the in-memory offsets/sizes in the HIRC
|
||||
Sound objects of that bank (and in any other bank whose sources use its media). Whether the game accepts a
|
||||
bank that changed size is untested.
|
||||
- **Embedded banks**: the package must be rewritten (decompress, patch every blob of the bank, recompress
|
||||
LZO, fix the package's chunk table and export offsets/sizes). A sound must be replaced in **every** package
|
||||
that carries a copy, otherwise the game may load the original from another package.
|
||||
- Either way: unverified in game so far; the same WEM rules apply (`wem.md`).
|
||||
## How ackit edits them
|
||||
- Catalog archive `BANK` (one row per media ID) + `data/catalog/BANK_locations.csv.gz` (every copy, and the
|
||||
prefetch heads of speech lines); built by `tools/build_catalog.py`, checked by `ackit verify --catalog`.
|
||||
- A replacement is written **in place into every copy** (no larger than the smallest copy, zero-padded), so
|
||||
DIDX/HIRC offsets and sizes and the package layout never change. Loose banks are patched directly; in
|
||||
packages only the touched blocks are re-encoded (`upk.md`). A sound left in one package only would let the
|
||||
game load the original from another one.
|
||||
- Replacing a speech line that has prefetch heads rewrites the heads (first bytes of the new, zero-padded line).
|
||||
- Design and revert rules: `docs/decisions.md` (4b). The same WEM rules apply (`wem.md`).
|
||||
|
||||
@@ -72,6 +72,19 @@ Conclusion: approach A works end-to-end in-game. Use >= 30 s probes for the intr
|
||||
UI, grunts) are in-memory WEMs inside soundbanks: 65 loose `SFX\*.bnk` (1,117) and 3,340 banks embedded in
|
||||
the `.upk` packages (15,641; many banks copied into several packages). None are in the catalog; the WAD
|
||||
side is complete. Inventory, layout and editing consequences: `docs/formats/banks.md`.
|
||||
17. (2026-10-09) Speech lines have **per-language media IDs**: every package carries one BKPK table per language
|
||||
for the same WAD entry. 396 lines (78-80 per language) have a prefetch head (first N bytes) in banks; each
|
||||
head equals the start of exactly one language's WEM.
|
||||
18. (2026-10-09) In-game test of in-memory sounds (`ackit verify --ingame --banks`): 77 probes written into the
|
||||
78 sounds stored in the Frontend*/Startup packages (54 files: patched LZO packages + loose banks). The game
|
||||
starts and runs, all 68 probe byte patterns are found in its memory (it loaded the patched packages), revert
|
||||
byte-identical: PASS. Not shown: a probe being *heard*. Only 7 Vorbis streams start on the title screen
|
||||
(setups 4f708975, 6ded1c65, 92f94a3f, d93b5944, d4b2e824: shared by hundreds of sounds, music included) and
|
||||
none is a probe. Tried: Enter/arrow keys posted to the game window (`debugger.post_key`, WM_KEYDOWN) to reach
|
||||
the main menu: no UI sound followed (the game probably reads input another way; global SendInput is not
|
||||
used on purpose); a breakpoint on the setup parser entry: no extra events. Hypotheses: no in-memory sound
|
||||
plays without input; the decode path is the same as for WAD sounds (same WEM rules, 300/300 in-memory WEMs
|
||||
rebuild byte-identically), so playback is expected to work like E4/E10.
|
||||
|
||||
## Old path -> new path
|
||||
`scripts/harness.py` -> `ackit verify --ingame` (ackit/verify/ingame.py); `scripts/validate_builder.py` ->
|
||||
|
||||
@@ -20,14 +20,16 @@ back with one click.
|
||||
- Type in the **search box** (Ctrl+F): words can be parts of the name, a category, a character, your own
|
||||
notes or tags, or an ID like `b49cdf48`. All words must match.
|
||||
- Narrow down with **Group** (Speech, Music, Ambience, SFX), **Category** (e.g. *Interview tapes*,
|
||||
*Combat barks*, *Cinematics*), **Character**, **Archive** (speech language or SFX), **Tag**, **Modified
|
||||
*Combat barks*, *Cinematics*), **Character**, **Archive** (speech language, SFX or BANK), **Tag**, **Modified
|
||||
only** and **Favorites only**. Combine them freely; results update instantly.
|
||||
- **Character** is the verified speaker from the curated catalog. Grey `~` names are only *mentioned* in the
|
||||
sound's name (useful hints for the many unlabelled lines); tick *Include characters mentioned in names*
|
||||
to filter on them too.
|
||||
- Click a column header to sort (length, size, status, ...). Ctrl/Shift-click selects several rows.
|
||||
- Most short effects (footsteps, foley, weapons, menu sounds) are stored inside soundbanks and are not listed
|
||||
yet; the list covers voice lines, music, ambience and the effects of the SFX archive.
|
||||
- Archive **BANK** holds the ~15,400 short effects stored inside soundbanks (footsteps, foley, combat,
|
||||
weapons, menu sounds, grunts). Many are named after the game event that plays them (`Play_...`), the
|
||||
rest after their bank (`CHR_Bane_3`). The editor shows the bank and how many copies the game keeps of
|
||||
the sound; a replacement is written into every copy.
|
||||
- **Language** (top left) switches the speech language. Speech is read from the language the game is set to.
|
||||
|
||||
|
||||
|
||||
@@ -89,9 +89,10 @@ def pack(media):
|
||||
return bytes(hdr) + bank, bytes(payload)
|
||||
|
||||
|
||||
def bank(bank_id, media, name=None, extra_sources=()):
|
||||
def bank(bank_id, media, name=None, extra_sources=(), prefetch=()):
|
||||
"""A v56 soundbank holding `media` {id: wem} in memory (DIDX/DATA, 16-byte aligned) with one HIRC
|
||||
Sound object per media; extra_sources: [(media id, bank id, off, size)] for sources of other banks."""
|
||||
Sound object per media; extra_sources: [(media id, bank id, off, size)] for sources of other banks;
|
||||
media ids in `prefetch` get a prefetch source (their DIDX data is the head of a streamed WEM)."""
|
||||
data = bytearray()
|
||||
didx = b""
|
||||
for m in sorted(media):
|
||||
@@ -104,11 +105,12 @@ def bank(bank_id, media, name=None, extra_sources=()):
|
||||
objs = []
|
||||
for i, m in enumerate(sorted(media)):
|
||||
off = struct.unpack_from("<I", didx, 12 * i + 4)[0]
|
||||
objs.append((m, bank_id, data_pos + off, len(media[m])))
|
||||
objs += list(extra_sources)
|
||||
objs.append((m, bank_id if m not in prefetch else 0x1234, data_pos + off, len(media[m]),
|
||||
2 if m in prefetch else 0))
|
||||
objs += [x + (0,) for x in extra_sources]
|
||||
hirc = struct.pack("<I", len(objs))
|
||||
for i, (m, b, off, size) in enumerate(objs):
|
||||
body = struct.pack("<7I", 0x100 + i, 0x00040001, 0, m, b, off, size) + bytes(6)
|
||||
for i, (m, b, off, size, stream) in enumerate(objs):
|
||||
body = struct.pack("<7I", 0x100 + i, 0x00040001, stream, m, b, off, size) + bytes(6)
|
||||
hirc += bytes([2]) + struct.pack("<I", len(body)) + body
|
||||
out = head + b"DATA" + struct.pack("<I", len(data)) + bytes(data) + b"HIRC" + struct.pack("<I", len(hirc)) + hirc
|
||||
if name:
|
||||
@@ -172,4 +174,44 @@ def make_game(root):
|
||||
originals = {"%08X:%d" % (SPEECH_ENTRY, i): w for i, w in enumerate(v)}
|
||||
originals["%08X" % SINGLE_ENTRY] = single
|
||||
originals.update({"%08X/%08X" % (PACK_ID, m): w for m, w in music.items()})
|
||||
originals.update(make_banks(cooked, v))
|
||||
return originals
|
||||
|
||||
|
||||
# soundbanks (docs/formats/banks.md): media ids of the in-memory sounds
|
||||
FOOT = (0x0A000001, 0x0A000002) # loose bank CHR_SynthFootsteps_Media
|
||||
FOLEY = (0x0B000001, 0x0B000002) # bank CHR_SynthFoley, copied into both packages
|
||||
GUI = (0x0C000001,) # bank GUI_Synth (package B only)
|
||||
LINE_MEDIA = (0x0D000000, 0x0D000001, 0x0D000002) # INT media ids of the SPEECH_ENTRY variations
|
||||
HEAD = 0x200 # prefetch head size of variation 1
|
||||
|
||||
|
||||
def make_banks(cooked, speech_wems):
|
||||
"""Loose bank + two packages (one bank in both) + a prefetch head of INT SPEECH_ENTRY variation 1."""
|
||||
from ackit.bnk import fnv1
|
||||
names = ["Init", "CHR_SynthFootsteps_Media", "CHR_SynthFoley", "GUI_Synth", "Speech_Synth"]
|
||||
with open(os.path.join(cooked, "CookedBanks.txt"), "w", newline="\r\n") as f:
|
||||
f.write("\n".join(names) + "\n")
|
||||
foot = {FOOT[0]: wem(0.4, 800), FOOT[1]: wem(0.3, 900)}
|
||||
foley = {FOLEY[0]: wem(0.5, 1000), FOLEY[1]: wem(0.6, 1100, 32000)}
|
||||
gui = {GUI[0]: wem(0.5, 1200)}
|
||||
open(os.path.join(cooked, "SFX", "%08X.bnk" % fnv1(names[1])), "wb").write(
|
||||
bank(fnv1(names[1]), foot, name=names[1]))
|
||||
# BKPK table of SPEECH_ENTRY for INT (media ids differ per language; offsets as in the WAD entry)
|
||||
offs, pos = [], 0
|
||||
for w in speech_wems:
|
||||
pos = _a(pos, 0x800)
|
||||
offs.append(pos)
|
||||
pos += len(w)
|
||||
tbl = bytearray(0x48)
|
||||
struct.pack_into("<4sII", tbl, 0, b"BKPK", 0x01020000, SPEECH_ENTRY)
|
||||
struct.pack_into("<IIIIII", tbl, 0x10, 3, 0, 36, 0x48, _a(pos, 0x800), 0)
|
||||
tbl += b"".join(struct.pack("<III", m, len(w), o) for m, w, o in zip(LINE_MEDIA, speech_wems, offs))
|
||||
speech = bank(fnv1(names[4]), {LINE_MEDIA[1]: speech_wems[1][:HEAD]}, prefetch=(LINE_MEDIA[1],))
|
||||
filler = bytes(range(256)) * 9
|
||||
synth_a = filler + bank(fnv1(names[2]), foley) + filler[:777] + bytes(tbl) + speech + filler
|
||||
synth_b = filler[:1000] + bank(fnv1(names[3]), gui) + filler + bank(fnv1(names[2]), foley) + filler
|
||||
package(os.path.join(cooked, "Synth_A.upk"), synth_a)
|
||||
package(os.path.join(cooked, "Synth_B.upk"), synth_b)
|
||||
out = {"%08X" % m: w for m, w in list(foot.items()) + list(foley.items()) + list(gui.items())}
|
||||
return out
|
||||
|
||||
@@ -93,6 +93,24 @@ def test_replace_apply_revert_byte_identical(win, tmp_path):
|
||||
assert not w.ws.store.entries
|
||||
|
||||
|
||||
def test_bank_sound_apply_revert_in_editor(win, tmp_path):
|
||||
w, root, originals = win
|
||||
before = files(root)
|
||||
w.cmb_arch.setCurrentText("BANK")
|
||||
assert w.proxy.rowCount() == 5
|
||||
assert w.select_name("CHR_SynthFoley_1")
|
||||
ed = w.editor
|
||||
assert "bank CHR_SynthFoley (2 copies)" in ed.meta.text()
|
||||
wav = str(tmp_path / "foley.wav")
|
||||
audio.save_wav(wav, *synth.tone(0.2, 44100, 1300))
|
||||
assert ed.load_file(wav)
|
||||
ed._update_new()
|
||||
ed.apply()
|
||||
assert w.ws.state(ed.sound) == W.MODIFIED and files(root) != before
|
||||
ed.revert()
|
||||
assert files(root) == before and not w.ws.store.entries
|
||||
|
||||
|
||||
def test_record_simulated_input(win):
|
||||
w, root, _ = win
|
||||
w.select_name("CHR_RadioScanner_1")
|
||||
|
||||
162
tests/test_banks.py
Normal file
162
tests/test_banks.py
Normal file
@@ -0,0 +1,162 @@
|
||||
"""In-memory sounds (archive BANK) and prefetch heads: catalog scan, apply/revert through Workspace."""
|
||||
import hashlib
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
import synth
|
||||
from ackit import banks
|
||||
from ackit.game import Game
|
||||
from ackit.inject import SizeLimitError
|
||||
from ackit.upk import Package
|
||||
from ackit.workspace import Workspace, ConflictError, MODIFIED, ORIGINAL
|
||||
|
||||
|
||||
def _sha(p):
|
||||
return hashlib.sha1(open(p, "rb").read()).hexdigest()
|
||||
|
||||
|
||||
def _ws(root, tmp_path):
|
||||
return Workspace(root, store_root=str(tmp_path / "store"), archives=["INT", "SFX", "BANK"])
|
||||
|
||||
|
||||
def _cooked(root):
|
||||
return Game(root).cooked
|
||||
|
||||
|
||||
def _copy(ws, l):
|
||||
return banks.open_file(ws.bank_path(l.file)).read(l.offset, l.size)
|
||||
|
||||
|
||||
def test_bank_catalog(synth_game, tmp_path):
|
||||
root, originals = synth_game
|
||||
ws = _ws(root, tmp_path)
|
||||
bank = {s.target: s for s in ws.catalog.sounds if s.archive == "BANK"}
|
||||
assert set(bank) == {"%08X" % m for m in synth.FOOT + synth.FOLEY + synth.GUI}
|
||||
foot = bank["%08X" % synth.FOOT[0]]
|
||||
assert foot.bank == "CHR_SynthFootsteps_Media" and foot.name == "CHR_SynthFootsteps_Media_1"
|
||||
assert (foot.group, foot.category) == ("SFX", "Footsteps")
|
||||
assert bank["%08X" % synth.GUI[0]].category == "Interface"
|
||||
foley = bank["%08X" % synth.FOLEY[1]]
|
||||
assert foley.rate == 32000 and foley.size == len(originals[foley.target])
|
||||
assert sorted(l.file for l in ws.locations(foley)) == ["Synth_A.upk", "Synth_B.upk"]
|
||||
for s in bank.values():
|
||||
assert ws.current_bytes(s) == originals[s.target]
|
||||
assert all(_copy(ws, l) == originals[s.target] for l in ws.locations(s))
|
||||
# the prefetch head belongs to INT variation 1 of the speech entry, not to the BANK archive
|
||||
line = ws.catalog.by_key["INT:%08X:1" % synth.SPEECH_ENTRY]
|
||||
heads = ws.locations(line)
|
||||
assert [(l.file, l.size, l.kind) for l in heads] == [("Synth_A.upk", synth.HEAD, "p")]
|
||||
assert _copy(ws, heads[0]) == originals["%08X:1" % synth.SPEECH_ENTRY][:synth.HEAD]
|
||||
assert ws.locations(ws.catalog.by_key["INT:%08X:0" % synth.SPEECH_ENTRY]) == []
|
||||
|
||||
|
||||
def test_bank_apply_every_copy_and_revert_byte_identical(synth_game, tmp_path):
|
||||
root, originals = synth_game
|
||||
ws = _ws(root, tmp_path)
|
||||
cooked = _cooked(root)
|
||||
files = [os.path.join(cooked, f) for f in ("Synth_A.upk", "Synth_B.upk")]
|
||||
before = {f: _sha(f) for f in files}
|
||||
s = ws.catalog.by_key["BANK:%08X" % synth.FOLEY[0]]
|
||||
new = synth.wem(0.2, 1500)
|
||||
assert len(new) < ws.limit(s)
|
||||
ws.apply([(s, new, {"source": "test"})])
|
||||
assert ws.state(s) == MODIFIED
|
||||
for l in ws.locations(s):
|
||||
assert _copy(ws, l) == new + bytes(l.size - len(new))
|
||||
other = ws.catalog.by_key["BANK:%08X" % synth.FOLEY[1]]
|
||||
assert all(_copy(ws, l) == originals[other.target] for l in ws.locations(other))
|
||||
assert all(_sha(f) != before[f] for f in files)
|
||||
assert Package(files[1]).size == Package(ws.store.full_copy(ws.game.rel(files[1]))).size # layout unchanged
|
||||
ws.revert([s])
|
||||
assert ws.state(s) == ORIGINAL and {f: _sha(f) for f in files} == before
|
||||
|
||||
|
||||
def test_loose_bank_in_place(synth_game, tmp_path):
|
||||
root, originals = synth_game
|
||||
ws = _ws(root, tmp_path)
|
||||
s = ws.catalog.by_key["BANK:%08X" % synth.FOOT[1]]
|
||||
path = ws.files(s)[0]
|
||||
before, size = _sha(path), os.path.getsize(path)
|
||||
ws.apply([(s, synth.wem(0.1, 2000), {})])
|
||||
assert os.path.getsize(path) == size and _sha(path) != before
|
||||
ws.revert_all()
|
||||
assert _sha(path) == before
|
||||
|
||||
|
||||
def test_partial_revert_keeps_other_changes(synth_game, tmp_path):
|
||||
root, originals = synth_game
|
||||
ws = _ws(root, tmp_path)
|
||||
path = os.path.join(_cooked(root), "Synth_B.upk")
|
||||
before = _sha(path)
|
||||
a = ws.catalog.by_key["BANK:%08X" % synth.FOLEY[0]]
|
||||
b = ws.catalog.by_key["BANK:%08X" % synth.GUI[0]]
|
||||
na, nb = synth.wem(0.2, 1500), synth.wem(0.2, 1700)
|
||||
ws.apply([(a, na, {}), (b, nb, {})])
|
||||
ws.revert([a])
|
||||
assert ws.current_bytes(a) == originals[a.target]
|
||||
assert ws.current_bytes(b).startswith(nb) and ws.state(b) == MODIFIED
|
||||
ws.revert([b])
|
||||
assert _sha(path) == before
|
||||
|
||||
|
||||
def test_speech_line_with_prefetch_head(synth_game, tmp_path):
|
||||
root, originals = synth_game
|
||||
ws = _ws(root, tmp_path)
|
||||
line = ws.catalog.by_key["INT:%08X:1" % synth.SPEECH_ENTRY]
|
||||
files = ws.files(line)
|
||||
assert [os.path.basename(f) for f in files] == ["28CCF006_LOC_INT.wad", "Synth_A.upk"]
|
||||
before = {f: _sha(f) for f in files}
|
||||
new = synth.wem(0.3, 640)
|
||||
ws.apply([(line, new, {})])
|
||||
head = ws.locations(line)[0]
|
||||
assert _copy(ws, head) == (new + bytes(synth.HEAD))[:synth.HEAD]
|
||||
assert ws.current_bytes(line).startswith(new)
|
||||
ws.revert([line])
|
||||
assert {f: _sha(f) for f in files} == before
|
||||
|
||||
|
||||
def test_bank_size_limit(synth_game, tmp_path):
|
||||
root, _ = synth_game
|
||||
ws = _ws(root, tmp_path)
|
||||
s = ws.catalog.by_key["BANK:%08X" % synth.FOOT[1]]
|
||||
with pytest.raises(SizeLimitError):
|
||||
ws.apply([(s, synth.wem(3.0, 440), {})])
|
||||
assert ws.store.entries == {}
|
||||
|
||||
|
||||
def test_bank_file_changed_underneath(synth_game, tmp_path):
|
||||
root, originals = synth_game
|
||||
ws = _ws(root, tmp_path)
|
||||
s = ws.catalog.by_key["BANK:%08X" % synth.GUI[0]]
|
||||
ws.apply([(s, synth.wem(0.2, 1500), {})])
|
||||
path = ws.files(s)[0]
|
||||
# another tool rewrites the package (different bytes elsewhere in it)
|
||||
pk = Package(path)
|
||||
other = pk.patched([(pk.header_size + 10, b"OTHER-TOOL")])
|
||||
open(path, "wb").write(other)
|
||||
with pytest.raises(ConflictError):
|
||||
ws.apply([(s, synth.wem(0.2, 1600), {})])
|
||||
ws.apply([(s, synth.wem(0.2, 1600), {})], force=True)
|
||||
ws.revert([s])
|
||||
assert _sha(path) == _sha(ws.store.full_copy(ws.game.rel(path)))
|
||||
|
||||
|
||||
def test_event_names():
|
||||
sc = banks.Scan()
|
||||
sc.names = {"Play_Synth_Punch", "Stop_Synth_Punch", "Unrelated"}
|
||||
sc.bank_names = {7: "CHR_SynthCombat"}
|
||||
for m in (1, 2, 3):
|
||||
sc.media[m] = {"size": 10, "sha1": {"x"}, "head": b"", "banks": {7}, "kinds": {"m"}}
|
||||
# event -> action -> random container (children 0x50, 0x51) -> sounds -> media 1, 2; media 3 unreferenced
|
||||
from ackit.bnk import fnv1
|
||||
import struct
|
||||
sc.objects = {0x50: 2, 0x51: 2, 0x60: 5, 0x70: 3, fnv1("Play_Synth_Punch"): 4}
|
||||
sc.containers = {0x60: struct.pack("<I", 0x60) + bytes(9) + struct.pack("<3I", 2, 0x50, 0x51) + bytes(5)}
|
||||
sc.actions = {0x70: 0x60}
|
||||
sc.events = {fnv1("Play_Synth_Punch"): [0x70]}
|
||||
sc.sound_media = {0x50: {1}, 0x51: {2}}
|
||||
names = banks.media_names(sc)
|
||||
assert names[1] == ("Play_Synth_Punch_1", "CHR_SynthCombat")
|
||||
assert names[2] == ("Play_Synth_Punch_2", "CHR_SynthCombat")
|
||||
assert names[3] == ("CHR_SynthCombat_1", "CHR_SynthCombat")
|
||||
@@ -4,7 +4,7 @@ import os
|
||||
|
||||
import pytest
|
||||
|
||||
from ackit import classify
|
||||
from ackit import banks, classify
|
||||
from ackit.catalog import Catalog, read_csv, load_characters
|
||||
from ackit.game import Game
|
||||
from ackit.paths import catalog_dir
|
||||
@@ -63,15 +63,24 @@ def test_shipped_csvs_are_consistent():
|
||||
keys = {r.key for r in rows}
|
||||
assert len(keys) == len(rows), f
|
||||
assert all(r.group in classify.GROUPS for r in rows)
|
||||
assert all(len(r.sha1) == 16 for r in rows)
|
||||
assert len({r.name.lower() for r in rows}) == len(rows), f
|
||||
# no checksum only where the copies of a BANK sound differ (1C898731)
|
||||
assert sum(len(r.sha1) != 16 for r in rows) <= (1 if f == "BANK.csv" else 0), f
|
||||
hdr = next(csv.reader(open(os.path.join(catalog_dir(), "ITA.csv"), encoding="utf-8")))
|
||||
assert hdr[:3] == ["archive", "target", "name"]
|
||||
locs = banks.read_locations(os.path.join(catalog_dir(), banks.LOCATIONS))
|
||||
bank = {r.target for r in read_csv(os.path.join(catalog_dir(), "BANK.csv"))}
|
||||
assert {"%08X" % m for m, ls in locs.items() if ls[0].kind == "m"} == bank
|
||||
assert all(l.sound for ls in locs.values() for l in ls if l.kind == "p")
|
||||
|
||||
|
||||
@pytest.mark.game
|
||||
def test_real_game_uses_shipped_catalog(real_game_dir):
|
||||
g = Game(real_game_dir)
|
||||
cat = Catalog(g, ["ITA", "SFX"])
|
||||
assert len(cat.sounds) == 10816 + 651
|
||||
cat = Catalog(g, ["ITA", "SFX", "BANK"])
|
||||
assert len(cat.sounds) == 10816 + 651 + 15404
|
||||
s = cat.resolve("Speech_TAPE_JOKER_1_4")
|
||||
assert s.category == "Interview tapes" and s.size == 44985
|
||||
assert len(cat.heads) == 396 and sum(len(v) for v in cat.heads.values()) == 2101
|
||||
b = cat.resolve("Play_BatmanDeviceImpact_Wood_1")
|
||||
assert b.archive == "BANK" and b.group == "SFX"
|
||||
|
||||
40
tests/test_catalogcheck.py
Normal file
40
tests/test_catalogcheck.py
Normal file
@@ -0,0 +1,40 @@
|
||||
"""`ackit verify --catalog`: the catalog in use against a fresh scan of the game."""
|
||||
import glob
|
||||
import os
|
||||
|
||||
from ackit.catalog import read_csv, write_csv
|
||||
from ackit.game import Game
|
||||
from ackit.paths import user_dir
|
||||
from ackit.verify import catalogcheck
|
||||
|
||||
|
||||
def test_clean_synthetic_catalog_passes(synth_game):
|
||||
root, _ = synth_game
|
||||
rep = catalogcheck.run(Game(root), log=lambda *a: None)
|
||||
assert rep.ok, rep.issues
|
||||
assert rep.checked["BANK"] == 5 and rep.checked["INT"] == 4
|
||||
|
||||
|
||||
def test_corrupted_catalog_is_reported(synth_game):
|
||||
root, _ = synth_game
|
||||
game = Game(root)
|
||||
catalogcheck.run(game, ["INT", "BANK"], log=lambda *a: None) # builds the cached scans
|
||||
cache = glob.glob(os.path.join(user_dir("cache"), "scan_INT_*.csv"))[0]
|
||||
rows = read_csv(cache)
|
||||
rows[1].category = "Weapons" # wrong category
|
||||
rows.append(rows[0]) # duplicate
|
||||
del rows[2] # missing
|
||||
write_csv(cache, rows)
|
||||
bank = glob.glob(os.path.join(user_dir("cache"), "scan_BANK_*.csv"))[0]
|
||||
brows = read_csv(bank)
|
||||
brows[0].name = brows[1].name # duplicate name
|
||||
write_csv(bank, brows[:-1]) # one sound missing
|
||||
rep = catalogcheck.run(game, ["INT", "BANK"], log=lambda *a: None)
|
||||
kinds = {(a, k) for a, k, _, _ in rep.issues}
|
||||
assert {("INT", "category"), ("INT", "duplicate"), ("INT", "missing"), ("BANK", "duplicate"),
|
||||
("BANK", "missing")} <= kinds
|
||||
|
||||
|
||||
def test_real_catalog_quick(real_game_dir):
|
||||
rep = catalogcheck.run(Game(real_game_dir), quick=True, log=lambda *a: None)
|
||||
assert rep.ok, rep.issues[:10]
|
||||
@@ -5,7 +5,9 @@
|
||||
Outputs
|
||||
data/catalog/characters.csv name -> verified speaker, from data/catalog/source/<lang>/*.txt
|
||||
data/catalog/<ARCHIVE>.csv every sound of one archive (schema: data/catalog/README.md)
|
||||
data/catalog/index.json WAD fingerprints the CSVs were built from + counts
|
||||
data/catalog/BANK.csv in-memory sounds (soundbanks in SFX/*.bnk and the .upk packages)
|
||||
data/catalog/BANK_locations.csv.gz every stored copy of them + the prefetch heads of speech lines
|
||||
data/catalog/index.json fingerprints the CSVs were built from + counts
|
||||
Run it only on clean game files (`ackit status` must say nothing is modified).
|
||||
"""
|
||||
import argparse
|
||||
@@ -15,10 +17,11 @@ import hashlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
from ackit import settings # noqa: E402
|
||||
from ackit.catalog import scan_archive, write_csv, hash_names # noqa: E402
|
||||
from ackit import banks, settings # noqa: E402
|
||||
from ackit.catalog import scan_archive, scan_banks, write_csv, hash_names # noqa: E402
|
||||
from ackit.classify import USER_LABELS # noqa: E402
|
||||
from ackit.game import Game, default_game_dir, wad_fingerprint # noqa: E402
|
||||
from ackit.inject import read_target # noqa: E402
|
||||
@@ -107,6 +110,24 @@ def main():
|
||||
"sounds": len(sounds), "groups": dict(g),
|
||||
"with_character": sum(1 for s in sounds if s.character)}
|
||||
print("%s.csv: %d sounds %s" % (arch, len(sounds), dict(g)))
|
||||
|
||||
# in-memory sounds: loose SFX\*.bnk + banks inside the .upk packages (a few minutes)
|
||||
t0 = time.time()
|
||||
sounds, locs, sc = scan_banks(game, progress=lambda k, n: print(" banks: %d/%d files" % (k, n), flush=True)
|
||||
if k % 500 == 0 else None, characters=charmap)
|
||||
write_csv(os.path.join(out_dir, "%s.csv" % banks.ARCHIVE), sounds)
|
||||
banks.write_locations(os.path.join(out_dir, banks.LOCATIONS), locs)
|
||||
g = collections.Counter(s.group for s in sounds)
|
||||
heads = [l for l in locs if l.kind == banks.PREFETCH_HEAD]
|
||||
index["archives"][banks.ARCHIVE] = {
|
||||
"fingerprint": banks.game_fingerprint(game.cooked), "sounds": len(sounds), "groups": dict(g),
|
||||
"copies": sum(1 for l in locs if l.kind == banks.MEMORY), "files": len(sc.files),
|
||||
"named_by_event": sum(1 for s in sounds if not s.name.lower().startswith(s.bank.lower() + "_")),
|
||||
"prefetch_heads": {"lines": len({l.media for l in heads}), "copies": len(heads),
|
||||
"unmapped": len({l.media for l in heads if not l.sound})},
|
||||
"issues": ["%s: %s" % i for i in sc.issues]}
|
||||
print("%s.csv: %d sounds %s, %d copies in %d files, %d prefetch heads (%.0fs)"
|
||||
% (banks.ARCHIVE, len(sounds), dict(g), len(locs) - len(heads), len(sc.files), len(heads), time.time() - t0))
|
||||
# flatten for catalog.load_archive
|
||||
flat = {a: v for a, v in index["archives"].items()}
|
||||
json.dump(dict(flat, _info=index["built_from"]), open(os.path.join(out_dir, "index.json"), "w", newline="\n"), indent=1)
|
||||
|
||||
Reference in New Issue
Block a user