gora/UnheardCity
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Version 3.2.1: whole bank copies of voice lines, unique names, docs figures checked by tools/check_docs.py

Browse Source
This commit is contained in:
Cristian Gora
2026-10-10 19:30:06 +02:00
parent 1586abb8e9
commit cac52e7597
29 changed files with 919 additions and 188 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
CLAUDE.md
View File

@@ -18,6 +18,9 @@ Batman: Arkham City (PC) audio toolkit + editor. Read `NOTES.md` first (status,
stops if a sound differs from the current catalog). Voice lines are identified by the game's Wwise Sound
objects (`ackit/lines.py`, column `line`; one recording per language), never by WAD position or name:
positions differ per language (`docs/decisions.md` 5, `docs/formats/banks.md` *Lines*).
- Figures in the docs are checked, not copied: after changing data, code or docs run `python tools/check_docs.py`
(`--scan`: also the bank survey and the catalog check's facts, ~10 min; `--wems`: every WEM rebuilt, ~10 min);
add a claim there for every new figure.
- Real-game write round trips: `python tools/e2e_features.py`, `python tools/e2e_editor.py` (they revert in a
`finally` block; still check `ackit status` after any interrupted run).
- Shell: git at `C:\Program Files\Git\cmd` (prepend to PATH). Write Python with the Write/Edit tools; avoid

25
NOTES.md
View File

@@ -20,7 +20,7 @@
`ackit/bnk.py` (bank parsing: DIDX/DATA/HIRC sources). Docs: formats/upk.md, banks.md. Findings: pure-Python
LZO ~47 MB/s (== decompress.exe on every block); 15,800 media IDs in 96,011 copies, 396 are prefetch heads
of WAD sounds (2,101 copies) -> 15,404 in-memory sounds; copies identical except 1C898731.
2. [x] Catalog archive `BANK` (15,404 sounds, 6,416 named by events, now 6,456) + `BANK_locations.csv.gz` (93,910 copies,
2. [x] Catalog archive `BANK` (15,404 sounds, 6,416 named by `Play_` events; now 15,355, 13,501 named by events) + `BANK_locations.csv.gz` (93,910 copies,
2,101 prefetch heads of 396 speech lines); `ackit verify --catalog` (full: catalog OK; fixed 2 duplicate
names per language: variation shown as NAME#i); categories for bank prefixes.
3. [x] Workspace writes BANK sounds into every copy and keeps speech-line prefetch heads in sync (bank files
@@ -87,6 +87,26 @@
from the rules: 54,079 / 54,080 identical (INT FE9963E8:74: avg-bytes/s field one lower, informational).
Removed unused code (helpers/constants nothing called, the key-posting debugger helper, a double refilter
after changes); old dist zips, logs/ and caches deleted.
- 2026-10-10 final check (user: "is everything fact checked ... functioning ... documentation correct?"):
every number in the docs recomputed from the catalog and a fresh game scan; stale ones fixed (positions differ
for 81.2-81.6 %, not 75 %; 8,092 neutral BANK sounds; 24 unlinked subtitle names; 637 position-named lines;
14,802 package media; subtitle coverage 9 % combat barks, 23 % cinematics; speaker coverage by line). Bugs
found and fixed (findings 23): 49 WAD recordings also stored whole in banks were separate BANK sounds (now
bank copies of the line, written with it); 8 BANK/SFX name clashes; name lookup preferred no name over an
alias (CLI "ambiguous" for e.g. `CHR_RadioScanner#1` in FRA). `ackit verify --lang` now applies (catalog: that
language only; in-game: must be the game's language). Subtitle error message says Windows-1252, not Latin-1.
Catalog rebuilt from the original files; `verify --catalog` OK (2,185 / 2,185 bank copies of WAD recordings
identical), e2e_features --lang all 89/89, e2e_editor, --ingame/--banks/--texts PASS, 132 tests; game original
after. Version 3.2.1: dist/UnheardCity-3.2.1-windows-amd64.zip smoke + GUI start PASS.
- 2026-10-10 re-check (user: "fact check everything again"): the catalog rebuilt from the game is identical to the
shipped one (every CSV and location index), and `verify --catalog` gives the same facts. Figures in the docs are
now checked by `tools/check_docs.py` (catalog, index.json, menu files; `--scan`: bank survey + catalog-check
facts; `--wems`: every WEM rebuilt); the build records the scan-only figures in index.json (names by event,
dialogue objects, WAD lines named after a recording). It found what hand-copied figures had wrong: 61,510
"recordings" counted the 25 second places (61,485 recordings; the stat was fixed too), 81.2-81.6 % (not 81-82),
the loose/embedded bank-name overlap, REPORT's test count, wem.md's SFX claim (626 / 651 rebuild, not "all but 4
looping tracks": 14 music seek tables, 5 music sample counts, 6 ambience `smpl` loops) and its extra-chunk count
(722 WEMs, not 37). New test: cross-references between the shipped files.
- NEXT (needs a person): play a mission with a replaced footstep/foley sound and an edited subtitle (REPORT.md).
- OPEN: channel order for >2 channels unverified; speech and in-memory sounds *heard* in-game (need gameplay;
posted keys do not drive the menu).
@@ -180,4 +200,5 @@
- 2026-10-10: source lists of another language (`source/<lang>/`) are resolved through THAT language's WAD
positions; ~1,270 speech sounds per language have no game line name (efforts 519, radio/TV 139, cinematics
134) and keep position names (`Sound.is_position_name`): their labels stay per language and mod packages from
another language are not matched to them.
another language are not matched to them. (Superseded the same day by the line identities of `ackit/lines.py`;
`is_position_name` was removed.)

5
README.md
View File

@@ -56,12 +56,13 @@ app/ Unheard City (PySide6)
tests/ pytest suite (synthetic game data; real-game tests run when the game is installed)
data/catalog/ sound catalogs (CSV), character lists (source/), schema in data/catalog/README.md
docs/ formats, reverse engineering, decisions, categories, user guide, building
tools/ catalog builder, real-game e2e checks (e2e_editor.py, e2e_features.py), packaging
tools/ catalog builder, docs figure check, real-game e2e checks (e2e_editor.py, e2e_features.py), packaging
(build_app.py, installer/), artwork, license and screenshot scripts
NOTES.md project status and working notes; CLAUDE.md: guidance for AI sessions
```
Tests: `pytest` (the real-game tests are skipped when the game is not installed).
Tests: `pytest` (the real-game tests are skipped when the game is not installed). Figures in the docs:
`python tools/check_docs.py [--scan] [--wems]`.
Real-game write round trips (no game launch): `python tools/e2e_features.py [--lang all]`,
`python tools/e2e_editor.py`.
In-game tests (Windows, Steam running, WinDbg installed): `ackit verify --ingame`, `--ingame --banks`,

51
REPORT.md
View File

@@ -5,17 +5,19 @@ editor Unheard City (mission 2).
## What works (verified)
- **Every sound of the game is cataloged and editable**: the speech WADs (5 languages x 10,816), the SFX WAD (651)
and, new, the **15,404 in-memory sounds** stored inside soundbanks (archive `BANK`: footsteps, foley, combat,
weapons, UI, efforts; 8,098 shared by all languages + ~1,460 localized speech lines per voice language) in 65
loose `.bnk` files and 975 `.upk` packages. ackit reads and patches the
and, new, the **15,355 in-memory sounds** stored inside soundbanks (archive `BANK`: footsteps, foley, combat,
weapons, UI, efforts; 8,092 shared by all languages, ~1,450 localized speech lines per voice language, 6
shared by several) in 65 loose `.bnk` files and 975 `.upk` packages. ackit reads and patches the
LZO-compressed UE3 packages itself (`ackit/upk.py`, `lzo.py`; no external tools). A replacement goes into every
copy (93,910 copies); 396 speech lines with prefetch heads in banks get their heads rewritten too.
Names: 6,456 BANK sounds are named after their Wwise event (`Play_...`), the rest after their bank.
copy (93,826 copies). 445 WAD voice recordings (89 lines) also have bank copies (396 prefetch heads, 49
whole copies; 2,185 copies), which are rewritten with them.
Names: 13,501 BANK sounds are named after the Wwise event that plays them (`Play_...`, `Speech_...`), the
other 1,854 after their bank.
- **Catalog check**: `ackit verify --catalog` compares the catalog with a fresh scan of the game (missing, extra,
duplicate, size/checksum, category, character, language, bank copies, subtitle copies/texts) and re-derives
every relation from the game files (lines, names, subtitle links, prefetch heads, copies): **catalog OK** on
the full game.
- **Subtitles**: found in `RDialogueEvent` objects inside the packages (9,016 speech lines x 11 languages) and
- **Subtitles**: found in `RDialogueEvent` objects inside the packages (9,032 speech lines x 11 languages) and
in 41 scripted `RSubtitle` objects (radio, encryption cards, Riddler; linked through `RSubtitleLookup`),
parsed, cataloged (`SUB_<LANG>.csv`) and linked to their voice recordings through the soundbanks: 103,304 of
103,658 strings in 11 languages (INT 9,373 of 9,404), each on the same line in every language; an event that
@@ -25,8 +27,8 @@ editor Unheard City (mission 2).
WAD recording to its media by its own language's BKPK table (10,816 / 10,816 in each language). Result:
12,434 lines (WAD + in-memory) with exactly one recording in each of the 5 voice languages, no gap and no
conflict; recordings stored twice (5 per language) are written together. Independent checks: every WAD
recording is a source in a bank of its own language (54,055 / 54,055), every prefetch head is the start of
its recording (2,101 / 2,101). Each line has one name in every language and carries its identity (`line`):
recording is a source in a bank of its own language (54,055 / 54,055), every prefetch head or whole bank copy
is the start of its recording (2,185 / 2,185). Each line has one name in every language and carries its identity (`line`):
speaker labels (3,366 in every language), subtitles and mod packages (another language: the same line)
follow it. Speaker labels vs the game's dialogue speaker field: 2,766 agree, 19 differ (categories.md).
- **Menu text**: the localization files (`BmGame\Localization\<LANG>\*.int`, ~7,000 strings per language),
@@ -40,13 +42,14 @@ editor Unheard City (mission 2).
- **Safety**: every changed file (WAD, bank, package, localization file) is rebuilt from a verified original copy
in the store + the edits applied now; reverting the last edit restores it byte-for-byte; files changed by
another tool or Steam are detected (whole-file SHA-1) and not overwritten without confirmation.
- **Tests**: 130 automated tests (unit, synthetic game with packages/banks/dialogues/localization, a
two-language game for the lines, real-game read-only, offscreen GUI incl. list order, selection and loading
overlays) pass. `ackit verify` PASS (incl.
- **Tests**: 133 automated tests (unit, synthetic game with packages/banks/dialogues/localization, a
two-language game for the lines (incl. a recording also stored whole in a bank), the shipped catalog's cross-references, real-game read-only (every
name of every language's catalog resolves to its own sound), offscreen GUI incl. list order, selection and
loading overlays) pass. `ackit verify` PASS (incl.
300/300 in-memory WEMs rebuilt byte-identically).
- **Real game** (2026-10-10), all reverting and checking byte-identity themselves:
`tools/e2e_features.py --lang all` 78/78 PASS in all 11 languages (every copy of an in-memory sound, each
language's localized line, prefetch heads, a multi-copy subtitle, a scripted subtitle, a menu text, sound +
`tools/e2e_features.py --lang all` 89/89 PASS in all 11 languages (every copy of an in-memory sound, each
language's localized line, prefetch heads, a line also stored whole in banks, a multi-copy subtitle, a scripted subtitle, a menu text, sound +
subtitle in one package, a mod moved to another voice language and matched to the same line at another
position); `ackit verify --ingame` PASS (music heard, ncc 1.0);
`--banks` PASS (77 in-memory sounds of the front-end packages replaced in 54 files: the game loads them, 68
@@ -54,9 +57,9 @@ editor Unheard City (mission 2).
found in the game's memory, no crash); `tools/e2e_editor.py` PASS. Game files original at the end
(`ackit status`). The catalog builder refuses to run on a modified game, so the catalog always describes the
original files.
- **Package**: `dist/UnheardCity-3.2.0-windows-amd64.zip` (86 MB, catalogs included). With `PATH` reduced to
- **Package**: `dist/UnheardCity-3.2.1-windows-amd64.zip` (86 MB, catalogs included). With `PATH` reduced to
the Windows folders the packaged exe passes its smoke test (in the game's language - Italian on the test PC:
21,025 sounds loaded, encode + inject, 9,377 subtitles and 7,020 menu texts loaded, an in-memory sound decoded from a package) and the GUI starts and
21,015 sounds loaded, encode + inject, 9,377 subtitles and 7,020 menu texts loaded, an in-memory sound decoded from a package) and the GUI starts and
quits cleanly. The installer is unchanged since mission 2 (install/upgrade/uninstall verified then).
## Unresolved / limitations
@@ -68,17 +71,21 @@ editor Unheard City (mission 2).
text's length ("room"); cp1252 subtitles accept cp1252 characters only. Lifting these limits means moving
objects inside the packages (new export offsets/sizes, added chunks): designed but not done, because it could
not be verified automatically in the game (decisions 4c).
- **Unlinked subtitles**: 29 speech names (~32 strings per language: Riddler trophy texts, Arkham City Stories,
radio reports and tapes) have no event in any bank; they are listed and editable but not linked.
- **Subtitle coverage is uneven in the game itself**: no subtitles exist for efforts/grunts, few for combat barks
(11 %) and cutscene lines (10 %; most cutscene lines are not played through speech events and have no dialogue
object). No other subtitle storage was found in the packages (all classes with subtitle data checked).
- 659 voice lines are played by no event whose name the game stores; they are named after their English
- **Unlinked subtitles**: 24 speech names (24-36 strings per language, INT 31: Riddler trophy texts, Arkham City
Stories, radio reports and tapes) have no event in any bank; they are listed and editable but not linked.
- **Subtitle coverage is uneven in the game itself** (INT voice lines with a subtitle): none for efforts/grunts,
few for combat barks (9 %), cinematics and boss fights (23 %; most cutscene lines are not played through speech
events and have no dialogue object), 89-97 % for story, predator, character and open-world lines. No other
subtitle storage was found in the packages (all classes with subtitle data checked).
- 637 voice lines are played by no event whose name the game stores; they are named after their English
recording's position (the same name in every language). Mod packages from versions before 3.1 carry no line
identity: their voice lines are not moved to another language (listed instead of guessed).
- Speaker labels: 2 names of the curated lists match two recordings each (wemscan name collision), both are
labelled; 19 labels differ from the game's dialogue speaker field (to be checked by listening).
- **Unnamed sounds**: ~9,000 BANK sounds have no event name and are named `<Bank>_<n>`.
- **Unnamed sounds**: 1,854 BANK sounds are reached by no event whose name the game stores and are named
`<Bank>_<n>`.
- 25 SFX originals (music seek tables and sample counts, looping ambience) are not reproduced byte-for-byte by
the WEM rules (`docs/formats/wem.md`); writing is not affected (replaced music plays in game).
- Two copies of media `1C898731` differ; its checksum is left empty. Channel order for 4-6 channel sounds is
unverified. M4A/AAC/WMA input needs an ffmpeg on the system.
- Another game build is scanned at first start (WADs: seconds; banks, lines and subtitles: a few minutes,

2
ackit/__init__.py
View File

@@ -8,4 +8,4 @@ Library layers (low -> high):
inject (WAD writing), originals (store), workspace (apply/revert), modpack (share)
tools cli (`ackit` command), verify/ (offline self-check, in-game test on Windows)
"""
__version__ = "3.2.0"
__version__ = "3.2.1"

11
ackit/banks.py
View File

@@ -156,12 +156,14 @@ class Scan:
self.events = {} # id -> [action ids]
self.actions = {} # id -> target id
self.sound_media = defaultdict(set) # sound object id -> media ids
self.prefetch = set() # media ids whose in-memory copy is a prefetch head
self.prefetch = set() # media ids whose bank copies belong to a WAD recording (prefetch
# heads; whole recordings: catalog._whole_recordings)
self.lang_media = defaultdict(set) # language -> media ids named by Sound objects of that language's banks
# BKPK tables: WAD entry id -> {language: {media id: (offset in entry, size)}}. A table belongs to the
# language its localized AkBank export declares ('' = outside such an export). The same media ID can be
# used by several languages (at a different position in each language's WAD), so tables never merge.
self.packs = defaultdict(_by_language)
self.subtitle_objects = {} # counts of dialogue / RSubtitle objects (catalog.scan_text)
self.issues = [] # (kind, detail)
@@ -368,11 +370,12 @@ def media_language(sc, m):
return next(iter(langs)) if len(langs) == 1 else ""
def media_names(sc):
def media_names(sc, reserved=()):
"""{media id: (display name, primary bank name)}: named after the most specific 'Play' event that
reaches the media, otherwise after its bank (<Bank>_<n>). Localized speech has one media per language
for the same line: they share the name, variations are numbered within a language. Names are unique
among the language-neutral media plus any one language (what a catalog shows at a time)."""
among the language-neutral media plus any one language (what a catalog shows at a time) and avoid the
lower-case names in `reserved` (other archives' sounds)."""
ev = event_media(sc)
best = {}
for name, ms in ev.items():
@@ -386,7 +389,7 @@ def media_names(sc):
if m not in best or rank < best[m][0]:
best[m] = (rank, name, i, len(g))
out = {}
taken = set() # (language, lower-case name)
taken = {("", n) for n in reserved} # (language, lower-case name)
def free(lang, nm):
k = nm.lower()

50
ackit/catalog.py
View File

@@ -234,10 +234,11 @@ def scan_archive(archive, wad_path, characters=None, progress=None):
_CAMEL = re.compile(r"(?<=[a-z])(?=[A-Z])")
def bank_sounds(sc, characters=None):
"""[Sound] of the in-memory sounds found by `banks.scan` (prefetch heads belong to WAD sounds)."""
def bank_sounds(sc, characters=None, reserved=()):
"""[Sound] of the in-memory sounds found by `banks.scan` (prefetch heads belong to WAD sounds). `reserved`:
lower-case names taken by other archives (sfx_names)."""
characters = characters if characters is not None else load_characters()
names = banks.media_names(sc)
names = banks.media_names(sc, reserved)
every = set().union(*(e.get("langs", set()) for e in sc.media.values())) - {""}
out = []
for m, e in sc.media.items():
@@ -275,13 +276,45 @@ def derive_banks(game, sc, characters=None):
"""(bank sounds, locations, Lines) from a Scan: the lines, the prefetch heads' lines, the in-memory sounds
with their line identities and names."""
lines = build_lines(game, sc)
mark_heads(sc, whole_recordings(sc, lines))
locs = [l._replace(sound=lines.head_key(l)) if l.kind == banks.PREFETCH_HEAD else l for l in sc.locations]
snd = bank_sounds(sc, characters)
snd = bank_sounds(sc, characters, sfx_names(game))
lines.names = line_names(game, sc, lines, {s.key: s.name for s in snd})
apply_lines(snd, lines, lines.names)
return snd, locs, lines
def whole_recordings(sc, lines):
"""Media stored in memory that are WAD recordings of their bank's language (the same media ID, identical
bytes: 49 in the game, e.g. game-over taunts). Like prefetch heads, these copies belong to the WAD line
and are written with it; they are not sounds of their own."""
keys = collections.defaultdict(list)
for l in sc.locations:
if l.kind == banks.MEMORY:
k = lines.head_key(l)
keys[l.media].append(bool(k) and not k.startswith(banks.ARCHIVE + ":"))
return {m for m, wad in keys.items() if all(wad)}
def mark_heads(sc, media):
"""Treat the bank copies of `media` as heads of WAD recordings (see whole_recordings)."""
if not media:
return
for m in media:
if m in sc.media:
sc.media[m]["kinds"] = {banks.PREFETCH_HEAD}
sc.prefetch.add(m)
sc.locations = [l._replace(kind=banks.PREFETCH_HEAD) if l.media in media else l for l in sc.locations]
def sfx_names(game):
"""Lower-case names and aliases of the SFX WAD's sounds: in-memory sounds of the same bank must not take
them (`<Bank>_<n>` is the position name of the bank's streamed sounds too)."""
if "SFX" not in game.archives:
return set()
return {n.lower() for s in scan_archive("SFX", game.wad("SFX"), characters={}) for n in s.names()}
def line_names(game, sc, lines, bank_names=None):
"""{line id: name} (lines.line_names). Lines no event names keep their English recording's name: the
position-based name in the INT WAD (`NAME_i`, or `NAME#i` where that is taken) or the bank-numbered name."""
@@ -378,6 +411,10 @@ def scan_text(game, progress=None, characters=None):
sc = banks.scan(game.cooked, progress, visit=col)
names = {n for n in sc.names if n.lower().startswith("speech_")}
names |= {n for n, _ in banks.media_names(sc).values() if n.lower().startswith("speech_")}
lookup = [n for n in col.dialogues if n.startswith("#lookup:")]
sc.subtitle_objects = {"dialogue_objects": sum(len(v) for n, v in col.dialogues.items() if n not in lookup),
"dialogue_lines": len(col.dialogues) - len(lookup),
"scripted_objects": sum(len(col.dialogues[n]) for n in lookup)}
col.resolve(names)
# the subtitle objects name their speech events too: an event ID is the FNV-1 hash of its name, so these
# names find events no package name table lists (e.g. Vicki Vale's radio reports)
@@ -519,6 +556,11 @@ class Catalog:
key = os.path.splitext(os.path.basename(name))[0].lower() if re.search(r"\.\w{2,4}$", name) else name.lower()
hits = self._by_name.get(key, [])
uniq = list({id(s): s for s in hits}.values())
if len(uniq) > 1:
# a name wins over another sound's alias: a line named after its INT position (NAME#i) can equal
# the position alias of another recording in another language's WAD
own = [s for s in uniq if s.name.lower() == key]
uniq = own if len(own) == 1 else uniq
if len(uniq) == 1:
return uniq[0]
if len(uniq) > 1:

8
ackit/cli.py
View File

@@ -294,7 +294,8 @@ def cmd_game(a):
def cmd_verify(a):
if a.catalog:
from .verify import catalogcheck
rep = catalogcheck.run(_game(a), [x.upper() for x in a.archive] or None, quick=a.quick)
archives = [x.upper() for x in a.archive] or ([a.lang] if a.lang else None)
rep = catalogcheck.run(_game(a), archives, quick=a.quick)
for (arch, kind), n in sorted(rep.counts().items()):
print("%-14s %-10s %d" % (arch, kind, n))
for arch, kind, key, detail in rep.issues[:a.show]:
@@ -310,6 +311,9 @@ def cmd_verify(a):
ok = selfcheck.run(ws, log=print)
sys.exit(0 if ok else 1)
from .verify import ingame
if a.lang and a.lang != ws.game.language():
raise GameError("the game plays its own language (%s); set it in Steam (game properties > Language) to "
"test %s in the game" % (ws.game.language(), a.lang))
if a.banks:
rep = ingame.run_banks(ws, a.sound or None)
elif a.texts:
@@ -385,7 +389,7 @@ def build_parser():
s = add("verify", cmd_verify, "self-check (offline), catalog cross-check (--catalog), or test in the real "
"game with --ingame (Windows)")
s.add_argument("--catalog", action="store_true", help="cross-check the catalog against the game files")
s.add_argument("--archive", action="append", default=[], help="with --catalog: only this archive (repeatable)")
s.add_argument("--archive", action="append", default=[], help="with --catalog: only this archive (repeatable; --lang X: only X)")
s.add_argument("--quick", action="store_true", help="with --catalog: loose banks + a sample of packages")
s.add_argument("--show", type=int, default=30, help="with --catalog: issues to list (default 30)")
s.add_argument("--ingame", action="store_true"); s.add_argument("--target", default="SFX:EDF036D6/393BC53A")

12
ackit/lines.py
View File

@@ -27,6 +27,7 @@ class Lines:
objects: dict = field(default_factory=dict) # line id -> sorted Sound object ids
obj_keys: dict = field(default_factory=dict) # Sound object id -> {lang: sound key}
names: dict = field(default_factory=dict) # line id -> name (catalog.line_names)
unnamed: set = field(default_factory=set) # line ids no named event plays (named after a recording)
copy_of: dict = field(default_factory=dict) # WAD key -> key of the same media stored in another entry
issues: list = field(default_factory=list) # (kind, detail): broken one-to-one relations
stats: dict = field(default_factory=dict)
@@ -161,10 +162,10 @@ def build(game, sc, walk):
for k, primary in copy_of.items(): # a second place of the same recording: the same line
if primary in lines.line_of:
lines.line_of[k] = lines.line_of[primary]
all_keys = set().union(*lines._keys.values()) if lines._keys else set()
all_keys |= set(copy_of)
recordings = set().union(*lines._keys.values()) if lines._keys else set()
all_keys = recordings | set(copy_of) # + the second places of recordings stored twice
full = sum(1 for per in lines.members.values() if len(per) == len(langs))
lines.stats = {"languages": langs, "wad_wems": wems, "recordings": len(all_keys),
lines.stats = {"languages": langs, "wad_wems": wems, "recordings": len(recordings),
"in_a_line": len(lines.line_of), "lines": len(lines.members), "lines_in_every_language": full,
"copies": sorted(copy_of),
"lines_missing_a_language": {lid: sorted(set(langs) - set(per))
@@ -215,8 +216,9 @@ def line_names(sc, lines, fallback):
out[lid] = name if i == 0 else "%s#%d" % (name, i + 1)
else:
out[lid] = name if len(lids) == 1 else "%s_%d" % (name, i + 1)
for lid, per in lines.members.items():
if lid not in out:
lines.unnamed = set(lines.members) - set(out)
for lid in lines.unnamed:
per = lines.members[lid]
k = per.get("INT") or per[next(L for L in lines.langs if L in per)]
out[lid] = fallback(k)
return out

7
ackit/subtitles.py
View File

@@ -9,7 +9,7 @@ a byte size; struct elements end with a u16 0).
A dialogue object can be stored in several packages; a subtitle edit is written into every copy, in place:
the new text replaces the bytes of the old FString and is padded with NUL characters (the game stops at the
first NUL), so no size field, export or package layout changes. The text must therefore fit the original
string's bytes (Latin-1/cp1252 strings: one byte per character; UTF-16 strings: two).
string's bytes (cp1252 strings: one byte per character; UTF-16 strings: two).
"""
import csv
import gzip
@@ -55,7 +55,8 @@ def encode_fstring(text, size, enc):
try:
raw = text.encode("cp1252")
except UnicodeEncodeError:
raise SubtitleError("this subtitle is stored as Latin-1 text; '%s' cannot be written"
raise SubtitleError("this subtitle is stored as Western European (Windows-1252) text; '%s' cannot be "
"written"
% next(c for c in text if not _cp1252(c))) from None
n = size - 4
if len(raw) + 1 > n:
@@ -175,6 +176,7 @@ class Collector:
def __init__(self):
self.dialogues = defaultdict(list) # speech name -> [(file, Dialogue)]
self.scripted = set() # speech names of RSubtitle objects (resolve)
self.issues = []
def __call__(self, rel, pkg, start, data):
@@ -201,6 +203,7 @@ class Collector:
self.issues.append(("subtitles", "no speech line for subtitle %s" % copies[0][1].export))
name = copies[0][1].export
self.dialogues[name].extend((rel, d._replace(speech=name)) for rel, d in copies)
self.scripted.add(name)
# --- catalog files: SUB_<LANG>.csv (texts) + SUB_locations.csv.gz (every copy) ------------------

27
ackit/verify/catalogcheck.py
View File

@@ -18,7 +18,8 @@ import random
from .. import banks, classify, subtitles
from ..catalog import (Catalog, apply_lines, available_archives, bank_names, bank_sounds, build_lines, character_of,
is_shipped, line_names, load_archive, load_banks, load_characters, read_csv, scan_archive)
is_shipped, line_names, load_archive, load_banks, load_characters, mark_heads, read_csv,
scan_archive, sfx_names, whole_recordings)
from ..inject import read_target
from ..paths import catalog_dir
from ..wad import Wad
@@ -133,14 +134,17 @@ def check_banks(rep, game, rows, locations, characters, quick=False, sample=40,
col.resolve({n for n in sc.names | known if n.lower().startswith("speech_")})
check_subtitles(rep, game, col, set(files), quick)
lines = None
if not quick:
sc.names.update(col.dialogues) # as the build does: subtitle objects name speech events too
lines = build_lines(game, sc)
index = collections.defaultdict(set)
for m, ls in locations.items():
for l in ls:
if l.file in set(files):
index[(l.media, l.kind)].add((l.file, l.offset, l.size, l.sound if l.kind == banks.PREFETCH_HEAD else ""))
if not quick:
sc.names.update(col.dialogues) # as the build does: subtitle objects name speech events too
lines = build_lines(game, sc)
mark_heads(sc, whole_recordings(sc, lines)) # WAD recordings also stored whole in banks
else: # a sample cannot derive them: the index says which they are
mark_heads(sc, {m for m, k in index if k == banks.PREFETCH_HEAD} - sc.prefetch)
found = collections.defaultdict(set)
for l in sc.locations:
head = l.kind == banks.PREFETCH_HEAD
@@ -152,7 +156,7 @@ def check_banks(rep, game, rows, locations, characters, quick=False, sample=40,
rep.add(banks.ARCHIVE, "copies", "%08X" % k[0], "%s: index %d, game %d"
% ("head" if k[1] == banks.PREFETCH_HEAD else "copies", len(index.get(k, ())), len(found.get(k, ()))))
cat = {s.target: s for s in rows}
fresh_list = bank_sounds(sc, characters)
fresh_list = bank_sounds(sc, characters, sfx_names(game))
if lines is not None:
lines.names = line_names(game, sc, lines, {s.key: s.name for s in fresh_list})
apply_lines(fresh_list, lines, lines.names)
@@ -253,11 +257,11 @@ def check_lines(rep, game, sc, lines, locations):
rep.add("LINES", "links", k, "media %08X is not a source in any %s bank" % (m, L))
rep.facts.append("WAD recordings named as a source by a Sound object of their own language's bank: %d of %d"
% (named, total))
rep.facts.append("voice languages %s: %d recordings (WAD %s + localized in-memory), %d lines, %d of them "
"with exactly one recording in every language"
% ("/".join(lines.langs), st["recordings"],
"/".join(str(st["wad_wems"].get(L, 0)) for L in lines.langs), st["lines"],
st["lines_in_every_language"]))
rep.facts.append("voice languages %s: %d recordings (WAD WEMs %s, %d of them second places of a recording "
"stored twice, + localized in-memory), %d lines, %d of them with exactly one recording in every "
"language" % ("/".join(lines.langs), st["recordings"],
"/".join(str(st["wad_wems"].get(L, 0)) for L in lines.langs), len(st["copies"]),
st["lines"], st["lines_in_every_language"]))
# catalog rows vs the game's lines
rows = {}
for L in lines.langs + [banks.ARCHIVE]:
@@ -305,7 +309,8 @@ def check_lines(rep, game, sc, lines, locations):
rep.add("LINES", "copies", k, "differs from %s (same media ID)" % primary)
rep.facts.append("recordings stored twice in a WAD (same media ID, written together): %d, byte-identical: %d"
% (len(lines.copy_of), same_copies))
rep.facts.append("prefetch heads: %d of %d stored copies are byte-identical to the start of their WAD recording"
rep.facts.append("prefetch heads and whole bank copies: %d of %d are byte-identical to the start of their WAD "
"recording"
% (same, len(heads)))
# independent evidence: the recordings of a line have similar lengths in every language
ratios = []

57
data/catalog/BANK.csv
View File

@@ -4084,9 +4084,9 @@ BANK,0191DF08,Play_RadioScanner_9,CHR_RadioScanner/0191DF08,Speech,Radio & TV,CH
BANK,24202946,Play_RadioScanner_Close_Gadget,CHR_RadioScanner/24202946,Speech,Radio & TV,CHR_RadioScanner,,16333BFB,,,0.384,48000,2,5860,2ac62f5c2dd982da,
BANK,0A66562F,Play_RadioScanner_Intercept_2,CHR_RadioScanner/0A66562F,Speech,Radio & TV,CHR_RadioScanner,,1590991C,,,0.200,16000,1,386,f91e17fbba8174b7,
BANK,1C8A9035,Play_RadioScanner_Intercept_4,CHR_RadioScanner/1C8A9035,Speech,Radio & TV,CHR_RadioScanner,,02C289C6,,,0.286,48000,2,5199,fa723f839dc998ea,
BANK,08514617,CHR_RasBossFight_1,CHR_RasBossFight/08514617,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,0.312,48000,1,2558,3a15689ad10607a9,
BANK,2721B526,CHR_RasBossFight_2,CHR_RasBossFight/2721B526,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,0.774,48000,1,5983,1baaa4b70ac728c3,
BANK,35F131FD,CHR_RasBossFight_3,CHR_RasBossFight/35F131FD,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,1.207,48000,1,8500,ea4b213c049d07d4,
BANK,08514617,CHR_RasBossFight_08514617,CHR_RasBossFight/08514617,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,0.312,48000,1,2558,3a15689ad10607a9,
BANK,2721B526,CHR_RasBossFight_2721B526,CHR_RasBossFight/2721B526,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,0.774,48000,1,5983,1baaa4b70ac728c3,
BANK,35F131FD,CHR_RasBossFight_35F131FD,CHR_RasBossFight/35F131FD,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,1.207,48000,1,8500,ea4b213c049d07d4,
BANK,2FA2124B,Play_DemonDeath,CHR_RasBossFight/2FA2124B,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,3.154,48000,2,36691,b919594f844829f4,
BANK,2D6FFCEF,Play_GiantRASEmergeLFE,CHR_RasBossFight/2D6FFCEF,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,1.770,48000,1,10530,7be35ff8a95b8a55,
BANK,00CCAAA6,Play_GiantSword_1,CHR_RasBossFight/00CCAAA6,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,2.760,48000,1,21059,777cec0b514a660c,
@@ -7258,7 +7258,7 @@ BANK,0F26C10C,Play_DT_VortexLoop01,SEQ_DemonTrials/0F26C10C,SFX,Scripted sequenc
BANK,1CBC4CD9,Play_TowerDescendLoop,SEQ_DemonTrials/1CBC4CD9,SFX,Scripted sequences,SEQ_DemonTrials,,,,,6.720,48000,1,61425,ba6c467e650f65a9,
BANK,17E4CD34,Play_VortexOn,SEQ_DemonTrials/17E4CD34,SFX,Scripted sequences,SEQ_DemonTrials,,,,,3.059,48000,3,51620,d32b50a9374a0329,
BANK,0BFFAE8F,Play_WallBurnLowSweep_1,SEQ_DemonTrials/0BFFAE8F,SFX,Scripted sequences,SEQ_DemonTrials,,,,,8.491,48000,2,82419,36d502760f2aac92,
BANK,14B78DC0,SEQ_DemonTrials_1,SEQ_DemonTrials/14B78DC0,SFX,Scripted sequences,SEQ_DemonTrials,,,,,1.619,48000,2,31218,435f399b85421da6,
BANK,14B78DC0,SEQ_DemonTrials_14B78DC0,SEQ_DemonTrials/14B78DC0,SFX,Scripted sequences,SEQ_DemonTrials,,,,,1.619,48000,2,31218,435f399b85421da6,
BANK,0EC21184,Play_BridgeExplosion_1,SEQ_GCPDExplosion/0EC21184,SFX,Scripted sequences,SEQ_GCPDExplosion,,,,,3.610,32000,2,39751,9b89913f89bc6e90,
BANK,03E5C32B,Play_BridgeExplosion_2,SEQ_GCPDExplosion/03E5C32B,SFX,Scripted sequences,SEQ_GCPDExplosion,,,,,6.344,32000,4,203846,0e8cdd46066c44c8,
BANK,2D667F1A,Play_GCPD_DistanExp01,SEQ_GCPDExplosion/2D667F1A,SFX,Scripted sequences,SEQ_GCPDExplosion,,,,Cop,7.801,11025,5,96518,94fa5c717169fd31,
@@ -7422,11 +7422,6 @@ BANK,1A3B7533,Speech_Bully_C_conv_05,Speech_Bully_C_conv/1A3B7533,Speech,Combat
BANK,076C0C7A,Speech_Bully_C_conv_05,Speech_Bully_C_conv/076C0C7A,Speech,Combat barks,Speech_Bully_C_conv,ITA,225159AB,,Thug,6.635,28000,1,50298,ac39d899bdb167cf,
BANK,0C5DBD6B,Speech_Bully_C_conv_05,Speech_Bully_C_conv/0C5DBD6B,Speech,Combat barks,Speech_Bully_C_conv,DEU,225159AB,,Thug,6.488,28000,1,47977,72b8e1e623c4b089,
BANK,16AD1A51,Speech_Bully_C_conv_05,Speech_Bully_C_conv/16AD1A51,Speech,Combat barks,Speech_Bully_C_conv,ESN,225159AB,,Thug,5.952,28000,1,43883,9711ec368d11e388,
BANK,3C7B953C,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3_01,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3/3C7B953C,Speech,Story missions,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3,INT,,Batman,,1.149,44100,1,11551,e25bb4cde803739d,
BANK,3653354B,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3_01,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3/3653354B,Speech,Story missions,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3,FRA,,Batman,,1.308,28000,1,10780,deb4215bfc476d40,
BANK,238B1AD2,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3_01,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3/238B1AD2,Speech,Story missions,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3,ITA,,Batman,,1.782,28000,1,13054,c8e2c2d687800345,
BANK,09ECFA75,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3_01,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3/09ECFA75,Speech,Story missions,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3,DEU,,Batman,,1.620,28000,1,10820,71bcff8ee2d30cab,
BANK,2540B067,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3_01,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3/2540B067,Speech,Story missions,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3,ESN,,Batman,,1.330,28000,1,11204,36e05933f5c12160,
BANK,235C3AB2,Speech_CH_ZERO_PENG_hands_PENG_fightcomments_01,Speech_CH_ZERO_PENG_hands/235C3AB2,Speech,Story missions,Speech_CH_ZERO_PENG_hands,INT,13EEB34A,,Penguin,4.681,44100,1,36231,0c9e76840c288b09,
BANK,32D15A18,Speech_CH_ZERO_PENG_hands_PENG_fightcomments_01,Speech_CH_ZERO_PENG_hands/32D15A18,Speech,Story missions,Speech_CH_ZERO_PENG_hands,FRA,13EEB34A,,Penguin,4.768,28000,1,33903,b5e01b7e7d017a30,
BANK,2F46312B,Speech_CH_ZERO_PENG_hands_PENG_fightcomments_01,Speech_CH_ZERO_PENG_hands/2F46312B,Speech,Story missions,Speech_CH_ZERO_PENG_hands,ITA,13EEB34A,,Penguin,4.545,28000,1,32718,a0a44cda507f015e,
@@ -7549,11 +7544,6 @@ BANK,3FA65C62,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN_19,Speech_CINEMATIC_CH3_MEET
BANK,2E1796AE,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN_19,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN/2E1796AE,Speech,Cinematics,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN,ITA,18D8058E,,Penguin,4.759,28000,1,27292,6a7810833a312bc3,
BANK,2D661420,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN_19,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN/2D661420,Speech,Cinematics,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN,DEU,18D8058E,,Penguin,3.802,28000,1,26244,9a68ba22c0a3d35d,
BANK,20DF687A,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN_19,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN/20DF687A,Speech,Cinematics,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN,ESN,18D8058E,,Penguin,4.586,28000,1,28616,949f43d5dd3b8823,
BANK,3D3E517A,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER_1,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER/3D3E517A,Speech,Cinematics,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER,INT,,,Batman; Joker,2.492,44100,1,20047,3a4ba2b9c63934ed,
BANK,0B42FF7D,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER_1,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER/0B42FF7D,Speech,Cinematics,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER,FRA,,,Batman; Joker,2.534,28000,1,17075,bc9ab36f0afa7b27,
BANK,1FC3A83F,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER_1,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER/1FC3A83F,Speech,Cinematics,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER,ITA,,,Batman; Joker,2.544,28000,1,17300,9af41c14c8540a89,
BANK,29B4097E,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER_1,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER/29B4097E,Speech,Cinematics,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER,DEU,,,Batman; Joker,2.534,28000,1,16926,736d5f57b8397512,
BANK,0472E7FB,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER_1,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER/0472E7FB,Speech,Cinematics,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER,ESN,,,Batman; Joker,2.304,28000,1,16967,7c83a67ca3aedb6a,
BANK,0D579006,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA_GRP_BM_01,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA/0D579006,Speech,Combat barks,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA,INT,261948D3,,Batman,1.942,44100,1,15561,804be70742e85c57,
BANK,3E88D860,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA_GRP_BM_01,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA/3E88D860,Speech,Combat barks,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA,FRA,261948D3,,Batman,1.791,28000,1,12938,e833a0135c168827,
BANK,1CA672D4,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA_GRP_BM_01,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA/1CA672D4,Speech,Combat barks,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA,ITA,261948D3,,Batman,2.006,28000,1,14634,d24da394444a4430,
@@ -12018,34 +12008,11 @@ BANK,2FAA3476,Speech_FORENSICS_Cold_Correct_FORENSICS_Cold_Correct,Speech_FORENS
BANK,25DB5842,Speech_FORENSICS_Cold_Correct_FORENSICS_Cold_Correct,Speech_FORENSICS_Cold_Correct/25DB5842,Speech,Gameplay & hints,Speech_FORENSICS_Cold_Correct,ITA,2B9791FF,,,2.628,28000,1,20518,ac2a3c7fdad562df,
BANK,2F9A9DEF,Speech_FORENSICS_Cold_Correct_FORENSICS_Cold_Correct,Speech_FORENSICS_Cold_Correct/2F9A9DEF,Speech,Gameplay & hints,Speech_FORENSICS_Cold_Correct,DEU,2B9791FF,,,2.691,28000,1,21795,236e4c26a88643e8,
BANK,35FAA3F3,Speech_FORENSICS_Cold_Correct_FORENSICS_Cold_Correct,Speech_FORENSICS_Cold_Correct/35FAA3F3,Speech,Gameplay & hints,Speech_FORENSICS_Cold_Correct,ESN,2B9791FF,,,2.400,28000,1,19029,25a18a794248f9dc,
BANK,02B02BF2,Speech_GAME_OVER_Harley_Dead2,Speech_GAME_OVER_Harley/02B02BF2,Speech,Gameplay & hints,Speech_GAME_OVER_Harley,INT,,Harley Quinn,Harley Quinn,2.034,44100,3,44118,3cd5d7056427b140,
BANK,3CCA3AAE,Speech_GAME_OVER_Harley_Dead2,Speech_GAME_OVER_Harley/3CCA3AAE,Speech,Gameplay & hints,Speech_GAME_OVER_Harley,FRA,,Harley Quinn,Harley Quinn,2.164,28000,1,15141,b1917825638115d7,
BANK,20AED5D2,Speech_GAME_OVER_Harley_Dead2,Speech_GAME_OVER_Harley/20AED5D2,Speech,Gameplay & hints,Speech_GAME_OVER_Harley,ITA,,Harley Quinn,Harley Quinn,1.787,28000,1,12397,38e40367a3c12f66,
BANK,043620D9,Speech_GAME_OVER_Harley_Dead2,Speech_GAME_OVER_Harley/043620D9,Speech,Gameplay & hints,Speech_GAME_OVER_Harley,DEU,,Harley Quinn,Harley Quinn,2.090,28000,1,13346,ae7986530de3cf1e,
BANK,068435EE,Speech_GAME_OVER_Joker_DEAD_13,Speech_GAME_OVER_Joker/068435EE,Speech,Gameplay & hints,Speech_GAME_OVER_Joker,INT,,Joker,Joker,1.539,44100,3,37363,e2adc1dc6a43ba18,
BANK,0E820C9F,Speech_GAME_OVER_Joker_DEAD_13,Speech_GAME_OVER_Joker/0E820C9F,Speech,Gameplay & hints,Speech_GAME_OVER_Joker,FRA,,Joker,Joker,1.593,28000,1,12567,ab5ef6041156eb76,
BANK,16D46180,Speech_GAME_OVER_Joker_DEAD_13,Speech_GAME_OVER_Joker/16D46180,Speech,Gameplay & hints,Speech_GAME_OVER_Joker,ITA,,Joker,Joker,1.029,28000,1,8680,1bd4cf5d6cc3d049,
BANK,1AD4AAB1,Speech_GAME_OVER_Joker_DEAD_13,Speech_GAME_OVER_Joker/1AD4AAB1,Speech,Gameplay & hints,Speech_GAME_OVER_Joker,DEU,,Joker,Joker,1.593,28000,1,10725,117399e19037e4a5,
BANK,3615E00B,Speech_GAME_OVER_Joker_DEAD_13,Speech_GAME_OVER_Joker/3615E00B,Speech,Gameplay & hints,Speech_GAME_OVER_Joker,ESN,,Joker,Joker,1.258,28000,1,9521,4282086256d079b9,
BANK,1AC4F6B4,Speech_GAME_OVER_Mr_Freeze_FREEZE_new,Speech_GAME_OVER_Mr_Freeze/1AC4F6B4,Speech,Gameplay & hints,Speech_GAME_OVER_Mr_Freeze,INT,1583D57C,,Mr. Freeze,2.750,44100,3,58665,0f7610cc9079b4d3,
BANK,1D9BDA58,Speech_GAME_OVER_Mr_Freeze_FREEZE_new,Speech_GAME_OVER_Mr_Freeze/1D9BDA58,Speech,Gameplay & hints,Speech_GAME_OVER_Mr_Freeze,FRA,1583D57C,,Mr. Freeze,2.447,44100,1,19577,3e34a16458bf6326,
BANK,3B38366F,Speech_GAME_OVER_Mr_Freeze_FREEZE_new,Speech_GAME_OVER_Mr_Freeze/3B38366F,Speech,Gameplay & hints,Speech_GAME_OVER_Mr_Freeze,ITA,1583D57C,,Mr. Freeze,2.826,44100,1,18356,002437e9e096ad72,
BANK,3481719C,Speech_GAME_OVER_Mr_Freeze_FREEZE_new,Speech_GAME_OVER_Mr_Freeze/3481719C,Speech,Gameplay & hints,Speech_GAME_OVER_Mr_Freeze,DEU,1583D57C,,Mr. Freeze,2.926,44100,1,21183,1ee5dc61590ea268,
BANK,0D2CE9E1,Speech_GAME_OVER_Mr_Freeze_FREEZE_new,Speech_GAME_OVER_Mr_Freeze/0D2CE9E1,Speech,Gameplay & hints,Speech_GAME_OVER_Mr_Freeze,ESN,1583D57C,,Mr. Freeze,2.576,44100,1,19639,55ded8c51a05fd16,
BANK,23647FF3,Speech_GAME_OVER_Talia_al_Ghul_Dead_Demon,Speech_GAME_OVER_Talia_al_Ghul/23647FF3,Speech,Gameplay & hints,Speech_GAME_OVER_Talia_al_Ghul,FRA,,Talia al Ghul,Talia al Ghul,1.836,28000,1,14387,a0487f77e650ec00,
BANK,0173EC0D,Speech_GAME_OVER_Talia_al_Ghul_Dead_Demon,Speech_GAME_OVER_Talia_al_Ghul/0173EC0D,Speech,Gameplay & hints,Speech_GAME_OVER_Talia_al_Ghul,ITA,,Talia al Ghul,Talia al Ghul,2.095,28000,1,15228,e60c97bbb7839e0a,
BANK,07AE2F63,Speech_GAME_OVER_Talia_al_Ghul_Dead_Demon,Speech_GAME_OVER_Talia_al_Ghul/07AE2F63,Speech,Gameplay & hints,Speech_GAME_OVER_Talia_al_Ghul,ESN,,Talia al Ghul,Talia al Ghul,1.924,28000,1,14521,b3b9c78725afa4d3,
BANK,2EF4A47C,Speech_GAME_OVER_Two_Face_Question,Speech_GAME_OVER_Two_Face/2EF4A47C,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,INT,,Two-Face,,1.463,44100,3,32957,0bc9366548a9daa2,
BANK,01ED1FEB,Speech_GAME_OVER_Two_Face_Question,Speech_GAME_OVER_Two_Face/01ED1FEB,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,FRA,,Two-Face,,1.597,28000,1,11484,ec2249942345ac00,
BANK,25E33254,Speech_GAME_OVER_Two_Face_Question,Speech_GAME_OVER_Two_Face/25E33254,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,ITA,,Two-Face,,1.390,28000,1,11300,c538dd16556b67d5,
BANK,34CCE5AC,Speech_GAME_OVER_Two_Face_Question,Speech_GAME_OVER_Two_Face/34CCE5AC,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,DEU,,Two-Face,,1.463,28000,1,11168,859d6637c812a421,
BANK,34F10C93,Speech_GAME_OVER_Two_Face_Question,Speech_GAME_OVER_Two_Face/34F10C93,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,ESN,,Two-Face,,1.236,28000,1,9068,d62fb0568bcc9b23,
BANK,38BD88BC,Speech_GAME_OVER_Two_Face_Reply_Bat_2,Speech_GAME_OVER_Two_Face/38BD88BC,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,INT,,Two-Face,,1.730,44100,3,39669,2b265c6efc35fd09,
BANK,2DD25DC9,Speech_GAME_OVER_Two_Face_Reply_Bat_2,Speech_GAME_OVER_Two_Face/2DD25DC9,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,FRA,,Two-Face,,1.862,28000,1,14695,c503e36bcf0b9ef0,
BANK,266B4A26,Speech_GAME_OVER_Two_Face_Reply_Bat_2,Speech_GAME_OVER_Two_Face/266B4A26,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,ITA,,Two-Face,,1.772,28000,1,13666,28f64eb52ca4cda3,
BANK,27071B28,Speech_GAME_OVER_Two_Face_Reply_Bat_2,Speech_GAME_OVER_Two_Face/27071B28,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,DEU,,Two-Face,,1.730,28000,1,13012,a849e1f91600de98,
BANK,309FF2EE,Speech_GAME_OVER_Two_Face_Reply_Bat_2,Speech_GAME_OVER_Two_Face/309FF2EE,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,ESN,,Two-Face,,1.861,28000,1,14258,3498e68019b17dc3,
BANK,07615F5F,Speech_GAME_OVER_Two_Face_Reply_Cat_1,Speech_GAME_OVER_Two_Face/07615F5F,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,ESN,,Two-Face,Catwoman,2.005,28000,1,14774,ba2ab6a3caf28194,
BANK,14F358D6,Speech_INTERROGATION_CH5_MAYOR_01_04b,Speech_INTERROGATION_CH5_Mayor_01/14F358D6,Speech,Interrogations,Speech_INTERROGATION_CH5_Mayor_01,,1BB09D3E,,,0.691,44100,1,4876,6c1dee7c8f96436f,
BANK,2FD9AB10,Speech_MAP_Deadshot_find_matching_1_CHAR_BATMAN_Deadshot_find_matching_01,Speech_MAP_Deadshot_find_matching_1/2FD9AB10,Speech,Gameplay & hints,Speech_MAP_Deadshot_find_matching_1,INT,139B2B14,,Deadshot; Batman,5.160,44100,1,48628,a7d5135f109446df,
BANK,36805E4A,Speech_MAP_Deadshot_find_matching_1_CHAR_BATMAN_Deadshot_find_matching_01,Speech_MAP_Deadshot_find_matching_1/36805E4A,Speech,Gameplay & hints,Speech_MAP_Deadshot_find_matching_1,FRA,139B2B14,,Deadshot; Batman,4.884,44100,1,41222,a6b0e04d967013af,
@@ -14817,22 +14784,6 @@ BANK,141EEA20,Speech_SEQUENCE_CH0_CATWOMAN_INTRO_08,Speech_SEQUENCE_CH0_CATWOMAN
BANK,0496BF7C,Speech_SEQUENCE_CH0_CATWOMAN_INTRO_08,Speech_SEQUENCE_CH0_CATWOMAN_INTRO/0496BF7C,Speech,Story missions,Speech_SEQUENCE_CH0_CATWOMAN_INTRO,ITA,18D7E643,,Catwoman,1.907,28000,1,14278,e63c8b66f70ecf54,
BANK,0A9933FA,Speech_SEQUENCE_CH0_CATWOMAN_INTRO_08,Speech_SEQUENCE_CH0_CATWOMAN_INTRO/0A9933FA,Speech,Story missions,Speech_SEQUENCE_CH0_CATWOMAN_INTRO,DEU,18D7E643,,Catwoman,1.907,28000,1,12844,20e101e6f36702c6,
BANK,1BD54DDE,Speech_SEQUENCE_CH0_CATWOMAN_INTRO_08,Speech_SEQUENCE_CH0_CATWOMAN_INTRO/1BD54DDE,Speech,Story missions,Speech_SEQUENCE_CH0_CATWOMAN_INTRO,ESN,18D7E643,,Catwoman,1.763,28000,1,12834,4d0ba4b1479279fe,
BANK,14112E39,Speech_SEQUENCE_CH2_RESCUE_DOC_01,Speech_SEQUENCE_CH2_RESCUE_DOC/14112E39,Speech,Story missions,Speech_SEQUENCE_CH2_RESCUE_DOC,INT,,Batman,Doctor,1.031,44100,1,8548,a75d7bcfa77efdb3,
BANK,045C2B58,Speech_SEQUENCE_CH2_RESCUE_DOC_01,Speech_SEQUENCE_CH2_RESCUE_DOC/045C2B58,Speech,Story missions,Speech_SEQUENCE_CH2_RESCUE_DOC,FRA,,Batman,Doctor,1.071,28000,1,6542,4b3e3f7aa9d1a83b,
BANK,3A312CE8,Speech_SEQUENCE_CH2_RESCUE_DOC_01,Speech_SEQUENCE_CH2_RESCUE_DOC/3A312CE8,Speech,Story missions,Speech_SEQUENCE_CH2_RESCUE_DOC,ITA,,Batman,Doctor,1.075,28000,1,8831,40ffdfe34956c897,
BANK,3808FEFE,Speech_SEQUENCE_CH2_RESCUE_DOC_01,Speech_SEQUENCE_CH2_RESCUE_DOC/3808FEFE,Speech,Story missions,Speech_SEQUENCE_CH2_RESCUE_DOC,DEU,,Batman,Doctor,1.071,28000,1,8140,8e5f47ab89719263,
BANK,3CEB24BB,Speech_SEQUENCE_CH2_RESCUE_DOC_01,Speech_SEQUENCE_CH2_RESCUE_DOC/3CEB24BB,Speech,Story missions,Speech_SEQUENCE_CH2_RESCUE_DOC,ESN,,Batman,Doctor,0.901,28000,1,8881,4f73e5c03f3693de,
BANK,2A2DEC79,Speech_SEQUENCE_CH3_MUSEUM_A1_BM_FREEZESUIT_01,Speech_SEQUENCE_CH3_MUSEUM_A1_BM_FREEZESUIT/2A2DEC79,Speech,Story missions,Speech_SEQUENCE_CH3_MUSEUM_A1_BM_FREEZESUIT,ESN,,,Batman; Mr. Freeze,2.024,28000,1,16548,e1430b84337ad106,
BANK,20FB1929,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH_01,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH/20FB1929,Speech,Story missions,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH,INT,,Penguin,Penguin,2.545,44100,1,21615,8c3745e89fec1a07,
BANK,32FEBF22,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH_01,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH/32FEBF22,Speech,Story missions,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH,FRA,,Penguin,Penguin,2.586,28000,1,17858,b6655fbf823b6a03,
BANK,1C5086AC,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH_01,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH/1C5086AC,Speech,Story missions,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH,ITA,,Penguin,Penguin,2.596,28000,1,18998,5d5e9cec61c04de0,
BANK,057E52DD,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH_01,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH/057E52DD,Speech,Story missions,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH,DEU,,Penguin,Penguin,2.586,28000,1,19398,6e98a57e0eb693c4,
BANK,3B667571,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH_01,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH/3B667571,Speech,Story missions,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH,ESN,,Penguin,Penguin,2.448,28000,1,18027,b99b40c9639f8983,
BANK,3B3AD732,Speech_SEQUENCE_TWOFACE_END2_05,Speech_SEQUENCE_TWOFACE_END2/3B3AD732,Speech,Story missions,Speech_SEQUENCE_TWOFACE_END2,INT,,Two-Face,Two-Face,0.960,44100,1,7760,ce1badd5de22d5d4,
BANK,3E7514E2,Speech_SEQUENCE_TWOFACE_END2_05,Speech_SEQUENCE_TWOFACE_END2/3E7514E2,Speech,Story missions,Speech_SEQUENCE_TWOFACE_END2,FRA,,Two-Face,Two-Face,1.049,28000,1,6311,99bef837e8302411,
BANK,14CC1583,Speech_SEQUENCE_TWOFACE_END2_05,Speech_SEQUENCE_TWOFACE_END2/14CC1583,Speech,Story missions,Speech_SEQUENCE_TWOFACE_END2,ITA,,Two-Face,Two-Face,0.960,28000,1,6860,f324fe563584eaed,
BANK,28F62D59,Speech_SEQUENCE_TWOFACE_END2_05,Speech_SEQUENCE_TWOFACE_END2/28F62D59,Speech,Story missions,Speech_SEQUENCE_TWOFACE_END2,DEU,,Two-Face,Two-Face,0.960,28000,1,6515,7108d991488807c6,
BANK,32D76E1C,Speech_SEQUENCE_TWOFACE_END2_05,Speech_SEQUENCE_TWOFACE_END2/32D76E1C,Speech,Story missions,Speech_SEQUENCE_TWOFACE_END2,ESN,,Two-Face,Two-Face,1.017,28000,1,7232,3eff15f4f9a7d8f5,
BANK,1F7A6D0F,Speech_STEEL_B1_CH2_GRUNT_01,Speech_STEEL_B1_CH2_GRUNT/1F7A6D0F,Speech,Story missions,Speech_STEEL_B1_CH2_GRUNT,INT,18146E8B,,,0.523,44100,1,4294,052817cde90ab5a0,
BANK,3BFCB495,Speech_STEEL_B1_CH2_GRUNT_01,Speech_STEEL_B1_CH2_GRUNT/3BFCB495,Speech,Story missions,Speech_STEEL_B1_CH2_GRUNT,FRA,18146E8B,,,0.541,44100,1,4343,40d1717706422c9b,
BANK,327E0165,Speech_STEEL_B1_CH2_GRUNT_01,Speech_STEEL_B1_CH2_GRUNT/327E0165,Speech,Story missions,Speech_STEEL_B1_CH2_GRUNT,ITA,18146E8B,,,0.541,44100,1,4343,40d1717706422c9b,
Can't render this file because it is too large.

BIN
data/catalog/BANK_locations.csv.gz
View File

Binary file not shown.

10
data/catalog/README.md
View File

@@ -3,7 +3,7 @@
| file | content |
|------|---------|
| `<ARCHIVE>.csv` | every sound of one audio archive: `INT`, `FRA`, `ITA`, `DEU`, `ESN` (speech WADs, one per language), `SFX` (music, ambience and effects streamed from the SFX WAD) and `BANK` (in-memory sounds stored inside soundbanks: footsteps, foley, combat, weapons, UI, efforts; `docs/formats/banks.md`) |
| `BANK_locations.csv.gz` | every stored copy of the `BANK` sounds, and the prefetch heads of speech lines (below) |
| `BANK_locations.csv.gz` | every stored copy of the `BANK` sounds, and the bank copies of WAD speech recordings (prefetch heads, whole copies; below) |
| `SUB_<LANG>.csv` | subtitles of the speech lines, one file per language (11 languages; `docs/formats/subtitles.md`) |
| `SUB_locations.csv.gz` | every stored copy of each subtitle string (dialogue objects in the packages) |
| `characters.csv` | `name,character,language`: verified speaker per line name (empty `language`: every language, the same line everywhere; a language code would label a sound of that language only - none needs it now) |
@@ -17,7 +17,7 @@ All CSVs are UTF-8 with a header row. They contain only names and metadata (no g
|--------|---------|
| `archive` | `INT`, `FRA`, ..., `SFX` or `BANK` |
| `target` | location: `HASH` (WAD entry with one WEM), `HASH:i` (i-th variation, 0-based), `PACK/MEDIA` (music stream pack), `MEDIA` (BANK: Wwise media ID) |
| `name` | display name. Speech: the line's name, the same in every language (`docs/decisions.md` 5): its speech event (e.g. `Speech_TAPE_JOKER_1_04`, `NAME#2` = second version an event plays) or, for lines no named event plays, the INT recording's position name. A recording stored twice keeps its position name in the second place. SFX: `NAME`, `NAME_1..n`. BANK: the Wwise event that plays it (`Play_..._i`) or `<Bank>_<n>` |
| `name` | display name. Speech: the line's name, the same in every language (`docs/decisions.md` 5): its speech event (e.g. `Speech_TAPE_JOKER_1_04`, `NAME#2` = second version an event plays) or, for lines no named event plays, the INT recording's position name. A recording stored twice keeps its position name in the second place. SFX: `NAME`, `NAME_1..n`. BANK: the Wwise event that plays it (`Play_..._i`) or `<Bank>_<n>` (`<Bank>_<MEDIA>` where that name is taken, e.g. by the SFX WAD's position names) |
| `aliases` | other accepted names, `\|`-separated: `NAME#i` = i-th WEM of the entry in this language's WAD (positions differ between languages), the entry name for single entries, `<Bank>/<MEDIA>` for BANK |
| `group` | Speech, Music, Ambience, SFX, Unknown (`docs/categories.md`) |
| `category` | finer category (`docs/categories.md`) |
@@ -38,11 +38,11 @@ All CSVs are UTF-8 with a header row. They contain only names and metadata (no g
| `file` | file under `BmGame/CookedPCConsole`, `/`-separated (`SFX/xxxxxxxx.bnk` or `Name.upk`) |
| `offset` | position of the WEM in the *uncompressed* file (hex; `docs/formats/upk.md`) |
| `size` | bytes reserved for it there |
| `kind` | `m` = copy of a BANK sound, `p` = prefetch head of a speech line |
| `sound` | `p` rows: key of that speech line (e.g. `INT:2B151272:0`); each language has its own lines and heads |
| `kind` | `m` = copy of a BANK sound, `p` = copy of (the start of) a WAD speech recording: a prefetch head, or a whole copy with the same media ID |
| `sound` | `p` rows: key of that speech recording (e.g. `INT:2B151272:0`); each language has its own recordings and heads |
A replacement is written into every `m` copy of a BANK sound; replacing a speech line also rewrites its `p`
heads (`docs/formats/banks.md`).
copies (`docs/formats/banks.md`).
## `SUB_<LANG>.csv` columns
| column | meaning |

21
data/catalog/index.json
View File

@@ -63,20 +63,21 @@
},
"BANK": {
"fingerprint": "3f9766af425b8eef",
"sounds": 15404,
"sounds": 15355,
"groups": {
"Ambience": 1015,
"SFX": 6912,
"Speech": 7471,
"Speech": 7422,
"Music": 2,
"Unknown": 4
},
"copies": 93910,
"named_by_event": 13501,
"named_by_bank": 1854,
"copies": 93826,
"files": 1499,
"named_by_event": 6456,
"prefetch_heads": {
"lines": 396,
"copies": 2101,
"recordings": 445,
"copies": 2185,
"unmapped": 0
},
"issues": []
@@ -111,6 +112,9 @@
"RUS": 9463
},
"copies": 168688,
"dialogue_objects": 14744,
"dialogue_lines": 9032,
"scripted_objects": 41,
"issues": []
},
"LINES": {
@@ -121,8 +125,11 @@
"INT",
"ITA"
],
"recordings": 61510,
"recordings": 61485,
"lines": 12434,
"wad_lines": 10811,
"wad_lines_named_by_event": 10174,
"wad_lines_named_after_int_recording": 637,
"lines_in_every_language": 12434,
"lines_missing_a_language": {},
"without_line": [],

12
docs/categories.md
View File

@@ -4,8 +4,9 @@ Implemented in `ackit/classify.py`; used both when building the shipped catalog
and when a game build is not in the catalog (runtime scan), so every language/build gets the same scheme.
## Groups and categories
Every sound has a **group** and a finer **category**, derived from its Wwise name (from the community hash
list; unnamed IDs show as hex). First matching rule wins:
Every sound has a **group** and a finer **category**, derived from its name (taken from the game files: the
soundbank names in `CookedBanks.txt` and the Wwise event names; the 3 SFX entries no name reaches show as hex).
First matching rule wins:
| name starts with | group | category |
|------------------|-------|----------|
@@ -78,9 +79,10 @@ alternates speakers between `NAME_1`, `NAME_2`, ...):
identified). A line in two lists gets both ("A; B").
- **characters_inferred** - characters and roles *mentioned by the name* (token rules in `classify._TOKENS`,
e.g. `BM`/`BATMAN` -> Batman, `PENG` -> Penguin, `TF` -> Two-Face, `THUG*`/`GENTHUG*` -> Thug, `TYGER` ->
TYGER guard; interview tapes add Hugo Strange). A hint for the ~7,500 lines without a verified speaker,
TYGER guard; interview tapes add Hugo Strange). A hint for the ~7,400 lines without a verified speaker,
never shown as the speaker. In the editor the character filter matches verified speakers, and optionally
inferred ones.
Coverage (every voice language, the same lines): 3,366 with a verified speaker, 6,263 more with inferred
characters, 1,186 with neither (mostly predator chatter and efforts of unnamed thugs).
Coverage of the 10,811 WAD voice lines (every voice language, the same lines): 3,366 with a verified speaker,
6,258 more with inferred characters, 1,187 with neither (combat barks 407, story missions 267, open-world
chatter 228, radio & TV 122, ...).

28
docs/decisions.md
View File

@@ -75,11 +75,13 @@ tables): high risk for little gain, not done.
replacements applied now", so reverting the last one restores the original bytes exactly, and an interrupted
operation is repaired by the next one. Conflicts: the whole-file SHA-1 must be the original's or the last one
ackit wrote.
- **Prefetch heads:** 396 speech lines (about 79 per language) have their first bytes copied into banks; a
replacement of such a line rewrites those heads too, otherwise the start would play the old line.
- **Prefetch heads and whole copies:** 396 voice recordings (81 lines, 78-80 recordings per language) have
their first bytes copied into banks, and 49 more are stored whole in banks (same media ID, identical bytes);
a replacement of such a recording rewrites those copies too, otherwise the game could play the old line.
They are not listed as sounds of their own.
- **Names:** no list names these media. Names come from Wwise events whose names are in the
packages' name and export tables (FNV-1 of the name = event ID), following event -> action -> containers ->
sounds (6,456 sounds); the rest are numbered per bank. Categories use the bank name. A localized speech line
sounds (13,501 of 15,355, voice lines included); the other 1,854 are numbered per bank. Categories use the bank name. A localized speech line
has the same name in every language (its line, section 5).
- **Localized speech:** speech banks hold one copy per voice language (4d); every media is labelled with the
language(s) whose banks hold it (a few recordings are shared by several languages) and the catalog shows
@@ -119,11 +121,11 @@ copy per voice language) is filtered the same way, so the sound list never mixes
*Lines*): the recordings that are sources of the same Wwise Sound objects are one line, with exactly one
recording per language (12,434 lines, all 5 languages). The catalog stores that identity (`line`), and
speaker labels, mod packages (another language: the same `line`) and subtitles follow it. Positions are
useless across languages (WEMs are stored in media-ID order; position i is another line in 75 % of the
multi-variation entries).
- A line has one name in every language: the most specific speech event that plays it (10,156 of 10,815 voice
lines), numbered `NAME#2` in the event's own order when it plays two versions; the other 659 keep the name of
their English (INT) recording's position. The position in each language stays an alias (`NAME#i`).
useless across languages (WEMs are stored in media-ID order; 81.2-81.6 % of the WEMs in multi-variation entries
are at another position than in INT).
- A line has one name in every language: the most specific speech event that plays it (10,174 of the 10,811
WAD voice lines), numbered `NAME#2` in the event's own order when it plays two versions; the other 637 keep
the name of their English (INT) recording's position. The position in each language stays an alias (`NAME#i`).
- A recording stored twice in a WAD (5 per language) is written and reverted in both places.
- The catalog records the ORIGINAL bytes: `tools/build_catalog.py` stops before writing anything if a sound
differs from the current catalog (a modified game), unless `--accept-changes` (a game update).
@@ -174,14 +176,18 @@ copy per voice language) is filtered the same way, so the sound list never mixes
## 8. Verification
- `pytest`: unit tests, a synthetic game (WADs, packs, packages with banks, localized banks, dialogue and subtitle
objects, localization files), a two-language game built like the real one (per-language BKPK tables, a media
ID shared by both languages, a recording stored twice), offscreen GUI tests, and read-only real-game tests
when the game is installed.
ID shared by both languages, a recording stored twice, one also stored whole in a bank), offscreen GUI
tests, and read-only real-game tests when the game is installed.
- `ackit verify` (WEMs rebuilt byte-for-byte, encode/decode), `ackit verify --catalog` (catalog vs game files,
and every relation re-derived from the game files with independent checks; it prints them as facts).
- Real-game write round trips without launching the game: `tools/e2e_features.py [--lang all]` (every copy of
an in-memory sound, localized line, prefetch heads, subtitles incl. RSubtitle, menu text, sound + subtitle in
an in-memory sound, localized line, prefetch heads and whole bank copies of a line, subtitles incl. RSubtitle, menu text, sound + subtitle in
one package, a mod moved to another voice language; `all` = each of the 11 languages) and
`tools/e2e_editor.py` (through the editor's code path).
- Documentation: `tools/check_docs.py [--scan] [--wems]` recomputes every figure the documents state (from the
catalog, the build's `index.json`, the menu files; with `--scan` from a fresh package scan and the catalog
check's facts; with `--wems` from rebuilding every WAD WEM) and fails on any difference, so figures are never
copied by hand.
- In the running game: `ackit verify --ingame` (music heard), `--banks` (patched packages loaded, probe data in
memory), `--texts` (menu texts and Startup subtitles found in memory). Every tool reverts in a `finally` block
and checks that every touched file is byte-identical afterwards.

58
docs/formats/banks.md
View File

@@ -10,29 +10,31 @@ Survey of the installed game (buildid 3718300), 2026-10-09.
| where | banks | unique in-memory WEMs |
|-------|-------|-----------------------|
| loose `SFX\*.bnk` (Batman/Catwoman footsteps, 32 surfaces each, + `TST_Prep_Media`) | 65 | 1,117 |
| banks embedded in `.upk` packages (in 1,432 of the 2,650 packages) | 3,340 | 15,641 (~276 MB) |
| banks embedded in `.upk` packages (in 1,432 of the 2,650 packages) | 3,340 | 14,802 |
All of them are in the catalog (archive `BANK`).
The two sets share 119 media IDs: 15,800 unique media IDs in 96,011 places (bank copies). 396 of them are
**prefetch heads** of streamed WAD sounds (below), so there are **15,404 in-memory sounds**. All copies of a
**prefetch heads** of streamed WAD sounds and 49 are **whole copies** of WAD recordings (below), so there are
**15,355 in-memory sounds**. All copies of a
media ID are byte-identical except `1C898731` (two variants). The WADs, by comparison, are fully covered: the
BKPK tables in the packages reference 2,543 WAD entries, all of them in the catalog.
Embedded WEMs by bank name prefix: `Speech` 7,872, `CHR` (character foley, combat, bosses) 3,896, `PSO` 781,
`WPN` 664, `NPC` 475, `OBJ` 439, `SEQ` 265, `GUI` 208, `PSA` 203, `MTN` 196, `PX` 195, `DVC` 100, `GEN` 80,
`PHY` 73, `AMB` 65, `SBJ` 50, others < 30. The `Speech_` banks (e.g. `Speech_EMOTE_Batman`, `..._GRUNT`) hold
speech: grunts and efforts (language-neutral, 165) and **localized lines** (about 1,460 per language, see
*Localized banks*).
All WEMs seen are Vorbis (format tag 0xFFFF), the same codec as the WADs: the footsteps are mono (4 stereo),
the embedded ones mono or stereo (~22 % stereo in a 2,533-WEM sample).
Embedded WEMs by bank name prefix (a media held by banks of two prefixes counts for both): `Speech` 7,827, `CHR`
(character foley, combat, bosses) 3,624, `PSO` 737, `WPN` 577, `NPC` 471, `OBJ` 411, `SEQ` 253, `PSA` 203,
`PX` 195, `MTN` 191, `GUI` 155, `DVC` 100, `GEN` 80, `PHY` 73, `AMB` 61, `SBJ` 50, others < 30. The `Speech_` banks (e.g. `Speech_EMOTE_Batman`, `..._GRUNT`) hold
speech: efforts, grunts and a few other lines stored once for every language (119 sounds) and **localized
lines** (about 1,450 per language, see *Localized banks*).
All WEMs seen are Vorbis (format tag 0xFFFF), the same codec as the WADs: the loose footsteps are mono (4
stereo); of all 15,355 in-memory sounds 13,982 are mono, 1,196 stereo, 177 have 3-6 channels.
## Names
- No list names these media IDs (the community hash list names only WAD entries and banks).
- Media IDs have no names of their own: a sound is named after the Wwise event that plays it (event names
are in the packages' name and export tables, event ID = FNV-1 of the name), else after its bank.
- `CookedPCConsole\CookedBanks.txt` lists all 3,345 cooked bank names. Bank ID = FNV-1 32 of the lowercased
name (the same hash Wwise uses for every name). 3,338 of them are embedded in packages, 4 are loose `.bnk`
files (`Init`, `Global`, `Music`, `Music_PDLC`); `AMB_OW_A6`, `AMB_OW_C1`, `AMB_OW_E1`,
`Speech_EMOTE_Thug_Muffled_Stop` and `Speech_GFxUI_ArkhamCityStories` were found nowhere.
name (the same hash Wwise uses for every name). 3,338 of them are embedded in packages (`Music`, `Music_PDLC`
also as loose `.bnk` files), 2 are only loose `.bnk` files (`Global`, `Init`); `AMB_OW_A6`, `AMB_OW_C1`,
`AMB_OW_E1`, `Speech_EMOTE_Thug_Muffled_Stop` and `Speech_GFxUI_ArkhamCityStories` were found nowhere.
- Loose footstep banks are named in their STID chunk (`CHR_BatmanFootsteps_Concrete_Media`, ...).
## Bank layout (version 56, Wwise 2011.1)
@@ -51,6 +53,12 @@ with bank id 0 whose media is not in the same bank (loaded by another bank); 133
(`wad.md`). A prefetch source's media is also in the bank's DIDX: that in-memory copy is the **head** of the
streamed WEM (its size = the source's size), which the game plays while the rest is read from the WAD. 396
WAD sounds have such heads (2,101 copies); replacing one of them must rewrite the heads too.
49 more WAD recordings (9-11 per language: game-over taunts, an encryption-card line, a cutscene line, scripted
story lines) are
also stored **whole** in banks: an in-memory source with the WAD recording's own media ID, byte-identical to
the WAD WEM (84 copies). They are handled like heads: part of the WAD line, written and reverted with it, not
sounds of their own (`catalog._whole_recordings`). Heads and whole copies together: 445 recordings, 2,185
copies (`BANK_locations.csv.gz` kind `p`).
## Localized banks
Speech banks are localized inside the (language-neutral) packages: one `AkBank` export starts with a language
@@ -58,9 +66,10 @@ list (FStrings `INT FRA ITA DEU ESN`) and then holds one block per language, in
of the speech WAD entry for that language (`wad.md`) and a complete bank, with the same bank ID in every
language. ackit gives each bank and each BKPK table the language of its position in that list (an export
whose block count differs from its language count would be reported; none does). Each in-memory media gets
the languages of the banks that hold it (`BANK.csv` column `language`): 8,098 sounds are language-neutral,
~1,460 belong to one voice language, 6 to several (a DLC intro line shared by INT/FRA/ITA/ESN while DEU has
its own); a media stored in the banks of every language is listed as `''` (the same recording everywhere).
the languages of the banks that hold it (`BANK.csv` column `language`): 8,092 sounds are language-neutral,
1,450-1,456 belong to one voice language, 6 to several (5 shared by INT/FRA/ITA/ESN while DEU has its own, 1
by INT/FRA/ITA); a media stored in the banks of every language is listed as `''` (the same recording
everywhere; 167 of these are voice lines, e.g. efforts and radio chatter).
A catalog shows the neutral sounds plus the loaded language's lines; names are unique per language.
## Lines: one recording per language
@@ -73,17 +82,18 @@ and the game ties the recordings together itself:
- Every recording, streamed, prefetched or in memory, is the source of a Wwise Sound object; Sound objects,
events, actions and containers have the same IDs in every language, only the media differ. A *line* = the
recordings that are sources of the same Sound objects: 12,434 lines (WAD and in-memory), each with exactly
one recording in each of the 5 voice languages (61,510 recordings), no Sound object with two recordings in
one recording in each of the 5 voice languages (61,485 recordings), no Sound object with two recordings in
one language. Line identity (`line` column) = the smallest Sound object ID.
- Prefetched sources count too: the 396 lines with a prefetch head (mostly the game-over taunts) are linked
only through them.
- Prefetched sources count too: the 81 lines with a prefetch head (396 recordings over the 5 languages; mostly
the game-over taunts) are linked only through them. A WAD recording that a bank also stores whole (49) is
the same recording (same media ID), so its line is unchanged and the bank copies are written with it.
- 5 recordings per language are stored twice in that language's WAD (same media ID in two entries, byte-
identical, e.g. `Speech_STEEL_A1_CH2_Thug_behinddoormuffled` and `DIA_SteelA1Muffled`): the second place
is a copy of the line (same `line`, keeps its position name) and is written and reverted with it, because
the game plays whichever place the loaded package's table names.
- Independent checks (no BKPK involved): every WAD recording is named as a source by a Sound object of a bank
of its own language (54,055 of 54,055); every stored prefetch head equals the start of the recording its
line gives it (2,101 of 2,101); a line's recordings have similar lengths across languages (11,761 of 12,434
of its own language (54,055 of 54,055); every stored head or whole copy equals the start of the recording
its line gives it (2,185 of 2,185); a line's recordings have similar lengths across languages (11,761 of 12,434
within a factor 1.5; the others are different edits, e.g. a 0.55 s ESN line vs 5.46 s elsewhere).
## Banks inside packages
@@ -99,10 +109,12 @@ and the game ties the recordings together itself:
## How ackit edits them
- Catalog archive `BANK` (one row per media ID) + `data/catalog/BANK_locations.csv.gz` (every copy, and the
prefetch heads of speech lines); built by `tools/build_catalog.py`, checked by `ackit verify --catalog`.
heads and whole copies of WAD speech recordings); built by `tools/build_catalog.py`, checked by `ackit verify
--catalog`.
- A replacement is written **in place into every copy** (no larger than the smallest copy, zero-padded), so
DIDX/HIRC offsets and sizes and the package layout never change. Loose banks are patched directly; in
packages only the touched blocks are re-encoded (`upk.md`). A sound left in one package only would let the
game load the original from another one.
- Replacing a speech line that has prefetch heads rewrites the heads (first bytes of the new, zero-padded line).
- Replacing a speech line that has prefetch heads or whole copies rewrites them (the first bytes of the new,
zero-padded line; for a whole copy, all of it).
- Design and revert rules: `docs/decisions.md` (4b). The same WEM rules apply (`wem.md`).

13
docs/formats/subtitles.md
View File

@@ -5,14 +5,15 @@ the UE3 packages (`upk.md`), holding the subtitle text of all 11 languages. acki
`ackit/subtitles.py` and writes them through `Workspace.texts` (`ackit/texts.py`).
## Inventory (buildid 3718300)
- 14,743 dialogue objects in the packages for 9,016 speech lines (a line's object is copied into every package
- 14,744 dialogue objects in the packages for 9,032 speech lines (a line's object is copied into every package
that uses it; all copies are identical), plus 41 scripted subtitles (below): ~9,400 subtitle strings and
168,688 stored copies.
- 11 languages: INT, FRA, ITA, DEU, ESN, ESM, POL, POR, RUS, JPN, KOR (the speech WADs exist only for INT, FRA,
ITA, DEU, ESN; the other languages play the INT voices with their own subtitles).
- ~9,400 subtitle strings per language: 94 % of the subtitles have one line, the rest up to 5 lines.
- Strings are stored as cp1252 (positive FString length) or UTF-16 (negative length): RUS, POL, JPN, KOR
and some INT/ESN/FRA/DEU strings are UTF-16.
- 9,035-9,667 subtitle strings per language: 94-98 % of the subtitles have one line (JPN: all), the rest up
to 5 lines.
- Strings are stored as cp1252 (positive FString length) or UTF-16 (negative length): almost all RUS, JPN and
KOR strings and 85 % of POL are UTF-16; the other languages have 17-56 UTF-16 strings each.
## Scripted subtitles (`RSubtitle`)
41 more subtitles (radio and encryption-card lines) are `RSubtitle` objects in `Startup.upk`: the same layout,
@@ -34,7 +35,7 @@ or property name from the name table, i32 byte size, i32 array index, then `size
| `0x09 LocalizedSubtitles` | u32 count, then per language: `0D 00 00 00` FString language code, `09 00 0C 00 Subtitles` struct, `00 00` |
| `Subtitles` | u32 line count, then per line: `0D 00 00 00` FString text, `04 00 0C 00` 4-byte value, `00 00` |
The sizes add up exactly in all 14,743 objects (checked by the parser: `SubtitleError` otherwise).
The sizes add up exactly in all 14,744 objects (checked by the parser: `SubtitleError` otherwise).
## Link to the speech sound
The speech event name is hashed (FNV-1) to the Wwise event ID and followed through the soundbanks (event ->
@@ -46,7 +47,7 @@ export names) and from the subtitle objects themselves: some events are listed i
dialogue / RSubtitle object names them (Vicki Vale's radio reports, the Hush tape reveal).
An event that plays two versions of a line (105 subtitled events, e.g. Mr. Freeze's lines) links its subtitle
to both (`sound` = `key1;key2`). Linked: 103,304 of 103,658 strings in the 11 languages (INT 9,373 / 9,404),
each to the same line in every language. The rest (29 speech names, ~32 strings per language: Riddler trophy
each to the same line in every language. The rest (24 speech names, 24-36 strings per language: Riddler trophy
texts, Arkham City Stories, radio reports and tapes the game does not ship audio events for) have no event in
any bank and stay editable without a sound link.

5
docs/formats/wad.md
View File

@@ -29,8 +29,9 @@ Payload kinds:
- **LOC WADs**: each entry is a group of WEMs, the variations of one line (every language: 10,816 WEMs in
2,464 entries; e.g. `CHR_RadioScanner` holds 91). Each WEM starts at the next 0x800 boundary after
the previous one (zero padding); 3 entries pack two WEMs back to back. Addressed `HASH:i`. The WEMs of an
entry are stored in media-ID order and the languages have different media IDs, so variation `i` is a
different line in each language (75 % of the lines). Which recording is which line comes from the banks
entry are stored in media-ID order and the languages have different media IDs, so variation `i` is
usually a different line in each language (81.2-81.6 % of the WEMs in multi-variation entries, compared with
INT; e.g. `Speech_TAPE_JOKER_1_04` is variation 1 in INT and ITA, 8 in FRA, 6 in DEU, 5 in ESN). Which recording is which line comes from the banks
(`banks.md`, *Lines*), not from positions.
- **SFX WAD**: 79 grouped ambience/effect entries (260 WEMs) plus 4 **stream packs** (below).

8
docs/formats/wem.md
View File

@@ -3,10 +3,14 @@
Derived from the game's WEMs; `ackit verify` / `tests/test_realgame.py` rebuild them byte-for-byte from these rules
alone. Checked on every WEM of the 5 speech WADs: 54,079 of 54,080 are identical; the exception (INT
`FE9963E8:74`) differs only in the informational avg-bytes/s field (one lower than the formula below). In SFX
only the 4 looping music tracks with loop points differ.
626 of 651 are identical; the others are 14 music tracks whose seek table follows another placement (one entry per
1,024-1,103 samples), 5 music tracks whose sample count exceeds what their packets hold, and 6 4-channel
ambience WEMs with a `smpl` (loop) chunk. These rules are not needed to write sounds the game plays (replaced
music, built with the rules here, is heard in game: `ackit verify --ingame`), only to reproduce those originals.
## RIFF layout
`RIFF` size `WAVE`, then `fmt ` (0x42 bytes), optional `cue `/`LIST` (37 files), `data`.
`RIFF` size `WAVE`, then `fmt ` (0x42 bytes), optional `LIST` + `cue ` (659 WEMs), `LIST` + `cue ` + `smpl` (1 WEM),
`cue ` (18 WEMs), `cue ` + `smpl` (3 WEMs), `smpl` (41 WEMs), `data`.
Odd-sized `data` chunks are **not** padded.
### `fmt ` (0x42)

16
docs/reverse-engineering/findings.md
View File

@@ -70,7 +70,7 @@ Conclusion: approach A works end-to-end in-game. Use >= 30 s probes for the intr
expects WAVEFORMATEX order (FL FR FC LFE BL BR) is unverified (identical-channel probes were used).
16. (2026-10-09) The WADs hold only streamed audio. About 15,800 more sounds (footsteps, foley, combat, weapons,
UI, grunts) are in-memory WEMs inside soundbanks: 65 loose `SFX\*.bnk` (1,117) and 3,340 banks embedded in
the `.upk` packages (15,641; many banks copied into several packages). None were in the catalog then (they
the `.upk` packages (14,802; many banks copied into several packages). None were in the catalog then (they
are now: archive BANK); the WAD side was complete. Inventory and layout: `docs/formats/banks.md`.
17. (2026-10-09) Speech lines have **per-language media IDs**: every package carries one BKPK table per language
for the same WAD entry. 396 lines (78-80 per language) have a prefetch head (first N bytes) in banks; each
@@ -95,12 +95,14 @@ Conclusion: approach A works end-to-end in-game. Use >= 30 s probes for the intr
Position names (NAME_i, wemscan) therefore mean different lines per language. Speech sounds are now named
after their speech event (same in every language); the curated ITA lists are mapped through it. Independent
check: the speaker field of the dialogue objects agrees with the mapped labels for 955 / 955 lines.
(Superseded by 22: the identity of a line now comes from the Sound objects, for every recording.)
(Superseded by 22: the identity of a line now comes from the Sound objects, for every recording; measured
that way, 81.2-81.6 % of the WEMs in multi-variation entries are at another position than in INT.)
21. (2026-10-10) In-memory speech banks are localized (language list + one bank per language in one AkBank
export, checked on 215 banks against the WAD languages of their streamed lines); 41 more subtitles are
RSubtitle objects linked through RSubtitleLookup (key = FNV-1 of the line name without `Speech_`). Subtitle
coverage by category is uneven in the game itself: efforts 0 %, combat barks 11 %, cinematics 10 % (cutscene
lines are mostly not reached by speech events and have no dialogue object), story/predator/open world ~90 %+.
(With every version of a line linked, item 22: combat barks 9 %, cinematics 23 %; REPORT.md.)
22. (2026-10-10) **One recording per language, from the game data alone** (`docs/formats/banks.md`, *Lines*).
The BKPK tables of a speech entry sit inside the localized AkBank export, one per declared language; they
@@ -113,6 +115,16 @@ Conclusion: approach A works end-to-end in-game. Use >= 30 s probes for the intr
recording in each of the 5 languages, no conflict; independent checks: 54,055 / 54,055 WAD recordings are
sources in a bank of their own language, 2,101 / 2,101 prefetch heads equal the start of their recording,
every linked subtitle is on the same line in all 11 languages.
23. (2026-10-10, final check) **Whole WAD recordings in banks.** 49 in-memory media (9-11 per language) have the
media ID of a WAD recording of their bank's language and are byte-identical to it (game-over taunts,
`Speech_CH_ZERO_CH0_ENCRYPTION_CARD3_01`, `Speech_SEQUENCE_CH2_RESCUE_DOC_01`, ...): the line's Sound
object names that media, and some bank also stores the whole WEM in memory. They were listed as separate BANK sounds (with the
line's name: the CLI called the name ambiguous), so a replacement of the line left the in-memory copies
old. Now they are bank copies of the WAD recording like prefetch heads (84 copies; 445 recordings with
heads or whole copies, 2,185 copies), written and reverted with the line. Also: 8 in-memory sounds per
catalog had the same `<Bank>_<n>` name as an SFX WAD sound of the same bank (`CHR_RasBossFight_1`,
`SEQ_DemonTrials_1`); they now take `<Bank>_<MEDIA>`, and a line's own name wins over another recording's
position alias (`CHR_RadioScanner#1` in FRA). Every name of every catalog resolves to its own sound.
## Old path -> new path
`scripts/harness.py` -> `ackit verify --ingame` (ackit/verify/ingame.py); `scripts/validate_builder.py` ->

8
docs/user-guide.md
View File

@@ -48,7 +48,7 @@ the same line and are changed and reverted together.
- Click a column header to sort (length, size, status, ...). Ctrl/Shift-click selects several rows.
- Archive **BANK** holds the short sounds stored inside soundbanks (footsteps, foley, combat,
weapons, menu sounds, grunts). Many are named after the game event that plays them (`Play_...`), the
rest after their bank (`CHR_Bane_3`). The editor shows the bank and how many copies the game keeps of
rest after their bank (`CHR_BatclawGrapple_3`). The editor shows the bank and how many copies the game keeps of
the sound; a replacement is written into every copy.
![Browsing by category](img/browse.png)
@@ -92,8 +92,10 @@ The **Subtitles** tab lists the subtitle of every speech line in the selected la
- Search by text or line name, filter by **Character** and **Category** (taken from the linked speech sound),
or show **Modified only**. Select a row to edit its text below and press **Apply** (or Enter).
- Each subtitle replaces the original *in place*, so it can be at most as long as the original (the counter
shows *used / room* characters). Latin-1 subtitles accept Latin-1 characters only (accents are fine);
Russian, Polish, Japanese and Korean subtitles accept any character.
shows *used / room* characters). Most subtitles are stored in the Western European character set
(Windows-1252: accents are fine, other scripts are not); those stored as Unicode (almost all Russian,
Japanese and Korean ones, most Polish ones) accept any character. The editor says so when a character does
not fit.
- **Go to the sound** opens the speech line in the Sounds tab. In the Sounds tab, the editor of a speech line
shows its subtitle in the **Subtitle** column next to the audio: edit both side by side, **Apply** /
**Revert** the subtitle there, or **Show in list** to jump back to the Subtitles tab.

2
pyproject.toml
View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "ackit"
version = "3.2.0"
version = "3.2.1"
description = "Batman: Arkham City (PC) audio toolkit and editor"
readme = "README.md"
requires-python = ">=3.10"

65
tests/test_catalog.py
View File

@@ -1,4 +1,5 @@
"""Catalog: classification rules, naming, character labels, search; the shipped CSVs."""
import collections
import csv
import os
@@ -104,6 +105,57 @@ def test_shipped_csvs_are_consistent():
assert all(l.sound for ls in locs.values() for l in ls if l.kind == "p")
def test_shipped_catalog_cross_references():
"""Every reference between the shipped files points to something that exists, and every line has one
recording and one name per language (copies aside)."""
from ackit import subtitles
voices = ["INT", "FRA", "ITA", "DEU", "ESN"]
rows = {a: read_csv(os.path.join(catalog_dir(), a + ".csv")) for a in voices + ["SFX", "BANK"]}
keys = {s.key: s for r in rows.values() for s in r}
bank = rows["BANK"]
# subtitles -> a voice recording of that language (INT voices for the others)
for f in os.listdir(catalog_dir()):
if f.startswith("SUB_") and f.endswith(".csv"):
lang = f[4:7]
voice = lang if lang in voices else "INT"
for s in subtitles.read_texts(os.path.join(catalog_dir(), f), lang):
for k in filter(None, s.sound.split(";")):
assert k in keys, (f, s.speech, k)
snd = keys[k]
assert snd.archive == voice or (snd.archive == "BANK" and snd.in_language(voice)), (f, k)
# heads of WAD recordings -> a WAD recording, never larger than it; no BANK sound is also a head
locs = banks.read_locations(os.path.join(catalog_dir(), banks.LOCATIONS))
for m, ls in locs.items():
for l in ls:
if l.kind == "p":
assert l.sound in keys and keys[l.sound].archive in voices and l.size <= keys[l.sound].size, l
assert "BANK:%08X" % m not in keys
# speaker labels -> line names that exist in every voice language
names = {L: {s.name for s in rows[L]} for L in voices}
with open(os.path.join(catalog_dir(), "characters.csv"), encoding="utf-8") as f:
for r in csv.DictReader(f):
assert all(r["name"] in names[L] for L in ([r["language"]] if r["language"] else voices)), r
# one recording per language per line (copies: a second place of the same recording, same line)
for L in voices:
per = collections.defaultdict(list)
for s in rows[L] + [b for b in bank if b.line and b.in_language(L)]:
if s.line:
per[s.line].append(s)
for lid, ss in per.items():
assert len({s.sha1 for s in ss}) == 1, (L, lid, [s.key for s in ss]) # copies are identical
assert len([s for s in ss if s.archive == "BANK"]) <= 1, (L, lid)
# the same line has the same name in every language (the second place of a copy keeps its position name)
common = {}
for L in voices:
per = collections.defaultdict(set)
for s in rows[L]:
if s.line:
per[s.line].add(s.name)
for lid, ns in per.items():
common[lid] = common[lid] & ns if lid in common else ns
assert all(common.values()), [k for k, v in common.items() if not v][:5]
@pytest.mark.game
def test_real_game_uses_shipped_catalog(real_game_dir):
g = Game(real_game_dir)
@@ -114,6 +166,17 @@ def test_real_game_uses_shipped_catalog(real_game_dir):
assert all(s.line for s in cat.sounds if s.archive == "ITA" and s.group == "Speech") # every voice line
s = cat.resolve("Speech_TAPE_JOKER_1#4") # position 4 in the ITA entry (the user's list name)
assert s.category == "Interview tapes" and s.size == 44985 and s.name.startswith("Speech_TAPE_JOKER_1_")
assert len(cat.heads) == 396 and sum(len(v) for v in cat.heads.values()) == 2101
# bank copies of WAD recordings: 396 prefetch heads + 49 recordings also stored whole in banks
assert len(cat.heads) == 445 and sum(len(v) for v in cat.heads.values()) == 2185
b = cat.resolve("Play_BatmanDeviceImpact_Wood_1")
assert b.archive == "BANK" and b.group == "SFX"
@pytest.mark.game
def test_every_name_resolves_to_its_sound(real_game_dir):
"""A line named after its INT position (CHR_RadioScanner#1) can equal another recording's position alias
in another language: the name wins, in every language."""
g = Game(real_game_dir)
for lang in g.languages:
cat = Catalog(g, [lang, "SFX", "BANK"])
assert all(cat.resolve(s.name) is s for s in cat.sounds), lang

25
tests/test_lines.py
View File

@@ -37,7 +37,8 @@ def _offsets(wems):
return out
def make(root):
def make(root, whole_copy=False):
"""whole_copy: INT's bank also stores INT's recording of line A whole, in memory (same media ID)."""
cooked = os.path.join(root, "BmGame", "CookedPCConsole")
a = {"INT": synth.wem(0.6, 400), "ITA": synth.wem(0.7, 420)}
b = {"INT": synth.wem(1.1, 600), "ITA": synth.wem(0.9, 640)}
@@ -61,7 +62,8 @@ def make(root):
bank_id = fnv1("Speech_Lines")
for L in ("INT", "ITA"):
blob += _table(ENTRY, rows[L])
blob += synth.bank(bank_id, {}, name="Speech_Lines", events=[(EV_A, A[L]), (EV_B, B)]) + bytes(16)
mem = {A[L]: a[L]} if whole_copy and L == "INT" else {}
blob += synth.bank(bank_id, mem, name="Speech_Lines", events=[(EV_A, A[L]), (EV_B, B)]) + bytes(16)
# the copy entry's table sits in a second localized export (INT and ITA; ITA has no such entry: no rows)
blob2 = bytes(4) + struct.pack("<HHiiii", 9, 0x84, 0, 0, len(lst), 0) + lst
blob2 += _table(COPY_ENTRY, [(A["INT"], a["INT"], 0)]) + bytes(32) + _table(COPY_ENTRY, [(0x999, b"x" * 8, 0)])
@@ -122,3 +124,22 @@ def test_names_and_copies_through_the_workspace(tmp_path):
assert ws.state(s) == ORIGINAL and ws.state(copy) == ORIGINAL and not ws.store.entries
assert hashlib.sha1(open(wad, "rb").read()).hexdigest() == before
assert ws.current_bytes(copy) == a["INT"]
def test_whole_recording_in_a_bank_is_written_with_its_line(tmp_path):
"""A streamed recording can also be stored whole in a bank (same media ID, e.g. the game-over taunts): like
a prefetch head, that copy belongs to the WAD line and is written with it; it is not a sound of its own."""
root = str(tmp_path / "game")
a, _ = make(root, whole_copy=True)
ws = Workspace(root, str(tmp_path / "store"), ["INT", "BANK"])
assert "BANK:%08X" % A["INT"] not in ws.catalog.by_key
s = ws.catalog.by_key["INT:%08X:0" % ENTRY]
(l,) = ws.locations(s)
assert l.kind == banks.PREFETCH_HEAD and l.size == len(a["INT"]) and l.file == "Lines.upk"
pkg = os.path.join(ws.game.cooked, "Lines.upk")
before = hashlib.sha1(open(pkg, "rb").read()).hexdigest()
new = synth.wem(0.3, 900)
ws.apply([(s, new, {})])
assert banks.open_file(pkg).read(l.offset, l.size) == new + bytes(l.size - len(new))
ws.revert([s])
assert hashlib.sha1(open(pkg, "rb").read()).hexdigest() == before

16
tools/build_catalog.py
View File

@@ -128,7 +128,7 @@ def main():
st = lines.stats
print("packages scanned (%.0fs); %d recordings in %d lines, %d of them in every language (%s); "
"%d recordings without a line; %d relation issues"
% (time.time() - t0, st["in_a_line"], st["lines"], st["lines_in_every_language"], "/".join(lines.langs),
% (time.time() - t0, st["recordings"], st["lines"], st["lines_in_every_language"], "/".join(lines.langs),
len(st["without_line"]), len(lines.issues)))
for k, d in lines.issues[:10]:
print(" issue:", k, d)
@@ -187,14 +187,17 @@ def main():
banks.write_locations(os.path.join(out_dir, banks.LOCATIONS), locs)
g = collections.Counter(s.group for s in sounds)
heads = [l for l in locs if l.kind == banks.PREFETCH_HEAD]
played = {m for ms in banks.event_media(sc).values() for m in ms} # reached by a named event
by_event = sum(1 for s in sounds if (s.line not in lines.unnamed if s.line else int(s.target, 16) in played))
index["archives"][banks.ARCHIVE] = {
"fingerprint": banks.game_fingerprint(game.cooked), "sounds": len(sounds), "groups": dict(g),
"named_by_event": by_event, "named_by_bank": len(sounds) - by_event,
"copies": sum(1 for l in locs if l.kind == banks.MEMORY), "files": len(sc.files),
"named_by_event": sum(1 for s in sounds if not s.name.lower().startswith(s.bank.lower() + "_")),
"prefetch_heads": {"lines": len({l.media for l in heads}), "copies": len(heads),
# bank copies of WAD recordings: prefetch heads and whole in-memory copies (catalog._whole_recordings)
"prefetch_heads": {"recordings": len({l.media for l in heads}), "copies": len(heads),
"unmapped": len({l.media for l in heads if not l.sound})},
"issues": ["%s: %s" % i for i in sc.issues]}
print("%s.csv: %d sounds %s, %d copies in %d files, %d prefetch heads"
print("%s.csv: %d sounds %s, %d copies in %d files, %d heads of WAD recordings"
% (banks.ARCHIVE, len(sounds), dict(g), len(locs) - len(heads), len(sc.files), len(heads)))
# subtitles (dialogue objects in the packages), one CSV per language + every stored copy
@@ -210,9 +213,12 @@ def main():
"lines": {lang: len(rows) for lang, rows in sorted(texts.items())},
"speech_lines": len({s.speech for s in any_lang}),
"linked": {lang: sum(1 for s in rows if s.sound) for lang, rows in sorted(texts.items())},
"copies": len(sublocs), "issues": ["%s: %s" % i for i in sub_issues]}
"copies": len(sublocs), **sc.subtitle_objects, "issues": ["%s: %s" % i for i in sub_issues]}
wad_lines = {lid for lid, per in lines.members.items() if any(not k.startswith("BANK:") for k in per.values())}
index["archives"]["LINES"] = {
"languages": lines.langs, "recordings": st["recordings"], "lines": st["lines"],
"wad_lines": len(wad_lines), "wad_lines_named_by_event": len(wad_lines - lines.unnamed),
"wad_lines_named_after_int_recording": len(wad_lines & lines.unnamed),
"lines_in_every_language": st["lines_in_every_language"],
"lines_missing_a_language": st["lines_missing_a_language"], "without_line": st["without_line"],
"issues": ["%s: %s" % i for i in lines.issues]}

545
tools/check_docs.py Normal file
View File

@@ -0,0 +1,545 @@
"""Check every figure the documentation states against the data it describes (read-only).
python tools/check_docs.py [--game DIR] [--scan] [--wems]
The figures are recomputed, never copied: from the shipped catalog (`data/catalog/`, built from the original
game files) and, when the game is installed, its menu files; with --scan also from a fresh scan of every
package (bank survey: blobs, source kinds, bank names; ~3 minutes) and from `ackit verify --catalog`'s facts
(relations re-derived from the game; ~6 minutes more); with --wems from rebuilding every WEM of the speech and
SFX WADs from the format rules (~10 minutes). Each claim is the exact text a document must contain
(line breaks and repeated spaces ignored). Exit code 1 when a document states anything else.
History logs (NOTES.md, findings.md, docs/history/) record what was true on their date and are not checked.
"""
import argparse
import collections
import csv
import glob
import gzip
import os
import re
import sys
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, ROOT)
from ackit import banks, settings, subtitles # noqa: E402
from ackit.catalog import read_csv # noqa: E402
from ackit.game import Game, default_game_dir # noqa: E402
from ackit.paths import catalog_dir # noqa: E402
VOICES = ["INT", "FRA", "ITA", "DEU", "ESN"]
CLAIMS = [] # (document, text, what it is checked against)
def n(x):
return "{:,}".format(x)
def rng(values, fmt=n):
lo, hi = min(values), max(values)
return fmt(lo) if lo == hi else "%s-%s" % (fmt(lo), fmt(hi))
def pct(a, b):
return int(round(100.0 * a / b))
def claim(doc, text, why=""):
CLAIMS.append((doc, text, why))
# --- figures from the shipped catalog --------------------------------------------------------------
def catalog_claims(cat):
rows = {a: read_csv(os.path.join(cat, a + ".csv")) for a in VOICES + ["SFX", "BANK"]}
bank, sfx = rows["BANK"], rows["SFX"]
wad_n = {len(rows[L]) for L in VOICES}
assert len(wad_n) == 1, "WADs differ in size: %s" % wad_n
W = wad_n.pop()
claim("REPORT.md", "(5 languages x %s), the SFX WAD (%s)" % (n(W), n(len(sfx))), "catalog rows")
claim("REPORT.md", "the **%s in-memory sounds**" % n(len(bank)), "BANK.csv rows")
claim("docs/formats/banks.md", "so there are **%s in-memory sounds**" % n(len(bank)), "BANK.csv rows")
# languages of the in-memory sounds
neutral = [s for s in bank if not s.language]
single = collections.Counter(s.language for s in bank if s.language and ";" not in s.language)
multi = [s for s in bank if ";" in s.language]
multi_c = collections.Counter(s.language for s in multi)
claim("REPORT.md", "%s shared by all languages, ~%s localized speech lines per voice language, %d shared by "
"several" % (n(len(neutral)), n(int(round(sum(single.values()) / len(single), -1))), len(multi)))
claim("docs/formats/banks.md", "%s sounds are language-neutral, %s belong to one voice language, %d to several"
% (n(len(neutral)), rng(single.values()), len(multi)))
a4 = multi_c.get("ESN;FRA;INT;ITA", 0)
a3 = multi_c.get("FRA;INT;ITA", 0)
claim("docs/formats/banks.md", "(%d shared by INT/FRA/ITA/ESN while DEU has its own, %d by INT/FRA/ITA)" % (a4, a3))
claim("docs/formats/banks.md", "everywhere; %d of these are voice lines" % sum(1 for s in neutral if s.line))
claim("docs/formats/banks.md", "stored once for every language (%d sounds)"
% sum(1 for s in neutral if s.bank.startswith("Speech")))
claim("docs/formats/banks.md", "lines** (about %s per language" % n(int(round(sum(single.values()) / len(single), -1))))
ch = collections.Counter(s.channels for s in bank)
claim("docs/formats/banks.md", "of all %s in-memory sounds %s are mono, %s stereo, %d have 3-6 channels"
% (n(len(bank)), n(ch[1]), n(ch[2]), sum(v for k, v in ch.items() if k >= 3)))
# copies and heads
with gzip.open(os.path.join(cat, "BANK_locations.csv.gz"), "rt", newline="") as f:
loc = list(csv.DictReader(f))
m_rows = [r for r in loc if r["kind"] == "m"]
p_rows = [r for r in loc if r["kind"] == "p"]
keys = {s.key: s for L in VOICES for s in rows[L]}
whole = [r for r in p_rows if int(r["size"]) >= keys[r["sound"]].size]
pref = [r for r in p_rows if r not in whole]
whole_rec = {r["sound"] for r in whole}
pref_rec = {r["sound"] for r in pref}
pref_lines = {keys[k].line for k in pref_rec}
all_lines = {keys[r["sound"]].line for r in p_rows}
per_lang_pref = collections.Counter(k.split(":")[0] for k in pref_rec)
per_lang_whole = collections.Counter(k.split(":")[0] for k in whole_rec)
claim("REPORT.md", "copy (%s copies). %s WAD voice recordings (%d lines) also have bank copies (%d prefetch heads, %d "
"whole copies; %s copies)" % (n(len(m_rows)), n(len(pref_rec | whole_rec)), len(all_lines), len(pref_rec),
len(whole_rec), n(len(p_rows))))
claim("docs/formats/banks.md", "%d of them are **prefetch heads** of streamed WAD sounds and %d are **whole copies**"
% (len({r["media"] for r in pref}), len({r["media"] for r in whole})))
claim("docs/formats/banks.md", "%d WAD sounds have such heads (%s copies)" % (len(pref_rec), n(len(pref))))
claim("docs/formats/banks.md", "%d more WAD recordings (%s per language" % (len(whole_rec), rng(per_lang_whole.values())))
claim("docs/formats/banks.md", "byte-identical to the WAD WEM (%d copies)" % len(whole))
claim("docs/formats/banks.md", "Heads and whole copies together: %d recordings, %s copies"
% (len(pref_rec | whole_rec), n(len(p_rows))))
claim("docs/formats/banks.md", "the %d lines with a prefetch head (%d recordings over the 5 languages"
% (len(pref_lines), len(pref_rec)))
claim("docs/formats/banks.md", "A WAD recording that a bank also stores whole (%d)" % len(whole_rec))
claim("docs/decisions.md", "%d voice recordings (%d lines, %s recordings per language) have their first bytes"
% (len(pref_rec), len(pref_lines), rng(per_lang_pref.values(), str)))
claim("docs/decisions.md", "and %d more are stored whole in banks" % len(whole_rec))
places = len(loc)
media_all = {r["media"] for r in loc}
loose = {r["media"] for r in loc if r["file"].startswith("SFX/")}
pk = {r["media"] for r in loc if not r["file"].startswith("SFX/")}
claim("docs/formats/banks.md", "The two sets share %d media IDs: %s unique media IDs in %s places"
% (len(loose & pk), n(len(media_all)), n(places)))
claim("docs/formats/banks.md", "+ `TST_Prep_Media`) | %d | %s |"
% (len({r["file"] for r in loc if r["file"].startswith("SFX/")}), n(len(loose))))
claim("docs/formats/banks.md", "packages) | 3,340 | %s |" % n(len(pk)))
claim("REPORT.md", "in %d loose `.bnk` files and %d `.upk` packages"
% (len({r["file"] for r in loc if r["file"].startswith("SFX/")}), len({r["file"] for r in loc if r["file"].endswith(".upk")})))
assert len(media_all) == len(bank) + len(pref_rec | whole_rec)
# lines
lines_all = {s.line for L in VOICES for s in rows[L] if s.line} | {s.line for s in bank if s.line}
wad_lines = {L: {s.line for s in rows[L] if s.line} for L in VOICES}
assert len({len(v) for v in wad_lines.values()}) == 1
WL = len(wad_lines["INT"])
copies = {L: sum(1 for s in rows[L] if s.line) - len(wad_lines[L]) for L in VOICES}
recs = sum(len(wad_lines[L]) for L in VOICES) + len({(s.target) for s in bank if s.line})
for doc, text in [("REPORT.md", "%s lines (WAD + in-memory) with exactly one recording in each of the 5 voice languages"),
("docs/decisions.md", "recording per language (%s lines, all 5 languages)"),
("docs/formats/banks.md", "recordings that are sources of the same Sound objects: %s lines")]:
claim(doc, text % n(len(lines_all)), "distinct line ids")
claim("docs/formats/banks.md", "in each of the 5 voice languages (%s recordings)" % n(recs))
claim("REPORT.md", "recordings stored twice (%s per language) are written together" % rng(copies.values(), str))
claim("docs/formats/banks.md", "- %s recordings per language are stored twice" % rng(copies.values(), str).capitalize())
claim("docs/categories.md", "Coverage of the %s WAD voice lines" % n(WL))
claim("docs/decisions.md", "of the %s WAD voice lines" % n(WL))
# positions differ between languages
it = {s.target: s.line for s in rows["INT"] if ":" in s.target}
diff = []
for L in ["FRA", "ITA", "DEU", "ESN"]:
other = {s.target: s.line for s in rows[L] if ":" in s.target}
diff.append(100.0 * sum(1 for t in it if other.get(t) != it[t]) / len(it))
span = "%.1f-%.1f %%" % (min(diff), max(diff))
claim("docs/formats/wad.md", "(%s of the WEMs in multi-variation entries, compared with INT" % span)
claim("docs/decisions.md", "%s of the WEMs in multi-variation entries are at another position than in INT" % span)
pos = {}
for L in VOICES:
s = next(s for s in rows[L] if s.name == "Speech_TAPE_JOKER_1_04")
pos[L] = int(s.target.split(":")[1])
claim("docs/formats/wad.md", "`Speech_TAPE_JOKER_1_04` is variation %d in INT and ITA, %d in FRA, %d in DEU, %d in ESN"
% (pos["INT"], pos["FRA"], pos["DEU"], pos["ESN"]) if pos["INT"] == pos["ITA"] else "(INT != ITA)")
ents = collections.Counter(s.target.split(":")[0] for s in rows["INT"])
claim("docs/formats/wad.md", "(every language: %s WEMs in %s entries" % (n(W), n(len(ents))))
radio = [s for s in rows["INT"] if s.aliases and any(a.startswith("CHR_RadioScanner#") for a in s.names())
or s.name.startswith("CHR_RadioScanner#")]
claim("docs/formats/wad.md", "e.g. `CHR_RadioScanner` holds %d" % len({s.target for s in radio}))
# SFX WAD
packs = collections.Counter(s.target.split("/")[0] for s in sfx if "/" in s.target)
grouped = [s for s in sfx if "/" not in s.target]
claim("docs/formats/wad.md", "**SFX WAD**: %d grouped ambience/effect entries (%d WEMs) plus %d **stream packs**"
% (len({s.target.split(":")[0] for s in grouped}), len(grouped), len(packs)))
claim("docs/formats/wad.md", "`EDF036D6` (Music, %d WEMs" % packs["EDF036D6"])
claim("docs/formats/wad.md", "`E8F980BE` (Music_PDLC, %d), `57572DAC` (Global, %d), `FCA4F778` (%d)"
% (packs["E8F980BE"], packs["57572DAC"], packs["FCA4F778"]))
# categories
cats = [collections.Counter(s.category for s in rows[L]) for L in VOICES]
assert all(c == cats[0] for c in cats), "category counts differ between languages"
order = ["Story missions", "Predator chatter", "Character lines", "Open-world chatter", "Side missions",
"Combat barks", "Efforts & emotes", "Interview tapes", "Riddler", "Radio & TV", "Gameplay & hints", "DLC",
"Boss fights", "Cinematics", "Reactions", "Interrogations"]
g = collections.Counter(s.group for s in sfx)
bc = collections.Counter(s.category for s in bank).most_common(8)
claim("docs/categories.md", "Counts per voice language (identical in all 5): " + ", ".join(
"%s %d" % (c, cats[0][c]) for c in order) + ". SFX archive: " + ", ".join(
"%s %d" % (k, g[k]) for k in ["Music", "Ambience", "SFX", "Unknown"]) + ". BANK (all languages): " + ", ".join(
"%s %d" % kv for kv in bc) + ", ...")
# characters (by line)
first = {}
for s in rows["INT"]:
first.setdefault(s.line, s)
wc = sum(1 for s in first.values() if s.character)
inf = sum(1 for s in first.values() if not s.character and s.characters_inferred)
nei = [s for s in first.values() if not s.character and not s.characters_inferred]
top = collections.Counter(s.category for s in nei).most_common(4)
assert all(sum(1 for s in rows[L] if s.character) == sum(1 for s in rows["INT"] if s.character) for L in VOICES)
claim("docs/categories.md", "(every voice language, the same lines): %s with a verified speaker, %s more with inferred "
"characters, %s with neither (%s, ...)" % (n(wc), n(inf), n(len(nei)), ", ".join(
"%s %d" % (c.lower().replace("radio & tv", "radio & TV"), k) for c, k in top)))
claim("docs/categories.md", "A hint for the ~%s lines without a verified speaker" % n(int(round(WL - wc, -2))))
claim("REPORT.md", "speaker labels (%s in every language)" % n(wc))
# subtitles
langs = sorted(f[4:7] for f in os.listdir(cat) if f.startswith("SUB_") and f.endswith(".csv"))
texts = {L: subtitles.read_texts(os.path.join(cat, "SUB_%s.csv" % L), L) for L in langs}
total = sum(len(t) for t in texts.values())
linked = sum(1 for t in texts.values() for s in t if s.sound)
i_tot, i_l = len(texts["INT"]), sum(1 for s in texts["INT"] if s.sound)
for doc in ("REPORT.md", "docs/formats/subtitles.md"):
claim(doc, "%s of %s strings in %d languages (INT %s" % (n(linked), n(total), len(langs), n(i_l))
if doc == "REPORT.md" else "Linked: %s of %s strings in the %d languages (INT %s / %s)"
% (n(linked), n(total), len(langs), n(i_l), n(i_tot)))
claim("REPORT.md", "(INT %s of %s)" % (n(i_l), n(i_tot)))
unl = {L: {s.speech for s in t if not s.sound} for L, t in texts.items()}
unl_n = {len(v) for v in unl.values()}
assert len(unl_n) == 1, unl_n
un_strings = [sum(1 for s in t if not s.sound) for t in texts.values()]
claim("REPORT.md", "%d speech names (%s strings per language, INT %d" % (unl_n.pop(), rng(un_strings),
i_tot - i_l))
claim("docs/formats/subtitles.md", "(%d speech names, %s strings per language" % (len(unl["INT"]), rng(un_strings)))
two = {s.speech for s in texts["INT"] if ";" in s.sound}
claim("docs/formats/subtitles.md", "(%d subtitled events" % len(two))
one = []
for L, t in texts.items():
c = collections.Counter(s.speech for s in t)
one.append(100.0 * sum(1 for v in c.values() if v == 1) / len(c))
claim("docs/formats/subtitles.md", "%s subtitle strings per language: %d-%d %% of the subtitles have one line (JPN: all)"
% (rng([len(t) for t in texts.values()]), int(min(one)), int(round(max(x for x in one if x < 100)))))
enc = {L: collections.Counter(s.enc for s in t) for L, t in texts.items()}
pol = pct(enc["POL"]["w"], len(texts["POL"]))
west = [enc[L]["w"] for L in langs if L not in ("RUS", "JPN", "KOR", "POL")]
claim("docs/formats/subtitles.md", "%d %% of POL are UTF-16; the other languages have %s UTF-16 strings each"
% (pol, rng(west)))
with gzip.open(os.path.join(cat, subtitles.LOCATIONS), "rt", newline="") as f:
sublocs = sum(1 for _ in csv.DictReader(f))
claim("REPORT.md", "Edited in place in every copy (%s copies)" % n(sublocs))
claim("docs/formats/subtitles.md", "%s stored copies" % n(sublocs))
claim("docs/user-guide.md", "About %d subtitles per language have no sound" % int(round(sum(un_strings) / len(un_strings), -1)))
# subtitle coverage of the INT voice lines
lk = {k for s in texts["INT"] for k in s.sound.split(";") if k}
cov, tot = collections.Counter(), collections.Counter()
for s in rows["INT"] + [b for b in bank if b.line and "INT" in b.language.split(";")]:
if s.line:
tot[s.category] += 1
cov[s.category] += s.key in lk
p = {c: pct(cov[c], tot[c]) for c in tot}
claim("REPORT.md", "few for combat barks (%d %%), cinematics and boss fights (%d %%;"
% (p["Combat barks"], p["Cinematics"]) if p["Cinematics"] == p["Boss fights"] else "(cinematics != boss)")
hi = [p[c] for c in ("Story missions", "Predator chatter", "Character lines", "Open-world chatter")]
claim("REPORT.md", "%d-%d %% for story, predator, character and open-world lines" % (min(hi), max(hi)))
claim("REPORT.md", "(INT voice lines with a subtitle): none for efforts/grunts" if p["Efforts & emotes"] == 0 else "!")
# what the editor loads (Italian on the test PC: the smoke test)
ita = len(rows["ITA"]) + len(sfx) + sum(1 for s in bank if s.in_language("ITA"))
claim("REPORT.md", "%s sounds loaded, encode + inject, %s subtitles" % (n(ita), n(len(texts["ITA"]))))
claim("README.md", "~%s sounds of one language" % n(int(round(ita, -3))))
return rows
# --- figures recorded by the catalog build (index.json: they need the full scan) ---------------------
def index_claims(cat):
import json
ix = json.load(open(os.path.join(cat, "index.json"), encoding="utf-8"))
b, ln, sub = ix["BANK"], ix["LINES"], ix["SUB"]
claim("REPORT.md", "Names: %s BANK sounds are named after the Wwise event that plays them (`Play_...`, `Speech_...`), "
"the other %s after their bank" % (n(b["named_by_event"]), n(b["named_by_bank"])))
claim("REPORT.md", "**Unnamed sounds**: %s BANK sounds are reached by no event" % n(b["named_by_bank"]))
claim("docs/decisions.md", "sounds (%s of %s, voice lines included); the other %s are numbered per bank"
% (n(b["named_by_event"]), n(b["sounds"]), n(b["named_by_bank"])))
claim("docs/decisions.md", "the most specific speech event that plays it (%s of the %s WAD voice lines)"
% (n(ln["wad_lines_named_by_event"]), n(ln["wad_lines"])))
claim("docs/decisions.md", "the other %s keep the name of their English (INT) recording's position"
% n(ln["wad_lines_named_after_int_recording"]))
claim("REPORT.md", "- %s voice lines are played by no event whose name the game stores"
% n(ln["wad_lines_named_after_int_recording"]))
claim("docs/user-guide.md", "About %d %% of the lines are named by no game event"
% pct(ln["wad_lines_named_after_int_recording"], ln["wad_lines"]))
claim("docs/formats/subtitles.md", "%s dialogue objects in the packages for %s speech lines"
% (n(sub["dialogue_objects"]), n(sub["dialogue_lines"])))
claim("docs/formats/subtitles.md", "The sizes add up exactly in all %s objects" % n(sub["dialogue_objects"]))
claim("docs/formats/subtitles.md", "plus %d scripted subtitles" % sub["scripted_objects"])
claim("docs/formats/subtitles.md", "%d more subtitles (radio and encryption-card lines) are `RSubtitle` objects"
% sub["scripted_objects"])
claim("REPORT.md", "(%s speech lines x %d languages) and" % (n(sub["dialogue_lines"]), len(sub["lines"])))
claim("REPORT.md", "in %d scripted `RSubtitle` objects" % sub["scripted_objects"])
claim("docs/formats/banks.md", "(in %s of the 2,650 packages)" % n(b["files"] - 67)) # 67 loose .bnk with banks
# --- the game's own files ----------------------------------------------------------------------------
def menu_claims(game):
from ackit import loctext
root = os.path.join(game.root, "BmGame", "Localization")
counts = {L: len(loctext.load(root, L)) for L in loctext.languages(root)}
files = {L: len(loctext.files(root, L)) for L in loctext.languages(root)}
approx = n(int(round(sum(counts.values()) / len(counts), -3)))
claim("docs/formats/text.md", "~%d files and ~%s strings each" % (collections.Counter(files.values()).most_common(1)[0][0], approx))
claim("REPORT.md", "~%s strings per language" % approx)
claim("docs/user-guide.md", "of the selected language (about\n%s)" % approx)
claim("REPORT.md", "%s menu texts loaded" % n(counts["ITA"]))
def scan_claims(game):
"""Bank survey from a fresh scan of every package, and the relation facts of `ackit verify --catalog`."""
from ackit.bnk import find_banks, Bank, BankError, IN_MEMORY, STREAMED, PREFETCH
blobs = collections.Counter()
media_sets = collections.defaultdict(set)
kinds = collections.Counter()
in_pkg = set()
def visit(rel, pkg, start, data):
for bk in find_banks(data):
blobs[bk.id] += 1
media_sets[bk.id].add(frozenset(m.id for m in bk.media()))
in_pkg.add(bk.id)
own = {m.id for m in bk.media()}
for s in bk.sources():
if s.stream == IN_MEMORY:
kinds["memory own" if s.media in own else "memory other"] += 1
elif s.stream == STREAMED:
kinds["streamed"] += 1
elif s.stream == PREFETCH:
kinds["prefetch"] += 1
sc = banks.scan(game.cooked, visit=visit)
for path in glob.glob(os.path.join(game.cooked, "SFX", "*.bnk")): # loose banks: source kinds too
try:
bk = Bank(open(path, "rb").read(), 0)
except BankError:
continue
own = {m.id for m in bk.media()}
for s in bk.sources():
if s.stream == IN_MEMORY:
kinds["memory own" if s.media in own else "memory other"] += 1
elif s.stream == STREAMED:
kinds["streamed"] += 1
elif s.stream == PREFETCH:
kinds["prefetch"] += 1
names = {}
cooked_list = [l.strip() for l in open(os.path.join(game.cooked, "CookedBanks.txt"), encoding="latin-1") if l.strip()]
from ackit.bnk import fnv1
for nm in cooked_list:
names[fnv1(nm)] = nm
loose_ids = {int(os.path.splitext(os.path.basename(p))[0], 16) for p in glob.glob(os.path.join(game.cooked, "SFX", "*.bnk"))
if re.fullmatch(r"[0-9A-Fa-f]{8}", os.path.splitext(os.path.basename(p))[0])}
embedded = [nm for h, nm in names.items() if h in in_pkg]
loose = sorted(nm for h, nm in names.items() if h in loose_ids and h not in in_pkg)
nowhere = sorted(nm for h, nm in names.items() if h not in in_pkg and h not in loose_ids)
claim("docs/formats/banks.md", "lists all %s cooked bank names" % n(len(cooked_list)))
both = sorted(nm for h, nm in names.items() if h in loose_ids and h in in_pkg)
claim("docs/formats/banks.md", "%s of them are embedded in packages (%s also as loose `.bnk` files), %d are only loose "
"`.bnk` files (%s); %s were found nowhere"
% (n(len(embedded)), ", ".join("`%s`" % x for x in both), len(loose), ", ".join("`%s`" % x for x in loose),
", ".join("`%s`" % x for x in nowhere[:-1]) + " and `%s`" % nowhere[-1]))
diff_sets = sum(1 for b, v in media_sets.items() if len(v) > 1)
claim("docs/formats/banks.md", "Bank blobs in packages: %s for %s bank IDs" % (n(sum(blobs.values())), n(len(blobs))))
by_name = {names.get(b, ""): c for b, c in blobs.items()}
claim("docs/formats/banks.md", "`CHR_CatwomanFoleySuit` %d times, `CHR_BatmanFoleyBelt` %d). %d bank IDs"
% (by_name.get("CHR_CatwomanFoleySuit", 0), by_name.get("CHR_BatmanFoleyBelt", 0), diff_sets))
claim("docs/formats/banks.md", "| banks embedded in `.upk` packages (in %s of the %s packages) | %s |"
% (n(sum(1 for f in sc.files if f.lower().endswith(".upk"))), n(len(glob.glob(os.path.join(game.cooked, "*.upk")))),
n(len(blobs))))
claim("docs/formats/banks.md", "Source kinds over all bank copies: %s in-memory sources of the bank's own media; %s "
"in-memory sources with bank id 0" % (n(kinds["memory own"]), n(kinds["memory other"])))
claim("docs/formats/banks.md", "(loaded by another bank); %s streamed sources; %s **prefetch** sources"
% (n(kinds["streamed"]), n(kinds["prefetch"])))
refs = {h for h, t in sc.packs.items() if t}
claim("docs/formats/banks.md", "the BKPK tables in the packages reference %s WAD entries" % n(len(refs)))
per_bank = collections.defaultdict(set)
for m, e in sc.media.items():
for b in e["banks"]:
per_bank[b].add(m)
big = sorted(per_bank.items(), key=lambda kv: -len(kv[1]))[:7]
nm = [(sc.bank_names.get(b, "%08X" % b), len(v)) for b, v in big]
claim("docs/formats/banks.md", "Largest: `%s` %d WEMs, " % nm[0] + ", ".join("`%s` %d" % x for x in nm[1:]))
pkm = {l.media for l in sc.locations if not l.file.startswith("SFX/")}
pre = collections.Counter()
for m in pkm:
for p_ in {sc.bank_names.get(b, "%08X" % b).split("_")[0] for b in sc.media[m]["banks"]}:
pre[p_] += 1
top = [k for k, v in pre.most_common() if v >= 30]
text = ", ".join("`%s` %s" % (k, n(pre[k])) if k != "CHR" else "`CHR`\n(character foley, combat, bosses) %s" % n(pre[k])
for k in top)
claim("docs/formats/banks.md", "(a media held by banks of two prefixes counts for both): " + text + ", others < 30")
# media IDs used by several languages' WAD tables (each at its own position)
langs_of = collections.defaultdict(set)
for h, tables in sc.packs.items():
for L, t in tables.items():
if L:
for m in t:
langs_of[m].add(L)
claim("docs/formats/banks.md", "%d media IDs are used by several\n languages" % sum(1 for v in langs_of.values() if len(v) > 1))
# relations, re-derived from the game by the catalog check
from ackit.verify import catalogcheck
rep = catalogcheck.run(game, log=lambda *a: None)
facts = "\n".join(rep.facts)
print("verify --catalog:", "catalog OK" if rep.ok else "%d issue(s)" % len(rep.issues))
def fact(pattern):
m = re.search(pattern, facts)
if not m:
raise SystemExit("fact not found: " + pattern)
return [int(x) for x in m.groups()]
src = fact(r"own language's bank: (\d+) of (\d+)")
claim("REPORT.md", "own language (%s / %s)" % (n(src[0]), n(src[1])))
claim("docs/formats/banks.md", "of its own language (%s of %s)" % (n(src[0]), n(src[1])))
hd = fact(r"whole bank copies: (\d+) of (\d+)")
claim("REPORT.md", "is the start of its recording (%s / %s)" % (n(hd[0]), n(hd[1])))
claim("docs/formats/banks.md", "line gives it (%s of %s)" % (n(hd[0]), n(hd[1])) if True else "")
rec, lns, full = fact(r"(\d+) recordings \(WAD .*?\), (\d+) lines, (\d+) of them")
claim("docs/formats/banks.md", "(%s recordings)" % n(rec))
ln15 = fact(r"(\d+) of (\d+) lines within a factor 1.5")
claim("docs/formats/banks.md", "(%s of %s\n within a factor 1.5" % (n(ln15[0]), n(ln15[1])))
sp = fact(r"speakers: (\d+) labelled voice lines agree .*?, (\d+) listed as Unknown .*?, (\d+) differ")
claim("REPORT.md", "%s agree, %d differ" % (n(sp[0]), sp[2]))
claim("docs/categories.md", "%s labelled\n lines agree" % n(sp[0]))
claim("docs/categories.md", "%d listed\n as Unknown are named there, %d differ" % (sp[1], sp[2]))
if not rep.ok:
claim("(game)", "catalog OK", "ackit verify --catalog")
def _rebuild_language(lang):
"""Keys of the WEMs of one speech WAD that the builder does not reproduce byte for byte."""
from ackit.inject import read_target
from ackit.verify.selfcheck import rebuild_identical
from ackit.wad import Wad
game = Game(default_game_dir(settings.load()))
path = game.wad(lang)
w = Wad(path)
from ackit.wem import Wem
bad, count, extra = [], 0, collections.Counter()
for s in read_csv(os.path.join(catalog_dir(), lang + ".csv")):
count += 1
r = read_target(path, s.target, w)
tags = {c.tag for c in Wem.parse(r).chunks} - {b"fmt ", b"data"}
if tags:
extra["/".join(sorted(t.decode() for t in tags))] += 1
try:
ok = rebuild_identical(r)
except Exception: # noqa: BLE001 - a WEM the builder cannot parse is a difference too
ok = False
if not ok:
bad.append(s.key)
return count, bad, extra
def wem_claims():
"""Every WEM of the 5 speech WADs and of the SFX WAD rebuilt from the format rules (docs/formats/wem.md;
~10 minutes)."""
import multiprocessing
from ackit.inject import read_target
from ackit.wem import Wem
with multiprocessing.Pool(len(VOICES) + 1) as p:
res = p.map(_rebuild_language, VOICES + ["SFX"])
extra = collections.Counter()
for _, _, e in res:
extra.update(e)
sfx_count, sfx_bad, _ = res.pop()
game = Game(default_game_dir(settings.load()))
sfx = {s.key: s for s in read_csv(os.path.join(catalog_dir(), "SFX.csv"))}
kinds = collections.Counter()
per = [] # samples per seek-table entry of the music tracks
from ackit.verify.selfcheck import rebuild_identical
for k in sfx_bad:
s = sfx[k]
r = read_target(game.wad("SFX"), s.target)
m = Wem.parse(r)
try:
rebuild_identical(r)
err = False
except ValueError: # "inconsistent total_frames": more samples than the packets hold
err = True
if any(c.tag == b"smpl" for c in m.chunks):
kinds["smpl"] += 1
elif err:
kinds["frames"] += 1
elif s.group == "Music" and m.vorb["seek_table_size"]:
kinds["seek"] += 1
per.append(m.vorb["total_frames"] / (m.vorb["seek_table_size"] // 4))
else:
kinds["other"] += 1
claim("REPORT.md", "%d SFX originals (music seek tables and sample counts, looping ambience)" % len(sfx_bad))
claim("docs/formats/wem.md", "In SFX %s of %s are identical; the others are %d music tracks whose seek table follows "
"another placement (one entry per %s samples), %d music tracks whose sample count exceeds what their "
"packets hold, and %d 4-channel ambience WEMs with a `smpl` (loop) chunk%s"
% (n(sfx_count - len(sfx_bad)), n(sfx_count), kinds["seek"], rng([int(x) for x in per]), kinds["frames"],
kinds["smpl"], "" if not kinds["other"] else " (+%d other)" % kinds["other"]))
total = sum(c for c, _, _ in res)
bad = [k for _, b, _ in res for k in b]
claim("docs/formats/wem.md", "`RIFF` size `WAVE`, then `fmt ` (0x42 bytes), optional %s, `data`" % ", ".join(
"%s (%s %s)" % (" + ".join("`%s`" % t for t in k.split("/")), n(v), "WEM" if v == 1 else "WEMs")
for k, v in sorted(extra.items())))
exceptions = ", ".join("%s `%s`" % tuple(k.split(":", 1)) for k in bad) # e.g. INT `FE9963E8:74`
claim("docs/formats/wem.md", "Checked on every WEM of the 5 speech WADs: %s of %s are identical; the exception (%s)"
% (n(total - len(bad)), n(total), exceptions))
def test_claims():
import subprocess
out = subprocess.run([sys.executable, "-m", "pytest", "--collect-only", "-q", "-p", "no:cacheprovider"],
cwd=ROOT, capture_output=True, text=True).stdout
m = re.search(r"(\d+) tests? collected", out)
claim("REPORT.md", "**Tests**: %s automated tests" % (m.group(1) if m else "?"), "pytest --collect-only")
def norm(t):
return re.sub(r"\s+", " ", t).strip()
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--game")
ap.add_argument("--scan", action="store_true", help="also scan every package and run the catalog check")
ap.add_argument("--wems", action="store_true", help="also rebuild every speech and SFX WEM from the rules")
a = ap.parse_args()
cat = catalog_dir()
catalog_claims(cat)
index_claims(cat)
test_claims()
gd = a.game or default_game_dir(settings.load())
game = Game(gd) if gd else None
if game:
menu_claims(game)
if a.scan:
if not game:
raise SystemExit("--scan needs the game")
scan_claims(game)
if a.wems:
if not game:
raise SystemExit("--wems needs the game")
wem_claims()
docs = {}
bad = 0
for doc, text, why in CLAIMS:
if doc not in docs:
path = os.path.join(ROOT, doc)
docs[doc] = norm(open(path, encoding="utf-8").read()) if os.path.exists(path) else ""
if norm(text) not in docs[doc]:
bad += 1
print("MISMATCH %s: expected \"%s\"%s" % (doc, norm(text), (" (" + why + ")") if why else ""))
print("%d claims checked in %d documents%s: %s" % (len(CLAIMS), len(docs), "" if a.scan else " (without --scan)",
"all match" if not bad else "%d mismatch(es)" % bad))
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())

13
tools/e2e_features.py
View File

@@ -8,7 +8,8 @@ For every check: the SHA-1 of each touched file is taken, the change is applied
of the editor and the CLI), read back from the game files, reverted, and the files must be byte-identical again.
1. in-memory sound with the most copies (every copy written; once, it is shared by all languages)
2. localized in-memory speech line of the language's voices
3. speech line of the language's voices with prefetch heads (WAD + every head copy)
3. speech line of the language's voices with prefetch heads (WAD + every head copy), and one that a bank
also stores whole (WAD + every whole copy)
4. subtitle with several copies (RDialogueEvent), and a scripted subtitle (RSubtitle, Startup.upk)
5. menu text
6. sound + subtitle in the same package, reverted one at a time
@@ -82,7 +83,7 @@ def sound_round_trip(ws, s, label):
for l, b in copies_of(ws, s):
if l.kind == banks.MEMORY:
ok &= b == wem + bytes(l.size - len(wem))
else: # prefetch head of a WAD line (stored zero-padded to its size)
else: # head or whole copy of a WAD line (zero-padded to its size)
full = wem + bytes(max(0, ws.store.get(s.key)["orig_size"] - len(wem)))
ok &= b == (full + bytes(l.size))[:l.size]
ws.revert([s])
@@ -140,7 +141,13 @@ def run_checks(game, lang, voices, td, ws, first):
heads = [ws.catalog.by_key[k] for k in ws.catalog.heads if k in ws.catalog.by_key
and ws.catalog.by_key[k].archive == voices]
sound_round_trip(ws, max(heads, key=lambda s: len(ws.locations(s))),
"3 %s speech line with prefetch heads" % lang)
"3a %s speech line with prefetch heads" % lang)
whole = [s for s in heads if any(l.size >= s.size for l in ws.locations(s))]
if whole:
sound_round_trip(ws, max(whole, key=lambda s: len(ws.locations(s))),
"3b %s speech line also stored whole in a bank" % lang)
else:
check("3b %s speech line also stored whole in a bank" % lang, False, "none found")
tx = ws.texts
subs = tx.subtitles(lang)