diff --git a/CLAUDE.md b/CLAUDE.md index f0ceddc..6ca1267 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,6 +18,9 @@ Batman: Arkham City (PC) audio toolkit + editor. Read `NOTES.md` first (status, stops if a sound differs from the current catalog). Voice lines are identified by the game's Wwise Sound objects (`ackit/lines.py`, column `line`; one recording per language), never by WAD position or name: positions differ per language (`docs/decisions.md` 5, `docs/formats/banks.md` *Lines*). +- Figures in the docs are checked, not copied: after changing data, code or docs run `python tools/check_docs.py` + (`--scan`: also the bank survey and the catalog check's facts, ~10 min; `--wems`: every WEM rebuilt, ~10 min); + add a claim there for every new figure. - Real-game write round trips: `python tools/e2e_features.py`, `python tools/e2e_editor.py` (they revert in a `finally` block; still check `ackit status` after any interrupted run). - Shell: git at `C:\Program Files\Git\cmd` (prepend to PATH). Write Python with the Write/Edit tools; avoid diff --git a/NOTES.md b/NOTES.md index 612239c..0468ced 100644 --- a/NOTES.md +++ b/NOTES.md @@ -20,7 +20,7 @@ `ackit/bnk.py` (bank parsing: DIDX/DATA/HIRC sources). Docs: formats/upk.md, banks.md. Findings: pure-Python LZO ~47 MB/s (== decompress.exe on every block); 15,800 media IDs in 96,011 copies, 396 are prefetch heads of WAD sounds (2,101 copies) -> 15,404 in-memory sounds; copies identical except 1C898731. - 2. [x] Catalog archive `BANK` (15,404 sounds, 6,416 named by events, now 6,456) + `BANK_locations.csv.gz` (93,910 copies, + 2. [x] Catalog archive `BANK` (15,404 sounds, 6,416 named by `Play_` events; now 15,355, 13,501 named by events) + `BANK_locations.csv.gz` (93,910 copies, 2,101 prefetch heads of 396 speech lines); `ackit verify --catalog` (full: catalog OK; fixed 2 duplicate names per language: variation shown as NAME#i); categories for bank prefixes. 3. [x] Workspace writes BANK sounds into every copy and keeps speech-line prefetch heads in sync (bank files @@ -87,6 +87,26 @@ from the rules: 54,079 / 54,080 identical (INT FE9963E8:74: avg-bytes/s field one lower, informational). Removed unused code (helpers/constants nothing called, the key-posting debugger helper, a double refilter after changes); old dist zips, logs/ and caches deleted. +- 2026-10-10 final check (user: "is everything fact checked ... functioning ... documentation correct?"): + every number in the docs recomputed from the catalog and a fresh game scan; stale ones fixed (positions differ + for 81.2-81.6 %, not 75 %; 8,092 neutral BANK sounds; 24 unlinked subtitle names; 637 position-named lines; + 14,802 package media; subtitle coverage 9 % combat barks, 23 % cinematics; speaker coverage by line). Bugs + found and fixed (findings 23): 49 WAD recordings also stored whole in banks were separate BANK sounds (now + bank copies of the line, written with it); 8 BANK/SFX name clashes; name lookup preferred no name over an + alias (CLI "ambiguous" for e.g. `CHR_RadioScanner#1` in FRA). `ackit verify --lang` now applies (catalog: that + language only; in-game: must be the game's language). Subtitle error message says Windows-1252, not Latin-1. + Catalog rebuilt from the original files; `verify --catalog` OK (2,185 / 2,185 bank copies of WAD recordings + identical), e2e_features --lang all 89/89, e2e_editor, --ingame/--banks/--texts PASS, 132 tests; game original + after. Version 3.2.1: dist/UnheardCity-3.2.1-windows-amd64.zip smoke + GUI start PASS. +- 2026-10-10 re-check (user: "fact check everything again"): the catalog rebuilt from the game is identical to the + shipped one (every CSV and location index), and `verify --catalog` gives the same facts. Figures in the docs are + now checked by `tools/check_docs.py` (catalog, index.json, menu files; `--scan`: bank survey + catalog-check + facts; `--wems`: every WEM rebuilt); the build records the scan-only figures in index.json (names by event, + dialogue objects, WAD lines named after a recording). It found what hand-copied figures had wrong: 61,510 + "recordings" counted the 25 second places (61,485 recordings; the stat was fixed too), 81.2-81.6 % (not 81-82), + the loose/embedded bank-name overlap, REPORT's test count, wem.md's SFX claim (626 / 651 rebuild, not "all but 4 + looping tracks": 14 music seek tables, 5 music sample counts, 6 ambience `smpl` loops) and its extra-chunk count + (722 WEMs, not 37). New test: cross-references between the shipped files. - NEXT (needs a person): play a mission with a replaced footstep/foley sound and an edited subtitle (REPORT.md). - OPEN: channel order for >2 channels unverified; speech and in-memory sounds *heard* in-game (need gameplay; posted keys do not drive the menu). @@ -180,4 +200,5 @@ - 2026-10-10: source lists of another language (`source//`) are resolved through THAT language's WAD positions; ~1,270 speech sounds per language have no game line name (efforts 519, radio/TV 139, cinematics 134) and keep position names (`Sound.is_position_name`): their labels stay per language and mod packages from - another language are not matched to them. + another language are not matched to them. (Superseded the same day by the line identities of `ackit/lines.py`; + `is_position_name` was removed.) diff --git a/README.md b/README.md index e0464c1..14f2f92 100644 --- a/README.md +++ b/README.md @@ -56,12 +56,13 @@ app/ Unheard City (PySide6) tests/ pytest suite (synthetic game data; real-game tests run when the game is installed) data/catalog/ sound catalogs (CSV), character lists (source/), schema in data/catalog/README.md docs/ formats, reverse engineering, decisions, categories, user guide, building -tools/ catalog builder, real-game e2e checks (e2e_editor.py, e2e_features.py), packaging +tools/ catalog builder, docs figure check, real-game e2e checks (e2e_editor.py, e2e_features.py), packaging (build_app.py, installer/), artwork, license and screenshot scripts NOTES.md project status and working notes; CLAUDE.md: guidance for AI sessions ``` -Tests: `pytest` (the real-game tests are skipped when the game is not installed). +Tests: `pytest` (the real-game tests are skipped when the game is not installed). Figures in the docs: +`python tools/check_docs.py [--scan] [--wems]`. Real-game write round trips (no game launch): `python tools/e2e_features.py [--lang all]`, `python tools/e2e_editor.py`. In-game tests (Windows, Steam running, WinDbg installed): `ackit verify --ingame`, `--ingame --banks`, diff --git a/REPORT.md b/REPORT.md index 81bf529..fee2772 100644 --- a/REPORT.md +++ b/REPORT.md @@ -5,17 +5,19 @@ editor Unheard City (mission 2). ## What works (verified) - **Every sound of the game is cataloged and editable**: the speech WADs (5 languages x 10,816), the SFX WAD (651) - and, new, the **15,404 in-memory sounds** stored inside soundbanks (archive `BANK`: footsteps, foley, combat, - weapons, UI, efforts; 8,098 shared by all languages + ~1,460 localized speech lines per voice language) in 65 - loose `.bnk` files and 975 `.upk` packages. ackit reads and patches the + and, new, the **15,355 in-memory sounds** stored inside soundbanks (archive `BANK`: footsteps, foley, combat, + weapons, UI, efforts; 8,092 shared by all languages, ~1,450 localized speech lines per voice language, 6 + shared by several) in 65 loose `.bnk` files and 975 `.upk` packages. ackit reads and patches the LZO-compressed UE3 packages itself (`ackit/upk.py`, `lzo.py`; no external tools). A replacement goes into every - copy (93,910 copies); 396 speech lines with prefetch heads in banks get their heads rewritten too. - Names: 6,456 BANK sounds are named after their Wwise event (`Play_...`), the rest after their bank. + copy (93,826 copies). 445 WAD voice recordings (89 lines) also have bank copies (396 prefetch heads, 49 + whole copies; 2,185 copies), which are rewritten with them. + Names: 13,501 BANK sounds are named after the Wwise event that plays them (`Play_...`, `Speech_...`), the + other 1,854 after their bank. - **Catalog check**: `ackit verify --catalog` compares the catalog with a fresh scan of the game (missing, extra, duplicate, size/checksum, category, character, language, bank copies, subtitle copies/texts) and re-derives every relation from the game files (lines, names, subtitle links, prefetch heads, copies): **catalog OK** on the full game. -- **Subtitles**: found in `RDialogueEvent` objects inside the packages (9,016 speech lines x 11 languages) and +- **Subtitles**: found in `RDialogueEvent` objects inside the packages (9,032 speech lines x 11 languages) and in 41 scripted `RSubtitle` objects (radio, encryption cards, Riddler; linked through `RSubtitleLookup`), parsed, cataloged (`SUB_.csv`) and linked to their voice recordings through the soundbanks: 103,304 of 103,658 strings in 11 languages (INT 9,373 of 9,404), each on the same line in every language; an event that @@ -25,8 +27,8 @@ editor Unheard City (mission 2). WAD recording to its media by its own language's BKPK table (10,816 / 10,816 in each language). Result: 12,434 lines (WAD + in-memory) with exactly one recording in each of the 5 voice languages, no gap and no conflict; recordings stored twice (5 per language) are written together. Independent checks: every WAD - recording is a source in a bank of its own language (54,055 / 54,055), every prefetch head is the start of - its recording (2,101 / 2,101). Each line has one name in every language and carries its identity (`line`): + recording is a source in a bank of its own language (54,055 / 54,055), every prefetch head or whole bank copy + is the start of its recording (2,185 / 2,185). Each line has one name in every language and carries its identity (`line`): speaker labels (3,366 in every language), subtitles and mod packages (another language: the same line) follow it. Speaker labels vs the game's dialogue speaker field: 2,766 agree, 19 differ (categories.md). - **Menu text**: the localization files (`BmGame\Localization\\*.int`, ~7,000 strings per language), @@ -40,13 +42,14 @@ editor Unheard City (mission 2). - **Safety**: every changed file (WAD, bank, package, localization file) is rebuilt from a verified original copy in the store + the edits applied now; reverting the last edit restores it byte-for-byte; files changed by another tool or Steam are detected (whole-file SHA-1) and not overwritten without confirmation. -- **Tests**: 130 automated tests (unit, synthetic game with packages/banks/dialogues/localization, a - two-language game for the lines, real-game read-only, offscreen GUI incl. list order, selection and loading - overlays) pass. `ackit verify` PASS (incl. +- **Tests**: 133 automated tests (unit, synthetic game with packages/banks/dialogues/localization, a + two-language game for the lines (incl. a recording also stored whole in a bank), the shipped catalog's cross-references, real-game read-only (every + name of every language's catalog resolves to its own sound), offscreen GUI incl. list order, selection and + loading overlays) pass. `ackit verify` PASS (incl. 300/300 in-memory WEMs rebuilt byte-identically). - **Real game** (2026-10-10), all reverting and checking byte-identity themselves: - `tools/e2e_features.py --lang all` 78/78 PASS in all 11 languages (every copy of an in-memory sound, each - language's localized line, prefetch heads, a multi-copy subtitle, a scripted subtitle, a menu text, sound + + `tools/e2e_features.py --lang all` 89/89 PASS in all 11 languages (every copy of an in-memory sound, each + language's localized line, prefetch heads, a line also stored whole in banks, a multi-copy subtitle, a scripted subtitle, a menu text, sound + subtitle in one package, a mod moved to another voice language and matched to the same line at another position); `ackit verify --ingame` PASS (music heard, ncc 1.0); `--banks` PASS (77 in-memory sounds of the front-end packages replaced in 54 files: the game loads them, 68 @@ -54,9 +57,9 @@ editor Unheard City (mission 2). found in the game's memory, no crash); `tools/e2e_editor.py` PASS. Game files original at the end (`ackit status`). The catalog builder refuses to run on a modified game, so the catalog always describes the original files. -- **Package**: `dist/UnheardCity-3.2.0-windows-amd64.zip` (86 MB, catalogs included). With `PATH` reduced to +- **Package**: `dist/UnheardCity-3.2.1-windows-amd64.zip` (86 MB, catalogs included). With `PATH` reduced to the Windows folders the packaged exe passes its smoke test (in the game's language - Italian on the test PC: - 21,025 sounds loaded, encode + inject, 9,377 subtitles and 7,020 menu texts loaded, an in-memory sound decoded from a package) and the GUI starts and + 21,015 sounds loaded, encode + inject, 9,377 subtitles and 7,020 menu texts loaded, an in-memory sound decoded from a package) and the GUI starts and quits cleanly. The installer is unchanged since mission 2 (install/upgrade/uninstall verified then). ## Unresolved / limitations @@ -68,17 +71,21 @@ editor Unheard City (mission 2). text's length ("room"); cp1252 subtitles accept cp1252 characters only. Lifting these limits means moving objects inside the packages (new export offsets/sizes, added chunks): designed but not done, because it could not be verified automatically in the game (decisions 4c). -- **Unlinked subtitles**: 29 speech names (~32 strings per language: Riddler trophy texts, Arkham City Stories, - radio reports and tapes) have no event in any bank; they are listed and editable but not linked. -- **Subtitle coverage is uneven in the game itself**: no subtitles exist for efforts/grunts, few for combat barks - (11 %) and cutscene lines (10 %; most cutscene lines are not played through speech events and have no dialogue - object). No other subtitle storage was found in the packages (all classes with subtitle data checked). -- 659 voice lines are played by no event whose name the game stores; they are named after their English +- **Unlinked subtitles**: 24 speech names (24-36 strings per language, INT 31: Riddler trophy texts, Arkham City + Stories, radio reports and tapes) have no event in any bank; they are listed and editable but not linked. +- **Subtitle coverage is uneven in the game itself** (INT voice lines with a subtitle): none for efforts/grunts, + few for combat barks (9 %), cinematics and boss fights (23 %; most cutscene lines are not played through speech + events and have no dialogue object), 89-97 % for story, predator, character and open-world lines. No other + subtitle storage was found in the packages (all classes with subtitle data checked). +- 637 voice lines are played by no event whose name the game stores; they are named after their English recording's position (the same name in every language). Mod packages from versions before 3.1 carry no line identity: their voice lines are not moved to another language (listed instead of guessed). - Speaker labels: 2 names of the curated lists match two recordings each (wemscan name collision), both are labelled; 19 labels differ from the game's dialogue speaker field (to be checked by listening). -- **Unnamed sounds**: ~9,000 BANK sounds have no event name and are named `_`. +- **Unnamed sounds**: 1,854 BANK sounds are reached by no event whose name the game stores and are named + `_`. +- 25 SFX originals (music seek tables and sample counts, looping ambience) are not reproduced byte-for-byte by + the WEM rules (`docs/formats/wem.md`); writing is not affected (replaced music plays in game). - Two copies of media `1C898731` differ; its checksum is left empty. Channel order for 4-6 channel sounds is unverified. M4A/AAC/WMA input needs an ffmpeg on the system. - Another game build is scanned at first start (WADs: seconds; banks, lines and subtitles: a few minutes, diff --git a/ackit/__init__.py b/ackit/__init__.py index 8cf80a2..d2d1743 100644 --- a/ackit/__init__.py +++ b/ackit/__init__.py @@ -8,4 +8,4 @@ Library layers (low -> high): inject (WAD writing), originals (store), workspace (apply/revert), modpack (share) tools cli (`ackit` command), verify/ (offline self-check, in-game test on Windows) """ -__version__ = "3.2.0" +__version__ = "3.2.1" diff --git a/ackit/banks.py b/ackit/banks.py index 0d47c75..a8fe037 100644 --- a/ackit/banks.py +++ b/ackit/banks.py @@ -156,12 +156,14 @@ class Scan: self.events = {} # id -> [action ids] self.actions = {} # id -> target id self.sound_media = defaultdict(set) # sound object id -> media ids - self.prefetch = set() # media ids whose in-memory copy is a prefetch head + self.prefetch = set() # media ids whose bank copies belong to a WAD recording (prefetch + # heads; whole recordings: catalog._whole_recordings) self.lang_media = defaultdict(set) # language -> media ids named by Sound objects of that language's banks # BKPK tables: WAD entry id -> {language: {media id: (offset in entry, size)}}. A table belongs to the # language its localized AkBank export declares ('' = outside such an export). The same media ID can be # used by several languages (at a different position in each language's WAD), so tables never merge. self.packs = defaultdict(_by_language) + self.subtitle_objects = {} # counts of dialogue / RSubtitle objects (catalog.scan_text) self.issues = [] # (kind, detail) @@ -368,11 +370,12 @@ def media_language(sc, m): return next(iter(langs)) if len(langs) == 1 else "" -def media_names(sc): +def media_names(sc, reserved=()): """{media id: (display name, primary bank name)}: named after the most specific 'Play' event that reaches the media, otherwise after its bank (_). Localized speech has one media per language for the same line: they share the name, variations are numbered within a language. Names are unique - among the language-neutral media plus any one language (what a catalog shows at a time).""" + among the language-neutral media plus any one language (what a catalog shows at a time) and avoid the + lower-case names in `reserved` (other archives' sounds).""" ev = event_media(sc) best = {} for name, ms in ev.items(): @@ -386,7 +389,7 @@ def media_names(sc): if m not in best or rank < best[m][0]: best[m] = (rank, name, i, len(g)) out = {} - taken = set() # (language, lower-case name) + taken = {("", n) for n in reserved} # (language, lower-case name) def free(lang, nm): k = nm.lower() diff --git a/ackit/catalog.py b/ackit/catalog.py index 1038a5c..a6e4efd 100644 --- a/ackit/catalog.py +++ b/ackit/catalog.py @@ -234,10 +234,11 @@ def scan_archive(archive, wad_path, characters=None, progress=None): _CAMEL = re.compile(r"(?<=[a-z])(?=[A-Z])") -def bank_sounds(sc, characters=None): - """[Sound] of the in-memory sounds found by `banks.scan` (prefetch heads belong to WAD sounds).""" +def bank_sounds(sc, characters=None, reserved=()): + """[Sound] of the in-memory sounds found by `banks.scan` (prefetch heads belong to WAD sounds). `reserved`: + lower-case names taken by other archives (sfx_names).""" characters = characters if characters is not None else load_characters() - names = banks.media_names(sc) + names = banks.media_names(sc, reserved) every = set().union(*(e.get("langs", set()) for e in sc.media.values())) - {""} out = [] for m, e in sc.media.items(): @@ -275,13 +276,45 @@ def derive_banks(game, sc, characters=None): """(bank sounds, locations, Lines) from a Scan: the lines, the prefetch heads' lines, the in-memory sounds with their line identities and names.""" lines = build_lines(game, sc) + mark_heads(sc, whole_recordings(sc, lines)) locs = [l._replace(sound=lines.head_key(l)) if l.kind == banks.PREFETCH_HEAD else l for l in sc.locations] - snd = bank_sounds(sc, characters) + snd = bank_sounds(sc, characters, sfx_names(game)) lines.names = line_names(game, sc, lines, {s.key: s.name for s in snd}) apply_lines(snd, lines, lines.names) return snd, locs, lines +def whole_recordings(sc, lines): + """Media stored in memory that are WAD recordings of their bank's language (the same media ID, identical + bytes: 49 in the game, e.g. game-over taunts). Like prefetch heads, these copies belong to the WAD line + and are written with it; they are not sounds of their own.""" + keys = collections.defaultdict(list) + for l in sc.locations: + if l.kind == banks.MEMORY: + k = lines.head_key(l) + keys[l.media].append(bool(k) and not k.startswith(banks.ARCHIVE + ":")) + return {m for m, wad in keys.items() if all(wad)} + + +def mark_heads(sc, media): + """Treat the bank copies of `media` as heads of WAD recordings (see whole_recordings).""" + if not media: + return + for m in media: + if m in sc.media: + sc.media[m]["kinds"] = {banks.PREFETCH_HEAD} + sc.prefetch.add(m) + sc.locations = [l._replace(kind=banks.PREFETCH_HEAD) if l.media in media else l for l in sc.locations] + + +def sfx_names(game): + """Lower-case names and aliases of the SFX WAD's sounds: in-memory sounds of the same bank must not take + them (`_` is the position name of the bank's streamed sounds too).""" + if "SFX" not in game.archives: + return set() + return {n.lower() for s in scan_archive("SFX", game.wad("SFX"), characters={}) for n in s.names()} + + def line_names(game, sc, lines, bank_names=None): """{line id: name} (lines.line_names). Lines no event names keep their English recording's name: the position-based name in the INT WAD (`NAME_i`, or `NAME#i` where that is taken) or the bank-numbered name.""" @@ -378,6 +411,10 @@ def scan_text(game, progress=None, characters=None): sc = banks.scan(game.cooked, progress, visit=col) names = {n for n in sc.names if n.lower().startswith("speech_")} names |= {n for n, _ in banks.media_names(sc).values() if n.lower().startswith("speech_")} + lookup = [n for n in col.dialogues if n.startswith("#lookup:")] + sc.subtitle_objects = {"dialogue_objects": sum(len(v) for n, v in col.dialogues.items() if n not in lookup), + "dialogue_lines": len(col.dialogues) - len(lookup), + "scripted_objects": sum(len(col.dialogues[n]) for n in lookup)} col.resolve(names) # the subtitle objects name their speech events too: an event ID is the FNV-1 hash of its name, so these # names find events no package name table lists (e.g. Vicki Vale's radio reports) @@ -519,6 +556,11 @@ class Catalog: key = os.path.splitext(os.path.basename(name))[0].lower() if re.search(r"\.\w{2,4}$", name) else name.lower() hits = self._by_name.get(key, []) uniq = list({id(s): s for s in hits}.values()) + if len(uniq) > 1: + # a name wins over another sound's alias: a line named after its INT position (NAME#i) can equal + # the position alias of another recording in another language's WAD + own = [s for s in uniq if s.name.lower() == key] + uniq = own if len(own) == 1 else uniq if len(uniq) == 1: return uniq[0] if len(uniq) > 1: diff --git a/ackit/cli.py b/ackit/cli.py index 193647d..8f8820c 100644 --- a/ackit/cli.py +++ b/ackit/cli.py @@ -294,7 +294,8 @@ def cmd_game(a): def cmd_verify(a): if a.catalog: from .verify import catalogcheck - rep = catalogcheck.run(_game(a), [x.upper() for x in a.archive] or None, quick=a.quick) + archives = [x.upper() for x in a.archive] or ([a.lang] if a.lang else None) + rep = catalogcheck.run(_game(a), archives, quick=a.quick) for (arch, kind), n in sorted(rep.counts().items()): print("%-14s %-10s %d" % (arch, kind, n)) for arch, kind, key, detail in rep.issues[:a.show]: @@ -310,6 +311,9 @@ def cmd_verify(a): ok = selfcheck.run(ws, log=print) sys.exit(0 if ok else 1) from .verify import ingame + if a.lang and a.lang != ws.game.language(): + raise GameError("the game plays its own language (%s); set it in Steam (game properties > Language) to " + "test %s in the game" % (ws.game.language(), a.lang)) if a.banks: rep = ingame.run_banks(ws, a.sound or None) elif a.texts: @@ -385,7 +389,7 @@ def build_parser(): s = add("verify", cmd_verify, "self-check (offline), catalog cross-check (--catalog), or test in the real " "game with --ingame (Windows)") s.add_argument("--catalog", action="store_true", help="cross-check the catalog against the game files") - s.add_argument("--archive", action="append", default=[], help="with --catalog: only this archive (repeatable)") + s.add_argument("--archive", action="append", default=[], help="with --catalog: only this archive (repeatable; --lang X: only X)") s.add_argument("--quick", action="store_true", help="with --catalog: loose banks + a sample of packages") s.add_argument("--show", type=int, default=30, help="with --catalog: issues to list (default 30)") s.add_argument("--ingame", action="store_true"); s.add_argument("--target", default="SFX:EDF036D6/393BC53A") diff --git a/ackit/lines.py b/ackit/lines.py index f7063a0..b0a1821 100644 --- a/ackit/lines.py +++ b/ackit/lines.py @@ -27,6 +27,7 @@ class Lines: objects: dict = field(default_factory=dict) # line id -> sorted Sound object ids obj_keys: dict = field(default_factory=dict) # Sound object id -> {lang: sound key} names: dict = field(default_factory=dict) # line id -> name (catalog.line_names) + unnamed: set = field(default_factory=set) # line ids no named event plays (named after a recording) copy_of: dict = field(default_factory=dict) # WAD key -> key of the same media stored in another entry issues: list = field(default_factory=list) # (kind, detail): broken one-to-one relations stats: dict = field(default_factory=dict) @@ -161,10 +162,10 @@ def build(game, sc, walk): for k, primary in copy_of.items(): # a second place of the same recording: the same line if primary in lines.line_of: lines.line_of[k] = lines.line_of[primary] - all_keys = set().union(*lines._keys.values()) if lines._keys else set() - all_keys |= set(copy_of) + recordings = set().union(*lines._keys.values()) if lines._keys else set() + all_keys = recordings | set(copy_of) # + the second places of recordings stored twice full = sum(1 for per in lines.members.values() if len(per) == len(langs)) - lines.stats = {"languages": langs, "wad_wems": wems, "recordings": len(all_keys), + lines.stats = {"languages": langs, "wad_wems": wems, "recordings": len(recordings), "in_a_line": len(lines.line_of), "lines": len(lines.members), "lines_in_every_language": full, "copies": sorted(copy_of), "lines_missing_a_language": {lid: sorted(set(langs) - set(per)) @@ -215,8 +216,9 @@ def line_names(sc, lines, fallback): out[lid] = name if i == 0 else "%s#%d" % (name, i + 1) else: out[lid] = name if len(lids) == 1 else "%s_%d" % (name, i + 1) - for lid, per in lines.members.items(): - if lid not in out: - k = per.get("INT") or per[next(L for L in lines.langs if L in per)] - out[lid] = fallback(k) + lines.unnamed = set(lines.members) - set(out) + for lid in lines.unnamed: + per = lines.members[lid] + k = per.get("INT") or per[next(L for L in lines.langs if L in per)] + out[lid] = fallback(k) return out diff --git a/ackit/subtitles.py b/ackit/subtitles.py index 283c546..2199711 100644 --- a/ackit/subtitles.py +++ b/ackit/subtitles.py @@ -9,7 +9,7 @@ a byte size; struct elements end with a u16 0). A dialogue object can be stored in several packages; a subtitle edit is written into every copy, in place: the new text replaces the bytes of the old FString and is padded with NUL characters (the game stops at the first NUL), so no size field, export or package layout changes. The text must therefore fit the original -string's bytes (Latin-1/cp1252 strings: one byte per character; UTF-16 strings: two). +string's bytes (cp1252 strings: one byte per character; UTF-16 strings: two). """ import csv import gzip @@ -55,7 +55,8 @@ def encode_fstring(text, size, enc): try: raw = text.encode("cp1252") except UnicodeEncodeError: - raise SubtitleError("this subtitle is stored as Latin-1 text; '%s' cannot be written" + raise SubtitleError("this subtitle is stored as Western European (Windows-1252) text; '%s' cannot be " + "written" % next(c for c in text if not _cp1252(c))) from None n = size - 4 if len(raw) + 1 > n: @@ -175,6 +176,7 @@ class Collector: def __init__(self): self.dialogues = defaultdict(list) # speech name -> [(file, Dialogue)] + self.scripted = set() # speech names of RSubtitle objects (resolve) self.issues = [] def __call__(self, rel, pkg, start, data): @@ -201,6 +203,7 @@ class Collector: self.issues.append(("subtitles", "no speech line for subtitle %s" % copies[0][1].export)) name = copies[0][1].export self.dialogues[name].extend((rel, d._replace(speech=name)) for rel, d in copies) + self.scripted.add(name) # --- catalog files: SUB_.csv (texts) + SUB_locations.csv.gz (every copy) ------------------ diff --git a/ackit/verify/catalogcheck.py b/ackit/verify/catalogcheck.py index 187e04f..1907f26 100644 --- a/ackit/verify/catalogcheck.py +++ b/ackit/verify/catalogcheck.py @@ -18,7 +18,8 @@ import random from .. import banks, classify, subtitles from ..catalog import (Catalog, apply_lines, available_archives, bank_names, bank_sounds, build_lines, character_of, - is_shipped, line_names, load_archive, load_banks, load_characters, read_csv, scan_archive) + is_shipped, line_names, load_archive, load_banks, load_characters, mark_heads, read_csv, + scan_archive, sfx_names, whole_recordings) from ..inject import read_target from ..paths import catalog_dir from ..wad import Wad @@ -133,14 +134,17 @@ def check_banks(rep, game, rows, locations, characters, quick=False, sample=40, col.resolve({n for n in sc.names | known if n.lower().startswith("speech_")}) check_subtitles(rep, game, col, set(files), quick) lines = None - if not quick: - sc.names.update(col.dialogues) # as the build does: subtitle objects name speech events too - lines = build_lines(game, sc) index = collections.defaultdict(set) for m, ls in locations.items(): for l in ls: if l.file in set(files): index[(l.media, l.kind)].add((l.file, l.offset, l.size, l.sound if l.kind == banks.PREFETCH_HEAD else "")) + if not quick: + sc.names.update(col.dialogues) # as the build does: subtitle objects name speech events too + lines = build_lines(game, sc) + mark_heads(sc, whole_recordings(sc, lines)) # WAD recordings also stored whole in banks + else: # a sample cannot derive them: the index says which they are + mark_heads(sc, {m for m, k in index if k == banks.PREFETCH_HEAD} - sc.prefetch) found = collections.defaultdict(set) for l in sc.locations: head = l.kind == banks.PREFETCH_HEAD @@ -152,7 +156,7 @@ def check_banks(rep, game, rows, locations, characters, quick=False, sample=40, rep.add(banks.ARCHIVE, "copies", "%08X" % k[0], "%s: index %d, game %d" % ("head" if k[1] == banks.PREFETCH_HEAD else "copies", len(index.get(k, ())), len(found.get(k, ())))) cat = {s.target: s for s in rows} - fresh_list = bank_sounds(sc, characters) + fresh_list = bank_sounds(sc, characters, sfx_names(game)) if lines is not None: lines.names = line_names(game, sc, lines, {s.key: s.name for s in fresh_list}) apply_lines(fresh_list, lines, lines.names) @@ -253,11 +257,11 @@ def check_lines(rep, game, sc, lines, locations): rep.add("LINES", "links", k, "media %08X is not a source in any %s bank" % (m, L)) rep.facts.append("WAD recordings named as a source by a Sound object of their own language's bank: %d of %d" % (named, total)) - rep.facts.append("voice languages %s: %d recordings (WAD %s + localized in-memory), %d lines, %d of them " - "with exactly one recording in every language" - % ("/".join(lines.langs), st["recordings"], - "/".join(str(st["wad_wems"].get(L, 0)) for L in lines.langs), st["lines"], - st["lines_in_every_language"])) + rep.facts.append("voice languages %s: %d recordings (WAD WEMs %s, %d of them second places of a recording " + "stored twice, + localized in-memory), %d lines, %d of them with exactly one recording in every " + "language" % ("/".join(lines.langs), st["recordings"], + "/".join(str(st["wad_wems"].get(L, 0)) for L in lines.langs), len(st["copies"]), + st["lines"], st["lines_in_every_language"])) # catalog rows vs the game's lines rows = {} for L in lines.langs + [banks.ARCHIVE]: @@ -305,7 +309,8 @@ def check_lines(rep, game, sc, lines, locations): rep.add("LINES", "copies", k, "differs from %s (same media ID)" % primary) rep.facts.append("recordings stored twice in a WAD (same media ID, written together): %d, byte-identical: %d" % (len(lines.copy_of), same_copies)) - rep.facts.append("prefetch heads: %d of %d stored copies are byte-identical to the start of their WAD recording" + rep.facts.append("prefetch heads and whole bank copies: %d of %d are byte-identical to the start of their WAD " + "recording" % (same, len(heads))) # independent evidence: the recordings of a line have similar lengths in every language ratios = [] diff --git a/data/catalog/BANK.csv b/data/catalog/BANK.csv index 5d15472..95893fd 100644 --- a/data/catalog/BANK.csv +++ b/data/catalog/BANK.csv @@ -4084,9 +4084,9 @@ BANK,0191DF08,Play_RadioScanner_9,CHR_RadioScanner/0191DF08,Speech,Radio & TV,CH BANK,24202946,Play_RadioScanner_Close_Gadget,CHR_RadioScanner/24202946,Speech,Radio & TV,CHR_RadioScanner,,16333BFB,,,0.384,48000,2,5860,2ac62f5c2dd982da, BANK,0A66562F,Play_RadioScanner_Intercept_2,CHR_RadioScanner/0A66562F,Speech,Radio & TV,CHR_RadioScanner,,1590991C,,,0.200,16000,1,386,f91e17fbba8174b7, BANK,1C8A9035,Play_RadioScanner_Intercept_4,CHR_RadioScanner/1C8A9035,Speech,Radio & TV,CHR_RadioScanner,,02C289C6,,,0.286,48000,2,5199,fa723f839dc998ea, -BANK,08514617,CHR_RasBossFight_1,CHR_RasBossFight/08514617,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,0.312,48000,1,2558,3a15689ad10607a9, -BANK,2721B526,CHR_RasBossFight_2,CHR_RasBossFight/2721B526,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,0.774,48000,1,5983,1baaa4b70ac728c3, -BANK,35F131FD,CHR_RasBossFight_3,CHR_RasBossFight/35F131FD,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,1.207,48000,1,8500,ea4b213c049d07d4, +BANK,08514617,CHR_RasBossFight_08514617,CHR_RasBossFight/08514617,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,0.312,48000,1,2558,3a15689ad10607a9, +BANK,2721B526,CHR_RasBossFight_2721B526,CHR_RasBossFight/2721B526,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,0.774,48000,1,5983,1baaa4b70ac728c3, +BANK,35F131FD,CHR_RasBossFight_35F131FD,CHR_RasBossFight/35F131FD,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,1.207,48000,1,8500,ea4b213c049d07d4, BANK,2FA2124B,Play_DemonDeath,CHR_RasBossFight/2FA2124B,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,3.154,48000,2,36691,b919594f844829f4, BANK,2D6FFCEF,Play_GiantRASEmergeLFE,CHR_RasBossFight/2D6FFCEF,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,1.770,48000,1,10530,7be35ff8a95b8a55, BANK,00CCAAA6,Play_GiantSword_1,CHR_RasBossFight/00CCAAA6,SFX,Character foley,CHR_RasBossFight,,,,Ra's al Ghul,2.760,48000,1,21059,777cec0b514a660c, @@ -7258,7 +7258,7 @@ BANK,0F26C10C,Play_DT_VortexLoop01,SEQ_DemonTrials/0F26C10C,SFX,Scripted sequenc BANK,1CBC4CD9,Play_TowerDescendLoop,SEQ_DemonTrials/1CBC4CD9,SFX,Scripted sequences,SEQ_DemonTrials,,,,,6.720,48000,1,61425,ba6c467e650f65a9, BANK,17E4CD34,Play_VortexOn,SEQ_DemonTrials/17E4CD34,SFX,Scripted sequences,SEQ_DemonTrials,,,,,3.059,48000,3,51620,d32b50a9374a0329, BANK,0BFFAE8F,Play_WallBurnLowSweep_1,SEQ_DemonTrials/0BFFAE8F,SFX,Scripted sequences,SEQ_DemonTrials,,,,,8.491,48000,2,82419,36d502760f2aac92, -BANK,14B78DC0,SEQ_DemonTrials_1,SEQ_DemonTrials/14B78DC0,SFX,Scripted sequences,SEQ_DemonTrials,,,,,1.619,48000,2,31218,435f399b85421da6, +BANK,14B78DC0,SEQ_DemonTrials_14B78DC0,SEQ_DemonTrials/14B78DC0,SFX,Scripted sequences,SEQ_DemonTrials,,,,,1.619,48000,2,31218,435f399b85421da6, BANK,0EC21184,Play_BridgeExplosion_1,SEQ_GCPDExplosion/0EC21184,SFX,Scripted sequences,SEQ_GCPDExplosion,,,,,3.610,32000,2,39751,9b89913f89bc6e90, BANK,03E5C32B,Play_BridgeExplosion_2,SEQ_GCPDExplosion/03E5C32B,SFX,Scripted sequences,SEQ_GCPDExplosion,,,,,6.344,32000,4,203846,0e8cdd46066c44c8, BANK,2D667F1A,Play_GCPD_DistanExp01,SEQ_GCPDExplosion/2D667F1A,SFX,Scripted sequences,SEQ_GCPDExplosion,,,,Cop,7.801,11025,5,96518,94fa5c717169fd31, @@ -7422,11 +7422,6 @@ BANK,1A3B7533,Speech_Bully_C_conv_05,Speech_Bully_C_conv/1A3B7533,Speech,Combat BANK,076C0C7A,Speech_Bully_C_conv_05,Speech_Bully_C_conv/076C0C7A,Speech,Combat barks,Speech_Bully_C_conv,ITA,225159AB,,Thug,6.635,28000,1,50298,ac39d899bdb167cf, BANK,0C5DBD6B,Speech_Bully_C_conv_05,Speech_Bully_C_conv/0C5DBD6B,Speech,Combat barks,Speech_Bully_C_conv,DEU,225159AB,,Thug,6.488,28000,1,47977,72b8e1e623c4b089, BANK,16AD1A51,Speech_Bully_C_conv_05,Speech_Bully_C_conv/16AD1A51,Speech,Combat barks,Speech_Bully_C_conv,ESN,225159AB,,Thug,5.952,28000,1,43883,9711ec368d11e388, -BANK,3C7B953C,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3_01,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3/3C7B953C,Speech,Story missions,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3,INT,,Batman,,1.149,44100,1,11551,e25bb4cde803739d, -BANK,3653354B,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3_01,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3/3653354B,Speech,Story missions,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3,FRA,,Batman,,1.308,28000,1,10780,deb4215bfc476d40, -BANK,238B1AD2,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3_01,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3/238B1AD2,Speech,Story missions,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3,ITA,,Batman,,1.782,28000,1,13054,c8e2c2d687800345, -BANK,09ECFA75,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3_01,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3/09ECFA75,Speech,Story missions,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3,DEU,,Batman,,1.620,28000,1,10820,71bcff8ee2d30cab, -BANK,2540B067,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3_01,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3/2540B067,Speech,Story missions,Speech_CH_ZERO_CH0_ENCRYPTION_CARD3,ESN,,Batman,,1.330,28000,1,11204,36e05933f5c12160, BANK,235C3AB2,Speech_CH_ZERO_PENG_hands_PENG_fightcomments_01,Speech_CH_ZERO_PENG_hands/235C3AB2,Speech,Story missions,Speech_CH_ZERO_PENG_hands,INT,13EEB34A,,Penguin,4.681,44100,1,36231,0c9e76840c288b09, BANK,32D15A18,Speech_CH_ZERO_PENG_hands_PENG_fightcomments_01,Speech_CH_ZERO_PENG_hands/32D15A18,Speech,Story missions,Speech_CH_ZERO_PENG_hands,FRA,13EEB34A,,Penguin,4.768,28000,1,33903,b5e01b7e7d017a30, BANK,2F46312B,Speech_CH_ZERO_PENG_hands_PENG_fightcomments_01,Speech_CH_ZERO_PENG_hands/2F46312B,Speech,Story missions,Speech_CH_ZERO_PENG_hands,ITA,13EEB34A,,Penguin,4.545,28000,1,32718,a0a44cda507f015e, @@ -7549,11 +7544,6 @@ BANK,3FA65C62,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN_19,Speech_CINEMATIC_CH3_MEET BANK,2E1796AE,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN_19,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN/2E1796AE,Speech,Cinematics,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN,ITA,18D8058E,,Penguin,4.759,28000,1,27292,6a7810833a312bc3, BANK,2D661420,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN_19,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN/2D661420,Speech,Cinematics,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN,DEU,18D8058E,,Penguin,3.802,28000,1,26244,9a68ba22c0a3d35d, BANK,20DF687A,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN_19,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN/20DF687A,Speech,Cinematics,Speech_CINEMATIC_CH3_MEET_THE_PENGUIN,ESN,18D8058E,,Penguin,4.586,28000,1,28616,949f43d5dd3b8823, -BANK,3D3E517A,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER_1,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER/3D3E517A,Speech,Cinematics,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER,INT,,,Batman; Joker,2.492,44100,1,20047,3a4ba2b9c63934ed, -BANK,0B42FF7D,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER_1,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER/0B42FF7D,Speech,Cinematics,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER,FRA,,,Batman; Joker,2.534,28000,1,17075,bc9ab36f0afa7b27, -BANK,1FC3A83F,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER_1,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER/1FC3A83F,Speech,Cinematics,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER,ITA,,,Batman; Joker,2.544,28000,1,17300,9af41c14c8540a89, -BANK,29B4097E,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER_1,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER/29B4097E,Speech,Cinematics,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER,DEU,,,Batman; Joker,2.534,28000,1,16926,736d5f57b8397512, -BANK,0472E7FB,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER_1,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER/0472E7FB,Speech,Cinematics,Speech_CINEMATIC_CH6_BATMAN_CONFRONTS_JOKER,ESN,,,Batman; Joker,2.304,28000,1,16967,7c83a67ca3aedb6a, BANK,0D579006,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA_GRP_BM_01,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA/0D579006,Speech,Combat barks,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA,INT,261948D3,,Batman,1.942,44100,1,15561,804be70742e85c57, BANK,3E88D860,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA_GRP_BM_01,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA/3E88D860,Speech,Combat barks,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA,FRA,261948D3,,Batman,1.791,28000,1,12938,e833a0135c168827, BANK,1CA672D4,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA_GRP_BM_01,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA/1CA672D4,Speech,Combat barks,Speech_COMBAT_BARKS_1_INSULT_GROUP_SetA,ITA,261948D3,,Batman,2.006,28000,1,14634,d24da394444a4430, @@ -12018,34 +12008,11 @@ BANK,2FAA3476,Speech_FORENSICS_Cold_Correct_FORENSICS_Cold_Correct,Speech_FORENS BANK,25DB5842,Speech_FORENSICS_Cold_Correct_FORENSICS_Cold_Correct,Speech_FORENSICS_Cold_Correct/25DB5842,Speech,Gameplay & hints,Speech_FORENSICS_Cold_Correct,ITA,2B9791FF,,,2.628,28000,1,20518,ac2a3c7fdad562df, BANK,2F9A9DEF,Speech_FORENSICS_Cold_Correct_FORENSICS_Cold_Correct,Speech_FORENSICS_Cold_Correct/2F9A9DEF,Speech,Gameplay & hints,Speech_FORENSICS_Cold_Correct,DEU,2B9791FF,,,2.691,28000,1,21795,236e4c26a88643e8, BANK,35FAA3F3,Speech_FORENSICS_Cold_Correct_FORENSICS_Cold_Correct,Speech_FORENSICS_Cold_Correct/35FAA3F3,Speech,Gameplay & hints,Speech_FORENSICS_Cold_Correct,ESN,2B9791FF,,,2.400,28000,1,19029,25a18a794248f9dc, -BANK,02B02BF2,Speech_GAME_OVER_Harley_Dead2,Speech_GAME_OVER_Harley/02B02BF2,Speech,Gameplay & hints,Speech_GAME_OVER_Harley,INT,,Harley Quinn,Harley Quinn,2.034,44100,3,44118,3cd5d7056427b140, -BANK,3CCA3AAE,Speech_GAME_OVER_Harley_Dead2,Speech_GAME_OVER_Harley/3CCA3AAE,Speech,Gameplay & hints,Speech_GAME_OVER_Harley,FRA,,Harley Quinn,Harley Quinn,2.164,28000,1,15141,b1917825638115d7, -BANK,20AED5D2,Speech_GAME_OVER_Harley_Dead2,Speech_GAME_OVER_Harley/20AED5D2,Speech,Gameplay & hints,Speech_GAME_OVER_Harley,ITA,,Harley Quinn,Harley Quinn,1.787,28000,1,12397,38e40367a3c12f66, -BANK,043620D9,Speech_GAME_OVER_Harley_Dead2,Speech_GAME_OVER_Harley/043620D9,Speech,Gameplay & hints,Speech_GAME_OVER_Harley,DEU,,Harley Quinn,Harley Quinn,2.090,28000,1,13346,ae7986530de3cf1e, -BANK,068435EE,Speech_GAME_OVER_Joker_DEAD_13,Speech_GAME_OVER_Joker/068435EE,Speech,Gameplay & hints,Speech_GAME_OVER_Joker,INT,,Joker,Joker,1.539,44100,3,37363,e2adc1dc6a43ba18, -BANK,0E820C9F,Speech_GAME_OVER_Joker_DEAD_13,Speech_GAME_OVER_Joker/0E820C9F,Speech,Gameplay & hints,Speech_GAME_OVER_Joker,FRA,,Joker,Joker,1.593,28000,1,12567,ab5ef6041156eb76, -BANK,16D46180,Speech_GAME_OVER_Joker_DEAD_13,Speech_GAME_OVER_Joker/16D46180,Speech,Gameplay & hints,Speech_GAME_OVER_Joker,ITA,,Joker,Joker,1.029,28000,1,8680,1bd4cf5d6cc3d049, -BANK,1AD4AAB1,Speech_GAME_OVER_Joker_DEAD_13,Speech_GAME_OVER_Joker/1AD4AAB1,Speech,Gameplay & hints,Speech_GAME_OVER_Joker,DEU,,Joker,Joker,1.593,28000,1,10725,117399e19037e4a5, -BANK,3615E00B,Speech_GAME_OVER_Joker_DEAD_13,Speech_GAME_OVER_Joker/3615E00B,Speech,Gameplay & hints,Speech_GAME_OVER_Joker,ESN,,Joker,Joker,1.258,28000,1,9521,4282086256d079b9, BANK,1AC4F6B4,Speech_GAME_OVER_Mr_Freeze_FREEZE_new,Speech_GAME_OVER_Mr_Freeze/1AC4F6B4,Speech,Gameplay & hints,Speech_GAME_OVER_Mr_Freeze,INT,1583D57C,,Mr. Freeze,2.750,44100,3,58665,0f7610cc9079b4d3, BANK,1D9BDA58,Speech_GAME_OVER_Mr_Freeze_FREEZE_new,Speech_GAME_OVER_Mr_Freeze/1D9BDA58,Speech,Gameplay & hints,Speech_GAME_OVER_Mr_Freeze,FRA,1583D57C,,Mr. Freeze,2.447,44100,1,19577,3e34a16458bf6326, BANK,3B38366F,Speech_GAME_OVER_Mr_Freeze_FREEZE_new,Speech_GAME_OVER_Mr_Freeze/3B38366F,Speech,Gameplay & hints,Speech_GAME_OVER_Mr_Freeze,ITA,1583D57C,,Mr. Freeze,2.826,44100,1,18356,002437e9e096ad72, BANK,3481719C,Speech_GAME_OVER_Mr_Freeze_FREEZE_new,Speech_GAME_OVER_Mr_Freeze/3481719C,Speech,Gameplay & hints,Speech_GAME_OVER_Mr_Freeze,DEU,1583D57C,,Mr. Freeze,2.926,44100,1,21183,1ee5dc61590ea268, BANK,0D2CE9E1,Speech_GAME_OVER_Mr_Freeze_FREEZE_new,Speech_GAME_OVER_Mr_Freeze/0D2CE9E1,Speech,Gameplay & hints,Speech_GAME_OVER_Mr_Freeze,ESN,1583D57C,,Mr. Freeze,2.576,44100,1,19639,55ded8c51a05fd16, -BANK,23647FF3,Speech_GAME_OVER_Talia_al_Ghul_Dead_Demon,Speech_GAME_OVER_Talia_al_Ghul/23647FF3,Speech,Gameplay & hints,Speech_GAME_OVER_Talia_al_Ghul,FRA,,Talia al Ghul,Talia al Ghul,1.836,28000,1,14387,a0487f77e650ec00, -BANK,0173EC0D,Speech_GAME_OVER_Talia_al_Ghul_Dead_Demon,Speech_GAME_OVER_Talia_al_Ghul/0173EC0D,Speech,Gameplay & hints,Speech_GAME_OVER_Talia_al_Ghul,ITA,,Talia al Ghul,Talia al Ghul,2.095,28000,1,15228,e60c97bbb7839e0a, -BANK,07AE2F63,Speech_GAME_OVER_Talia_al_Ghul_Dead_Demon,Speech_GAME_OVER_Talia_al_Ghul/07AE2F63,Speech,Gameplay & hints,Speech_GAME_OVER_Talia_al_Ghul,ESN,,Talia al Ghul,Talia al Ghul,1.924,28000,1,14521,b3b9c78725afa4d3, -BANK,2EF4A47C,Speech_GAME_OVER_Two_Face_Question,Speech_GAME_OVER_Two_Face/2EF4A47C,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,INT,,Two-Face,,1.463,44100,3,32957,0bc9366548a9daa2, -BANK,01ED1FEB,Speech_GAME_OVER_Two_Face_Question,Speech_GAME_OVER_Two_Face/01ED1FEB,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,FRA,,Two-Face,,1.597,28000,1,11484,ec2249942345ac00, -BANK,25E33254,Speech_GAME_OVER_Two_Face_Question,Speech_GAME_OVER_Two_Face/25E33254,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,ITA,,Two-Face,,1.390,28000,1,11300,c538dd16556b67d5, -BANK,34CCE5AC,Speech_GAME_OVER_Two_Face_Question,Speech_GAME_OVER_Two_Face/34CCE5AC,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,DEU,,Two-Face,,1.463,28000,1,11168,859d6637c812a421, -BANK,34F10C93,Speech_GAME_OVER_Two_Face_Question,Speech_GAME_OVER_Two_Face/34F10C93,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,ESN,,Two-Face,,1.236,28000,1,9068,d62fb0568bcc9b23, -BANK,38BD88BC,Speech_GAME_OVER_Two_Face_Reply_Bat_2,Speech_GAME_OVER_Two_Face/38BD88BC,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,INT,,Two-Face,,1.730,44100,3,39669,2b265c6efc35fd09, -BANK,2DD25DC9,Speech_GAME_OVER_Two_Face_Reply_Bat_2,Speech_GAME_OVER_Two_Face/2DD25DC9,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,FRA,,Two-Face,,1.862,28000,1,14695,c503e36bcf0b9ef0, -BANK,266B4A26,Speech_GAME_OVER_Two_Face_Reply_Bat_2,Speech_GAME_OVER_Two_Face/266B4A26,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,ITA,,Two-Face,,1.772,28000,1,13666,28f64eb52ca4cda3, -BANK,27071B28,Speech_GAME_OVER_Two_Face_Reply_Bat_2,Speech_GAME_OVER_Two_Face/27071B28,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,DEU,,Two-Face,,1.730,28000,1,13012,a849e1f91600de98, -BANK,309FF2EE,Speech_GAME_OVER_Two_Face_Reply_Bat_2,Speech_GAME_OVER_Two_Face/309FF2EE,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,ESN,,Two-Face,,1.861,28000,1,14258,3498e68019b17dc3, -BANK,07615F5F,Speech_GAME_OVER_Two_Face_Reply_Cat_1,Speech_GAME_OVER_Two_Face/07615F5F,Speech,Gameplay & hints,Speech_GAME_OVER_Two_Face,ESN,,Two-Face,Catwoman,2.005,28000,1,14774,ba2ab6a3caf28194, BANK,14F358D6,Speech_INTERROGATION_CH5_MAYOR_01_04b,Speech_INTERROGATION_CH5_Mayor_01/14F358D6,Speech,Interrogations,Speech_INTERROGATION_CH5_Mayor_01,,1BB09D3E,,,0.691,44100,1,4876,6c1dee7c8f96436f, BANK,2FD9AB10,Speech_MAP_Deadshot_find_matching_1_CHAR_BATMAN_Deadshot_find_matching_01,Speech_MAP_Deadshot_find_matching_1/2FD9AB10,Speech,Gameplay & hints,Speech_MAP_Deadshot_find_matching_1,INT,139B2B14,,Deadshot; Batman,5.160,44100,1,48628,a7d5135f109446df, BANK,36805E4A,Speech_MAP_Deadshot_find_matching_1_CHAR_BATMAN_Deadshot_find_matching_01,Speech_MAP_Deadshot_find_matching_1/36805E4A,Speech,Gameplay & hints,Speech_MAP_Deadshot_find_matching_1,FRA,139B2B14,,Deadshot; Batman,4.884,44100,1,41222,a6b0e04d967013af, @@ -14817,22 +14784,6 @@ BANK,141EEA20,Speech_SEQUENCE_CH0_CATWOMAN_INTRO_08,Speech_SEQUENCE_CH0_CATWOMAN BANK,0496BF7C,Speech_SEQUENCE_CH0_CATWOMAN_INTRO_08,Speech_SEQUENCE_CH0_CATWOMAN_INTRO/0496BF7C,Speech,Story missions,Speech_SEQUENCE_CH0_CATWOMAN_INTRO,ITA,18D7E643,,Catwoman,1.907,28000,1,14278,e63c8b66f70ecf54, BANK,0A9933FA,Speech_SEQUENCE_CH0_CATWOMAN_INTRO_08,Speech_SEQUENCE_CH0_CATWOMAN_INTRO/0A9933FA,Speech,Story missions,Speech_SEQUENCE_CH0_CATWOMAN_INTRO,DEU,18D7E643,,Catwoman,1.907,28000,1,12844,20e101e6f36702c6, BANK,1BD54DDE,Speech_SEQUENCE_CH0_CATWOMAN_INTRO_08,Speech_SEQUENCE_CH0_CATWOMAN_INTRO/1BD54DDE,Speech,Story missions,Speech_SEQUENCE_CH0_CATWOMAN_INTRO,ESN,18D7E643,,Catwoman,1.763,28000,1,12834,4d0ba4b1479279fe, -BANK,14112E39,Speech_SEQUENCE_CH2_RESCUE_DOC_01,Speech_SEQUENCE_CH2_RESCUE_DOC/14112E39,Speech,Story missions,Speech_SEQUENCE_CH2_RESCUE_DOC,INT,,Batman,Doctor,1.031,44100,1,8548,a75d7bcfa77efdb3, -BANK,045C2B58,Speech_SEQUENCE_CH2_RESCUE_DOC_01,Speech_SEQUENCE_CH2_RESCUE_DOC/045C2B58,Speech,Story missions,Speech_SEQUENCE_CH2_RESCUE_DOC,FRA,,Batman,Doctor,1.071,28000,1,6542,4b3e3f7aa9d1a83b, -BANK,3A312CE8,Speech_SEQUENCE_CH2_RESCUE_DOC_01,Speech_SEQUENCE_CH2_RESCUE_DOC/3A312CE8,Speech,Story missions,Speech_SEQUENCE_CH2_RESCUE_DOC,ITA,,Batman,Doctor,1.075,28000,1,8831,40ffdfe34956c897, -BANK,3808FEFE,Speech_SEQUENCE_CH2_RESCUE_DOC_01,Speech_SEQUENCE_CH2_RESCUE_DOC/3808FEFE,Speech,Story missions,Speech_SEQUENCE_CH2_RESCUE_DOC,DEU,,Batman,Doctor,1.071,28000,1,8140,8e5f47ab89719263, -BANK,3CEB24BB,Speech_SEQUENCE_CH2_RESCUE_DOC_01,Speech_SEQUENCE_CH2_RESCUE_DOC/3CEB24BB,Speech,Story missions,Speech_SEQUENCE_CH2_RESCUE_DOC,ESN,,Batman,Doctor,0.901,28000,1,8881,4f73e5c03f3693de, -BANK,2A2DEC79,Speech_SEQUENCE_CH3_MUSEUM_A1_BM_FREEZESUIT_01,Speech_SEQUENCE_CH3_MUSEUM_A1_BM_FREEZESUIT/2A2DEC79,Speech,Story missions,Speech_SEQUENCE_CH3_MUSEUM_A1_BM_FREEZESUIT,ESN,,,Batman; Mr. Freeze,2.024,28000,1,16548,e1430b84337ad106, -BANK,20FB1929,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH_01,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH/20FB1929,Speech,Story missions,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH,INT,,Penguin,Penguin,2.545,44100,1,21615,8c3745e89fec1a07, -BANK,32FEBF22,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH_01,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH/32FEBF22,Speech,Story missions,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH,FRA,,Penguin,Penguin,2.586,28000,1,17858,b6655fbf823b6a03, -BANK,1C5086AC,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH_01,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH/1C5086AC,Speech,Story missions,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH,ITA,,Penguin,Penguin,2.596,28000,1,18998,5d5e9cec61c04de0, -BANK,057E52DD,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH_01,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH/057E52DD,Speech,Story missions,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH,DEU,,Penguin,Penguin,2.586,28000,1,19398,6e98a57e0eb693c4, -BANK,3B667571,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH_01,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH/3B667571,Speech,Story missions,Speech_SEQUENCE_CH3_MUSEUM_B1_PENGUIN_VENOM_HENCH,ESN,,Penguin,Penguin,2.448,28000,1,18027,b99b40c9639f8983, -BANK,3B3AD732,Speech_SEQUENCE_TWOFACE_END2_05,Speech_SEQUENCE_TWOFACE_END2/3B3AD732,Speech,Story missions,Speech_SEQUENCE_TWOFACE_END2,INT,,Two-Face,Two-Face,0.960,44100,1,7760,ce1badd5de22d5d4, -BANK,3E7514E2,Speech_SEQUENCE_TWOFACE_END2_05,Speech_SEQUENCE_TWOFACE_END2/3E7514E2,Speech,Story missions,Speech_SEQUENCE_TWOFACE_END2,FRA,,Two-Face,Two-Face,1.049,28000,1,6311,99bef837e8302411, -BANK,14CC1583,Speech_SEQUENCE_TWOFACE_END2_05,Speech_SEQUENCE_TWOFACE_END2/14CC1583,Speech,Story missions,Speech_SEQUENCE_TWOFACE_END2,ITA,,Two-Face,Two-Face,0.960,28000,1,6860,f324fe563584eaed, -BANK,28F62D59,Speech_SEQUENCE_TWOFACE_END2_05,Speech_SEQUENCE_TWOFACE_END2/28F62D59,Speech,Story missions,Speech_SEQUENCE_TWOFACE_END2,DEU,,Two-Face,Two-Face,0.960,28000,1,6515,7108d991488807c6, -BANK,32D76E1C,Speech_SEQUENCE_TWOFACE_END2_05,Speech_SEQUENCE_TWOFACE_END2/32D76E1C,Speech,Story missions,Speech_SEQUENCE_TWOFACE_END2,ESN,,Two-Face,Two-Face,1.017,28000,1,7232,3eff15f4f9a7d8f5, BANK,1F7A6D0F,Speech_STEEL_B1_CH2_GRUNT_01,Speech_STEEL_B1_CH2_GRUNT/1F7A6D0F,Speech,Story missions,Speech_STEEL_B1_CH2_GRUNT,INT,18146E8B,,,0.523,44100,1,4294,052817cde90ab5a0, BANK,3BFCB495,Speech_STEEL_B1_CH2_GRUNT_01,Speech_STEEL_B1_CH2_GRUNT/3BFCB495,Speech,Story missions,Speech_STEEL_B1_CH2_GRUNT,FRA,18146E8B,,,0.541,44100,1,4343,40d1717706422c9b, BANK,327E0165,Speech_STEEL_B1_CH2_GRUNT_01,Speech_STEEL_B1_CH2_GRUNT/327E0165,Speech,Story missions,Speech_STEEL_B1_CH2_GRUNT,ITA,18146E8B,,,0.541,44100,1,4343,40d1717706422c9b, diff --git a/data/catalog/BANK_locations.csv.gz b/data/catalog/BANK_locations.csv.gz index 8321136..63b5a36 100644 Binary files a/data/catalog/BANK_locations.csv.gz and b/data/catalog/BANK_locations.csv.gz differ diff --git a/data/catalog/README.md b/data/catalog/README.md index 327ec6c..a63e3eb 100644 --- a/data/catalog/README.md +++ b/data/catalog/README.md @@ -3,7 +3,7 @@ | file | content | |------|---------| | `.csv` | every sound of one audio archive: `INT`, `FRA`, `ITA`, `DEU`, `ESN` (speech WADs, one per language), `SFX` (music, ambience and effects streamed from the SFX WAD) and `BANK` (in-memory sounds stored inside soundbanks: footsteps, foley, combat, weapons, UI, efforts; `docs/formats/banks.md`) | -| `BANK_locations.csv.gz` | every stored copy of the `BANK` sounds, and the prefetch heads of speech lines (below) | +| `BANK_locations.csv.gz` | every stored copy of the `BANK` sounds, and the bank copies of WAD speech recordings (prefetch heads, whole copies; below) | | `SUB_.csv` | subtitles of the speech lines, one file per language (11 languages; `docs/formats/subtitles.md`) | | `SUB_locations.csv.gz` | every stored copy of each subtitle string (dialogue objects in the packages) | | `characters.csv` | `name,character,language`: verified speaker per line name (empty `language`: every language, the same line everywhere; a language code would label a sound of that language only - none needs it now) | @@ -17,7 +17,7 @@ All CSVs are UTF-8 with a header row. They contain only names and metadata (no g |--------|---------| | `archive` | `INT`, `FRA`, ..., `SFX` or `BANK` | | `target` | location: `HASH` (WAD entry with one WEM), `HASH:i` (i-th variation, 0-based), `PACK/MEDIA` (music stream pack), `MEDIA` (BANK: Wwise media ID) | -| `name` | display name. Speech: the line's name, the same in every language (`docs/decisions.md` 5): its speech event (e.g. `Speech_TAPE_JOKER_1_04`, `NAME#2` = second version an event plays) or, for lines no named event plays, the INT recording's position name. A recording stored twice keeps its position name in the second place. SFX: `NAME`, `NAME_1..n`. BANK: the Wwise event that plays it (`Play_..._i`) or `_` | +| `name` | display name. Speech: the line's name, the same in every language (`docs/decisions.md` 5): its speech event (e.g. `Speech_TAPE_JOKER_1_04`, `NAME#2` = second version an event plays) or, for lines no named event plays, the INT recording's position name. A recording stored twice keeps its position name in the second place. SFX: `NAME`, `NAME_1..n`. BANK: the Wwise event that plays it (`Play_..._i`) or `_` (`_` where that name is taken, e.g. by the SFX WAD's position names) | | `aliases` | other accepted names, `\|`-separated: `NAME#i` = i-th WEM of the entry in this language's WAD (positions differ between languages), the entry name for single entries, `/` for BANK | | `group` | Speech, Music, Ambience, SFX, Unknown (`docs/categories.md`) | | `category` | finer category (`docs/categories.md`) | @@ -38,11 +38,11 @@ All CSVs are UTF-8 with a header row. They contain only names and metadata (no g | `file` | file under `BmGame/CookedPCConsole`, `/`-separated (`SFX/xxxxxxxx.bnk` or `Name.upk`) | | `offset` | position of the WEM in the *uncompressed* file (hex; `docs/formats/upk.md`) | | `size` | bytes reserved for it there | -| `kind` | `m` = copy of a BANK sound, `p` = prefetch head of a speech line | -| `sound` | `p` rows: key of that speech line (e.g. `INT:2B151272:0`); each language has its own lines and heads | +| `kind` | `m` = copy of a BANK sound, `p` = copy of (the start of) a WAD speech recording: a prefetch head, or a whole copy with the same media ID | +| `sound` | `p` rows: key of that speech recording (e.g. `INT:2B151272:0`); each language has its own recordings and heads | A replacement is written into every `m` copy of a BANK sound; replacing a speech line also rewrites its `p` -heads (`docs/formats/banks.md`). +copies (`docs/formats/banks.md`). ## `SUB_.csv` columns | column | meaning | diff --git a/data/catalog/index.json b/data/catalog/index.json index f336c49..2933181 100644 --- a/data/catalog/index.json +++ b/data/catalog/index.json @@ -63,20 +63,21 @@ }, "BANK": { "fingerprint": "3f9766af425b8eef", - "sounds": 15404, + "sounds": 15355, "groups": { "Ambience": 1015, "SFX": 6912, - "Speech": 7471, + "Speech": 7422, "Music": 2, "Unknown": 4 }, - "copies": 93910, + "named_by_event": 13501, + "named_by_bank": 1854, + "copies": 93826, "files": 1499, - "named_by_event": 6456, "prefetch_heads": { - "lines": 396, - "copies": 2101, + "recordings": 445, + "copies": 2185, "unmapped": 0 }, "issues": [] @@ -111,6 +112,9 @@ "RUS": 9463 }, "copies": 168688, + "dialogue_objects": 14744, + "dialogue_lines": 9032, + "scripted_objects": 41, "issues": [] }, "LINES": { @@ -121,8 +125,11 @@ "INT", "ITA" ], - "recordings": 61510, + "recordings": 61485, "lines": 12434, + "wad_lines": 10811, + "wad_lines_named_by_event": 10174, + "wad_lines_named_after_int_recording": 637, "lines_in_every_language": 12434, "lines_missing_a_language": {}, "without_line": [], diff --git a/docs/categories.md b/docs/categories.md index 91af39a..61d8326 100644 --- a/docs/categories.md +++ b/docs/categories.md @@ -4,8 +4,9 @@ Implemented in `ackit/classify.py`; used both when building the shipped catalog and when a game build is not in the catalog (runtime scan), so every language/build gets the same scheme. ## Groups and categories -Every sound has a **group** and a finer **category**, derived from its Wwise name (from the community hash -list; unnamed IDs show as hex). First matching rule wins: +Every sound has a **group** and a finer **category**, derived from its name (taken from the game files: the +soundbank names in `CookedBanks.txt` and the Wwise event names; the 3 SFX entries no name reaches show as hex). +First matching rule wins: | name starts with | group | category | |------------------|-------|----------| @@ -78,9 +79,10 @@ alternates speakers between `NAME_1`, `NAME_2`, ...): identified). A line in two lists gets both ("A; B"). - **characters_inferred** - characters and roles *mentioned by the name* (token rules in `classify._TOKENS`, e.g. `BM`/`BATMAN` -> Batman, `PENG` -> Penguin, `TF` -> Two-Face, `THUG*`/`GENTHUG*` -> Thug, `TYGER` -> - TYGER guard; interview tapes add Hugo Strange). A hint for the ~7,500 lines without a verified speaker, + TYGER guard; interview tapes add Hugo Strange). A hint for the ~7,400 lines without a verified speaker, never shown as the speaker. In the editor the character filter matches verified speakers, and optionally inferred ones. -Coverage (every voice language, the same lines): 3,366 with a verified speaker, 6,263 more with inferred -characters, 1,186 with neither (mostly predator chatter and efforts of unnamed thugs). +Coverage of the 10,811 WAD voice lines (every voice language, the same lines): 3,366 with a verified speaker, +6,258 more with inferred characters, 1,187 with neither (combat barks 407, story missions 267, open-world +chatter 228, radio & TV 122, ...). diff --git a/docs/decisions.md b/docs/decisions.md index 18638d8..981a185 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -75,11 +75,13 @@ tables): high risk for little gain, not done. replacements applied now", so reverting the last one restores the original bytes exactly, and an interrupted operation is repaired by the next one. Conflicts: the whole-file SHA-1 must be the original's or the last one ackit wrote. -- **Prefetch heads:** 396 speech lines (about 79 per language) have their first bytes copied into banks; a - replacement of such a line rewrites those heads too, otherwise the start would play the old line. +- **Prefetch heads and whole copies:** 396 voice recordings (81 lines, 78-80 recordings per language) have + their first bytes copied into banks, and 49 more are stored whole in banks (same media ID, identical bytes); + a replacement of such a recording rewrites those copies too, otherwise the game could play the old line. + They are not listed as sounds of their own. - **Names:** no list names these media. Names come from Wwise events whose names are in the packages' name and export tables (FNV-1 of the name = event ID), following event -> action -> containers -> - sounds (6,456 sounds); the rest are numbered per bank. Categories use the bank name. A localized speech line + sounds (13,501 of 15,355, voice lines included); the other 1,854 are numbered per bank. Categories use the bank name. A localized speech line has the same name in every language (its line, section 5). - **Localized speech:** speech banks hold one copy per voice language (4d); every media is labelled with the language(s) whose banks hold it (a few recordings are shared by several languages) and the catalog shows @@ -119,11 +121,11 @@ copy per voice language) is filtered the same way, so the sound list never mixes *Lines*): the recordings that are sources of the same Wwise Sound objects are one line, with exactly one recording per language (12,434 lines, all 5 languages). The catalog stores that identity (`line`), and speaker labels, mod packages (another language: the same `line`) and subtitles follow it. Positions are - useless across languages (WEMs are stored in media-ID order; position i is another line in 75 % of the - multi-variation entries). -- A line has one name in every language: the most specific speech event that plays it (10,156 of 10,815 voice - lines), numbered `NAME#2` in the event's own order when it plays two versions; the other 659 keep the name of - their English (INT) recording's position. The position in each language stays an alias (`NAME#i`). + useless across languages (WEMs are stored in media-ID order; 81.2-81.6 % of the WEMs in multi-variation entries + are at another position than in INT). +- A line has one name in every language: the most specific speech event that plays it (10,174 of the 10,811 + WAD voice lines), numbered `NAME#2` in the event's own order when it plays two versions; the other 637 keep + the name of their English (INT) recording's position. The position in each language stays an alias (`NAME#i`). - A recording stored twice in a WAD (5 per language) is written and reverted in both places. - The catalog records the ORIGINAL bytes: `tools/build_catalog.py` stops before writing anything if a sound differs from the current catalog (a modified game), unless `--accept-changes` (a game update). @@ -174,14 +176,18 @@ copy per voice language) is filtered the same way, so the sound list never mixes ## 8. Verification - `pytest`: unit tests, a synthetic game (WADs, packs, packages with banks, localized banks, dialogue and subtitle objects, localization files), a two-language game built like the real one (per-language BKPK tables, a media - ID shared by both languages, a recording stored twice), offscreen GUI tests, and read-only real-game tests - when the game is installed. + ID shared by both languages, a recording stored twice, one also stored whole in a bank), offscreen GUI + tests, and read-only real-game tests when the game is installed. - `ackit verify` (WEMs rebuilt byte-for-byte, encode/decode), `ackit verify --catalog` (catalog vs game files, and every relation re-derived from the game files with independent checks; it prints them as facts). - Real-game write round trips without launching the game: `tools/e2e_features.py [--lang all]` (every copy of - an in-memory sound, localized line, prefetch heads, subtitles incl. RSubtitle, menu text, sound + subtitle in + an in-memory sound, localized line, prefetch heads and whole bank copies of a line, subtitles incl. RSubtitle, menu text, sound + subtitle in one package, a mod moved to another voice language; `all` = each of the 11 languages) and `tools/e2e_editor.py` (through the editor's code path). +- Documentation: `tools/check_docs.py [--scan] [--wems]` recomputes every figure the documents state (from the + catalog, the build's `index.json`, the menu files; with `--scan` from a fresh package scan and the catalog + check's facts; with `--wems` from rebuilding every WAD WEM) and fails on any difference, so figures are never + copied by hand. - In the running game: `ackit verify --ingame` (music heard), `--banks` (patched packages loaded, probe data in memory), `--texts` (menu texts and Startup subtitles found in memory). Every tool reverts in a `finally` block and checks that every touched file is byte-identical afterwards. diff --git a/docs/formats/banks.md b/docs/formats/banks.md index 3125db1..c5bd471 100644 --- a/docs/formats/banks.md +++ b/docs/formats/banks.md @@ -10,29 +10,31 @@ Survey of the installed game (buildid 3718300), 2026-10-09. | where | banks | unique in-memory WEMs | |-------|-------|-----------------------| | loose `SFX\*.bnk` (Batman/Catwoman footsteps, 32 surfaces each, + `TST_Prep_Media`) | 65 | 1,117 | -| banks embedded in `.upk` packages (in 1,432 of the 2,650 packages) | 3,340 | 15,641 (~276 MB) | +| banks embedded in `.upk` packages (in 1,432 of the 2,650 packages) | 3,340 | 14,802 | All of them are in the catalog (archive `BANK`). The two sets share 119 media IDs: 15,800 unique media IDs in 96,011 places (bank copies). 396 of them are -**prefetch heads** of streamed WAD sounds (below), so there are **15,404 in-memory sounds**. All copies of a +**prefetch heads** of streamed WAD sounds and 49 are **whole copies** of WAD recordings (below), so there are +**15,355 in-memory sounds**. All copies of a media ID are byte-identical except `1C898731` (two variants). The WADs, by comparison, are fully covered: the BKPK tables in the packages reference 2,543 WAD entries, all of them in the catalog. -Embedded WEMs by bank name prefix: `Speech` 7,872, `CHR` (character foley, combat, bosses) 3,896, `PSO` 781, -`WPN` 664, `NPC` 475, `OBJ` 439, `SEQ` 265, `GUI` 208, `PSA` 203, `MTN` 196, `PX` 195, `DVC` 100, `GEN` 80, -`PHY` 73, `AMB` 65, `SBJ` 50, others < 30. The `Speech_` banks (e.g. `Speech_EMOTE_Batman`, `..._GRUNT`) hold -speech: grunts and efforts (language-neutral, 165) and **localized lines** (about 1,460 per language, see -*Localized banks*). -All WEMs seen are Vorbis (format tag 0xFFFF), the same codec as the WADs: the footsteps are mono (4 stereo), -the embedded ones mono or stereo (~22 % stereo in a 2,533-WEM sample). +Embedded WEMs by bank name prefix (a media held by banks of two prefixes counts for both): `Speech` 7,827, `CHR` +(character foley, combat, bosses) 3,624, `PSO` 737, `WPN` 577, `NPC` 471, `OBJ` 411, `SEQ` 253, `PSA` 203, +`PX` 195, `MTN` 191, `GUI` 155, `DVC` 100, `GEN` 80, `PHY` 73, `AMB` 61, `SBJ` 50, others < 30. The `Speech_` banks (e.g. `Speech_EMOTE_Batman`, `..._GRUNT`) hold +speech: efforts, grunts and a few other lines stored once for every language (119 sounds) and **localized +lines** (about 1,450 per language, see *Localized banks*). +All WEMs seen are Vorbis (format tag 0xFFFF), the same codec as the WADs: the loose footsteps are mono (4 +stereo); of all 15,355 in-memory sounds 13,982 are mono, 1,196 stereo, 177 have 3-6 channels. ## Names -- No list names these media IDs (the community hash list names only WAD entries and banks). +- Media IDs have no names of their own: a sound is named after the Wwise event that plays it (event names + are in the packages' name and export tables, event ID = FNV-1 of the name), else after its bank. - `CookedPCConsole\CookedBanks.txt` lists all 3,345 cooked bank names. Bank ID = FNV-1 32 of the lowercased - name (the same hash Wwise uses for every name). 3,338 of them are embedded in packages, 4 are loose `.bnk` - files (`Init`, `Global`, `Music`, `Music_PDLC`); `AMB_OW_A6`, `AMB_OW_C1`, `AMB_OW_E1`, - `Speech_EMOTE_Thug_Muffled_Stop` and `Speech_GFxUI_ArkhamCityStories` were found nowhere. + name (the same hash Wwise uses for every name). 3,338 of them are embedded in packages (`Music`, `Music_PDLC` + also as loose `.bnk` files), 2 are only loose `.bnk` files (`Global`, `Init`); `AMB_OW_A6`, `AMB_OW_C1`, + `AMB_OW_E1`, `Speech_EMOTE_Thug_Muffled_Stop` and `Speech_GFxUI_ArkhamCityStories` were found nowhere. - Loose footstep banks are named in their STID chunk (`CHR_BatmanFootsteps_Concrete_Media`, ...). ## Bank layout (version 56, Wwise 2011.1) @@ -51,6 +53,12 @@ with bank id 0 whose media is not in the same bank (loaded by another bank); 133 (`wad.md`). A prefetch source's media is also in the bank's DIDX: that in-memory copy is the **head** of the streamed WEM (its size = the source's size), which the game plays while the rest is read from the WAD. 396 WAD sounds have such heads (2,101 copies); replacing one of them must rewrite the heads too. +49 more WAD recordings (9-11 per language: game-over taunts, an encryption-card line, a cutscene line, scripted +story lines) are +also stored **whole** in banks: an in-memory source with the WAD recording's own media ID, byte-identical to +the WAD WEM (84 copies). They are handled like heads: part of the WAD line, written and reverted with it, not +sounds of their own (`catalog._whole_recordings`). Heads and whole copies together: 445 recordings, 2,185 +copies (`BANK_locations.csv.gz` kind `p`). ## Localized banks Speech banks are localized inside the (language-neutral) packages: one `AkBank` export starts with a language @@ -58,9 +66,10 @@ list (FStrings `INT FRA ITA DEU ESN`) and then holds one block per language, in of the speech WAD entry for that language (`wad.md`) and a complete bank, with the same bank ID in every language. ackit gives each bank and each BKPK table the language of its position in that list (an export whose block count differs from its language count would be reported; none does). Each in-memory media gets -the languages of the banks that hold it (`BANK.csv` column `language`): 8,098 sounds are language-neutral, -~1,460 belong to one voice language, 6 to several (a DLC intro line shared by INT/FRA/ITA/ESN while DEU has -its own); a media stored in the banks of every language is listed as `''` (the same recording everywhere). +the languages of the banks that hold it (`BANK.csv` column `language`): 8,092 sounds are language-neutral, +1,450-1,456 belong to one voice language, 6 to several (5 shared by INT/FRA/ITA/ESN while DEU has its own, 1 +by INT/FRA/ITA); a media stored in the banks of every language is listed as `''` (the same recording +everywhere; 167 of these are voice lines, e.g. efforts and radio chatter). A catalog shows the neutral sounds plus the loaded language's lines; names are unique per language. ## Lines: one recording per language @@ -73,17 +82,18 @@ and the game ties the recordings together itself: - Every recording, streamed, prefetched or in memory, is the source of a Wwise Sound object; Sound objects, events, actions and containers have the same IDs in every language, only the media differ. A *line* = the recordings that are sources of the same Sound objects: 12,434 lines (WAD and in-memory), each with exactly - one recording in each of the 5 voice languages (61,510 recordings), no Sound object with two recordings in + one recording in each of the 5 voice languages (61,485 recordings), no Sound object with two recordings in one language. Line identity (`line` column) = the smallest Sound object ID. -- Prefetched sources count too: the 396 lines with a prefetch head (mostly the game-over taunts) are linked - only through them. +- Prefetched sources count too: the 81 lines with a prefetch head (396 recordings over the 5 languages; mostly + the game-over taunts) are linked only through them. A WAD recording that a bank also stores whole (49) is + the same recording (same media ID), so its line is unchanged and the bank copies are written with it. - 5 recordings per language are stored twice in that language's WAD (same media ID in two entries, byte- identical, e.g. `Speech_STEEL_A1_CH2_Thug_behinddoormuffled` and `DIA_SteelA1Muffled`): the second place is a copy of the line (same `line`, keeps its position name) and is written and reverted with it, because the game plays whichever place the loaded package's table names. - Independent checks (no BKPK involved): every WAD recording is named as a source by a Sound object of a bank - of its own language (54,055 of 54,055); every stored prefetch head equals the start of the recording its - line gives it (2,101 of 2,101); a line's recordings have similar lengths across languages (11,761 of 12,434 + of its own language (54,055 of 54,055); every stored head or whole copy equals the start of the recording + its line gives it (2,185 of 2,185); a line's recordings have similar lengths across languages (11,761 of 12,434 within a factor 1.5; the others are different edits, e.g. a 0.55 s ESN line vs 5.46 s elsewhere). ## Banks inside packages @@ -99,10 +109,12 @@ and the game ties the recordings together itself: ## How ackit edits them - Catalog archive `BANK` (one row per media ID) + `data/catalog/BANK_locations.csv.gz` (every copy, and the - prefetch heads of speech lines); built by `tools/build_catalog.py`, checked by `ackit verify --catalog`. + heads and whole copies of WAD speech recordings); built by `tools/build_catalog.py`, checked by `ackit verify + --catalog`. - A replacement is written **in place into every copy** (no larger than the smallest copy, zero-padded), so DIDX/HIRC offsets and sizes and the package layout never change. Loose banks are patched directly; in packages only the touched blocks are re-encoded (`upk.md`). A sound left in one package only would let the game load the original from another one. -- Replacing a speech line that has prefetch heads rewrites the heads (first bytes of the new, zero-padded line). +- Replacing a speech line that has prefetch heads or whole copies rewrites them (the first bytes of the new, + zero-padded line; for a whole copy, all of it). - Design and revert rules: `docs/decisions.md` (4b). The same WEM rules apply (`wem.md`). diff --git a/docs/formats/subtitles.md b/docs/formats/subtitles.md index f915757..1035d84 100644 --- a/docs/formats/subtitles.md +++ b/docs/formats/subtitles.md @@ -5,14 +5,15 @@ the UE3 packages (`upk.md`), holding the subtitle text of all 11 languages. acki `ackit/subtitles.py` and writes them through `Workspace.texts` (`ackit/texts.py`). ## Inventory (buildid 3718300) -- 14,743 dialogue objects in the packages for 9,016 speech lines (a line's object is copied into every package +- 14,744 dialogue objects in the packages for 9,032 speech lines (a line's object is copied into every package that uses it; all copies are identical), plus 41 scripted subtitles (below): ~9,400 subtitle strings and 168,688 stored copies. - 11 languages: INT, FRA, ITA, DEU, ESN, ESM, POL, POR, RUS, JPN, KOR (the speech WADs exist only for INT, FRA, ITA, DEU, ESN; the other languages play the INT voices with their own subtitles). -- ~9,400 subtitle strings per language: 94 % of the subtitles have one line, the rest up to 5 lines. -- Strings are stored as cp1252 (positive FString length) or UTF-16 (negative length): RUS, POL, JPN, KOR - and some INT/ESN/FRA/DEU strings are UTF-16. +- 9,035-9,667 subtitle strings per language: 94-98 % of the subtitles have one line (JPN: all), the rest up + to 5 lines. +- Strings are stored as cp1252 (positive FString length) or UTF-16 (negative length): almost all RUS, JPN and + KOR strings and 85 % of POL are UTF-16; the other languages have 17-56 UTF-16 strings each. ## Scripted subtitles (`RSubtitle`) 41 more subtitles (radio and encryption-card lines) are `RSubtitle` objects in `Startup.upk`: the same layout, @@ -34,7 +35,7 @@ or property name from the name table, i32 byte size, i32 array index, then `size | `0x09 LocalizedSubtitles` | u32 count, then per language: `0D 00 00 00` FString language code, `09 00 0C 00 Subtitles` struct, `00 00` | | `Subtitles` | u32 line count, then per line: `0D 00 00 00` FString text, `04 00 0C 00` 4-byte value, `00 00` | -The sizes add up exactly in all 14,743 objects (checked by the parser: `SubtitleError` otherwise). +The sizes add up exactly in all 14,744 objects (checked by the parser: `SubtitleError` otherwise). ## Link to the speech sound The speech event name is hashed (FNV-1) to the Wwise event ID and followed through the soundbanks (event -> @@ -46,7 +47,7 @@ export names) and from the subtitle objects themselves: some events are listed i dialogue / RSubtitle object names them (Vicki Vale's radio reports, the Hush tape reveal). An event that plays two versions of a line (105 subtitled events, e.g. Mr. Freeze's lines) links its subtitle to both (`sound` = `key1;key2`). Linked: 103,304 of 103,658 strings in the 11 languages (INT 9,373 / 9,404), -each to the same line in every language. The rest (29 speech names, ~32 strings per language: Riddler trophy +each to the same line in every language. The rest (24 speech names, 24-36 strings per language: Riddler trophy texts, Arkham City Stories, radio reports and tapes the game does not ship audio events for) have no event in any bank and stay editable without a sound link. diff --git a/docs/formats/wad.md b/docs/formats/wad.md index 6889ebd..2b39f01 100644 --- a/docs/formats/wad.md +++ b/docs/formats/wad.md @@ -29,8 +29,9 @@ Payload kinds: - **LOC WADs**: each entry is a group of WEMs, the variations of one line (every language: 10,816 WEMs in 2,464 entries; e.g. `CHR_RadioScanner` holds 91). Each WEM starts at the next 0x800 boundary after the previous one (zero padding); 3 entries pack two WEMs back to back. Addressed `HASH:i`. The WEMs of an - entry are stored in media-ID order and the languages have different media IDs, so variation `i` is a - different line in each language (75 % of the lines). Which recording is which line comes from the banks + entry are stored in media-ID order and the languages have different media IDs, so variation `i` is + usually a different line in each language (81.2-81.6 % of the WEMs in multi-variation entries, compared with + INT; e.g. `Speech_TAPE_JOKER_1_04` is variation 1 in INT and ITA, 8 in FRA, 6 in DEU, 5 in ESN). Which recording is which line comes from the banks (`banks.md`, *Lines*), not from positions. - **SFX WAD**: 79 grouped ambience/effect entries (260 WEMs) plus 4 **stream packs** (below). diff --git a/docs/formats/wem.md b/docs/formats/wem.md index 142e401..f538c40 100644 --- a/docs/formats/wem.md +++ b/docs/formats/wem.md @@ -3,10 +3,14 @@ Derived from the game's WEMs; `ackit verify` / `tests/test_realgame.py` rebuild them byte-for-byte from these rules alone. Checked on every WEM of the 5 speech WADs: 54,079 of 54,080 are identical; the exception (INT `FE9963E8:74`) differs only in the informational avg-bytes/s field (one lower than the formula below). In SFX -only the 4 looping music tracks with loop points differ. +626 of 651 are identical; the others are 14 music tracks whose seek table follows another placement (one entry per +1,024-1,103 samples), 5 music tracks whose sample count exceeds what their packets hold, and 6 4-channel +ambience WEMs with a `smpl` (loop) chunk. These rules are not needed to write sounds the game plays (replaced +music, built with the rules here, is heard in game: `ackit verify --ingame`), only to reproduce those originals. ## RIFF layout -`RIFF` size `WAVE`, then `fmt ` (0x42 bytes), optional `cue `/`LIST` (37 files), `data`. +`RIFF` size `WAVE`, then `fmt ` (0x42 bytes), optional `LIST` + `cue ` (659 WEMs), `LIST` + `cue ` + `smpl` (1 WEM), +`cue ` (18 WEMs), `cue ` + `smpl` (3 WEMs), `smpl` (41 WEMs), `data`. Odd-sized `data` chunks are **not** padded. ### `fmt ` (0x42) diff --git a/docs/reverse-engineering/findings.md b/docs/reverse-engineering/findings.md index 1549ee6..e633e52 100644 --- a/docs/reverse-engineering/findings.md +++ b/docs/reverse-engineering/findings.md @@ -70,7 +70,7 @@ Conclusion: approach A works end-to-end in-game. Use >= 30 s probes for the intr expects WAVEFORMATEX order (FL FR FC LFE BL BR) is unverified (identical-channel probes were used). 16. (2026-10-09) The WADs hold only streamed audio. About 15,800 more sounds (footsteps, foley, combat, weapons, UI, grunts) are in-memory WEMs inside soundbanks: 65 loose `SFX\*.bnk` (1,117) and 3,340 banks embedded in - the `.upk` packages (15,641; many banks copied into several packages). None were in the catalog then (they + the `.upk` packages (14,802; many banks copied into several packages). None were in the catalog then (they are now: archive BANK); the WAD side was complete. Inventory and layout: `docs/formats/banks.md`. 17. (2026-10-09) Speech lines have **per-language media IDs**: every package carries one BKPK table per language for the same WAD entry. 396 lines (78-80 per language) have a prefetch head (first N bytes) in banks; each @@ -95,12 +95,14 @@ Conclusion: approach A works end-to-end in-game. Use >= 30 s probes for the intr Position names (NAME_i, wemscan) therefore mean different lines per language. Speech sounds are now named after their speech event (same in every language); the curated ITA lists are mapped through it. Independent check: the speaker field of the dialogue objects agrees with the mapped labels for 955 / 955 lines. - (Superseded by 22: the identity of a line now comes from the Sound objects, for every recording.) + (Superseded by 22: the identity of a line now comes from the Sound objects, for every recording; measured + that way, 81.2-81.6 % of the WEMs in multi-variation entries are at another position than in INT.) 21. (2026-10-10) In-memory speech banks are localized (language list + one bank per language in one AkBank export, checked on 215 banks against the WAD languages of their streamed lines); 41 more subtitles are RSubtitle objects linked through RSubtitleLookup (key = FNV-1 of the line name without `Speech_`). Subtitle coverage by category is uneven in the game itself: efforts 0 %, combat barks 11 %, cinematics 10 % (cutscene lines are mostly not reached by speech events and have no dialogue object), story/predator/open world ~90 %+. + (With every version of a line linked, item 22: combat barks 9 %, cinematics 23 %; REPORT.md.) 22. (2026-10-10) **One recording per language, from the game data alone** (`docs/formats/banks.md`, *Lines*). The BKPK tables of a speech entry sit inside the localized AkBank export, one per declared language; they @@ -113,6 +115,16 @@ Conclusion: approach A works end-to-end in-game. Use >= 30 s probes for the intr recording in each of the 5 languages, no conflict; independent checks: 54,055 / 54,055 WAD recordings are sources in a bank of their own language, 2,101 / 2,101 prefetch heads equal the start of their recording, every linked subtitle is on the same line in all 11 languages. +23. (2026-10-10, final check) **Whole WAD recordings in banks.** 49 in-memory media (9-11 per language) have the + media ID of a WAD recording of their bank's language and are byte-identical to it (game-over taunts, + `Speech_CH_ZERO_CH0_ENCRYPTION_CARD3_01`, `Speech_SEQUENCE_CH2_RESCUE_DOC_01`, ...): the line's Sound + object names that media, and some bank also stores the whole WEM in memory. They were listed as separate BANK sounds (with the + line's name: the CLI called the name ambiguous), so a replacement of the line left the in-memory copies + old. Now they are bank copies of the WAD recording like prefetch heads (84 copies; 445 recordings with + heads or whole copies, 2,185 copies), written and reverted with the line. Also: 8 in-memory sounds per + catalog had the same `_` name as an SFX WAD sound of the same bank (`CHR_RasBossFight_1`, + `SEQ_DemonTrials_1`); they now take `_`, and a line's own name wins over another recording's + position alias (`CHR_RadioScanner#1` in FRA). Every name of every catalog resolves to its own sound. ## Old path -> new path `scripts/harness.py` -> `ackit verify --ingame` (ackit/verify/ingame.py); `scripts/validate_builder.py` -> diff --git a/docs/user-guide.md b/docs/user-guide.md index 1418093..8012e49 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -48,7 +48,7 @@ the same line and are changed and reverted together. - Click a column header to sort (length, size, status, ...). Ctrl/Shift-click selects several rows. - Archive **BANK** holds the short sounds stored inside soundbanks (footsteps, foley, combat, weapons, menu sounds, grunts). Many are named after the game event that plays them (`Play_...`), the - rest after their bank (`CHR_Bane_3`). The editor shows the bank and how many copies the game keeps of + rest after their bank (`CHR_BatclawGrapple_3`). The editor shows the bank and how many copies the game keeps of the sound; a replacement is written into every copy. ![Browsing by category](img/browse.png) @@ -92,8 +92,10 @@ The **Subtitles** tab lists the subtitle of every speech line in the selected la - Search by text or line name, filter by **Character** and **Category** (taken from the linked speech sound), or show **Modified only**. Select a row to edit its text below and press **Apply** (or Enter). - Each subtitle replaces the original *in place*, so it can be at most as long as the original (the counter - shows *used / room* characters). Latin-1 subtitles accept Latin-1 characters only (accents are fine); - Russian, Polish, Japanese and Korean subtitles accept any character. + shows *used / room* characters). Most subtitles are stored in the Western European character set + (Windows-1252: accents are fine, other scripts are not); those stored as Unicode (almost all Russian, + Japanese and Korean ones, most Polish ones) accept any character. The editor says so when a character does + not fit. - **Go to the sound** opens the speech line in the Sounds tab. In the Sounds tab, the editor of a speech line shows its subtitle in the **Subtitle** column next to the audio: edit both side by side, **Apply** / **Revert** the subtitle there, or **Show in list** to jump back to the Subtitles tab. diff --git a/pyproject.toml b/pyproject.toml index 27fd014..d4ee5fc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "ackit" -version = "3.2.0" +version = "3.2.1" description = "Batman: Arkham City (PC) audio toolkit and editor" readme = "README.md" requires-python = ">=3.10" diff --git a/tests/test_catalog.py b/tests/test_catalog.py index 269cc56..c345ab4 100644 --- a/tests/test_catalog.py +++ b/tests/test_catalog.py @@ -1,4 +1,5 @@ """Catalog: classification rules, naming, character labels, search; the shipped CSVs.""" +import collections import csv import os @@ -104,6 +105,57 @@ def test_shipped_csvs_are_consistent(): assert all(l.sound for ls in locs.values() for l in ls if l.kind == "p") +def test_shipped_catalog_cross_references(): + """Every reference between the shipped files points to something that exists, and every line has one + recording and one name per language (copies aside).""" + from ackit import subtitles + voices = ["INT", "FRA", "ITA", "DEU", "ESN"] + rows = {a: read_csv(os.path.join(catalog_dir(), a + ".csv")) for a in voices + ["SFX", "BANK"]} + keys = {s.key: s for r in rows.values() for s in r} + bank = rows["BANK"] + # subtitles -> a voice recording of that language (INT voices for the others) + for f in os.listdir(catalog_dir()): + if f.startswith("SUB_") and f.endswith(".csv"): + lang = f[4:7] + voice = lang if lang in voices else "INT" + for s in subtitles.read_texts(os.path.join(catalog_dir(), f), lang): + for k in filter(None, s.sound.split(";")): + assert k in keys, (f, s.speech, k) + snd = keys[k] + assert snd.archive == voice or (snd.archive == "BANK" and snd.in_language(voice)), (f, k) + # heads of WAD recordings -> a WAD recording, never larger than it; no BANK sound is also a head + locs = banks.read_locations(os.path.join(catalog_dir(), banks.LOCATIONS)) + for m, ls in locs.items(): + for l in ls: + if l.kind == "p": + assert l.sound in keys and keys[l.sound].archive in voices and l.size <= keys[l.sound].size, l + assert "BANK:%08X" % m not in keys + # speaker labels -> line names that exist in every voice language + names = {L: {s.name for s in rows[L]} for L in voices} + with open(os.path.join(catalog_dir(), "characters.csv"), encoding="utf-8") as f: + for r in csv.DictReader(f): + assert all(r["name"] in names[L] for L in ([r["language"]] if r["language"] else voices)), r + # one recording per language per line (copies: a second place of the same recording, same line) + for L in voices: + per = collections.defaultdict(list) + for s in rows[L] + [b for b in bank if b.line and b.in_language(L)]: + if s.line: + per[s.line].append(s) + for lid, ss in per.items(): + assert len({s.sha1 for s in ss}) == 1, (L, lid, [s.key for s in ss]) # copies are identical + assert len([s for s in ss if s.archive == "BANK"]) <= 1, (L, lid) + # the same line has the same name in every language (the second place of a copy keeps its position name) + common = {} + for L in voices: + per = collections.defaultdict(set) + for s in rows[L]: + if s.line: + per[s.line].add(s.name) + for lid, ns in per.items(): + common[lid] = common[lid] & ns if lid in common else ns + assert all(common.values()), [k for k, v in common.items() if not v][:5] + + @pytest.mark.game def test_real_game_uses_shipped_catalog(real_game_dir): g = Game(real_game_dir) @@ -114,6 +166,17 @@ def test_real_game_uses_shipped_catalog(real_game_dir): assert all(s.line for s in cat.sounds if s.archive == "ITA" and s.group == "Speech") # every voice line s = cat.resolve("Speech_TAPE_JOKER_1#4") # position 4 in the ITA entry (the user's list name) assert s.category == "Interview tapes" and s.size == 44985 and s.name.startswith("Speech_TAPE_JOKER_1_") - assert len(cat.heads) == 396 and sum(len(v) for v in cat.heads.values()) == 2101 + # bank copies of WAD recordings: 396 prefetch heads + 49 recordings also stored whole in banks + assert len(cat.heads) == 445 and sum(len(v) for v in cat.heads.values()) == 2185 b = cat.resolve("Play_BatmanDeviceImpact_Wood_1") assert b.archive == "BANK" and b.group == "SFX" + + +@pytest.mark.game +def test_every_name_resolves_to_its_sound(real_game_dir): + """A line named after its INT position (CHR_RadioScanner#1) can equal another recording's position alias + in another language: the name wins, in every language.""" + g = Game(real_game_dir) + for lang in g.languages: + cat = Catalog(g, [lang, "SFX", "BANK"]) + assert all(cat.resolve(s.name) is s for s in cat.sounds), lang diff --git a/tests/test_lines.py b/tests/test_lines.py index 1171918..3cb82b8 100644 --- a/tests/test_lines.py +++ b/tests/test_lines.py @@ -37,7 +37,8 @@ def _offsets(wems): return out -def make(root): +def make(root, whole_copy=False): + """whole_copy: INT's bank also stores INT's recording of line A whole, in memory (same media ID).""" cooked = os.path.join(root, "BmGame", "CookedPCConsole") a = {"INT": synth.wem(0.6, 400), "ITA": synth.wem(0.7, 420)} b = {"INT": synth.wem(1.1, 600), "ITA": synth.wem(0.9, 640)} @@ -61,7 +62,8 @@ def make(root): bank_id = fnv1("Speech_Lines") for L in ("INT", "ITA"): blob += _table(ENTRY, rows[L]) - blob += synth.bank(bank_id, {}, name="Speech_Lines", events=[(EV_A, A[L]), (EV_B, B)]) + bytes(16) + mem = {A[L]: a[L]} if whole_copy and L == "INT" else {} + blob += synth.bank(bank_id, mem, name="Speech_Lines", events=[(EV_A, A[L]), (EV_B, B)]) + bytes(16) # the copy entry's table sits in a second localized export (INT and ITA; ITA has no such entry: no rows) blob2 = bytes(4) + struct.pack("= 3))) + + # copies and heads + with gzip.open(os.path.join(cat, "BANK_locations.csv.gz"), "rt", newline="") as f: + loc = list(csv.DictReader(f)) + m_rows = [r for r in loc if r["kind"] == "m"] + p_rows = [r for r in loc if r["kind"] == "p"] + keys = {s.key: s for L in VOICES for s in rows[L]} + whole = [r for r in p_rows if int(r["size"]) >= keys[r["sound"]].size] + pref = [r for r in p_rows if r not in whole] + whole_rec = {r["sound"] for r in whole} + pref_rec = {r["sound"] for r in pref} + pref_lines = {keys[k].line for k in pref_rec} + all_lines = {keys[r["sound"]].line for r in p_rows} + per_lang_pref = collections.Counter(k.split(":")[0] for k in pref_rec) + per_lang_whole = collections.Counter(k.split(":")[0] for k in whole_rec) + claim("REPORT.md", "copy (%s copies). %s WAD voice recordings (%d lines) also have bank copies (%d prefetch heads, %d " + "whole copies; %s copies)" % (n(len(m_rows)), n(len(pref_rec | whole_rec)), len(all_lines), len(pref_rec), + len(whole_rec), n(len(p_rows)))) + claim("docs/formats/banks.md", "%d of them are **prefetch heads** of streamed WAD sounds and %d are **whole copies**" + % (len({r["media"] for r in pref}), len({r["media"] for r in whole}))) + claim("docs/formats/banks.md", "%d WAD sounds have such heads (%s copies)" % (len(pref_rec), n(len(pref)))) + claim("docs/formats/banks.md", "%d more WAD recordings (%s per language" % (len(whole_rec), rng(per_lang_whole.values()))) + claim("docs/formats/banks.md", "byte-identical to the WAD WEM (%d copies)" % len(whole)) + claim("docs/formats/banks.md", "Heads and whole copies together: %d recordings, %s copies" + % (len(pref_rec | whole_rec), n(len(p_rows)))) + claim("docs/formats/banks.md", "the %d lines with a prefetch head (%d recordings over the 5 languages" + % (len(pref_lines), len(pref_rec))) + claim("docs/formats/banks.md", "A WAD recording that a bank also stores whole (%d)" % len(whole_rec)) + claim("docs/decisions.md", "%d voice recordings (%d lines, %s recordings per language) have their first bytes" + % (len(pref_rec), len(pref_lines), rng(per_lang_pref.values(), str))) + claim("docs/decisions.md", "and %d more are stored whole in banks" % len(whole_rec)) + places = len(loc) + media_all = {r["media"] for r in loc} + loose = {r["media"] for r in loc if r["file"].startswith("SFX/")} + pk = {r["media"] for r in loc if not r["file"].startswith("SFX/")} + claim("docs/formats/banks.md", "The two sets share %d media IDs: %s unique media IDs in %s places" + % (len(loose & pk), n(len(media_all)), n(places))) + claim("docs/formats/banks.md", "+ `TST_Prep_Media`) | %d | %s |" + % (len({r["file"] for r in loc if r["file"].startswith("SFX/")}), n(len(loose)))) + claim("docs/formats/banks.md", "packages) | 3,340 | %s |" % n(len(pk))) + claim("REPORT.md", "in %d loose `.bnk` files and %d `.upk` packages" + % (len({r["file"] for r in loc if r["file"].startswith("SFX/")}), len({r["file"] for r in loc if r["file"].endswith(".upk")}))) + assert len(media_all) == len(bank) + len(pref_rec | whole_rec) + + # lines + lines_all = {s.line for L in VOICES for s in rows[L] if s.line} | {s.line for s in bank if s.line} + wad_lines = {L: {s.line for s in rows[L] if s.line} for L in VOICES} + assert len({len(v) for v in wad_lines.values()}) == 1 + WL = len(wad_lines["INT"]) + copies = {L: sum(1 for s in rows[L] if s.line) - len(wad_lines[L]) for L in VOICES} + recs = sum(len(wad_lines[L]) for L in VOICES) + len({(s.target) for s in bank if s.line}) + for doc, text in [("REPORT.md", "%s lines (WAD + in-memory) with exactly one recording in each of the 5 voice languages"), + ("docs/decisions.md", "recording per language (%s lines, all 5 languages)"), + ("docs/formats/banks.md", "recordings that are sources of the same Sound objects: %s lines")]: + claim(doc, text % n(len(lines_all)), "distinct line ids") + claim("docs/formats/banks.md", "in each of the 5 voice languages (%s recordings)" % n(recs)) + claim("REPORT.md", "recordings stored twice (%s per language) are written together" % rng(copies.values(), str)) + claim("docs/formats/banks.md", "- %s recordings per language are stored twice" % rng(copies.values(), str).capitalize()) + claim("docs/categories.md", "Coverage of the %s WAD voice lines" % n(WL)) + claim("docs/decisions.md", "of the %s WAD voice lines" % n(WL)) + + # positions differ between languages + it = {s.target: s.line for s in rows["INT"] if ":" in s.target} + diff = [] + for L in ["FRA", "ITA", "DEU", "ESN"]: + other = {s.target: s.line for s in rows[L] if ":" in s.target} + diff.append(100.0 * sum(1 for t in it if other.get(t) != it[t]) / len(it)) + span = "%.1f-%.1f %%" % (min(diff), max(diff)) + claim("docs/formats/wad.md", "(%s of the WEMs in multi-variation entries, compared with INT" % span) + claim("docs/decisions.md", "%s of the WEMs in multi-variation entries are at another position than in INT" % span) + pos = {} + for L in VOICES: + s = next(s for s in rows[L] if s.name == "Speech_TAPE_JOKER_1_04") + pos[L] = int(s.target.split(":")[1]) + claim("docs/formats/wad.md", "`Speech_TAPE_JOKER_1_04` is variation %d in INT and ITA, %d in FRA, %d in DEU, %d in ESN" + % (pos["INT"], pos["FRA"], pos["DEU"], pos["ESN"]) if pos["INT"] == pos["ITA"] else "(INT != ITA)") + ents = collections.Counter(s.target.split(":")[0] for s in rows["INT"]) + claim("docs/formats/wad.md", "(every language: %s WEMs in %s entries" % (n(W), n(len(ents)))) + radio = [s for s in rows["INT"] if s.aliases and any(a.startswith("CHR_RadioScanner#") for a in s.names()) + or s.name.startswith("CHR_RadioScanner#")] + claim("docs/formats/wad.md", "e.g. `CHR_RadioScanner` holds %d" % len({s.target for s in radio})) + + # SFX WAD + packs = collections.Counter(s.target.split("/")[0] for s in sfx if "/" in s.target) + grouped = [s for s in sfx if "/" not in s.target] + claim("docs/formats/wad.md", "**SFX WAD**: %d grouped ambience/effect entries (%d WEMs) plus %d **stream packs**" + % (len({s.target.split(":")[0] for s in grouped}), len(grouped), len(packs))) + claim("docs/formats/wad.md", "`EDF036D6` (Music, %d WEMs" % packs["EDF036D6"]) + claim("docs/formats/wad.md", "`E8F980BE` (Music_PDLC, %d), `57572DAC` (Global, %d), `FCA4F778` (%d)" + % (packs["E8F980BE"], packs["57572DAC"], packs["FCA4F778"])) + + # categories + cats = [collections.Counter(s.category for s in rows[L]) for L in VOICES] + assert all(c == cats[0] for c in cats), "category counts differ between languages" + order = ["Story missions", "Predator chatter", "Character lines", "Open-world chatter", "Side missions", + "Combat barks", "Efforts & emotes", "Interview tapes", "Riddler", "Radio & TV", "Gameplay & hints", "DLC", + "Boss fights", "Cinematics", "Reactions", "Interrogations"] + g = collections.Counter(s.group for s in sfx) + bc = collections.Counter(s.category for s in bank).most_common(8) + claim("docs/categories.md", "Counts per voice language (identical in all 5): " + ", ".join( + "%s %d" % (c, cats[0][c]) for c in order) + ". SFX archive: " + ", ".join( + "%s %d" % (k, g[k]) for k in ["Music", "Ambience", "SFX", "Unknown"]) + ". BANK (all languages): " + ", ".join( + "%s %d" % kv for kv in bc) + ", ...") + + # characters (by line) + first = {} + for s in rows["INT"]: + first.setdefault(s.line, s) + wc = sum(1 for s in first.values() if s.character) + inf = sum(1 for s in first.values() if not s.character and s.characters_inferred) + nei = [s for s in first.values() if not s.character and not s.characters_inferred] + top = collections.Counter(s.category for s in nei).most_common(4) + assert all(sum(1 for s in rows[L] if s.character) == sum(1 for s in rows["INT"] if s.character) for L in VOICES) + claim("docs/categories.md", "(every voice language, the same lines): %s with a verified speaker, %s more with inferred " + "characters, %s with neither (%s, ...)" % (n(wc), n(inf), n(len(nei)), ", ".join( + "%s %d" % (c.lower().replace("radio & tv", "radio & TV"), k) for c, k in top))) + claim("docs/categories.md", "A hint for the ~%s lines without a verified speaker" % n(int(round(WL - wc, -2)))) + claim("REPORT.md", "speaker labels (%s in every language)" % n(wc)) + + # subtitles + langs = sorted(f[4:7] for f in os.listdir(cat) if f.startswith("SUB_") and f.endswith(".csv")) + texts = {L: subtitles.read_texts(os.path.join(cat, "SUB_%s.csv" % L), L) for L in langs} + total = sum(len(t) for t in texts.values()) + linked = sum(1 for t in texts.values() for s in t if s.sound) + i_tot, i_l = len(texts["INT"]), sum(1 for s in texts["INT"] if s.sound) + for doc in ("REPORT.md", "docs/formats/subtitles.md"): + claim(doc, "%s of %s strings in %d languages (INT %s" % (n(linked), n(total), len(langs), n(i_l)) + if doc == "REPORT.md" else "Linked: %s of %s strings in the %d languages (INT %s / %s)" + % (n(linked), n(total), len(langs), n(i_l), n(i_tot))) + claim("REPORT.md", "(INT %s of %s)" % (n(i_l), n(i_tot))) + unl = {L: {s.speech for s in t if not s.sound} for L, t in texts.items()} + unl_n = {len(v) for v in unl.values()} + assert len(unl_n) == 1, unl_n + un_strings = [sum(1 for s in t if not s.sound) for t in texts.values()] + claim("REPORT.md", "%d speech names (%s strings per language, INT %d" % (unl_n.pop(), rng(un_strings), + i_tot - i_l)) + claim("docs/formats/subtitles.md", "(%d speech names, %s strings per language" % (len(unl["INT"]), rng(un_strings))) + two = {s.speech for s in texts["INT"] if ";" in s.sound} + claim("docs/formats/subtitles.md", "(%d subtitled events" % len(two)) + one = [] + for L, t in texts.items(): + c = collections.Counter(s.speech for s in t) + one.append(100.0 * sum(1 for v in c.values() if v == 1) / len(c)) + claim("docs/formats/subtitles.md", "%s subtitle strings per language: %d-%d %% of the subtitles have one line (JPN: all)" + % (rng([len(t) for t in texts.values()]), int(min(one)), int(round(max(x for x in one if x < 100))))) + enc = {L: collections.Counter(s.enc for s in t) for L, t in texts.items()} + pol = pct(enc["POL"]["w"], len(texts["POL"])) + west = [enc[L]["w"] for L in langs if L not in ("RUS", "JPN", "KOR", "POL")] + claim("docs/formats/subtitles.md", "%d %% of POL are UTF-16; the other languages have %s UTF-16 strings each" + % (pol, rng(west))) + with gzip.open(os.path.join(cat, subtitles.LOCATIONS), "rt", newline="") as f: + sublocs = sum(1 for _ in csv.DictReader(f)) + claim("REPORT.md", "Edited in place in every copy (%s copies)" % n(sublocs)) + claim("docs/formats/subtitles.md", "%s stored copies" % n(sublocs)) + claim("docs/user-guide.md", "About %d subtitles per language have no sound" % int(round(sum(un_strings) / len(un_strings), -1))) + + # subtitle coverage of the INT voice lines + lk = {k for s in texts["INT"] for k in s.sound.split(";") if k} + cov, tot = collections.Counter(), collections.Counter() + for s in rows["INT"] + [b for b in bank if b.line and "INT" in b.language.split(";")]: + if s.line: + tot[s.category] += 1 + cov[s.category] += s.key in lk + p = {c: pct(cov[c], tot[c]) for c in tot} + claim("REPORT.md", "few for combat barks (%d %%), cinematics and boss fights (%d %%;" + % (p["Combat barks"], p["Cinematics"]) if p["Cinematics"] == p["Boss fights"] else "(cinematics != boss)") + hi = [p[c] for c in ("Story missions", "Predator chatter", "Character lines", "Open-world chatter")] + claim("REPORT.md", "%d-%d %% for story, predator, character and open-world lines" % (min(hi), max(hi))) + claim("REPORT.md", "(INT voice lines with a subtitle): none for efforts/grunts" if p["Efforts & emotes"] == 0 else "!") + + # what the editor loads (Italian on the test PC: the smoke test) + ita = len(rows["ITA"]) + len(sfx) + sum(1 for s in bank if s.in_language("ITA")) + claim("REPORT.md", "%s sounds loaded, encode + inject, %s subtitles" % (n(ita), n(len(texts["ITA"])))) + claim("README.md", "~%s sounds of one language" % n(int(round(ita, -3)))) + return rows + + +# --- figures recorded by the catalog build (index.json: they need the full scan) --------------------- +def index_claims(cat): + import json + ix = json.load(open(os.path.join(cat, "index.json"), encoding="utf-8")) + b, ln, sub = ix["BANK"], ix["LINES"], ix["SUB"] + claim("REPORT.md", "Names: %s BANK sounds are named after the Wwise event that plays them (`Play_...`, `Speech_...`), " + "the other %s after their bank" % (n(b["named_by_event"]), n(b["named_by_bank"]))) + claim("REPORT.md", "**Unnamed sounds**: %s BANK sounds are reached by no event" % n(b["named_by_bank"])) + claim("docs/decisions.md", "sounds (%s of %s, voice lines included); the other %s are numbered per bank" + % (n(b["named_by_event"]), n(b["sounds"]), n(b["named_by_bank"]))) + claim("docs/decisions.md", "the most specific speech event that plays it (%s of the %s WAD voice lines)" + % (n(ln["wad_lines_named_by_event"]), n(ln["wad_lines"]))) + claim("docs/decisions.md", "the other %s keep the name of their English (INT) recording's position" + % n(ln["wad_lines_named_after_int_recording"])) + claim("REPORT.md", "- %s voice lines are played by no event whose name the game stores" + % n(ln["wad_lines_named_after_int_recording"])) + claim("docs/user-guide.md", "About %d %% of the lines are named by no game event" + % pct(ln["wad_lines_named_after_int_recording"], ln["wad_lines"])) + claim("docs/formats/subtitles.md", "%s dialogue objects in the packages for %s speech lines" + % (n(sub["dialogue_objects"]), n(sub["dialogue_lines"]))) + claim("docs/formats/subtitles.md", "The sizes add up exactly in all %s objects" % n(sub["dialogue_objects"])) + claim("docs/formats/subtitles.md", "plus %d scripted subtitles" % sub["scripted_objects"]) + claim("docs/formats/subtitles.md", "%d more subtitles (radio and encryption-card lines) are `RSubtitle` objects" + % sub["scripted_objects"]) + claim("REPORT.md", "(%s speech lines x %d languages) and" % (n(sub["dialogue_lines"]), len(sub["lines"]))) + claim("REPORT.md", "in %d scripted `RSubtitle` objects" % sub["scripted_objects"]) + claim("docs/formats/banks.md", "(in %s of the 2,650 packages)" % n(b["files"] - 67)) # 67 loose .bnk with banks + + +# --- the game's own files ---------------------------------------------------------------------------- +def menu_claims(game): + from ackit import loctext + root = os.path.join(game.root, "BmGame", "Localization") + counts = {L: len(loctext.load(root, L)) for L in loctext.languages(root)} + files = {L: len(loctext.files(root, L)) for L in loctext.languages(root)} + approx = n(int(round(sum(counts.values()) / len(counts), -3))) + claim("docs/formats/text.md", "~%d files and ~%s strings each" % (collections.Counter(files.values()).most_common(1)[0][0], approx)) + claim("REPORT.md", "~%s strings per language" % approx) + claim("docs/user-guide.md", "of the selected language (about\n%s)" % approx) + claim("REPORT.md", "%s menu texts loaded" % n(counts["ITA"])) + + +def scan_claims(game): + """Bank survey from a fresh scan of every package, and the relation facts of `ackit verify --catalog`.""" + from ackit.bnk import find_banks, Bank, BankError, IN_MEMORY, STREAMED, PREFETCH + blobs = collections.Counter() + media_sets = collections.defaultdict(set) + kinds = collections.Counter() + in_pkg = set() + + def visit(rel, pkg, start, data): + for bk in find_banks(data): + blobs[bk.id] += 1 + media_sets[bk.id].add(frozenset(m.id for m in bk.media())) + in_pkg.add(bk.id) + own = {m.id for m in bk.media()} + for s in bk.sources(): + if s.stream == IN_MEMORY: + kinds["memory own" if s.media in own else "memory other"] += 1 + elif s.stream == STREAMED: + kinds["streamed"] += 1 + elif s.stream == PREFETCH: + kinds["prefetch"] += 1 + + sc = banks.scan(game.cooked, visit=visit) + for path in glob.glob(os.path.join(game.cooked, "SFX", "*.bnk")): # loose banks: source kinds too + try: + bk = Bank(open(path, "rb").read(), 0) + except BankError: + continue + own = {m.id for m in bk.media()} + for s in bk.sources(): + if s.stream == IN_MEMORY: + kinds["memory own" if s.media in own else "memory other"] += 1 + elif s.stream == STREAMED: + kinds["streamed"] += 1 + elif s.stream == PREFETCH: + kinds["prefetch"] += 1 + names = {} + cooked_list = [l.strip() for l in open(os.path.join(game.cooked, "CookedBanks.txt"), encoding="latin-1") if l.strip()] + from ackit.bnk import fnv1 + for nm in cooked_list: + names[fnv1(nm)] = nm + loose_ids = {int(os.path.splitext(os.path.basename(p))[0], 16) for p in glob.glob(os.path.join(game.cooked, "SFX", "*.bnk")) + if re.fullmatch(r"[0-9A-Fa-f]{8}", os.path.splitext(os.path.basename(p))[0])} + embedded = [nm for h, nm in names.items() if h in in_pkg] + loose = sorted(nm for h, nm in names.items() if h in loose_ids and h not in in_pkg) + nowhere = sorted(nm for h, nm in names.items() if h not in in_pkg and h not in loose_ids) + claim("docs/formats/banks.md", "lists all %s cooked bank names" % n(len(cooked_list))) + both = sorted(nm for h, nm in names.items() if h in loose_ids and h in in_pkg) + claim("docs/formats/banks.md", "%s of them are embedded in packages (%s also as loose `.bnk` files), %d are only loose " + "`.bnk` files (%s); %s were found nowhere" + % (n(len(embedded)), ", ".join("`%s`" % x for x in both), len(loose), ", ".join("`%s`" % x for x in loose), + ", ".join("`%s`" % x for x in nowhere[:-1]) + " and `%s`" % nowhere[-1])) + diff_sets = sum(1 for b, v in media_sets.items() if len(v) > 1) + claim("docs/formats/banks.md", "Bank blobs in packages: %s for %s bank IDs" % (n(sum(blobs.values())), n(len(blobs)))) + by_name = {names.get(b, ""): c for b, c in blobs.items()} + claim("docs/formats/banks.md", "`CHR_CatwomanFoleySuit` %d times, `CHR_BatmanFoleyBelt` %d). %d bank IDs" + % (by_name.get("CHR_CatwomanFoleySuit", 0), by_name.get("CHR_BatmanFoleyBelt", 0), diff_sets)) + claim("docs/formats/banks.md", "| banks embedded in `.upk` packages (in %s of the %s packages) | %s |" + % (n(sum(1 for f in sc.files if f.lower().endswith(".upk"))), n(len(glob.glob(os.path.join(game.cooked, "*.upk")))), + n(len(blobs)))) + claim("docs/formats/banks.md", "Source kinds over all bank copies: %s in-memory sources of the bank's own media; %s " + "in-memory sources with bank id 0" % (n(kinds["memory own"]), n(kinds["memory other"]))) + claim("docs/formats/banks.md", "(loaded by another bank); %s streamed sources; %s **prefetch** sources" + % (n(kinds["streamed"]), n(kinds["prefetch"]))) + refs = {h for h, t in sc.packs.items() if t} + claim("docs/formats/banks.md", "the BKPK tables in the packages reference %s WAD entries" % n(len(refs))) + per_bank = collections.defaultdict(set) + for m, e in sc.media.items(): + for b in e["banks"]: + per_bank[b].add(m) + big = sorted(per_bank.items(), key=lambda kv: -len(kv[1]))[:7] + nm = [(sc.bank_names.get(b, "%08X" % b), len(v)) for b, v in big] + claim("docs/formats/banks.md", "Largest: `%s` %d WEMs, " % nm[0] + ", ".join("`%s` %d" % x for x in nm[1:])) + pkm = {l.media for l in sc.locations if not l.file.startswith("SFX/")} + pre = collections.Counter() + for m in pkm: + for p_ in {sc.bank_names.get(b, "%08X" % b).split("_")[0] for b in sc.media[m]["banks"]}: + pre[p_] += 1 + top = [k for k, v in pre.most_common() if v >= 30] + text = ", ".join("`%s` %s" % (k, n(pre[k])) if k != "CHR" else "`CHR`\n(character foley, combat, bosses) %s" % n(pre[k]) + for k in top) + claim("docs/formats/banks.md", "(a media held by banks of two prefixes counts for both): " + text + ", others < 30") + # media IDs used by several languages' WAD tables (each at its own position) + langs_of = collections.defaultdict(set) + for h, tables in sc.packs.items(): + for L, t in tables.items(): + if L: + for m in t: + langs_of[m].add(L) + claim("docs/formats/banks.md", "%d media IDs are used by several\n languages" % sum(1 for v in langs_of.values() if len(v) > 1)) + + # relations, re-derived from the game by the catalog check + from ackit.verify import catalogcheck + rep = catalogcheck.run(game, log=lambda *a: None) + facts = "\n".join(rep.facts) + print("verify --catalog:", "catalog OK" if rep.ok else "%d issue(s)" % len(rep.issues)) + + def fact(pattern): + m = re.search(pattern, facts) + if not m: + raise SystemExit("fact not found: " + pattern) + return [int(x) for x in m.groups()] + src = fact(r"own language's bank: (\d+) of (\d+)") + claim("REPORT.md", "own language (%s / %s)" % (n(src[0]), n(src[1]))) + claim("docs/formats/banks.md", "of its own language (%s of %s)" % (n(src[0]), n(src[1]))) + hd = fact(r"whole bank copies: (\d+) of (\d+)") + claim("REPORT.md", "is the start of its recording (%s / %s)" % (n(hd[0]), n(hd[1]))) + claim("docs/formats/banks.md", "line gives it (%s of %s)" % (n(hd[0]), n(hd[1])) if True else "") + rec, lns, full = fact(r"(\d+) recordings \(WAD .*?\), (\d+) lines, (\d+) of them") + claim("docs/formats/banks.md", "(%s recordings)" % n(rec)) + ln15 = fact(r"(\d+) of (\d+) lines within a factor 1.5") + claim("docs/formats/banks.md", "(%s of %s\n within a factor 1.5" % (n(ln15[0]), n(ln15[1]))) + sp = fact(r"speakers: (\d+) labelled voice lines agree .*?, (\d+) listed as Unknown .*?, (\d+) differ") + claim("REPORT.md", "%s agree, %d differ" % (n(sp[0]), sp[2])) + claim("docs/categories.md", "%s labelled\n lines agree" % n(sp[0])) + claim("docs/categories.md", "%d listed\n as Unknown are named there, %d differ" % (sp[1], sp[2])) + if not rep.ok: + claim("(game)", "catalog OK", "ackit verify --catalog") + + +def _rebuild_language(lang): + """Keys of the WEMs of one speech WAD that the builder does not reproduce byte for byte.""" + from ackit.inject import read_target + from ackit.verify.selfcheck import rebuild_identical + from ackit.wad import Wad + game = Game(default_game_dir(settings.load())) + path = game.wad(lang) + w = Wad(path) + from ackit.wem import Wem + bad, count, extra = [], 0, collections.Counter() + for s in read_csv(os.path.join(catalog_dir(), lang + ".csv")): + count += 1 + r = read_target(path, s.target, w) + tags = {c.tag for c in Wem.parse(r).chunks} - {b"fmt ", b"data"} + if tags: + extra["/".join(sorted(t.decode() for t in tags))] += 1 + try: + ok = rebuild_identical(r) + except Exception: # noqa: BLE001 - a WEM the builder cannot parse is a difference too + ok = False + if not ok: + bad.append(s.key) + return count, bad, extra + + +def wem_claims(): + """Every WEM of the 5 speech WADs and of the SFX WAD rebuilt from the format rules (docs/formats/wem.md; + ~10 minutes).""" + import multiprocessing + from ackit.inject import read_target + from ackit.wem import Wem + with multiprocessing.Pool(len(VOICES) + 1) as p: + res = p.map(_rebuild_language, VOICES + ["SFX"]) + extra = collections.Counter() + for _, _, e in res: + extra.update(e) + sfx_count, sfx_bad, _ = res.pop() + game = Game(default_game_dir(settings.load())) + sfx = {s.key: s for s in read_csv(os.path.join(catalog_dir(), "SFX.csv"))} + kinds = collections.Counter() + per = [] # samples per seek-table entry of the music tracks + from ackit.verify.selfcheck import rebuild_identical + for k in sfx_bad: + s = sfx[k] + r = read_target(game.wad("SFX"), s.target) + m = Wem.parse(r) + try: + rebuild_identical(r) + err = False + except ValueError: # "inconsistent total_frames": more samples than the packets hold + err = True + if any(c.tag == b"smpl" for c in m.chunks): + kinds["smpl"] += 1 + elif err: + kinds["frames"] += 1 + elif s.group == "Music" and m.vorb["seek_table_size"]: + kinds["seek"] += 1 + per.append(m.vorb["total_frames"] / (m.vorb["seek_table_size"] // 4)) + else: + kinds["other"] += 1 + claim("REPORT.md", "%d SFX originals (music seek tables and sample counts, looping ambience)" % len(sfx_bad)) + claim("docs/formats/wem.md", "In SFX %s of %s are identical; the others are %d music tracks whose seek table follows " + "another placement (one entry per %s samples), %d music tracks whose sample count exceeds what their " + "packets hold, and %d 4-channel ambience WEMs with a `smpl` (loop) chunk%s" + % (n(sfx_count - len(sfx_bad)), n(sfx_count), kinds["seek"], rng([int(x) for x in per]), kinds["frames"], + kinds["smpl"], "" if not kinds["other"] else " (+%d other)" % kinds["other"])) + total = sum(c for c, _, _ in res) + bad = [k for _, b, _ in res for k in b] + claim("docs/formats/wem.md", "`RIFF` size `WAVE`, then `fmt ` (0x42 bytes), optional %s, `data`" % ", ".join( + "%s (%s %s)" % (" + ".join("`%s`" % t for t in k.split("/")), n(v), "WEM" if v == 1 else "WEMs") + for k, v in sorted(extra.items()))) + exceptions = ", ".join("%s `%s`" % tuple(k.split(":", 1)) for k in bad) # e.g. INT `FE9963E8:74` + claim("docs/formats/wem.md", "Checked on every WEM of the 5 speech WADs: %s of %s are identical; the exception (%s)" + % (n(total - len(bad)), n(total), exceptions)) + + +def test_claims(): + import subprocess + out = subprocess.run([sys.executable, "-m", "pytest", "--collect-only", "-q", "-p", "no:cacheprovider"], + cwd=ROOT, capture_output=True, text=True).stdout + m = re.search(r"(\d+) tests? collected", out) + claim("REPORT.md", "**Tests**: %s automated tests" % (m.group(1) if m else "?"), "pytest --collect-only") + + +def norm(t): + return re.sub(r"\s+", " ", t).strip() + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--game") + ap.add_argument("--scan", action="store_true", help="also scan every package and run the catalog check") + ap.add_argument("--wems", action="store_true", help="also rebuild every speech and SFX WEM from the rules") + a = ap.parse_args() + cat = catalog_dir() + catalog_claims(cat) + index_claims(cat) + test_claims() + gd = a.game or default_game_dir(settings.load()) + game = Game(gd) if gd else None + if game: + menu_claims(game) + if a.scan: + if not game: + raise SystemExit("--scan needs the game") + scan_claims(game) + if a.wems: + if not game: + raise SystemExit("--wems needs the game") + wem_claims() + docs = {} + bad = 0 + for doc, text, why in CLAIMS: + if doc not in docs: + path = os.path.join(ROOT, doc) + docs[doc] = norm(open(path, encoding="utf-8").read()) if os.path.exists(path) else "" + if norm(text) not in docs[doc]: + bad += 1 + print("MISMATCH %s: expected \"%s\"%s" % (doc, norm(text), (" (" + why + ")") if why else "")) + print("%d claims checked in %d documents%s: %s" % (len(CLAIMS), len(docs), "" if a.scan else " (without --scan)", + "all match" if not bad else "%d mismatch(es)" % bad)) + return 1 if bad else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/e2e_features.py b/tools/e2e_features.py index 4fd05d8..700fc93 100644 --- a/tools/e2e_features.py +++ b/tools/e2e_features.py @@ -8,7 +8,8 @@ For every check: the SHA-1 of each touched file is taken, the change is applied of the editor and the CLI), read back from the game files, reverted, and the files must be byte-identical again. 1. in-memory sound with the most copies (every copy written; once, it is shared by all languages) 2. localized in-memory speech line of the language's voices - 3. speech line of the language's voices with prefetch heads (WAD + every head copy) + 3. speech line of the language's voices with prefetch heads (WAD + every head copy), and one that a bank + also stores whole (WAD + every whole copy) 4. subtitle with several copies (RDialogueEvent), and a scripted subtitle (RSubtitle, Startup.upk) 5. menu text 6. sound + subtitle in the same package, reverted one at a time @@ -82,7 +83,7 @@ def sound_round_trip(ws, s, label): for l, b in copies_of(ws, s): if l.kind == banks.MEMORY: ok &= b == wem + bytes(l.size - len(wem)) - else: # prefetch head of a WAD line (stored zero-padded to its size) + else: # head or whole copy of a WAD line (zero-padded to its size) full = wem + bytes(max(0, ws.store.get(s.key)["orig_size"] - len(wem))) ok &= b == (full + bytes(l.size))[:l.size] ws.revert([s]) @@ -140,7 +141,13 @@ def run_checks(game, lang, voices, td, ws, first): heads = [ws.catalog.by_key[k] for k in ws.catalog.heads if k in ws.catalog.by_key and ws.catalog.by_key[k].archive == voices] sound_round_trip(ws, max(heads, key=lambda s: len(ws.locations(s))), - "3 %s speech line with prefetch heads" % lang) + "3a %s speech line with prefetch heads" % lang) + whole = [s for s in heads if any(l.size >= s.size for l in ws.locations(s))] + if whole: + sound_round_trip(ws, max(whole, key=lambda s: len(ws.locations(s))), + "3b %s speech line also stored whole in a bank" % lang) + else: + check("3b %s speech line also stored whole in a bank" % lang, False, "none found") tx = ws.texts subs = tx.subtitles(lang)