Version 3.2.3: editor layout with aligned controls and full-width button bars; docs on recordings shared between languages

CLAUDE.md

@@ -4,13 +4,25 @@ Batman: Arkham City (PC) audio toolkit + editor. Read `NOTES.md` first (status,
 
 - Layout: `ackit/` library + CLI (`python -m ackit -h`), `app/` PySide6 editor, `tests/` pytest, `data/catalog/`
   CSVs (+ user's original lists in `data/catalog/source/`, never edit them), `docs/`, `tools/` build scripts.
-- Dev env: `.venv` (Python 3.14, `pip install -e .[app,dev]`). Run tests with `.venv/Scripts/python -m pytest -q`.
+- Dev env: `.venv` (Python 3.14, `pip install -e .[app,dev,verify]`). Run tests with `.venv/Scripts/python -m pytest -q`.
 - Writing game files goes through `ackit.workspace.Workspace` only (originals store, atomic writes, conflict checks).
   Speech/ambience: in place, never larger than the original WEM; music packs: any size (`docs/formats/wad.md`).
+  In-memory sounds (archive BANK, loose `.bnk` + `.upk` packages): in place into every copy; a bank file is always
+  rebuilt from the store's original copy + the applied replacements (`docs/formats/banks.md`, `upk.md`).
+  Texts go through `Workspace.texts` (subtitles in packages, in place and NUL-padded; menu `.int` files).
 - Never use Steam "verify integrity". Keep game files original after tests (`ackit status`, `ackit revert --all`).
-- Real-game checks: `ackit verify` (offline), `ackit verify --ingame` (Windows; needs Steam running and logged in,
-  WinDbg; ~1.5 min; reverts and checks byte-identity itself).
-- Rebuild catalogs only from original game files: `python tools/build_catalog.py`.
+- Real-game checks: `ackit verify` (offline), `ackit verify --catalog [--quick]` (catalog vs game files),
+  `ackit verify --ingame [--banks | --texts]` (Windows; needs Steam running and logged in, WinDbg; ~1.5 min; reverts and
+  checks byte-identity itself).
+- Rebuild catalogs only from original game files: `python tools/build_catalog.py` (~5 min, scans all packages;
+  stops if a sound differs from the current catalog). Voice lines are identified by the game's Wwise Sound
+  objects (`ackit/lines.py`, column `line`; one recording per language), never by WAD position or name:
+  positions differ per language (`docs/decisions.md` 5, `docs/formats/banks.md` *Lines*).
+- Figures in the docs are checked, not copied: after changing data, code or docs run `python tools/check_docs.py`
+  (`--scan`: also the bank survey and the catalog check's facts, ~10 min; `--wems`: every WEM rebuilt, ~10 min);
+  add a claim there for every new figure.
+- Real-game write round trips: `python tools/e2e_features.py`, `python tools/e2e_editor.py` (they revert in a
+  `finally` block; still check `ackit status` after any interrupted run).
 - Shell: git at `C:\Program Files\Git\cmd` (prepend to PATH). Write Python with the Write/Edit tools; avoid
   PowerShell here-strings and heredoc string surgery with backslash escapes (they produced NUL bytes before).
 - The speaker loopback can contain other apps' audio (voice chat): never save raw captures.

NOTES.md

@@ -14,10 +14,121 @@
   Packaging: `python tools/build_app.py` -> dist/UnheardCity (self-contained, licenses); packaged exe passes
   `--smoke-test` with PATH = C:\Windows only. Real e2e through the editor code path (tools/e2e_editor.py): PASS
   (music heard in game ncc 1.0, reverts byte-identical, speech line apply/revert byte-identical).
-- NEXT: editing of the ~15,800 in-memory sounds in soundbanks (loose `SFX\*.bnk` footsteps first, then banks
-  embedded in `.upk` packages); not listed or editable yet, survey in `docs/formats/banks.md`.
-- NEXT (optional): verify >2-channel channel order in game; speech heard in-game (needs gameplay).
-- OPEN: channel order for >2 channels unverified.
+- Mission 3 (2026-10-09, brief: `docs/history/mission3_audio_text_prompt.md`): full-game audio, then subtitle
+  and menu text editing. Plan (one commit per stage; a stage = code + tests + docs + this status):
+  1. [x] `ackit/lzo.py`, `ackit/upk.py` (UE3 package summary, compressed chunks, block-level read/patch),
+     `ackit/bnk.py` (bank parsing: DIDX/DATA/HIRC sources). Docs: formats/upk.md, banks.md. Findings: pure-Python
+     LZO ~47 MB/s (== decompress.exe on every block); 15,800 media IDs in 96,011 copies, 396 are prefetch heads
+     of WAD sounds (2,101 copies) -> 15,404 in-memory sounds; copies identical except 1C898731.
+  2. [x] Catalog archive `BANK` (15,404 sounds, 6,416 named by `Play_` events; now 15,355, 13,501 named by events) + `BANK_locations.csv.gz` (93,910 copies,
+     2,101 prefetch heads of 396 speech lines); `ackit verify --catalog` (full: catalog OK; fixed 2 duplicate
+     names per language: variation shown as NAME#i); categories for bank prefixes.
+  3. [x] Workspace writes BANK sounds into every copy and keeps speech-line prefetch heads in sync (bank files
+     rebuilt from a mandatory full original copy); editor shows bank + copies. `ackit verify` PASS (300/300
+     BANK WEMs rebuilt identically); `ackit verify --ingame --banks` PASS (loaded, no crash, revert identical;
+     audible check not possible without input: findings 18).
+  4. [x] Subtitles = RDialogueEvent objects in packages (9,016 lines x 11 languages, 168k copies; parser
+     `ackit/subtitles.py`, catalog SUB_<LANG>.csv + SUB_locations.csv.gz, 92 % linked to speech sounds via
+     events; now ~99 %); menu text = Localization/<LANG>/*.int (`ackit/loctext.py`, ~7,000 strings/language).
+     `Workspace.texts` (ackit/texts.py): states, apply/revert, conflicts, mod packages; CLI `ackit text`.
+     Subtitles in place (NUL-padded, limit = original length); decision + reasons in decisions.md 4c.
+  5. [x] Editor: tabs Sounds / Subtitles / Menu text (app/textpanel.py: search, filters, edit with room
+     counter, apply/revert/revert all, find & replace on the filtered rows, go to sound); the sound editor
+     shows and edits the sound's subtitle (SubtitleBox); texts in mod export/import and the changed-outside
+     banner; smoke test loads texts. 15 GUI tests.
+  6. [x] Final pass 2026-10-09: `ackit verify` PASS, `--catalog` OK, `--ingame` PASS (ncc 1.0), `--ingame
+     --banks` PASS, `--ingame --texts` PASS (12/12 menu markers in memory), tools/e2e_editor.py PASS; game
+     original after. Version 3.0.0; dist/UnheardCity-3.0.0-windows-amd64.zip (86 MB) smoke + GUI start PASS
+     with PATH = C:\Windows. 121 tests. REPORT.md rewritten (unresolved items).
+- 2026-10-09 (after user test): in-memory speech is localized (one AkBank export = language list + one bank
+  per language; ~1,460 lines per voice language): BANK.csv has a `language` column, the catalog shows shared
+  sounds + the loaded language, names unique per language, subtitles of in-memory lines link to them (links
+  92 % -> 99 %). Editor: one Language setting at the top (always starts at the game's language; languages
+  without voices use INT voices), text tabs follow it.
+- 2026-10-10 (user asked "is it really all in order?"): variation positions differ per language (75 % of lines)
+  -> speech sounds named after the game's line (speech event), curated labels mapped through it (characters.csv
+  gets a language column), verified 955/955 against dialogue speaker fields; RSubtitle + RSubtitleLookup
+  (key = FNV-1 of the line name without Speech_) added; editor redesigned: Listen | Replace | Subtitle columns,
+  editor spans the full window width (no scrolling at 1000x700 and up).
+- 2026-10-10 final check: build_catalog refuses to run on a modified game (`--accept-changes` after an update);
+  every test tool reverts in `finally`; mods from another language are not matched to position-named lines;
+  `tools/e2e_features.py` (8 real-game round trips) PASS; `--ingame --texts` also finds 23/23 Startup.upk
+  subtitle markers in memory; `--ingame`, `--banks`, e2e_editor PASS; `verify --catalog` OK; 124 tests; game
+  original after. Docs brought up to date.
+- 2026-10-10 (user: "make it complete for every language, fact check everything"): voice lines are identified
+  by the game data alone (`ackit/lines.py`, catalog column `line`): per-language BKPK tables (they were merged
+  by media ID, and 657 shared IDs let ESN overwrite the others), prefetch sources in the Sound-object map,
+  recordings stored twice written together, multi-language in-memory media, subtitle objects naming events.
+  12,434 lines, each with exactly one recording in all 5 voice languages; 3,366 speaker labels in every language;
+  subtitles link every version of a line; mods move between languages by line identity. `ackit verify
+  --catalog` re-derives all relations and prints facts (independent checks: bank language of every source,
+  prefetch heads = start of their recording, copies identical, speaker field of the dialogue objects, menu keys
+  per language). `tools/e2e_features.py --lang all` 78/78; `--ingame`, `--banks`, `--texts`, e2e_editor PASS;
+  127 tests; game original after. Speaker labels vs the dialogue speaker field: 2,766 agree, 19 differ (listed in
+  categories.md, for the user to check by listening; the source lists are never edited). The user's guide files
+  (`ArkhamCityModding/Audio/AbdAshes' ...`) hold no relation data: their hashlist was ackit/data/hashlist.txt,
+  the scripts only unpack (wemscan = position names). `data/catalog/source/ita/Checkpoint.txt` was found deleted
+  (no code touches source/; cause unknown) and restored from git. Version 3.1.0 (minor):
+  dist/UnheardCity-3.1.0-windows-amd64.zip smoke PASS. e2e_features now deletes its temporary stores (GBs of
+  full copies) when everything reverted.
+- 2026-10-10 (user: order in the browser "messed up" after filter changes; redundant files): the proxy had
+  dynamic sorting off, so rows a filter showed again were appended at the end -> dynamic sort on (+ GUI test).
+  ackit/data/hashlist.txt removed: its 2,544 names the catalog uses are exactly the game's CookedBanks.txt bank
+  names (WAD entry ID = FNV-1 of the bank name), so names now come from the game (`catalog.bank_names`);
+  rebuilt catalogs identical.
+- 2026-10-10 (user: filtering/sorting choppy, add spinners like the audio preview): QSortFilterProxyModel
+  replaced by `app/models.py: RowView` (Python filter + one sort over cached keys): sound list 1.3-1.5 s ->
+  10-20 ms per sort/filter, text lists 0.26-0.7 s -> ~10 ms. Spinner overlay (`spinner.BusyOverlay`) over the
+  sound list while it loads and over the subtitle/menu lists, which now load as background jobs (a selection
+  asked for meanwhile is applied when the rows arrive). GUI tests for order, selection and the overlays.
+- 2026-10-10 version 3.2.0: docs use INT (the game's English) for examples and figures; facts measured on this
+  PC's Italian install stay labelled as such. User-guide screenshots in INT (`tools/screenshots.py --lang`,
+  default INT; the app itself still starts in the game's language). Every WEM of the 5 speech WADs rebuilt
+  from the rules: 54,079 / 54,080 identical (INT FE9963E8:74: avg-bytes/s field one lower, informational).
+  Removed unused code (helpers/constants nothing called, the key-posting debugger helper, a double refilter
+  after changes); old dist zips, logs/ and caches deleted.
+- 2026-10-10 final check (user: "is everything fact checked ... functioning ... documentation correct?"):
+  every number in the docs recomputed from the catalog and a fresh game scan; stale ones fixed (positions differ
+  for 81.2-81.6 %, not 75 %; 8,092 neutral BANK sounds; 24 unlinked subtitle names; 637 position-named lines;
+  14,802 package media; subtitle coverage 9 % combat barks, 23 % cinematics; speaker coverage by line). Bugs
+  found and fixed (findings 23): 49 WAD recordings also stored whole in banks were separate BANK sounds (now
+  bank copies of the line, written with it); 8 BANK/SFX name clashes; name lookup preferred no name over an
+  alias (CLI "ambiguous" for e.g. `CHR_RadioScanner#1` in FRA). `ackit verify --lang` now applies (catalog: that
+  language only; in-game: must be the game's language). Subtitle error message says Windows-1252, not Latin-1.
+  Catalog rebuilt from the original files; `verify --catalog` OK (2,185 / 2,185 bank copies of WAD recordings
+  identical), e2e_features --lang all 89/89, e2e_editor, --ingame/--banks/--texts PASS, 132 tests; game original
+  after. Version 3.2.1: dist/UnheardCity-3.2.1-windows-amd64.zip smoke + GUI start PASS.
+- 2026-10-10 re-check (user: "fact check everything again"): the catalog rebuilt from the game is identical to the
+  shipped one (every CSV and location index), and `verify --catalog` gives the same facts. Figures in the docs are
+  now checked by `tools/check_docs.py` (catalog, index.json, menu files; `--scan`: bank survey + catalog-check
+  facts; `--wems`: every WEM rebuilt); the build records the scan-only figures in index.json (names by event,
+  dialogue objects, WAD lines named after a recording). It found what hand-copied figures had wrong: 61,510
+  "recordings" counted the 25 second places (61,485 recordings; the stat was fixed too), 81.2-81.6 % (not 81-82),
+  the loose/embedded bank-name overlap, REPORT's test count, wem.md's SFX claim (626 / 651 rebuild, not "all but 4
+  looping tracks": 14 music seek tables, 5 music sample counts, 6 ambience `smpl` loops) and its extra-chunk count
+  (722 WEMs, not 37). New test: cross-references between the shipped files.
+- 2026-10-10 user gameplay test: replaced voice lines and subtitles WORK in game. Batman footsteps do NOT:
+  all 568 BANK sounds of the CHR_BatmanFootsteps* banks were replaced with Downloads\boom.mp3 (trimmed per sound,
+  still APPLIED, with the user's own test subtitle SUB:ITA:Speech_CHURCH_B1_CH2_HQ_INTRO2_08; revert with
+  `ackit revert --all` when the user says so). In game the boom was heard once, briefly; otherwise the original
+  steps. Checked: every copy (40 files) holds the new bytes; replacements are loud (-8.8 dB RMS vs -20..-33);
+  no other media ID has the same original audio; no game data outside CookedPCConsole. The named footstep
+  events (Play_BM_Footstep_Walk/Run/Land/...) reach only the 46 media of bank CHR_BatmanFootsteps (replaced);
+  the 522 surface media (CHR_BatmanFootsteps_<Surface>_Media, loose SFX\*.bnk) are reached by no named event
+  (switch containers not followed by event_objects?). NEXT: find where the game gets the step audio it plays
+  (switch/random containers and their children, banks loaded at runtime, a memory capture of a step in game).
+- 2026-10-10 footsteps solved (findings 24): in-memory replacements were zero-padded in their slot and the game
+  plays that as silence; ackit now fills each in-memory copy exactly with a LIST/adtl/labl chunk (`wem.fill`,
+  `Workspace.fits/fill_sizes/prepare`). Verified by the user in game (a beep on every step), e2e_features 89/89.
+  Version 3.2.2. The footstep test was reverted (game original).
+- 2026-10-11 version 3.2.3: editor layout (user: controls "placed in line and go to new line, not pretty"):
+  aligned grid for the edit fields, every button row fills its column, each column ends with a full-width
+  action bar level with the others; A/B and Loop are icons next to the volume. Fits a 1000-px window
+  without horizontal scrolling (minimum 984 px). Docs: recordings the game shares between languages (user saw
+  English lines "leaking" into other languages: the game's own data, banks.md *Lines*).
+- DONE by the user in game: replaced voice lines, edited subtitles and replaced footsteps (after the fill fix).
+- OPEN: channel order for >2 channels unverified; 19 speaker labels to check by listening; the Linux build was
+  never built or run (no Linux here).
 
 ## Environment facts
 - Game: Steam AppID 200260 "Batman: Arkham City GOTY", `C:\Program Files (x86)\Steam\steamapps\common\Batman Arkham
@@ -29,7 +140,7 @@
   `%USERPROFILE%\Desktop\ArkhamCityModding\Audio\49D84887_LOC_ITA\WAD\` (sha1 1238172C...). The game is clean now
   (fingerprints match the old `_backup/`, deleted on 2026-10-08).
 - Dev tools: `.venv` (Python 3.14.6, PySide6 6.10.3, soundfile 0.14.0 / libsndfile 1.2.2, soxr 1.1.0, numpy,
-  pytest, PyInstaller 6.22.3, soundcard 0.4.6). git at `C:\Program Files\Git\cmd`. WinDbg Store (path changes on
+  pytest, pyflakes, PyInstaller 6.22.3, soundcard 0.4.6). git at `C:\Program Files\Git\cmd`. WinDbg Store (path changes on
   update; `ackit/verify/debugger.py: find_cdb`). No WSL distro, no Docker.
 - User data (this PC): `%APPDATA%\UnheardCity\` (settings.json, stores\<id>\), cache `%LOCALAPPDATA%\UnheardCity\cache`.
 - The user's own material: `%USERPROFILE%\Desktop\ArkhamCityModding` (old mod work, Ghidra project, dump);
@@ -39,13 +150,16 @@
 ## Where to look
 - Findings, failed hypotheses, in-game experiment log E0-E14: `docs/reverse-engineering/findings.md`.
 - Formats: `docs/formats/`. Decoder addresses: `docs/reverse-engineering/decoder.md`.
-- Decisions (stack, deps, licenses, originals store, size rule): `docs/decisions.md`.
+- Decisions (stack, deps, licenses, originals store, size rule, catalog, verification): `docs/decisions.md`.
+- Voice lines across languages (one recording per language, from the game data): `docs/formats/banks.md`
+  (*Lines*), `ackit/lines.py`; checked by `ackit verify --catalog` (facts).
 
 ## Session log
 - 2026-10-08: user added `my_catalog_for_ita/` (per-character lists, wemscan names) -> moved to
   `data/catalog/source/ita/`; 3364 names labelled (2 wemscan name collisions label both candidates; one line had a
   `.wem.wav` typo). Built catalogs for ITA/INT/FRA/DEU/ESN/SFX (identical names and counts across languages).
-  Assumption: speakers are the same in every language, so ITA labels apply to all.
+  Assumption: speakers are the same in every language, so ITA labels apply to all. (Wrong for multi-variation
+  entries, whose WEM order differs per language: since 2026-10-10 labels go through the game's line names.)
 - 2026-10-08: libsndfile's Vorbis writer crashed (C stack overflow) on a 30 s x 6 ch single write -> block writes.
 - 2026-10-08: wad_fingerprint made structural (IDs + non-pack sizes) so music edits don't invalidate the catalog;
   scan mode never borrows checksums from the shipped CSV (other builds / synthetic tests).
@@ -54,8 +168,10 @@
   SoundFilter.refilter uses begin/endFilterChange when available.
 - 2026-10-08: search haystack must not contain the variation index (":2" matched "2"); IDs are matched separately
   (words of >= 6 chars against the target id).
-- Verification commands: `pytest -q`; `ackit verify`; `ackit verify --ingame`; `python tools/e2e_editor.py`
-  (needs Steam logged in); packaged: `dist/UnheardCity/UnheardCity.exe --smoke-test --report f.txt`.
+- Verification commands: `pytest -q`; `ackit verify`; `ackit verify --catalog`; `python tools/e2e_features.py
+  [--lang all]`;
+  `python tools/e2e_editor.py`; `ackit verify --ingame [--banks | --texts]` (needs Steam logged in); packaged:
+  `dist/UnheardCity/UnheardCity.exe --smoke-test --report f.txt`.
 - 2026-10-08: UI restyle on request: vector icon set (app/icons.py, no emoji), "Arkham" theme (app/theme.py: QSS +
   palette, translucent panels over a wallpaper backdrop).
 - 2026-10-08: user rejected the procedural icon/backdrop and supplied images (~/Downloads/Arkham City:
@@ -86,3 +202,22 @@
   soundbanks are missing (docs/formats/banks.md). Packages were unpacked with Gildor's decompress.exe from
   `%USERPROFILE%\Desktop\ArkhamCityModding\Audio\decompress\` (outside the repo). User: document now, implement
   editing later.
+- 2026-10-09 (mission 3): BANK = one row per media ID, written into every copy in place (assumption: the game
+  may load any copy; a partial write would be inconsistent). Package writes store changed blocks as LZO literal
+  runs (no compressor needed). Bank-file conflicts use the whole-file SHA-1 (the cheap guid+chunk-table
+  fingerprint missed same-size edits in tests). Name collisions in the WAD catalogs (2 per language) are shown as
+  NAME#i. `soundcard` comes from the `verify` extra (was missing from the venv).
+- 2026-10-09 (mission 3, stage 4): subtitle links by name were wrong for most multi-variation entries (WEMs are
+  stored in media-ID order, not _NN order): links come only from events. Numbered event names (Speech_X_04)
+  live in export tables, not in name tables: the bank scan now collects export names (+40 named BANK sounds,
+  links 71 % -> 92 %). Version*.int (build IDs) are not treated as UI text.
+- 2026-10-10 INCIDENT: a crashed e2e_features run (exception before its revert) left ITA:8108E77E:0 modified,
+  and a catalog rebuild then recorded the modified bytes. Recovered: reverted from the run's temporary store,
+  ITA WAD sha1 1238172C... again, every other full copy in that store equal to the game, ITA.csv/SUB_ITA.csv
+  restored from git and an identical rebuild. Prevention: build_catalog stops when a sound differs from the
+  catalog (`changed_since_catalog`), all test tools revert in `finally`, e2e_features writes only ORIGINAL sounds.
+- 2026-10-10: source lists of another language (`source/<lang>/`) are resolved through THAT language's WAD
+  positions; ~1,270 speech sounds per language have no game line name (efforts 519, radio/TV 139, cinematics
+  134) and keep position names (`Sound.is_position_name`): their labels stay per language and mod packages from
+  another language are not matched to them. (Superseded the same day by the line identities of `ackit/lines.py`;
+  `is_position_name` was removed.)

README.md

@@ -1,18 +1,22 @@
 # Arkham City Audio Toolkit
 
-Replace any voice line, music track or ambience in **Batman: Arkham City (PC, Steam)** with your own audio:
-by name, from a file or a recording, with one-click revert. No Wwise, no game patch, no external programs.
+Replace any voice line, music track, ambience or sound effect in **Batman: Arkham City (PC, Steam)** with your own
+audio (by name, from a file or a recording) and edit subtitles and menu text, with one-click revert. No Wwise, no
+game patch, no external programs.
 
 - **Unheard City** (`app/`, [user guide](docs/user-guide.md)): the desktop editor for Windows and Linux. Browse and
-  search the ~11,500 sounds of the game's audio archives, filter by category and character, listen, replace or
-  record, apply and revert. Most short effects (footsteps, foley, weapons, UI) live inside soundbanks and are
-  not listed yet ([docs/formats/banks.md](docs/formats/banks.md)).
+  search all ~21,000 sounds of one language (its voice lines, music, ambience, and the effects stored inside
+  soundbanks: footsteps, foley, combat, weapons, UI), filter by category and character, listen, replace or
+  record, apply and revert. Edit subtitles (linked to their speech lines) and the menu text in their own
+  sections. One Language setting (default: the game's) applies to sounds, subtitles and menu text.
 - **`ackit`** (`ackit/`): the Python library and command-line tool behind it.
 
 How it works in one paragraph: the game's Wwise 2011.1 decoder only knows the standard Vorbis codebooks, so ackit
 encodes with libvorbis and rewrites the stream into exactly the layout the game expects (codebook IDs, packet
-format, seek table, header hash and memory size reversed from the game), then writes it into the `.wad`
-archives atomically, keeping the original bytes in a store for exact reverts. Details in `docs/`.
+format, seek table, header hash and memory size reversed from the game), then writes it in place into the
+`.wad` archives, the soundbanks and the game's compressed `.upk` packages (ackit reads and patches them itself),
+keeping the original files in a store for exact reverts. Subtitles live in the packages too, menu text in the
+localization files. Details in `docs/`.
 
 ## Install
 - **Editor (users), Windows:** unzip `UnheardCity-<version>-windows-amd64.zip` and double-click **Install.cmd**.
@@ -23,23 +27,26 @@ archives atomically, keeping the original bytes in a store for exact reverts. De
 - **Toolkit (developers):** Python 3.10+:
   ```
   python -m venv .venv && .venv/Scripts/activate        # Linux: source .venv/bin/activate
-  pip install -e .[app,dev]                              # app = editor, dev = tests/packaging
+  pip install -e .[app,dev,verify]                       # app = editor, dev = tests/packaging, verify = in-game test
   python -m app.main                                     # run the editor from source
   ```
 
 ## Quick start (command line)
 ```
-ackit find joker tape                            # search by name, category or character
-ackit extract Speech_TAPE_JOKER_1_4 orig.wav     # listen to the original
-ackit replace Speech_TAPE_JOKER_1_4 my_line.wav --match-loudness
-ackit revert Speech_TAPE_JOKER_1_4               # or: ackit revert --all
+ackit find joker tape                             # search by name, category or character
+ackit extract Speech_TAPE_JOKER_1_04 orig.wav     # listen to the original (a line has one name in every language)
+ackit replace Speech_TAPE_JOKER_1_04 my_line.wav --match-loudness
+ackit revert Speech_TAPE_JOKER_1_04               # or: ackit revert --all
+ackit text find --lang INT protocol             # subtitles and menu texts (keys SUB:... / MENU:...)
+ackit text set "MENU:INT:FRONTEND.int:Buttonpress:DoubleTap" "Tap twice"
 ```
 More: `ackit status`, `ackit list --character Joker`, `ackit export-mod my.acmod` / `ackit import-mod`,
-`ackit verify` (offline self-check), `ackit game` (detected install, language, store). `ackit -h` lists all
-commands. The game folder is found through Steam (Windows, Linux, Flatpak, Proton); otherwise pass `--game DIR`.
+`ackit verify` (offline self-check), `ackit verify --catalog` (catalog vs game files), `ackit game` (detected
+install, language, store). `ackit -h` lists all commands. The game folder is found through Steam (Windows, Linux, Flatpak, Proton); otherwise pass `--game DIR`.
 
 Rules worth knowing: speech/effects replacements must fit in the original's size (the tool squeezes or refuses
-with the maximum length); music can be any length. Close the game before writing.
+with the maximum length); music can be any length. Subtitles must fit the original text's length (menu text:
+any length). Close the game before writing.
 
 ## Where things are
 ```
@@ -49,9 +56,14 @@ app/            Unheard City (PySide6)
 tests/          pytest suite (synthetic game data; real-game tests run when the game is installed)
 data/catalog/   sound catalogs (CSV), character lists (source/), schema in data/catalog/README.md
 docs/           formats, reverse engineering, decisions, categories, user guide, building
-tools/          catalog builder, packaging (build_app.py, installer/), artwork and license scripts
+tools/          catalog builder, docs figure check, real-game e2e checks (e2e_editor.py, e2e_features.py), packaging
+                (build_app.py, installer/), artwork, license and screenshot scripts
 NOTES.md        project status and working notes; CLAUDE.md: guidance for AI sessions
 ```
 
-Tests: `pytest` (the real-game tests are skipped when the game is not installed).
-In-game test (Windows, Steam running, WinDbg installed): `ackit verify --ingame`.
+Tests: `pytest` (the real-game tests are skipped when the game is not installed). Figures in the docs:
+`python tools/check_docs.py [--scan] [--wems]`.
+Real-game write round trips (no game launch): `python tools/e2e_features.py [--lang all]`,
+`python tools/e2e_editor.py`.
+In-game tests (Windows, Steam running, WinDbg installed): `ackit verify --ingame`, `--ingame --banks`,
+`--ingame --texts`.

REPORT.md

@@ -1,38 +1,99 @@
-# Report — repository cleanup and the editor, now Unheard City (mission 2)
+# Report — full-game audio, subtitles and menu text (mission 3)
+
+Brief: `docs/history/mission3_audio_text_prompt.md`. Earlier missions: the WAD/WEM toolkit (mission 1) and the
+editor Unheard City (mission 2).
 
 ## What works (verified)
-- **Repository**: clean layout (`ackit/`, `app/`, `tests/`, `data/catalog/`, `docs/`, `tools/`); old state in git
-  tag `pre-cleanup`. One CLI (`ackit`) and one app (`UnheardCity`).
-- **Toolkit**: portable encoder (libvorbis via libsndfile; no Wwise, ffmpeg or vgmstream), WAD/pack injector,
-  originals store with byte-exact revert, conflict detection, mod packages, Steam/Linux/Proton game discovery.
-- **Catalog**: CSVs for ITA/INT/FRA/DEU/ESN/SFX (11,467 sounds per speech language set), categories, verified
-  speakers from the user's lists (3,366 lines in every language), inferred characters for the rest, original
-  sizes and checksums. Schema in `data/catalog/README.md`.
-- **Editor**: browse/search/filter, listen (A/B, loop, waveforms), replace from file or recording, trim/gain/
-  normalize/loudness, apply/revert/revert all, changed-underneath warnings, notes/tags/favorites, mod
-  export/import, settings.
-- **Tests**: 75 automated tests (synthetic game, real-game read-only tests, offscreen GUI tests) pass.
-  `ackit verify` (offline self-check on the installed game) PASS.
-- **Real game**: `ackit verify --ingame` PASS (ncc 1.0); `tools/e2e_editor.py` — replacement applied through the
-  editor is heard in the game (ncc 1.0, no crash), revert leaves WAD and pack index byte-identical; a speech line
-  applied and reverted byte-identically. Game files are original at the end.
-- **Package (Windows x86-64)**: `dist/UnheardCity-2.0.0-windows-amd64.zip` (78 MB) with
-  `THIRD_PARTY_LICENSES.txt`; runs its smoke test and starts the GUI with `PATH` reduced to `C:\Windows`.
-  Contains a per-user installer (`Install.cmd`): install, upgrade and uninstall (Settings > Apps) verified.
+- **Every sound of the game is cataloged and editable**: the speech WADs (5 languages x 10,816), the SFX WAD (651)
+  and, new, the **15,355 in-memory sounds** stored inside soundbanks (archive `BANK`: footsteps, foley, combat,
+  weapons, UI, efforts; 8,092 shared by all languages, ~1,450 localized speech lines per voice language, 6
+  shared by several) in 65 loose `.bnk` files and 975 `.upk` packages. ackit reads and patches the
+  LZO-compressed UE3 packages itself (`ackit/upk.py`, `lzo.py`; no external tools). A replacement goes into every
+  copy (93,826 copies). 445 WAD voice recordings (89 lines) also have bank copies (396 prefetch heads, 49
+  whole copies; 2,185 copies), which are rewritten with them.
+  Names: 13,501 BANK sounds are named after the Wwise event that plays them (`Play_...`, `Speech_...`), the
+  other 1,854 after their bank.
+- **Catalog check**: `ackit verify --catalog` compares the catalog with a fresh scan of the game (missing, extra,
+  duplicate, size/checksum, category, character, language, bank copies, subtitle copies/texts) and re-derives
+  every relation from the game files (lines, names, subtitle links, prefetch heads, copies): **catalog OK** on
+  the full game.
+- **Subtitles**: found in `RDialogueEvent` objects inside the packages (9,032 speech lines x 11 languages) and
+  in 41 scripted `RSubtitle` objects (radio, encryption cards, Riddler; linked through `RSubtitleLookup`),
+  parsed, cataloged (`SUB_<LANG>.csv`) and linked to their voice recordings through the soundbanks: 103,304 of
+  103,658 strings in 11 languages (INT 9,373 of 9,404), each on the same line in every language; an event that
+  plays two versions of a line links both. Edited in place in every copy (168,688 copies).
+- **One recording per language, from the game data alone** (`docs/formats/banks.md`, *Lines*): every voice
+  recording is tied to its line by the game's Wwise Sound objects (same IDs in every language), and each
+  WAD recording to its media by its own language's BKPK table (10,816 / 10,816 in each language). Result:
+  12,434 lines (WAD + in-memory) with exactly one recording in each of the 5 voice languages, no gap and no
+  conflict; recordings stored twice (5 per language) are written together. Independent checks: every WAD
+  recording is a source in a bank of its own language (54,055 / 54,055), every prefetch head or whole bank copy
+  is the start of its recording (2,185 / 2,185). Each line has one name in every language and carries its identity (`line`):
+  speaker labels (3,366 in every language), subtitles and mod packages (another language: the same line)
+  follow it. Speaker labels vs the game's dialogue speaker field: 2,766 agree, 19 differ (categories.md).
+- **Menu text**: the localization files (`BmGame\Localization\<LANG>\*.int`, ~7,000 strings per language),
+  edited value by value; everything else in the files stays byte-identical.
+- **Editor**: one *Language* setting (default: the game's language) for voices, subtitles and menu text
+  (languages without voices use the INT ones, like the game); tabs *Sounds* / *Subtitles* / *Menu text*. The sound
+  editor shows Listen / Replace / Subtitle side by side (resizable columns, no scrolling at usual window sizes);
+  the subtitle list links back to the sound. Search, filters (character,
+  category, file, modified), room counter, apply/revert/revert all, find & replace on the filtered rows, mod
+  export/import with texts, changed-outside banner for sounds and texts.
+- **Safety**: every changed file (WAD, bank, package, localization file) is rebuilt from a verified original copy
+  in the store + the edits applied now; reverting the last edit restores it byte-for-byte; files changed by
+  another tool or Steam are detected (whole-file SHA-1) and not overwritten without confirmation.
+- **Tests**: 135 automated tests (unit, synthetic game with packages/banks/dialogues/localization, a
+  two-language game for the lines (incl. a recording also stored whole in a bank), the shipped catalog's cross-references, real-game read-only (every
+  name of every language's catalog resolves to its own sound), offscreen GUI incl. list order, selection and
+  loading overlays) pass. `ackit verify` PASS (incl.
+  300/300 in-memory WEMs rebuilt byte-identically).
+- **Real game** (2026-10-10), all reverting and checking byte-identity themselves:
+  `tools/e2e_features.py --lang all` 89/89 PASS in all 11 languages (every copy of an in-memory sound, each
+  language's localized line, prefetch heads, a line also stored whole in banks, a multi-copy subtitle, a scripted subtitle, a menu text, sound +
+  subtitle in one package, a mod moved to another voice language and matched to the same line at another
+  position); `ackit verify --ingame` PASS (music heard, ncc 1.0);
+  `--banks` PASS (77 in-memory sounds of the front-end packages replaced in 54 files: the game loads them, 71
+  probes found in its memory, no crash); `--texts` PASS (12/12 menu texts and 23/23 subtitles of Startup.upk
+  found in the game's memory, no crash); `tools/e2e_editor.py` PASS. Game files original at the end
+  (`ackit status`). The catalog builder refuses to run on a modified game, so the catalog always describes the
+  original files.
+- **Package**: `dist/UnheardCity-3.2.3-windows-amd64.zip` (86 MB, catalogs included). With `PATH` reduced to
+  the Windows folders the packaged exe passes its smoke test (in the game's language - Italian on the test PC:
+  21,015 sounds loaded, encode + inject, 9,377 subtitles and 7,020 menu texts loaded, an in-memory sound decoded from a package) and the GUI starts and
+  quits cleanly. The installer is unchanged since mission 2 (install/upgrade/uninstall verified then).
 
-## What is not verified / limitations
-- **Coverage**: only sounds in the WAD archives are listed and editable. About 15,800 short effects (footsteps,
-  foley, weapons, UI, grunts) are stored inside soundbanks (loose `.bnk` and banks embedded in `.upk`
-  packages) and are not covered yet (`docs/formats/banks.md`).
-- Speech lines being *heard* in the game needs gameplay; verified indirectly (same decoder/WAD path as the music
-  test, byte-level apply/revert, boot test with modified speech WAD).
-- Speech/effects replacements must fit the original's size (index lives in compressed `.upk` packages).
-- Channel order for 4-6 channel sounds is unverified (tests used identical channels).
-- Inputs: WAV/FLAC/OGG/OPUS/MP3/AIFF built in; M4A/AAC/WMA only with an ffmpeg installed on the system.
-- Character labels outside the user's lists are inferred from names (marked as such), not verified speakers.
+## Unresolved / limitations
+- **Checked by a person in game** (2026-10-10), not automatically (nothing of this plays or shows on the title
+  screen without input, findings 18): replaced voice lines and edited subtitles are heard and shown; replaced
+  footsteps play on every step. That test found the one defect the automatic checks could not: in-memory
+  replacements followed by zero padding in their slot played as silence (fixed in 3.2.2, findings 24).
+- **Size limits**: in-memory sounds and speech must fit their original size; subtitles must fit the original
+  text's length ("room"); cp1252 subtitles accept cp1252 characters only. Lifting these limits means moving
+  objects inside the packages (new export offsets/sizes, added chunks): designed but not done, because it could
+  not be verified automatically in the game (decisions 4c).
+- **Unlinked subtitles**: 24 speech names (24-36 strings per language, INT 31: Riddler trophy texts, Arkham City
+  Stories, radio reports and tapes) have no event in any bank; they are listed and editable but not linked.
+- **Subtitle coverage is uneven in the game itself** (INT voice lines with a subtitle): none for efforts/grunts,
+  few for combat barks (9 %), cinematics and boss fights (23 %; most cutscene lines are not played through speech
+  events and have no dialogue object), 89-97 % for story, predator, character and open-world lines. No other
+  subtitle storage was found in the packages (all classes with subtitle data checked).
+- 637 voice lines are played by no event whose name the game stores; they are named after their English
+  recording's position (the same name in every language). Mod packages from versions before 3.1 carry no line
+  identity: their voice lines are not moved to another language (listed instead of guessed).
+- Speaker labels: 2 names of the curated lists match two recordings each (wemscan name collision), both are
+  labelled; 19 labels differ from the game's dialogue speaker field (to be checked by listening).
+- **Unnamed sounds**: 1,854 BANK sounds are reached by no event whose name the game stores and are named
+  `<Bank>_<n>`.
+- 25 SFX originals (music seek tables and sample counts, looping ambience) are not reproduced byte-for-byte by
+  the WEM rules (`docs/formats/wem.md`); writing is not affected (replaced music plays in game).
+- Two copies of media `1C898731` differ; its checksum is left empty. Channel order for 4-6 channel sounds is
+  unverified. M4A/AAC/WMA input needs an ffmpeg on the system.
+- Another game build is scanned at first start (WADs: seconds; banks, lines and subtitles: a few minutes,
+  cached); it gets the same line identities, but no original checksums.
 
 ## Next steps
-1. Support the in-memory sounds in soundbanks: the loose footstep banks first, then the banks embedded in
-   `.upk` packages (`docs/formats/banks.md`).
-2. Extend the curated speaker lists (`data/catalog/source/<lang>/*.txt`) and rebuild with `tools/build_catalog.py`.
-3. Optional: rewrite the BKPK indexes inside `.upk` packages (LZO) to lift the size limit for speech.
+1. Optional: lift the subtitle length limit by relocating dialogue objects (checked in game by a person).
+2. Build and smoke-test the Linux package once on a Linux machine.
+3. Listen to the 19 lines whose speaker label differs from the game's dialogue speaker field
+   (`docs/categories.md`) and fix the curated lists if needed; extend them (`data/catalog/source/<lang>/*.txt`)
+   and rebuild the catalog.

ackit/__init__.py

@@ -1,10 +1,11 @@
 """ackit - Batman: Arkham City (PC) audio toolkit.
 
 Library layers (low -> high):
-  formats   wad, pack, wem, ogg, bits, codebooks, wwvorbis, allocsize, builder, convert, known_setups
+  formats   wad, pack, bnk (soundbanks), upk + lzo (UE3 packages), wem, ogg, bits, codebooks, wwvorbis,
+            allocsize, builder, convert, known_setups
   audio     audio (load/convert/decode), encoder (PCM -> game WEM)
   game      paths, game (install discovery), catalog + classify (names/categories/characters),
             inject (WAD writing), originals (store), workspace (apply/revert), modpack (share)
   tools     cli (`ackit` command), verify/ (offline self-check, in-game test on Windows)
 """
-__version__ = "2.0.0"
+__version__ = "3.2.3"

ackit/audio.py

@@ -12,9 +12,6 @@ import subprocess
 import numpy as np
 import soundfile as sf
 
-READABLE = (".wav", ".flac", ".ogg", ".oga", ".opus", ".mp3", ".aif", ".aiff", ".au", ".w64", ".caf")
-FFMPEG_ONLY = (".m4a", ".aac", ".wma", ".mp4", ".webm", ".mkv", ".ac3")
-
 
 class AudioError(Exception):
     pass

ackit/banks.py

@@ -0,0 +1,463 @@
+"""In-memory sounds: WEMs held inside soundbanks, i.e. the loose SFX\\*.bnk files and the banks embedded
+in the .upk packages (docs/formats/banks.md, upk.md). Catalog archive `BANK`, target = media ID.
+
+A media ID can be stored many times (the same bank copied into several packages, or several banks
+holding the same media). The location index (data/catalog/BANK_locations.csv.gz) lists every copy:
+  media, file (relative to CookedPCConsole, '/'-separated), offset (in the *uncompressed* file), size, kind
+kind 'm' = an in-memory sound, 'p' = the in-memory head of a streamed WAD sound (prefetch); for 'p' rows
+`sound` is the WAD sound's key (e.g. ITA:2B14E472:3): speech lines have per-language media IDs and heads.
+Replacements are written in place into every copy (<= original size, zero-padded), so bank and package
+layouts never change.
+"""
+import csv
+import gzip
+import hashlib
+import io
+import os
+import re
+import struct
+from collections import defaultdict, namedtuple
+
+from .bnk import Bank, BankError, find_banks, fnv1, PREFETCH
+from .upk import Package, PackageError, is_package
+
+ARCHIVE = "BANK"
+LOCATIONS = "BANK_locations.csv.gz"
+SOUND, MEMORY, PREFETCH_HEAD = 2, "m", "p"
+EVENT, ACTION = 4, 3
+CONTAINERS = (5, 6, 7, 9)            # random/sequence, switch, actor-mixer, layer
+
+# lang: language of the bank holding this copy ('' = language-neutral bank); known after a scan only
+Location = namedtuple("Location", "media file offset size kind sound lang", defaults=("", ""))
+
+
+# --- files ----------------------------------------------------------------------------------------
+def bank_files(cooked):
+    """Relative paths ('/'-separated) of every loose regular bank and every package."""
+    out = []
+    sfx = os.path.join(cooked, "SFX")
+    if os.path.isdir(sfx):
+        for f in sorted(os.listdir(sfx)):
+            if f.lower().endswith(".bnk"):
+                with open(os.path.join(sfx, f), "rb") as fh:
+                    if fh.read(4) == b"BKHD":
+                        out.append("SFX/" + f)
+    for f in sorted(os.listdir(cooked)):
+        if f.lower().endswith(".upk") and is_package(os.path.join(cooked, f)):
+            out.append(f)
+    return out
+
+
+class _Plain:
+    """Uniform read access for a loose bank (offsets = file offsets)."""
+
+    def __init__(self, path):
+        self.path = path
+
+    def read(self, off, n):
+        with open(self.path, "rb") as f:
+            f.seek(off)
+            return f.read(n)
+
+    def read_all(self):
+        return 0, open(self.path, "rb").read()
+
+    def fingerprint(self):
+        return file_fingerprint(self.path)
+
+    def patched(self, edits):
+        out = bytearray(open(self.path, "rb").read())
+        for off, data in edits:
+            out[off:off + len(data)] = data
+        return bytes(out)
+
+
+def file_fingerprint(path):
+    """Identity of a loose bank (small files: content hash)."""
+    return hashlib.sha1(open(path, "rb").read()).hexdigest()[:16]
+
+
+_open_cache = {}
+
+
+def open_file(path):
+    """A reader for a bank file or package (cached while the file is unchanged)."""
+    st = os.stat(path)
+    key = (os.path.normcase(os.path.abspath(path)), st.st_size, st.st_mtime_ns)
+    r = _open_cache.get(key)
+    if r is None:
+        r = Package(path) if path.lower().endswith(".upk") else _Plain(path)
+        if len(_open_cache) > 64:
+            _open_cache.clear()
+        _open_cache[key] = r
+    return r
+
+
+def file_path(cooked, rel):
+    return os.path.join(cooked, *rel.split("/"))
+
+
+def riff_trim(b):
+    """A WEM slot without the zero padding after a smaller replacement (RIFF length)."""
+    if len(b) >= 8 and b[:4] == b"RIFF":
+        n = 8 + struct.unpack_from("<I", b, 4)[0]
+        if n <= len(b):
+            return b[:n]
+    return b
+
+
+# --- HIRC graph (for names) ---------------------------------------------------------------------
+def _hirc_objects(buf, bank):
+    """(type, id, payload memoryview) of every HIRC object of a bank."""
+    if "HIRC" not in bank.chunks:
+        return
+    o, n = bank.chunks["HIRC"]
+    p = bank.at + o
+    end = p + n
+    count = struct.unpack_from("<I", buf, p)[0]
+    p += 4
+    for _ in range(count):
+        if p + 9 > end:
+            return
+        t = buf[p]
+        ln = struct.unpack_from("<I", buf, p + 1)[0]
+        yield t, struct.unpack_from("<I", buf, p + 5)[0], bytes(buf[p + 5:p + 5 + ln])
+        p += 5 + ln
+
+
+def _children(payload, known):
+    """A container's child list: the first `u32 count, count x u32 id` whose ids are all known objects."""
+    n = len(payload)
+    for p in range(4, n - 7):
+        c = struct.unpack_from("<I", payload, p)[0]
+        if 0 < c <= 4096 and p + 4 + 4 * c <= n:
+            ids = struct.unpack_from("<%dI" % c, payload, p + 4)
+            if all(i in known for i in ids):
+                return ids
+    return ()
+
+
+# --- scan ---------------------------------------------------------------------------------------
+def _by_language():
+    return defaultdict(dict)
+
+
+class Scan:
+    """Everything found in the game's banks (built by `scan`)."""
+
+    def __init__(self):
+        self.locations = []                  # [Location]
+        self.files = {}                      # rel -> fingerprint
+        self.media = {}                      # id -> {"size", "sha1": set, "head": bytes, "banks": set}
+        self.bank_names = {}                 # bank id -> name
+        self.names = set()                   # identifier-like names seen in packages (for event names)
+        self.objects = {}                    # HIRC id -> type
+        self.containers = {}                 # id -> payload
+        self.events = {}                     # id -> [action ids]
+        self.actions = {}                    # id -> target id
+        self.sound_media = defaultdict(set)  # sound object id -> media ids
+        self.prefetch = set()                # media ids whose bank copies belong to a WAD recording (prefetch
+                                             # heads; whole recordings: catalog._whole_recordings)
+        self.lang_media = defaultdict(set)   # language -> media ids named by Sound objects of that language's banks
+        # BKPK tables: WAD entry id -> {language: {media id: (offset in entry, size)}}. A table belongs to the
+        # language its localized AkBank export declares ('' = outside such an export). The same media ID can be
+        # used by several languages (at a different position in each language's WAD), so tables never merge.
+        self.packs = defaultdict(_by_language)
+        self.subtitle_objects = {}           # counts of dialogue / RSubtitle objects (catalog.scan_text)
+        self.issues = []                     # (kind, detail)
+
+
+_LANG = re.compile(rb"([\x01-\x10])\x00\x00\x00\x04\x00\x00\x00[A-Z]{3}\x00")
+
+
+def localized_exports(data, start, imports, exports):
+    """[(begin, end, [languages])] of the localized AkBank exports of a package (offsets in data). Such an
+    export starts with its list of languages (FStrings, e.g. INT FRA ITA DEU ESN) and then holds one block per
+    language, in that order: a bank and/or the BKPK table of the speech WAD entry (docs/formats/banks.md)."""
+    out = []
+    akbank = {-(i + 1) for i, (c, o, _) in enumerate(imports) if c == "Class" and o == "AkBank"}
+    for x in exports:
+        if x.cls not in akbank or x.size <= 0:
+            continue
+        o0 = x.offset - start
+        m = _LANG.search(data, o0, min(o0 + 256, o0 + x.size))
+        if not m:
+            continue
+        n = m.group(1)[0]
+        langs, o = [], m.start() + 4
+        for _ in range(n):
+            ln = struct.unpack_from("<i", data, o)[0]
+            langs.append(bytes(data[o + 4:o + 3 + ln]).decode("latin-1"))
+            o += 4 + ln
+        out.append((o0, o0 + x.size, langs, x.name))
+    return out
+
+
+def _assign(positions, regions, issues, rel, what):
+    """{position: language} for the blocks (banks or BKPK tables) inside localized exports: the k-th block of
+    an export belongs to its k-th language. An export whose block count differs is reported, not guessed."""
+    out = {}
+    for o0, o1, langs, name in regions:
+        inside = [a for a in positions if o0 <= a < o1]
+        if not inside:
+            continue
+        if len(inside) != len(langs):
+            if issues is not None:
+                issues.append(("languages", "%s: %s has %d %s for %d languages" % (rel, name, len(inside), what, len(langs))))
+            continue
+        out.update(zip(inside, langs))
+    return out
+
+
+def bkpk_positions(data):
+    """Offsets of the BKPK tables (docs/formats/wad.md) embedded in a package."""
+    out = []
+    for m in re.finditer(b"BKPK", data):
+        p = m.start()
+        if p + 0x48 > len(data) or struct.unpack_from("<I", data, p + 0x1C)[0] != 0x48:
+            continue
+        n = struct.unpack_from("<I", data, p + 0x10)[0]
+        if n <= 100000 and p + 0x48 + 12 * n <= len(data):
+            out.append(p)
+    return out
+
+
+def read_bkpk(data, p):
+    """(WAD entry id, {media id: (offset in entry, size)}) of the BKPK table at p."""
+    entry, n = struct.unpack_from("<I", data, p + 8)[0], struct.unpack_from("<I", data, p + 0x10)[0]
+    table = {}
+    for i in range(n):
+        mid, size, off = struct.unpack_from("<III", data, p + 0x48 + 12 * i)
+        table[mid] = (off, size)
+    return entry, table
+
+
+def _bkpk_tables(data, packs, regions, issues=None, rel=""):
+    """Add a package's BKPK tables to `packs` under the language their export declares. Copies of a table
+    in several packages must agree (reported otherwise)."""
+    pos = bkpk_positions(data)
+    lang_at = _assign(pos, regions, issues, rel, "BKPK tables")
+    for p in pos:
+        entry, table = read_bkpk(data, p)
+        lang = lang_at.get(p, "")
+        have = packs[entry].get(lang)
+        if have and have != table and issues is not None:
+            issues.append(("bkpk", "%s: table of %08X (%s) differs from another copy" % (rel, entry, lang or "-")))
+        packs[entry][lang] = table
+
+
+def scan(cooked, progress=None, cooked_banks=None, files=None, visit=None):
+    """Scan every loose bank and package of the game (minutes), or only `files`. Returns a `Scan`.
+    `visit(rel, package, start, data)` is called for every package (other scans share the decompression)."""
+    sc = Scan()
+    files = bank_files(cooked) if files is None else files
+    if cooked_banks is None:
+        p = os.path.join(cooked, "CookedBanks.txt")
+        cooked_banks = [l.strip() for l in open(p, encoding="latin-1")] if os.path.exists(p) else []
+    sc.bank_names.update({fnv1(n): n for n in cooked_banks if n})
+    for k, rel in enumerate(files):
+        if progress and k % 50 == 0:
+            progress(k, len(files))
+        path = file_path(cooked, rel)
+        try:
+            r = open_file(path)
+            start, data = r.read_all()
+        except (PackageError, OSError) as e:
+            sc.issues.append(("unreadable", "%s: %s" % (rel, e)))
+            continue
+        lang_of = {}
+        if isinstance(r, Package):
+            names, imports, exports = r.tables(data, start)
+            sc.names.update(names)                      # base names ...
+            sc.names.update(x.name for x in exports)    # ... and numbered object names (Speech_X_04)
+            regions = localized_exports(data, start, imports, exports)
+            _bkpk_tables(data, sc.packs, regions, sc.issues, rel)
+            if visit:
+                visit(rel, r, start, data)
+            banks = list(find_banks(data))
+            lang_of = _assign(sorted(bk.at for bk in banks), regions, sc.issues, rel, "banks")
+        else:
+            try:
+                banks = [Bank(data, 0)]
+            except BankError:
+                continue
+        if banks:
+            sc.files[rel] = r.fingerprint()
+        for bk in banks:
+            sc.bank_names.update(bk.names())
+            media = {m.id: m for m in bk.media()}
+            prefetched = set()
+            for s in bk.sources():
+                if s.stream == PREFETCH and s.media in media:
+                    prefetched.add(s.media)
+            for m in media.values():
+                w = bytes(data[bk.at + m.off:bk.at + m.off + m.size])
+                kind = PREFETCH_HEAD if m.id in prefetched else MEMORY
+                if kind == PREFETCH_HEAD:
+                    sc.prefetch.add(m.id)
+                e = sc.media.setdefault(m.id, {"size": m.size, "sha1": set(), "head": w[:0x60], "banks": set(),
+                                               "kinds": set(), "langs": set()})
+                e["langs"].add(lang_of.get(bk.at, ""))
+                e["sha1"].add(hashlib.sha1(w).hexdigest()[:16])
+                e["banks"].add(bk.id)
+                e["kinds"].add(kind)
+                sc.locations.append(Location(m.id, rel, start + bk.at + m.off, m.size, kind, "", lang_of.get(bk.at, "")))
+            for t, oid, pl in _hirc_objects(data, bk):
+                sc.objects[oid] = t
+                if t == EVENT and len(pl) >= 8:
+                    n = struct.unpack_from("<I", pl, 4)[0]
+                    if len(pl) >= 8 + 4 * n:
+                        sc.events[oid] = list(struct.unpack_from("<%dI" % n, pl, 8))
+                elif t == ACTION and len(pl) >= 12:
+                    sc.actions[oid] = struct.unpack_from("<I", pl, 8)[0]
+                elif t in CONTAINERS:
+                    sc.containers[oid] = pl
+            for s in bk.sources():          # every source, prefetched ones too (streamed lines whose start
+                sc.sound_media[s.object].add(s.media)          # is kept in the bank: the game-over taunts)
+                if lang_of.get(bk.at):
+                    sc.lang_media[lang_of[bk.at]].add(s.media)
+    if progress:
+        progress(len(files), len(files))
+    return sc
+
+
+# --- names --------------------------------------------------------------------------------------
+def event_objects(sc):
+    """{event name: [sound object ids in traversal order]} for every event whose name is known. Events,
+    actions, containers and sound objects have the same IDs in every language (only the media differ), so
+    this order is the same in every language."""
+    known = set(sc.objects)
+    kids = {cid: _children(pl, known) for cid, pl in sc.containers.items()}
+    by_hash = {}
+    for n in sc.names:
+        if n and len(n) < 128:
+            by_hash.setdefault(fnv1(n), n)
+    out = {}
+    for eid, acts in sc.events.items():
+        name = by_hash.get(eid)
+        if not name:
+            continue
+        seen, order, stack = set(), [], [sc.actions.get(a) for a in acts][::-1]
+        while stack:
+            o = stack.pop()
+            if o is None or o in seen:
+                continue
+            seen.add(o)
+            if o in sc.sound_media:
+                order.append(o)
+            stack.extend(list(kids.get(o, ()))[::-1])
+        if order:
+            out[name] = order
+    return out
+
+
+def event_media(sc):
+    """{event name: [media ids in traversal order]} for every event whose name is known."""
+    out = {}
+    for name, objs in event_objects(sc).items():
+        order = []
+        for o in objs:
+            for m in sorted(sc.sound_media[o]):
+                if m not in order:
+                    order.append(m)
+        out[name] = order
+    return out
+
+
+def media_language(sc, m):
+    """Language of a localized in-memory media ('' = language-neutral)."""
+    langs = sc.media[m].get("langs", set()) - {""}
+    return next(iter(langs)) if len(langs) == 1 else ""
+
+
+def media_names(sc, reserved=()):
+    """{media id: (display name, primary bank name)}: named after the most specific 'Play' event that
+    reaches the media, otherwise after its bank (<Bank>_<n>). Localized speech has one media per language
+    for the same line: they share the name, variations are numbered within a language. Names are unique
+    among the language-neutral media plus any one language (what a catalog shows at a time) and avoid the
+    lower-case names in `reserved` (other archives' sounds)."""
+    ev = event_media(sc)
+    best = {}
+    for name, ms in ev.items():
+        groups = defaultdict(list)
+        for m in ms:
+            if m in sc.media and PREFETCH_HEAD not in sc.media[m]["kinds"]:     # heads belong to WAD lines
+                groups[media_language(sc, m)].append(m)
+        for g in groups.values():
+            rank = (0 if name.lower().startswith("play") else 1, len(g), name.lower())
+            for i, m in enumerate(g):
+                if m not in best or rank < best[m][0]:
+                    best[m] = (rank, name, i, len(g))
+    out = {}
+    taken = {("", n) for n in reserved}  # (language, lower-case name)
+
+    def free(lang, nm):
+        k = nm.lower()
+        if lang:
+            return (lang, k) not in taken and ("", k) not in taken
+        return not any(n == k for _, n in taken)
+
+    def take(lang, nm, m, base):
+        if not free(lang, nm):
+            nm = "%s_%08X" % (base, m)
+        taken.add((lang, nm.lower()))
+        return nm
+
+    for m, (_, name, i, n) in sorted(best.items(), key=lambda x: (x[1][1].lower(), x[1][2], x[0])):
+        out[m] = take(media_language(sc, m), name if n == 1 else "%s_%d" % (name, i + 1), m, name)
+    # the rest: by bank (and language), numbered in media id order
+    groups = defaultdict(list)
+    for m, e in sc.media.items():
+        bank = sorted(sc.bank_names.get(b, "%08X" % b) for b in e["banks"])[0]
+        if m in out:
+            out[m] = (out[m], bank)
+        else:
+            groups[(bank, media_language(sc, m))].append(m)
+    for (bank, lang), ms in sorted(groups.items()):
+        for i, m in enumerate(sorted(ms)):
+            out[m] = (take(lang, "%s_%d" % (bank, i + 1), m, bank), bank)
+    return out
+
+
+# --- location index -------------------------------------------------------------------------------
+def write_locations(path, locs):
+    raw = io.StringIO()
+    w = csv.writer(raw, lineterminator="\n")
+    w.writerow(["media", "file", "offset", "size", "kind", "sound"])
+    for l in sorted(locs, key=lambda l: (l.media, l.file, l.offset)):
+        w.writerow(["%08X" % l.media, l.file, "%X" % l.offset, l.size, l.kind, l.sound])
+    data = raw.getvalue().encode("utf-8")
+    with open(path + ".tmp", "wb") as f:
+        # mtime=0: identical content gives identical bytes (no spurious git changes)
+        with gzip.GzipFile(fileobj=f, mode="wb", mtime=0) as gz:
+            gz.write(data)
+    os.replace(path + ".tmp", path)
+
+
+def read_locations(path):
+    """{media id: [Location]}"""
+    out = defaultdict(list)
+    with gzip.open(path, "rt", encoding="utf-8", newline="") as f:
+        for r in csv.DictReader(f):
+            m = int(r["media"], 16)
+            out[m].append(Location(m, r["file"], int(r["offset"], 16), int(r["size"]), r["kind"], r["sound"]))
+    return dict(out)
+
+
+def game_fingerprint(cooked):
+    """Cheap identity of the game's bank set: CookedBanks.txt + names/sizes of the loose banks.
+    (Each file is checked again before it is written: `file_fingerprint` / Package.fingerprint.)"""
+    h = hashlib.sha1()
+    p = os.path.join(cooked, "CookedBanks.txt")
+    if os.path.exists(p):
+        h.update(open(p, "rb").read())
+    sfx = os.path.join(cooked, "SFX")
+    if os.path.isdir(sfx):
+        for f in sorted(os.listdir(sfx)):
+            if f.lower().endswith(".bnk"):
+                h.update(("%s:%d;" % (f.upper(), os.path.getsize(os.path.join(sfx, f)))).encode())
+    return h.hexdigest()[:16]
+
+
+def has_banks(cooked):
+    return os.path.exists(os.path.join(cooked, "CookedBanks.txt"))

ackit/bits.py

@@ -14,9 +14,6 @@ class BitReader:
             self.pos += 1
         return v
 
-    def bits_left(self) -> int:
-        return len(self.data) * 8 - self.pos
-
 
 class BitWriter:
     def __init__(self):
@@ -31,13 +28,6 @@ class BitWriter:
                 self.buf[self.pos >> 3] |= 1 << (self.pos & 7)
             self.pos += 1
 
-    def put_bits_from(self, r: BitReader, n: int):
-        """Copy n bits from a reader (fast path in 32-bit chunks)."""
-        while n > 0:
-            k = 32 if n >= 32 else n
-            self.put(k, r.get(k))
-            n -= k
-
     def bytes(self) -> bytes:
         return bytes(self.buf)
 

ackit/bnk.py

@@ -0,0 +1,108 @@
+"""Wwise soundbanks (bank version 56): chunks, in-memory media (DIDX/DATA) and HIRC sound sources.
+
+Works on a buffer that may hold a bank anywhere (a loose .bnk file, a BKPK pack's embedded bank, or
+an uncompressed .upk package with many banks). Offsets are relative to the start of the bank.
+Format: docs/formats/banks.md.
+"""
+import re
+import struct
+from collections import namedtuple
+
+CHUNKS = (b"BKHD", b"DIDX", b"DATA", b"HIRC", b"STID", b"STMG", b"ENVS", b"FXPR")
+SOUND = 2                         # HIRC object type of a Sound
+IN_MEMORY, STREAMED, PREFETCH = 0, 1, 2
+
+Media = namedtuple("Media", "id off size")                     # off: from bank start
+Source = namedtuple("Source", "object stream media bank off size pos")  # pos: of `off` field, from bank start
+
+
+class BankError(ValueError):
+    pass
+
+
+class Bank:
+    def __init__(self, buf, at=0):
+        self.buf, self.at = buf, at
+        self.chunks = {}
+        p = at
+        while p + 8 <= len(buf) and bytes(buf[p:p + 4]) in CHUNKS:
+            tag = bytes(buf[p:p + 4]).decode()
+            n = struct.unpack_from("<I", buf, p + 4)[0]
+            if p + 8 + n > len(buf) or tag in self.chunks:
+                break
+            self.chunks[tag] = (p + 8 - at, n)
+            p += 8 + n
+        if "BKHD" not in self.chunks or self.chunks["BKHD"][1] < 8:
+            raise BankError("no bank at 0x%X" % at)
+        self.size = p - at
+        o = at + self.chunks["BKHD"][0]
+        self.version, self.id = struct.unpack_from("<II", buf, o)
+
+    def _abs(self, tag):
+        o, n = self.chunks[tag]
+        return self.at + o, n
+
+    def media(self):
+        """In-memory WEMs: [Media(id, offset from bank start, size)]."""
+        if "DIDX" not in self.chunks or "DATA" not in self.chunks:
+            return []
+        o, n = self._abs("DIDX")
+        data = self.chunks["DATA"][0]
+        out = []
+        for i in range(n // 12):
+            mid, off, size = struct.unpack_from("<III", self.buf, o + 12 * i)
+            out.append(Media(mid, data + off, size))
+        return out
+
+    def sources(self):
+        """Sources of the HIRC Sound objects: [Source]. In-memory sources carry offset/size."""
+        if "HIRC" not in self.chunks:
+            return []
+        o, n = self._abs("HIRC")
+        end = o + n
+        count = struct.unpack_from("<I", self.buf, o)[0]
+        p = o + 4
+        out = []
+        for _ in range(count):
+            if p + 5 > end:
+                raise BankError("HIRC overruns its chunk in bank %08X" % self.id)
+            t = self.buf[p]
+            ln = struct.unpack_from("<I", self.buf, p + 1)[0]
+            if t == SOUND and ln >= 28:
+                obj, _plugin, stream, mid, bank = struct.unpack_from("<5I", self.buf, p + 5)
+                off, size = struct.unpack_from("<II", self.buf, p + 25)
+                out.append(Source(obj, stream, mid, bank, off, size, p + 25 - self.at))
+            p += 5 + ln
+        return out
+
+    def names(self):
+        """STID: {bank id: name}."""
+        if "STID" not in self.chunks:
+            return {}
+        o, n = self._abs("STID")
+        count = struct.unpack_from("<I", self.buf, o + 4)[0]
+        p = o + 8
+        out = {}
+        for _ in range(count):
+            bid, ln = struct.unpack_from("<IB", self.buf, p)
+            out[bid] = bytes(self.buf[p + 5:p + 5 + ln]).decode("latin-1")
+            p += 5 + ln
+        return out
+
+
+def find_banks(buf, start=0):
+    """Every bank in `buf` (e.g. an uncompressed package): yields Bank."""
+    for m in re.finditer(b"BKHD", buf[start:] if start else buf):
+        try:
+            yield Bank(buf, m.start() + start)
+        except (BankError, struct.error):
+            continue
+
+
+def fnv1(name):
+    """Wwise ID of a name (bank, event, ...): FNV-1 32 of the lower-cased name."""
+    h = 2166136261
+    for c in name.lower().encode("utf-8"):
+        h = (h * 16777619) & 0xFFFFFFFF
+        h ^= c
+    return h

ackit/catalog.py

@@ -5,10 +5,12 @@ Sources, in order:
   1. data/catalog/<ARCHIVE>.csv  -- shipped catalog (instant) when the WAD fingerprint matches;
   2. otherwise a header scan of the WAD (a few seconds, cached in the user cache), classified with
      `classify` and labelled with data/catalog/characters.csv (speaker per sound name).
-Names follow the community hash list (ackit/data/hashlist.txt): an entry with one WEM is NAME
+Names come from the game: a WAD entry ID is the Wwise ID (FNV-1) of a soundbank listed in the game's
+CookedPCConsole/CookedBanks.txt (`bank_names`); an entry with one WEM is NAME
 (alias NAME_1), entries with n variations are NAME_1..NAME_n (alias NAME#i); stream-pack media are
 named by their ID or <Pack>_<ID>.
 """
+import collections
 import csv
 import difflib
 import json
@@ -17,16 +19,19 @@ import re
 import struct
 from dataclasses import dataclass, asdict
 
-from . import classify
+from . import banks, classify, subtitles
+from . import lines as lines_mod
 from .game import wad_fingerprint
+from .bnk import fnv1
+from .lines import bank_langs
 from .inject import pack_bnk_path
 from .pack import Pack
-from .paths import catalog_dir, package_data, user_dir
+from .paths import catalog_dir, user_dir
 from .wad import Wad
 
-FIELDS = ["archive", "target", "name", "aliases", "group", "category", "character",
+FIELDS = ["archive", "target", "name", "aliases", "group", "category", "bank", "language", "line", "character",
           "characters_inferred", "seconds", "rate", "channels", "size", "sha1", "notes"]
-SCAN_VERSION = 1
+SCAN_VERSION = 3
 
 
 @dataclass
@@ -37,6 +42,9 @@ class Sound:
     aliases: tuple = ()
     group: str = "Unknown"
     category: str = "Unknown"
+    bank: str = ""                  # BANK sounds: the soundbank holding it (name); WAD sounds: ''
+    language: str = ""              # BANK sounds: language(s) of a localized speech line, ';'-separated ('' = every language)
+    line: str = ""                  # speech: the game's identity of the line in every language (ackit/lines.py)
     character: str = ""             # verified speaker(s), '; '-separated (curated catalog)
     characters_inferred: str = ""   # characters/roles mentioned by the name (hint)
     seconds: float = 0.0
@@ -62,41 +70,67 @@ class Sound:
     def names(self):
         return (self.name,) + tuple(self.aliases)
 
+    @property
+    def languages(self):
+        """Voice languages of a localized in-memory sound (() = every language)."""
+        return tuple(self.language.split(";")) if self.language else ()
+
+    def in_language(self, lang):
+        return not self.language or lang in self.languages
+
 
 # --- static data ---------------------------------------------------------------------------
-_names = None
+_names = {}                     # cooked folder -> {Wwise ID: bank name}
 
 
-def hash_names():
-    global _names
-    if _names is None:
-        _names = {}
-        for ln in open(package_data("hashlist.txt"), encoding="latin-1"):
-            p = ln.rstrip("\r\n").split("\t")
-            if len(p) >= 2 and p[1]:
-                try:
-                    _names[int(p[0], 16)] = p[1]
-                except ValueError:
-                    pass
-    return _names
+def bank_names(cooked):
+    """{Wwise ID: name} of the game's soundbanks, from its own CookedBanks.txt (ID = FNV-1 of the name). They
+    name the WAD entries (each entry holds the streamed media of one bank) and the stream packs."""
+    key = os.path.normcase(os.path.abspath(cooked))
+    if key not in _names:
+        out = {}
+        p = os.path.join(cooked, "CookedBanks.txt")
+        if os.path.exists(p):
+            for ln in open(p, encoding="latin-1"):
+                n = ln.strip()
+                if n:
+                    out.setdefault(fnv1(n), n)
+        _names[key] = out
+    return _names[key]
 
 
-def load_characters(path=None):
-    """{sound name (lower): 'Speaker' or 'A; B'} from data/catalog/characters.csv."""
+def cooked_of(wad_path):
+    """CookedPCConsole folder of a WAD (WADs are in CookedPCConsole/<language or SFX>/)."""
+    return os.path.dirname(os.path.dirname(os.path.abspath(wad_path)))
+
+
+def load_characters(path=None, lang=None):
+    """{sound name (lower): 'Speaker' or 'A; B'} from data/catalog/characters.csv: the rows valid in every
+    language (keyed by line names, the same line everywhere) plus the rows of `lang` (a sound of that
+    language only; none at present)."""
     path = path or os.path.join(catalog_dir(), "characters.csv")
     out = {}
     if os.path.exists(path):
         for row in csv.DictReader(open(path, encoding="utf-8")):
-            out[row["name"].lower()] = row["character"]
+            if not row.get("language") or row["language"] == lang:
+                out[row["name"].lower()] = row["character"]
     return out
 
 
+def character_of(s, characters):
+    """Verified speaker of a sound from a load_characters() map, by its name only: names are unique in a
+    language and a line has the same name in every language, while aliases (positions) are local."""
+    return characters.get(s.name.lower(), "")
+
+
 def read_csv(path):
     out = []
     for r in csv.DictReader(open(path, encoding="utf-8", newline="")):
         out.append(Sound(archive=r["archive"], target=r["target"], name=r["name"],
                          aliases=tuple(a for a in r["aliases"].split("|") if a), group=r["group"],
-                         category=r["category"], character=r["character"],
+                         category=r["category"], bank=r.get("bank") or "", language=r.get("language") or "",
+                         line=r.get("line") or "",
+                         character=r["character"],
                          characters_inferred=r["characters_inferred"], seconds=float(r["seconds"] or 0),
                          rate=int(r["rate"] or 0), channels=int(r["channels"] or 0), size=int(r["size"] or 0),
                          sha1=r["sha1"], notes=r["notes"]))
@@ -155,8 +189,8 @@ def _walk_entry(f, offset, size):
 
 def scan_archive(archive, wad_path, characters=None, progress=None):
     """Header scan of one WAD -> [Sound]."""
-    names = hash_names()
-    characters = characters if characters is not None else load_characters()
+    names = bank_names(cooked_of(wad_path))
+    characters = characters if characters is not None else load_characters(lang=archive)
     wad = Wad(wad_path)
     out = []
     with open(wad_path, "rb") as f:
@@ -185,12 +219,255 @@ def scan_archive(archive, wad_path, characters=None, progress=None):
                 s.seconds, s.rate, s.channels, s.size = secs, rate, ch, ln
                 s.group, s.category = classify.classify(s.name)
                 out.append(s)
+    # a variation named NAME_i can collide with an entry whose own name is NAME_i: the variation is
+    # shown as NAME#i and keeps NAME_i as alias (both stay findable, display names stay unique)
+    taken = collections.Counter(s.name.lower() for s in out)
+    for s in out:
+        if taken[s.name.lower()] > 1 and ":" in s.target:
+            s.name, s.aliases = s.aliases[0], (s.name,)
     for s in out:
         s.characters_inferred = "; ".join(classify.infer_characters(s.name))
-        s.character = next((characters[a.lower()] for a in s.names() if a.lower() in characters), "")
+        s.character = character_of(s, characters)
     return out
 
 
+_CAMEL = re.compile(r"(?<=[a-z])(?=[A-Z])")
+
+
+def bank_sounds(sc, characters=None, reserved=()):
+    """[Sound] of the in-memory sounds found by `banks.scan` (prefetch heads belong to WAD sounds). `reserved`:
+    lower-case names taken by other archives (sfx_names)."""
+    characters = characters if characters is not None else load_characters()
+    names = banks.media_names(sc, reserved)
+    every = set().union(*(e.get("langs", set()) for e in sc.media.values())) - {""}
+    out = []
+    for m, e in sc.media.items():
+        if banks.PREFETCH_HEAD in e["kinds"]:
+            continue
+        name, bank = names[m]
+        rate, ch, secs = _hdr(e["head"])
+        ls = bank_langs(sc, m)                  # localized: the languages whose banks hold it
+        s = Sound(banks.ARCHIVE, "%08X" % m, name, ("%s/%08X" % (bank, m),), bank=bank,
+                  language="" if set(ls) == every else ";".join(ls), seconds=secs, rate=rate, channels=ch,
+                  size=min(l.size for l in sc.locations if l.media == m) if len(e["sha1"]) > 1 else e["size"],
+                  sha1=next(iter(e["sha1"])) if len(e["sha1"]) == 1 else "")
+        s.group, s.category = classify.classify(name, bank=bank)
+        s.characters_inferred = "; ".join(classify.infer_characters(_CAMEL.sub("_", name + "_" + bank)))
+        s.character = character_of(s, characters)
+        out.append(s)
+    out.sort(key=lambda s: (s.bank.lower(), s.name.lower()))
+    return out
+
+
+def build_lines(game, sc):
+    """The game's speech lines across languages (ackit/lines.py)."""
+    return lines_mod.build(game, sc, _walk_entry)
+
+
+def scan_banks(game, progress=None, characters=None, visit=None):
+    """Full scan of the game's banks -> ([Sound], [Location] with WAD keys on prefetch heads, Scan, Lines).
+    Localized in-memory speech lines carry their line identity and the name shared by every language."""
+    sc = banks.scan(game.cooked, progress, visit=visit)
+    snd, locs, lines = derive_banks(game, sc, characters)
+    return snd, locs, sc, lines
+
+
+def derive_banks(game, sc, characters=None):
+    """(bank sounds, locations, Lines) from a Scan: the lines, the prefetch heads' lines, the in-memory sounds
+    with their line identities and names."""
+    lines = build_lines(game, sc)
+    mark_heads(sc, whole_recordings(sc, lines))
+    locs = [l._replace(sound=lines.head_key(l)) if l.kind == banks.PREFETCH_HEAD else l for l in sc.locations]
+    snd = bank_sounds(sc, characters, sfx_names(game))
+    lines.names = line_names(game, sc, lines, {s.key: s.name for s in snd})
+    apply_lines(snd, lines, lines.names)
+    return snd, locs, lines
+
+
+def whole_recordings(sc, lines):
+    """Media stored in memory that are WAD recordings of their bank's language (the same media ID, identical
+    bytes: 49 in the game, e.g. game-over taunts). Like prefetch heads, these copies belong to the WAD line
+    and are written with it; they are not sounds of their own."""
+    keys = collections.defaultdict(list)
+    for l in sc.locations:
+        if l.kind == banks.MEMORY:
+            k = lines.head_key(l)
+            keys[l.media].append(bool(k) and not k.startswith(banks.ARCHIVE + ":"))
+    return {m for m, wad in keys.items() if all(wad)}
+
+
+def mark_heads(sc, media):
+    """Treat the bank copies of `media` as heads of WAD recordings (see whole_recordings)."""
+    if not media:
+        return
+    for m in media:
+        if m in sc.media:
+            sc.media[m]["kinds"] = {banks.PREFETCH_HEAD}
+            sc.prefetch.add(m)
+    sc.locations = [l._replace(kind=banks.PREFETCH_HEAD) if l.media in media else l for l in sc.locations]
+
+
+def sfx_names(game):
+    """Lower-case names and aliases of the SFX WAD's sounds: in-memory sounds of the same bank must not take
+    them (`<Bank>_<n>` is the position name of the bank's streamed sounds too)."""
+    if "SFX" not in game.archives:
+        return set()
+    return {n.lower() for s in scan_archive("SFX", game.wad("SFX"), characters={}) for n in s.names()}
+
+
+def line_names(game, sc, lines, bank_names=None):
+    """{line id: name} (lines.line_names). Lines no event names keep their English recording's name: the
+    position-based name in the INT WAD (`NAME_i`, or `NAME#i` where that is taken) or the bank-numbered name."""
+    pos = {}
+    for L in {k.split(":")[0] for per in lines.members.values() for k in per.values()} - {banks.ARCHIVE}:
+        for s in scan_archive(L, game.wad(L), characters={}):
+            pos[s.key] = s
+    bank_names = bank_names if bank_names is not None else {}
+
+    def fallback(k):
+        return pos[k].name if k in pos else bank_names.get(k, k)
+
+    out = lines_mod.line_names(sc, lines, fallback)
+    taken = collections.Counter(n.lower() for n in out.values())
+    for lid, n in out.items():           # a fallback name equal to an event-based one -> NAME#i of the INT WEM
+        if taken[n.lower()] > 1:
+            k = lines.members[lid].get("INT") or next(iter(lines.members[lid].values()))
+            alt = [a for a in (pos[k].names() if k in pos else ()) if "#" in a]
+            if alt and alt[0] != n:
+                out[lid] = alt[0]
+    return out
+
+
+def apply_lines(sounds, lines, names):
+    """Give every speech sound of one archive (a WAD language or BANK) its line identity and the line's name,
+    the same in every language. A WAD sound keeps its position-based names as aliases (NAME#i: position in
+    this language's WAD; single-WEM entries: the entry name). Names stay unique in the archive (and, for
+    BANK, per language together with the language-neutral sounds): a clash is resolved by the sound that
+    is not the line's own name falling back to its position name / media-ID name."""
+    for s in sounds:
+        lid = lines.line_of.get(s.key)
+        if not lid:
+            continue
+        s.line = lid
+        if s.key in lines.copy_of:     # a second place of a recording: keeps its position name, written with it
+            continue
+        name = names[lid]
+        if s.archive != banks.ARCHIVE:
+            if ":" in s.target:
+                s.aliases = tuple(a for a in s.names() if "#" in a)
+            else:
+                s.aliases = (s.name,) + tuple(a for a in s.aliases if a != s.name)
+            if name in s.aliases:
+                s.aliases = tuple(a for a in s.aliases if a != name)
+        s.name = name
+    neutral, by_lang = set(), collections.defaultdict(set)     # lower-case names taken
+
+    def clash(s, n):
+        if s.archive != banks.ARCHIVE or not s.languages:
+            return n in neutral or any(n in v for v in by_lang.values())
+        return n in neutral or any(n in by_lang[L] for L in s.languages)
+
+    # a line's own recordings first: they keep the line's name in a clash (copies and others give way)
+    for s in sorted(sounds, key=lambda s: (not s.line or s.key in lines.copy_of, s.key)):
+        if clash(s, s.name.lower()):
+            if s.archive == banks.ARCHIVE:
+                s.name = "%s_%s" % (s.name, s.target)
+            else:
+                alt = [a for a in s.aliases if "#" in a and a != s.name]
+                s.aliases, s.name = (s.name,), alt[0] if alt else "%s@%s" % (s.name, s.target)
+        if s.archive == banks.ARCHIVE and s.languages:
+            for L in s.languages:
+                by_lang[L].add(s.name.lower())
+        else:
+            neutral.add(s.name.lower())
+    for s in sounds:
+        if s.archive != banks.ARCHIVE:
+            s.group, s.category = classify.classify(s.name)
+            s.characters_inferred = "; ".join(classify.infer_characters(s.name))
+    return sounds
+
+
+def subtitle_links(sc, lines, speech_names):
+    """{speech event name: {language: sound keys}}: event -> actions -> containers -> Sound objects (the same
+    in every language) -> that language's recording of each line (a WAD line or a localized in-memory one).
+    An event that plays several lines (two versions) links its subtitle to all of them: 'k1;k2', in order."""
+    ev = banks.event_objects(sc)
+    out = {}
+    for n in speech_names:
+        keys = collections.defaultdict(list)
+        for o in ev.get(n, ()):
+            for L, k in lines.obj_keys.get(o, {}).items():
+                if k not in keys[L]:
+                    keys[L].append(k)
+        if keys:
+            out[n] = {L: ";".join(ks) for L, ks in keys.items()}
+    return out
+
+
+def scan_text(game, progress=None, characters=None):
+    """One pass over every package: (bank sounds, bank locations, Scan, {lang: [SubLine]}, [SubLoc], issues,
+    Lines)."""
+    col = subtitles.Collector()
+    sc = banks.scan(game.cooked, progress, visit=col)
+    names = {n for n in sc.names if n.lower().startswith("speech_")}
+    names |= {n for n, _ in banks.media_names(sc).values() if n.lower().startswith("speech_")}
+    lookup = [n for n in col.dialogues if n.startswith("#lookup:")]
+    sc.subtitle_objects = {"dialogue_objects": sum(len(v) for n, v in col.dialogues.items() if n not in lookup),
+                           "dialogue_lines": len(col.dialogues) - len(lookup),
+                           "scripted_objects": sum(len(col.dialogues[n]) for n in lookup)}
+    col.resolve(names)
+    # the subtitle objects name their speech events too: an event ID is the FNV-1 hash of its name, so these
+    # names find events no package name table lists (e.g. Vicki Vale's radio reports)
+    sc.names.update(col.dialogues)
+    snd, locs, lines = derive_banks(game, sc, characters)
+    links = subtitle_links(sc, lines, list(col.dialogues))
+    texts, sublocs = subtitles.rows(col, links)
+    return snd, locs, sc, texts, sublocs, col.issues, lines
+
+
+def load_subtitles(game, lang, progress=None):
+    """([SubLine], {SUB key: [SubLoc]}) of one language: shipped catalog when the game matches it, else a
+    cached or fresh scan of the packages (several minutes)."""
+    fp = banks.game_fingerprint(game.cooked)
+    shipped = _shipped_index().get("SUB", {})
+    t = os.path.join(catalog_dir(), "SUB_%s.csv" % lang)
+    loc = os.path.join(catalog_dir(), subtitles.LOCATIONS)
+    if shipped.get("fingerprint") == fp and os.path.exists(t) and os.path.exists(loc):
+        return subtitles.read_texts(t, lang), subtitles.read_locations(loc, lang)
+    cache = os.path.join(user_dir("cache"), "scan_SUB_%s_v%d" % (fp, SCAN_VERSION))
+    if not os.path.exists(cache + ".locations.csv.gz"):
+        _, _, _, texts, sublocs, _, _ = scan_text(game, progress)
+        for lg, rows in texts.items():
+            subtitles.write_texts(cache + "_%s.csv" % lg, rows)
+        subtitles.write_locations(cache + ".locations.csv.gz", sublocs)
+    t = cache + "_%s.csv" % lang
+    return (subtitles.read_texts(t, lang) if os.path.exists(t) else []), \
+        subtitles.read_locations(cache + ".locations.csv.gz", lang)
+
+
+def load_banks(game, progress=None):
+    """([Sound], {media: [Location]}) of the in-memory sounds: shipped catalog when the game's bank set
+    matches it, else a cached or fresh scan (several minutes)."""
+    fp = banks.game_fingerprint(game.cooked)
+    shipped = _shipped_index().get(banks.ARCHIVE, {})
+    csv_path = os.path.join(catalog_dir(), "%s.csv" % banks.ARCHIVE)
+    loc_path = os.path.join(catalog_dir(), banks.LOCATIONS)
+    if shipped.get("fingerprint") == fp and os.path.exists(csv_path) and os.path.exists(loc_path):
+        return read_csv(csv_path), banks.read_locations(loc_path)
+    cache = os.path.join(user_dir("cache"), "scan_%s_%s_v%d" % (banks.ARCHIVE, fp, SCAN_VERSION))
+    if not (os.path.exists(cache + ".csv") and os.path.exists(cache + ".locations.csv.gz")):
+        sounds, locs, _, lines = scan_banks(game, progress)
+        for s in sounds:
+            s.sha1 = ""              # unknown build: no trusted original checksums
+        # the speech WADs of this build with the lines found now (load_archive reads these caches)
+        for a in game.languages:
+            snd = apply_lines(scan_archive(a, game.wad(a), progress=progress), lines, lines.names)
+            write_csv(_wad_cache(game, a), snd)
+        banks.write_locations(cache + ".locations.csv.gz", locs)
+        write_csv(cache + ".csv", sounds)
+    return read_csv(cache + ".csv"), banks.read_locations(cache + ".locations.csv.gz")
+
+
 # --- catalog -------------------------------------------------------------------------------
 def _shipped_index():
     p = os.path.join(catalog_dir(), "index.json")
@@ -200,6 +477,18 @@ def _shipped_index():
         return {}
 
 
+def is_shipped(game, archive):
+    """True if the game's archive matches the shipped catalog (not a scan of another build)."""
+    shipped = _shipped_index().get(archive, {})
+    if archive == banks.ARCHIVE:
+        return shipped.get("fingerprint") == banks.game_fingerprint(game.cooked)
+    return shipped.get("fingerprint") == wad_fingerprint(game.wad(archive))
+
+
+def _wad_cache(game, archive):
+    return os.path.join(user_dir("cache"), "scan_%s_%s_v%d.csv" % (archive, wad_fingerprint(game.wad(archive)), SCAN_VERSION))
+
+
 def load_archive(game, archive, progress=None):
     """[Sound] for one archive of a game (shipped CSV, user-cache scan, or fresh scan)."""
     path = game.wad(archive)
@@ -208,7 +497,7 @@ def load_archive(game, archive, progress=None):
     csv_path = os.path.join(catalog_dir(), "%s.csv" % archive)
     if shipped.get("fingerprint") == fp and os.path.exists(csv_path):
         return read_csv(csv_path)
-    cache = os.path.join(user_dir("cache"), "scan_%s_%s_v%d.csv" % (archive, fp, SCAN_VERSION))
+    cache = _wad_cache(game, archive)
     if os.path.exists(cache):
         return read_csv(cache)
     sounds = scan_archive(archive, path, progress=progress)
@@ -217,16 +506,41 @@ def load_archive(game, archive, progress=None):
     return sounds
 
 
+def available_archives(game):
+    """WAD archives of the game plus BANK when it has soundbanks."""
+    return list(game.archives) + ([banks.ARCHIVE] if banks.has_banks(game.cooked) else [])
+
+
 class Catalog:
-    """All sounds of the selected archives (default: the game's speech language + SFX)."""
+    """All sounds of the selected archives (default: the game's speech language, SFX and BANK)."""
 
     def __init__(self, game, archives=None, progress=None):
         self.game = game
-        self.archives = archives or [a for a in (game.language(), "SFX") if a in game.archives]
+        avail = available_archives(game)
+        self.archives = archives or [a for a in (game.language(), "SFX", banks.ARCHIVE) if a in avail]
         self.sounds = []
+        self.locations = {}          # BANK media id -> [Location]
+        self.heads = {}              # WAD sound key -> [Location] of its prefetch heads
+        bank_sounds_ = []
+        if banks.ARCHIVE in avail:
+            # always loaded: speech lines may have prefetch heads that must be written with them
+            bank_sounds_, self.locations = load_banks(game, progress)
+            for ls in self.locations.values():
+                for l in ls:
+                    if l.kind == banks.PREFETCH_HEAD and l.sound:
+                        self.heads.setdefault(l.sound, []).append(l)
+        # localized in-memory speech: only the languages loaded here (all of them if none is)
+        langs = [a for a in self.archives if a in game.languages] or None
+        bank_sounds_ = [s for s in bank_sounds_ if not langs or any(s.in_language(L) for L in langs)]
         for a in self.archives:
-            self.sounds += load_archive(game, a, progress)
+            self.sounds += bank_sounds_ if a == banks.ARCHIVE else load_archive(game, a, progress)
         self.by_key = {s.key: s for s in self.sounds}
+        # copies: one recording stored in two places of a WAD (same line, same archive); written together
+        same = collections.defaultdict(list)
+        for s in self.sounds:
+            if s.line and s.archive != banks.ARCHIVE:
+                same[(s.archive, s.line)].append(s)
+        self.copies = {s.key: [c for c in ss if c is not s] for ss in same.values() if len(ss) > 1 for s in ss}
         self._by_name = {}
         for s in self.sounds:
             for n in s.names():
@@ -242,6 +556,11 @@ class Catalog:
         key = os.path.splitext(os.path.basename(name))[0].lower() if re.search(r"\.\w{2,4}$", name) else name.lower()
         hits = self._by_name.get(key, [])
         uniq = list({id(s): s for s in hits}.values())
+        if len(uniq) > 1:
+            # a name wins over another sound's alias: a line named after its INT position (NAME#i) can equal
+            # the position alias of another recording in another language's WAD
+            own = [s for s in uniq if s.name.lower() == key]
+            uniq = own if len(own) == 1 else uniq
         if len(uniq) == 1:
             return uniq[0]
         if len(uniq) > 1:

ackit/classify.py

@@ -28,17 +28,26 @@ _RULES = [
     (r"^SPEECH_CHAR_", "Speech", "Character lines"),
     (r"^SPEECH_(CH|SEQUENCE|STEEL|MUSEUM|CHURCH|COURT|UNDER|GCPD|ZP1)_", "Speech", "Story missions"),
     (r"^SPEECH_(GAME|MAP|AUGMENTED|FORENSICS|TEMP)_", "Speech", "Gameplay & hints"),
-    (r"^NPC_", "Ambience", "Crowds"),
+    (r"^NPC_.*CROWD", "Ambience", "Crowds"),
     (r"^SPEECH_", "Speech", "Other speech"),
     (r"^MUSIC_PDLC", "Music", "Music (DLC)"),
-    (r"^MUSIC", "Music", "Music"),
+    (r"^(MUSIC|MUS_)", "Music", "Music"),
     (r"^(AMB|PSA)_", "Ambience", "Ambience"),
+    (r"^PSO_", "Ambience", "Ambient emitters"),
     (r"^WND_", "Ambience", "Wind"),
+    (r"^WTR_", "Ambience", "Weather"),
     (r"^P?VOL_", "Ambience", "Room tone"),
-    (r"^SEQ_", "SFX", "Scripted sequences"),
+    (r"FOOTSTEP", "SFX", "Footsteps"),
+    (r"^(CHR|NPC)_.*(COMBAT|BONEBREAK|BODYFALL)", "SFX", "Combat"),
+    (r"^(SEQ|MTN)_", "SFX", "Scripted sequences"),
     (r"^OBJ_", "SFX", "Objects"),
+    (r"^(PX|PHY)_", "SFX", "Physics"),
     (r"^WPN_", "SFX", "Weapons"),
-    (r"^CHR_", "SFX", "Character foley"),
+    (r"^DVC_", "SFX", "Gadgets"),
+    (r"^(GUI|SBJ)_", "SFX", "Interface"),
+    (r"^VFX_", "SFX", "Visual effects"),
+    (r"^NPC_", "SFX", "Enemies & NPCs"),
+    (r"^(CHR|GEN)_", "SFX", "Character foley"),
     (r"^GLOBAL", "SFX", "Global"),
 ]
 _RULES = [(re.compile(p), g, c) for p, g, c in _RULES]
@@ -46,9 +55,10 @@ _RULES = [(re.compile(p), g, c) for p, g, c in _RULES]
 GROUPS = ("Speech", "Music", "Ambience", "SFX", "Unknown")
 
 
-def classify(name: str, pack: str = None):
-    """-> (group, category). `pack` = name of the stream pack a media belongs to, if any."""
-    n = name.upper()
+def classify(name: str, pack: str = None, bank: str = None):
+    """-> (group, category). `pack` = name of the stream pack a media belongs to, if any; `bank` = name of
+    the soundbank holding an in-memory sound (its prefix is more reliable than event names)."""
+    n = (bank or name).upper()
     if pack:
         p = pack.upper()
         if p.startswith("MUSIC"):

ackit/cli.py

@@ -19,8 +19,7 @@ from . import audio, settings
 from .game import Game, GameError, find_installs, default_game_dir
 
 
-def _workspace(a, archives=None):
-    from .workspace import Workspace
+def _game(a):
     st = settings.load()
     gd = a.game or default_game_dir(st)
     if not gd:
@@ -29,9 +28,15 @@ def _workspace(a, archives=None):
     if a.game:
         st["game_dir"] = os.path.abspath(a.game)
         settings.save(st)
-    game = Game(gd)
+    return Game(gd)
+
+
+def _workspace(a, archives=None):
+    from .workspace import Workspace
+    st = settings.load()
+    game = _game(a)
     if archives is None and getattr(a, "lang", None):
-        archives = [a.lang, "SFX"]
+        archives = [a.lang, "SFX", "BANK"]
     return Workspace(game, a.store or st.get("store_root"), archives,
                      full_backup=not getattr(a, "no_full_backup", False) and st.get("full_backup", True),
                      progress=lambda n, t: None)
@@ -166,37 +171,82 @@ def cmd_revert(a):
     ws = _workspace(a)
     if a.all:
         done = ws.revert_all(progress=print)
-    else:
-        done = ws.revert([ws.sound(n) for n in a.names], progress=print)
+        texts = ws.texts.revert_all(progress=print)
+        print("reverted %d sound(s) and %d text(s)" % (len(done), len(texts)))
+        return
+    done = ws.revert([ws.sound(n) for n in a.names], progress=print)
     print("reverted %d sound(s)" % len(done))
 
 
 def cmd_status(a):
     ws = _workspace(a)
     keys = ws.modified_keys()
-    if not keys:
-        print("no sounds modified; game audio is original")
+    texts = ws.texts.modified()
+    if not keys and not texts:
+        print("no sounds or texts modified; game files are original")
         return
     for k in keys:
         s = ws.catalog.by_key.get(k)
         st = ws.state(s) if s else "(other language)"
         print("%-50s %s" % (s.name if s else k, st))
+    for k in texts:
+        print("%-50s %s" % (k, ws.texts.state(k)))
 
 
 def cmd_export_mod(a):
     from .modpack import export_mod
     ws = _workspace(a)
     keys = [ws.sound(n).key for n in a.names] if a.names else ws.modified_keys()
+    texts = [] if a.names else ws.texts.modified()
     n = export_mod(ws, a.out, keys, a.title or os.path.splitext(os.path.basename(a.out))[0], a.author or "",
-                   a.description or "")
-    print("wrote %s (%d sounds)" % (a.out, n))
+                   a.description or "", texts=texts)
+    print("wrote %s (%d sounds, %d texts)" % (a.out, len(keys), n - len(keys)))
 
 
 def cmd_import_mod(a):
     from .modpack import import_mod
     ws = _workspace(a)
-    for im in import_mod(ws, a.file, force=a.force, progress=print):
+    items, texts = import_mod(ws, a.file, force=a.force, progress=print)
+    for im in items:
         print("%-8s %-46s %s" % (im.how, im.item.get("name"), im.note))
+    for k, _, problem in texts:
+        print("%-8s %-46s %s" % ("text", k, problem or "applied"))
+
+
+def cmd_text(a):
+    """Subtitles and menu text: find, show, set, revert."""
+    ws = _workspace(a)
+    tx = ws.texts
+    lang = (a.lang or ws.game.language()).upper()
+    if a.action == "find":
+        words = [w.lower() for w in a.args]
+        rows = []
+        if a.kind in ("all", "subs"):
+            rows += [(subtitles_key(s), s.text) for s in tx.subtitles(lang)]
+        if a.kind in ("all", "menu"):
+            rows += [("MENU:" + e.key, e.value) for e in tx.menu(lang)]
+        hits = [(k, v) for k, v in rows if all(w in (k + " " + v).lower() for w in words)]
+        for k, v in hits[:a.limit]:
+            print("%s\n    %s" % (k, v))
+        print("%d match(es)%s" % (len(hits), "" if len(hits) <= a.limit else " (first %d shown)" % a.limit))
+    elif a.action == "show":
+        for k in a.args:
+            print("%s\n    now: %s\n    original: %s\n    state: %s%s" % (
+                k, tx.current(k), tx.original(k), tx.state(k),
+                "" if tx.limit(k) is None else "\n    room: %d characters" % tx.limit(k)))
+    elif a.action == "set":
+        if len(a.args) != 2:
+            raise GameError("usage: ackit text set KEY \"new text\"")
+        tx.apply({a.args[0]: a.args[1]}, force=a.force, progress=print)
+        print("set %s" % a.args[0])
+    elif a.action == "revert":
+        done = tx.revert_all(progress=print) if a.all else tx.revert(a.args, progress=print)
+        print("reverted %d text(s)" % len(done))
+
+
+def subtitles_key(s):
+    from .subtitles import key
+    return key(s.lang, s.speech, s.line)
 
 
 # --- advanced ---------------------------------------------------------------------------------
@@ -242,13 +292,33 @@ def cmd_game(a):
 
 
 def cmd_verify(a):
+    if a.catalog:
+        from .verify import catalogcheck
+        archives = [x.upper() for x in a.archive] or ([a.lang] if a.lang else None)
+        rep = catalogcheck.run(_game(a), archives, quick=a.quick)
+        for (arch, kind), n in sorted(rep.counts().items()):
+            print("%-14s %-10s %d" % (arch, kind, n))
+        for arch, kind, key, detail in rep.issues[:a.show]:
+            print("  %s %s %s %s" % (arch, kind, key, detail))
+        print("checked: %s" % ", ".join("%s %d" % kv for kv in sorted(rep.checked.items())))
+        for f in rep.facts:
+            print("fact: " + f)
+        print("catalog OK" if rep.ok else "%d issue(s)" % len(rep.issues))
+        sys.exit(0 if rep.ok else 1)
     from .verify import selfcheck
     ws = _workspace(a)
     if not a.ingame:
         ok = selfcheck.run(ws, log=print)
         sys.exit(0 if ok else 1)
     from .verify import ingame
-    if a.baseline:
+    if a.lang and a.lang != ws.game.language():
+        raise GameError("the game plays its own language (%s); set it in Steam (game properties > Language) to "
+                        "test %s in the game" % (ws.game.language(), a.lang))
+    if a.banks:
+        rep = ingame.run_banks(ws, a.sound or None)
+    elif a.texts:
+        rep = ingame.run_texts(ws)
+    elif a.baseline:
         rep = ingame.run(ws, a.target, mode="baseline", seconds=a.seconds)
     elif a.installed:
         rep = ingame.run(ws, a.target, mode="installed", seconds=a.seconds)
@@ -304,16 +374,31 @@ def build_parser():
     s = add("revert", cmd_revert, "put original sounds back", aliases=["restore"])
     s.add_argument("names", nargs="*"); s.add_argument("--all", action="store_true")
     add("status", cmd_status, "list the sounds you modified")
+    s = add("text", cmd_text, "subtitles and menu text: find WORDS | show KEY | set KEY TEXT | revert KEY/--all")
+    s.add_argument("action", choices=["find", "show", "set", "revert"])
+    s.add_argument("args", nargs="*")
+    s.add_argument("--kind", choices=["all", "subs", "menu"], default="all", help="with find: what to search")
+    s.add_argument("--limit", type=int, default=40); s.add_argument("--all", action="store_true")
+    s.add_argument("--force", action="store_true", help="overwrite texts changed outside this tool")
     s = add("export-mod", cmd_export_mod, "save your changes as a shareable .acmod package")
     s.add_argument("out"); s.add_argument("names", nargs="*", help="default: everything you modified")
     s.add_argument("--title"); s.add_argument("--author"); s.add_argument("--description")
     s = add("import-mod", cmd_import_mod, "install a .acmod package", (enc,))
     s.add_argument("file")
     add("game", cmd_game, "show the detected game folder, archives, language and store")
-    s = add("verify", cmd_verify, "self-check (offline), or test in the real game with --ingame (Windows)")
+    s = add("verify", cmd_verify, "self-check (offline), catalog cross-check (--catalog), or test in the real "
+            "game with --ingame (Windows)")
+    s.add_argument("--catalog", action="store_true", help="cross-check the catalog against the game files")
+    s.add_argument("--archive", action="append", default=[], help="with --catalog: only this archive (repeatable; --lang X: only X)")
+    s.add_argument("--quick", action="store_true", help="with --catalog: loose banks + a sample of packages")
+    s.add_argument("--show", type=int, default=30, help="with --catalog: issues to list (default 30)")
     s.add_argument("--ingame", action="store_true"); s.add_argument("--target", default="SFX:EDF036D6/393BC53A")
     s.add_argument("--audio"); s.add_argument("--baseline", action="store_true")
     s.add_argument("--installed", action="store_true"); s.add_argument("--seconds", type=float, default=40)
+    s.add_argument("--banks", action="store_true",
+                   help="with --ingame: test in-memory sounds (default: the front-end banks, or --sound NAME ...)")
+    s.add_argument("--sound", action="append", default=[], help="with --banks: a sound to test (repeatable)")
+    s.add_argument("--texts", action="store_true", help="with --ingame: test menu text and subtitle edits")
     # advanced
     s = add("encode", cmd_encode, "[advanced] audio -> .wem file", (enc,))
     s.add_argument("audio"); s.add_argument("out"); s.add_argument("--like", metavar="NAME")

ackit/codebooks.py

@@ -20,17 +20,6 @@ def load_library(path: str = DEFAULT_LIB) -> list:
     return [b[offs[i]:offs[i + 1]] for i in range(n - 1)]
 
 
-def save_library(path: str, books: list):
-    out = bytearray()
-    offs = []
-    for bk in books:
-        offs.append(len(out))
-        out += bk
-    offs.append(len(out))  # table start == end of last book
-    out += struct.pack("<%dI" % len(offs), *offs)
-    open(path, "wb").write(bytes(out))
-
-
 def wwise_book_to_vorbis(blob: bytes, w: BitWriter):
     """Expand a compact Wwise codebook into a standard Vorbis codebook (ww2ogg rebuild)."""
     r = BitReader(blob)

ackit/game.py

@@ -11,7 +11,6 @@ import sys
 
 APPID = 200260
 FOLDER_NAMES = ("Batman Arkham City GOTY", "Batman Arkham City")
-LANGS = ("INT", "ITA", "FRA", "DEU", "ESN", "ESM", "POL", "RUS", "JPN", "KOR", "POR")
 
 
 class GameError(Exception):

ackit/lines.py

@@ -0,0 +1,224 @@
+"""Speech lines across languages, derived only from the game's own data (docs/formats/banks.md, "Lines").
+
+A voice recording exists once per language. The game ties them together itself:
+  - every recording of a speech WAD has exactly one row in its language's BKPK table (media ID, offset, size);
+    the table's language is the one its localized AkBank export declares (banks.py: localized_exports);
+  - every recording, streamed, prefetched or in memory, is the source of a Wwise Sound object. Sound objects
+    (and the events, actions and containers above them) have the same IDs in every language; only the media
+    differ (a few media IDs are shared by several languages, each at its own position in each WAD).
+So a *line* is the set of recordings whose media are the sources of the same Sound objects: one recording per
+language. Its identity (`Sound.line`) is the smallest Sound object ID (8 hex digits), its name is chosen once
+for all languages (`line_names`). A media stored in two entries of one WAD is one recording with a copy
+(`copy_of`), written together. Nothing here depends on file positions or on names.
+"""
+import collections
+from dataclasses import dataclass, field
+
+from . import banks
+from .wad import Wad
+
+
+@dataclass
+class Lines:
+    langs: list                                        # voice languages (speech WADs)
+    owners: dict = field(default_factory=dict)         # lang -> {media id: sound key} (WAD line or BANK sound)
+    line_of: dict = field(default_factory=dict)        # sound key -> line id
+    members: dict = field(default_factory=dict)        # line id -> {lang: sound key}
+    objects: dict = field(default_factory=dict)        # line id -> sorted Sound object ids
+    obj_keys: dict = field(default_factory=dict)       # Sound object id -> {lang: sound key}
+    names: dict = field(default_factory=dict)          # line id -> name (catalog.line_names)
+    unnamed: set = field(default_factory=set)          # line ids no named event plays (named after a recording)
+    copy_of: dict = field(default_factory=dict)        # WAD key -> key of the same media stored in another entry
+    issues: list = field(default_factory=list)         # (kind, detail): broken one-to-one relations
+    stats: dict = field(default_factory=dict)
+    _keys: dict = field(default_factory=dict, repr=False)  # lang -> set of sound keys
+
+    def key_langs(self, key):
+        return [L for L in self.langs if key in self._keys.get(L, ())]
+
+    def head_key(self, loc):
+        """Speech line (sound key) of a prefetch-head copy: the line of the copy's bank language that has
+        this media. A head in a language-neutral bank belongs to the one line that uses the media."""
+        if loc.lang:
+            return self.owners.get(loc.lang, {}).get(loc.media, "")
+        ks = {d[loc.media] for d in self.owners.values() if loc.media in d}
+        return ks.pop() if len(ks) == 1 else ""
+
+
+def wad_owners(game, sc, langs, walk):
+    """({lang: {media id: WAD sound key}}, {copy key: key}, issues, WEM counts) through each language's own BKPK
+    tables. Issues: a WEM without a table row, a row without a WEM. A media stored in two entries of one
+    language's WAD (a few lines, byte-identical) has one key; the other place is a copy written with it.
+    `walk(f, offset, size)` -> [(offset, length, header)] (catalog._walk_entry)."""
+    owners, issues, wems = {}, [], collections.Counter()
+    places = {}
+    for L in langs:
+        owners[L] = {}
+        places[L] = collections.defaultdict(list)
+        wad = Wad(game.wad(L))
+        with open(game.wad(L), "rb") as f:
+            for e in wad.entries:
+                tables = sc.packs.get(e.hash, {})
+                table = tables.get(L) if L in tables else tables.get("")
+                ws = walk(f, e.offset, e.size)
+                wems[L] += len(ws)
+                if table is None:
+                    issues.append(("no table", "%s:%08X: %d WEMs, no BKPK table" % (L, e.hash, len(ws))))
+                    continue
+                at = {(o, ln): i for i, (o, ln, _) in enumerate(ws)}
+                used = set()
+                for m, v in table.items():
+                    if v not in at:
+                        issues.append(("row without WEM", "%s:%08X: media %08X" % (L, e.hash, m)))
+                        continue
+                    used.add(at[v])
+                    places[L][m].append("%s:%08X" % (L, e.hash) + (":%d" % at[v] if len(ws) > 1 else ""))
+                for i in sorted(set(range(len(ws))) - used):
+                    issues.append(("WEM without row", "%s:%08X:%d" % (L, e.hash, i)))
+    copy_of = {}
+    for L in langs:
+        for m, ks in places[L].items():
+            ks = sorted(ks)
+            owners[L][m] = ks[0]
+            for k in ks[1:]:
+                copy_of[k] = ks[0]
+    return owners, copy_of, issues, dict(wems)
+
+
+def bank_langs(sc, m):
+    """Voice languages of a localized in-memory media: the languages of the banks holding it ('' = neutral)."""
+    ls = sc.media[m].get("langs", set())
+    return () if "" in ls else tuple(sorted(ls))
+
+
+def build(game, sc, walk):
+    """Lines of every voice language of the game (see module doc)."""
+    in_banks = sorted(set().union(*(e.get("langs", set()) for e in sc.media.values())) - {""} - set(game.languages))
+    langs = list(game.languages) + in_banks            # voice languages: speech WADs + localized banks
+    lines = Lines(langs)
+    owners, copy_of, issues, wems = wad_owners(game, sc, game.languages, walk)
+    for L in in_banks:
+        owners[L] = {}
+    lines.issues += issues
+    lines.copy_of = copy_of
+    for m, e in sc.media.items():                      # localized in-memory recordings (not prefetch heads)
+        if banks.PREFETCH_HEAD in e["kinds"]:
+            continue
+        for L in bank_langs(sc, m):
+            if L in owners:
+                owners[L].setdefault(m, "%s:%08X" % (banks.ARCHIVE, m))
+    lines.owners = owners
+    lines._keys = {L: set(d.values()) for L, d in owners.items()}
+
+    # Sound object -> its recording in each language (exactly one, or the relation is broken)
+    for o, ms in sc.sound_media.items():
+        per = {}
+        for L in langs:
+            ks = {owners[L][m] for m in ms if m in owners[L]}
+            if len(ks) > 1:
+                lines.issues.append(("object", "Sound %08X has %d recordings in %s: %s" % (o, len(ks), L, sorted(ks))))
+            elif ks:
+                per[L] = ks.pop()
+        if per:
+            lines.obj_keys[o] = per
+
+    # lines = connected recordings (union-find over the objects' recordings)
+    parent = {}
+
+    def find(x):
+        while parent.get(x, x) != x:
+            parent[x] = parent.get(parent[x], parent[x])
+            x = parent[x]
+        return x
+
+    for o, per in lines.obj_keys.items():
+        ks = sorted(set(per.values()))
+        for k in ks[1:]:
+            a, b = find(ks[0]), find(k)
+            if a != b:
+                parent[max(a, b)] = min(a, b)
+    groups = collections.defaultdict(set)
+    objs = collections.defaultdict(set)
+    for o, per in lines.obj_keys.items():
+        for k in per.values():
+            groups[find(k)].add(k)
+            objs[find(k)].add(o)
+    for root, ks in groups.items():
+        per = {}
+        ok = True
+        for k in ks:
+            for L in lines.key_langs(k):
+                if L in per and per[L] != k:
+                    ok = False
+                per.setdefault(L, k)
+        if not ok:
+            lines.issues.append(("line", "recordings %s share Sound objects but several are in one language" % sorted(ks)))
+            continue
+        lid = "%08X" % min(objs[root])
+        lines.members[lid] = per
+        lines.objects[lid] = sorted(objs[root])
+        for k in ks:
+            lines.line_of[k] = lid
+    for k, primary in copy_of.items():                 # a second place of the same recording: the same line
+        if primary in lines.line_of:
+            lines.line_of[k] = lines.line_of[primary]
+    recordings = set().union(*lines._keys.values()) if lines._keys else set()
+    all_keys = recordings | set(copy_of)               # + the second places of recordings stored twice
+    full = sum(1 for per in lines.members.values() if len(per) == len(langs))
+    lines.stats = {"languages": langs, "wad_wems": wems, "recordings": len(recordings),
+                   "in_a_line": len(lines.line_of), "lines": len(lines.members), "lines_in_every_language": full,
+                   "copies": sorted(copy_of),
+                   "lines_missing_a_language": {lid: sorted(set(langs) - set(per))
+                                                for lid, per in lines.members.items() if len(per) < len(langs)},
+                   "without_line": sorted(all_keys - set(lines.line_of))}
+    return lines
+
+
+def line_names(sc, lines, fallback):
+    """{line id: name}, the same in every language. A line is named after the game event that plays it: for
+    WAD lines the most specific `Speech_` event (fewest lines), for in-memory lines the most specific event
+    (`Play_` first), as the in-memory sounds always were. When an event plays several lines they are numbered
+    in the event's own order (the same in every language): NAME, NAME#2, ... (WAD) or NAME_1, NAME_2, ... (in
+    memory). A line no named event plays keeps the name its English recording has (`fallback(key)`: the
+    position-based name in the INT WAD, or the bank-numbered name), else the first voice language's."""
+    obj_line = {o: lid for lid, os_ in lines.objects.items() for o in os_}
+    is_wad = {lid: any(not k.startswith(banks.ARCHIVE + ":") for k in per.values()) for lid, per in lines.members.items()}
+    best = {}
+    reach = {}
+    for name, objs in banks.event_objects(sc).items():
+        order = []
+        for o in objs:
+            lid = obj_line.get(o)
+            if lid and lid not in order:
+                order.append(lid)
+        if order:
+            reach[name] = order
+    for name, order in reach.items():
+        low = name.lower()
+        for lid in order:
+            if is_wad[lid]:
+                if not low.startswith("speech_"):
+                    continue
+                rank = (len(order), low)
+            else:
+                rank = (0 if low.startswith("play") else 1, len(order), low)
+            if lid not in best or rank < best[lid][0]:
+                best[lid] = (rank, name)
+    named = collections.defaultdict(list)                     # event -> lines it names, in the event's order
+    for name, order in reach.items():
+        for lid in order:
+            if lid in best and best[lid][1] == name:
+                named[name].append(lid)
+    out = {}
+    for name, lids in named.items():
+        for i, lid in enumerate(lids):
+            if is_wad[lid]:
+                out[lid] = name if i == 0 else "%s#%d" % (name, i + 1)
+            else:
+                out[lid] = name if len(lids) == 1 else "%s_%d" % (name, i + 1)
+    lines.unnamed = set(lines.members) - set(out)
+    for lid in lines.unnamed:
+        per = lines.members[lid]
+        k = per.get("INT") or per[next(L for L in lines.langs if L in per)]
+        out[lid] = fallback(k)
+    return out

ackit/loctext.py

@@ -0,0 +1,108 @@
+"""Menu and UI text: the game's localization files BmGame/Localization/<LANG>/<File>.<lang>
+(UE3 .ini format; docs/formats/text.md).
+
+Files are UTF-16LE with a BOM and CRLF line ends (two small ones are plain ASCII). Lines are `[Section]`,
+`Key=Value` (the value may be in double quotes) or comments (`;`). A key can repeat inside a section, so an
+entry is identified by (file, section, key, occurrence). Writing changes only the value part of the edited
+lines; every other byte of the file (BOM, encoding, quotes, line ends, comments) stays as it was.
+"""
+import os
+import re
+from collections import namedtuple
+
+# key: "<LANG>:<File>:<Section>:<Key>" + ("#n" for the n-th repeat, n >= 2)
+Entry = namedtuple("Entry", "key lang file section name occurrence value quoted line")
+
+
+class TextError(ValueError):
+    pass
+
+
+def _decode(raw):
+    if raw[:2] == b"\xff\xfe":
+        return raw[2:].decode("utf-16-le"), "utf-16"
+    if raw[:3] == b"\xef\xbb\xbf":
+        return raw[3:].decode("utf-8"), "utf-8"
+    return raw.decode("cp1252"), "cp1252"
+
+
+def _encode(text, enc):
+    if enc == "utf-16":
+        return b"\xff\xfe" + text.encode("utf-16-le")
+    if enc == "utf-8":
+        return b"\xef\xbb\xbf" + text.encode("utf-8")
+    return text.encode("cp1252")
+
+
+_KV = re.compile(r"^([^=;\[\s][^=]*?)=(.*)$")
+
+
+def parse(raw, lang, fname):
+    """Entries of one file's bytes."""
+    text, _ = _decode(raw)
+    out = []
+    section = ""
+    seen = {}
+    for i, line in enumerate(re.split(r"(?<=\n)", text)):
+        body = line.rstrip("\r\n")
+        s = body.strip()
+        if s.startswith("[") and s.endswith("]"):
+            section = s[1:-1]
+            continue
+        m = _KV.match(body)
+        if not m or s.startswith(";"):
+            continue
+        name, value = m.group(1), m.group(2)
+        quoted = len(value) >= 2 and value[0] == '"' and value[-1] == '"'
+        if quoted:
+            value = value[1:-1]
+        k = (section.lower(), name.lower())
+        seen[k] = seen.get(k, 0) + 1
+        n = seen[k]
+        key = "%s:%s:%s:%s" % (lang, fname, section, name) + ("#%d" % n if n > 1 else "")
+        out.append(Entry(key, lang, fname, section, name, n, value, quoted, i))
+    return out
+
+
+def write(raw, lang, fname, values):
+    """New file bytes with {entry key: new value} applied (keys not in the file raise TextError)."""
+    text, enc = _decode(raw)
+    lines = re.split(r"(?<=\n)", text)
+    by_key = {e.key: e for e in parse(raw, lang, fname)}
+    for key, value in values.items():
+        e = by_key.get(key)
+        if e is None:
+            raise TextError("no text %s in %s" % (key, fname))
+        if "\r" in value or "\n" in value:
+            raise TextError("a menu text cannot contain line breaks (use the game's own markers, e.g. \\n)")
+        line = lines[e.line]
+        end = line[len(line.rstrip("\r\n")):]
+        head = line[:line.index("=") + 1]
+        lines[e.line] = head + ('"%s"' % value if e.quoted else value) + end
+    out = "".join(lines)
+    try:
+        return _encode(out, enc)
+    except UnicodeEncodeError:
+        raise TextError("%s is stored as %s and cannot hold these characters" % (fname, enc)) from None
+
+
+def files(loc_root, lang):
+    """{file name: path} of one language folder (case-insensitive): localization files, without the
+    Version*.int build IDs (not UI text)."""
+    d = next((os.path.join(loc_root, x) for x in os.listdir(loc_root) if x.upper() == lang.upper()), None)
+    if not d:
+        return {}
+    return {f: os.path.join(d, f) for f in sorted(os.listdir(d))
+            if os.path.splitext(f)[1].lower() == "." + lang.lower() and not f.lower().startswith("version")}
+
+
+def languages(loc_root):
+    return sorted(x.upper() for x in os.listdir(loc_root) if os.path.isdir(os.path.join(loc_root, x)))
+
+
+def load(loc_root, lang):
+    """All entries of a language: [Entry]."""
+    out = []
+    for f, p in files(loc_root, lang).items():
+        out += parse(open(p, "rb").read(), lang.upper(), f)
+    return out

ackit/lzo.py

@@ -0,0 +1,142 @@
+"""LZO1X, as used by the game's compressed UE3 packages (docs/formats/upk.md).
+
+`decompress` is a pure-Python LZO1X decoder (same semantics as lzo1x_decompress_safe; the packages
+use 128 KiB blocks, so speed is fine for block-level access). `compress_stored` writes a valid LZO1X
+stream that holds the data as one literal run (no matches): bigger than real compression, but any
+LZO1X decoder (the game's included) reads it back exactly, and nothing else has to change.
+"""
+M2_MAX_OFFSET = 0x0800
+
+
+class LzoError(ValueError):
+    pass
+
+
+def _copy_match(out, dist, n):
+    start = len(out) - dist
+    if start < 0:
+        raise LzoError("match distance beyond output start")
+    if dist >= n:
+        out += out[start:start + n]
+    else:  # overlapping: repeat the last `dist` bytes
+        pat = bytes(out[start:])
+        out += (pat * (n // dist + 1))[:n]
+
+
+def decompress(src, out_len=None):
+    """LZO1X block -> bytes. `out_len` (if given) is checked."""
+    src = bytes(src)
+    n = len(src)
+    out = bytearray()
+    ip = 0
+
+    def ext(t, base):
+        nonlocal ip
+        while src[ip] == 0:
+            t += 255
+            ip += 1
+        t += base + src[ip]
+        ip += 1
+        return t
+
+    try:
+        state = "loop"
+        if src[0] > 17:
+            t = src[0] - 17
+            ip = 1
+            if t < 4:
+                state = "match_next"
+            else:
+                out += src[ip:ip + t]
+                ip += t
+                state = "first_literal_run"
+        while True:
+            if state == "loop":
+                t = src[ip]
+                ip += 1
+                if t >= 16:
+                    state = "match"
+                    continue
+                if t == 0:
+                    t = ext(0, 15)
+                out += src[ip:ip + t + 3]
+                ip += t + 3
+                state = "first_literal_run"
+                continue
+            if state == "first_literal_run":
+                t = src[ip]
+                ip += 1
+                if t >= 16:
+                    state = "match"
+                    continue
+                dist = 1 + M2_MAX_OFFSET + (t >> 2) + (src[ip] << 2)
+                ip += 1
+                _copy_match(out, dist, 3)
+                state = "match_done"
+                continue
+            if state == "match_next":
+                out += src[ip:ip + t]
+                ip += t
+                t = src[ip]
+                ip += 1
+                state = "match"
+                continue
+            if state == "match":
+                if t >= 64:                       # M2
+                    dist = 1 + ((t >> 2) & 7) + (src[ip] << 3)
+                    ip += 1
+                    _copy_match(out, dist, (t >> 5) + 1)
+                elif t >= 32:                     # M3
+                    t &= 31
+                    if t == 0:
+                        t = ext(0, 31)
+                    dist = 1 + ((src[ip] | (src[ip + 1] << 8)) >> 2)
+                    ip += 2
+                    _copy_match(out, dist, t + 2)
+                elif t >= 16:                     # M4 (distance 0 = end of stream)
+                    hi = (t & 8) << 11
+                    t &= 7
+                    if t == 0:
+                        t = ext(0, 7)
+                    dist = hi + ((src[ip] | (src[ip + 1] << 8)) >> 2)
+                    ip += 2
+                    if dist == 0:
+                        break
+                    _copy_match(out, dist + 0x4000, t + 2)
+                else:                             # M1 after a match: 2 bytes
+                    dist = 1 + (t >> 2) + (src[ip] << 2)
+                    ip += 1
+                    _copy_match(out, dist, 2)
+                state = "match_done"
+                continue
+            # match_done: the low 2 bits of the instruction's last byte = trailing literals
+            t = src[ip - 2] & 3
+            if t == 0:
+                state = "loop"
+            else:
+                state = "match_next"
+    except IndexError:
+        raise LzoError("truncated LZO stream") from None
+    if ip != n:
+        raise LzoError("%d bytes after the end of the LZO stream" % (n - ip))
+    if out_len is not None and len(out) != out_len:
+        raise LzoError("LZO block decoded to %d bytes, expected %d" % (len(out), out_len))
+    return bytes(out)
+
+
+EOF = b"\x11\x00\x00"
+
+
+def compress_stored(data):
+    """bytes -> LZO1X stream made of one literal run (decodes back to `data`)."""
+    data = bytes(data)
+    n = len(data)
+    if n == 0:
+        return EOF
+    if n <= 238:
+        return bytes([n + 17]) + data + EOF
+    # long literal run: 0x00, k zero bytes (255 each), last byte; length = 18 + 255*k + last
+    r = n - 18
+    k = (r - 1) // 255
+    last = r - 255 * k
+    return b"\x00" + b"\x00" * k + bytes([last]) + data + EOF

ackit/modpack.py

@@ -5,7 +5,10 @@ A package is a zip:
   items/NNN.wem      the exact WEM that was applied (used as-is when the target matches)
   items/NNN.flac     the same audio, decoded (used to re-encode when the target differs, e.g. another
                      language whose original has a different size)
-Each item records archive, target, name, group and the SHA-1 of the original it replaced.
+Each item records archive, target, name, group, the SHA-1 of the original it replaced and, for a voice line,
+its line identity (the same in every language, ackit/lines.py): that is how another language finds it.
+  "texts": [{"key", "text", "original"}]   edited subtitles / menu texts (keys: ackit/texts.py); optional,
+                                           so packages without texts stay valid
 """
 import io
 import json
@@ -22,9 +25,17 @@ from .originals import sha1_16
 FORMAT = 1
 
 
-def export_mod(ws, path, keys, name="", author="", description=""):
-    """Write the applied versions of `keys` (from the originals store) into a package."""
+def export_mod(ws, path, keys, name="", author="", description="", texts=()):
+    """Write the applied versions of sound `keys` and text keys `texts` (from the originals store) into a
+    package. Returns the number of entries."""
     items = []
+    recs = ws.texts.records()
+    text_items = []
+    for k in texts:
+        r = recs.get(k)
+        if not r or r.get("applied") is None:
+            raise ValueError("%s is not modified" % k)
+        text_items.append({"key": k, "text": r["applied"], "original": r["orig"]})
     with zipfile.ZipFile(path + ".tmp", "w", zipfile.ZIP_DEFLATED) as z:
         for i, key in enumerate(keys):
             e = ws.store.get(key)
@@ -39,20 +50,22 @@ def export_mod(ws, path, keys, name="", author="", description=""):
             z.writestr(base + ".wem", wem)
             z.writestr(base + ".flac", buf.getvalue())
             items.append({"archive": e["archive"], "target": e["target"], "name": s.name if s else "",
+                          "line": s.line if s else "",
                           "group": s.group if s else "", "original_sha1": e["orig_sha1"],
                           "wem": base + ".wem", "audio": base + ".flac", "meta": e.get("meta", {})})
         z.writestr("manifest.json", json.dumps({"format": FORMAT, "name": name, "author": author,
                                                 "description": description, "created": time.time(),
-                                                "game": "Batman: Arkham City (PC)", "items": items}, indent=1))
+                                                "game": "Batman: Arkham City (PC)", "items": items,
+                                                "texts": text_items}, indent=1, ensure_ascii=False))
     os.replace(path + ".tmp", path)
-    return len(items)
+    return len(items) + len(text_items)
 
 
 @dataclass
 class ImportItem:
     item: dict
     sound: object = None      # matched Sound or None
-    how: str = ""             # 'exact' | 'by name' | 'missing'
+    how: str = ""             # 'exact' | 'same line' | 'by name' | 'missing'
     wem: bytes = None         # ready-to-apply bytes (after `prepare_import`)
     note: str = ""
 
@@ -69,19 +82,34 @@ def match_mod(ws, path):
     """Match package items to sounds of this workspace. -> (manifest, [ImportItem])"""
     z, man = read_mod(path)
     out = []
+    by_line = {}
+    for c in ws.catalog.sounds:
+        if c.line:
+            by_line.setdefault(c.line, []).append(c)
     for it in man["items"]:
         key = "%s:%s" % (it["archive"], it["target"])
         s = ws.catalog.by_key.get(key)
-        if s and it.get("name") and s.name != it["name"]:
-            s = None
-        if s:
+        # the very same recording: same position and the same name or the same original bytes
+        if s and (s.name == it.get("name") or (s.sha1 and s.sha1 == it.get("original_sha1"))):
             out.append(ImportItem(it, s, "exact"))
             continue
+        if it.get("line"):                    # a voice line: its recording in the language loaded here
+            cands = sorted(by_line.get(it["line"], []), key=lambda c: (c.name != it.get("name"), c.key))
+            if cands:                         # several only for one recording stored twice (written together)
+                out.append(ImportItem(it, cands[0], "same line"))
+            else:
+                out.append(ImportItem(it, None, "missing", note="the line %r has no recording in the language "
+                                      "loaded here" % it.get("name")))
+            continue
         try:
             cands = [c for c in ws.catalog.search(it["name"]) if c.name == it["name"]] if it.get("name") else []
         except LookupError:
             cands = []
-        if len(cands) == 1:
+        if len(cands) == 1 and cands[0].line:
+            # a voice line from a package without line identities (older version): names alone are not proof
+            out.append(ImportItem(it, None, "missing", note="%r is a voice line from a package of an older "
+                                  "version; export the mod again to move it to another language" % it["name"]))
+        elif len(cands) == 1:
             out.append(ImportItem(it, cands[0], "by name"))
         else:
             out.append(ImportItem(it, None, "missing", note="no sound named %r in this game/language" % it.get("name")))
@@ -89,6 +117,20 @@ def match_mod(ws, path):
     return man, out
 
 
+def match_texts(ws, man):
+    """[(key, text, problem or '')] for the package's texts in this game (problem: why it cannot be applied)."""
+    out = []
+    for it in man.get("texts", []):
+        k, text = it["key"], it["text"]
+        try:
+            ws.texts.current(k)
+            ws.texts.check(k, text)
+            out.append((k, text, ""))
+        except Exception as e:  # missing key, too long, characters the file cannot hold
+            out.append((k, text, str(e)))
+    return out
+
+
 def prepare_import(ws, path, items, progress=None):
     """Fill `wem` for matched items: reuse the packaged WEM when it targets the very same original,
     otherwise re-encode the packaged audio for the local sound."""
@@ -98,8 +140,7 @@ def prepare_import(ws, path, items, progress=None):
             continue
         orig = ws.original_bytes(im.sound)
         wem = z.read(im.item["wem"])
-        lim = ws.limit(im.sound)
-        if im.how == "exact" and sha1_16(orig) == im.item.get("original_sha1") and (lim is None or len(wem) <= lim):
+        if im.how == "exact" and sha1_16(orig) == im.item.get("original_sha1") and ws.fits(im.sound, wem):
             im.wem, im.note = wem, "exact copy"
         else:
             x, sr = sf.read(io.BytesIO(z.read(im.item["audio"])), dtype="float32", always_2d=True)
@@ -110,11 +151,15 @@ def prepare_import(ws, path, items, progress=None):
 
 
 def import_mod(ws, path, force=False, progress=None):
-    """Match, prepare and apply a package. Returns the list of ImportItem (see .how/.note)."""
-    _, items = match_mod(ws, path)
+    """Match, prepare and apply a package. Returns ([ImportItem] (see .how/.note), [(key, text, problem)])."""
+    man, items = match_mod(ws, path)
     prepare_import(ws, path, items, progress)
-    changes = [(im.sound, im.wem, {"source": "mod:" + path.replace("\\", "/").split("/")[-1]})
-               for im in items if im.wem]
+    source = {"source": "mod:" + path.replace("\\", "/").split("/")[-1]}
+    changes = [(im.sound, im.wem, source) for im in items if im.wem]
     if changes:
         ws.apply(changes, force=force, progress=progress)
-    return items
+    texts = match_texts(ws, man)
+    ok = {k: v for k, v, problem in texts if not problem}
+    if ok:
+        ws.texts.apply(ok, force=force, progress=progress, meta=source)
+    return items, texts

ackit/paths.py

@@ -16,11 +16,6 @@ def resource_root():
     return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 
 
-def package_data(*parts):
-    """Files shipped inside the package (codebooks, known setups, hash list)."""
-    return os.path.join(os.path.dirname(os.path.abspath(__file__)), "data", *parts)
-
-
 def catalog_dir():
     return os.path.join(resource_root(), "data", "catalog")
 

ackit/settings.py

@@ -4,10 +4,10 @@ import os
 
 from .paths import user_dir
 
-DEFAULTS = {"game_dir": None, "store_root": None, "full_backup": True, "language": None,
+DEFAULTS = {"game_dir": None, "store_root": None, "full_backup": True,
             "input_device": None, "output_device": None, "last_dir": None,
             "background_enabled": True, "background_opacity": 0.1}
-OBSOLETE = ("theme", "background_image")
+OBSOLETE = ("theme", "background_image", "language")   # the editor starts in the game's language
 
 
 def path():

ackit/subtitles.py

@@ -0,0 +1,335 @@
+"""Subtitles: the RDialogueEvent objects inside the game's packages (docs/formats/subtitles.md).
+
+Each dialogue object references its speech event (an export named like the speech line, e.g.
+'Speech_CHURCH_B1_Ch2_POST_Explosion_BM_Cash_04') and holds the subtitle text of every language:
+  LocalizedSubtitles: array of {Language: FString, Subtitles: array of {Text: FString, (float @0x0c)}}
+in Rocksteady's compact tagged-property format (u16 type, u16 field offset, value; structs and arrays with
+a byte size; struct elements end with a u16 0).
+
+A dialogue object can be stored in several packages; a subtitle edit is written into every copy, in place:
+the new text replaces the bytes of the old FString and is padded with NUL characters (the game stops at the
+first NUL), so no size field, export or package layout changes. The text must therefore fit the original
+string's bytes (cp1252 strings: one byte per character; UTF-16 strings: two).
+"""
+import csv
+import gzip
+import io
+import os
+import re
+import struct
+from collections import defaultdict, namedtuple
+
+
+CLASS = "RDialogueEvent"
+SUBTITLE_CLASS, LOOKUP_CLASS = "RSubtitle", "RSubtitleLookup"   # scripted radio/encryption subtitles
+UNRESOLVED = "#lookup:%08X"     # speech of an RSubtitle until its lookup key is matched to a line name
+T_STRUCT, T_STRING, T_OBJECT = 0x09, 0x0D, 0x10
+
+# one subtitle string inside a package: pos = absolute uncompressed offset of the FString length field,
+# size = bytes of the whole FString (length field included), enc 'a' (cp1252) or 'w' (UTF-16)
+Field = namedtuple("Field", "lang index text pos size enc")
+# speech: name of the referenced speech event; speaker: the object's speaker field (RDialogueEvent only)
+Dialogue = namedtuple("Dialogue", "speech export fields speaker", defaults=("",))
+
+
+class SubtitleError(ValueError):
+    pass
+
+
+def read_fstring(b, o):
+    """-> (text, end offset, enc)"""
+    n = struct.unpack_from("<i", b, o)[0]
+    if n >= 0:
+        raw = bytes(b[o + 4:o + 4 + n])
+        return raw.split(b"\0", 1)[0].decode("cp1252", "replace"), o + 4 + n, "a"
+    raw = bytes(b[o + 4:o + 4 - 2 * n])
+    return raw.decode("utf-16-le", "replace").split("\0", 1)[0], o + 4 - 2 * n, "w"
+
+
+def encode_fstring(text, size, enc):
+    """The FString bytes (length field included) for `text`, exactly `size` bytes long (NUL padded).
+    Raises SubtitleError when the text does not fit."""
+    if "\0" in text:
+        raise SubtitleError("the text must not contain NUL characters")
+    if enc == "a":
+        try:
+            raw = text.encode("cp1252")
+        except UnicodeEncodeError:
+            raise SubtitleError("this subtitle is stored as Western European (Windows-1252) text; '%s' cannot be "
+                                "written"
+                                % next(c for c in text if not _cp1252(c))) from None
+        n = size - 4
+        if len(raw) + 1 > n:
+            raise SubtitleError("the text is %d characters long but this subtitle has room for %d"
+                                % (len(raw), n - 1))
+        return struct.pack("<i", n) + raw + bytes(n - len(raw))
+    raw = text.encode("utf-16-le")
+    n = (size - 4) // 2
+    if len(raw) // 2 + 1 > n:
+        raise SubtitleError("the text is %d characters long but this subtitle has room for %d" % (len(raw) // 2, n - 1))
+    return struct.pack("<i", -n) + raw + bytes(2 * n - len(raw))
+
+
+def _cp1252(c):
+    try:
+        c.encode("cp1252")
+        return True
+    except UnicodeEncodeError:
+        return False
+
+
+def room(field):
+    """Maximum number of characters for a subtitle string."""
+    return (field.size - 4 - 1) if field.enc == "a" else (field.size - 4) // 2 - 1
+
+
+def _parse_lines(b, o, end, base, lang):
+    """Subtitles array content [o, end) -> [Field]."""
+    count = struct.unpack_from("<I", b, o)[0]
+    o += 4
+    out = []
+    for i in range(count):
+        # element: props until u16 0; the text is the string at field offset 0
+        while True:
+            t, fo = struct.unpack_from("<HH", b, o)
+            if t == 0:
+                o += 2
+                break
+            if t == T_STRING:
+                text, e, enc = read_fstring(b, o + 4)
+                if fo == 0:
+                    out.append(Field(lang, i, text, base + o + 4, e - (o + 4), enc))
+                o = e
+            elif t == 0x04:               # 4-byte value (the line's time)
+                o += 8
+            else:
+                raise SubtitleError("unexpected property type %#x in a subtitle line" % t)
+            if o > end:
+                raise SubtitleError("subtitle line overruns its array")
+    if o != end:
+        raise SubtitleError("subtitle array size mismatch")
+    return out
+
+
+def parse_dialogue(b, base, names_idx):
+    """Fields of one dialogue/subtitle export (bytes `b` starting at uncompressed offset `base`).
+    names_idx: (name indices of the per-language array: LocalizedSubtitles or ArraySubtitles, of Subtitles)."""
+    arrays, sub = names_idx
+    i = -1
+    for ls in arrays:
+        i = b.find(struct.pack("<II", ls, 0))
+        if i >= 4 and struct.unpack_from("<H", b, i - 4)[0] == T_STRUCT:
+            break
+    if i < 4 or struct.unpack_from("<H", b, i - 4)[0] != T_STRUCT:
+        return []
+    size = struct.unpack_from("<i", b, i + 8)[0]
+    o, end = i + 16, i + 16 + size
+    count = struct.unpack_from("<I", b, o)[0]
+    o += 4
+    fields = []
+    for _ in range(count):
+        lang = None
+        while True:
+            t, fo = struct.unpack_from("<HH", b, o)
+            if t == 0:
+                o += 2
+                break
+            if t == T_STRING:
+                lang, o, _ = read_fstring(b, o + 4)
+            elif t == T_STRUCT:
+                nm, _nn, ssize, _ai = struct.unpack_from("<iiii", b, o + 4)
+                c0 = o + 20
+                if nm == sub:
+                    fields += _parse_lines(b, c0, c0 + ssize, base, lang)
+                o = c0 + ssize
+            else:
+                raise SubtitleError("unexpected property type %#x in a localized subtitle" % t)
+    if o != end:
+        raise SubtitleError("localized subtitle size mismatch")
+    return fields
+
+
+def _speaker(b):
+    """The speaker FString of a dialogue object (property 0x0D at field offset 0x34), '' if none."""
+    m = re.search(rb"\x0d\x00\x34\x00", b)
+    try:
+        return read_fstring(b, m.start() + 4)[0] if m else ""
+    except (struct.error, UnicodeDecodeError, ValueError):
+        return ""
+
+
+def _speech_ref(b, exports, imports):
+    """Name of the object referenced by the dialogue's first object property (its speech event)."""
+    m = re.search(rb"\x10\x00\x2c\x00", b)
+    if not m:
+        return ""
+    ref = struct.unpack_from("<i", b, m.start() + 4)[0]
+    if ref > 0 and ref <= len(exports):
+        return exports[ref - 1].name
+    if ref < 0 and -ref <= len(imports):
+        return imports[-ref - 1][1]
+    return ""
+
+
+class Collector:
+    """`visit` hook for banks.scan: collects the dialogue objects of every package."""
+
+    def __init__(self):
+        self.dialogues = defaultdict(list)       # speech name -> [(file, Dialogue)]
+        self.scripted = set()                    # speech names of RSubtitle objects (resolve)
+        self.issues = []
+
+    def __call__(self, rel, pkg, start, data):
+        try:
+            for d in scan_package(pkg, data, start):
+                self.dialogues[d.speech].append((rel, d))
+        except (SubtitleError, struct.error, ValueError, IndexError) as e:
+            self.issues.append(("subtitles", "%s: %s" % (rel, e)))
+
+    PREFIXES = ("", "RADIO_SCAN_", "RADIO_", "TV_", "CH_ZERO_", "GCPD_", "CHAR_", "SEQUENCE_")
+
+    def resolve(self, line_names):
+        """Give RSubtitle objects their speech line name: the name whose lookup_key equals the object's key,
+        from `line_names` or built from the object's own name (Speech_ + prefix + name; exact hash match)."""
+        by_key = {lookup_key(n): n for n in line_names}
+        for sp in [s for s in self.dialogues if s.startswith("#lookup:")]:
+            key = int(sp.split(":")[1], 16)
+            name = by_key.get(key)
+            if not name:
+                exp = self.dialogues[sp][0][1].export
+                name = next(("Speech_" + p + exp for p in self.PREFIXES if lookup_key("Speech_" + p + exp) == key), None)
+            copies = self.dialogues.pop(sp)
+            if not name:
+                self.issues.append(("subtitles", "no speech line for subtitle %s" % copies[0][1].export))
+                name = copies[0][1].export
+            self.dialogues[name].extend((rel, d._replace(speech=name)) for rel, d in copies)
+            self.scripted.add(name)
+
+
+# --- catalog files: SUB_<LANG>.csv (texts) + SUB_locations.csv.gz (every copy) ------------------
+LOCATIONS = "SUB_locations.csv.gz"
+TEXT_FIELDS = ["speech", "line", "text", "room", "enc", "sound"]
+class SubLine(namedtuple("SubLine", "lang speech line text room enc sound")):
+    """A subtitle string. sound: the voice recording(s) its speech event plays in that language (sound keys,
+    ';'-separated in the event's order: an event can play one of two versions of a line)."""
+    __slots__ = ()
+
+    @property
+    def sounds(self):
+        return tuple(self.sound.split(";")) if self.sound else ()
+SubLoc = namedtuple("SubLoc", "lang speech line file pos size enc")
+
+
+def key(lang, speech, line):
+    return "SUB:%s:%s:%d" % (lang, speech, line)
+
+
+def rows(collector, links):
+    """({lang: [SubLine]}, [SubLoc]) from a Collector; links: {speech: {lang: sound key}}."""
+    texts = defaultdict(list)
+    locs = []
+    for speech in sorted(collector.dialogues):
+        copies = collector.dialogues[speech]
+        rel0, d0 = copies[0]
+        for f in d0.fields:
+            ln = links.get(speech, {})
+            snd = ln.get(f.lang) or ln.get("INT", "")
+            texts[f.lang].append(SubLine(f.lang, speech, f.index, f.text, room(f), f.enc, snd))
+        for rel, d in copies:
+            for f in d.fields:
+                locs.append(SubLoc(f.lang, speech, f.index, rel, f.pos, f.size, f.enc))
+    return dict(texts), locs
+
+
+def write_texts(path, lines):
+    with open(path + ".tmp", "w", encoding="utf-8", newline="") as fh:
+        w = csv.writer(fh, lineterminator="\n")
+        w.writerow(TEXT_FIELDS)
+        for s in lines:
+            w.writerow([s.speech, s.line, s.text, s.room, s.enc, s.sound])
+    os.replace(path + ".tmp", path)
+
+
+def read_texts(path, lang):
+    with open(path, encoding="utf-8", newline="") as fh:
+        return [SubLine(lang, r["speech"], int(r["line"]), r["text"], int(r["room"]), r["enc"], r["sound"])
+                for r in csv.DictReader(fh)]
+
+
+def write_locations(path, locs):
+    raw = io.StringIO()
+    w = csv.writer(raw, lineterminator="\n")
+    w.writerow(["lang", "speech", "line", "file", "pos", "size", "enc"])
+    for l in sorted(locs, key=lambda l: (l.lang, l.speech, l.line, l.file, l.pos)):
+        w.writerow([l.lang, l.speech, l.line, l.file, "%X" % l.pos, l.size, l.enc])
+    with open(path + ".tmp", "wb") as fh:
+        with gzip.GzipFile(fileobj=fh, mode="wb", mtime=0) as gz:
+            gz.write(raw.getvalue().encode("utf-8"))
+    os.replace(path + ".tmp", path)
+
+
+def read_locations(path, lang=None):
+    """{SUB key: [SubLoc]} (only `lang` if given)."""
+    out = defaultdict(list)
+    with gzip.open(path, "rt", encoding="utf-8", newline="") as fh:
+        for r in csv.DictReader(fh):
+            if lang and r["lang"] != lang:
+                continue
+            l = SubLoc(r["lang"], r["speech"], int(r["line"]), r["file"], int(r["pos"], 16), int(r["size"]), r["enc"])
+            out[key(l.lang, l.speech, l.line)].append(l)
+    return dict(out)
+
+
+def lookup_key(line_name):
+    """Key of a speech line in the RSubtitleLookup table: FNV-1 of its name without 'Speech_'."""
+    from .bnk import fnv1
+    return fnv1(line_name[7:] if line_name.lower().startswith("speech_") else line_name)
+
+
+def _lookup_table(b):
+    """RSubtitleLookup export data -> {export index: key}: array of {int key @0, object @4}."""
+    count = struct.unpack_from("<I", b, 24)[0]
+    out, o = {}, 28
+    for _ in range(count):
+        if b[o:o + 4] != b"\x02\x00\x00\x00" or b[o + 8:o + 12] != b"\x10\x00\x04\x00":
+            raise SubtitleError("unexpected subtitle lookup layout")
+        key, ref = struct.unpack_from("<Ii", b, o + 4)[0], struct.unpack_from("<i", b, o + 12)[0]
+        out[ref] = key
+        o += 18
+    return out
+
+
+def scan_package(pkg, data=None, start=None, tables=None):
+    """[Dialogue] of one package (empty if it has none): RDialogueEvent objects (speech = the referenced
+    speech event) and RSubtitle objects (speech = UNRESOLVED % lookup key, see Collector.resolve)."""
+    if data is None:
+        start, data = pkg.read_all()
+    names, imports, exports = tables or pkg.tables(data, start)
+    by_class = {}
+    for i, (c, o, _) in enumerate(imports):
+        if c == "Class" and o in (CLASS, SUBTITLE_CLASS, LOOKUP_CLASS):
+            by_class[-(i + 1)] = o
+    arrays = [names.index(a) for a in ("LocalizedSubtitles", "ArraySubtitles") if a in names]
+    if not by_class or not arrays or "Subtitles" not in names:
+        return []
+    idx = (arrays, names.index("Subtitles"))
+    keys = {}
+    for e in exports:
+        if by_class.get(e.cls) == LOOKUP_CLASS and e.size > 0:
+            keys.update(_lookup_table(data[e.offset - start:e.offset - start + e.size]))
+    out = []
+    for e in exports:
+        kind = by_class.get(e.cls)
+        if kind in (CLASS, SUBTITLE_CLASS) and e.size > 0:
+            b = data[e.offset - start:e.offset - start + e.size]
+            fields = parse_dialogue(b, e.offset, idx)
+            if not fields:
+                continue
+            if kind == CLASS:
+                speech = _speech_ref(b, exports, imports)
+            elif e.index in keys:
+                speech = UNRESOLVED % keys[e.index]
+            else:
+                speech = e.name                       # not in a lookup table: keep the object's name
+            out.append(Dialogue(speech, e.name, fields, _speaker(b) if kind == CLASS else ""))
+    return out

ackit/texts.py

@@ -0,0 +1,223 @@
+"""Subtitles and menu text: reading, states, apply and revert (shared by the CLI and the editor).
+
+Keys (also used in mod packages):
+  SUB:<LANG>:<speech event>:<line>                  a subtitle string (docs/formats/subtitles.md)
+  MENU:<LANG>:<File>:<Section>:<Key>[#n]            a menu/UI text (docs/formats/text.md)
+Records live in the originals store (data["texts"]): {key: {"orig", "applied", "files", ...}}. The files
+holding the texts are derived files of the Workspace: each is rebuilt from its original copy plus every edit
+applied now (sounds and texts), so reverting the last edit restores it byte-for-byte.
+"""
+import os
+import time
+
+from . import loctext, subtitles
+from .catalog import load_subtitles
+from .game import GameError
+
+ORIGINAL, MODIFIED, LOST, CHANGED, FOREIGN = "original", "modified", "lost", "changed", "foreign"
+
+
+class TextConflictError(GameError):
+    def __init__(self, keys):
+        self.keys = keys
+        super().__init__("These texts were changed outside this tool (Steam update or 'verify', or another mod "
+                         "tool): %s.\nRevert them or confirm to overwrite." % ", ".join(keys[:5])
+                         + (" ..." if len(keys) > 5 else ""))
+
+
+class Texts:
+    def __init__(self, ws):
+        self.ws = ws
+        self._subs = {}          # lang -> ([SubLine], {key: [SubLoc]})
+        self._menu = {}          # lang -> [Entry]
+        self.loc_root = _find_dir(ws.game.root, "BmGame", "Localization")
+
+    # --- languages and lists ---------------------------------------------------------------------
+    def menu_languages(self):
+        return loctext.languages(self.loc_root) if self.loc_root else []
+
+    def subtitle_languages(self):
+        from .catalog import _shipped_index
+        langs = sorted(_shipped_index().get("SUB", {}).get("lines", {}))
+        return langs or self.menu_languages()
+
+    def subtitles(self, lang, progress=None):
+        """[SubLine] of a language (original texts from the catalog; use `current` for the game's text)."""
+        if lang not in self._subs:
+            self._subs[lang] = load_subtitles(self.ws.game, lang, progress)
+        return self._subs[lang][0]
+
+    def subtitle_locations(self, key):
+        lang = key.split(":")[1]
+        self.subtitles(lang)
+        return self._subs[lang][1].get(key, [])
+
+    def for_sound(self, sound, lang=None):
+        """Subtitle lines of a speech sound, in `lang` (default: the sound's own language). Languages without
+        their own voices link their subtitles to the INT sounds, so e.g. JPN subtitles are found for INT lines."""
+        lang = lang or (sound.languages[0] if sound.languages else sound.archive)
+        if lang not in self.subtitle_languages():
+            return []
+        return [s for s in self.subtitles(lang) if sound.key in s.sounds]
+
+    def menu(self, lang):
+        """[loctext.Entry] of the game's current files (the values the game shows now)."""
+        if lang not in self._menu:
+            self._menu[lang] = loctext.load(self.loc_root, lang) if self.loc_root else []
+        return self._menu[lang]
+
+    def is_menu_file(self, path):
+        return bool(self.loc_root) and os.path.normcase(os.path.abspath(path)).startswith(
+            os.path.normcase(os.path.abspath(self.loc_root)) + os.sep)
+
+    # --- values and states ----------------------------------------------------------------------
+    def records(self):
+        return self.ws.store.data.setdefault("texts", {})
+
+    def modified(self):
+        return [k for k, r in self.records().items() if r.get("applied") is not None]
+
+    def _menu_entry(self, key):
+        lang = key.split(":")[1]
+        raw = key[len("MENU:"):]
+        for e in loctext.load(self.loc_root, lang):
+            if e.key == raw:
+                return e
+        raise GameError("no menu text %s" % raw)
+
+    def files(self, key):
+        """Absolute paths of the files holding a text."""
+        if key.startswith("SUB:"):
+            return sorted({self.ws.bank_path(l.file) for l in self.subtitle_locations(key)})
+        e = self._menu_entry(key)
+        return [loctext.files(self.loc_root, e.lang)[e.file]]
+
+    def current(self, key):
+        """The text in the game files now."""
+        if key.startswith("SUB:"):
+            locs = self.subtitle_locations(key)
+            if not locs:
+                raise GameError("no subtitle %s" % key)
+            from .banks import open_file
+            l = locs[0]
+            b = open_file(self.ws.bank_path(l.file)).read(l.pos, l.size)
+            return subtitles.read_fstring(b, 0)[0]
+        return self._menu_entry(key).value
+
+    def catalog_text(self, key):
+        """Original text according to the shipped catalog (subtitles only), or None."""
+        if not key.startswith("SUB:"):
+            return None
+        _, lang, rest = key.split(":", 2)
+        speech, line = rest.rsplit(":", 1)
+        for s in self.subtitles(lang):
+            if s.speech == speech and s.line == int(line):
+                return s.text
+        return None
+
+    def original(self, key):
+        r = self.records().get(key)
+        return r["orig"] if r else self.current(key)
+
+    def state(self, key, current=None):
+        cur = self.current(key) if current is None else current
+        r = self.records().get(key)
+        if r:
+            if r.get("applied") is not None and cur == r["applied"]:
+                return MODIFIED
+            if cur == r["orig"]:
+                return LOST if r.get("applied") is not None else ORIGINAL
+            return CHANGED
+        cat = self.catalog_text(key)
+        if cat is not None and cur != cat:
+            return FOREIGN
+        return ORIGINAL
+
+    def limit(self, key):
+        """Maximum characters for a subtitle (None = no limit: menu text)."""
+        if not key.startswith("SUB:"):
+            return None
+        return min((subtitles.room(subtitles.Field(l.lang, l.line, "", l.pos, l.size, l.enc))
+                    for l in self.subtitle_locations(key)), default=0)
+
+    def check(self, key, text):
+        """Raise SubtitleError / TextError if `text` cannot be written for `key`."""
+        if key.startswith("SUB:"):
+            for l in self.subtitle_locations(key):
+                subtitles.encode_fstring(text, l.size, l.enc)
+        else:
+            e = self._menu_entry(key)
+            path = loctext.files(self.loc_root, e.lang)[e.file]
+            loctext.write(open(path, "rb").read(), e.lang, e.file, {e.key: text})
+
+    # --- writing --------------------------------------------------------------------------------
+    def apply(self, changes, force=False, progress=None, meta=None):
+        """changes: {key: new text}. Every file holding one of them is rebuilt once."""
+        for k, v in changes.items():
+            self.check(k, v)
+        conflicts = [k for k in changes if self.state(k) in (CHANGED, FOREIGN)]
+        paths = {p for k in changes for p in self.files(k)}
+        changed = set(self.ws.changed_files(paths))
+        conflicts += [k for k in changes if k not in conflicts and set(self.files(k)) & changed]
+        if conflicts and not force:
+            raise TextConflictError(conflicts)
+        for p in paths:
+            self.ws._keep_original(p)
+        recs = self.records()
+        now = time.time()
+        for k, v in changes.items():
+            if k not in recs:
+                recs[k] = {"orig": self.current(k), "files": [self.ws.game.rel(p) for p in self.files(k)],
+                           "first_modified": now, "applied": None}
+            recs[k].update(applied=v, applied_at=now, meta=meta or {})
+        self.ws.store.save()
+        self._sync(paths, progress)
+
+    def revert(self, keys, progress=None):
+        recs = self.records()
+        keys = [k for k in keys if k in recs]
+        paths = {p for k in keys for p in self.files(k)}
+        for k in keys:
+            recs.pop(k)
+        self.ws.store.save()
+        self._sync(paths, progress)
+        return keys
+
+    def revert_all(self, progress=None):
+        return self.revert(list(self.records()), progress)
+
+    def _sync(self, paths, progress):
+        self._menu.clear()
+        self.ws.sync_files(paths, progress)
+
+    # --- derived-file hooks (called by Workspace.sync_files) ------------------------------------
+    def subtitle_edits(self, rel):
+        out = []
+        for k, r in self.records().items():
+            if r.get("applied") is None or not k.startswith("SUB:"):
+                continue
+            for l in self.subtitle_locations(k):
+                if l.file == rel:
+                    out.append((l.pos, subtitles.encode_fstring(r["applied"], l.size, l.enc)))
+        return out
+
+    def menu_content(self, path, orig_raw):
+        fname = os.path.basename(path)
+        lang = os.path.basename(os.path.dirname(path)).upper()
+        values = {}
+        for k, r in self.records().items():
+            if r.get("applied") is None or not k.startswith("MENU:"):
+                continue
+            _, lg, f, _rest = k.split(":", 3)
+            if lg == lang and f == fname:
+                values[k[len("MENU:"):]] = r["applied"]
+        return loctext.write(orig_raw, lang, fname, values) if values else orig_raw
+
+
+def _find_dir(root, *parts):
+    d = root
+    for p in parts:
+        if not d or not os.path.isdir(d):
+            return None
+        d = next((os.path.join(d, x) for x in os.listdir(d) if x.lower() == p.lower()), None)
+    return d if d and os.path.isdir(d) else None

ackit/upk.py

@@ -0,0 +1,290 @@
+"""The game's UE3 packages (.upk): summary, LZO-compressed chunks, block-level reads and patches.
+
+Offsets used by the rest of ackit are positions in the *uncompressed* package (what Gildor's
+decompress.exe writes), so a sound's location does not depend on how the file is compressed.
+Format and rules: docs/formats/upk.md.
+
+Patching rewrites only the blocks that contain changed bytes (stored as one LZO literal run, see
+lzo.compress_stored) and moves the chunks after them; every other byte of the file stays identical.
+"""
+import struct
+from collections import OrderedDict, namedtuple
+
+from . import lzo
+
+TAG = 0x9E2A83C1
+COMPRESS_LZO = 2
+
+
+class PackageError(ValueError):
+    pass
+
+
+Export = namedtuple("Export", "index cls outer name size offset entry")   # entry: table position (uncompressed)
+
+
+class Block:
+    __slots__ = ("chunk", "uoff", "usize", "coff", "csize")
+
+    def __init__(self, chunk, uoff, usize, coff, csize):
+        self.chunk, self.uoff, self.usize, self.coff, self.csize = chunk, uoff, usize, coff, csize
+
+
+class Package:
+    """Read access to one package file. `blocks` covers the uncompressed range [header_size, size)."""
+
+    def __init__(self, path, cache_blocks=8):
+        self.path = path
+        with open(path, "rb") as f:
+            head = f.read(0x10000)
+            self._parse_summary(head, f)
+            self.file_size = f.seek(0, 2)
+        self._cache = OrderedDict()
+        self._cache_n = cache_blocks
+
+    # summary -----------------------------------------------------------------------------------
+    def _parse_summary(self, b, f):
+        if len(b) < 64 or struct.unpack_from("<I", b, 0)[0] != TAG:
+            raise PackageError("%s is not an Unreal package" % self.path)
+        self.version, self.licensee = struct.unpack_from("<HH", b, 4)
+        o = 12
+        n = struct.unpack_from("<i", b, o)[0]
+        o += 4 + max(n, 0)
+        # PackageFlags, NameCount/Offset, ExportCount/Offset, ImportCount/Offset, DependsOffset,
+        # ImportExportGuidsOffset, ImportGuidsCount, ExportGuidsCount, ThumbnailTableOffset
+        f12 = struct.unpack_from("<12i", b, o)
+        self.name_count, self.name_offset = f12[1], f12[2]
+        self.export_count, self.export_offset = f12[3], f12[4]
+        self.import_count, self.import_offset = f12[5], f12[6]
+        o += 48
+        self.guid = b[o:o + 16]
+        o += 16
+        ng = struct.unpack_from("<i", b, o)[0]
+        o += 4 + 12 * ng
+        self.engine, self.cooker, self.compression = struct.unpack_from("<3I", b, o)
+        o += 12
+        nc = struct.unpack_from("<i", b, o)[0]
+        o += 4
+        if nc and len(b) < o + 16 * nc:
+            f.seek(0)
+            b = f.read(o + 16 * nc)
+        self.chunk_table_offset = o
+        self.chunks = [list(struct.unpack_from("<4i", b, o + 16 * i)) for i in range(nc)]  # uoff usize coff csize
+        if nc and self.compression != COMPRESS_LZO:
+            raise PackageError("%s: unsupported compression %d" % (self.path, self.compression))
+        self.blocks = []
+        if not nc:
+            return
+        for ci, (uoff, usize, coff, csize) in enumerate(self.chunks):
+            f.seek(coff)
+            h = f.read(16)
+            tag, bsize, sum_c, sum_u = struct.unpack("<4I", h)
+            if tag != TAG or sum_u != usize:
+                raise PackageError("%s: bad chunk header at 0x%X" % (self.path, coff))
+            nb = (sum_u + bsize - 1) // bsize
+            tbl = f.read(8 * nb)
+            p = coff + 16 + 8 * nb
+            u = uoff
+            for i in range(nb):
+                c, uu = struct.unpack_from("<2I", tbl, 8 * i)
+                self.blocks.append(Block(ci, u, uu, p, c))
+                p += c
+                u += uu
+            if p != coff + csize or u != uoff + usize:
+                raise PackageError("%s: inconsistent chunk %d" % (self.path, ci))
+        self.block_size = bsize
+
+    @property
+    def compressed(self):
+        return bool(self.chunks)
+
+    @property
+    def header_size(self):
+        """Uncompressed size of the summary area (= start of the first chunk's data)."""
+        return self.chunks[0][0] if self.chunks else 0
+
+    @property
+    def size(self):
+        """Uncompressed package size."""
+        if not self.chunks:
+            return self.file_size
+        return self.chunks[-1][0] + self.chunks[-1][1]
+
+    # tables (need the uncompressed data: `read_all`) ---------------------------------------------
+    def tables(self, data=None, start=None):
+        """(names, imports, exports) of the package. imports: [(class name, object name, outer)],
+        exports: [Export]. Layouts: docs/formats/upk.md (licensee 101 adds one u32 to each export)."""
+        if data is None:
+            start, data = self.read_all()
+        buf = memoryview(data)
+        names = []
+        o = self.name_offset - start
+        for _ in range(self.name_count):
+            n = struct.unpack_from("<i", buf, o)[0]
+            o += 4
+            if n >= 0:
+                names.append(bytes(buf[o:o + n - 1]).decode("latin-1"))
+                o += n
+            else:
+                names.append(bytes(buf[o:o - 2 * n - 2]).decode("utf-16-le", "replace"))
+                o += -2 * n
+            o += 8                                   # name flags
+        imports = []
+        o = self.import_offset - start
+        for _ in range(self.import_count):
+            _pkg, _pn, cls, _cn, outer, nm, nn = struct.unpack_from("<7i", buf, o)
+            imports.append((names[cls], names[nm], outer))
+            o += 28
+        exports = []
+        o = self.export_offset - start
+        for i in range(self.export_count):
+            cls, sup, outer, nm, nn, arch, _x = struct.unpack_from("<7i", buf, o)
+            size, off = struct.unpack_from("<ii", buf, o + 36)
+            ngen = struct.unpack_from("<i", buf, o + 48)[0]
+            exports.append(Export(i + 1, cls, outer, names[nm] + ("_%d" % (nn - 1) if nn else ""), size, off,
+                                  o + start))
+            o += 72 + 4 * ngen
+        return names, imports, exports
+
+    def fingerprint(self):
+        """Cheap identity of the package contents: guid + chunk table (changes with any re-cook/patch)."""
+        import hashlib
+        h = hashlib.sha1(self.guid)
+        h.update(struct.pack("<q", self.file_size))
+        for c in self.chunks:
+            h.update(struct.pack("<4i", *c))
+        return h.hexdigest()[:16]
+
+    # reading -----------------------------------------------------------------------------------
+    def _block(self, i, f=None):
+        if i in self._cache:
+            self._cache.move_to_end(i)
+            return self._cache[i]
+        bl = self.blocks[i]
+        own = f is None
+        f = f or open(self.path, "rb")
+        try:
+            f.seek(bl.coff)
+            data = lzo.decompress(f.read(bl.csize), bl.usize)
+        finally:
+            if own:
+                f.close()
+        self._cache[i] = data
+        if len(self._cache) > self._cache_n:
+            self._cache.popitem(last=False)
+        return data
+
+    def _block_index(self, off):
+        lo, hi = 0, len(self.blocks) - 1
+        while lo <= hi:
+            mid = (lo + hi) // 2
+            bl = self.blocks[mid]
+            if off < bl.uoff:
+                hi = mid - 1
+            elif off >= bl.uoff + bl.usize:
+                lo = mid + 1
+            else:
+                return mid
+        raise PackageError("offset 0x%X is outside the package data" % off)
+
+    def read(self, off, n):
+        """Uncompressed bytes [off, off+n)."""
+        if not self.chunks:
+            with open(self.path, "rb") as f:
+                f.seek(off)
+                return f.read(n)
+        if off < self.header_size:
+            raise PackageError("reading the package summary area is not supported")
+        out = bytearray()
+        i = self._block_index(off)
+        with open(self.path, "rb") as f:
+            while len(out) < n:
+                bl = self.blocks[i]
+                d = self._block(i, f)
+                s = off + len(out) - bl.uoff
+                out += d[s:s + n - len(out)]
+                i += 1
+        return bytes(out)
+
+    def iter_data(self):
+        """(uncompressed offset, bytes) per block, in order (whole-package scans)."""
+        if not self.chunks:
+            with open(self.path, "rb") as f:
+                yield 0, f.read()
+            return
+        with open(self.path, "rb") as f:
+            for bl in self.blocks:
+                f.seek(bl.coff)
+                yield bl.uoff, lzo.decompress(f.read(bl.csize), bl.usize)
+
+    def read_all(self):
+        """The whole uncompressed data area as one bytes object (offset `data_start`)."""
+        parts = list(self.iter_data())
+        return (parts[0][0] if parts else 0), b"".join(p for _, p in parts)
+
+    # patching ----------------------------------------------------------------------------------
+    def patched(self, edits):
+        """edits: [(uncompressed offset, bytes)] (same length as what they overwrite) -> new file bytes.
+
+        Uncompressed packages are patched in place. For compressed ones the touched blocks are
+        re-encoded and the chunk headers and the summary's chunk table are updated."""
+        with open(self.path, "rb") as f:
+            raw = f.read()
+        if not self.chunks:
+            out = bytearray(raw)
+            for off, data in edits:
+                if off + len(data) > len(out):
+                    raise PackageError("patch beyond the end of %s" % self.path)
+                out[off:off + len(data)] = data
+            return bytes(out)
+        touched = {}
+        for off, data in edits:
+            if off < self.header_size or off + len(data) > self.size:
+                raise PackageError("patch outside the package data (0x%X)" % off)
+            pos = 0
+            while pos < len(data):
+                i = self._block_index(off + pos)
+                bl = self.blocks[i]
+                if i not in touched:
+                    touched[i] = bytearray(lzo.decompress(raw[bl.coff:bl.coff + bl.csize], bl.usize))
+                s = off + pos - bl.uoff
+                k = min(len(data) - pos, bl.usize - s)
+                touched[i][s:s + k] = data[pos:pos + k]
+                pos += k
+        # rebuild chunk by chunk
+        head = bytearray(raw[:self.chunks[0][2]])
+        body = bytearray()
+        by_chunk = {}
+        for i, bl in enumerate(self.blocks):
+            by_chunk.setdefault(bl.chunk, []).append(i)
+        for ci, (uoff, usize, coff, csize) in enumerate(self.chunks):
+            new_coff = len(head) + len(body)
+            idx = by_chunk[ci]
+            if not any(i in touched for i in idx):
+                body += raw[coff:coff + csize]
+                self._set_chunk(head, ci, uoff, usize, new_coff, csize)
+                continue
+            datas = []
+            for i in idx:
+                bl = self.blocks[i]
+                if i in touched:
+                    datas.append((lzo.compress_stored(touched[i]), bl.usize))
+                else:
+                    datas.append((raw[bl.coff:bl.coff + bl.csize], bl.usize))
+            hdr = struct.pack("<4I", TAG, self.block_size, sum(len(d) for d, _ in datas), usize)
+            hdr += b"".join(struct.pack("<2I", len(d), u) for d, u in datas)
+            chunk = hdr + b"".join(d for d, _ in datas)
+            body += chunk
+            self._set_chunk(head, ci, uoff, usize, new_coff, len(chunk))
+        return bytes(head + body)
+
+    def _set_chunk(self, head, ci, uoff, usize, coff, csize):
+        struct.pack_into("<4i", head, self.chunk_table_offset + 16 * ci, uoff, usize, coff, csize)
+
+
+def is_package(path):
+    try:
+        with open(path, "rb") as f:
+            return struct.unpack("<I", f.read(4))[0] == TAG
+    except (OSError, struct.error):
+        return False

ackit/verify/catalogcheck.py

@@ -0,0 +1,453 @@
+"""Cross-check the catalog against the installed game (`ackit verify --catalog`).
+
+For every archive the catalog in use (shipped CSV or the user-cache scan) is compared with a fresh scan:
+  missing     a sound of the game that is not in the catalog
+  extra       a catalog row that is not in the game
+  duplicate   a target or a name used twice
+  size/sha1   the catalog's original size or checksum differs from the game (game modified or other build)
+  category    the row's group/category is not what the classification rules give for its name
+  character   the verified speaker differs from data/catalog/characters.csv
+  language    the archive code does not match the WAD's language folder
+  copies      (BANK) stored copies or prefetch heads differ from the location index
+Run it on unmodified game files (`ackit status`), otherwise modified sounds show up as size/sha1 issues.
+"""
+import collections
+import hashlib
+import os
+import random
+
+from .. import banks, classify, subtitles
+from ..catalog import (Catalog, apply_lines, available_archives, bank_names, bank_sounds, build_lines, character_of,
+                       is_shipped, line_names, load_archive, load_banks, load_characters, mark_heads, read_csv,
+                       scan_archive, sfx_names, whole_recordings)
+from ..inject import read_target
+from ..paths import catalog_dir
+from ..wad import Wad
+
+FOLDER_LANG = {"ENGLISH(US)": "INT", "ITALIAN": "ITA", "FRENCH(FRANCE)": "FRA", "GERMAN": "DEU",
+               "SPANISH(SPAIN)": "ESN"}
+
+
+class Report:
+    def __init__(self):
+        self.checked = collections.Counter()
+        self.issues = []                       # (archive, kind, key, detail)
+        self.facts = []                        # what was established about the game's relations (full check)
+
+    def add(self, archive, kind, key, detail=""):
+        self.issues.append((archive, kind, key, detail))
+
+    def counts(self):
+        return collections.Counter((a, k) for a, k, _, _ in self.issues)
+
+    @property
+    def ok(self):
+        return not self.issues
+
+
+def _expected_category(s, names):
+    if s.archive == banks.ARCHIVE:
+        return classify.classify(s.name, bank=s.bank)
+    pack = None
+    if s.is_pack_media:
+        pack = names.get(int(s.target.split("/")[0], 16), s.target.split("/")[0])
+    return classify.classify(s.name, pack=pack)
+
+
+def _check_rows(rep, archive, rows, characters, names):
+    seen_t, seen_n = set(), {}
+    for s in rows:
+        rep.checked[archive] += 1
+        if s.archive != archive:
+            rep.add(archive, "language", s.key, "row says archive %s" % s.archive)
+        if s.target in seen_t:
+            rep.add(archive, "duplicate", s.key, "target")
+        seen_t.add(s.target)
+        n = (s.language, s.name.lower())          # BANK: names are unique per language
+        if n in seen_n or (s.language and ("", n[1]) in seen_n):
+            rep.add(archive, "duplicate", s.key, "name %s (also %s)" % (s.name, seen_n.get(n) or seen_n[("", n[1])]))
+        seen_n[n] = s.key
+        g, c = _expected_category(s, names)
+        if (s.group, s.category) != (g, c):
+            rep.add(archive, "category", s.key, "%s/%s, rules give %s/%s" % (s.group, s.category, g, c))
+        want = character_of(s, characters)
+        if s.character != want:
+            rep.add(archive, "character", s.key, "%r, curated list says %r" % (s.character, want))
+
+
+def check_wad(rep, game, archive, rows, characters, lines=None):
+    """Rows vs a fresh scan of the WAD. With `lines` (full check) the fresh sounds get the game's line
+    identities and names, so names, aliases and lines are compared too."""
+    path = game.wad(archive)
+    if archive != "SFX":
+        folder = os.path.basename(os.path.dirname(path)).upper()
+        if FOLDER_LANG.get(folder, archive) != archive:
+            rep.add(archive, "language", archive, "WAD in folder %s" % folder)
+    _check_rows(rep, archive, rows, characters, bank_names(game.cooked))
+    fresh_list = scan_archive(archive, path, characters=characters)
+    if lines is not None and archive in lines.langs:
+        apply_lines(fresh_list, lines, lines.names)
+    fresh = {s.target: s for s in fresh_list}
+    if lines is None and archive in game.languages:    # quick check: line names need the banks, not rescanned
+        byk = {s.target: s for s in rows}
+        for t, s in fresh.items():
+            if t in byk:
+                s.name, s.aliases, s.line = byk[t].name, byk[t].aliases, byk[t].line
+    byk = {s.target: s for s in rows}
+    for t in sorted(set(fresh) & set(byk)):
+        f, s = fresh[t], byk[t]
+        if (f.name, tuple(f.aliases), f.line) != (s.name, tuple(s.aliases), s.line):
+            rep.add(archive, "name", s.key, "catalog %s %s line %s, game %s %s line %s"
+                    % (s.name, list(s.aliases), s.line or "-", f.name, list(f.aliases), f.line or "-"))
+    cat = {s.target: s for s in rows}
+    for t in sorted(set(fresh) - set(cat)):
+        rep.add(archive, "missing", "%s:%s" % (archive, t), fresh[t].name)
+    for t in sorted(set(cat) - set(fresh)):
+        rep.add(archive, "extra", "%s:%s" % (archive, t), cat[t].name)
+    wad = Wad(path)
+    for t in sorted(set(cat) & set(fresh)):
+        s = cat[t]
+        if s.size and s.size != fresh[t].size and not s.is_pack_media:
+            rep.add(archive, "size", s.key, "catalog %d, game %d" % (s.size, fresh[t].size))
+        if s.sha1:
+            h = hashlib.sha1(read_target(path, t, wad)).hexdigest()[:16]
+            if h != s.sha1:
+                rep.add(archive, "sha1", s.key, "game file differs from the catalog's original")
+
+
+def check_banks(rep, game, rows, locations, characters, quick=False, sample=40, progress=None):
+    """BANK rows, copies, prefetch heads and subtitles vs a scan of the packages. The full check (not quick)
+    also derives the game's lines from that scan and returns them (else None)."""
+    _check_rows(rep, banks.ARCHIVE, rows, characters, bank_names(game.cooked))
+    files = banks.bank_files(game.cooked)
+    if quick:
+        pk = [f for f in files if f.lower().endswith(".upk")]
+        rnd = random.Random(1)
+        files = [f for f in files if f.startswith("SFX/")] + sorted(rnd.sample(pk, min(sample, len(pk))))
+    col = subtitles.Collector()
+    sc = banks.scan(game.cooked, progress, files=files, visit=col)
+    known = {n.split("#")[0] for s in rows for n in s.names()}           # BANK names
+    for a in game.languages:                                               # + speech line names of the catalogs
+        p = os.path.join(catalog_dir(), "%s.csv" % a)
+        if os.path.exists(p):
+            known.update(n.split("#")[0] for s in read_csv(p) for n in s.names())
+    col.resolve({n for n in sc.names | known if n.lower().startswith("speech_")})
+    check_subtitles(rep, game, col, set(files), quick)
+    lines = None
+    index = collections.defaultdict(set)
+    for m, ls in locations.items():
+        for l in ls:
+            if l.file in set(files):
+                index[(l.media, l.kind)].add((l.file, l.offset, l.size, l.sound if l.kind == banks.PREFETCH_HEAD else ""))
+    if not quick:
+        sc.names.update(col.dialogues)        # as the build does: subtitle objects name speech events too
+        lines = build_lines(game, sc)
+        mark_heads(sc, whole_recordings(sc, lines))       # WAD recordings also stored whole in banks
+    else:                                     # a sample cannot derive them: the index says which they are
+        mark_heads(sc, {m for m, k in index if k == banks.PREFETCH_HEAD} - sc.prefetch)
+    found = collections.defaultdict(set)
+    for l in sc.locations:
+        head = l.kind == banks.PREFETCH_HEAD
+        found[(l.media, l.kind)].add((l.file, l.offset, l.size, lines.head_key(l) if head and lines else ""))
+    if lines is None:                       # quick: a subset of the packages cannot resolve the heads' lines
+        index = {k: {(f, o, s, "") for f, o, s, _ in v} for k, v in index.items()}
+    for k in sorted(set(found) | set(index)):
+        if found.get(k, set()) != index.get(k, set()):
+            rep.add(banks.ARCHIVE, "copies", "%08X" % k[0], "%s: index %d, game %d"
+                    % ("head" if k[1] == banks.PREFETCH_HEAD else "copies", len(index.get(k, ())), len(found.get(k, ()))))
+    cat = {s.target: s for s in rows}
+    fresh_list = bank_sounds(sc, characters, sfx_names(game))
+    if lines is not None:
+        lines.names = line_names(game, sc, lines, {s.key: s.name for s in fresh_list})
+        apply_lines(fresh_list, lines, lines.names)
+        for s in fresh_list:
+            s.character = character_of(s, characters)
+    fresh = {s.target: s for s in fresh_list}
+    for t in sorted(set(fresh) - set(cat)):
+        rep.add(banks.ARCHIVE, "missing", "BANK:" + t, fresh[t].bank)
+    if not quick:
+        for t in sorted(set(cat) - set(fresh)):
+            rep.add(banks.ARCHIVE, "extra", "BANK:" + t, cat[t].name)
+    for t in sorted(set(cat) & set(fresh)):
+        s, f = cat[t], fresh[t]
+        if s.sha1 and f.sha1 and s.sha1 != f.sha1:
+            rep.add(banks.ARCHIVE, "sha1", s.key, "game file differs from the catalog's original")
+        if f.size < s.size:
+            rep.add(banks.ARCHIVE, "size", s.key, "catalog %d, game %d" % (s.size, f.size))
+        if lines is not None and (f.name, f.language, f.line) != (s.name, s.language, s.line):
+            rep.add(banks.ARCHIVE, "name", s.key, "catalog %s [%s] line %s, game %s [%s] line %s"
+                    % (s.name, s.language, s.line or "-", f.name, f.language, f.line or "-"))
+    for m, ls in locations.items():
+        for l in ls:
+            if l.kind == banks.PREFETCH_HEAD and not l.sound:
+                rep.add(banks.ARCHIVE, "copies", "%08X" % m, "prefetch head without a speech line")
+    return sc, lines, col
+
+
+def check_subtitles(rep, game, col, files, quick):
+    """Subtitle copies found in the scanned packages vs SUB_locations + the SUB_<LANG>.csv texts."""
+    from ..catalog import _shipped_index, catalog_dir
+    loc_path = os.path.join(catalog_dir(), subtitles.LOCATIONS)
+    if not os.path.exists(loc_path) or not is_shipped(game, banks.ARCHIVE):
+        return
+    index = collections.defaultdict(set)
+    for k, ls in subtitles.read_locations(loc_path).items():
+        for l in ls:
+            if l.file in files:
+                index[k].add((l.file, l.pos, l.size))
+    found, texts = collections.defaultdict(set), {}
+    for speech, copies in col.dialogues.items():
+        for rel, d in copies:
+            for f in d.fields:
+                k = subtitles.key(f.lang, speech, f.index)
+                found[k].add((rel, f.pos, f.size))
+                texts[k] = f.text
+    for k in sorted(set(found) | set(index)):
+        if found.get(k, set()) != index.get(k, set()):
+            rep.add("SUB", "copies", k, "index %d, game %d" % (len(index.get(k, ())), len(found.get(k, ()))))
+    langs = sorted(_shipped_index().get("SUB", {}).get("lines", {}))
+    bank_lang = {s.key: s.language for s in read_csv(os.path.join(catalog_dir(), "BANK.csv"))}
+    for lang in langs:
+        voice = lang if lang in game.languages else "INT"      # languages without own voices: INT voices
+        p = os.path.join(catalog_dir(), "SUB_%s.csv" % lang)
+        rows = subtitles.read_texts(p, lang)
+        rep.checked["SUB_" + lang] += len(rows)
+        seen = set()
+        for s in rows:
+            k = subtitles.key(lang, s.speech, s.line)
+            if k in seen:
+                rep.add("SUB", "duplicate", k)
+            seen.add(k)
+            if k in texts and texts[k] != s.text:
+                rep.add("SUB", "text", k, "catalog %r, game %r" % (s.text[:40], texts[k][:40]))
+            for snd in s.sounds:
+                if snd.startswith(banks.ARCHIVE + ":"):
+                    b = bank_lang.get(snd)
+                    if b and voice not in b.split(";"):             # '' = stored for every language
+                        rep.add("SUB", "language", k, "linked to %s (%s)" % (snd, b))
+                elif not snd.startswith(voice + ":"):
+                    rep.add("SUB", "language", k, "linked to %s, not a %s voice line" % (snd, voice))
+
+
+def check_lines(rep, game, sc, lines, locations):
+    """The game's relations, re-derived from the game files (ackit/lines.py) and compared with the catalogs:
+      links   a broken one-to-one relation in the game data (a WEM without its table row, a Sound object with
+              two recordings in one language, a line without a recording in some language, ...)
+      line    a catalog row whose line identity / name differs from the game's
+      subtitle  a subtitle linked to a different line in one language than in INT
+    Facts (rep.facts) give the counts, prefetch heads compared byte by byte with the start of their WAD
+    recording, and the length agreement of a line's recordings across languages (independent evidence)."""
+    for kind, detail in lines.issues:
+        rep.add("LINES", "links", kind, detail)
+    st = lines.stats
+    for lid, miss in sorted(st["lines_missing_a_language"].items()):
+        rep.facts.append("line %s (%s) has no recording in %s" % (lid, lines.names.get(lid, "?"), ", ".join(miss)))
+    for k in st["without_line"]:
+        rep.add("LINES", "links", k, "a recording of no line")
+    # independent of the BKPK tables: each WAD recording is the source of a Sound object in a bank of the
+    # language the tables give it (localized banks: one per declared language)
+    named = total = 0
+    for L in lines.langs:
+        for m, k in lines.owners.get(L, {}).items():
+            if not k.startswith(banks.ARCHIVE + ":"):
+                total += 1
+                if m in sc.lang_media.get(L, ()):
+                    named += 1
+                else:
+                    rep.add("LINES", "links", k, "media %08X is not a source in any %s bank" % (m, L))
+    rep.facts.append("WAD recordings named as a source by a Sound object of their own language's bank: %d of %d"
+                     % (named, total))
+    rep.facts.append("voice languages %s: %d recordings (WAD WEMs %s, %d of them second places of a recording "
+                     "stored twice, + localized in-memory), %d lines, %d of them with exactly one recording in every "
+                     "language" % ("/".join(lines.langs), st["recordings"],
+                                   "/".join(str(st["wad_wems"].get(L, 0)) for L in lines.langs), len(st["copies"]),
+                                   st["lines"], st["lines_in_every_language"]))
+    # catalog rows vs the game's lines
+    rows = {}
+    for L in lines.langs + [banks.ARCHIVE]:
+        p = os.path.join(catalog_dir(), "%s.csv" % L)
+        if os.path.exists(p):
+            for s in read_csv(p):
+                rows[s.key] = s
+    for lid, per in lines.members.items():
+        for L, k in per.items():
+            s = rows.get(k)
+            if s is None:
+                rep.add("LINES", "line", k, "recording of line %s is not in the catalog" % lid)
+            elif s.line != lid or s.name != lines.names[lid]:
+                rep.add("LINES", "line", k, "catalog %s/%s, game %s/%s" % (s.line, s.name, lid, lines.names[lid]))
+    for k, s in rows.items():
+        if s.line and lines.line_of.get(k) != s.line:
+            rep.add("LINES", "line", k, "catalog says line %s, the game data %s" % (s.line, lines.line_of.get(k)))
+    # prefetch heads: every stored head is the start of the WAD recording it is mapped to
+    wads = {L: (game.wad(L), Wad(game.wad(L))) for L in lines.langs}
+    heads = [l for ls in locations.values() for l in ls if l.kind == banks.PREFETCH_HEAD]
+    by_file = collections.defaultdict(list)
+    for l in heads:
+        by_file[l.file].append(l)
+    same = 0
+    for rel, ls in by_file.items():
+        r = banks.open_file(banks.file_path(game.cooked, rel))
+        for l in ls:
+            L = l.sound.split(":")[0]
+            if L not in wads:
+                rep.add("LINES", "head", "%08X" % l.media, "head in %s maps to %r" % (rel, l.sound))
+                continue
+            w = read_target(wads[L][0], l.sound.split(":", 1)[1], wads[L][1])
+            if r.read(l.offset, l.size) == (w + bytes(l.size))[:l.size]:
+                same += 1
+            else:
+                rep.add("LINES", "head", "%08X" % l.media, "head in %s is not the start of %s" % (rel, l.sound))
+    same_copies = 0
+    for k, primary in sorted(lines.copy_of.items()):          # one recording stored in two WAD entries
+        L = k.split(":")[0]
+        a = read_target(wads[L][0], k.split(":", 1)[1], wads[L][1])
+        b = read_target(wads[L][0], primary.split(":", 1)[1], wads[L][1])
+        if a == b:
+            same_copies += 1
+        else:
+            rep.add("LINES", "copies", k, "differs from %s (same media ID)" % primary)
+    rep.facts.append("recordings stored twice in a WAD (same media ID, written together): %d, byte-identical: %d"
+                     % (len(lines.copy_of), same_copies))
+    rep.facts.append("prefetch heads and whole bank copies: %d of %d are byte-identical to the start of their WAD "
+                     "recording"
+                     % (same, len(heads)))
+    # independent evidence: the recordings of a line have similar lengths in every language
+    ratios = []
+    for lid, per in lines.members.items():
+        secs = [rows[k].seconds for k in per.values() if k in rows and rows[k].seconds > 0]
+        if len(secs) > 1:
+            ratios.append((max(secs) / min(secs), lid))
+    if ratios:
+        ratios.sort()
+        within = sum(1 for r, _ in ratios if r <= 1.5)
+        rep.facts.append("lengths across languages: %d of %d lines within a factor 1.5, median %.2f, longest ratio "
+                         "%.1f (%s)" % (within, len(ratios), ratios[len(ratios) // 2][0], ratios[-1][0],
+                                        lines.names.get(ratios[-1][1], "")))
+    return rows
+
+
+def check_subtitle_lines(rep, rows):
+    """A subtitle line (speech event + line index) must point to the same line in every language."""
+    from ..catalog import _shipped_index
+    langs = sorted(_shipped_index().get("SUB", {}).get("lines", {}))
+    ref = {}
+    p = os.path.join(catalog_dir(), "SUB_INT.csv")
+    if os.path.exists(p):
+        for s in subtitles.read_texts(p, "INT"):
+            if s.sounds and all(k in rows for k in s.sounds):
+                ref[(s.speech, s.line)] = [rows[k].line for k in s.sounds]
+    linked = total = 0
+    for lang in langs:
+        for s in subtitles.read_texts(os.path.join(catalog_dir(), "SUB_%s.csv" % lang), lang):
+            total += 1
+            if not s.sound:
+                continue
+            linked += 1
+            unknown = [k for k in s.sounds if k not in rows]
+            want = ref.get((s.speech, s.line))
+            if unknown:
+                rep.add("SUB", "link", subtitles.key(lang, s.speech, s.line), "linked to unknown %s" % unknown)
+            elif want and [rows[k].line for k in s.sounds] != want:
+                rep.add("SUB", "link", subtitles.key(lang, s.speech, s.line),
+                        "lines %s, INT subtitle is on lines %s" % ([rows[k].line for k in s.sounds], want))
+    rep.facts.append("subtitles: %d strings in %d languages, %d linked to a voice line, each to the same line as in INT"
+                     % (total, len(langs), linked))
+
+
+SAME_PERSON = {("mr freeze", "mister freeze"), ("batman", "bruce wayne"), ("two face", "twoface"),
+               ("azrael", "mysterious stranger")}
+
+
+def _person(s):
+    return " ".join(s.lower().replace("'", "").replace(".", "").replace("_", " ").replace("-", " ").split())
+
+
+def check_speakers(rep, rows, col):
+    """Independent evidence for the curated speaker labels: the dialogue object of a subtitled line names its
+    speaker. Facts only (the lists are curated by listening; differences are for a person to judge)."""
+    speaker = {}
+    for speech, copies in col.dialogues.items():
+        for _, d in copies:
+            if d.speaker:
+                speaker.setdefault(speech, d.speaker)
+    p = os.path.join(catalog_dir(), "SUB_INT.csv")
+    if not os.path.exists(p):
+        return
+    agree, unknown, differ = 0, 0, []
+    for s in subtitles.read_texts(p, "INT"):
+        spk = speaker.get(s.speech, "")
+        if s.line != 0 or not spk:
+            continue
+        for k in s.sounds:
+            r = rows.get(k)
+            if not r or not r.character:
+                continue
+            who = [_person(c) for c in r.character.split("; ")]
+            g = _person(spk)
+            if any(g in w or w in g or (w, g) in SAME_PERSON for w in who):
+                agree += 1
+            elif who == ["unknown"]:
+                unknown += 1
+            else:
+                differ.append("%s (list: %s, game: %s)" % (r.name, r.character, spk))
+    rep.facts.append("speakers: %d labelled voice lines agree with the speaker field of their dialogue object, "
+                     "%d listed as Unknown are named there, %d differ%s"
+                     % (agree, unknown, len(differ), (": " + "; ".join(differ)) if differ else ""))
+
+
+def check_menu(rep, game):
+    """Menu texts: the same keys in every language (facts; differences are the game's own)."""
+    from .. import loctext
+    root = os.path.join(game.root, "BmGame", "Localization")
+    if not os.path.isdir(root):
+        return
+    keys = {}
+    for lang in loctext.languages(root):
+        try:
+            keys[lang] = {(os.path.splitext(e.file)[0].upper(), e.section, e.name, e.occurrence)
+                          for e in loctext.load(root, lang)}
+        except Exception as e:  # unreadable folder: report, do not stop
+            rep.add("MENU", "read", lang, str(e))
+    if "INT" not in keys:
+        return
+    for lang, ks in sorted(keys.items()):
+        extra, miss = ks - keys["INT"], keys["INT"] - ks
+        rep.facts.append("menu text %s: %d keys%s" % (lang, len(ks), "" if not (extra or miss) else
+                         " (%d not in INT, %d of INT missing)" % (len(extra), len(miss))))
+
+
+def run(game, archives=None, quick=False, log=print, progress=None):
+    """Check the given archives (default: all). Returns a Report."""
+    rep = Report()
+    archives = archives or available_archives(game)
+    cat = Catalog(game, archives, progress)
+    by_arch = collections.defaultdict(list)
+    for s in cat.sounds:
+        by_arch[s.archive].append(s)
+    lines = None
+    if banks.ARCHIVE in archives:           # first: the full check derives the game's lines from this scan
+        log("checking %s ..." % banks.ARCHIVE)
+        rows = load_banks(game, progress)[0]              # every language, not only the loaded one
+        sc, lines, col = check_banks(rep, game, rows, cat.locations, load_characters(lang=None), quick=quick,
+                               progress=progress)
+    for a in archives:
+        if a == banks.ARCHIVE:
+            continue
+        log("checking %s ..." % a)
+        rows_a = by_arch[a] if a in cat.archives else load_archive(game, a, progress)
+        check_wad(rep, game, a, rows_a, load_characters(lang=a), lines)
+    if lines is not None and is_shipped(game, banks.ARCHIVE):
+        log("checking the relations ...")
+        rows_all = check_lines(rep, game, sc, lines, cat.locations)
+        check_subtitle_lines(rep, rows_all)
+        check_speakers(rep, rows_all, col)
+        check_menu(rep, game)
+    characters = load_characters(lang=game.language())
+    known = {n.lower() for s in cat.sounds for n in s.names()}
+    # speaker names are the same in every language; only meaningful for the build the lists were made for
+    if any(a in game.languages and is_shipped(game, a) for a in archives):
+        for n in sorted(characters):
+            if n not in known:
+                rep.add("characters.csv", "extra", n, "no sound with this name")
+    return rep

ackit/verify/debugger.py

@@ -61,39 +61,43 @@ def read_mem(pid, addr, n):
         k32.CloseHandle(h)
 
 
-u32 = ctypes.WinDLL("user32", use_last_error=True)
-WM_KEYDOWN, WM_KEYUP = 0x0100, 0x0101
-VK = dict(enter=0x0D, space=0x20, esc=0x1B, up=0x26, down=0x28, left=0x25, right=0x27, e=0x45)
+class _MBI(ctypes.Structure):
+    _fields_ = [("BaseAddress", ctypes.c_void_p), ("AllocationBase", ctypes.c_void_p),
+                ("AllocationProtect", wt.DWORD), ("RegionSize", ctypes.c_size_t), ("State", wt.DWORD),
+                ("Protect", wt.DWORD), ("Type", wt.DWORD)]
 
 
-def game_windows(pid):
-    """Top-level windows of the game process."""
-    found = []
-    cb_t = ctypes.WINFUNCTYPE(wt.BOOL, wt.HWND, wt.LPARAM)
+k32.VirtualQueryEx.argtypes = [wt.HANDLE, ctypes.c_void_p, ctypes.POINTER(_MBI), ctypes.c_size_t]
+k32.VirtualQueryEx.restype = ctypes.c_size_t
+MEM_COMMIT, PAGE_NOACCESS, PAGE_GUARD = 0x1000, 0x01, 0x100
 
-    def cb(hwnd, _):
-        p = wt.DWORD()
-        u32.GetWindowThreadProcessId(hwnd, ctypes.byref(p))
-        if p.value == pid and u32.IsWindowVisible(hwnd):
-            found.append(hwnd)
-        return True
-    u32.EnumWindows(cb_t(cb), 0)
+
+def search_memory(pid, needles, limit=1 << 32):
+    """Which of `needles` (bytes) occur anywhere in the committed memory of the process -> set."""
+    h = k32.OpenProcess(PROCESS_VM_READ | PROCESS_QUERY_INFORMATION, False, pid)
+    if not h:
+        return set()
+    found = set()
+    try:
+        addr, mbi = 0, _MBI()
+        while addr < limit and len(found) < len(needles):
+            if not k32.VirtualQueryEx(h, ctypes.c_void_p(addr), ctypes.byref(mbi), ctypes.sizeof(mbi)):
+                break
+            size = mbi.RegionSize
+            if mbi.State == MEM_COMMIT and not mbi.Protect & (PAGE_NOACCESS | PAGE_GUARD):
+                buf = ctypes.create_string_buffer(size)
+                got = ctypes.c_size_t()
+                if k32.ReadProcessMemory(h, ctypes.c_void_p(addr), buf, size, ctypes.byref(got)):
+                    data = buf.raw[:got.value]
+                    for n in needles:
+                        if n not in found and n in data:
+                            found.add(n)
+            addr = (mbi.BaseAddress or 0) + size
+    finally:
+        k32.CloseHandle(h)
     return found
 
 
-def post_key(pid, key, hold=0.08):
-    """Post a key press to the game's window only (never global SendInput)."""
-    vk = VK[key] if isinstance(key, str) else key
-    sc = u32.MapVirtualKeyW(vk, 0)
-    n = 0
-    for hwnd in game_windows(pid):
-        u32.PostMessageW(hwnd, WM_KEYDOWN, vk, 1 | (sc << 16))
-        time.sleep(hold)
-        u32.PostMessageW(hwnd, WM_KEYUP, vk, 1 | (sc << 16) | (3 << 30))
-        n += 1
-    return n
-
-
 def running_game_pids():
     out = subprocess.run(["tasklist", "/FI", "IMAGENAME eq BatmanAC.exe", "/FO", "CSV", "/NH"],
                          capture_output=True, text=True).stdout

ackit/verify/ingame.py

@@ -69,18 +69,18 @@ def run(ws, key=DEFAULT_TARGET, wem=None, mode="modded", seconds=40.0, threshold
     report = dict(target=s.key, name=s.name, mode=mode)
     before = {f: _sha1_file(f) for f in files} if mode == "modded" else {}
 
-    if mode == "modded":
-        ws.apply([(s, wem, {"source": "ingame-test"})], progress=log)
-    expect = ws.current_bytes(s)
-    orig = ws.original_bytes(s)
-    from ..wem import Wem
-    report["expect_hash"] = "%08x" % Wem.parse(expect).vorb["codebook_hash"]
-    ref_expect, ref_orig = _mono48(expect), _mono48(orig)
-
-    rec = LoopbackRecorder(seconds + 10)
-    rec.start()
-    runner = Run(ws.game.exe, logp)
-    try:
+    rec = runner = None
+    try:                                   # from the first write on: whatever happens, the finally block reverts
+        if mode == "modded":
+            ws.apply([(s, wem, {"source": "ingame-test"})], progress=log)
+        expect = ws.current_bytes(s)
+        orig = ws.original_bytes(s)
+        from ..wem import Wem
+        report["expect_hash"] = "%08x" % Wem.parse(expect).vorb["codebook_hash"]
+        ref_expect, ref_orig = _mono48(expect), _mono48(orig)
+        rec = LoopbackRecorder(seconds + 10)
+        rec.start()
+        runner = Run(ws.game.exe, logp)
         report.update(runner.start())
         while time.time() - runner.t0 < seconds and runner.alive():
             time.sleep(0.5)
@@ -88,9 +88,11 @@ def run(ws, key=DEFAULT_TARGET, wem=None, mode="modded", seconds=40.0, threshold
                 break
         report["alive_at_end"] = runner.alive()
     finally:
-        runner.stop()
-        rec.stop_flag = True
-        rec.join(5)
+        if runner:
+            runner.stop()
+        if rec:
+            rec.stop_flag = True
+            rec.join(5)
         if mode == "modded" and revert_after:
             ws.revert([s], progress=log)
             report["reverted_byte_identical"] = all(_sha1_file(f) == h for f, h in before.items())
@@ -119,3 +121,155 @@ def run(ws, key=DEFAULT_TARGET, wem=None, mode="modded", seconds=40.0, threshold
     report["result"] = "PASS" if ok else "FAIL"
     json.dump(report, open(logp.replace(".log", ".json"), "w"), indent=1)
     return report
+
+
+# --- texts --------------------------------------------------------------------------------------
+def run_texts(ws, seconds=45.0, menu_files=("FRONTEND", "GFXUI"), n_menu=12, n_subs=12, logdir=None, log=print):
+    """Write markers into menu texts of the game's language (front-end files, loaded at startup), into the
+    subtitles stored in Startup.upk (RSubtitle objects, loaded at startup) and into some other subtitles, start
+    the game, then check: no crash, the menu and Startup subtitle markers are in the game's memory (it loaded the
+    edited files), and after the revert every touched file is byte-identical. The other subtitles are not loaded
+    on the title screen; for them the check is that the patched packages do not break the game."""
+    import random
+    from .debugger import Run, kill_game, search_memory
+    tx = ws.texts
+    lang = ws.game.language()
+    menu = [e for e in tx.menu(lang) if e.file.split(".")[0].upper() in menu_files and len(e.value) >= 4]
+    rnd = random.Random(3)
+    picks = rnd.sample(menu, min(n_menu, len(menu)))
+    changes = {"MENU:" + e.key: "ACKIT%02d %s" % (i, e.value) for i, e in enumerate(picks)}
+    subs = [s for s in tx.subtitles(lang) if s.room >= 12] if lang in tx.subtitle_languages() else []
+    startup = [s for s in subs if {l.file for l in tx.subtitle_locations("SUB:%s:%s:%d" % (lang, s.speech, s.line))}
+               == {"Startup.upk"}]
+    for i, s in enumerate(startup):
+        changes["SUB:%s:%s:%d" % (lang, s.speech, s.line)] = ("AKST%02d " % i + s.text)[:s.room]
+    others = [s for s in subs if s not in startup]
+    for i, s in enumerate(rnd.sample(others, min(n_subs, len(others)))):
+        changes["SUB:%s:%s:%d" % (lang, s.speech, s.line)] = ("AKS%02d " % i + s.text)[:s.room]
+    files = sorted({p for k in changes for p in tx.files(k)})
+    logdir = logdir or os.path.join(os.getcwd(), "logs")
+    os.makedirs(logdir, exist_ok=True)
+    logp = os.path.join(logdir, "ingame_texts_%s.log" % time.strftime("%Y%m%d_%H%M%S"))
+    report = dict(mode="texts", lang=lang, menu_texts=len(picks), startup_subtitles=len(startup),
+                  subtitles=len(changes) - len(picks), files=len(files))
+    kill_game()
+    before = {f: _sha1_file(f) for f in files}
+    needles = {("ACKIT%02d " % i).encode("utf-16-le") for i in range(len(picks))}
+    sub_needles = {}                       # marker -> its byte forms (the game may keep cp1252 or UTF-16)
+    for i in range(len(startup)):
+        m = "AKST%02d " % i
+        sub_needles[m] = {m.encode("utf-16-le"), m.encode("cp1252")}
+    needles |= {n for v in sub_needles.values() for n in v}
+    runner = Run(ws.game.exe, logp)
+    found = set()
+    try:                                   # from the first write on: the finally block reverts
+        tx.apply(changes, progress=log)
+        report.update(runner.start())
+        while time.time() - runner.t0 < seconds and runner.alive():
+            time.sleep(0.5)
+            if re.search(r"^ACKIT_CRASH", runner.log(), re.M):
+                break
+        if runner.alive():
+            found = search_memory(report["pid"], needles)
+        report["alive_at_end"] = runner.alive()
+    finally:
+        runner.stop()
+        tx.revert(list(changes), progress=log)
+        report["reverted_byte_identical"] = all(_sha1_file(f) == h for f, h in before.items())
+    report["crash"] = bool(re.search(r"^ACKIT_CRASH", runner.log(), re.M))
+    report["menu_markers_in_memory"] = sum(1 for i in range(len(picks)) if ("ACKIT%02d " % i).encode("utf-16-le") in found)
+    report["startup_subtitles_in_memory"] = sum(1 for v in sub_needles.values() if v & found)
+    ok = (not report["crash"] and report["alive_at_end"] and report["menu_markers_in_memory"] > 0
+          and (not startup or report["startup_subtitles_in_memory"] > 0) and report["reverted_byte_identical"])
+    report["result"] = "PASS" if ok else "FAIL"
+    json.dump(report, open(logp.replace(".log", ".json"), "w"), indent=1)
+    return report
+
+
+# --- in-memory sounds (soundbanks inside the packages) ------------------------------------------
+PROBE_RATE, PROBE_Q = 28000, 4          # a codebook setup the game's own WEMs do not use (E10)
+
+
+def bank_probes(ws, sounds):
+    """[(sound, probe WEM)]: the same probe format for every sound, shortened to fit its slot."""
+    from ..encoder import encode
+    x, sr = probe_signal(10.0)
+    out = []
+    for s in sounds:
+        lim = ws.limit(s)
+        secs = min(10.0, max(0.4, s.seconds or 1.0))
+        while secs >= 0.1:
+            wem, _ = encode(x[:int(secs * sr)], sr, PROBE_RATE, 1, PROBE_Q)
+            if ws.fits(s, wem):                 # within the slot, and able to fill it exactly (wem.fill)
+                out.append((s, wem))
+                break
+            secs *= 0.7 if len(wem) > lim else 0.98
+    return out
+
+
+def run_banks(ws, keys=None, seconds=45.0, logdir=None, log=print):
+    """Replace in-memory sounds (default: every sound stored in the Frontend*/Startup packages) with probes, start the game,
+    then check: no crash, the probe data is in the game's memory (the modified packages were loaded), and
+    after the revert every touched file is byte-identical. Probes decoded (codec setup log) and heard
+    (loopback correlation) are reported too, but no in-memory sound plays on the title screen without input,
+    so they are not required. Returns a report dict (report['result'] PASS/FAIL)."""
+    from .debugger import Run, kill_game, search_memory
+    from .loopback import LoopbackRecorder, detect_chunks
+    from ..wem import Wem
+    if keys:
+        sounds = [ws.catalog.resolve(k) for k in keys]
+    else:  # everything stored in the packages loaded for the title screen (rain ambience plays without input)
+        sounds = [s for s in ws.catalog.sounds if s.archive == "BANK" and any(
+            l.file.lower().startswith(("frontend", "startup")) for l in ws.locations(s))]
+    probes = bank_probes(ws, sounds)
+    files = sorted({f for s, _ in probes for f in ws.files(s)})
+    logdir = logdir or os.path.join(os.getcwd(), "logs")
+    os.makedirs(logdir, exist_ok=True)
+    logp = os.path.join(logdir, "ingame_banks_%s.log" % time.strftime("%Y%m%d_%H%M%S"))
+    report = dict(mode="banks", sounds=len(sounds), probes=len(probes), files=len(files))
+    kill_game()
+    before = {f: _sha1_file(f) for f in files}
+    hashes = {"%08x" % Wem.parse(w).vorb["codebook_hash"] for _, w in probes}
+    needles = set()
+    for _, w in probes:
+        if len(w) > 400:
+            needles.add(w[len(w) // 2:len(w) // 2 + 32])
+    report.update(probe_hashes=sorted(hashes), needles=len(needles))
+    from .debugger import ADDR
+    # also log every setup parse (in-memory sources may not go through the stream setup cache)
+    extra = 'bp %#x ".printf \\"ACKIT_PARSE2 hash=%%08x\\\\n\\", poi(poi(@esp+4)+0x9c); gc"' % ADDR["parse_setup"]
+    runner = Run(ws.game.exe, logp, script_extra=extra)
+    found = set()
+    rec = LoopbackRecorder(seconds + 10)
+    rec.start()
+    try:                                   # from the first write on: the finally block reverts
+        ws.apply([(s, w, {"source": "ingame-test"}) for s, w in probes], progress=log)
+        report.update(runner.start())
+        pid = report["pid"]
+        while time.time() - runner.t0 < seconds and runner.alive():
+            time.sleep(0.5)
+            if re.search(r"^ACKIT_CRASH", runner.log(), re.M):
+                break
+        if runner.alive():
+            found = search_memory(pid, needles)
+        report["alive_at_end"] = runner.alive()
+    finally:
+        runner.stop()
+        rec.stop_flag = True
+        rec.join(5)
+        ws.revert([s for s, _ in probes], progress=log)
+        report["reverted_byte_identical"] = all(_sha1_file(f) == h for f, h in before.items())
+    text = runner.log()
+    report["crash"] = bool(re.search(r"^ACKIT_CRASH", text, re.M))
+    ref = probe_signal(0.35)[0][:, 0]
+    report["capture_error"] = rec.error
+    report["ncc_probe"] = round(detect_chunks(rec.audio(), ref)[0], 3)
+    seen = re.findall(r"ACKIT_(?:SETUP|PARSE2) hash=([0-9a-f]+)", text)
+    report["probe_streams_decoded"] = sum(1 for h in seen if h in hashes)
+    report["setups_logged"] = len(seen)
+    report["probes_in_memory"] = len(found)
+    ok = (not report["crash"] and report["alive_at_end"] and report["probes_in_memory"] > 0
+          and report["reverted_byte_identical"])
+    report["result"] = "PASS" if ok else "FAIL"
+    json.dump(report, open(logp.replace(".log", ".json"), "w"), indent=1)
+    return report

ackit/verify/selfcheck.py

@@ -66,11 +66,23 @@ def run(ws, sample=300, log=print):
                     break
         log("%-4s layout %s, rebuilt %d/%d sample WEMs byte-identical" % (arch, "OK" if not probs else probs[:2], good, n))
         ok &= not probs and good >= n - 2  # 4 looping music tracks are known exceptions
+    bank = [s for s in ws.catalog.sounds if s.archive == "BANK"]
+    if bank:
+        n = good = 0
+        for s in random.Random(1).sample(bank, min(len(bank), sample)):
+            n += 1
+            try:
+                good += rebuild_identical(ws.current_bytes(s))
+            except Exception:
+                pass
+        log("BANK in-memory sounds: rebuilt %d/%d sample WEMs byte-identical" % (good, n))
+        ok &= good == n
     # 3
     t = np.arange(48000 * 2) / 48000.0
     sig = (0.3 * np.sin(2 * np.pi * 330 * t) * np.sin(2 * np.pi * 1.5 * t)).astype(np.float32)[:, None]
     for s in [x for x in ws.catalog.sounds if x.group == "Speech"][:1] + \
-             [x for x in ws.catalog.sounds if x.group == "Music"][:1]:
+             [x for x in ws.catalog.sounds if x.group == "Music"][:1] + \
+             [x for x in ws.catalog.sounds if x.archive == "BANK" and x.seconds >= 1.5][:1]:
         wem, info = ws.prepare(s, sig, 48000)
         y, sr = audio.decode_wem(wem)
         ref = audio.to_channels(audio.resample(sig, 48000, sr), y.shape[1])

ackit/wad.py

@@ -58,12 +58,6 @@ class Wad:
             f.seek(e.offset)
             return f.read(e.size)
 
-    def iter_entries(self):
-        with open(self.path, "rb") as f:
-            for e in self.entries:
-                f.seek(e.offset)
-                yield e, f.read(e.size)
-
     def check(self):
         """Validate structural invariants; returns list of problems (empty = OK)."""
         probs = []

ackit/wem.py

@@ -66,10 +66,6 @@ class Wem:
     def data(self) -> bytes:
         return self.get(b"data")
 
-    def seek_table(self):
-        n = self.vorb["seek_table_size"] // 4
-        return [struct.unpack_from("<HH", self.data, 4 * i) for i in range(n)]
-
     def setup_packet(self) -> bytes:
         d = self.data
         off = self.vorb["seek_table_size"]
@@ -105,6 +101,60 @@ class Wem:
         return b"RIFF" + struct.pack("<I", len(body)) + body
 
 
+# ---- filling an in-memory slot exactly ----
+# The game ships no in-memory WEM with bytes after its RIFF inside its slot, and plays a replacement that is
+# followed by zero padding there as silence (findings 24). A smaller replacement is therefore grown to the
+# slot's exact size with a label list chunk before `data`, the form the game's own WEMs use for markers:
+# LIST{'adtl', labl{u32 cue id 0, zero text}}. Even sizes only (the game's chunks before `data` are even).
+FILL_MIN = 8 + 4 + 8 + 4 + 2
+
+
+def fillable(length, size):
+    """True if a WEM of `length` bytes can fill a slot of `size` bytes exactly."""
+    pad = size - length
+    return pad == 0 or (pad >= FILL_MIN and pad % 2 == 0)
+
+
+def fill(wem: bytes, size: int) -> bytes:
+    """`wem` grown to exactly `size` bytes (see above). ValueError if `fillable` is False."""
+    if not fillable(len(wem), size):
+        raise ValueError("a %d-byte WEM cannot fill a %d-byte slot exactly" % (len(wem), size))
+    pad = size - len(wem)
+    if not pad:
+        return wem
+    i = _data_chunk(wem)
+    text = pad - 8 - 4 - 8 - 4
+    chunk = (b"LIST" + struct.pack("<I", pad - 8) + b"adtl" + b"labl" + struct.pack("<II", text + 4, 0)
+             + bytes(text))
+    out = wem[:i] + chunk + wem[i:]
+    return out[:4] + struct.pack("<I", len(out) - 8) + out[8:]
+
+
+def unfill(wem: bytes) -> bytes:
+    """The WEM without a filler chunk added by `fill` (a game WEM is returned unchanged)."""
+    i = _data_chunk(wem)
+    if i is None:
+        return wem
+    for j in range(12, i, 2):
+        if wem[j:j + 4] == b"LIST" and wem[j + 8:j + 16] == b"adtllabl":
+            n = 8 + struct.unpack_from("<I", wem, j + 4)[0]
+            if j + n == i and struct.unpack_from("<I", wem, j + 20)[0] == 0 and not wem[j + 24:i].strip(b"\0"):
+                out = wem[:j] + wem[i:]
+                return out[:4] + struct.pack("<I", len(out) - 8) + out[8:]
+    return wem
+
+
+def _data_chunk(wem: bytes):
+    """Offset of the `data` chunk header (walking the chunks), or None."""
+    pos = 12
+    while pos + 8 <= len(wem):
+        tag, ln = struct.unpack_from("<4sI", wem, pos)
+        if tag == b"data":
+            return pos
+        pos += 8 + ln
+    return None
+
+
 SUB_ALIGN = 0x800
 _NONZERO = re.compile(rb"[^\x00]")
 

ackit/workspace.py

@@ -6,19 +6,27 @@ Safety rules (docs/decisions.md):
   * each WAD is rewritten atomically; several sounds of one WAD are written in one pass;
   * if a sound in the game is neither its original nor what we applied (Steam update, "verify
     integrity", another tool), we refuse to write it unless forced, and report it.
+In-memory sounds (archive BANK) and the prefetch heads of speech lines live in loose banks and .upk
+packages (ackit/banks.py); a replacement fills each in-memory slot exactly (wem.fill). Before the first
+change to such a file a verified full copy is kept, and the
+file is always rebuilt as "original copy + every replacement currently applied", so reverting the last
+change restores it byte-for-byte and any later operation repairs an interrupted one.
 """
+import hashlib
 import os
 from collections import defaultdict
 
-from . import audio
+from . import audio, banks
 from .catalog import Catalog
-from .encoder import encode_fit
+from .encoder import EncodeError, encode_fit
 from .game import Game, GameError
-from .inject import inject, read_target, parse_target, pack_bnk_path
+from .inject import inject, read_target, parse_target, pack_bnk_path, SizeLimitError
 from .originals import OriginalsStore, sha256, sha1_16
 from .pack import Pack
+from .texts import Texts
 from .paths import user_dir
 from .wad import Wad
+from .wem import FILL_MIN, fill, fillable, unfill
 
 # sound states
 ORIGINAL = "original"      # untouched
@@ -46,6 +54,7 @@ class Workspace:
         self.store = OriginalsStore(store_root or default_store_root(self.game), self.game.root)
         self.catalog = Catalog(self.game, archives, progress)
         self.full_backup = full_backup
+        self.texts = Texts(self)
 
     # reading -----------------------------------------------------------------------------------
     def sound(self, name_or_key):
@@ -54,7 +63,39 @@ class Workspace:
     def wad_path(self, s):
         return self.game.wad(s.archive)
 
+    def bank_path(self, rel):
+        return banks.file_path(self.game.cooked, rel)
+
+    def locations(self, s_or_key):
+        """Bank copies written together with a sound: every copy of a BANK sound, or the prefetch heads
+        of a WAD speech line (usually none)."""
+        key = s_or_key if isinstance(s_or_key, str) else s_or_key.key
+        archive, _, target = key.partition(":")
+        if archive == banks.ARCHIVE:
+            return self.catalog.locations.get(int(target, 16), [])
+        return self.catalog.heads.get(key, [])
+
+    def copies(self, s):
+        """Other places of the same recording in the WAD (written and reverted together with `s`)."""
+        return self.catalog.copies.get(s.key, [])
+
+    def files(self, s):
+        """Absolute paths of the game files that writing `s` changes."""
+        out = []
+        if s.archive != banks.ARCHIVE:
+            out.append(self.wad_path(s))
+            if s.is_pack_media:
+                out.append(pack_bnk_path(self.wad_path(s), parse_target(s.target)[0]))
+        out += sorted({self.bank_path(l.file) for x in [s] + self.copies(s) for l in self.locations(x)})
+        return out
+
     def current_bytes(self, s):
+        if s.archive == banks.ARCHIVE:
+            locs = self.locations(s)
+            if not locs:
+                raise GameError("no copy of %s was found in the game's banks" % s.name)
+            l = locs[0]
+            return unfill(banks.riff_trim(banks.open_file(self.bank_path(l.file)).read(l.offset, l.size)))
         return read_target(self.wad_path(s), s.target)
 
     def original_bytes(self, s):
@@ -81,6 +122,9 @@ class Workspace:
         """Max size in bytes for a replacement (None = any size)."""
         if s.is_pack_media:
             return None
+        if s.archive == banks.ARCHIVE:
+            locs = self.locations(s)
+            return min(l.size for l in locs) if locs else s.size
         e = self.store.get(s.key)
         if e:
             return e["orig_size"]
@@ -91,18 +135,34 @@ class Workspace:
     def original_audio(self, s):
         return audio.decode_wem(self.original_bytes(s))
 
-    def current_audio(self, s):
-        return audio.decode_wem(self.current_bytes(s))
+    def fill_sizes(self, s):
+        """Slot sizes a replacement of `s` must fill exactly: its in-memory copies, and for a WAD recording the
+        bank copies that hold it whole (wem.fill: the game plays a padded in-memory WEM as silence)."""
+        if s.archive == banks.ARCHIVE:
+            return sorted({l.size for l in self.locations(s) if l.kind == banks.MEMORY})
+        orig = self.limit(s)
+        return sorted({l.size for l in self.locations(s) if orig is not None and l.size >= orig})
+
+    def fits(self, s, wem):
+        """True if `wem` can be written for `s`: within its size limit and able to fill its in-memory slots."""
+        lim = self.limit(s)
+        return (lim is None or len(wem) <= lim) and all(fillable(len(wem), n) for n in self.fill_sizes(s))
 
     # encoding ----------------------------------------------------------------------------------
     def prepare(self, s, x, src_rate, quality="auto", progress=None):
-        """Encode PCM for sound `s` (its rate/channels, fitted to its size limit) -> (wem, info)."""
-        return encode_fit(x, src_rate, s.rate or src_rate, s.channels or x.shape[1], self.limit(s),
-                          quality, progress)
-
-    def prepare_file(self, s, path, quality="auto", progress=None):
-        x, sr = audio.load(path)
-        return self.prepare(s, x, sr, quality, progress)
+        """Encode PCM for sound `s` (its rate/channels, fitted to its size limit) -> (wem, info). A sound with
+        in-memory slots gets a WEM that fills them exactly (`fits`): at least FILL_MIN bytes smaller, and an
+        even number of bytes smaller; the end is shortened by a few samples until the size allows that."""
+        sizes = self.fill_sizes(s)
+        lim = self.limit(s)
+        if sizes:
+            lim = min(([lim] if lim is not None else []) + sizes) - FILL_MIN
+        rate, ch = s.rate or src_rate, s.channels or x.shape[1]
+        for k in range(32):
+            wem, info = encode_fit(x[:len(x) - 16 * k] if k else x, src_rate, rate, ch, lim, quality, progress)
+            if self.fits(s, wem):
+                return wem, info
+        raise EncodeError("could not encode a size that fills this sound's slots exactly; shorten the audio")
 
     # writing -----------------------------------------------------------------------------------
     def _backup(self, wad, pack_hashes):
@@ -115,6 +175,11 @@ class Workspace:
 
     def _write(self, archive, items, progress=None):
         """items: [(sound, new_bytes, meta, is_revert)] all in one archive."""
+        if archive == banks.ARCHIVE:
+            return self._write_banks(items, progress)
+        heads = sorted({self.bank_path(l.file) for s, _, _, _ in items for l in self.locations(s)})
+        for path in heads:
+            self._keep_original(path)
         wad = self.game.wad(archive)
         self.game.check_writable(wad)
         packs = {parse_target(s.target)[0] for s, _, _, _ in items if s.is_pack_media}
@@ -140,13 +205,130 @@ class Workspace:
             else:
                 self.store.record_applied(s.key, data, meta)
         self.store.save()
+        self.sync_files(heads, progress)
+
+    # derived files: bank files and packages (in-memory sounds, prefetch heads, subtitles) and the
+    # localization files (menu text). Each is rebuilt from its original copy + every edit applied now.
+    def _derived(self):
+        return self.store.data.setdefault("derived", {})
+
+    def _fingerprint(self, path):
+        """Content hash of a file (whole file: any change by another tool is detected)."""
+        h = hashlib.sha1()
+        with open(path, "rb") as f:
+            for c in iter(lambda: f.read(1 << 22), b""):
+                h.update(c)
+        return h.hexdigest()
+
+    def _keep_original(self, path):
+        """Before the first change to a derived file: verified full copy + its fingerprint."""
+        rel = self.game.rel(path)
+        rec = self._derived().get(rel)
+        if rec:
+            return rec
+        self.game.check_writable(path)
+        self.store.full_backup(path, rel)
+        rec = self._derived()[rel] = {"orig_fp": self._fingerprint(path), "written_fp": None}
+        self.store.save()
+        return rec
+
+    def changed_files(self, paths):
+        """Derived files changed since we last wrote them (Steam update/verify, another tool)."""
+        out = []
+        for path in sorted(set(paths)):
+            rec = self._derived().get(self.game.rel(path))
+            if rec and self._fingerprint(path) not in (rec["orig_fp"], rec["written_fp"]):
+                out.append(path)
+        return out
+
+    def _audio_edits(self, rel):
+        """[(offset, bytes)] that the sound replacements applied now need in bank file `rel` (cooked-relative)."""
+        out = []
+        for key, e in self.store.entries.items():
+            if not e.get("applied"):
+                continue
+            locs = [l for l in self.locations(key) if l.file == rel]
+            if not locs:
+                continue
+            data = unfill(self.store.get_blob(e["applied"]))
+            for l in locs:
+                whole = l.kind == banks.MEMORY or l.size >= e["orig_size"]    # a whole WEM in memory
+                if whole and fillable(len(data), l.size):
+                    payload = fill(data, l.size)
+                elif whole:     # recorded by a version before 3.2.2 (zero padding: the game plays silence)
+                    payload = data + bytes(l.size - len(data))
+                else:           # prefetch head: the start of the WAD recording as it is stored there
+                    full = data + bytes(max(0, e["orig_size"] - len(data)))
+                    payload = (full + bytes(l.size))[:l.size]
+                out.append((l.offset, payload))
+        return out
+
+    def _build(self, path, orig):
+        """Content of derived file `path` from its original copy `orig`."""
+        if self.texts.is_menu_file(path):
+            return self.texts.menu_content(path, open(orig, "rb").read())
+        rel = os.path.relpath(path, self.game.cooked).replace(os.sep, "/")
+        edits = self._audio_edits(rel) + self.texts.subtitle_edits(rel)
+        return banks.open_file(orig).patched(edits) if edits else open(orig, "rb").read()
+
+    def sync_files(self, paths, progress=None):
+        """Rebuild each derived file from its original copy and the edits applied now (atomic writes)."""
+        for path in sorted(set(paths)):
+            rec = self._keep_original(path)
+            orig = self.store.full_copy(self.game.rel(path))
+            if orig is None:
+                raise GameError("the original copy of %s is missing from the store" % self.game.rel(path))
+            if progress:
+                progress("writing %s" % os.path.basename(path))
+            data = self._build(path, orig)
+            self.game.check_writable(path)
+            tmp = path + ".ackit-tmp"
+            with open(tmp, "wb") as f:
+                f.write(data)
+            os.replace(tmp, path)
+            same = data == open(orig, "rb").read()
+            rec["written_fp"] = None if same else self._fingerprint(path)
+        self.store.save()
+
+    def _write_banks(self, items, progress=None):
+        for s, data, meta, is_revert in items:
+            locs = self.locations(s)
+            if not locs:
+                raise GameError("no copy of %s was found in the game's banks" % s.name)
+            if not is_revert and len(data) > min(l.size for l in locs):
+                raise SizeLimitError(s.target, min(l.size for l in locs), len(data))
+        files = sorted({self.bank_path(l.file) for s, _, _, _ in items for l in self.locations(s)})
+        for path in files:
+            self._keep_original(path)
+        for s, data, meta, is_revert in items:
+            if is_revert:
+                self.store.forget(s.key)
+            else:
+                self.store.remember_original(s.key, s.archive, s.target, self.locations(s)[0].file,
+                                             self.current_bytes(s))
+                self.store.record_applied(s.key, data, meta)
+        self.store.save()  # the store says what the files must contain before they change
+        self.sync_files(files, progress)
 
     def apply(self, changes, force=False, progress=None):
-        """changes: [(sound, wem_bytes, meta dict)]. One atomic rewrite per WAD."""
+        """changes: [(sound, wem_bytes, meta dict)]. One atomic rewrite per WAD / bank file. A recording stored
+        in two places of a WAD gets the same bytes in both (Workspace.copies)."""
+        keys = {s.key for s, _, _ in changes}
+        changes = [(s, unfill(data), meta) for s, data, meta in changes]
+        for s, data, _ in changes:
+            if not self.fits(s, data) and len(data) <= (self.limit(s) or len(data)):
+                raise GameError("%s: a %d-byte WEM cannot fill this sound's in-memory slots (%s bytes) exactly; "
+                                "encode it for this sound (Workspace.prepare / the editor / ackit replace)"
+                                % (s.name, len(data), ", ".join(map(str, self.fill_sizes(s)))))
+        changes = list(changes) + [(c, data, meta) for s, data, meta in changes for c in self.copies(s)
+                                   if c.key not in keys]
         conflicts = []
         for s, _, _ in changes:
             if self.state(s) in (CHANGED, FOREIGN):
                 conflicts.append(s)
+        changed = set(self.changed_files([self.bank_path(l.file) for s, _, _ in changes for l in self.locations(s)]))
+        conflicts += [s for s, _, _ in changes
+                      if s not in conflicts and {self.bank_path(l.file) for l in self.locations(s)} & changed]
         if conflicts and not force:
             raise ConflictError(conflicts)
         by_arch = defaultdict(list)
@@ -159,6 +341,8 @@ class Workspace:
         """Put the stored originals back (sounds without a record are skipped). Returns reverted."""
         by_arch = defaultdict(list)
         done = []
+        keys = {s.key for s in sounds}
+        sounds = list(sounds) + [c for s in sounds for c in self.copies(s) if c.key not in keys]
         for s in sounds:
             orig = self.store.original(s.key)
             if orig is None:
@@ -191,6 +375,9 @@ class Workspace:
             shutil.copy2(src, tmp)
             os.replace(tmp, dst)
             out.append(rel)
+        for rec in self._derived().values():
+            rec["written_fp"] = None
+        self.store.data["texts"] = {}
         self.store.data["entries"] = {}
         self.store.save()
         self.store.gc()

app/dialogs.py

@@ -133,7 +133,7 @@ class ExportModDialog(QDialog):
 class ImportModDialog(QDialog):
     """Shows how each item of a package matches this game before installing."""
 
-    def __init__(self, manifest, items, parent=None):
+    def __init__(self, manifest, items, parent=None, texts=()):
         super().__init__(parent)
         self.setWindowTitle("Import mod package")
         lay = QVBoxLayout(self)
@@ -147,13 +147,20 @@ class ImportModDialog(QDialog):
         self.tree.setHeaderLabels(["Sound", "Match", "Note"])
         n_ok = 0
         for im in items:
-            note = {"exact": "same sound", "by name": "matched by name (other language) - will be re-encoded",
+            note = {"exact": "same sound", "same line": "the same line in this language - will be re-encoded",
+                    "by name": "matched by name - will be re-encoded",
                     "missing": im.note}[im.how]
             QTreeWidgetItem(self.tree, [im.item.get("name") or im.item["target"], im.how, note])
             n_ok += im.how != "missing"
+        t_ok = 0
+        for key, _text, problem in texts:
+            QTreeWidgetItem(self.tree, [key, "text", problem or "will be applied"])
+            t_ok += not problem
         self.tree.resizeColumnToContents(0)
         lay.addWidget(self.tree)
-        lay.addWidget(QLabel("%d of %d sounds will be installed." % (n_ok, len(items))))
+        lay.addWidget(QLabel("%d of %d sounds and %d of %d texts will be installed."
+                             % (n_ok, len(items), t_ok, len(texts))))
+        n_ok += t_ok
         bb = QDialogButtonBox(QDialogButtonBox.StandardButton.Ok | QDialogButtonBox.StandardButton.Cancel)
         bb.button(QDialogButtonBox.StandardButton.Ok).setText("Install")
         bb.button(QDialogButtonBox.StandardButton.Ok).setEnabled(n_ok > 0)

app/editor.py

@@ -4,7 +4,7 @@ import os
 import numpy as np
 from PySide6.QtCore import Qt, QTimer, Signal
 from PySide6.QtGui import QFont
-from PySide6.QtWidgets import (QCheckBox, QComboBox, QDoubleSpinBox, QFileDialog, QGridLayout, QGroupBox,
+from PySide6.QtWidgets import (QCheckBox, QComboBox, QDoubleSpinBox, QFileDialog, QGridLayout, QGroupBox, QSplitter,
                                QHBoxLayout, QLabel, QMessageBox, QProgressBar, QPushButton, QSlider,
                                QVBoxLayout, QWidget)
 
@@ -54,15 +54,26 @@ class EditorPanel(QWidget):
             b.setToolTip(tip)
         return b
 
+    @staticmethod
+    def _fill(*items):
+        """A row of buttons filling the column's width: [(widget, share)], equal edges in every column."""
+        row = QHBoxLayout()
+        row.setSpacing(6)
+        for w, share in items:
+            row.addWidget(w, share)
+        return row
+
     def _wave(self, kind, selectable=False):
         w = Waveform(selectable=selectable)
         w.accent, w.bg = theme.WAVE[kind], theme.WAVE_BG
         return w
 
     def _build(self):
+        """Header on top; below, side by side (resizable): LISTEN | REPLACE (with the apply buttons) | SUBTITLE."""
         self.setObjectName("EditorPanel")
         lay = QVBoxLayout(self)
-        lay.setContentsMargins(10, 8, 10, 8)
+        lay.setContentsMargins(8, 6, 8, 6)
+        lay.setSpacing(4)
         self.title = QLabel()
         self.title.setObjectName("EditorTitle")
         self.title.setTextInteractionFlags(Qt.TextInteractionFlag.TextSelectableByMouse)
@@ -71,136 +82,182 @@ class EditorPanel(QWidget):
         f.setBold(True)
         f.setLetterSpacing(QFont.SpacingType.PercentageSpacing, 104)
         self.title.setFont(f)
-        lay.addWidget(self.title)
+        self.state = QLabel()
+        self.state.setWordWrap(True)
+        head = QHBoxLayout()
+        head.addWidget(self.title, 1)
+        head.addWidget(self.state, 0, Qt.AlignmentFlag.AlignRight | Qt.AlignmentFlag.AlignVCenter)
+        lay.addLayout(head)
         self.meta = QLabel()
         self.meta.setWordWrap(True)
         self.meta.setObjectName("Muted")
-        self.state = QLabel()
-        self.state.setWordWrap(True)
         lay.addWidget(self.meta)
-        lay.addWidget(self.state)
 
         # listen ----------------------------------------------------------------------------------
         g = QGroupBox("LISTEN")
         gl = QGridLayout(g)
+        gl.setContentsMargins(8, 8, 8, 8)
+        gl.setVerticalSpacing(4)
         self.wf_orig = self._wave("orig")
         self.wf_inst = self._wave("inst")
         self.b_play_orig = self._btn("Original", "play", lambda: self.play("orig"), "Play the original (Space)", icons.STEEL)
         self.b_play_inst = self._btn("Installed", "play", lambda: self.play("inst"), "Play what the game has now", "#8f4a4e")
         self.b_stop = self._btn("Stop", "stop", self.stop, "Stop (Esc)")
-        self.b_ab = self._btn("A/B", "ab", self.ab_switch, "Switch between original and replacement at the same position",
-                              icons.ACCENT)
-        self.cb_loop = self._btn("Loop", "loop", None, "Repeat playback")
+        # playback options next to the volume: icons with tooltips
+        self.b_ab = self._btn("", "ab", self.ab_switch, "A/B: switch between the original and the replacement at "
+                              "the same position", icons.ACCENT)
+        self.cb_loop = self._btn("", "loop", None, "Loop: repeat playback")
         self.cb_loop.setCheckable(True)
-        self.vol_icon = QLabel()
-        self.vol_icon.setPixmap(icons.icon("play", icons.MUTED).pixmap(14, 14))
+        for b in (self.b_ab, self.cb_loop):
+            b.setFixedWidth(b.sizeHint().height() + 4)
         self.sl_vol = QSlider(Qt.Orientation.Horizontal)
         self.sl_vol.setRange(0, 100)
         self.sl_vol.setValue(80)
-        self.sl_vol.setMaximumWidth(120)
+        self.sl_vol.setMinimumWidth(80)
         self.sl_vol.setToolTip("Volume")
         self.sl_vol.valueChanged.connect(lambda v: self.ctx.player.set_volume(v / 100.0))
         self.lbl_time = QLabel("0:00.00")
         self.lbl_time.setObjectName("Muted")
+        self.lbl_time.setAlignment(Qt.AlignmentFlag.AlignRight | Qt.AlignmentFlag.AlignVCenter)
+        self.lbl_time.setMinimumWidth(self.lbl_time.fontMetrics().horizontalAdvance("00:00.00") + 4)
         gl.addWidget(QLabel("Original"), 0, 0)
-        gl.addWidget(self.wf_orig, 0, 1, 1, 6)
+        gl.addWidget(self.wf_orig, 0, 1)
         self.lbl_inst = QLabel("Installed")
         gl.addWidget(self.lbl_inst, 1, 0)
-        gl.addWidget(self.wf_inst, 1, 1, 1, 6)
-        row = QHBoxLayout()
-        for w in (self.b_play_orig, self.b_play_inst, self.b_stop, self.b_ab, self.cb_loop, QLabel("  Volume"),
-                  self.sl_vol, self.lbl_time):
-            row.addWidget(w)
-        row.addStretch(1)
-        gl.addLayout(row, 2, 0, 1, 7)
+        gl.addWidget(self.wf_inst, 1, 1)
+        gl.setRowStretch(2, 1)                 # controls at the bottom, level with the other columns' actions
+        gl.addLayout(self._fill((self.b_play_orig, 1), (self.b_play_inst, 1), (self.b_stop, 1)), 4, 0, 1, 2)
+        vol = QHBoxLayout()                    # playback options: A/B, loop, volume, position
+        vol.setSpacing(6)
+        vol.addWidget(self.b_ab)
+        vol.addWidget(self.cb_loop)
+        vol.addSpacing(6)
+        vol.addWidget(QLabel("Volume"))
+        vol.addWidget(self.sl_vol, 1)
+        vol.addWidget(self.lbl_time)
+        gl.addLayout(vol, 3, 0, 1, 2)
         gl.setColumnStretch(1, 1)
-        lay.addWidget(g)
         self.wf_orig.seekRequested.connect(lambda t: self.play("orig", t))
         self.wf_inst.seekRequested.connect(lambda t: self.play("inst", t))
+        listen = g
 
         # replace ---------------------------------------------------------------------------------
         g = QGroupBox("REPLACE")
         rl = QVBoxLayout(g)
+        rl.setContentsMargins(8, 8, 8, 8)
+        rl.setSpacing(6)
         src = QHBoxLayout()
+        src.setSpacing(6)
         self.b_load = self._btn("Load file…", "open", self.load_file_dialog,
                                 "WAV, FLAC, OGG, OPUS, MP3, AIFF (M4A/WMA need ffmpeg installed)  (Ctrl+O)")
         self.cmb_in = QComboBox()
-        self.cmb_in.setMinimumWidth(180)
+        self.cmb_in.setMinimumWidth(110)
         self.cmb_in.setToolTip("Recording device")
         self.b_rec = self._btn("Record", "record", self.toggle_record, "Record from the selected input")
         self.meter = QProgressBar()
         self.meter.setRange(0, 100)
         self.meter.setTextVisible(False)
-        self.meter.setMaximumWidth(120)
+        self.meter.setFixedWidth(60)
         self.meter.setToolTip("Input level")
         self.lbl_rec = QLabel("")
         self.lbl_rec.setObjectName("Muted")
-        for w in (self.b_load, QLabel("  or"), self.cmb_in, self.b_rec, self.meter, self.lbl_rec):
+        src.addWidget(self.b_load, 1)          # the row fills the column: file | microphone, level
+        src.addSpacing(10)
+        src.addWidget(self.cmb_in, 2)
+        src.addWidget(self.b_rec, 1)
+        for w in (self.meter, self.lbl_rec):
             src.addWidget(w)
-        src.addStretch(1)
         rl.addLayout(src)
         self.wf_new = self._wave("new", selectable=True)
         self.wf_new.setToolTip("Click: play from here. Right-drag (or Shift+drag): select the part to keep.")
         self.wf_new.seekRequested.connect(lambda t: self.play("new", t))
         self.wf_new.selectionChanged.connect(self._selection)
         rl.addWidget(self.wf_new)
-        proc = QHBoxLayout()
         self.sp_a = QDoubleSpinBox()
         self.sp_b = QDoubleSpinBox()
         for s_ in (self.sp_a, self.sp_b):
             s_.setDecimals(2)
             s_.setSuffix(" s")
             s_.setRange(0, 36000)
+            s_.setMaximumWidth(80)
             s_.valueChanged.connect(self._changed)
         self.sp_gain = QDoubleSpinBox()
         self.sp_gain.setRange(-40, 40)
         self.sp_gain.setSuffix(" dB")
+        self.sp_gain.setMaximumWidth(80)
         self.sp_gain.valueChanged.connect(self._changed)
         self.cb_norm = QCheckBox("Normalize")
-        self.cb_match = QCheckBox("Match original loudness")
+        self.cb_match = QCheckBox("Match loudness")
+        self.cb_match.setToolTip("Match the loudness of the original sound")
         self.cb_norm.toggled.connect(self._changed)
         self.cb_match.toggled.connect(self._changed)
         self.b_silence = self._btn("Trim silence", None, self.trim_silence, "Cut silence at the start and end")
         self.b_play_new = self._btn("Replacement", "play", lambda: self.play("new"), "Play the processed replacement",
                                     icons.ACCENT)
-        for w in (QLabel("Keep from"), self.sp_a, QLabel("to"), self.sp_b, self.b_silence, QLabel("  Gain"),
-                  self.sp_gain, self.cb_norm, self.cb_match, self.b_play_new):
-            proc.addWidget(w)
-        proc.addStretch(1)
+        # edit controls in a grid: labels in one column, the fields aligned, playback on the right
+        proc = QGridLayout()
+        proc.setHorizontalSpacing(6)
+        proc.setVerticalSpacing(6)
+        proc.addWidget(QLabel("Keep"), 0, 0)
+        proc.addWidget(self.sp_a, 0, 1)
+        proc.addWidget(QLabel("to"), 0, 2, Qt.AlignmentFlag.AlignCenter)
+        proc.addWidget(self.sp_b, 0, 3)
+        proc.addLayout(self._fill((self.b_silence, 1), (self.b_play_new, 1)), 0, 4)   # the rest of the width
+        proc.addWidget(QLabel("Gain"), 1, 0)
+        proc.addWidget(self.sp_gain, 1, 1)
+        checks = QHBoxLayout()
+        checks.setSpacing(12)
+        checks.addWidget(self.cb_norm)
+        checks.addWidget(self.cb_match)
+        checks.addStretch(1)
+        proc.addLayout(checks, 1, 2, 1, 3)
+        proc.setColumnStretch(4, 1)
         rl.addLayout(proc)
         info = QHBoxLayout()
         self.lbl_new = QLabel()
-        self.lbl_new.setWordWrap(True)
         self.lbl_new.setObjectName("Muted")
+        self.lbl_new.setWordWrap(True)
         self.warn_icon = QLabel()
         self.warn_icon.setPixmap(icons.icon("warning", icons.ACCENT).pixmap(18, 18))
         self.warn_icon.hide()
         self.lbl_warn = QLabel()
         self.lbl_warn.setWordWrap(True)
         self.lbl_warn.setStyleSheet("color: %s;" % icons.ACCENT)
-        self.lbl_new.setWordWrap(False)
-        info.addWidget(self.lbl_new, 0, Qt.AlignmentFlag.AlignTop)
-        info.addSpacing(12)
+        info.addWidget(self.lbl_new, 1, Qt.AlignmentFlag.AlignTop)
         info.addWidget(self.warn_icon, 0, Qt.AlignmentFlag.AlignTop)
         info.addWidget(self.lbl_warn, 1)
         rl.addLayout(info)
-        lay.addWidget(g)
-
-        # apply -----------------------------------------------------------------------------------
-        act = QHBoxLayout()
+        # apply (belongs to the replacement) ------------------------------------------------------
         self.b_apply = self._btn("Apply to game", "apply", self.apply,
                                  "Encode the replacement and install it in the game files (Ctrl+S)", "#ffffff")
         self.b_apply.setObjectName("Primary")
-        self.b_revert = self._btn("Revert to original", "revert", self.revert, "Put the game's original back")
-        self.b_export = self._btn("Save as WAV…", "save", self.export_wav,
+        self.b_revert = self._btn("Revert", "revert", self.revert, "Put the game's original back")
+        self.b_export = self._btn("Save WAV…", "save", self.export_wav,
                                   "Save the original (or the replacement) as a WAV file")
         self.busy = Spinner(16)
         self.lbl_busy = QLabel()
-        for w in (self.b_apply, self.b_revert, self.b_export, self.busy, self.lbl_busy):
-            act.addWidget(w)
-        act.addStretch(1)
-        lay.addLayout(act)
-        lay.addStretch(1)
+        info.addWidget(self.busy, 0, Qt.AlignmentFlag.AlignTop)      # encoding/writing: next to the status
+        info.addWidget(self.lbl_busy, 0, Qt.AlignmentFlag.AlignTop)
+        act = self._fill((self.b_apply, 2), (self.b_revert, 1), (self.b_export, 1))
+        rl.addStretch(1)
+        rl.addLayout(act)
+        replace = g
+
+        # subtitle of this sound (edited side by side with the audio) -------------------------------
+        from .textpanel import SubtitleBox
+        gs = QGroupBox("SUBTITLE")
+        gsl = QVBoxLayout(gs)
+        gsl.setContentsMargins(8, 8, 8, 8)
+        self.subtitle = SubtitleBox(self.ctx, gs)
+        gsl.addWidget(self.subtitle, 1)
+
+        self.columns = QSplitter(Qt.Orientation.Horizontal)
+        self.columns.setChildrenCollapsible(False)
+        self.columns.setHandleWidth(4)
+        for w, stretch in ((listen, 3), (replace, 4), (gs, 3)):
+            self.columns.addWidget(w)
+            self.columns.setStretchFactor(self.columns.count() - 1, stretch)
+        lay.addWidget(self.columns, 1)
 
         self.recorder.level.connect(lambda v: self.meter.setValue(int(min(1.0, v) * 100)))
         self.recorder.elapsed.connect(lambda t: self.lbl_rec.setText("recording %s" % fmt_s(t)))
@@ -252,6 +309,7 @@ class EditorPanel(QWidget):
         self.wf_inst.set_audio(None, 1, "")
         self.wf_new.set_audio(None, 1, "load a file or record a replacement")
         self.lbl_new.setText("")
+        self.subtitle.set_sound(s)
         if s is None:
             self.title.setText("No sound selected")
             self.meta.setText("Search and select a sound in the list.")
@@ -260,10 +318,14 @@ class EditorPanel(QWidget):
         self.title.setText(s.name)
         who = s.character or (("mentions " + s.characters_inferred) if s.characters_inferred else "—")
         lim = self.ctx.ws.limit(s)
-        self.meta.setText("%s / %s   •   %s   •   %.2f s, %d Hz, %s   •   max size: %s   •   %s"
+        where = ""
+        if s.bank:
+            n = len(self.ctx.ws.locations(s))
+            where = "   •   bank %s (%d %s)" % (s.bank, n, "copy" if n == 1 else "copies")
+        self.meta.setText("%s / %s   •   %s   •   %.2f s, %d Hz, %s   •   max size: %s   •   %s%s"
                           % (s.group, s.category, who, s.seconds, s.rate,
                              {1: "mono", 2: "stereo"}.get(s.channels, "%d ch" % s.channels),
-                             "any" if lim is None else "%.1f KB" % (lim / 1024.0), s.key))
+                             "any" if lim is None else "%.1f KB" % (lim / 1024.0), s.key, where))
         key = s.key
         ws = self.ctx.ws
 

app/main.py

@@ -50,6 +50,13 @@ def smoke_test(game_dir=None, report=None):
         assert read_target(out, s.target) == wem
         print("pipeline OK: %s (%s), %d sounds loaded, re-encoded %d bytes (limit %d), injected into a temp copy"
               % (s.name, s.archive, len(ws.catalog.sounds), len(wem), ws.limit(s)))
+        lang = ws.game.language()
+        b = next((x for x in ws.catalog.sounds if x.archive == "BANK"), None)
+        if b:
+            audio.decode_wem(ws.current_bytes(b))
+        subs = ws.texts.subtitles(lang) if lang in ws.texts.subtitle_languages() else []
+        print("texts OK: %d subtitles, %d menu texts (%s); in-memory sound decoded: %s"
+              % (len(subs), len(ws.texts.menu(lang)), lang, b.name if b else "none"))
     return 0
 
 

app/mainwindow.py

@@ -1,21 +1,26 @@
-"""Main window: filters + sound table (browser) on top, single-sound editor below."""
+"""Main window. Tabs: Sounds (filters + sound table on top, single-sound editor below, with the sound's
+subtitle), Subtitles and Menu text (app/textpanel.py)."""
 import os
 
 from PySide6.QtCore import QByteArray, QEvent, QItemSelectionModel, QSize, Qt, QTimer
 from PySide6.QtGui import QFont, QKeySequence, QShortcut
 from PySide6.QtWidgets import (QAbstractItemView, QCheckBox, QComboBox, QDialog, QFileDialog, QFormLayout,
                                QHBoxLayout, QHeaderView, QLabel, QLineEdit, QMainWindow, QMenu, QMessageBox,
-                               QPushButton, QScrollArea, QSplitter, QTableView, QToolBar, QVBoxLayout, QWidget)
+                               QPushButton, QScrollArea, QSplitter, QTableView, QTabWidget, QToolBar, QVBoxLayout,
+                               QWidget)
 
 from ackit import settings as settings_mod, workspace as W
+from ackit.catalog import available_archives
 from ackit.game import Game, GameError, default_game_dir
-from ackit.modpack import export_mod, match_mod, prepare_import
+from ackit.modpack import export_mod, match_mod, match_texts, prepare_import
+from ackit.texts import TextConflictError
 from . import icons, theme
 from .audioio import Player, find_device, output_devices
 from .dialogs import ExportModDialog, ImportModDialog, SettingsDialog, pick_game_dir
 from .editor import EditorPanel
 from .models import C_FAV, C_NAME, COLS, SoundFilter, SoundModel
-from .spinner import Spinner
+from .spinner import BusyOverlay, Spinner
+from .textpanel import TextPanel
 from .userdata import UserData
 from .workers import Runner
 
@@ -25,6 +30,19 @@ TABLE_MIN = 160                     # the sound list never gets smaller than thi
 ANY = "Any"
 
 
+def all_languages(ws):
+    """Every language the editor offers: speech WADs + subtitle and menu text languages."""
+    return sorted(set(ws.game.languages) | set(ws.texts.menu_languages()) | set(ws.texts.subtitle_languages()))
+
+
+def editor_languages(game, language=None):
+    """(language, voices): the selected language (default: the game's) and the speech archive used for it
+    (the language itself, or INT for languages the game has no voices for)."""
+    lang = (language or game.language() or "INT").upper()
+    voices = lang if lang in game.languages else ("INT" if "INT" in game.languages else game.languages[0])
+    return lang, voices
+
+
 class MainWindow(QMainWindow):
     def __init__(self, settings=None, runner=None, game_dir=None):
         super().__init__()
@@ -35,6 +53,7 @@ class MainWindow(QMainWindow):
         self.player.error.connect(lambda m: self.statusBar().showMessage(m, 5000))
         self.userdata = UserData()
         self.ws = None
+        self.language = None              # the editor's language (voices, subtitles, menu text)
         self.loading = False
         self._build()
         self._apply_output_device()
@@ -45,7 +64,7 @@ class MainWindow(QMainWindow):
             self.resize(1400, 900)
         gd = game_dir or default_game_dir(self.settings)
         if gd:
-            QTimer.singleShot(0, lambda: self.load_game(gd, self.settings.get("language")))
+            QTimer.singleShot(0, lambda: self.load_game(gd))     # always starts in the game's language
         else:
             self.show_no_game()
 
@@ -69,11 +88,16 @@ class MainWindow(QMainWindow):
         tb.addWidget(title)
         tb.addSeparator()
         self.cmb_lang = QComboBox()
-        self.cmb_lang.setToolTip("Speech language archive to browse")
+        self.cmb_lang.setToolTip("The one language setting of the editor: the voice lines in the Sounds tab, the "
+                                 "subtitles and the menu text.\nLanguages without their own voices (e.g. JPN, RUS) "
+                                 "use the English (INT) voice lines, like the game.")
         self.cmb_lang.activated.connect(lambda _: self.load_game(self.ws.game.root, self.cmb_lang.currentText())
                                         if self.ws else None)
         tb.addWidget(QLabel(" Language "))
         tb.addWidget(self.cmb_lang)
+        self.lbl_voices = QLabel()
+        self.lbl_voices.setObjectName("Muted")
+        tb.addWidget(self.lbl_voices)
         tb.addSeparator()
         self.act_revert_all = tb.addAction(icons.icon("revert"), "Revert all", self.revert_all)
         self.act_revert_all.setToolTip("Put every original sound back")
@@ -157,6 +181,7 @@ class MainWindow(QMainWindow):
         self.proxy.setSourceModel(self.model)
         self.table = QTableView()
         self.table.setModel(self.proxy)
+        self.table_busy = BusyOverlay(self.table)   # while the sound list loads (start, language change)
         self.table.setSortingEnabled(True)
         self.table.setSelectionBehavior(QAbstractItemView.SelectionBehavior.SelectRows)
         self.table.setSelectionMode(QAbstractItemView.SelectionMode.ExtendedSelection)
@@ -185,8 +210,14 @@ class MainWindow(QMainWindow):
         scroll.setWidget(self.editor)
         scroll.setWidgetResizable(True)
 
+        # sidebar | sound list on top, the editor below across the whole width
+        hs = QSplitter(Qt.Orientation.Horizontal)
+        hs.addWidget(side)
+        hs.addWidget(self.table)
+        hs.setStretchFactor(1, 1)
+        hs.setSizes([SIDEBAR_MAX, 100000])
         self.vsplit = QSplitter(Qt.Orientation.Vertical)
-        self.vsplit.addWidget(self.table)
+        self.vsplit.addWidget(hs)
         self.vsplit.addWidget(scroll)
         self.vsplit.setStretchFactor(0, 1)       # extra height goes to the list; the editor keeps its fitted height
         self.vsplit.setStretchFactor(1, 0)
@@ -195,17 +226,28 @@ class MainWindow(QMainWindow):
         self._user_sized = False                  # once the divider is dragged, keep the user's choice
         self.vsplit.splitterMoved.connect(lambda *_: setattr(self, "_user_sized", True))
         self.editor.installEventFilter(self)
-        hs = QSplitter(Qt.Orientation.Horizontal)
-        hs.addWidget(side)
-        hs.addWidget(self.vsplit)
-        hs.setStretchFactor(1, 1)
-        hs.setSizes([SIDEBAR_MAX, 100000])
+        # sections: sounds, subtitles, menu text
+        self.tabs = QTabWidget()
+        self.tabs.setDocumentMode(True)
+        self.tabs.addTab(self.vsplit, icons.icon("wave"), "Sounds")
+        self.subs = TextPanel(self, "subs")
+        self.menu = TextPanel(self, "menu")
+        self.tabs.addTab(self.subs, icons.icon("ab"), "Subtitles")
+        self.tabs.addTab(self.menu, icons.icon("settings"), "Menu text")
+        self._text_loaded = set()
+        for panel in (self.subs, self.menu):
+            panel.message.connect(lambda m: self.statusBar().showMessage(m, 8000))
+            panel.changed.connect(self._texts_changed)
+        self.subs.go_to_sound.connect(self.show_sound)
+        self.editor.subtitle.changed.connect(self._texts_changed)
+        self.editor.subtitle.open_in_list.connect(self.show_subtitle)
+        self.tabs.currentChanged.connect(self._tab_changed)
         central = QWidget()
         cl = QVBoxLayout(central)
         cl.setContentsMargins(0, 0, 0, 0)
         cl.setSpacing(0)
         cl.addWidget(self.banner)
-        cl.addWidget(hs, 1)
+        cl.addWidget(self.tabs, 1)
         self.setCentralWidget(central)
         self.spinner = Spinner(14)
         self.statusBar().addWidget(self.spinner)
@@ -249,30 +291,36 @@ class MainWindow(QMainWindow):
         except GameError as e:
             self.show_no_game(str(e))
             return
-        lang = language if language in game.languages else game.language()
+        lang, voices = editor_languages(game, language)
         self.loading = True
         self.statusBar().showMessage("Loading sound list…")
         self.spinner.show()
+        self.table_busy.start("Loading sounds…")
         store = self.settings.get("store_root")
         full = self.settings.get("full_backup", True)
 
         def work(job):
-            return W.Workspace(game, store, [lang, "SFX"] if "SFX" in game.archives else [lang], full_backup=full,
-                               progress=lambda n, t: job.report("scanning %s: %d / %d" % (lang, n, t)))
+            avail = available_archives(game)
+            return W.Workspace(game, store, [a for a in (voices, "SFX", "BANK") if a in avail], full_backup=full,
+                               progress=lambda n, t: job.report("scanning game audio: %d / %d" % (n, t)))
 
         def done(ws):
             self.loading = False
             self.spinner.hide()
+            self.table_busy.stop()
             self.ws = ws
             self.settings["game_dir"] = game.root
-            self.settings["language"] = lang
             self.save_settings()
+            self.language = lang
             self.cmb_lang.blockSignals(True)
             self.cmb_lang.clear()
-            self.cmb_lang.addItems(game.languages)
+            self.cmb_lang.addItems(all_languages(ws))
             self.cmb_lang.setCurrentText(lang)
             self.cmb_lang.blockSignals(False)
+            self.lbl_voices.setText("" if voices == lang else "  voices: %s (no %s voices in the game)" % (voices, lang))
             self.populate()
+            self._text_loaded.clear()
+            self._tab_changed(self.tabs.currentIndex())
             self.banner.hide()
             self.lbl_game.setText("%s  •  store: %s" % (game.root, ws.store.root))
             self.statusBar().showMessage("Loaded %d sounds (%s)." % (len(ws.catalog.sounds), ", ".join(ws.catalog.archives)), 6000)
@@ -282,6 +330,7 @@ class MainWindow(QMainWindow):
         def failed(e, tb):
             self.loading = False
             self.spinner.hide()
+            self.table_busy.stop()
             self.show_no_game("Could not load the game audio: %s" % e)
 
         self.runner.start(work, done, failed, lambda m: self.statusBar().showMessage(m))
@@ -432,10 +481,42 @@ class MainWindow(QMainWindow):
         self.cmb_tag.setCurrentText(cur)
         self.cmb_tag.blockSignals(False)
 
+    # --- text sections --------------------------------------------------------------------------
+    def _tab_changed(self, i):
+        panel = self.tabs.widget(i)
+        if self.ws and panel in (self.subs, self.menu) and panel.kind not in self._text_loaded:
+            self._text_loaded.add(panel.kind)
+            panel.load()
+
+    def _texts_changed(self):
+        for panel in (self.subs, self.menu):
+            if panel.kind in self._text_loaded:
+                panel.refresh()
+        self.editor.subtitle.set_sound(self.editor.sound)
+
+    def show_sound(self, key):
+        """Sounds tab with `key` selected (from a subtitle)."""
+        s = self.ws.catalog.by_key.get(key) if self.ws else None
+        if not s:
+            self.inform("Sound", "%s is not in the sounds loaded now (choose its language at the top)." % key)
+            return
+        self.tabs.setCurrentIndex(0)
+        self.clear_filters()
+        self.select_name(s.name)
+
+    def show_subtitle(self, key):
+        """Subtitles tab with `key` selected (from the sound editor)."""
+        self.tabs.setCurrentWidget(self.subs)
+        self.subs.search.clear()
+        self.subs.cb_mod.setChecked(False)
+        for cb in self.subs.filters.values():
+            cb.setCurrentIndex(0)
+        self.subs._filter()
+        self.subs.select_key(key)
+
     # --- changes ----------------------------------------------------------------------------------
     def _after_change(self, key=None):
-        self.model.refresh_keys(modified=self.ws.modified_keys())
-        self.proxy.refilter()
+        self.model.refresh_keys(modified=self.ws.modified_keys())     # the view refilters/re-sorts itself
         self._update_count()
 
     def revert_sounds(self, sounds):
@@ -466,6 +547,7 @@ class MainWindow(QMainWindow):
             self.spinner.hide()
             self.statusBar().showMessage(ok_msg, 8000)
             self._after_change()
+            self._texts_changed()
             if self.editor.sound:
                 self.editor.set_sound(self.editor.sound)
             self.check_problems()
@@ -491,11 +573,20 @@ class MainWindow(QMainWindow):
                 st = ws.state(s)
                 if st in (W.LOST, W.CHANGED):
                     out.append((s, st))
-            return out
+            texts = []
+            for key in list(ws.texts.records()):
+                try:
+                    st = ws.texts.state(key)
+                except (GameError, OSError, ValueError):
+                    continue
+                if st in (W.LOST, W.CHANGED):
+                    texts.append((key, st))
+            return out, texts
 
-        def done(problems):
-            self.problems = problems
-            if not problems:
+        def done(res):
+            problems, texts = res
+            self.problems, self.text_problems = problems, texts
+            if not problems and not texts:
                 if self.banner.isVisible() and not self.b_choose.isVisible():
                     self.banner.hide()
                 return
@@ -508,6 +599,9 @@ class MainWindow(QMainWindow):
             if changed:
                 txt.append("%d sound(s) were changed by something else since you modified them: %s. "
                            "Revert puts the original back." % (len(changed), ", ".join(changed[:4])))
+            if texts:
+                txt.append("%d of your text changes were undone or changed outside this editor: %s"
+                           % (len(texts), ", ".join(k for k, _ in texts[:3]) + (" …" if len(texts) > 3 else "")))
             self.banner_text.setText("\n".join(txt))
             self.b_reapply.show()
             self.b_forget.show()
@@ -520,15 +614,24 @@ class MainWindow(QMainWindow):
         ws = self.ws
         items = [(s, ws.store.get_blob(ws.store.get(s.key)["applied"]), ws.store.get(s.key).get("meta", {}))
                  for s, st in getattr(self, "problems", []) if ws.store.get(s.key) and ws.store.get(s.key).get("applied")]
-        if items:
-            self._run_change("Re-applying…", lambda job: ws.apply(items, force=True, progress=job.report),
-                             "Re-applied %d change(s)." % len(items))
+        texts = {k: ws.texts.records()[k]["applied"] for k, st in getattr(self, "text_problems", [])
+                 if k in ws.texts.records() and ws.texts.records()[k].get("applied") is not None}
+
+        def work(job):
+            if items:
+                ws.apply(items, force=True, progress=job.report)
+            if texts:
+                ws.texts.apply(texts, force=True, progress=job.report)
+        if items or texts:
+            self._run_change("Re-applying…", work, "Re-applied %d change(s)." % (len(items) + len(texts)))
 
     def forget_problems(self):
-        if not self.confirm("Forget", "Stop tracking these sounds? Their current game files are kept as they are."):
+        if not self.confirm("Forget", "Stop tracking these changes? The current game files are kept as they are."):
             return
         for s, _ in getattr(self, "problems", []):
             self.ws.store.forget(s.key)
+        for k, _ in getattr(self, "text_problems", []):
+            self.ws.texts.records().pop(k, None)
         self.ws.store.save()
         self.ws.store.gc()
         self.banner.hide()
@@ -539,11 +642,13 @@ class MainWindow(QMainWindow):
         if not self.ws:
             return
         mods = self.ws.modified_keys()
-        if not mods:
-            self.inform("Export mod", "You have not changed any sound yet.")
+        texts = self.ws.texts.modified()
+        if not mods and not texts:
+            self.inform("Export mod", "You have not changed any sound or text yet.")
             return
         if keys is None:
             names = [(k, self.ws.catalog.by_key[k].name if k in self.ws.catalog.by_key else k) for k in mods]
+            names += [(k, "Text  " + k) for k in texts]
             dlg = ExportModDialog(names, self)
             if dlg.exec() != QDialog.DialogCode.Accepted:
                 return
@@ -553,8 +658,10 @@ class MainWindow(QMainWindow):
                 self.settings.get("last_dir") or "", (title or "my_mod") + ".acmod"), "Unheard City mod (*.acmod)")
             if not path:
                 return
-        self._run_change("Exporting…", lambda job: export_mod(self.ws, path, keys, title, author, description),
-                         "Saved %s" % path)
+        text_keys = [k for k in keys if k.startswith(("SUB:", "MENU:"))]
+        keys = [k for k in keys if k not in text_keys]
+        self._run_change("Exporting…", lambda job: export_mod(self.ws, path, keys, title, author, description,
+                                                               texts=text_keys), "Saved %s" % path)
 
     def import_mod_dialog(self, path=None, confirm=True):
         if not self.ws:
@@ -569,18 +676,27 @@ class MainWindow(QMainWindow):
         except Exception as e:  # noqa: BLE001
             self.inform("Import mod", "Cannot read this package: %s" % e)
             return
-        if confirm and ImportModDialog(man, items, self).exec() != QDialog.DialogCode.Accepted:
+        dlg = ImportModDialog(man, items, self, match_texts(self.ws, man)) if confirm else None
+        if dlg and dlg.exec() != QDialog.DialogCode.Accepted:
             return
         ws = self.ws
 
         def work(job):
             prepare_import(ws, path, items, progress=job.report)
-            changes = [(im.sound, im.wem, {"source": "mod:" + os.path.basename(path)}) for im in items if im.wem]
-            try:
-                ws.apply(changes, progress=job.report)
-            except W.ConflictError:
-                ws.apply(changes, force=True, progress=job.report)
-            return len(changes)
+            source = {"source": "mod:" + os.path.basename(path)}
+            changes = [(im.sound, im.wem, source) for im in items if im.wem]
+            if changes:
+                try:
+                    ws.apply(changes, progress=job.report)
+                except W.ConflictError:
+                    ws.apply(changes, force=True, progress=job.report)
+            texts = {k: v for k, v, problem in match_texts(ws, man) if not problem}
+            if texts:
+                try:
+                    ws.texts.apply(texts, progress=job.report, meta=source)
+                except TextConflictError:
+                    ws.texts.apply(texts, force=True, progress=job.report, meta=source)
+            return len(changes) + len(texts)
 
         self._run_change("Installing mod…", work, "Installed %s" % os.path.basename(path))
 
@@ -597,7 +713,7 @@ class MainWindow(QMainWindow):
         if (self.settings.get("game_dir") != old.get("game_dir") or self.settings.get("store_root") != old.get("store_root")
                 or self.settings.get("full_backup") != old.get("full_backup")):
             if self.settings.get("game_dir"):
-                self.load_game(self.settings["game_dir"], self.settings.get("language"))
+                self.load_game(self.settings["game_dir"], self.language)
 
     def _apply_output_device(self):
         name = self.settings.get("output_device")

app/models.py

@@ -1,5 +1,5 @@
 """Table model for the sound browser + a combined filter (search text and facet filters)."""
-from PySide6.QtCore import QAbstractTableModel, QModelIndex, QSortFilterProxyModel, Qt
+from PySide6.QtCore import QAbstractProxyModel, QAbstractTableModel, QModelIndex, Qt
 from PySide6.QtGui import QFont, QPalette
 from PySide6.QtWidgets import QApplication
 
@@ -40,6 +40,44 @@ class SoundModel(QAbstractTableModel):
         if self.sounds:
             self.dataChanged.emit(self.index(0, 0), self.index(len(self.sounds) - 1, len(COLS) - 1))
 
+    def sort_key(self, row, c):
+        """Value a column sorts by (plain Python values: the view sorts all rows in one Python sort)."""
+        s, ud = self.sounds[row], self.userdata
+        if c == C_FAV:
+            return 0 if ud and ud.favorite(s.key) else 1
+        if c == C_LEN:
+            return s.seconds
+        if c == C_SIZE:
+            return 1e12 if s.is_pack_media else s.size
+        if c == C_FMT:
+            return (s.rate, s.channels)
+        if c == C_STATUS:
+            return 0 if s.key in self.modified else 1
+        return self.display(s, c).lower()
+
+    def display(self, s, c):
+        ud = self.userdata
+        if c == C_NAME:
+            return s.name
+        if c == C_CAT:
+            return s.category
+        if c == C_CHAR:
+            return s.character or (("~ " + s.characters_inferred) if s.characters_inferred else "")
+        if c == C_LEN:
+            return "%.2f s" % s.seconds
+        if c == C_ARCH:
+            return s.archive
+        if c == C_FMT:
+            return "%d Hz %s" % (s.rate, {1: "mono", 2: "stereo"}.get(s.channels, "%d ch" % s.channels))
+        if c == C_SIZE:
+            return "any" if s.is_pack_media else "%.1f KB" % (s.size / 1024.0)
+        if c == C_STATUS:
+            return "Modified" if s.key in self.modified else ""
+        if c == C_NOTES and ud:
+            t = ud.tags(s.key)
+            return " ".join("#" + x for x in t) + ("  " if t else "") + ud.note(s.key)
+        return ""
+
     def rowCount(self, parent=QModelIndex()):
         return 0 if parent.isValid() else len(self.sounds)
 
@@ -66,41 +104,9 @@ class SoundModel(QAbstractTableModel):
         if role == SOUND_ROLE:
             return s
         if role == Qt.ItemDataRole.DisplayRole:
-            if c == C_FAV:
-                return ""
-            if c == C_NAME:
-                return s.name
-            if c == C_CAT:
-                return s.category
-            if c == C_CHAR:
-                return s.character or (("~ " + s.characters_inferred) if s.characters_inferred else "")
-            if c == C_LEN:
-                return "%.2f s" % s.seconds
-            if c == C_ARCH:
-                return s.archive
-            if c == C_FMT:
-                return "%d Hz %s" % (s.rate, {1: "mono", 2: "stereo"}.get(s.channels, "%d ch" % s.channels))
-            if c == C_SIZE:
-                return "any" if s.is_pack_media else "%.1f KB" % (s.size / 1024.0)
-            if c == C_STATUS:
-                return "Modified" if s.key in self.modified else ""
-            if c == C_NOTES:
-                if not ud:
-                    return ""
-                t = ud.tags(s.key)
-                return " ".join("#" + x for x in t) + ("  " if t else "") + ud.note(s.key)
+            return self.display(s, c)
         if role == SORT_ROLE:
-            if c == C_FAV:
-                return 0 if ud and ud.favorite(s.key) else 1
-            if c == C_LEN:
-                return s.seconds
-            if c == C_SIZE:
-                return 1e12 if s.is_pack_media else s.size
-            if c == C_FMT:
-                return (s.rate, s.channels)
-            if c == C_STATUS:
-                return 0 if s.key in self.modified else 1
-            return (self.data(index, Qt.ItemDataRole.DisplayRole) or "").lower()
+            return self.sort_key(index.row(), c)
         if role == Qt.ItemDataRole.DecorationRole and c == C_FAV:
             fav = bool(ud and ud.favorite(s.key))
             return icons.icon("star" if fav else "star-off", icons.ACCENT if fav else "#4a4a4a")
@@ -122,9 +128,122 @@ class SoundModel(QAbstractTableModel):
         return None
 
 
-class SoundFilter(QSortFilterProxyModel):
+class RowView(QAbstractProxyModel):
+    """The rows a table shows: the source rows a subclass accepts, in the chosen order. Computed in Python in
+    one pass (filter, then one sort over cached keys): ~10 ms for 21,000 sounds, where QSortFilterProxyModel
+    needed up to ~1.5 s per sort or widened filter (each comparison called back into Python). Selection and the
+    current row survive every change (persistent indexes are moved, as Qt's own proxies do).
+    Subclasses: source_len(), accepts(row), sort_key(row, column)."""
+
     def __init__(self, parent=None):
         super().__init__(parent)
+        self.sort_column, self.sort_order = -1, Qt.SortOrder.AscendingOrder
+        self.rows = []                  # proxy row -> source row
+        self.pos = {}                   # source row -> proxy row
+        self._keys = {}                 # column -> [sort key per source row] (until the source changes)
+
+    def source_len(self):
+        return self.sourceModel().rowCount()
+
+    def accepts(self, row):
+        return True
+
+    def sort_key(self, row, column):
+        v = self.sourceModel().index(row, column).data()
+        return v.lower() if isinstance(v, str) else v
+
+    # --- source model --------------------------------------------------------------------------------
+    def setSourceModel(self, model):
+        """One source model for the view's lifetime."""
+        self.beginResetModel()
+        super().setSourceModel(model)
+        model.modelAboutToBeReset.connect(self.beginResetModel)
+        model.modelReset.connect(self._source_reset)
+        model.dataChanged.connect(self._source_changed)
+        self._recompute_all()
+        self.endResetModel()
+
+    def _recompute_all(self):
+        self._keys = {}
+        self.rows = self._compute()
+        self.pos = {r: i for i, r in enumerate(self.rows)}
+
+    def _source_reset(self):
+        self._recompute_all()
+        self.endResetModel()
+
+    def _source_changed(self, top_left, bottom_right, roles=()):
+        """Values changed (status, notes, favorites, texts): sort keys and filters may depend on them."""
+        self._keys = {}
+        self.refilter()
+        if self.rows:
+            self.dataChanged.emit(self.index(0, 0), self.index(len(self.rows) - 1, self.columnCount() - 1))
+
+    # --- filter and order ----------------------------------------------------------------------------
+    def refilter(self):
+        """Recompute the rows; the view keeps its selection and current row (layout change)."""
+        rows = self._compute()
+        if rows == self.rows:
+            return
+        self.layoutAboutToBeChanged.emit()
+        old = self.persistentIndexList()
+        src = [(self.rows[i.row()], i.column()) if i.isValid() and i.row() < len(self.rows) else None for i in old]
+        self.rows = rows
+        self.pos = {r: i for i, r in enumerate(rows)}
+        new = [self.index(self.pos[s[0]], s[1]) if s and s[0] in self.pos else QModelIndex() for s in src]
+        self.changePersistentIndexList(old, new)
+        self.layoutChanged.emit()
+
+    def sort(self, column, order=Qt.SortOrder.AscendingOrder):
+        self.sort_column, self.sort_order = column, order
+        self.refilter()
+
+    def _compute(self):
+        if self.sourceModel() is None:
+            return []
+        n = self.source_len()
+        rows = [r for r in range(n) if self.accepts(r)]
+        c = self.sort_column
+        if c >= 0:
+            if c not in self._keys:
+                self._keys[c] = [self.sort_key(r, c) for r in range(n)]
+            rows.sort(key=self._keys[c].__getitem__, reverse=self.sort_order == Qt.SortOrder.DescendingOrder)
+        return rows
+
+    # --- QAbstractProxyModel -------------------------------------------------------------------------
+    def index(self, row, column, parent=QModelIndex()):
+        if parent.isValid() or not (0 <= row < len(self.rows)) or not (0 <= column < self.columnCount()):
+            return QModelIndex()
+        return self.createIndex(row, column)
+
+    def parent(self, index=QModelIndex()):
+        return QModelIndex()
+
+    def rowCount(self, parent=QModelIndex()):
+        return 0 if parent.isValid() else len(self.rows)
+
+    def columnCount(self, parent=QModelIndex()):
+        m = self.sourceModel()
+        return 0 if parent.isValid() or m is None else m.columnCount()
+
+    def mapToSource(self, proxy_index):
+        if not proxy_index.isValid() or proxy_index.row() >= len(self.rows):
+            return QModelIndex()
+        return self.sourceModel().index(self.rows[proxy_index.row()], proxy_index.column())
+
+    def mapFromSource(self, source_index):
+        if not source_index.isValid() or source_index.row() not in self.pos:
+            return QModelIndex()
+        return self.index(self.pos[source_index.row()], source_index.column())
+
+    def headerData(self, section, orientation, role=Qt.ItemDataRole.DisplayRole):
+        return self.sourceModel().headerData(section, orientation, role)
+
+
+class SoundFilter(RowView):
+    """The sound browser's rows: search words and facet filters, sorted by any column."""
+
+    def __init__(self, parent=None):
         self.words = []
         self.group = None
         self.category = None
@@ -134,15 +253,7 @@ class SoundFilter(QSortFilterProxyModel):
         self.modified_only = False
         self.favorites_only = False
         self.tag = None
-        self.setSortRole(SORT_ROLE)
-        self.setDynamicSortFilter(False)
-
-    def refilter(self):
-        if hasattr(self, "beginFilterChange"):  # Qt >= 6.10
-            self.beginFilterChange()
-            self.endFilterChange(QSortFilterProxyModel.Direction.Rows)
-        else:
-            self.invalidateRowsFilter()
+        super().__init__(parent)
 
     def set(self, **kw):
         for k, v in kw.items():
@@ -153,7 +264,13 @@ class SoundFilter(QSortFilterProxyModel):
         self.words = [w for w in text.lower().replace("_", " ").split() if w]
         self.refilter()
 
-    def filterAcceptsRow(self, row, parent):
+    def source_len(self):
+        return len(self.sourceModel().sounds)
+
+    def sort_key(self, row, column):
+        return self.sourceModel().sort_key(row, column)
+
+    def accepts(self, row):
         m = self.sourceModel()
         s = m.sounds[row]
         if self.group and s.group != self.group:
@@ -181,4 +298,4 @@ class SoundFilter(QSortFilterProxyModel):
         return True
 
     def sound_at(self, proxy_index):
-        return self.sourceModel().sounds[self.mapToSource(proxy_index).row()]
+        return self.sourceModel().sounds[self.rows[proxy_index.row()]]

app/spinner.py

@@ -1,6 +1,6 @@
 """Small activity spinner: a rotating accent-coloured arc. Runs only while visible."""
-from PySide6.QtCore import QRectF, QTimer, Qt
-from PySide6.QtGui import QColor, QPainter, QPen
+from PySide6.QtCore import QEvent, QRectF, QTimer, Qt
+from PySide6.QtGui import QColor, QPainter, QPalette, QPen
 from PySide6.QtWidgets import QWidget
 
 from . import icons
@@ -51,3 +51,68 @@ class Spinner(QWidget):
     def paintEvent(self, _):
         p = QPainter(self)
         paint_arc(p, QRectF(self.rect()), self.angle)
+
+
+class BusyOverlay(QWidget):
+    """A list being (re)loaded: a light veil over the widget with the spinner and a message in its centre, the
+    same arc as the waveform's loading state. Covers the target and follows its size; blocks clicks while on."""
+
+    def __init__(self, target):
+        super().__init__(target)
+        self.target = target
+        self.message = ""
+        self.angle = 0
+        self.timer = QTimer(self)
+        self.timer.setInterval(STEP_MS)
+        self.timer.timeout.connect(self._step)
+        target.installEventFilter(self)
+        self.hide()
+
+    def start(self, message):
+        self.message = message
+        self.setGeometry(self.target.rect())
+        self.raise_()
+        self.show()
+        self.timer.start()
+        self.update()
+
+    def stop(self):
+        self.timer.stop()
+        self.hide()
+
+    @property
+    def busy(self):
+        return not self.isHidden()          # on, whether or not the window is on screen yet
+
+    def _step(self):
+        self.angle = (self.angle + STEP_DEG) % 360
+        if self.isVisible():
+            self.update()
+
+    def eventFilter(self, obj, e):
+        if obj is self.target and e.type() == QEvent.Type.Resize:
+            self.setGeometry(self.target.rect())
+        return False
+
+    def paintEvent(self, _):
+        p = QPainter(self)
+        p.setRenderHint(QPainter.RenderHint.Antialiasing)
+        pal = self.palette()
+        veil = QColor(pal.color(QPalette.ColorRole.Base))
+        veil.setAlpha(210)
+        p.fillRect(self.rect(), veil)
+        fm = p.fontMetrics()
+        d, gap, pad = 20, 12, 16
+        tw = fm.horizontalAdvance(self.message)
+        w, h = pad + d + gap + tw + pad, max(d, fm.height()) + 2 * pad - 8
+        box = QRectF((self.width() - w) / 2, (self.height() - h) / 2, w, h)
+        border = QColor(pal.color(QPalette.ColorRole.Text))
+        border.setAlpha(40)
+        p.setPen(QPen(border, 1))
+        p.setBrush(pal.color(QPalette.ColorRole.Window))
+        p.drawRoundedRect(box, 8, 8)
+        y = box.center().y()
+        paint_arc(p, QRectF(box.left() + pad, y - d / 2, d, d), self.angle)
+        p.setPen(pal.color(QPalette.ColorRole.Text))
+        p.drawText(QRectF(box.left() + pad + d + gap, y - fm.height() / 2, tw + 2, fm.height()),
+                   Qt.AlignmentFlag.AlignVCenter | Qt.AlignmentFlag.AlignLeft, self.message)

app/textpanel.py

@@ -0,0 +1,598 @@
+"""Text sections of the main window: Subtitles (linked to the speech sounds) and Menu text.
+
+One TextPanel per section: filters + table on top, the editor of the selected text below. Texts are written
+through Workspace.texts (ackit/texts.py): subtitles in place into every copy (limited to the original's
+length), menu texts into the localization files. Revert per text or all; changed-underneath warnings.
+"""
+from PySide6.QtCore import QAbstractTableModel, QModelIndex, Qt, QTimer, Signal
+from PySide6.QtWidgets import (QAbstractItemView, QCheckBox, QComboBox, QDialog, QDialogButtonBox, QFormLayout,
+                               QHBoxLayout, QHeaderView, QLabel, QLineEdit, QPushButton, QSplitter, QTableView,
+                               QVBoxLayout, QWidget)
+
+from ackit import subtitles
+from ackit.texts import CHANGED, FOREIGN, LOST, MODIFIED, ORIGINAL, TextConflictError
+from . import icons
+from .models import RowView
+from .spinner import BusyOverlay
+
+ANY = "Any"
+STATE_TEXT = {ORIGINAL: "", MODIFIED: "modified", LOST: "lost", CHANGED: "changed outside", FOREIGN: "not original"}
+
+
+class Row:
+    __slots__ = ("key", "cols", "text", "sound", "search")
+
+    def __init__(self, key, cols, text, sound=None):
+        self.key, self.cols, self.text, self.sound = key, cols, text, sound
+        self.search = (" ".join(str(c) for c in cols) + " " + text).lower()
+
+
+class TextModel(QAbstractTableModel):
+    def __init__(self, headers, parent=None):
+        super().__init__(parent)
+        self.headers = headers + ["Text", "State"]
+        self.rows = []
+        self.modified = {}            # key -> applied text
+
+    def reset(self, rows, modified):
+        self.beginResetModel()
+        self.rows, self.modified = rows, dict(modified)
+        self.endResetModel()
+
+    def set_modified(self, modified):
+        self.modified = dict(modified)
+        if self.rows:
+            self.dataChanged.emit(self.index(0, 0), self.index(len(self.rows) - 1, len(self.headers) - 1))
+
+    def rowCount(self, parent=QModelIndex()):
+        return 0 if parent.isValid() else len(self.rows)
+
+    def columnCount(self, parent=QModelIndex()):
+        return len(self.headers)
+
+    def headerData(self, section, orientation, role=Qt.ItemDataRole.DisplayRole):
+        if orientation == Qt.Orientation.Horizontal and role == Qt.ItemDataRole.DisplayRole:
+            return self.headers[section]
+        return None
+
+    def text_of(self, r):
+        return self.modified.get(r.key, r.text)
+
+    def data(self, index, role=Qt.ItemDataRole.DisplayRole):
+        r = self.rows[index.row()]
+        c = index.column()
+        n = len(r.cols)
+        if role in (Qt.ItemDataRole.DisplayRole, Qt.ItemDataRole.ToolTipRole):
+            if c < n:
+                return r.cols[c]
+            if c == n:
+                return self.text_of(r)
+            return "modified" if r.key in self.modified else ""
+        if role == Qt.ItemDataRole.UserRole:
+            return r
+        if role == Qt.ItemDataRole.ForegroundRole and c == n + 1 and r.key in self.modified:
+            from PySide6.QtGui import QColor
+            return QColor(icons.ACCENT)
+        return None
+
+
+class TextFilter(RowView):
+    """The text table's rows: search words, column filters, modified only; sorted case-insensitively."""
+
+    def __init__(self, parent=None):
+        self.words, self.col_filters, self.only_modified = [], {}, False
+        super().__init__(parent)
+
+    def source_len(self):
+        return len(self.sourceModel().rows)
+
+    def sort_key(self, row, column):
+        m = self.sourceModel()
+        r = m.rows[row]
+        n = len(r.cols)
+        v = r.cols[column] if column < n else m.text_of(r) if column == n else ("modified" if r.key in m.modified else "")
+        return v.lower() if isinstance(v, str) else v
+
+    def accepts(self, row):
+        m = self.sourceModel()
+        r = m.rows[row]
+        if self.only_modified and r.key not in m.modified:
+            return False
+        for c, v in self.col_filters.items():
+            if v != ANY and r.cols[c] != v:
+                return False
+        if self.words:
+            hay = r.search + " " + m.modified.get(r.key, "").lower()
+            return all(w in hay for w in self.words)
+        return True
+
+    def row_at(self, index):
+        return self.sourceModel().rows[self.rows[index.row()]]
+
+
+class FindReplaceDialog(QDialog):
+    def __init__(self, count, parent=None):
+        super().__init__(parent)
+        self.setWindowTitle("Find and replace")
+        f = QFormLayout(self)
+        self.find, self.repl = QLineEdit(), QLineEdit()
+        self.case = QCheckBox("Match case")
+        f.addRow(QLabel("In the %d text(s) shown by the current filter:" % count))
+        f.addRow("Find", self.find)
+        f.addRow("Replace with", self.repl)
+        f.addRow(self.case)
+        bb = QDialogButtonBox(QDialogButtonBox.StandardButton.Ok | QDialogButtonBox.StandardButton.Cancel)
+        bb.accepted.connect(self.accept)
+        bb.rejected.connect(self.reject)
+        f.addRow(bb)
+
+
+def replace_in(text, find, repl, case):
+    if case:
+        return text.replace(find, repl)
+    import re
+    return re.sub(re.escape(find), lambda _: repl, text, flags=re.IGNORECASE)
+
+
+class SubtitleBox(QWidget):
+    """The subtitle of the sound open in the sound editor (side by side with the audio)."""
+    changed = Signal()
+    open_in_list = Signal(str)            # SUB key
+
+    def __init__(self, win, parent=None):
+        super().__init__(parent)
+        self.win = win
+        self.keys = []
+        outer = QVBoxLayout(self)
+        outer.setContentsMargins(0, 0, 0, 0)
+        outer.setSpacing(6)
+        self.lay = QFormLayout()               # one line edit per subtitle line
+        self.lay.setContentsMargins(0, 0, 0, 0)
+        self.edits = []
+        self.lbl = QLabel()
+        self.lbl.setObjectName("Muted")
+        row = QHBoxLayout()
+        row.setSpacing(6)
+        self.b_apply = QPushButton(icons.icon("apply"), "Apply")
+        self.b_revert = QPushButton(icons.icon("revert"), "Revert")
+        self.b_open = QPushButton(icons.icon("search"), "Show in list")
+        self.b_apply.setToolTip("Write this subtitle into the game (every copy)")
+        self.b_revert.setToolTip("Put the original subtitle back")
+        self.b_open.setToolTip("Open this subtitle in the Subtitles tab")
+        self.b_apply.clicked.connect(self.apply)
+        self.b_revert.clicked.connect(self.revert)
+        self.b_open.clicked.connect(lambda: self.keys and self.open_in_list.emit(self.keys[0]))
+        for b in (self.b_apply, self.b_revert):    # two halves of the column, like the sound's action bar
+            row.addWidget(b, 1)
+        self.lbl.setWordWrap(True)
+        outer.addLayout(self.lay)
+        outer.addWidget(self.lbl)              # room counter
+        outer.addWidget(self.b_open)           # the way to the Subtitles tab, across the column
+        outer.addStretch(1)                    # actions at the bottom, level with the sound's
+        outer.addLayout(row)
+
+    def set_sound(self, s):
+        for e in self.edits:
+            self.lay.removeRow(e)
+        self.edits, self.keys = [], []
+        ws = self.win.ws
+        lines = ws.texts.for_sound(s, self.win.language) if ws and s else []
+        if not lines:
+            self.lbl.setText("No subtitle for this sound." if s else "")
+            for b in (self.b_apply, self.b_revert, self.b_open):
+                b.setEnabled(False)
+            return
+        tx = ws.texts
+        for i, ln in enumerate(lines):
+            k = subtitles.key(ln.lang, ln.speech, ln.line)
+            e = QLineEdit(tx.records().get(k, {}).get("applied") or ln.text)
+            e.setMinimumWidth(80)
+            e.setCursorPosition(0)                     # long subtitles: show their beginning
+            e.setToolTip("Original: %s\nRoom: %d characters" % (ln.text, ln.room))
+            e.textChanged.connect(self._check)
+            self.lay.insertRow(i, "Line %d" % (i + 1) if len(lines) > 1 else "Subtitle", e)
+            self.edits.append(e)
+            self.keys.append(k)
+        self.b_open.setEnabled(True)
+        self.b_revert.setEnabled(any(k in tx.records() for k in self.keys))
+        self._check()
+
+    def _check(self):
+        tx = self.win.ws.texts
+        msgs = []
+        ok = True
+        for k, e in zip(self.keys, self.edits):
+            try:
+                tx.check(k, e.text())
+            except ValueError as ex:
+                ok = False
+                msgs.append(str(ex))
+        rooms = [tx.limit(k) for k in self.keys]
+        self.lbl.setText("   •   ".join(msgs) if msgs else "Room: %s characters (%s)" % (
+            " / ".join(str(r) for r in rooms), self.keys[0].split(":")[1]))
+        self.b_apply.setEnabled(ok)
+
+    def values(self):
+        return {k: e.text() for k, e in zip(self.keys, self.edits)}
+
+    def apply(self):
+        tx = self.win.ws.texts
+        changes = {k: v for k, v in self.values().items() if v != tx.original(k) or k in tx.records()}
+        if not changes:
+            return
+        self._run(lambda job, force: tx.apply(changes, force=force, progress=job.report), "Subtitle applied.")
+
+    def revert(self):
+        tx = self.win.ws.texts
+        self._run(lambda job, force: tx.revert(self.keys, progress=job.report), "Subtitle reverted.")
+
+    def _run(self, fn, ok_msg):
+        def done(_):
+            self.win.statusBar().showMessage(ok_msg, 6000)
+            self.changed.emit()
+
+        def failed(e, tb):
+            if isinstance(e, TextConflictError) and self.win.confirm("Changed outside", "%s\n\nOverwrite anyway?" % e):
+                self.win.runner.start(lambda job: fn(job, True), done, failed)
+                return
+            self.win.inform("Error", str(e))
+
+        self.win.runner.start(lambda job: fn(job, False), done, failed)
+
+
+class TextPanel(QWidget):
+    """kind 'subs' (subtitles) or 'menu' (menu text)."""
+    changed = Signal()
+    message = Signal(str)
+    go_to_sound = Signal(str)            # sound key (subtitles)
+
+    def __init__(self, win, kind):
+        super().__init__(win)
+        self.win, self.kind = win, kind
+        self.lang = None
+        self.current = None              # selected Row
+        self._build()
+
+    # --- UI ---------------------------------------------------------------------------------------
+    def _build(self):
+        v = QVBoxLayout(self)
+        v.setContentsMargins(6, 6, 6, 6)
+        top = QHBoxLayout()
+        self.search = QLineEdit()
+        self.search.setClearButtonEnabled(True)
+        self.search.setPlaceholderText("Search texts, names, characters…" if self.kind == "subs" else
+                                       "Search texts, files, keys…")
+        self.search.addAction(icons.icon("search", icons.MUTED), QLineEdit.ActionPosition.LeadingPosition)
+        self._timer = QTimer(self)
+        self._timer.setSingleShot(True)
+        self._timer.setInterval(150)
+        self._timer.timeout.connect(self._filter)
+        self.search.textChanged.connect(lambda _: self._timer.start())
+        self.lbl_lang = QLabel()
+        self.lbl_lang.setObjectName("Muted")
+        self.lbl_lang.setToolTip("Change the language with the Language list at the top of the window")
+        top.addWidget(self.search, 1)
+        top.addWidget(self.lbl_lang)
+        self.filters = {}
+        names = ("Character", "Category") if self.kind == "subs" else ("File",)
+        for name in names:
+            cb = QComboBox()
+            cb.setMinimumWidth(120)
+            cb.currentTextChanged.connect(lambda _: self._filter())
+            top.addWidget(QLabel(name))
+            top.addWidget(cb)
+            self.filters[name] = cb
+        self.cb_mod = QCheckBox("Modified only")
+        self.cb_mod.toggled.connect(lambda _: self._filter())
+        top.addWidget(self.cb_mod)
+        self.lbl_count = QLabel()
+        top.addWidget(self.lbl_count)
+        v.addLayout(top)
+
+        headers = ["Speech line", "#", "Character", "Category", "Room"] if self.kind == "subs" else \
+            ["File", "Section", "Key"]
+        self.model = TextModel(headers, self)
+        self.proxy = TextFilter(self)
+        self.proxy.setSourceModel(self.model)
+        self.table = QTableView()
+        self.table.setModel(self.proxy)
+        self.busy = BusyOverlay(self.table)    # while the texts of a language load (background job)
+        self._load_gen = 0
+        self._pending_key = None
+        self.table.setSortingEnabled(True)
+        self.table.setSelectionBehavior(QAbstractItemView.SelectionBehavior.SelectRows)
+        self.table.setSelectionMode(QAbstractItemView.SelectionMode.SingleSelection)
+        self.table.verticalHeader().hide()
+        self.table.verticalHeader().setDefaultSectionSize(22)
+        self.table.setAlternatingRowColors(True)
+        self.table.setWordWrap(False)
+        h = self.table.horizontalHeader()
+        h.setSectionResizeMode(QHeaderView.ResizeMode.Interactive)
+        h.setStretchLastSection(False)
+        widths = (330, 30, 120, 130, 50) if self.kind == "subs" else (140, 150, 220)
+        for c, w in enumerate(widths):
+            self.table.setColumnWidth(c, w)
+        self.table.setColumnWidth(len(widths), 560)
+        self.table.selectionModel().currentRowChanged.connect(lambda cur, _: self._select(cur))
+
+        # editor
+        ed = QWidget()
+        f = QFormLayout(ed)
+        self.title = QLabel("No text selected")
+        self.title.setObjectName("EditorTitle")
+        self.lbl_orig = QLabel()
+        self.lbl_orig.setWordWrap(True)
+        self.lbl_orig.setTextInteractionFlags(Qt.TextInteractionFlag.TextSelectableByMouse)
+        self.edit = QLineEdit()
+        self.edit.textChanged.connect(self._edited)
+        self.edit.returnPressed.connect(self.apply_current)
+        self.lbl_info = QLabel()
+        row = QHBoxLayout()
+        self.b_apply = QPushButton(icons.icon("apply"), "Apply")
+        self.b_revert = QPushButton(icons.icon("revert"), "Revert")
+        self.b_replace = QPushButton(icons.icon("search"), "Find and replace…")
+        self.b_revert_all = QPushButton(icons.icon("revert"), "Revert all texts here")
+        self.b_apply.clicked.connect(self.apply_current)
+        self.b_revert.clicked.connect(self.revert_current)
+        self.b_replace.clicked.connect(self.find_replace)
+        self.b_revert_all.clicked.connect(self.revert_all)
+        for b in (self.b_apply, self.b_revert):
+            row.addWidget(b)
+        if self.kind == "subs":
+            self.b_sound = QPushButton(icons.icon("play"), "Go to the sound")
+            self.b_sound.clicked.connect(lambda: self.current and self.current.sound and
+                                         self.go_to_sound.emit(self.current.sound))
+            row.addWidget(self.b_sound)
+        row.addStretch(1)
+        row.addWidget(self.b_replace)
+        row.addWidget(self.b_revert_all)
+        f.addRow(self.title)
+        f.addRow("Original", self.lbl_orig)
+        f.addRow("Text", self.edit)
+        f.addRow("", self.lbl_info)
+        f.addRow(row)
+        split = QSplitter(Qt.Orientation.Vertical)
+        split.addWidget(self.table)
+        split.addWidget(ed)
+        split.setStretchFactor(0, 1)
+        split.setStretchFactor(1, 0)
+        v.addWidget(split, 1)
+        self._select(None)
+
+    # --- data ---------------------------------------------------------------------------------------
+    def ws(self):
+        return self.win.ws
+
+    def languages(self):
+        tx = self.ws().texts
+        return tx.subtitle_languages() if self.kind == "subs" else tx.menu_languages()
+
+    def load(self, lang=None):
+        """(Re)load the rows of a language (default: the window's language)."""
+        ws = self.ws()
+        if not ws:
+            return
+        lang = lang or self.win.language
+        if lang not in self.languages():
+            lang = None
+        self.lang = lang
+        self.lbl_lang.setText("Language: %s" % lang if lang else "No %s in this language" % (
+            "subtitles" if self.kind == "subs" else "menu text"))
+        self._load_gen += 1
+        gen = self._load_gen
+        if not lang:
+            self.busy.stop()
+            self.model.reset([], {})
+            return
+        self.busy.start("Loading subtitles…" if self.kind == "subs" else "Loading menu text…")
+
+        def done(rows):
+            if gen == self._load_gen:              # a newer load replaces this one
+                self.busy.stop()
+                self._show_rows(rows)
+
+        def failed(e, tb):
+            if gen == self._load_gen:
+                self.busy.stop()
+                self.message.emit("Could not load the texts: %s" % e)
+
+        self.win.runner.start(lambda job: self._rows(ws, lang), done, failed)
+
+    def _rows(self, ws, lang):
+        """The table rows of a language (runs as a background job)."""
+        tx = ws.texts
+        rows = []
+        if self.kind == "subs":
+            by_key = ws.catalog.by_key
+            for s in tx.subtitles(lang):
+                snd = by_key.get(s.sounds[0]) if s.sounds else None   # None: the line's archive is not loaded
+                who = (snd.character or snd.characters_inferred) if snd else ""
+                cat = snd.category if snd else ""
+                rows.append(Row(subtitles.key(lang, s.speech, s.line),
+                                [s.speech, s.line + 1, who, cat, s.room], s.text, s.sounds[0] if s.sounds else None))
+        else:
+            for e in tx.menu(lang):
+                rows.append(Row("MENU:" + e.key, [e.file, e.section, e.name + ("#%d" % e.occurrence if e.occurrence > 1 else "")],
+                                self._menu_original(e)))
+        return rows
+
+    def _show_rows(self, rows):
+        self.model.reset(rows, self._modified())
+        for i, cb in enumerate(self.filters.values()):
+            col = (2 + i) if self.kind == "subs" else 0
+            cur = cb.currentText()
+            cb.blockSignals(True)
+            cb.clear()
+            cb.addItems([ANY] + sorted({str(r.cols[col]) for r in rows if r.cols[col]}))
+            cb.setCurrentText(cur if cur else ANY)
+            cb.blockSignals(False)
+        self._filter()
+        if self._pending_key:                       # asked for while the rows were loading
+            key, self._pending_key = self._pending_key, None
+            self.select_key(key)
+
+    def _menu_original(self, e):
+        r = self.ws().texts.records().get("MENU:" + e.key)
+        return r["orig"] if r else e.value
+
+    def _modified(self):
+        return {k: r["applied"] for k, r in self.ws().texts.records().items()
+                if r.get("applied") is not None and k.split(":")[1] == self.lang and
+                k.startswith("SUB:" if self.kind == "subs" else "MENU:")}
+
+    def refresh(self):
+        self.model.set_modified(self._modified())                    # the view refilters/re-sorts itself
+        self._update_count()
+        if self.current:
+            self._show(self.current)
+
+    def _filter(self):
+        self.proxy.words = [w for w in self.search.text().lower().split() if w]
+        self.proxy.only_modified = self.cb_mod.isChecked()
+        cols = (2, 3) if self.kind == "subs" else (0,)
+        self.proxy.col_filters = {c: cb.currentText() for c, cb in zip(cols, self.filters.values())}
+        self.proxy.refilter()
+        self._update_count()
+
+    def _update_count(self):
+        self.lbl_count.setText("%d / %d" % (self.proxy.rowCount(), self.model.rowCount()))
+
+    def visible_rows(self):
+        return [self.proxy.row_at(self.proxy.index(i, 0)) for i in range(self.proxy.rowCount())]
+
+    def select_key(self, key):
+        if self.busy.busy:                          # rows still loading: select when they arrive
+            self._pending_key = key
+            return True
+        for i in range(self.proxy.rowCount()):
+            if self.proxy.row_at(self.proxy.index(i, 0)).key == key:
+                self.table.selectRow(i)
+                self.table.scrollTo(self.proxy.index(i, 0))
+                return True
+        return False
+
+    # --- editing ------------------------------------------------------------------------------------
+    def _select(self, index):
+        r = self.proxy.row_at(index) if index is not None and index.isValid() else None
+        self.current = r
+        self._show(r)
+
+    def _show(self, r):
+        enabled = r is not None
+        for w in (self.edit, self.b_apply, self.b_revert):
+            w.setEnabled(enabled)
+        if self.kind == "subs":
+            self.b_sound.setEnabled(bool(r and r.sound))
+        if not r:
+            self.title.setText("No text selected")
+            self.lbl_orig.setText("")
+            self.edit.setText("")
+            self.lbl_info.setText("")
+            return
+        tx = self.ws().texts
+        self.title.setText(r.key)
+        self.lbl_orig.setText(r.text)
+        try:
+            cur = tx.current(r.key)
+            st = tx.state(r.key, cur)
+        except Exception as e:  # noqa: BLE001 - unreadable file: show it, do not crash
+            cur, st = r.text, "error: %s" % e
+        self.edit.blockSignals(True)
+        self.edit.setText(cur)
+        self.edit.blockSignals(False)
+        self._state = st
+        self.b_revert.setEnabled(r.key in tx.records())
+        self._edited()
+
+    def _edited(self):
+        r = self.current
+        if not r:
+            return
+        tx = self.ws().texts
+        lim = tx.limit(r.key)
+        n = len(self.edit.text())
+        info = []
+        if lim is not None:
+            info.append("%d / %d characters" % (n, lim))
+        st = STATE_TEXT.get(self._state, self._state)
+        if st:
+            info.append("state: " + st)
+        try:
+            tx.check(r.key, self.edit.text())
+            ok = True
+        except ValueError as e:
+            info.append(str(e))
+            ok = False
+        self.lbl_info.setText("   •   ".join(info))
+        self.b_apply.setEnabled(ok)
+
+    def _run(self, busy, fn, ok_msg):
+        def done(_):
+            self.message.emit(ok_msg)
+            self.refresh()
+            self.changed.emit()
+
+        def failed(e, tb):
+            if isinstance(e, TextConflictError) and self.win.confirm(
+                    "Changed outside", "%s\n\nOverwrite anyway?" % e):
+                self.win.runner.start(lambda job: fn(job, True), done, failed)
+                return
+            self.win.inform("Error", str(e))
+
+        self.message.emit(busy)
+        self.win.runner.start(lambda job: fn(job, False), done, failed)
+
+    def apply_current(self):
+        r = self.current
+        if not r or not self.b_apply.isEnabled():
+            return
+        text = self.edit.text()
+        self._run("Writing…", lambda job, force: self.ws().texts.apply({r.key: text}, force=force, progress=job.report),
+                  "Applied %s" % r.key)
+
+    def revert_current(self):
+        r = self.current
+        if r:
+            self._run("Reverting…", lambda job, force: self.ws().texts.revert([r.key], progress=job.report),
+                      "Reverted %s" % r.key)
+
+    def revert_all(self):
+        keys = list(self._modified())
+        if not keys:
+            self.win.inform("Revert", "No %s text is modified." % self.lang)
+            return
+        if not self.win.confirm("Revert", "Put the original back for %d text(s)?" % len(keys)):
+            return
+        self._run("Reverting…", lambda job, force: self.ws().texts.revert(keys, progress=job.report),
+                  "Reverted %d text(s)." % len(keys))
+
+    def find_replace(self, find=None, repl=None, case=False):
+        rows = self.visible_rows()
+        if find is None:
+            dlg = FindReplaceDialog(len(rows), self)
+            if dlg.exec() != QDialog.DialogCode.Accepted or not dlg.find.text():
+                return
+            find, repl, case = dlg.find.text(), dlg.repl.text(), dlg.case.isChecked()
+        tx = self.ws().texts
+        changes, skipped = {}, []
+        for r in rows:
+            old = self.model.text_of(r)
+            new = replace_in(old, find, repl, case)
+            if new == old:
+                continue
+            try:
+                tx.check(r.key, new)
+                changes[r.key] = new
+            except ValueError as e:
+                skipped.append("%s: %s" % (r.key, e))
+        if not changes:
+            self.win.inform("Find and replace", "Nothing to replace." + ("\n" + "\n".join(skipped[:10]) if skipped else ""))
+            return
+        if skipped and not self.win.confirm("Find and replace", "%d text(s) cannot take the replacement and are "
+                                            "skipped:\n%s\n\nReplace in the other %d?" % (
+                                                len(skipped), "\n".join(skipped[:10]), len(changes))):
+            return
+        self._run("Replacing in %d text(s)…" % len(changes),
+                  lambda job, force: tx.apply(changes, force=force, progress=job.report),
+                  "Replaced in %d text(s)." % len(changes))
+        return len(changes)

data/catalog/README.md

@@ -2,9 +2,12 @@
 
 | file | content |
 |------|---------|
-| `<ARCHIVE>.csv` | every sound of one audio archive: `ITA`, `INT`, `FRA`, `DEU`, `ESN` (speech, one per language) and `SFX` (music, ambience, effects of the SFX WAD; sounds stored inside soundbanks are not listed, see `docs/formats/banks.md`) |
-| `characters.csv` | `name,character`: verified speaker per sound name (language independent) |
-| `index.json` | fingerprint of the game WAD each CSV was built from (`ackit/game.py: wad_fingerprint`) and counts |
+| `<ARCHIVE>.csv` | every sound of one audio archive: `INT`, `FRA`, `ITA`, `DEU`, `ESN` (speech WADs, one per language), `SFX` (music, ambience and effects streamed from the SFX WAD) and `BANK` (in-memory sounds stored inside soundbanks: footsteps, foley, combat, weapons, UI, efforts; `docs/formats/banks.md`) |
+| `BANK_locations.csv.gz` | every stored copy of the `BANK` sounds, and the bank copies of WAD speech recordings (prefetch heads, whole copies; below) |
+| `SUB_<LANG>.csv` | subtitles of the speech lines, one file per language (11 languages; `docs/formats/subtitles.md`) |
+| `SUB_locations.csv.gz` | every stored copy of each subtitle string (dialogue objects in the packages) |
+| `characters.csv` | `name,character,language`: verified speaker per line name (empty `language`: every language, the same line everywhere; a language code would label a sound of that language only - none needs it now) |
+| `index.json` | per archive: fingerprint of the game files the CSV was built from (`ackit/game.py: wad_fingerprint`, `ackit/banks.py: game_fingerprint`) and counts |
 | `source/<lang>/*.txt` | **the curated original lists** (one file per character, one wemscan-style file name per line, e.g. `Speech_TAPE_JOKER_1_4.wem`). `Files.txt` = all files, `Stats.txt` = counts, `Checkpoint.txt` = progress marker. Never edited by tools. |
 
 All CSVs are UTF-8 with a header row. They contain only names and metadata (no game audio).
@@ -12,26 +15,60 @@ All CSVs are UTF-8 with a header row. They contain only names and metadata (no g
 ## `<ARCHIVE>.csv` columns
 | column | meaning |
 |--------|---------|
-| `archive` | `ITA`, `INT`, ... or `SFX` |
-| `target` | location inside the archive: `HASH` (entry with one WEM), `HASH:i` (i-th variation, 0-based), `PACK/MEDIA` (music stream pack) |
-| `name` | display name (`NAME`, `NAME_1..n` for variations, see `ackit/catalog.py`) |
-| `aliases` | other accepted names, `\|`-separated (`NAME_1` for single entries, `NAME#i` for variations) |
+| `archive` | `INT`, `FRA`, ..., `SFX` or `BANK` |
+| `target` | location: `HASH` (WAD entry with one WEM), `HASH:i` (i-th variation, 0-based), `PACK/MEDIA` (music stream pack), `MEDIA` (BANK: Wwise media ID) |
+| `name` | display name. Speech: the line's name, the same in every language (`docs/decisions.md` 5): its speech event (e.g. `Speech_TAPE_JOKER_1_04`, `NAME#2` = second version an event plays) or, for lines no named event plays, the INT recording's position name. A recording stored twice keeps its position name in the second place. SFX: `NAME`, `NAME_1..n`. BANK: the Wwise event that plays it (`Play_..._i`) or `<Bank>_<n>` (`<Bank>_<MEDIA>` where that name is taken, e.g. by the SFX WAD's position names) |
+| `aliases` | other accepted names, `\|`-separated: `NAME#i` = i-th WEM of the entry in this language's WAD (positions differ between languages), the entry name for single entries, `<Bank>/<MEDIA>` for BANK |
 | `group` | Speech, Music, Ambience, SFX, Unknown (`docs/categories.md`) |
 | `category` | finer category (`docs/categories.md`) |
+| `bank` | BANK: the soundbank holding the sound (first by name if several do); empty for WAD sounds |
+| `language` | BANK: language(s) of a localized speech line, `;`-separated (`INT`, or `ESN;FRA;INT;ITA` for one recording shared by those); empty = the same sound in every language. The editor shows the sounds of the selected language; names are unique per language |
+| `line` | speech (WAD and BANK): the line's identity, the same in every language (smallest Wwise Sound object ID of the line; `ackit/lines.py`). Two rows of one archive with the same `line` are one recording stored twice (written together). Empty for non-speech sounds |
 | `character` | verified speaker(s) from `characters.csv`, `; `-separated, empty if unknown |
-| `characters_inferred` | characters/roles mentioned by the name (hint only) |
+| `characters_inferred` | characters/roles mentioned by the name (and bank) (hint only) |
 | `seconds`, `rate`, `channels` | of the original sound |
 | `size` | size of the original WEM in bytes = maximum size of a replacement (except music packs: any size) |
-| `sha1` | first 16 hex digits of the original WEM's SHA-1 (detects files changed outside the tool) |
+| `sha1` | first 16 hex digits of the original WEM's SHA-1 (detects files changed outside the tool; empty when the copies of a BANK sound differ) |
 | `notes` | free text (empty; for future curation, e.g. transcripts) |
 
-The editor uses these files directly when the installed WAD matches `index.json`; for another game build it
-scans the WAD itself and applies the same rules (no checksums then).
+## `BANK_locations.csv.gz` columns (gzip-compressed CSV)
+| column | meaning |
+|--------|---------|
+| `media` | Wwise media ID (hex) |
+| `file` | file under `BmGame/CookedPCConsole`, `/`-separated (`SFX/xxxxxxxx.bnk` or `Name.upk`) |
+| `offset` | position of the WEM in the *uncompressed* file (hex; `docs/formats/upk.md`) |
+| `size` | bytes reserved for it there |
+| `kind` | `m` = copy of a BANK sound, `p` = copy of (the start of) a WAD speech recording: a prefetch head, or a whole copy with the same media ID |
+| `sound` | `p` rows: key of that speech recording (e.g. `INT:2B151272:0`); each language has its own recordings and heads |
+
+A replacement is written into every `m` copy of a BANK sound; replacing a speech line also rewrites its `p`
+copies (`docs/formats/banks.md`).
+
+## `SUB_<LANG>.csv` columns
+| column | meaning |
+|--------|---------|
+| `speech` | the speech event of the line (e.g. `Speech_CHURCH_B1_Ch2_POST_Explosion_BM_Cash_04`) |
+| `line` | 0-based line of a multi-line subtitle |
+| `text` | original subtitle text |
+| `room` | maximum length of a replacement in characters (it is written in place) |
+| `enc` | `a` = cp1252 string, `w` = UTF-16 string |
+| `sound` | key(s) of the linked voice recording(s), `;`-separated when the event plays two versions: this language's WAD line or in-memory `BANK:` line (the INT ones for languages without voices); empty when no event reaches a sound |
+
+`SUB_locations.csv.gz`: `lang, speech, line, file, pos` (hex, uncompressed offset of the FString), `size`, `enc`.
+Menu text has no catalog: the localization files are read directly (`docs/formats/text.md`).
+
+The editor uses these files directly when the installed game matches `index.json`; for another game build it
+scans the game itself and applies the same rules (no checksums then; the bank scan takes a few minutes).
+`ackit verify --catalog` cross-checks the catalog in use against the game files (`--quick` samples packages).
 
 ## Rebuilding / adding a language
 1. Make sure the game files are original (`ackit status`; `ackit revert --all`).
-2. Optional: add listening-based lists for another language in `source/<lang>/<Character>.txt` (same format;
-   names are the same in every language, so the Italian lists already label all languages).
-3. Run `python tools/build_catalog.py` - it scans every archive of the installed game (all languages present)
-   and rewrites the CSVs, `characters.csv` and `index.json`.
+2. Optional: add listening-based lists for another language in `source/<lang>/<Character>.txt` (same format:
+   wemscan names of THAT language's WAD). Each list is resolved against its own language's positions and moved
+   to the lines, so a label then applies to the same recording in every language.
+3. Run `python tools/build_catalog.py` - it scans every archive and all soundbanks of the installed game (all
+   languages present; ~5 minutes) and rewrites the CSVs, the location indexes, `characters.csv` and `index.json`.
+   It stops before writing anything if a sound differs from the current catalog (a modified game); after a game
+   update run it with `--accept-changes`.
+4. Run `ackit verify --catalog` (no issues expected).
 New character labels need a display name in `ackit/classify.py: USER_LABELS` (otherwise the file name is used).

data/catalog/index.json

@@ -61,5 +61,79 @@
   },
   "with_character": 0
  },
+ "BANK": {
+  "fingerprint": "3f9766af425b8eef",
+  "sounds": 15355,
+  "groups": {
+   "Ambience": 1015,
+   "SFX": 6912,
+   "Speech": 7422,
+   "Music": 2,
+   "Unknown": 4
+  },
+  "named_by_event": 13501,
+  "named_by_bank": 1854,
+  "copies": 93826,
+  "files": 1499,
+  "prefetch_heads": {
+   "recordings": 445,
+   "copies": 2185,
+   "unmapped": 0
+  },
+  "issues": []
+ },
+ "SUB": {
+  "fingerprint": "3f9766af425b8eef",
+  "lines": {
+   "DEU": 9667,
+   "ESM": 9504,
+   "ESN": 9495,
+   "FRA": 9570,
+   "INT": 9404,
+   "ITA": 9377,
+   "JPN": 9035,
+   "KOR": 9252,
+   "POL": 9379,
+   "POR": 9478,
+   "RUS": 9497
+  },
+  "speech_lines": 9035,
+  "linked": {
+   "DEU": 9632,
+   "ESM": 9470,
+   "ESN": 9462,
+   "FRA": 9534,
+   "INT": 9373,
+   "ITA": 9346,
+   "JPN": 9011,
+   "KOR": 9223,
+   "POL": 9346,
+   "POR": 9444,
+   "RUS": 9463
+  },
+  "copies": 168688,
+  "dialogue_objects": 14744,
+  "dialogue_lines": 9032,
+  "scripted_objects": 41,
+  "issues": []
+ },
+ "LINES": {
+  "languages": [
+   "DEU",
+   "ESN",
+   "FRA",
+   "INT",
+   "ITA"
+  ],
+  "recordings": 61485,
+  "lines": 12434,
+  "wad_lines": 10811,
+  "wad_lines_named_by_event": 10174,
+  "wad_lines_named_after_int_recording": 637,
+  "lines_in_every_language": 12434,
+  "lines_missing_a_language": {},
+  "without_line": [],
+  "issues": []
+ },
  "_info": "unmodified game files"
 }

docs/README.md

@@ -3,13 +3,16 @@
 | document | content |
 |----------|---------|
 | [user-guide.md](user-guide.md) | installing and using Unheard City (screenshots) |
-| [decisions.md](decisions.md) | stack, dependencies/licenses, originals store, size rule, trade-offs |
+| [decisions.md](decisions.md) | stack, dependencies/licenses, originals store, size rule, in-memory sounds, texts, language, verification |
 | [categories.md](categories.md) | group/category scheme and character fields |
 | [building.md](building.md) | building the editor and its installer (Windows, Linux) |
 | [formats/wad.md](formats/wad.md) | WWAD archives, grouped entries, BKPK stream packs |
-| [formats/banks.md](formats/banks.md) | soundbanks and the in-memory sounds not covered yet (footsteps, foley, effects) |
+| [formats/banks.md](formats/banks.md) | soundbanks and the in-memory sounds (footsteps, foley, effects, localized speech); how each voice line is linked across languages |
+| [formats/subtitles.md](formats/subtitles.md) | subtitles: dialogue objects in the packages, link to the speech sounds |
+| [formats/text.md](formats/text.md) | menu and UI text: the localization files |
+| [formats/upk.md](formats/upk.md) | UE3 packages: summary, LZO chunks, how ackit reads and patches them |
 | [formats/wem.md](formats/wem.md) | WEM layout the game accepts, setup stripping, hash and decode-alloc formulas |
 | [reverse-engineering/decoder.md](reverse-engineering/decoder.md) | BatmanAC.exe decoder functions and addresses |
 | [reverse-engineering/findings.md](reverse-engineering/findings.md) | root cause, findings, in-game experiment log (incl. failed hypotheses) |
 | [../data/catalog/README.md](../data/catalog/README.md) | catalog CSV schema, adding languages |
-| [history/](history/) | the two mission briefs this project was built from |
+| [history/](history/) | the mission briefs this project was built from |

docs/building.md

@@ -15,8 +15,8 @@ That one command does everything, offline. If `app/assets/` (committed icon and
 made from `bac_logo_full.png` and `wallpaper.jpg` in `Downloads\Arkham City` (or `--art FOLDER`); without those
 images the build still works, with a plain icon and no wallpaper.
 Output: `dist\UnheardCity\UnheardCity.exe` and `dist\UnheardCity-<ver>-windows-amd64.zip`
-(~78 MB zip, ~190 MB unpacked). The build also writes `THIRD_PARTY_LICENSES.txt` (license texts come from the
-wheels and from `tools/licenses/`).
+(~86 MB zip, ~205 MB unpacked, catalogs included). The build also writes `THIRD_PARTY_LICENSES.txt` (license
+texts come from the wheels and from `tools/licenses/`).
 
 **Installer.** On Windows the folder (and zip) also contains `Install.cmd`, `install.ps1` and `uninstall.ps1`
 (sources in `tools/installer/`, version filled in by the build). It is a plain PowerShell installer, so no

docs/categories.md

@@ -4,8 +4,9 @@ Implemented in `ackit/classify.py`; used both when building the shipped catalog
 and when a game build is not in the catalog (runtime scan), so every language/build gets the same scheme.
 
 ## Groups and categories
-Every sound has a **group** and a finer **category**, derived from its Wwise name (from the community hash
-list; unnamed IDs show as hex). First matching rule wins:
+Every sound has a **group** and a finer **category**, derived from its name (taken from the game files: the
+soundbank names in `CookedBanks.txt` and the Wwise event names; the 3 SFX entries no name reaches show as hex).
+First matching rule wins:
 
 | name starts with | group | category |
 |------------------|-------|----------|
@@ -25,40 +26,63 @@ list; unnamed IDs show as hex). First matching rule wins:
 | `Speech_CHAR_` | Speech | Character lines |
 | `Speech_CH_`, `Speech_SEQUENCE_`, `Speech_STEEL_`, `_MUSEUM_`, `_CHURCH_`, `_COURT_`, `_UNDER_`, `_GCPD_`, `_ZP1_` | Speech | Story missions |
 | `Speech_GAME_`, `Speech_MAP_`, `Speech_AUGMENTED_`, `Speech_FORENSICS_`, `Speech_TEMP_` | Speech | Gameplay & hints |
-| `NPC_` | Ambience | Crowds |
+| `NPC_...Crowd...` | Ambience | Crowds |
 | other `Speech_` | Speech | Other speech |
-| media of the `Music` / `Music_PDLC` stream packs, `Music*` | Music | Music / Music (DLC) |
+| media of the `Music` / `Music_PDLC` stream packs, `Music*`, `MUS_` | Music | Music / Music (DLC) |
 | `AMB_`, `PSA_` | Ambience | Ambience |
+| `PSO_` (flies, drips, fans, lights) | Ambience | Ambient emitters |
 | `WND_` | Ambience | Wind |
+| `WTR_` | Ambience | Weather |
 | `VOL_`, `PVOL_` | Ambience | Room tone |
-| `SEQ_` | SFX | Scripted sequences |
+| anything containing `Footstep` | SFX | Footsteps |
+| `CHR_`/`NPC_` containing `Combat`, `BoneBreak`, `Bodyfall` | SFX | Combat |
+| `SEQ_`, `MTN_` | SFX | Scripted sequences |
 | `OBJ_` | SFX | Objects |
+| `PX_`, `PHY_` | SFX | Physics |
 | `WPN_` | SFX | Weapons |
-| `CHR_` (in SFX) | SFX | Character foley |
+| `DVC_` | SFX | Gadgets |
+| `GUI_`, `SBJ_` | SFX | Interface |
+| `VFX_` | SFX | Visual effects |
+| other `NPC_` | SFX | Enemies & NPCs |
+| `CHR_`, `GEN_` | SFX | Character foley |
 | `Global` | SFX | Global |
 | anything else (e.g. unnamed hex IDs) | Unknown | Unknown |
 
-Counts in the ITA catalog: Story missions 1993, Predator chatter 1796, Character lines 1537, Open-world chatter
-1295, Side missions 808, Combat barks 801, Efforts & emotes 616, Interview tapes 514, Riddler 360, Radio & TV 226,
-Gameplay & hints 199, DLC 167, Boss fights 150, Cinematics 150, Reactions 108, Interrogations 95. SFX archive:
-Music 389, Ambience 225, SFX 31, Unknown 6.
+BANK sounds (in-memory, `docs/formats/banks.md`) are classified by the name of their **soundbank** (e.g.
+`CHR_CatwomanFoleySuit`, `Speech_EMOTE_Batman`), which is more systematic than their event names.
+
+Counts per voice language (identical in all 5): Story missions 1996, Predator chatter 1796, Character lines 1537,
+Open-world chatter 1295, Side missions 808, Combat barks 801, Efforts & emotes 616, Interview tapes 514,
+Riddler 360, Radio & TV 223, Gameplay & hints 199, DLC 167, Boss fights 150, Cinematics 150, Reactions 108,
+Interrogations 95. SFX archive: Music 389, Ambience 225, SFX 31, Unknown 6. BANK (all languages): Combat barks
+4360, Character foley 3051, Other speech 2590, Footsteps 1256, Ambient emitters 705, Weapons 494, Combat 412,
+Objects 401, ...
 
 ## Characters
 Two separate fields, because Wwise names describe a *scene*, not the speaker of each variation (a conversation
 alternates speakers between `NAME_1`, `NAME_2`, ...):
 
 - **character** - the verified speaker, only from the curated lists in `data/catalog/source/<lang>/*.txt`
-  (one file per character, listening-based). Mapped by sound name, so labels made on the Italian files apply to
-  every language (the five language WADs have identical names and variation counts). Display names:
+  (one file per character, listening-based; the lists so far were made on the Italian voices, `source/ita/`).
+  A list names sounds by their position in its language's WAD (wemscan names); positions differ between
+  languages, so each listed recording is found at its position in that language and the label goes to its line
+  (`characters.csv`, keyed by the line's name), i.e. to the same recording in every language. A label applies to exactly the listed recordings (not to other versions of the same event).
+  Checked by `ackit verify --catalog` against the speaker field of the game's dialogue objects: 2,766 labelled
+  lines agree (spelling aside: Mister Freeze, Bruce Wayne, TwoFace, Mysterious_stranger = Azrael), 25 listed
+  as Unknown are named there, 19 differ (to listen to: `Batman_sword_raised_01-04` game: Ra's al Ghul,
+  `ENCRYPTION_CARD2_02` game: Hugo Strange, `thug1_heckle_01_3` game: Thug 01, `SIDE_CAT_TF_intro1_03` game:
+  Thug 06; the 12 Wonder City lines say "Announcement").
+  Display names:
   Calendar -> Calendar Man, Croc -> Killer Croc, Freeze -> Mr. Freeze, Ivy -> Poison Ivy, Madhatter -> Mad
   Hatter, Quinn -> Harley Quinn, Ras -> Ra's al Ghul, Strange -> Hugo Strange, Talia -> Talia al Ghul,
   Twoface -> Two-Face, Vicky -> Vicki Vale, Zsazz -> Victor Zsasz, Unknown -> Unknown (speaker checked but not
   identified). A line in two lists gets both ("A; B").
 - **characters_inferred** - characters and roles *mentioned by the name* (token rules in `classify._TOKENS`,
   e.g. `BM`/`BATMAN` -> Batman, `PENG` -> Penguin, `TF` -> Two-Face, `THUG*`/`GENTHUG*` -> Thug, `TYGER` ->
-  TYGER guard; interview tapes add Hugo Strange). A hint for the ~7,500 lines without a verified speaker,
+  TYGER guard; interview tapes add Hugo Strange). A hint for the ~7,400 lines without a verified speaker,
   never shown as the speaker. In the editor the character filter matches verified speakers, and optionally
   inferred ones.
 
-Coverage (every language): 3,366 sounds with a verified speaker, 5,930 more with inferred characters, 1,520 with
-neither (mostly predator/efforts from unnamed thugs).
+Coverage of the 10,811 WAD voice lines (every voice language, the same lines): 3,366 with a verified speaker,
+6,258 more with inferred characters, 1,187 with neither (combat barks 407, story missions 267, open-world
+chatter 228, radio & TV 122, ...).

docs/decisions.md

@@ -3,7 +3,7 @@
 ## 1. Stack: Python + Qt (PySide6), packaged with PyInstaller
 - **Reuse:** the toolkit (`ackit`) is Python and proven against the real game; the editor imports it unchanged.
 - **Cross-platform GUI:** Qt Widgets via PySide6 (LGPL-3.0, official Qt for Python) looks native on
-  Windows and Linux, has a fast model/view table for 11,000+ rows, and runs offscreen for automated GUI tests.
+  Windows and Linux, has a fast model/view table for 20,000+ rows, and runs offscreen for automated GUI tests.
 - **Audio I/O:** Qt Multimedia (`QMediaDevices`, `QAudioSink`, `QAudioSource`) for playback, recording and
   device selection. It ships inside the PySide6 wheels with its own backends (WASAPI on Windows,
   PulseAudio/PipeWire on Linux), so unlike PortAudio/`sounddevice` nothing has to be installed on Linux.
@@ -42,6 +42,7 @@ loopback); it is a developer tool and not part of the editor.
   every record. Stream-pack padding bytes are stored so pack reverts are byte-identical.
 - **Full-file fallback:** before the first change to a WAD (and its pack index), one verified full copy is kept
   in the store (`full/`), default on, can be disabled. Used by "Restore from full backup" if the store is damaged.
+  Soundbank files, packages and localization files always get this copy (they are rebuilt from it, 4b/4c).
 - **Why the user data folder:** it survives app updates and reinstalls, needs no write access next to the game,
   and works the same on Linux/Proton. Steam never touches it.
 - **Safe writes:** every WAD is rebuilt into a temp file and atomically renamed over the original, pack indexes
@@ -53,15 +54,84 @@ loopback); it is a developer tool and not part of the editor.
 
 ## 4. Size rule
 Speech/ambience variations are addressed by offset+size from BKPK tables inside compressed `.upk` packages,
-which ackit does not rewrite. Replacements are written in place and must not exceed the original WEM's size;
+which ackit leaves unchanged (it patches packages only in place). Replacements are written in place and must not
+exceed the original WEM's size;
 the encoder lowers quality, then sample rate, to fit, and refuses otherwise. Music stream packs have a loose
-index (`SFX\*.bnk`) that ackit rewrites, so music can be any length. (Rewriting `.upk` packages would lift the
-limit but means LZO recompression of large engine packages: out of scope, high risk.)
+index (`SFX\*.bnk`) that ackit rewrites, so music can be any length. In-memory sounds follow the same rule
+(below). Lifting the limit would mean re-laying out the uncompressed packages (export offsets, BKPK/DIDX
+tables): high risk for little gain, not done.
+
+## 4b. In-memory sounds (soundbanks in loose `.bnk` files and `.upk` packages)
+- **Model:** one catalog row per Wwise media ID (archive `BANK`, target = media ID), not per bank: the same
+  media is stored in up to dozens of bank copies (banks are cooked into every package that uses them), and the
+  game may load any of them. A replacement is written into **every** copy (`BANK_locations.csv.gz`).
+- **In place:** each copy keeps its size; DIDX/HIRC offsets and the package layout never change, so nothing
+  else in the engine data has to be rewritten. Unlike WAD speech, an in-memory copy is never zero-padded:
+  the replacement fills its slot exactly with a label list chunk (the game plays a padded in-memory WEM as
+  silence; `docs/formats/banks.md`). Shrinking the declared sizes (DIDX and every Sound object's in-memory
+  size) would also work, but needs every HIRC source of the media in every bank copy located and rewritten.
+- **Packages:** ackit reads the LZO packages itself (pure Python, ~47 MB/s, block-level access) and writes
+  only the changed 128 KiB blocks, stored as a single LZO literal run (valid LZO1X without matches), then
+  fixes the chunk table. No compressor and no external tool are needed; untouched blocks stay byte-identical.
+- **Revert:** before the first change to a bank file a verified full copy goes to the store (mandatory, not
+  optional like the WAD copies: the file is rebuilt from it). The file content is always "original copy +
+  replacements applied now", so reverting the last one restores the original bytes exactly, and an interrupted
+  operation is repaired by the next one. Conflicts: the whole-file SHA-1 must be the original's or the last one
+  ackit wrote.
+- **Prefetch heads and whole copies:** 396 voice recordings (81 lines, 78-80 recordings per language) have
+  their first bytes copied into banks, and 49 more are stored whole in banks (same media ID, identical bytes);
+  a replacement of such a recording rewrites those copies too, otherwise the game could play the old line.
+  They are not listed as sounds of their own.
+- **Names:** no list names these media. Names come from Wwise events whose names are in the
+  packages' name and export tables (FNV-1 of the name = event ID), following event -> action -> containers ->
+  sounds (13,501 of 15,355, voice lines included); the other 1,854 are numbered per bank. Categories use the bank name. A localized speech line
+  has the same name in every language (its line, section 5).
+- **Localized speech:** speech banks hold one copy per voice language (4d); every media is labelled with the
+  language(s) whose banks hold it (a few recordings are shared by several languages) and the catalog shows
+  only the selected language's lines.
+
+## 4c. Subtitles and menu text
+- **Subtitles** are written in place into every copy of the dialogue object, NUL-padded inside the original
+  string (the game stops at the first NUL), so nothing in the package moves; the limit is the original
+  string's length ("room"). Moving the object to the end of the package (new export offset/size and an extra
+  chunk) would lift the limit, but no dialogue is loaded on the title screen, so the result could not be
+  checked automatically in the game: not done.
+- **Links** subtitle -> speech sound come from the soundbanks (event -> ... -> per-language media -> WAD line
+  or localized in-memory line), not from names: the `_NN` suffixes do not follow the WEM order inside WAD
+  entries. Scripted subtitles (`RSubtitle`) are linked through the game's own lookup table (FNV-1 of the line
+  name without `Speech_`). The subtitle objects' names also find events that no package name table lists, and
+  an event that plays two versions of a line links its subtitle to both.
+- **Menu text** files are plain UTF-16 text: no size limit; only the edited values change.
+- Both use the originals store like sounds: texts are recorded as strings (`texts` in store.json) and every
+  file is rebuilt from its original copy + the applied edits (sound and text edits combine in one package).
+- Keys (`SUB:<LANG>:<speech>:<line>`, `MENU:<LANG>:<File>:<Section>:<Key>`) are stable and used in mod packages.
+
+## 4d. One language setting
+The editor has a single *Language* list (top bar) for voices, subtitles and menu text, starting at the language
+the game is set to on every launch (not remembered: the game's setting is what the player hears). The game
+has voices in 5 languages and texts in 11; for the other 6 it plays the INT voices, so the editor loads the INT
+voice lines with that language's texts and says so next to the list. Localized in-memory speech (one bank
+copy per voice language) is filtered the same way, so the sound list never mixes languages.
 
 ## 5. Catalog
 - Shipped CSVs per archive (names, categories, characters, original size + checksum) make loading instant and
   enable change detection; any other build is scanned at runtime (header reads only, cached).
+- Names come from the game files only: a WAD entry or stream pack is named after its soundbank in the game's
+  `CookedBanks.txt` (entry ID = FNV-1 of the bank name; this gives exactly the names of the community hash list
+  the toolkit used to ship, which was removed), speech lines after their events (below).
 - Verified speakers come only from the user's lists; name-based character inference is kept in a separate field.
+- Voice lines are identified by the game's own data, not by names or positions (`docs/formats/banks.md`,
+  *Lines*): the recordings that are sources of the same Wwise Sound objects are one line, with exactly one
+  recording per language (12,434 lines, all 5 languages). The catalog stores that identity (`line`), and
+  speaker labels, mod packages (another language: the same `line`) and subtitles follow it. Positions are
+  useless across languages (WEMs are stored in media-ID order; 81.2-81.6 % of the WEMs in multi-variation entries
+  are at another position than in INT).
+- A line has one name in every language: the most specific speech event that plays it (10,174 of the 10,811
+  WAD voice lines), numbered `NAME#2` in the event's own order when it plays two versions; the other 637 keep
+  the name of their English (INT) recording's position. The position in each language stays an alias (`NAME#i`).
+- A recording stored twice in a WAD (5 per language) is written and reverted in both places.
+- The catalog records the ORIGINAL bytes: `tools/build_catalog.py` stops before writing anything if a sound
+  differs from the current catalog (a modified game), unless `--accept-changes` (a game update).
 - Category scheme: `docs/categories.md`.
 
 ## 6. Repository cleanup choices
@@ -78,7 +148,14 @@ limit but means LZO recompression of large engine packages: out of scope, high r
   store does not need it.
 
 ## 7. Editor UX choices
-- One window: filters (left), sound table (top), editor of the selected sound (bottom). The primary flow
+- One window with three tabs. *Sounds*: filters (left) and sound table on top, the editor of the selected
+  sound below across the full width, in three resizable columns (Listen | Replace | Subtitle). Inside each
+  column the controls are aligned, not packed in rows that wrap: Replace's edit fields in a grid with one
+  label column, and every column's actions in one full-width bar at its bottom, level across the three
+  columns (Listen: Original | Installed | Stop; Replace: Apply to game, twice as wide, | Revert | Save WAV;
+  Subtitle: Apply | Revert). Every row of buttons fills its column's width (Load file / device / Record; Trim
+  silence / Replacement; Show in list). A/B and Loop are icon buttons with tooltips next to the volume. The editor fits a 1000-pixel-wide window without horizontal scrolling (measured:
+  minimum 984 px with the window's vertical scroll bar). *Subtitles* and *Menu text*: search, filters, table, editor. The primary flow
   (search -> select -> Space to listen -> Load file/Record -> Apply) needs no menus; power features (tags,
   favorites, multi-select revert, mod packages, settings) are in the toolbar and context menu.
 - Long work (loading, encoding, writing 0.5 GB WADs) runs in background jobs with status messages; the
@@ -95,6 +172,30 @@ limit but means LZO recompression of large engine packages: out of scope, high r
 - Background: the wallpaper sits only behind the filter panel, scaled to the panel height so its left part fills
   the panel, under a light veil. Settings -> Background toggles it and sets its strength (default 10 %).
 - Loading feedback stays where the work is, never modal: a spinner inside the waveform being loaded (sound or
-  replacement file, decoded off the GUI thread), next to the Apply/Revert buttons while encoding/writing, and in
-  the status bar for game loading and bulk changes.
+  replacement file, decoded off the GUI thread), next to the Apply/Revert buttons while encoding/writing, over
+  a list while it loads (the sound list at start and on a language change, the subtitles / menu text of a
+  language, read off the GUI thread), and in the status bar for game loading and bulk changes.
+- Lists filter and sort in Python, not with Qt's QSortFilterProxyModel (`app/models.py: RowView`): the view
+  computes the visible rows in one pass and one sort over cached keys (~10-20 ms for 21,000 sounds; the Qt
+  proxy called back into Python for every comparison: up to 1.5 s per sort or widened filter). Rows a filter
+  shows again keep their sorted place; selection and the current row survive every change.
   All filter selectors (including Character) are plain, non-editable drop-downs so they look and behave alike.
+
+## 8. Verification
+- `pytest`: unit tests, a synthetic game (WADs, packs, packages with banks, localized banks, dialogue and subtitle
+  objects, localization files), a two-language game built like the real one (per-language BKPK tables, a media
+  ID shared by both languages, a recording stored twice, one also stored whole in a bank), offscreen GUI
+  tests, and read-only real-game tests when the game is installed.
+- `ackit verify` (WEMs rebuilt byte-for-byte, encode/decode), `ackit verify --catalog` (catalog vs game files,
+  and every relation re-derived from the game files with independent checks; it prints them as facts).
+- Real-game write round trips without launching the game: `tools/e2e_features.py [--lang all]` (every copy of
+  an in-memory sound, localized line, prefetch heads and whole bank copies of a line, subtitles incl. RSubtitle, menu text, sound + subtitle in
+  one package, a mod moved to another voice language; `all` = each of the 11 languages) and
+  `tools/e2e_editor.py` (through the editor's code path).
+- Documentation: `tools/check_docs.py [--scan] [--wems]` recomputes every figure the documents state (from the
+  catalog, the build's `index.json`, the menu files; with `--scan` from a fresh package scan and the catalog
+  check's facts; with `--wems` from rebuilding every WAD WEM) and fails on any difference, so figures are never
+  copied by hand.
+- In the running game: `ackit verify --ingame` (music heard), `--banks` (patched packages loaded, probe data in
+  memory), `--texts` (menu texts and Startup subtitles found in memory). Every tool reverts in a `finally` block
+  and checks that every touched file is byte-identical afterwards.

docs/formats/banks.md

@@ -2,32 +2,39 @@
 
 Most short sound effects (footsteps, foley, combat, weapons, objects, UI, grunts) are **not in the WADs**: they
 are WEMs stored inside Wwise soundbanks, either loose in `SFX\*.bnk` or embedded in the UE3 packages (`.upk`).
-The catalog and the editor only cover the WADs (`wad.md`), so none of these sounds are listed or editable yet.
-Survey of the installed game (buildid 3718300) on 2026-10-09, read-only.
+ackit lists them as the catalog archive `BANK` and writes them (see the end of this page).
+Survey of the installed game (buildid 3718300), 2026-10-09.
 
 ## Inventory
 
-| where | banks | unique in-memory WEMs | in the catalog |
-|-------|-------|-----------------------|----------------|
-| loose `SFX\*.bnk` (Batman/Catwoman footsteps, 32 surfaces each, + `TST_Prep_Media`) | 65 | 1,117 | 0 |
-| banks embedded in `.upk` packages (in 1,432 of the 2,650 packages) | 3,340 | 15,641 (~276 MB) | 0 |
+| where | banks | unique in-memory WEMs |
+|-------|-------|-----------------------|
+| loose `SFX\*.bnk` (Batman/Catwoman footsteps, 32 surfaces each, + `TST_Prep_Media`) | 65 | 1,117 |
+| banks embedded in `.upk` packages (in 1,432 of the 2,650 packages) | 3,340 | 14,802 |
 
-The two sets share 119 media IDs: about **15,800 unique sounds** in total. The WADs, by comparison, are fully
-covered: the BKPK tables in the packages reference 2,543 WAD entries, all of them in the catalog.
+All of them are in the catalog (archive `BANK`).
 
-Embedded WEMs by bank name prefix: `Speech` 7,872, `CHR` (character foley, combat, bosses) 3,896, `PSO` 781,
-`WPN` 664, `NPC` 475, `OBJ` 439, `SEQ` 265, `GUI` 208, `PSA` 203, `MTN` 196, `PX` 195, `DVC` 100, `GEN` 80,
-`PHY` 73, `AMB` 65, `SBJ` 50, others < 30. The `Speech_` banks (e.g. `Speech_EMOTE_Batman`, `..._GRUNT`) are
-probably efforts/grunts rather than dialogue (there are no per-language packages); not verified by listening.
-All WEMs seen are Vorbis (format tag 0xFFFF), the same codec as the WADs: the footsteps are mono (4 stereo),
-the embedded ones mono or stereo (~22 % stereo in a 2,533-WEM sample).
+The two sets share 119 media IDs: 15,800 unique media IDs in 96,011 places (bank copies). 396 of them are
+**prefetch heads** of streamed WAD sounds and 49 are **whole copies** of WAD recordings (below), so there are
+**15,355 in-memory sounds**. All copies of a
+media ID are byte-identical except `1C898731` (two variants). The WADs, by comparison, are fully covered: the
+BKPK tables in the packages reference 2,543 WAD entries, all of them in the catalog.
+
+Embedded WEMs by bank name prefix (a media held by banks of two prefixes counts for both): `Speech` 7,827, `CHR`
+(character foley, combat, bosses) 3,624, `PSO` 737, `WPN` 577, `NPC` 471, `OBJ` 411, `SEQ` 253, `PSA` 203,
+`PX` 195, `MTN` 191, `GUI` 155, `DVC` 100, `GEN` 80, `PHY` 73, `AMB` 61, `SBJ` 50, others < 30. The `Speech_` banks (e.g. `Speech_EMOTE_Batman`, `..._GRUNT`) hold
+speech: efforts, grunts and a few other lines stored once for every language (119 sounds) and **localized
+lines** (about 1,450 per language, see *Localized banks*).
+All WEMs seen are Vorbis (format tag 0xFFFF), the same codec as the WADs: the loose footsteps are mono (4
+stereo); of all 15,355 in-memory sounds 13,982 are mono, 1,196 stereo, 177 have 3-6 channels.
 
 ## Names
-- The community hash list names none of these media IDs; only the bank names are known.
+- Media IDs have no names of their own: a sound is named after the Wwise event that plays it (event names
+  are in the packages' name and export tables, event ID = FNV-1 of the name), else after its bank.
 - `CookedPCConsole\CookedBanks.txt` lists all 3,345 cooked bank names. Bank ID = FNV-1 32 of the lowercased
-  name (the same hash Wwise uses for every name). 3,338 of them are embedded in packages, 4 are loose `.bnk`
-  files (`Init`, `Global`, `Music`, `Music_PDLC`); `AMB_OW_A6`, `AMB_OW_C1`, `AMB_OW_E1`,
-  `Speech_EMOTE_Thug_Muffled_Stop` and `Speech_GFxUI_ArkhamCityStories` were found nowhere.
+  name (the same hash Wwise uses for every name). 3,338 of them are embedded in packages (`Music`, `Music_PDLC`
+  also as loose `.bnk` files), 2 are only loose `.bnk` files (`Global`, `Init`); `AMB_OW_A6`, `AMB_OW_C1`,
+  `AMB_OW_E1`, `Speech_EMOTE_Thug_Muffled_Stop` and `Speech_GFxUI_ArkhamCityStories` were found nowhere.
 - Loose footstep banks are named in their STID chunk (`CHR_BatmanFootsteps_Concrete_Media`, ...).
 
 ## Bank layout (version 56, Wwise 2011.1)
@@ -40,14 +47,65 @@ HIRC Sound object (type 2) payload: u32 object id, u32 plugin id (`0x00040001` =
 (0 = in memory, 1 = streamed, 2 = prefetch), u32 media id, u32 bank id of the media; for in-memory sources
 then u32 offset, u32 size, where offset = `DATA` payload position (from the start of the bank) + DIDX offset
 and size = DIDX size (checked on every in-memory source of the footstep banks and of `BmGame.upk`).
-A few sources use media of another bank. Every streamed media ID found is in a BKPK table, i.e. a WAD sound
-(`wad.md`).
+Source kinds over all bank copies: 115,861 in-memory sources of the bank's own media; 2,233 in-memory sources
+with bank id 0 whose media is not in the same bank (loaded by another bank); 133,075 streamed sources;
+2,101 **prefetch** sources. Every streamed or prefetched media ID is in a BKPK table, i.e. a WAD sound
+(`wad.md`). A prefetch source's media is also in the bank's DIDX: that in-memory copy is the **head** of the
+streamed WEM (its size = the source's size), which the game plays while the rest is read from the WAD. 396
+WAD sounds have such heads (2,101 copies); replacing one of them must rewrite the heads too.
+49 more WAD recordings (9-11 per language: game-over taunts, an encryption-card line, a cutscene line, scripted
+story lines) are
+also stored **whole** in banks: an in-memory source with the WAD recording's own media ID, byte-identical to
+the WAD WEM (84 copies). They are handled like heads: part of the WAD line, written and reverted with it, not
+sounds of their own (`catalog._whole_recordings`). Heads and whole copies together: 445 recordings, 2,185
+copies (`BANK_locations.csv.gz` kind `p`).
+
+## Localized banks
+Speech banks are localized inside the (language-neutral) packages: one `AkBank` export starts with a language
+list (FStrings `INT FRA ITA DEU ESN`) and then holds one block per language, in that order: the BKPK table(s)
+of the speech WAD entry for that language (`wad.md`) and a complete bank, with the same bank ID in every
+language. ackit gives each bank and each BKPK table the language of its position in that list (an export
+whose block count differs from its language count would be reported; none does). Each in-memory media gets
+the languages of the banks that hold it (`BANK.csv` column `language`): 8,092 sounds are language-neutral,
+1,450-1,456 belong to one voice language, 6 to several (5 shared by INT/FRA/ITA/ESN while DEU has its own, 1
+by INT/FRA/ITA); a media stored in the banks of every language is listed as `''` (the same recording
+everywhere; 167 of these are voice lines, e.g. efforts and radio chatter).
+A catalog shows the neutral sounds plus the loaded language's lines; names are unique per language.
+
+## Lines: one recording per language
+Implemented in `ackit/lines.py`, checked by `ackit verify --catalog`. A voice line exists once per language,
+and the game ties the recordings together itself:
+- Every WEM of a speech WAD has exactly one row in its own language's BKPK table (media ID, offset, size):
+  10,816 of 10,816 in each of the 5 languages, no row without a WEM. 657 media IDs are used by several
+  languages (cinematics, emotes, Clayface), each at another position in each language's WAD, so the tables
+  are kept per language and never merged.
+- Every recording, streamed, prefetched or in memory, is the source of a Wwise Sound object; Sound objects,
+  events, actions and containers have the same IDs in every language, only the media differ. A *line* = the
+  recordings that are sources of the same Sound objects: 12,434 lines (WAD and in-memory), each with exactly
+  one recording in each of the 5 voice languages (61,485 recordings), no Sound object with two recordings in
+  one language. Line identity (`line` column) = the smallest Sound object ID.
+- Prefetched sources count too: the 81 lines with a prefetch head (396 recordings over the 5 languages; mostly
+  the game-over taunts) are linked only through them. A WAD recording that a bank also stores whole (49) is
+  the same recording (same media ID), so its line is unchanged and the bank copies are written with it.
+- 5 recordings per language are stored twice in that language's WAD (same media ID in two entries, byte-
+  identical, e.g. `Speech_STEEL_A1_CH2_Thug_behinddoormuffled` and `DIA_SteelA1Muffled`): the second place
+  is a copy of the line (same `line`, keeps its position name) and is written and reverted with it, because
+  the game plays whichever place the loaded package's table names.
+- Not every line has its own recording in every language; the game shares some itself: in each voice WAD
+  616-651 lines are byte-identical to the INT recording (ITA 651: efforts and emotes 509, cinematics 117,
+  boss fights 10, story 8, radio/TV 4, character lines 2), 167 in-memory voice lines are stored once for every
+  language (efforts, radio chatter, crowds) and 6 DLC intro lines are one recording shared by INT/FRA/ITA(/ESN).
+  So some lines play the English recording in every language: that is the game's own data, which the catalog
+  shows as it is.
+- Independent checks (no BKPK involved): every WAD recording is named as a source by a Sound object of a bank
+  of its own language (54,055 of 54,055); every stored head or whole copy equals the start of the recording
+  its line gives it (2,185 of 2,185); a line's recordings have similar lengths across languages (11,761 of 12,434
+  within a factor 1.5; the others are different edits, e.g. a 0.55 s ESN line vs 5.46 s elsewhere).
 
 ## Banks inside packages
-- The packages are fully LZO-compressed (package version 805, licensee 101). For the survey they were unpacked
-  with Gildor's `decompress.exe` (gildor.org; used from the user's old modding folder, not part of this
-  repository) and searched for `BKHD` (banks) and `BKPK` (WAD indexes). `UnrealX.upk` (512 bytes) is not a
-  readable package (wrong tag, probably encrypted).
+- The packages are LZO-compressed UE3 packages (`upk.md`); ackit decompresses them itself and searches the
+  uncompressed data for `BKHD` (banks) and `BKPK` (WAD indexes). (The first survey used Gildor's
+  decompress.exe; `ackit/upk.py` gives identical data.)
 - Bank blobs in packages: 41,798 for 3,340 bank IDs. Many banks are copied into several packages (e.g.
   `CHR_CatwomanFoleySuit` 31 times, `CHR_BatmanFoleyBelt` 26). 302 bank IDs occur as several blobs with
   *different* media subsets (often one WEM each, several per package), so a bank's media set is the union
@@ -55,11 +113,20 @@ A few sources use media of another bank. Every streamed media ID found is in a B
 - Largest: `CHR_RobinFoleyCape` 300 WEMs, `CHR_BatmanFoleyCape` 282, `CHR_Clayface` 263,
   `CHR_NightwingFoley` 242, `CHR_CatwomanFoleySuit` 225, `CHR_MrFreeze` 176, `CHR_Combat` 166.
 
-## Consequences for editing (not implemented)
-- **Loose banks**: a replacement means rebuilding `DATA`/`DIDX` and the in-memory offsets/sizes in the HIRC
-  Sound objects of that bank (and in any other bank whose sources use its media). Whether the game accepts a
-  bank that changed size is untested.
-- **Embedded banks**: the package must be rewritten (decompress, patch every blob of the bank, recompress
-  LZO, fix the package's chunk table and export offsets/sizes). A sound must be replaced in **every** package
-  that carries a copy, otherwise the game may load the original from another package.
-- Either way: unverified in game so far; the same WEM rules apply (`wem.md`).
+## How ackit edits them
+- Catalog archive `BANK` (one row per media ID) + `data/catalog/BANK_locations.csv.gz` (every copy, and the
+  heads and whole copies of WAD speech recordings); built by `tools/build_catalog.py`, checked by `ackit verify
+  --catalog`.
+- A replacement is written **in place into every copy** (no larger than the smallest copy), so DIDX/HIRC
+  offsets and sizes and the package layout never change. Loose banks are patched directly; in packages only
+  the touched blocks are re-encoded (`upk.md`). A sound left in one package only would let the game load the
+  original from another one.
+- **Each copy is filled exactly**, never zero-padded: every in-memory WEM the game ships ends exactly at its
+  slot's end, and the game plays a replacement followed by zero padding there as **silence** (findings 24:
+  footsteps replaced that way were silent in game; filled, they play). A smaller replacement is grown to the
+  slot size with a label list chunk before `data` (`LIST{adtl, labl}`, the form the game's own WEMs use for
+  markers; `ackit/wem.py: fill`), so the encoder leaves at least 26 spare bytes, an even number.
+- Replacing a speech line that has prefetch heads or whole copies rewrites them: a head gets the first bytes
+  of the new, zero-padded line (the start of the streamed WEM as the WAD stores it); a whole copy is filled
+  like an in-memory sound.
+- Design and revert rules: `docs/decisions.md` (4b). The same WEM rules apply (`wem.md`).

docs/formats/subtitles.md

@@ -0,0 +1,60 @@
+# Subtitles (`RDialogueEvent` objects in the packages)
+
+There are no subtitle files: every speech line has a dialogue object (class `RDialogueEvent`, Rocksteady) inside
+the UE3 packages (`upk.md`), holding the subtitle text of all 11 languages. ackit reads them with
+`ackit/subtitles.py` and writes them through `Workspace.texts` (`ackit/texts.py`).
+
+## Inventory (buildid 3718300)
+- 14,744 dialogue objects in the packages for 9,032 speech lines (a line's object is copied into every package
+  that uses it; all copies are identical), plus 41 scripted subtitles (below): ~9,400 subtitle strings and
+  168,688 stored copies.
+- 11 languages: INT, FRA, ITA, DEU, ESN, ESM, POL, POR, RUS, JPN, KOR (the speech WADs exist only for INT, FRA,
+  ITA, DEU, ESN; the other languages play the INT voices with their own subtitles).
+- 9,035-9,667 subtitle strings per language: 94-98 % of the subtitles have one line (JPN: all), the rest up
+  to 5 lines.
+- Strings are stored as cp1252 (positive FString length) or UTF-16 (negative length): almost all RUS, JPN and
+  KOR strings and 85 % of POL are UTF-16; the other languages have 17-56 UTF-16 strings each.
+
+## Scripted subtitles (`RSubtitle`)
+41 more subtitles (radio and encryption-card lines) are `RSubtitle` objects in `Startup.upk`: the same layout,
+with the per-language array named `ArraySubtitles` and a float (duration) at `0x2C` instead of a speech
+reference. `RSubtitleLookup` (`Startup.upk`, `Lookups`) is an array of `{int key @0, object @4}`, key =
+FNV-1 of the speech line name without `Speech_` (e.g. `CH_ZERO_CH0_ENCRYPTION_CARD2_01`), which links each
+`RSubtitle` exactly to its line. ackit reads and writes both kinds the same way.
+
+## Object layout (compact tagged properties)
+The export data is a list of properties `u16 type, u16 field offset, value`, ended by `u16 0`. Types seen:
+`0x10` object reference (i32 export/import index), `0x0D` FString, `0x09` struct/array (name i32+i32 = struct
+or property name from the name table, i32 byte size, i32 array index, then `size` bytes), `0x04` 4-byte value,
+`0x03` bool, `0x01` enum. Relevant properties:
+
+| field | content |
+|-------|---------|
+| `0x10` @ `0x2C` | the speech event this line plays, e.g. export `Speech_CHURCH_B1_Ch2_POST_Explosion_BM_Cash_04` |
+| `0x0D` @ `0x34` | speaker, e.g. `Batman` (`ackit verify --catalog` compares it with the curated speaker labels) |
+| `0x09 LocalizedSubtitles` | u32 count, then per language: `0D 00 00 00` FString language code, `09 00 0C 00 Subtitles` struct, `00 00` |
+| `Subtitles` | u32 line count, then per line: `0D 00 00 00` FString text, `04 00 0C 00` 4-byte value, `00 00` |
+
+The sizes add up exactly in all 14,744 objects (checked by the parser: `SubtitleError` otherwise).
+
+## Link to the speech sound
+The speech event name is hashed (FNV-1) to the Wwise event ID and followed through the soundbanks (event ->
+action -> containers -> Sound objects, the same in every language) to that language's recording of the line
+(`banks.md`, *Lines*): a WAD variation or a localized in-memory sound. This is exact; the `_NN` suffix of the
+event name does **not** match the variation order inside the WAD entry (WEMs are stored in media-ID order).
+The names come from the packages' name and export tables (numbered names like `Speech_X_04` exist only as
+export names) and from the subtitle objects themselves: some events are listed in no name table, but their
+dialogue / RSubtitle object names them (Vicki Vale's radio reports, the Hush tape reveal).
+An event that plays two versions of a line (105 subtitled events, e.g. Mr. Freeze's lines) links its subtitle
+to both (`sound` = `key1;key2`). Linked: 103,304 of 103,658 strings in the 11 languages (INT 9,373 / 9,404),
+each to the same line in every language. The rest (24 speech names, 24-36 strings per language: Riddler trophy
+texts, Arkham City Stories, radio reports and tapes the game does not ship audio events for) have no event in
+any bank and stay editable without a sound link.
+
+## How ackit writes them
+- In place, into **every copy** (`data/catalog/SUB_locations.csv.gz`): the new text replaces the FString's
+  bytes and is padded with NUL characters, keeping the length field. The game stops at the first NUL, so no
+  size field, export or package layout changes. Limit: the new text must fit the original string's bytes
+  (cp1252: one byte per character; UTF-16: two) - shown as "room" in the CLI and the editor.
+- A cp1252 string can only hold cp1252 characters; UTF-16 strings can hold any character.
+- Packages are rebuilt from the store's original copy + all applied sound and text edits (`docs/decisions.md`).

docs/formats/text.md

@@ -0,0 +1,19 @@
+# Menu and UI text (`BmGame\Localization\<LANG>\*.<lang>`)
+
+The menus, HUD, character bios, riddles, credits and other UI strings are UE3 localization files, one folder
+per language (INT, ITA, FRA, DEU, ESN, ESM, POL, POR, RUS, JPN, KOR), ~37 files and ~7,000 strings each
+(`FRONTEND`, `GFxUI`, `Bios`, `Credits`, `PC`, `RIDDLER`, ...). ackit reads and writes them with
+`ackit/loctext.py` through `Workspace.texts`.
+
+## Format
+- UTF-16LE with a BOM (`FF FE`), CRLF line ends; the first line is empty. (`INT\Version.int` and
+  `VersionData.int` are plain ASCII build IDs, not UI text: ackit ignores them.)
+- `[Section]` lines, `Key=Value` lines, `;` comments. A few values are in double quotes (`Quit="Quit game"`).
+- File extension = language code (`FRONTEND.int`, `FRONTEND.FRA`); case varies, lookups are case-insensitive.
+- Markers inside values (e.g. `\n`, button tags) are the game's own; ackit keeps them as plain text.
+
+## Keys and writing
+- Key: `MENU:<LANG>:<File>:<Section>:<Key>` (`#n` for a repeated key; none in the shipped files).
+- Writing replaces only the value part of the edited lines (quotes kept); BOM, encoding, other lines and line
+  ends stay byte-identical. Values cannot contain line breaks; characters must fit the file's encoding.
+- The files are rebuilt from the store's original copy + the applied edits, so reverting restores them exactly.

docs/formats/upk.md

@@ -0,0 +1,48 @@
+# UE3 packages (`*.upk`)
+
+Arkham City's 2,650 packages in `BmGame\CookedPCConsole` are Unreal Engine 3 packages, version 805,
+licensee 101. 2,647 are fully LZO-compressed, 2 are stored uncompressed, and `UnrealX.upk` (512 bytes)
+is not a readable package. ackit reads and patches them with `ackit/upk.py` and `ackit/lzo.py` (no external
+tools). Soundbanks inside them: `banks.md`.
+
+## Summary (file start, little-endian)
+
+| field | notes |
+|-------|-------|
+| u32 tag `0x9E2A83C1`, u16 version 805, u16 licensee 101, i32 TotalHeaderSize | |
+| FString FolderName | i32 length incl. NUL, then bytes (`None`) |
+| 12 x i32 | PackageFlags, NameCount, NameOffset, ExportCount, ExportOffset, ImportCount, ImportOffset, DependsOffset, ImportExportGuidsOffset, ImportGuidsCount, ExportGuidsCount, ThumbnailTableOffset |
+| 16 bytes Guid | |
+| i32 N + N x {ExportCount, NameCount, NetObjectCount} | generations (N = 1) |
+| u32 EngineVersion 7748, u32 CookerVersion, u32 CompressionFlags | 2 = LZO, 0 = none |
+| i32 C + C x {i32 UncompressedOffset, UncompressedSize, CompressedOffset, CompressedSize} | chunk table |
+| u32 PackageSource, ... | not used by ackit |
+
+All offsets in the summary (names, exports, ...) refer to the **uncompressed** package. The uncompressed
+package = the summary without the chunk table, followed by the chunks' data at their UncompressedOffset
+(the first chunk starts right after the summary, at NameOffset).
+
+## Name, import and export tables (uncompressed package, at NameOffset / ImportOffset / ExportOffset)
+- Name: FString (positive length: ANSI incl. NUL; negative: UTF-16) + u64 flags.
+- Import (28 bytes): ClassPackage (name index, number), ClassName (index, number), i32 OuterIndex,
+  ObjectName (index, number).
+- Export: i32 ClassIndex, SuperIndex, OuterIndex, ObjectName (index, number), ArchetypeIndex, **one extra u32
+  (licensee 101)**, u64 ObjectFlags, i32 SerialSize, SerialOffset, u32 ExportFlags, i32 count + count x i32
+  (net objects), 16-byte guid, u32 package flags: 72 bytes when the net-object list is empty. Checked on every
+  export of the game. A name with number n is shown as `NAME_<n-1>` (e.g. the speech event `..._04`): such full
+  names exist only in the export table. `Package.tables()` returns names, imports and exports.
+
+## Compressed chunks
+Each chunk (at CompressedOffset): `u32 tag, u32 block size (0x20000), u32 compressed sum, u32 uncompressed
+sum`, then one `{u32 compressed size, u32 uncompressed size}` per block, then the blocks. Every block is an
+independent LZO1X stream. In all game packages the chunks follow each other without gaps, start right
+after the summary and end at the end of the file (checked for every package).
+
+## How ackit reads and writes them
+- Positions of sounds are offsets in the uncompressed package, the same as in a decompressed copy
+  (e.g. Gildor's decompress.exe output). Reading decompresses only the blocks that are needed.
+- Writing (`Package.patched`): every changed block is stored again as a single LZO literal run (a valid
+  LZO1X stream without matches, `lzo.compress_stored`), the chunk headers of the changed chunks and the
+  CompressedOffset/CompressedSize of all following chunks are updated; every other byte is copied, so a
+  package without changes is rebuilt byte-for-byte. Replacements never change the uncompressed layout
+  (sounds are replaced in place), so names, exports and their offsets stay valid.

docs/formats/wad.md

@@ -21,19 +21,27 @@ Rules:
 - Each payload starts at the next **0x1000-aligned** offset after the previous one.
 - `size` = payload length rounded up to **0x800**; padding is zero.
 - File ends 0x1000-aligned; header area after the table is zero.
-- `hash` = Wwise media/file ID (names: `hashlist.txt`, e.g. `CHR_RadioScanner`, `AMB_Steel_B3`, `Music`).
+- `hash` = Wwise ID of the soundbank whose streamed media the entry holds: FNV-1 of a bank name listed in
+  `CookedPCConsole\CookedBanks.txt` (e.g. `CHR_RadioScanner`, `AMB_Steel_B3`, `Music`); this names every
+  entry the game uses (2,544 of 2,547; ackit takes the names from there).
 
 Payload kinds:
-- **LOC WADs**: each entry is a group of WEMs, the variations of one line (ITA: 10816 WEMs in
-  2464 entries; e.g. `CHR_RadioScanner` holds 91). Each WEM starts at the next 0x800 boundary after
-  the previous one (zero padding); 3 entries pack two WEMs back to back. Addressed `HASH:i`.
+- **LOC WADs**: each entry is a group of WEMs, the variations of one line (every language: 10,816 WEMs in
+  2,464 entries; e.g. `CHR_RadioScanner` holds 91). Each WEM starts at the next 0x800 boundary after
+  the previous one (zero padding); 3 entries pack two WEMs back to back. Addressed `HASH:i`. The WEMs of an
+  entry are stored in media-ID order and the languages have different media IDs, so variation `i` is
+  usually a different line in each language (81.2-81.6 % of the WEMs in multi-variation entries, compared with
+  INT; e.g. `Speech_TAPE_JOKER_1_04` is variation 1 in INT and ITA, 8 in FRA, 6 in DEU, 5 in ESN). Which recording is which line comes from the banks
+  (`banks.md`, *Lines*), not from positions.
 - **SFX WAD**: 79 grouped ambience/effect entries (260 WEMs) plus 4 **stream packs** (below).
 
 Where the game gets the inner offsets: the entry IDs appear next to `BKPK` headers inside the
-LZO-compressed UE3 packages (e.g. `BmGame.upk`, `OW.upk`, `Church_B1_*.upk`). Those embedded BKPK
-tables give each WEM's offset **and size**. ackit does not rewrite `.upk` files, so these WEMs are
-replaced **in place, no larger than the original** (zero-filled to the original size), and the WAD
-layout never changes. The user's earlier quickbms-reimport mod did the same and was loaded by the game.
+LZO-compressed UE3 packages (e.g. `BmGame.upk`, `OW.upk`, `Church_B1_*.upk`). A speech entry has one table
+per language, inside a localized `AkBank` export (its k-th table belongs to the k-th language the export lists;
+the SFX WAD's 79 tables are outside such exports). Those embedded BKPK tables give each WEM's offset **and
+size**; in every language each WEM has exactly one row (`banks.md`, *Lines*). ackit never changes them,
+so these WEMs are replaced **in place, no larger than the original** (zero-filled to the original size), and
+the WAD layout never changes. The user's earlier quickbms-reimport mod did the same and was loaded by the game.
 Only the 4 SFX stream packs, whose BKPK index is a loose `.bnk`, can change size.
 
 ## BKPK stream packs (`SFX\<ID>.bnk` + SFX WAD entry `<ID>`)
@@ -61,9 +69,9 @@ end rounded up to 0x800. Padding is mostly zero (a few non-zero bytes in `EDF036
 original bytes where nothing moved).
 
 Replacing a pack media (`ackit` target `EDF036D6/393BC53A`) rewrites the pack table (`.bnk`), the
-WAD entry payload, and the WAD table/offsets. Verified in-game (NOTES.md, E2).
+WAD entry payload, and the WAD table/offsets. Verified in-game (findings E2, E14).
 
 ## Regular soundbanks
 
 The other `SFX\*.bnk` files and the banks embedded in `.upk` packages hold their WEMs in memory (not in a
-WAD); see `banks.md`. The toolkit does not read or modify them yet.
+WAD); ackit lists and writes them as the catalog archive `BANK` (`banks.md`).

docs/formats/wem.md

@@ -1,10 +1,16 @@
 # WEM format accepted by Arkham City (Wwise 2011.1, Vorbis)
 
 Derived from the game's WEMs; `ackit verify` / `tests/test_realgame.py` rebuild them byte-for-byte from these rules
-alone (all 10816 ITA WEMs; in SFX only the 4 looping music tracks with loop points differ).
+alone. Checked on every WEM of the 5 speech WADs: 54,079 of 54,080 are identical; the exception (INT
+`FE9963E8:74`) differs only in the informational avg-bytes/s field (one lower than the formula below). In SFX
+626 of 651 are identical; the others are 14 music tracks whose seek table follows another placement (one entry per
+1,024-1,103 samples), 5 music tracks whose sample count exceeds what their packets hold, and 6 4-channel
+ambience WEMs with a `smpl` (loop) chunk. These rules are not needed to write sounds the game plays (replaced
+music, built with the rules here, is heard in game: `ackit verify --ingame`), only to reproduce those originals.
 
 ## RIFF layout
-`RIFF` size `WAVE`, then `fmt ` (0x42 bytes), optional `cue `/`LIST` (37 files), `data`.
+`RIFF` size `WAVE`, then `fmt ` (0x42 bytes), optional `LIST` + `cue ` (659 WEMs), `LIST` + `cue ` + `smpl` (1 WEM),
+`cue ` (18 WEMs), `cue ` + `smpl` (3 WEMs), `smpl` (41 WEMs), `data`.
 Odd-sized `data` chunks are **not** padded.
 
 ### `fmt ` (0x42)

docs/history/mission3_audio_text_prompt.md

@@ -0,0 +1,35 @@
+# CONTINUE: Finish full-game audio editing, then add subtitle and menu-text editing
+
+Same autonomous rules as before: I won't answer questions, so decide, log assumptions in `NOTES.md`, and continue. Start by reading `NOTES.md`, `CLAUDE.md`, `docs/` and `docs/decisions.md` to restore context, and apply the new findings from the last session. Keep the Steam, backup and byte-identical-originals rules.
+
+## 1. Complete audio editing for every sound in the game
+- Make editing work for every audio file in the game (all WADs and languages), using the new findings. Handle edge cases explicitly (different codecs/variants, streamed vs. in-memory, sizes, alignment) in `ackit`, not in app-side workarounds.
+- Make sure every single audio file is cataloged and the catalog is correct: add a verification command and test that cross-checks the catalog CSVs against the real WADs (missing, extra, duplicate, wrong character, wrong category, wrong language). Fix the data, and report remaining unknowns in `docs/`.
+
+## 2. Subtitles and menu text editing
+- Reverse engineer where and how the game stores subtitles and menu/UI text (they are text files in the game folders). Document the formats and encodings in `docs/`, and implement parsing and writing in `ackit` (with safe backup/revert, as for audio).
+- In the editor:
+  - **Subtitles section**, linked to audio. Each sound shows its subtitle, and each subtitle links back to its sound. Editing both together should be convenient: search by text, character and category, show both side by side in the single-file editor, and bulk edit/find-and-replace.
+  - **Separate "Menu text" section**, kept apart from audio and subtitles, with its own search, filters and per-string editing.
+- Support revert (per entry and all), mod export/import for text, and detect when game files changed underneath.
+
+## Quality rules
+- Do it properly: no hacks. If the editor UI or `ackit` need major restructuring to fit this cleanly, do it, and keep the repository organised as it is now (clear modules, no duplicated code, no leftover scripts).
+- Add the necessary tests: unit, headless app-logic, GUI smoke tests, and a real end-to-end test against the game for audio and for text (apply, check the game starts and shows the change, revert, byte-identical).
+- Document everything and update the existing documents (`README`, `docs/`, `decisions.md`, user guide, `NOTES.md`, `CLAUDE.md`), including the new catalog schema and the text file formats.
+- Update the relevant docs and README in the same commit as each change, not at the end. A stage is not finished until its docs, tests and `NOTES.md` status are updated.
+- Rebuild the packaged distribution (with bundled tools and licenses) and re-verify it.
+
+## Saving quota (I'm on a limited Claude Pro quota; this is a hard requirement)
+- Plan before acting: write a short ordered plan in `NOTES.md`, work in small stages, and commit after each. Finish and verify one stage before starting the next, so a usage pause never leaves broken work.
+- Keep context small: never print whole files, big logs, WAD listings or catalogs. Use grep, head/tail, counts and summaries, and read only the line ranges you need. Write long outputs to files and inspect them selectively.
+- Don't re-read what you already know: rely on `NOTES.md` and `docs/` instead of re-deriving or re-reading large sources. Update them as you go so a context reset is cheap.
+- Use cheaper subagents (Sonnet or Haiku) for mechanical and bulk work: catalog cross-checks, CSV fixes, log triage, repetitive edits, test runs, and doc updates. Keep the main Opus context for reverse engineering, design decisions and tricky debugging.
+- Run the cheapest check that proves the point: run the specific failing test first, not the whole suite after every edit. Run the full suite only at stage boundaries.
+- Run the expensive real-game end-to-end tests (launching the game) sparingly: once per stage after the headless tests pass, plus the full pass at the end. Batch many changes into one game run where possible.
+- Avoid GUI screenshots and image reads unless a layout problem needs them. Prefer headless assertions.
+- Don't rebuild the distribution until the end, except for a single early build to check the pipeline.
+- Avoid repeated failed attempts: after two failures on the same problem, write down the hypotheses tried, then change approach instead of retrying.
+- Order of priority if the quota runs short: (1) correct, complete audio editing and catalog, (2) subtitle editing linked to audio, (3) menu text section, (4) tests and docs for each as you go, (5) packaging. Always leave `NOTES.md` with a clear "Current status / Next steps" so work can resume cleanly after a pause.
+
+Finish with a short summary and a `REPORT.md` of anything unresolved.

docs/reverse-engineering/findings.md

@@ -70,8 +70,70 @@ Conclusion: approach A works end-to-end in-game. Use >= 30 s probes for the intr
     expects WAVEFORMATEX order (FL FR FC LFE BL BR) is unverified (identical-channel probes were used).
 16. (2026-10-09) The WADs hold only streamed audio. About 15,800 more sounds (footsteps, foley, combat, weapons,
     UI, grunts) are in-memory WEMs inside soundbanks: 65 loose `SFX\*.bnk` (1,117) and 3,340 banks embedded in
-    the `.upk` packages (15,641; many banks copied into several packages). None are in the catalog; the WAD
-    side is complete. Inventory, layout and editing consequences: `docs/formats/banks.md`.
+    the `.upk` packages (14,802; many banks copied into several packages). None were in the catalog then (they
+    are now: archive BANK); the WAD side was complete. Inventory and layout: `docs/formats/banks.md`.
+17. (2026-10-09) Speech lines have **per-language media IDs**: every package carries one BKPK table per language
+    for the same WAD entry. 396 lines (78-80 per language) have a prefetch head (first N bytes) in banks; each
+    head equals the start of exactly one language's WEM.
+18. (2026-10-09) In-game test of in-memory sounds (`ackit verify --ingame --banks`): 77 probes written into the
+    78 sounds stored in the Frontend*/Startup packages (54 files: patched LZO packages + loose banks). The game
+    starts and runs, all 68 probe byte patterns are found in its memory (it loaded the patched packages), revert
+    byte-identical: PASS. Not shown: a probe being *heard*. Only 7 Vorbis streams start on the title screen
+    (setups 4f708975, 6ded1c65, 92f94a3f, d93b5944, d4b2e824: shared by hundreds of sounds, music included) and
+    none is a probe. Tried: Enter/arrow keys posted to the game window (WM_KEYDOWN/WM_KEYUP; that helper was removed later) to reach
+    the main menu: no UI sound followed (the game probably reads input another way; global SendInput is not
+    used on purpose); a breakpoint on the setup parser entry: no extra events. Hypotheses: no in-memory sound
+    plays without input; the decode path is the same as for WAD sounds (same WEM rules, 300/300 in-memory WEMs
+    rebuild byte-identically), so playback is expected to work like E4/E10.
+19. (2026-10-09) In-game text test (`ackit verify --ingame --texts`, ITA): 12 menu texts of FRONTEND/GFxUI
+    prefixed with markers and 12 subtitles patched in place (21 files). All 12 markers are found as UTF-16 in the
+    game's memory after ~45 s (the localization files are read at startup), no crash, revert byte-identical:
+    PASS. Subtitles are not loaded on the title screen (no dialogue object in Frontend*/Startup/BmGame).
+20. (2026-10-10) **Variation positions differ between languages.** A WAD entry stores its WEMs in media-ID order
+    and each language has its own media IDs, so for 7,010 of 9,366 lines (75 %) the same game line is at a
+    different position in the ITA and INT entries (e.g. Speech_CH_ZERO_CH0_fight_intro_01 = ITA :0, INT :2).
+    Position names (NAME_i, wemscan) therefore mean different lines per language. Speech sounds are now named
+    after their speech event (same in every language); the curated ITA lists are mapped through it. Independent
+    check: the speaker field of the dialogue objects agrees with the mapped labels for 955 / 955 lines.
+    (Superseded by 22: the identity of a line now comes from the Sound objects, for every recording; measured
+    that way, 81.2-81.6 % of the WEMs in multi-variation entries are at another position than in INT.)
+21. (2026-10-10) In-memory speech banks are localized (language list + one bank per language in one AkBank
+    export, checked on 215 banks against the WAD languages of their streamed lines); 41 more subtitles are
+    RSubtitle objects linked through RSubtitleLookup (key = FNV-1 of the line name without `Speech_`). Subtitle
+    coverage by category is uneven in the game itself: efforts 0 %, combat barks 11 %, cinematics 10 % (cutscene
+    lines are mostly not reached by speech events and have no dialogue object), story/predator/open world ~90 %+.
+    (With every version of a line linked, item 22: combat barks 9 %, cinematics 23 %; REPORT.md.)
+
+22. (2026-10-10) **One recording per language, from the game data alone** (`docs/formats/banks.md`, *Lines*).
+    The BKPK tables of a speech entry sit inside the localized AkBank export, one per declared language; they
+    used to be merged by media ID, but 657 media IDs are used by several languages at different positions, so
+    the last language read (ESN) overwrote the others (120-157 WEMs per language looked unreferenced). Kept per
+    language: 10,816 / 10,816 WEMs matched in every language. Prefetch sources were left out of the Sound-object
+    map, which hid the 396 prefetched lines (game-over taunts). Five recordings per language are stored twice
+    (same media ID in two entries, identical bytes). Subtitle objects name events no package lists (FNV-1 of the
+    name = event ID: Vicki Vale's radio reports, Hush tape reveal). Result: 12,434 lines with exactly one
+    recording in each of the 5 languages, no conflict; independent checks: 54,055 / 54,055 WAD recordings are
+    sources in a bank of their own language, 2,101 / 2,101 prefetch heads equal the start of their recording,
+    every linked subtitle is on the same line in all 11 languages.
+23. (2026-10-10, final check) **Whole WAD recordings in banks.** 49 in-memory media (9-11 per language) have the
+    media ID of a WAD recording of their bank's language and are byte-identical to it (game-over taunts,
+    `Speech_CH_ZERO_CH0_ENCRYPTION_CARD3_01`, `Speech_SEQUENCE_CH2_RESCUE_DOC_01`, ...): the line's Sound
+    object names that media, and some bank also stores the whole WEM in memory. They were listed as separate BANK sounds (with the
+    line's name: the CLI called the name ambiguous), so a replacement of the line left the in-memory copies
+    old. Now they are bank copies of the WAD recording like prefetch heads (84 copies; 445 recordings with
+    heads or whole copies, 2,185 copies), written and reverted with the line. Also: 8 in-memory sounds per
+    catalog had the same `<Bank>_<n>` name as an SFX WAD sound of the same bank (`CHR_RasBossFight_1`,
+    `SEQ_DemonTrials_1`); they now take `<Bank>_<MEDIA>`, and a line's own name wins over another recording's
+    position alias (`CHR_RadioScanner#1` in FRA). Every name of every catalog resolves to its own sound.
+24. (2026-10-10, gameplay test by the user) **Padded in-memory replacements play as silence.** All 568 Batman
+    footsteps were replaced; in game the original steps were heard and the replaced cape-open sound was silent.
+    Memory scan while walking: no original footstep WEM loaded, 137 replacements loaded; debugger: the
+    replacements' Vorbis setups are created and parsed OK (ret 0). Every untouched in-memory WEM fills its slot
+    exactly (RIFF size = DIDX size, 500 / 500 sampled), while the replacements were zero-padded. Filled with a
+    `LIST{adtl, labl}` chunk before `data` (55 in-memory originals carry cue+LIST chunks), every step plays the
+    replacement. ackit now fills every in-memory copy and every whole bank copy of a WAD recording
+    (`wem.fill`); WAD entries and prefetch heads keep zero padding (streamed playback is unaffected). The
+    remaining sound after each step is Batman's gear and cape foley (`CHR_BatmanFoleyBelt`, `..Cape`).
 
 ## Old path -> new path
 `scripts/harness.py` -> `ackit verify --ingame` (ackit/verify/ingame.py); `scripts/validate_builder.py` ->

docs/user-guide.md

@@ -1,7 +1,7 @@
 # Unheard City - user guide
 
-Replace voice lines, music and ambience in **Batman: Arkham City (PC, Steam)**. Everything you change can be put
-back with one click.
+Replace voice lines, music, ambience and sound effects, and edit subtitles and menu text in **Batman: Arkham
+City (PC, Steam)**. Everything you change can be put back with one click.
 
 ## Install and start
 1. **Windows:** unzip the download and double-click **Install.cmd**. It installs Unheard City for your user (no
@@ -16,35 +16,62 @@ back with one click.
 
 ![Main window](img/main.png)
 
+## Language
+There is **one language setting**: the **Language** list at the top of the window. It starts at the language the
+game is set to (Steam > game properties > Language) and applies to everything you edit:
+- **Sounds**: the voice lines of that language, plus the sounds every language shares (music, ambience,
+  effects). The game has voices in English (INT), French, Italian, German and Spanish; for the other languages
+  (Latin American Spanish, Polish, Portuguese, Russian, Japanese, Korean) it plays the English voices, so the
+  editor shows those and says *voices: INT* next to the list.
+- **Subtitles** and **Menu text**: the texts of that language.
+Changing it reloads the lists; the next start is again in the game's language.
+Some voice lines sound English in every language: the game itself ships the English recording for most
+efforts and grunts, many cinematics and a few other lines, and some in-memory lines exist only once for all
+languages. The editor shows what each language's game files contain.
+
+## Names
+Every voice line has one name in every language, taken from the game itself (e.g. `Speech_TAPE_JOKER_1_04`, the
+same as in the Subtitles tab): the line `Speech_X` in English is the same line `Speech_X` in French, Italian,
+German and Spanish. Inside a WAD entry the game stores the variations of a line in a different order in each
+language, so the position-based name is kept only as a second name (`Speech_TAPE_JOKER_1#4` = the 4th WEM of
+that entry in this language; search finds both). When the game plays one of two versions of a line, the
+second is shown as `NAME#2`. About 6 % of the lines are named by no game event; they take the name of their
+English recording in every language. A few recordings are stored twice in the game files: both rows belong to
+the same line and are changed and reverted together.
+
 ## Find a sound
 - Type in the **search box** (Ctrl+F): words can be parts of the name, a category, a character, your own
   notes or tags, or an ID like `b49cdf48`. All words must match.
 - Narrow down with **Group** (Speech, Music, Ambience, SFX), **Category** (e.g. *Interview tapes*,
-  *Combat barks*, *Cinematics*), **Character**, **Archive** (speech language or SFX), **Tag**, **Modified
+  *Combat barks*, *Cinematics*), **Character**, **Archive** (speech language, SFX or BANK), **Tag**, **Modified
   only** and **Favorites only**. Combine them freely; results update instantly.
 - **Character** is the verified speaker from the curated catalog. Grey `~` names are only *mentioned* in the
   sound's name (useful hints for the many unlabelled lines); tick *Include characters mentioned in names*
   to filter on them too.
 - Click a column header to sort (length, size, status, ...). Ctrl/Shift-click selects several rows.
-- Most short effects (footsteps, foley, weapons, menu sounds) are stored inside soundbanks and are not listed
-  yet; the list covers voice lines, music, ambience and the effects of the SFX archive.
-- **Language** (top left) switches the speech language. Speech is read from the language the game is set to.
+- Archive **BANK** holds the short sounds stored inside soundbanks (footsteps, foley, combat,
+  weapons, menu sounds, grunts). Many are named after the game event that plays them (`Play_...`), the
+  rest after their bank (`CHR_BatclawGrapple_3`). The editor shows the bank and how many copies the game keeps of
+  the sound; a replacement is written into every copy.
 
 ![Browsing by category](img/browse.png)
 
 ## Listen
-Select a row and press **Space** (or double-click) to play the original. In the editor below:
-- **Original** / **Installed** play the game's original and what is currently in the game. Click on a
-  waveform to play from that point. **Loop**, **Volume** and **Stop** (Esc) work for everything.
-- **A/B** switches between the original and your replacement at the same position, to compare.
+The editor under the list shows the selected sound in three columns side by side (drag the dividers to
+resize them): **Listen**, **Replace** and **Subtitle**. Select a row and press **Space** (or double-click) to
+play the original. Each column ends with its own buttons on one line. In **Listen**:
+- **Original** / **Installed** play the game's original and what is currently in the game; **Stop** (Esc).
+  Click on a waveform to play from that point.
+- Above them: **A/B** (switch between the original and your replacement at the same position, to compare),
+  **Loop**, **Volume** and the playback position. The icon buttons show their names when you point at them.
 
 ## Replace
 ![Editor](img/editor.png)
 1. **Load file…** (Ctrl+O): WAV, FLAC, OGG, OPUS, MP3 or AIFF. (M4A/WMA work only if ffmpeg is installed.)
    Or choose a microphone in the list and press **Record**, then **Stop recording**. Listen with
    **Replacement**; record again as often as you like.
-2. Adjust: **Keep from / to** trims (or right-drag on the replacement waveform), **Trim silence**, **Gain**,
-   **Normalize**, **Match original loudness** (recommended for voice lines).
+2. Adjust: **Keep … to** trims (or right-drag on the replacement waveform), **Trim silence**, **Gain**,
+   **Normalize**, **Match loudness** (to the original; recommended for voice lines).
 3. **Apply to game** (Ctrl+S). The audio is converted to the original's format and written into the game files.
    The row turns bold and shows *Modified*.
 
@@ -54,23 +81,53 @@ you are asked to shorten it. Music (`Music_…`) can be any length. A warning ap
 than the original, since the game may cut it off.
 
 ## Undo
-- **Revert to original** in the editor puts that sound back exactly.
+- **Revert** in the editor puts that sound back exactly.
 - Right-click rows → **Revert selected**; toolbar **Revert all** puts every original back.
 - Originals are kept in your user data folder (`%APPDATA%\UnheardCity` on Windows,
   `~/.local/share/unheardcity` on Linux), so reverting works after updating or reinstalling the app too.
-  Uninstalling keeps this folder; use **Revert all** before uninstalling if you want the game back to original. Before the first
-  change to an audio archive, one full copy of it is also kept there as a fallback (Settings).
+  Uninstalling keeps this folder; use **Revert all** before uninstalling if you want the game back to original.
+  Before the first change to a game file, a full copy of it is kept there too: optional for the WAD archives
+  (Settings), always for soundbanks, packages and text files (they are rebuilt from that copy).
 - If Steam "Verify integrity of game files" or an update undoes your changes, a bar offers **Re-apply my
   changes** or **Forget them**. Do not use "verify" to undo mods; use Revert all.
 
+## Subtitles
+The **Subtitles** tab lists the subtitle of every speech line in the selected language (see *Language*).
+- Search by text or line name, filter by **Character** and **Category** (taken from the linked speech sound),
+  or show **Modified only**. Select a row to edit its text below and press **Apply** (or Enter).
+- Each subtitle replaces the original *in place*, so it can be at most as long as the original (the counter
+  shows *used / room* characters). Most subtitles are stored in the Western European character set
+  (Windows-1252: accents are fine, other scripts are not); those stored as Unicode (almost all Russian,
+  Japanese and Korean ones, most Polish ones) accept any character. The editor says so when a character does
+  not fit.
+- **Go to the sound** opens the speech line in the Sounds tab. In the Sounds tab, the editor of a speech line
+  shows its subtitle in the **Subtitle** column next to the audio: edit both side by side, **Apply** /
+  **Revert** the subtitle there, or **Show in list** to jump back to the Subtitles tab.
+- **Find and replace…** works on the rows the filters show (texts that would not fit are listed and skipped).
+- **Revert** / **Revert all texts here** put the original text back (every copy in every game file).
+- About 30 subtitles per language have no sound in the game (texts such as the Arkham City Stories, unused
+  radio reports); they have no linked sound. A subtitle of a line the game plays in two versions belongs to
+  both. The game itself has no subtitles for efforts and grunts and for most combat barks and cutscene lines.
+
+## Menu text
+The **Menu text** tab lists every menu, HUD, bio, riddle and credits string of the selected language (about
+7,000), grouped by **File** (e.g. `FRONTEND`, `GFxUI`, `Bios`). Search, edit, apply and revert work like
+subtitles; menu texts can be of any length but cannot contain line breaks (keep the game's own markers such
+as `\n` as they are).
+
 ## Notes, tags, favorites
 Right-click → **Notes & tags…** (F2): free text and comma-separated tags, searchable and filterable.
 Click the **★** column to mark favorites. These are stored in your user data folder, not in the game.
 
 ## Share your changes (mods)
-- **Export mod…**: choose which changed sounds to include and save a `.acmod` file (the exact game data plus
-  the audio, so it also works for another language whose original has a different size).
+- **Export mod…**: choose which changed sounds and texts to include and save a `.acmod` file (the exact game
+  data plus the audio, so it also works for another language whose original has a different size).
 - **Import mod…**: shows which sounds match this game, then installs them. You can revert them like any change.
+- Changed subtitles and menu texts are exported and imported with the sounds (listed as *Text*); a text stays
+  in its language (a French subtitle edit changes the French subtitle).
+- In another language, voice lines are matched by the game's own identity of the line, so the same line is
+  replaced even where its position differs. Voice lines from packages made with versions before 3.1 carry no
+  such identity: the import lists them instead of guessing (export the mod again with this version).
 
 ## Settings
 ![Settings](img/settings.png)
@@ -85,6 +142,7 @@ The editor panel opens tall enough to show all its controls; drag the divider ab
 | "Cannot write … close the game" | quit Arkham City (and anything else using the file) and try again |
 | "not a Batman: Arkham City folder" | pick the folder that contains `BmGame` and `Binaries` |
 | "audio is too long for this sound" | shorten the audio (see the original's length) |
+| "the text is … characters long but this subtitle has room for …" | shorten the subtitle (the counter shows the room) |
 | "Cannot read … Supported: WAV, FLAC, …" | convert the file to WAV/MP3/FLAC, or install ffmpeg |
 | "changed outside this tool" | the game files were changed by an update or another tool; revert or confirm to overwrite |
 | no sound when playing | choose another playback device in Settings |

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "ackit"
-version = "2.0.0"
+version = "3.2.3"
 description = "Batman: Arkham City (PC) audio toolkit and editor"
 readme = "README.md"
 requires-python = ">=3.10"

tests/synth.py

@@ -25,6 +25,16 @@ def wem(seconds, freq, rate=24000, ch=1):
     return encoder.encode(x, sr, rate, ch)[0]
 
 
+def fitting_wem(ws, s, seconds, freq):
+    """A tone WEM that `ws` accepts for `s` (Workspace.fits: in-memory slots are filled exactly, which needs
+    an even number of spare bytes): the tone is shortened a little until the size allows it."""
+    for k in range(40):
+        w = wem(seconds * (1 - 0.01 * k), freq)
+        if ws.fits(s, w):
+            return w
+    raise AssertionError("no fitting WEM for %s" % s.key)
+
+
 def _a(x, n):
     return (x + n - 1) // n * n
 
@@ -89,6 +99,153 @@ def pack(media):
     return bytes(hdr) + bank, bytes(payload)
 
 
+def bank(bank_id, media, name=None, extra_sources=(), prefetch=(), events=(), mem_events=()):
+    """A v56 soundbank holding `media` {id: wem} in memory (DIDX/DATA, 16-byte aligned) with one HIRC
+    Sound object per media; extra_sources: [(media id, bank id, off, size)] for sources of other banks;
+    media ids in `prefetch` get a prefetch source (their DIDX data is the head of a streamed WEM);
+    events: [(event name, streamed media id)] -> event -> action -> streamed Sound object;
+    mem_events: [(event name, media id of this bank)] -> event -> action -> that media's in-memory Sound."""
+    data = bytearray()
+    didx = b""
+    for m in sorted(media):
+        data += bytes(_a(len(data), 16) - len(data))
+        didx += struct.pack("<III", m, len(data), len(media[m]))
+        data += media[m]
+    bkhd = struct.pack("<II", 56, bank_id) + bytes(8)
+    head = b"BKHD" + struct.pack("<I", len(bkhd)) + bkhd + b"DIDX" + struct.pack("<I", len(didx)) + didx
+    data_pos = len(head) + 8                       # DATA payload, from the start of the bank
+    objs = []
+    for i, m in enumerate(sorted(media)):
+        off = struct.unpack_from("<I", didx, 12 * i + 4)[0]
+        objs.append((m, bank_id if m not in prefetch else 0x1234, data_pos + off, len(media[m]),
+                     2 if m in prefetch else 0))
+    objs += [x + (0,) for x in extra_sources]
+    from ackit.bnk import fnv1
+    obj = (bank_id & 0xFFFFF000) + 0x100         # HIRC object ids: unique per bank, like real Wwise ids
+    hirc_objs = []
+    for i, (m, b, off, size, stream) in enumerate(objs):
+        hirc_objs.append((2, struct.pack("<7I", obj + i, 0x00040001, stream, m, b, off, size) + bytes(6)))
+    for i, (ev, m) in enumerate(events):
+        sid, aid = 0x300 + i, 0x400 + i
+        hirc_objs.append((2, struct.pack("<7I", sid, 0x00040001, 1, m, 0, 0, 0) + bytes(6)))
+        hirc_objs.append((3, struct.pack("<III", aid, 0x4011, sid) + bytes(8)))
+        hirc_objs.append((4, struct.pack("<III", fnv1(ev), 1, aid)))
+    for i, (ev, m) in enumerate(mem_events):
+        aid = 0x500 + i
+        hirc_objs.append((3, struct.pack("<III", aid, 0x4011, obj + sorted(media).index(m)) + bytes(8)))
+        hirc_objs.append((4, struct.pack("<III", fnv1(ev), 1, aid)))
+    hirc = struct.pack("<I", len(hirc_objs))
+    for t, body in hirc_objs:
+        hirc += bytes([t]) + struct.pack("<I", len(body)) + body
+    out = head + b"DATA" + struct.pack("<I", len(data)) + bytes(data) + b"HIRC" + struct.pack("<I", len(hirc)) + hirc
+    if name:
+        nm = name.encode()
+        stid = struct.pack("<II", 1, 1) + struct.pack("<IB", bank_id, len(nm)) + nm
+        out += b"STID" + struct.pack("<I", len(stid)) + stid
+    return out
+
+
+def package(path, body, block_size=0x1000, chunk_blocks=3, tables=(0, 0, 0, 0, 0, 0)):
+    """A fully LZO-compressed UE3 package (docs/formats/upk.md) whose uncompressed data area is `body`.
+    Small blocks so that tests cross block and chunk boundaries. `tables`: (NameCount, NameOffset,
+    ExportCount, ExportOffset, ImportCount, ImportOffset) as absolute uncompressed offsets."""
+    from ackit import lzo
+    folder = b"None\x00"
+    nblocks = max(1, _a(len(body), block_size) // block_size)
+    nchunks = _a(nblocks, chunk_blocks) // chunk_blocks
+
+    def summary(chunks):
+        s = struct.pack("<IHHi", 0x9E2A83C1, 805, 101, 0) + struct.pack("<i", len(folder)) + folder
+        s += struct.pack("<12i", 0, *tables, 0, 0, 0, 0, 0) + b"GUID-synthetic!!"
+        s += struct.pack("<i3i", 1, 0, 0, 0) + struct.pack("<3I", 7748, 0, 2)
+        s += struct.pack("<i", len(chunks)) + b"".join(struct.pack("<4i", *c) for c in chunks)
+        return s + struct.pack("<Ii", 0x12345678, 0)
+    header_u = len(summary([])) + 16                       # uncompressed header (some spare bytes)
+    chunks, blobs = [], []
+    coff = len(summary([(0, 0, 0, 0)] * nchunks)) + 16
+    u = header_u
+    for c in range(nchunks):
+        part = body[(u - header_u):(u - header_u) + block_size * chunk_blocks]
+        blocks = [part[i:i + block_size] for i in range(0, len(part), block_size)] or [b""]
+        comp = [lzo.compress_stored(b) for b in blocks]
+        hdr = struct.pack("<4I", 0x9E2A83C1, block_size, sum(map(len, comp)), len(part))
+        hdr += b"".join(struct.pack("<2I", len(cb), len(b)) for cb, b in zip(comp, blocks))
+        blob = hdr + b"".join(comp)
+        chunks.append((u, len(part), coff, len(blob)))
+        blobs.append(blob)
+        coff += len(blob)
+        u += len(part)
+    s = summary(chunks)
+    out = s + bytes(16) + b"".join(blobs)
+    assert len(s) + 16 == chunks[0][2]
+    os.makedirs(os.path.dirname(path), exist_ok=True)
+    open(path, "wb").write(out)
+    return header_u
+
+
+HEADER_U = None   # uncompressed summary size of synthetic packages (fixed; computed on first use)
+
+
+def _header_u():
+    global HEADER_U
+    if HEADER_U is None:
+        import tempfile
+        with tempfile.TemporaryDirectory() as d:
+            HEADER_U = package(os.path.join(d, "x.upk"), b"\0" * 16)
+    return HEADER_U
+
+
+def ue_package(path, objects, prefix=b""):
+    """A package with name/import/export tables. objects: [(class name, object name, bytes or callable)]:
+    a callable gets {name: index} and the export index map and returns the object's bytes.
+    Returns {object name: (absolute offset, size)}."""
+    classes = sorted({c for c, _, _ in objects})
+    names = ["None", "Core", "Class", "LocalizedSubtitles", "Subtitles"] + classes + [n for _, n, _ in objects]
+    idx = {n: i for i, n in enumerate(names)}
+    exp_index = {n: i + 1 for i, (_, n, _) in enumerate(objects)}
+    blobs = [o(idx, exp_index) if callable(o) else o for _, _, o in objects]
+    nt = b"".join(struct.pack("<i", len(n) + 1) + n.encode() + b"\0" + bytes(8) for n in names)
+    it = b"".join(struct.pack("<7i", idx["Core"], 0, idx["Class"], 0, 0, idx[c], 0) for c in classes)
+    base = _header_u() + len(prefix)
+    name_off, imp_off = base, base + len(nt)
+    exp_off = imp_off + len(it)
+    data_off = exp_off + 72 * len(objects)
+    et, pos, offs = b"", data_off, {}
+    for (c, n, _), b in zip(objects, blobs):
+        et += struct.pack("<7i", -(classes.index(c) + 1), 0, 0, idx[n], 0, 0, 0) + bytes(8)
+        et += struct.pack("<ii", len(b), pos) + bytes(4 + 4 + 16 + 4)
+        offs[n] = (pos, len(b))
+        pos += len(b)
+    body = prefix + nt + it + et + b"".join(blobs) + bytes(64)
+    package(path, body, tables=(len(names), name_off, len(objects), exp_off, len(classes), imp_off))
+    return offs
+
+
+def _fstr(text, wide=False):
+    if wide:
+        return struct.pack("<i", -(len(text) + 1)) + text.encode("utf-16-le") + b"\0\0"
+    return struct.pack("<i", len(text) + 1) + text.encode("cp1252") + b"\0"
+
+
+def dialogue(speech_object, texts, wide=()):
+    """An RDialogueEvent export (compact tagged properties, docs/formats/subtitles.md) referencing the
+    export `speech_object`; texts: {lang: [line, ...]}; languages in `wide` are stored as UTF-16."""
+    def build(idx, exp_index):
+        content = struct.pack("<I", len(texts))
+        for lang, lines in texts.items():
+            sub = struct.pack("<I", len(lines))
+            for ln in lines:
+                sub += struct.pack("<HH", 0x0D, 0) + _fstr(ln, lang in wide) + struct.pack("<HHf", 4, 0x0C, 1.5)
+                sub += struct.pack("<H", 0)
+            content += struct.pack("<HH", 0x0D, 0) + _fstr(lang)
+            content += struct.pack("<HHiiii", 9, 0x0C, idx["Subtitles"], 0, len(sub), 0) + sub + struct.pack("<H", 0)
+        b = bytes(4) + struct.pack("<HHi", 0x10, 0x2C, exp_index[speech_object])
+        b += struct.pack("<HH", 0x0D, 0x34) + _fstr("Batman")
+        b += struct.pack("<HHiiii", 9, 0x9C, idx["LocalizedSubtitles"], 0, len(content), 0) + content
+        return b + struct.pack("<H", 0)
+    return build
+
+
 def make_game(root):
     """Create the folder; returns the dict of original WEMs by target."""
     cooked = os.path.join(root, "BmGame", "CookedPCConsole")
@@ -106,4 +263,97 @@ def make_game(root):
     originals = {"%08X:%d" % (SPEECH_ENTRY, i): w for i, w in enumerate(v)}
     originals["%08X" % SINGLE_ENTRY] = single
     originals.update({"%08X/%08X" % (PACK_ID, m): w for m, w in music.items()})
+    originals.update(make_banks(cooked, v))
+    make_localization(root)
     return originals
+
+
+# soundbanks (docs/formats/banks.md): media ids of the in-memory sounds
+FOOT = (0x0A000001, 0x0A000002)       # loose bank CHR_SynthFootsteps_Media
+FOLEY = (0x0B000001, 0x0B000002)      # bank CHR_SynthFoley, copied into both packages
+GUI = (0x0C000001,)                   # bank GUI_Synth (package B only)
+LINE_MEDIA = (0x0D000000, 0x0D000001, 0x0D000002)   # INT media ids of the SPEECH_ENTRY variations
+HEAD = 0x200                          # prefetch head size of variation 1
+
+
+def make_banks(cooked, speech_wems):
+    """Loose bank + two packages (one bank in both) + a prefetch head of INT SPEECH_ENTRY variation 1."""
+    from ackit.bnk import fnv1
+    names = ["Init", "CHR_SynthFootsteps_Media", "CHR_SynthFoley", "GUI_Synth", "Speech_Synth",
+             # banks whose streamed media are the WAD entries (entry ID = FNV-1 of the bank name)
+             "CHR_RadioScanner", "Speech_TAPE_JOKER_1", "Music", "AMB_Steel_B3"]
+    with open(os.path.join(cooked, "CookedBanks.txt"), "w", newline="\r\n") as f:
+        f.write("\n".join(names) + "\n")
+    foot = {FOOT[0]: wem(0.4, 800), FOOT[1]: wem(0.3, 900)}
+    foley = {FOLEY[0]: wem(0.5, 1000), FOLEY[1]: wem(0.6, 1100, 32000)}
+    gui = {GUI[0]: wem(0.5, 1200)}
+    open(os.path.join(cooked, "SFX", "%08X.bnk" % fnv1(names[1])), "wb").write(
+        bank(fnv1(names[1]), foot, name=names[1]))
+    # BKPK table of SPEECH_ENTRY for INT (media ids differ per language; offsets as in the WAD entry)
+    offs, pos = [], 0
+    for w in speech_wems:
+        pos = _a(pos, 0x800)
+        offs.append(pos)
+        pos += len(w)
+    tbl = bytearray(0x48)
+    struct.pack_into("<4sII", tbl, 0, b"BKPK", 0x01020000, SPEECH_ENTRY)
+    struct.pack_into("<IIIIII", tbl, 0x10, 3, 0, 36, 0x48, _a(pos, 0x800), 0)
+    tbl += b"".join(struct.pack("<III", m, len(w), o) for m, w, o in zip(LINE_MEDIA, speech_wems, offs))
+    speech = bank(fnv1(names[4]), {LINE_MEDIA[1]: speech_wems[1][:HEAD]}, prefetch=(LINE_MEDIA[1],))
+    filler = bytes(range(256)) * 9
+    synth_a = filler + bank(fnv1(names[2]), foley) + filler[:777] + bytes(tbl) + speech + filler
+    synth_b = filler[:1000] + bank(fnv1(names[3]), gui) + filler + bank(fnv1(names[2]), foley) + filler
+    package(os.path.join(cooked, "Synth_A.upk"), synth_a)
+    package(os.path.join(cooked, "Synth_B.upk"), synth_b)
+    # dialogue (subtitles) in two packages; the first one also holds a bank with an in-memory sound and the
+    # event that links the dialogue's speech event to INT variation 1 of SPEECH_ENTRY
+    dlg = dialogue(SPEECH_EVENT, SUBTITLES, wide=("RUS",))
+    dlg_wem = wem(0.3, 1400)
+    ev_bank = bank(fnv1("NPC_SynthDialogue"), {DLG_MEDIA: dlg_wem}, name="NPC_SynthDialogue",
+                   events=[(SPEECH_EVENT, LINE_MEDIA[1])])
+    objs = [("AkEvent", SPEECH_EVENT, bytes(8)), ("RDialogueEvent", "Radio_02", dlg)]
+    # a localized bank (one AkBank export: language list + one bank per language) with an in-memory line
+    loc = localized_bank(GRUNT_BANK, {lang: {m: wem(0.4, 500 + 100 * i)} for i, (lang, m) in enumerate(GRUNT_MEDIA.items())},
+                         GRUNT_EVENT)
+    grunt = [("AkEvent", GRUNT_EVENT, bytes(8)), ("RDialogueEvent", "Grunt_01", dialogue(GRUNT_EVENT, GRUNT_SUBS)),
+             ("AkBank", GRUNT_BANK, loc)]
+    ue_package(os.path.join(cooked, "Synth_Dialogue.upk"),
+               objs + [("AkBank", "NPC_SynthDialogue", ev_bank)] + grunt)
+    ue_package(os.path.join(cooked, "Synth_Dialogue2.upk"), objs, prefix=bytes(3000))
+    with open(os.path.join(cooked, "CookedBanks.txt"), "a", newline="\r\n") as f:
+        f.write("NPC_SynthDialogue\n%s\n" % GRUNT_BANK)
+    out = {"%08X" % m: w for m, w in list(foot.items()) + list(foley.items()) + list(gui.items())}
+    out["%08X" % DLG_MEDIA] = dlg_wem
+    return out
+
+
+SPEECH_EVENT = "Speech_Synth_Radio_02"
+GRUNT_BANK, GRUNT_EVENT = "Speech_Synth_Grunt", "Speech_Synth_Grunt_01"
+GRUNT_MEDIA = {"INT": 0x0F000001, "ITA": 0x0F000002}       # one media per language (localized bank)
+GRUNT_SUBS = {"INT": ["Hey you!"], "ITA": ["Ehi, tu!"]}
+
+
+def localized_bank(name, per_lang, event):
+    """AkBank export data of a localized bank: language list, then one bank per language (same id)."""
+    from ackit.bnk import fnv1
+    langs = list(per_lang)
+    lst = struct.pack("<I", len(langs)) + b"".join(_fstr(l) for l in langs)
+    head = bytes(4) + struct.pack("<HHiiii", 9, 0x84, 0, 0, len(lst), 0) + lst
+    blobs = b"".join(bank(fnv1(name), media, name=name, mem_events=[(event, next(iter(media)))]) + bytes(16)
+                     for media in per_lang.values())
+    return head + blobs
+SUBTITLES = {"INT": ["Scanner line two.", "Second part."], "ITA": ["Riga due dello scanner."],
+             "RUS": ["Строка два"]}
+DLG_MEDIA = 0x0E000001
+MENU = {"INT": ("FRONTEND.int", "\r\n[Menu]\r\nStart=Press any key\r\nQuit=\"Quit game\"\r\n; comment\r\n"
+                                "[Tips]\r\nTip=Use the grapnel\r\nTip=Glide over the city\r\n"),
+        "ITA": ("FRONTEND.ITA", "\r\n[Menu]\r\nStart=Premi un tasto\r\nQuit=\"Esci\"\r\n"
+                                "[Tips]\r\nTip=Usa il rampino\r\nTip=Plana sulla citta\r\n")}
+
+
+def make_localization(root):
+    for lang, (fname, text) in MENU.items():
+        d = os.path.join(root, "BmGame", "Localization", lang)
+        os.makedirs(d, exist_ok=True)
+        open(os.path.join(d, fname), "wb").write(b"\xff\xfe" + text.encode("utf-16-le"))
+    open(os.path.join(root, "BmGame", "Localization", "INT", "Version.int"), "wb").write(b"[ID]\r\nUnreal=SYNTH\r\n")

tests/test_app.py

@@ -56,7 +56,7 @@ def type_search(w, text):
 def test_browse_search_filter(win):
     w, root, _ = win
     assert w.model.rowCount() == len(w.ws.catalog.sounds)
-    type_search(w, "radio scanner 2")
+    type_search(w, "synth radio 02")
     assert w.proxy.rowCount() == 1
     type_search(w, "")
     w.cmb_group.setCurrentText("Music")
@@ -66,13 +66,35 @@ def test_browse_search_filter(win):
     assert w.proxy.rowCount() == w.model.rowCount()
 
 
+def test_sort_order_survives_filter_changes(win):
+    """Rows that a filter shows again go to their sorted place (not appended at the end)."""
+    from PySide6.QtCore import Qt
+    from app.models import C_NAME, SORT_ROLE
+    w, _, _ = win
+
+    def keys():
+        return [w.proxy.data(w.proxy.index(r, C_NAME), SORT_ROLE) for r in range(w.proxy.rowCount())]
+
+    w.table.sortByColumn(C_NAME, Qt.SortOrder.AscendingOrder)
+    assert keys() == sorted(keys())
+    w.cmb_group.setCurrentText("Music")
+    w.clear_filters()
+    type_search(w, "radio")
+    type_search(w, "")
+    assert w.proxy.rowCount() == w.model.rowCount() and keys() == sorted(keys())
+    w.table.sortByColumn(C_NAME, Qt.SortOrder.DescendingOrder)
+    w.cmb_group.setCurrentText("Speech")
+    w.clear_filters()
+    assert keys() == sorted(keys(), reverse=True)
+
+
 def test_replace_apply_revert_byte_identical(win, tmp_path):
     w, root, originals = win
     before = files(root)
-    type_search(w, "CHR_RadioScanner_2")
-    assert w.select_name("CHR_RadioScanner_2")
+    type_search(w, "Speech_Synth_Radio_02")
+    assert w.select_name("Speech_Synth_Radio_02")
     ed = w.editor
-    assert ed.sound.name == "CHR_RadioScanner_2" and ed.orig is not None
+    assert ed.sound.name == "Speech_Synth_Radio_02" and ed.orig is not None
     wav = str(tmp_path / "new.wav")
     x, sr = synth.tone(1.0, 44100, 990)
     audio.save_wav(wav, x, sr)
@@ -93,6 +115,128 @@ def test_replace_apply_revert_byte_identical(win, tmp_path):
     assert not w.ws.store.entries
 
 
+def test_bank_sound_apply_revert_in_editor(win, tmp_path):
+    w, root, originals = win
+    before = files(root)
+    w.cmb_arch.setCurrentText("BANK")
+    assert w.proxy.rowCount() == 7                                  # 6 neutral + the INT copy of a localized line
+    assert w.select_name("CHR_SynthFoley_1")
+    ed = w.editor
+    assert "bank CHR_SynthFoley (2 copies)" in ed.meta.text()
+    wav = str(tmp_path / "foley.wav")
+    audio.save_wav(wav, *synth.tone(0.2, 44100, 1300))
+    assert ed.load_file(wav)
+    ed._update_new()
+    ed.apply()
+    assert w.ws.state(ed.sound) == W.MODIFIED and files(root) != before
+    ed.revert()
+    assert files(root) == before and not w.ws.store.entries
+
+
+def test_subtitles_tab_edit_and_link(win):
+    w, root, _ = win
+    before = files(root)
+    w.tabs.setCurrentWidget(w.subs)
+    p = w.subs
+    assert p.lang == "INT" and p.model.rowCount() == 3
+    p.search.setText("scanner")
+    p._filter()
+    assert p.proxy.rowCount() == 1
+    p.table.selectRow(0)
+    assert p.edit.text() == "Scanner line two." and "/ 17 characters" in p.lbl_info.text()
+    p.edit.setText("x" * 30)
+    assert not p.b_apply.isEnabled()                               # longer than the room
+    p.edit.setText("Scan two.")
+    p.apply_current()
+    key = p.current.key
+    assert w.ws.texts.state(key) == "modified" and p.model.modified[key] == "Scan two."
+    p.go_to_sound.emit(p.current.sound)                             # "Go to the sound"
+    assert w.tabs.currentIndex() == 0 and w.editor.sound.name == "Speech_Synth_Radio_02"
+    box = w.editor.subtitle
+    assert [e.text() for e in box.edits] == ["Scan two.", "Second part."]
+    box.edits[1].setText("Part two.")
+    box.apply()
+    assert w.ws.texts.current(key.replace(":0", ":1")) == "Part two."
+    box.open_in_list.emit(box.keys[1])                              # back to the Subtitles tab
+    assert w.tabs.currentWidget() is p and p.current.key == box.keys[1]
+    box.revert()
+    assert files(root) == before and not w.ws.texts.records()
+
+
+def test_one_language_for_everything(win):
+    w, root, _ = win
+    assert w.language == "INT" and w.cmb_lang.currentText() == "INT" and not w.lbl_voices.text()
+    assert {"INT", "ITA", "RUS"} <= {w.cmb_lang.itemText(i) for i in range(w.cmb_lang.count())}
+    w.load_game(root, "RUS")                                  # no Russian voices: INT speech, RUS texts
+    assert w.language == "RUS" and "voices: INT" in w.lbl_voices.text()
+    assert "INT" in w.ws.catalog.archives
+    w.tabs.setCurrentWidget(w.subs)
+    assert w.subs.lang == "RUS" and "RUS" in w.subs.lbl_lang.text()
+    w.tabs.setCurrentIndex(0)
+    w.select_name("Speech_Synth_Radio_02")
+    assert [e.text() for e in w.editor.subtitle.edits] == ["Строка два"]
+    w.tabs.setCurrentWidget(w.menu)
+    assert w.menu.lang is None and w.menu.model.rowCount() == 0   # no RUS menu files in the synthetic game
+
+
+def test_subtitles_find_replace(win):
+    w, root, _ = win
+    before = files(root)
+    w.tabs.setCurrentWidget(w.subs)
+    p = w.subs
+    assert p.find_replace("line", "row") == 1
+    assert w.ws.texts.current("SUB:INT:%s:0" % synth.SPEECH_EVENT) == "Scanner row two."
+    p.revert_all()
+    assert files(root) == before
+
+
+def test_menu_text_tab(win):
+    w, root, _ = win
+    before = files(root)
+    w.tabs.setCurrentWidget(w.menu)
+    p = w.menu
+    assert p.model.rowCount() == 4
+    p.filters["File"].setCurrentText("FRONTEND.int")
+    p.search.setText("press")
+    p._filter()
+    assert p.proxy.rowCount() == 1
+    p.table.selectRow(0)
+    p.edit.setText("Press Enter to start")
+    p.apply_current()
+    assert w.ws.texts.current("MENU:INT:FRONTEND.int:Menu:Start") == "Press Enter to start"
+    p.cb_mod.setChecked(True)
+    assert p.proxy.rowCount() == 1
+    p.revert_all()
+    assert files(root) == before
+
+
+def test_mod_with_texts(win, tmp_path):
+    w, root, _ = win
+    tx = w.ws.texts
+    tx.apply({"SUB:INT:%s:0" % synth.SPEECH_EVENT: "Mod line.", "MENU:INT:FRONTEND.int:Menu:Quit": "Leave"})
+    pkg = str(tmp_path / "texts.acmod")
+    w.export_mod_dialog(path=pkg, keys=tx.modified(), title="texts")
+    tx.revert_all()
+    w.import_mod_dialog(path=pkg, confirm=False)
+    assert tx.current("MENU:INT:FRONTEND.int:Menu:Quit") == "Leave"
+    assert tx.current("SUB:INT:%s:0" % synth.SPEECH_EVENT) == "Mod line."
+    tx.revert_all()
+
+
+def test_text_undone_outside_shows_banner(win):
+    w, root, _ = win
+    tx = w.ws.texts
+    k = "MENU:INT:FRONTEND.int:Menu:Start"
+    tx.apply({k: "Go"})
+    path = tx.files(k)[0]
+    open(path, "wb").write(open(w.ws.store.full_copy(w.ws.game.rel(path)), "rb").read())   # like 'verify files'
+    w.check_problems()
+    assert not w.banner.isHidden() and "text changes" in w.banner_text.text()
+    w.reapply_lost()
+    assert tx.current(k) == "Go" and tx.state(k) == "modified"
+    tx.revert_all()
+
+
 def test_record_simulated_input(win):
     w, root, _ = win
     w.select_name("CHR_RadioScanner_1")
@@ -129,7 +273,7 @@ def test_notes_tags_favorites(win):
 
 def test_mod_export_import(win, tmp_path):
     w, root, _ = win
-    w.select_name("CHR_RadioScanner_2")
+    w.select_name("Speech_Synth_Radio_02")
     x, sr = synth.tone(1.0, 24000, 1200)
     w.editor.set_source(x, sr, "tone")
     w.editor.apply()
@@ -174,6 +318,39 @@ class HeldRunner:
             self.real.start(*self.held.pop(0))
 
 
+def test_selection_survives_sort_and_filter(win):
+    from PySide6.QtCore import Qt
+    from app.models import C_LEN
+    w, _, _ = win
+    assert w.select_name("Speech_Synth_Radio_02")
+    s = w.editor.sound
+    w.table.sortByColumn(C_LEN, Qt.SortOrder.DescendingOrder)
+    type_search(w, "radio")
+    cur = w.table.currentIndex()
+    assert cur.isValid() and w.proxy.sound_at(cur) is s and [x.key for x in w.selected_sounds()] == [s.key]
+    type_search(w, "music")                              # the row is filtered out: no current row
+    assert not w.table.currentIndex().isValid() and not w.selected_sounds()
+
+
+def test_list_spinners_while_loading(win):
+    """The sound list and the text lists show the loading overlay while they load (background jobs); a subtitle
+    asked for meanwhile is selected when the rows arrive."""
+    from ackit import subtitles
+    w, root, _ = win
+    held = w.runner = HeldRunner(w.runner)
+    w.load_game(root, "INT")
+    assert w.table_busy.busy and w.table_busy.timer.isActive()
+    held.release()
+    assert not w.table_busy.busy and w.model.rowCount() == len(w.ws.catalog.sounds)
+    w._text_loaded.clear()
+    sub = w.ws.texts.subtitles("INT")[0]
+    key = subtitles.key("INT", sub.speech, sub.line)
+    w.show_subtitle(key)
+    assert w.subs.busy.busy and w.subs.busy.timer.isActive()
+    held.release()
+    assert not w.subs.busy.busy and w.subs.current is not None and w.subs.current.key == key
+
+
 def test_spinners_while_loading(win, tmp_path):
     w, root, _ = win
     held = w.runner = HeldRunner(w.runner)

tests/test_banks.py

@@ -0,0 +1,207 @@
+"""In-memory sounds (archive BANK) and prefetch heads: catalog scan, apply/revert through Workspace."""
+import hashlib
+import os
+
+import pytest
+
+import synth
+from ackit import banks
+from ackit.game import Game
+from ackit.inject import SizeLimitError
+from ackit.upk import Package
+from ackit.wem import fill, unfill
+from ackit.workspace import Workspace, ConflictError, MODIFIED, ORIGINAL
+
+
+def _sha(p):
+    return hashlib.sha1(open(p, "rb").read()).hexdigest()
+
+
+def _ws(root, tmp_path):
+    return Workspace(root, store_root=str(tmp_path / "store"), archives=["INT", "SFX", "BANK"])
+
+
+def _cooked(root):
+    return Game(root).cooked
+
+
+def _copy(ws, l):
+    return banks.open_file(ws.bank_path(l.file)).read(l.offset, l.size)
+
+
+def test_bank_catalog(synth_game, tmp_path):
+    root, originals = synth_game
+    ws = _ws(root, tmp_path)
+    bank = {s.target: s for s in ws.catalog.sounds if s.archive == "BANK"}
+    neutral = synth.FOOT + synth.FOLEY + synth.GUI + (synth.DLG_MEDIA,)
+    assert set(bank) == {"%08X" % m for m in neutral + (synth.GRUNT_MEDIA["INT"],)}   # INT catalog: no ITA copy
+    grunt = bank["%08X" % synth.GRUNT_MEDIA["INT"]]
+    assert grunt.language == "INT" and grunt.name == synth.GRUNT_EVENT and grunt.group == "Speech"
+    ita = Workspace(root, store_root=str(tmp_path / "s2"), archives=["BANK"]).catalog   # no language: all
+    assert {s.language for s in ita.sounds if s.bank == synth.GRUNT_BANK} == {"INT", "ITA"}
+    foot = bank["%08X" % synth.FOOT[0]]
+    assert foot.bank == "CHR_SynthFootsteps_Media" and foot.name == "CHR_SynthFootsteps_Media_1"
+    assert (foot.group, foot.category) == ("SFX", "Footsteps")
+    assert bank["%08X" % synth.GUI[0]].category == "Interface"
+    foley = bank["%08X" % synth.FOLEY[1]]
+    assert foley.rate == 32000 and foley.size == len(originals[foley.target])
+    assert sorted(l.file for l in ws.locations(foley)) == ["Synth_A.upk", "Synth_B.upk"]
+    for s in bank.values():
+        if s.language:
+            continue
+        assert ws.current_bytes(s) == originals[s.target]
+        assert all(_copy(ws, l) == originals[s.target] for l in ws.locations(s))
+    # the prefetch head belongs to INT variation 1 of the speech entry, not to the BANK archive
+    line = ws.catalog.by_key["INT:%08X:1" % synth.SPEECH_ENTRY]
+    heads = ws.locations(line)
+    assert [(l.file, l.size, l.kind) for l in heads] == [("Synth_A.upk", synth.HEAD, "p")]
+    assert _copy(ws, heads[0]) == originals["%08X:1" % synth.SPEECH_ENTRY][:synth.HEAD]
+    assert ws.locations(ws.catalog.by_key["INT:%08X:0" % synth.SPEECH_ENTRY]) == []
+
+
+def test_bank_apply_every_copy_and_revert_byte_identical(synth_game, tmp_path):
+    root, originals = synth_game
+    ws = _ws(root, tmp_path)
+    cooked = _cooked(root)
+    files = [os.path.join(cooked, f) for f in ("Synth_A.upk", "Synth_B.upk")]
+    before = {f: _sha(f) for f in files}
+    s = ws.catalog.by_key["BANK:%08X" % synth.FOLEY[0]]
+    new = synth.fitting_wem(ws, s, 0.2, 1500)
+    assert len(new) < ws.limit(s)
+    ws.apply([(s, new, {"source": "test"})])
+    assert ws.state(s) == MODIFIED and ws.current_bytes(s) == new
+    for l in ws.locations(s):               # every copy filled exactly: no zero padding after the WEM
+        assert _copy(ws, l) == fill(new, l.size) and len(_copy(ws, l)) == l.size
+        assert unfill(_copy(ws, l)) == new
+    other = ws.catalog.by_key["BANK:%08X" % synth.FOLEY[1]]
+    assert all(_copy(ws, l) == originals[other.target] for l in ws.locations(other))
+    assert all(_sha(f) != before[f] for f in files)
+    assert Package(files[1]).size == Package(ws.store.full_copy(ws.game.rel(files[1]))).size  # layout unchanged
+    ws.revert([s])
+    assert ws.state(s) == ORIGINAL and {f: _sha(f) for f in files} == before
+
+
+def test_fill_slot_exactly():
+    """The game plays an in-memory WEM followed by zero padding as silence: a smaller replacement is grown to
+    its slot with a label list chunk (wem.fill), which the game's own WEMs use too."""
+    from ackit import audio
+    from ackit.wem import FILL_MIN, Wem, fillable
+    w = synth.wem(0.2, 1500)
+    for size in (len(w), len(w) + FILL_MIN, len(w) + 1000):
+        f = fill(w, size)
+        assert len(f) == size and 8 + int.from_bytes(f[4:8], "little") == size     # RIFF covers the slot
+        tags = [b"fmt ", b"data"] if size == len(w) else [b"fmt ", b"LIST", b"data"]
+        assert unfill(f) == w and [c.tag for c in Wem.parse(f).chunks] == tags
+        assert (audio.decode_wem(f)[0] == audio.decode_wem(w)[0]).all()
+    assert not fillable(len(w), len(w) + 3) and not fillable(len(w), len(w) + FILL_MIN - 2)
+    with pytest.raises(ValueError):
+        fill(w, len(w) + 1)
+    f = fill(w, len(w) + 40)
+    a = f.index(b"adtllabl")
+    game_like = f[:a + 12] + (1).to_bytes(4, "little") + b"END\0" + f[a + 20:]   # a real marker label: kept
+    assert len(game_like) == len(f) and unfill(game_like) == game_like
+
+
+def test_bank_apply_refuses_unfillable_and_prepare_fits(synth_game, tmp_path):
+    from ackit.game import GameError
+    root, _ = synth_game
+    ws = _ws(root, tmp_path)
+    s = ws.catalog.by_key["BANK:%08X" % synth.FOOT[1]]
+    bad = next(w for w in (synth.wem(0.1 * (1 - 0.01 * k), 2000) for k in range(40)) if not ws.fits(s, w)
+               and len(w) <= ws.limit(s))
+    with pytest.raises(GameError):
+        ws.apply([(s, bad, {})])
+    assert ws.store.entries == {}
+    x, sr = synth.tone(0.1, 24000, 2000, 1)
+    wem, _ = ws.prepare(s, x, sr)
+    assert ws.fits(s, wem)
+
+
+def test_loose_bank_in_place(synth_game, tmp_path):
+    root, originals = synth_game
+    ws = _ws(root, tmp_path)
+    s = ws.catalog.by_key["BANK:%08X" % synth.FOOT[1]]
+    path = ws.files(s)[0]
+    before, size = _sha(path), os.path.getsize(path)
+    ws.apply([(s, synth.fitting_wem(ws, s, 0.1, 2000), {})])
+    assert os.path.getsize(path) == size and _sha(path) != before
+    ws.revert_all()
+    assert _sha(path) == before
+
+
+def test_partial_revert_keeps_other_changes(synth_game, tmp_path):
+    root, originals = synth_game
+    ws = _ws(root, tmp_path)
+    path = os.path.join(_cooked(root), "Synth_B.upk")
+    before = _sha(path)
+    a = ws.catalog.by_key["BANK:%08X" % synth.FOLEY[0]]
+    b = ws.catalog.by_key["BANK:%08X" % synth.GUI[0]]
+    na, nb = synth.fitting_wem(ws, a, 0.2, 1500), synth.fitting_wem(ws, b, 0.2, 1700)
+    ws.apply([(a, na, {}), (b, nb, {})])
+    ws.revert([a])
+    assert ws.current_bytes(a) == originals[a.target]
+    assert ws.current_bytes(b) == nb and ws.state(b) == MODIFIED
+    ws.revert([b])
+    assert _sha(path) == before
+
+
+def test_speech_line_with_prefetch_head(synth_game, tmp_path):
+    root, originals = synth_game
+    ws = _ws(root, tmp_path)
+    line = ws.catalog.by_key["INT:%08X:1" % synth.SPEECH_ENTRY]
+    files = ws.files(line)
+    assert [os.path.basename(f) for f in files] == ["28CCF006_LOC_INT.wad", "Synth_A.upk"]
+    before = {f: _sha(f) for f in files}
+    new = synth.wem(0.3, 640)
+    ws.apply([(line, new, {})])
+    head = ws.locations(line)[0]
+    assert _copy(ws, head) == (new + bytes(synth.HEAD))[:synth.HEAD]
+    assert ws.current_bytes(line).startswith(new)
+    ws.revert([line])
+    assert {f: _sha(f) for f in files} == before
+
+
+def test_bank_size_limit(synth_game, tmp_path):
+    root, _ = synth_game
+    ws = _ws(root, tmp_path)
+    s = ws.catalog.by_key["BANK:%08X" % synth.FOOT[1]]
+    with pytest.raises(SizeLimitError):
+        ws.apply([(s, synth.wem(3.0, 440), {})])
+    assert ws.store.entries == {}
+
+
+def test_bank_file_changed_underneath(synth_game, tmp_path):
+    root, originals = synth_game
+    ws = _ws(root, tmp_path)
+    s = ws.catalog.by_key["BANK:%08X" % synth.GUI[0]]
+    ws.apply([(s, synth.fitting_wem(ws, s, 0.2, 1500), {})])
+    path = ws.files(s)[0]
+    # another tool rewrites the package (different bytes elsewhere in it)
+    pk = Package(path)
+    other = pk.patched([(pk.header_size + 10, b"OTHER-TOOL")])
+    open(path, "wb").write(other)
+    with pytest.raises(ConflictError):
+        ws.apply([(s, synth.fitting_wem(ws, s, 0.2, 1600), {})])
+    ws.apply([(s, synth.fitting_wem(ws, s, 0.2, 1600), {})], force=True)
+    ws.revert([s])
+    assert _sha(path) == _sha(ws.store.full_copy(ws.game.rel(path)))
+
+
+def test_event_names():
+    sc = banks.Scan()
+    sc.names = {"Play_Synth_Punch", "Stop_Synth_Punch", "Unrelated"}
+    sc.bank_names = {7: "CHR_SynthCombat"}
+    for m in (1, 2, 3):
+        sc.media[m] = {"size": 10, "sha1": {"x"}, "head": b"", "banks": {7}, "kinds": {"m"}}
+    # event -> action -> random container (children 0x50, 0x51) -> sounds -> media 1, 2; media 3 unreferenced
+    from ackit.bnk import fnv1
+    import struct
+    sc.objects = {0x50: 2, 0x51: 2, 0x60: 5, 0x70: 3, fnv1("Play_Synth_Punch"): 4}
+    sc.containers = {0x60: struct.pack("<I", 0x60) + bytes(9) + struct.pack("<3I", 2, 0x50, 0x51) + bytes(5)}
+    sc.actions = {0x70: 0x60}
+    sc.events = {fnv1("Play_Synth_Punch"): [0x70]}
+    sc.sound_media = {0x50: {1}, 0x51: {2}}
+    names = banks.media_names(sc)
+    assert names[1] == ("Play_Synth_Punch_1", "CHR_SynthCombat")
+    assert names[2] == ("Play_Synth_Punch_2", "CHR_SynthCombat")
+    assert names[3] == ("CHR_SynthCombat_1", "CHR_SynthCombat")

tests/test_catalog.py

@@ -1,10 +1,11 @@
 """Catalog: classification rules, naming, character labels, search; the shipped CSVs."""
+import collections
 import csv
 import os
 
 import pytest
 
-from ackit import classify
+from ackit import banks, classify
 from ackit.catalog import Catalog, read_csv, load_characters
 from ackit.game import Game
 from ackit.paths import catalog_dir
@@ -38,9 +39,10 @@ def test_synthetic_catalog_names_and_search(synth_game):
     root, _ = synth_game
     cat = Catalog(Game(root), ["INT", "SFX"])
     names = {s.name for s in cat.sounds}
-    assert {"CHR_RadioScanner_1", "CHR_RadioScanner_2", "CHR_RadioScanner_3", "Music_393BC53A"} <= names
+    assert {"CHR_RadioScanner_1", "Speech_Synth_Radio_02", "CHR_RadioScanner_3", "Music_393BC53A"} <= names
     s = cat.resolve("CHR_RadioScanner#2")
-    assert s.target.endswith(":1") and cat.resolve("chr_radioscanner_2.wav") is s
+    assert s.target.endswith(":1") and cat.resolve("speech_synth_radio_02.wav") is s
+    assert s.line and s.name == "Speech_Synth_Radio_02"        # the game's line: named after its event
     assert cat.resolve(s.key) is s
     assert [x.name for x in cat.search("radio scanner 3")] == ["CHR_RadioScanner_3"]
     with pytest.raises(LookupError) as e:
@@ -49,29 +51,132 @@ def test_synthetic_catalog_names_and_search(synth_game):
 
 
 def test_characters_csv_covers_user_lists():
-    chars = load_characters()
-    assert len(chars) > 3000
+    chars = load_characters(lang="ITA")
+    assert len(chars) > 3000 and len(load_characters()) > 2500     # most labels apply in every language
+    # the lists name ITA sounds by position (wemscan): NAME_i = variation i of the ITA WAD entry (alias NAME#i)
     src = os.path.join(catalog_dir(), "source", "ita", "Joker.txt")
     first = open(src).readline().strip()[:-4]
-    assert "Joker" in chars[first.lower()]
+    base, i = first.rsplit("_", 1)
+    ita = read_csv(os.path.join(catalog_dir(), "ITA.csv"))
+    s = next(s for s in ita if first.lower() in {n.lower() for n in s.names()} or
+             "%s#%s" % (base, i) in s.aliases)
+    assert "Joker" in s.character
+    # every labelled ITA sound keeps its label (takes of one line, NAME#k, share it), and most labels reach INT
+    assert sum(1 for x in ita if x.character) >= 3366
+    assert sum(1 for x in read_csv(os.path.join(catalog_dir(), "INT.csv")) if x.character) > 2900
+
+
+def test_line_names_agree_across_languages_and_subtitles():
+    """Speech sounds carry the game's line name: the same names in every language, and a linked subtitle's
+    sound has the subtitle's line name (positions inside WAD entries differ per language: findings 20)."""
+    from ackit import subtitles
+    ita = {s.name for s in read_csv(os.path.join(catalog_dir(), "ITA.csv")) if s.group == "Speech"}
+    intl = {s.name for s in read_csv(os.path.join(catalog_dir(), "INT.csv")) if s.group == "Speech"}
+    assert len(ita & intl) > 0.97 * len(ita)
+    snd = {s.key: s for s in read_csv(os.path.join(catalog_dir(), "ITA.csv"))}
+    rows = [r for r in subtitles.read_texts(os.path.join(catalog_dir(), "SUB_ITA.csv"), "ITA") if r.sound in snd]
+    same = sum(1 for r in rows if snd[r.sound].name.split("#")[0] == r.speech)
+    assert len(rows) > 8000 and same > 0.99 * len(rows)
+    hdr = next(csv.reader(open(os.path.join(catalog_dir(), "characters.csv"), encoding="utf-8")))
+    assert hdr == ["name", "character", "language"]
 
 
 def test_shipped_csvs_are_consistent():
+    from ackit import subtitles
     for f in os.listdir(catalog_dir()):
+        if f.startswith("SUB_") and f.endswith(".csv"):
+            lines = subtitles.read_texts(os.path.join(catalog_dir(), f), f[4:7])
+            assert len({(s.speech, s.line) for s in lines}) == len(lines), f
+            assert all(len(s.text) <= s.room for s in lines), f
+            continue
         if f.endswith(".csv") and f != "characters.csv":
             rows = read_csv(os.path.join(catalog_dir(), f))
             keys = {r.key for r in rows}
             assert len(keys) == len(rows), f
             assert all(r.group in classify.GROUPS for r in rows)
-            assert all(len(r.sha1) == 16 for r in rows)
+            assert len({(r.language, r.name.lower()) for r in rows}) == len(rows), f   # per language
+            # no checksum only where the copies of a BANK sound differ (1C898731)
+            assert sum(len(r.sha1) != 16 for r in rows) <= (1 if f == "BANK.csv" else 0), f
     hdr = next(csv.reader(open(os.path.join(catalog_dir(), "ITA.csv"), encoding="utf-8")))
     assert hdr[:3] == ["archive", "target", "name"]
+    locs = banks.read_locations(os.path.join(catalog_dir(), banks.LOCATIONS))
+    bank = {r.target for r in read_csv(os.path.join(catalog_dir(), "BANK.csv"))}
+    assert {"%08X" % m for m, ls in locs.items() if ls[0].kind == "m"} == bank
+    assert all(l.sound for ls in locs.values() for l in ls if l.kind == "p")
+
+
+def test_shipped_catalog_cross_references():
+    """Every reference between the shipped files points to something that exists, and every line has one
+    recording and one name per language (copies aside)."""
+    from ackit import subtitles
+    voices = ["INT", "FRA", "ITA", "DEU", "ESN"]
+    rows = {a: read_csv(os.path.join(catalog_dir(), a + ".csv")) for a in voices + ["SFX", "BANK"]}
+    keys = {s.key: s for r in rows.values() for s in r}
+    bank = rows["BANK"]
+    # subtitles -> a voice recording of that language (INT voices for the others)
+    for f in os.listdir(catalog_dir()):
+        if f.startswith("SUB_") and f.endswith(".csv"):
+            lang = f[4:7]
+            voice = lang if lang in voices else "INT"
+            for s in subtitles.read_texts(os.path.join(catalog_dir(), f), lang):
+                for k in filter(None, s.sound.split(";")):
+                    assert k in keys, (f, s.speech, k)
+                    snd = keys[k]
+                    assert snd.archive == voice or (snd.archive == "BANK" and snd.in_language(voice)), (f, k)
+    # heads of WAD recordings -> a WAD recording, never larger than it; no BANK sound is also a head
+    locs = banks.read_locations(os.path.join(catalog_dir(), banks.LOCATIONS))
+    for m, ls in locs.items():
+        for l in ls:
+            if l.kind == "p":
+                assert l.sound in keys and keys[l.sound].archive in voices and l.size <= keys[l.sound].size, l
+                assert "BANK:%08X" % m not in keys
+    # speaker labels -> line names that exist in every voice language
+    names = {L: {s.name for s in rows[L]} for L in voices}
+    with open(os.path.join(catalog_dir(), "characters.csv"), encoding="utf-8") as f:
+        for r in csv.DictReader(f):
+            assert all(r["name"] in names[L] for L in ([r["language"]] if r["language"] else voices)), r
+    # one recording per language per line (copies: a second place of the same recording, same line)
+    for L in voices:
+        per = collections.defaultdict(list)
+        for s in rows[L] + [b for b in bank if b.line and b.in_language(L)]:
+            if s.line:
+                per[s.line].append(s)
+        for lid, ss in per.items():
+            assert len({s.sha1 for s in ss}) == 1, (L, lid, [s.key for s in ss])    # copies are identical
+            assert len([s for s in ss if s.archive == "BANK"]) <= 1, (L, lid)
+    # the same line has the same name in every language (the second place of a copy keeps its position name)
+    common = {}
+    for L in voices:
+        per = collections.defaultdict(set)
+        for s in rows[L]:
+            if s.line:
+                per[s.line].add(s.name)
+        for lid, ns in per.items():
+            common[lid] = common[lid] & ns if lid in common else ns
+    assert all(common.values()), [k for k, v in common.items() if not v][:5]
 
 
 @pytest.mark.game
 def test_real_game_uses_shipped_catalog(real_game_dir):
     g = Game(real_game_dir)
-    cat = Catalog(g, ["ITA", "SFX"])
-    assert len(cat.sounds) == 10816 + 651
-    s = cat.resolve("Speech_TAPE_JOKER_1_4")
-    assert s.category == "Interview tapes" and s.size == 44985
+    cat = Catalog(g, ["ITA", "SFX", "BANK"])
+    bank = read_csv(os.path.join(catalog_dir(), "BANK.csv"))
+    assert len(cat.sounds) == 10816 + 651 + sum(1 for s in bank if s.in_language("ITA"))
+    assert all(s.in_language("ITA") for s in cat.sounds if s.archive == "BANK")
+    assert all(s.line for s in cat.sounds if s.archive == "ITA" and s.group == "Speech")   # every voice line
+    s = cat.resolve("Speech_TAPE_JOKER_1#4")             # position 4 in the ITA entry (the user's list name)
+    assert s.category == "Interview tapes" and s.size == 44985 and s.name.startswith("Speech_TAPE_JOKER_1_")
+    # bank copies of WAD recordings: 396 prefetch heads + 49 recordings also stored whole in banks
+    assert len(cat.heads) == 445 and sum(len(v) for v in cat.heads.values()) == 2185
+    b = cat.resolve("Play_BatmanDeviceImpact_Wood_1")
+    assert b.archive == "BANK" and b.group == "SFX"
+
+
+@pytest.mark.game
+def test_every_name_resolves_to_its_sound(real_game_dir):
+    """A line named after its INT position (CHR_RadioScanner#1) can equal another recording's position alias
+    in another language: the name wins, in every language."""
+    g = Game(real_game_dir)
+    for lang in g.languages:
+        cat = Catalog(g, [lang, "SFX", "BANK"])
+        assert all(cat.resolve(s.name) is s for s in cat.sounds), lang

tests/test_catalogcheck.py

@@ -0,0 +1,40 @@
+"""`ackit verify --catalog`: the catalog in use against a fresh scan of the game."""
+import glob
+import os
+
+from ackit.catalog import read_csv, write_csv
+from ackit.game import Game
+from ackit.paths import user_dir
+from ackit.verify import catalogcheck
+
+
+def test_clean_synthetic_catalog_passes(synth_game):
+    root, _ = synth_game
+    rep = catalogcheck.run(Game(root), log=lambda *a: None)
+    assert rep.ok, rep.issues
+    assert rep.checked["BANK"] == 8 and rep.checked["INT"] == 4      # all languages of the localized line
+
+
+def test_corrupted_catalog_is_reported(synth_game):
+    root, _ = synth_game
+    game = Game(root)
+    catalogcheck.run(game, ["INT", "BANK"], log=lambda *a: None)       # builds the cached scans
+    cache = glob.glob(os.path.join(user_dir("cache"), "scan_INT_*.csv"))[0]
+    rows = read_csv(cache)
+    rows[1].category = "Weapons"                                       # wrong category
+    rows.append(rows[0])                                               # duplicate
+    del rows[2]                                                        # missing
+    write_csv(cache, rows)
+    bank = glob.glob(os.path.join(user_dir("cache"), "scan_BANK_*.csv"))[0]
+    brows = read_csv(bank)
+    brows[0].name = brows[1].name                                      # duplicate name
+    write_csv(bank, brows[:-1])                                        # one sound missing
+    rep = catalogcheck.run(game, ["INT", "BANK"], log=lambda *a: None)
+    kinds = {(a, k) for a, k, _, _ in rep.issues}
+    assert {("INT", "category"), ("INT", "duplicate"), ("INT", "missing"), ("BANK", "duplicate"),
+            ("BANK", "missing")} <= kinds
+
+
+def test_real_catalog_quick(real_game_dir):
+    rep = catalogcheck.run(Game(real_game_dir), quick=True, log=lambda *a: None)
+    assert rep.ok, rep.issues[:10]

tests/test_cli.py

@@ -15,24 +15,24 @@ def run(capsys, *args):
 def test_cli_workflow(synth_game, tmp_path, capsys):
     root, originals = synth_game
     g = ["--game", root, "--lang", "INT"]
-    out = run(capsys, "find", "radioscanner", *g)
-    assert "CHR_RadioScanner_2" in out
+    out = run(capsys, "find", "synth radio", *g)
+    assert "Speech_Synth_Radio_02" in out
     wav = str(tmp_path / "line.wav")
     x, sr = synth.tone(1.0, 44100, 660)
     audio.save_wav(wav, x, sr)
-    out = run(capsys, "replace", "CHR_RadioScanner_2", wav, *g)
-    assert "replaced CHR_RadioScanner_2" in out
-    assert "CHR_RadioScanner_2" in run(capsys, "status", *g)
-    assert "modified" in run(capsys, "info", "CHR_RadioScanner_2", *g)
+    out = run(capsys, "replace", "Speech_Synth_Radio_02", wav, *g)
+    assert "replaced Speech_Synth_Radio_02" in out
+    assert "Speech_Synth_Radio_02" in run(capsys, "status", *g)
+    assert "modified" in run(capsys, "info", "Speech_Synth_Radio_02", *g)
     orig = str(tmp_path / "orig.wem")
-    run(capsys, "extract", "CHR_RadioScanner_2", orig, *g)
+    run(capsys, "extract", "Speech_Synth_Radio_02", orig, *g)
     assert open(orig, "rb").read() == originals["%08X:1" % synth.SPEECH_ENTRY]
     mod = str(tmp_path / "m.acmod")
     assert "1 sounds" in run(capsys, "export-mod", mod, *g)
     assert "reverted 1" in run(capsys, "revert", "--all", *g)
     assert "original" in run(capsys, "status", *g)
     assert "exact" in run(capsys, "import-mod", mod, *g)
-    run(capsys, "revert", "CHR_RadioScanner_2", *g)
+    run(capsys, "revert", "Speech_Synth_Radio_02", *g)
 
 
 def test_cli_errors_are_friendly(synth_game, capsys):

tests/test_containers.py

@@ -0,0 +1,110 @@
+"""LZO, UE3 packages and soundbanks (ackit/lzo.py, upk.py, bnk.py)."""
+import os
+
+import pytest
+
+import synth
+from ackit import lzo
+from ackit.bnk import Bank, find_banks, fnv1, IN_MEMORY
+from ackit.upk import Package
+
+
+# --- LZO ----------------------------------------------------------------------------------------
+@pytest.mark.parametrize("n", [0, 1, 3, 4, 17, 18, 19, 238, 239, 273, 274, 529, 0x20000])
+def test_lzo_stored_roundtrip(n):
+    data = os.urandom(n)
+    assert lzo.decompress(lzo.compress_stored(data), n) == data
+
+
+def test_lzo_matches():
+    # "abc" + M3 match (9 bytes at distance 3) + end
+    assert lzo.decompress(bytes([20]) + b"abc" + bytes([39, 0x08, 0x00]) + lzo.EOF) == b"abc" * 4
+    # "abcd" + M2 match (4 bytes at distance 4) + end
+    assert lzo.decompress(bytes([21]) + b"abcd" + bytes([108, 0]) + lzo.EOF) == b"abcd" * 2
+
+
+def test_lzo_errors():
+    with pytest.raises(lzo.LzoError):
+        lzo.decompress(bytes([25]) + b"abc")                       # truncated
+    with pytest.raises(lzo.LzoError):
+        lzo.decompress(lzo.compress_stored(b"hello") + b"x")        # trailing bytes
+    with pytest.raises(lzo.LzoError):
+        lzo.decompress(lzo.compress_stored(b"hello"), 6)             # wrong size
+
+
+# --- packages -----------------------------------------------------------------------------------
+@pytest.fixture
+def pkg(tmp_path):
+    body = bytes(range(256)) * 80                                     # 20480 bytes: 5 blocks, 2 chunks
+    p = str(tmp_path / "Synth.upk")
+    start = synth.package(p, body)
+    return p, body, start
+
+
+def test_package_read(pkg):
+    p, body, start = pkg
+    pk = Package(p)
+    assert pk.compressed and pk.version == 805 and pk.header_size == start
+    assert pk.size == start + len(body) and len(pk.chunks) == 2 and len(pk.blocks) == 5
+    assert pk.read(start + 4000, 300) == body[4000:4300]                # crosses a block boundary
+    assert pk.read_all() == (start, body)
+
+
+def test_package_patch(pkg):
+    p, body, start = pkg
+    raw = open(p, "rb").read()
+    pk = Package(p)
+    assert pk.patched([]) == raw                                       # nothing changed -> same bytes
+    new = pk.patched([(start + 16334, b"X" * 100)])                    # second chunk, across two blocks
+    q = p + ".new"
+    open(q, "wb").write(new)
+    pk2 = Package(q)
+    exp = bytearray(body)
+    exp[16334:16434] = b"X" * 100
+    assert pk2.read_all()[1] == bytes(exp)
+    # the first chunk is copied verbatim
+    c0 = pk.chunks[0]
+    assert new[c0[2]:c0[2] + c0[3]] == raw[c0[2]:c0[2] + c0[3]]
+    with pytest.raises(ValueError):
+        pk.patched([(start + len(body) - 10, b"too long" * 4)])
+
+
+# --- banks --------------------------------------------------------------------------------------
+def test_bank_parse():
+    media = {0x111: b"RIFF" + bytes(41), 0x222: b"RIFF" + bytes(100)}
+    b = synth.bank(fnv1("CHR_Test"), media, name="CHR_Test")
+    bk = Bank(b)
+    assert bk.id == fnv1("CHR_Test") and bk.version == 56 and bk.size == len(b)
+    assert bk.names() == {bk.id: "CHR_Test"}
+    m = {x.id: x for x in bk.media()}
+    assert b[m[0x222].off:m[0x222].off + m[0x222].size] == media[0x222]
+    assert m[0x222].off % 16 == (bk.chunks["DATA"][0]) % 16
+    src = {s.media: s for s in bk.sources()}
+    assert src[0x111].stream == IN_MEMORY and (src[0x111].off, src[0x111].size) == (m[0x111].off, m[0x111].size)
+
+
+def test_find_banks_in_buffer():
+    a = synth.bank(1, {5: b"x" * 20})
+    b = synth.bank(2, {6: b"y" * 30})
+    buf = b"junkBKHDjunk" + a + bytes(7) + b
+    assert [x.id for x in find_banks(buf)] == [1, 2]
+
+
+def test_fnv1():
+    assert fnv1("Init") == 0x50C63A23 and fnv1("Music") == 0xEDF036D6
+
+
+# --- real game (read-only) ----------------------------------------------------------------------
+def test_real_package_blocks(real_game_dir):
+    from ackit.game import Game
+    p = os.path.join(Game(real_game_dir).cooked, "BmGame.upk")
+    pk = Package(p)
+    assert pk.compressed and pk.patched([]) == open(p, "rb").read()
+    start, data = pk.read_all()
+    banks = list(find_banks(data))
+    assert len(banks) > 20
+    bk = banks[0]
+    m = bk.media()[0]
+    off = start + bk.at + m.off
+    assert pk.read(off, m.size) == data[bk.at + m.off:bk.at + m.off + m.size]
+    assert pk.read(off, 4) == b"RIFF"

tests/test_lines.py

@@ -0,0 +1,147 @@
+"""Speech lines across languages (ackit/lines.py) on a small two-language game built like the real one:
+one localized AkBank export holds, per declared language, the BKPK table of the speech entry and a bank whose
+Sound objects (same IDs in every language) name that language's recordings."""
+import hashlib
+import os
+import struct
+
+from ackit import banks
+from ackit.bnk import fnv1
+from ackit.catalog import build_lines
+from ackit.game import Game
+from ackit.wem import fill
+from ackit.workspace import ORIGINAL, MODIFIED, Workspace
+import synth
+
+ENTRY, COPY_ENTRY = 0x51000001, 0x52000001
+EV_A, EV_B = "Speech_Lines_A", "Speech_Lines_B"
+# media: line A has its own ID per language; line B uses one ID in both languages (as cinematics do)
+A = {"INT": 0x10, "ITA": 0x20}
+B = 0x77
+
+
+def _table(entry, rows):
+    """BKPK table: rows [(media, wem bytes, offset in the entry)]."""
+    t = bytearray(0x48)
+    struct.pack_into("<4sII", t, 0, b"BKPK", 0x01020000, entry)
+    end = max(o + len(w) for _, w, o in rows)
+    struct.pack_into("<IIIIII", t, 0x10, len(rows), 0, 36, 0x48, synth._a(end, 0x800), 0)
+    return bytes(t) + b"".join(struct.pack("<III", m, len(w), o) for m, w, o in rows)
+
+
+def _offsets(wems):
+    out, pos = [], 0
+    for w in wems:
+        pos = synth._a(pos, 0x800)
+        out.append(pos)
+        pos += len(w)
+    return out
+
+
+def make(root, whole_copy=False):
+    """whole_copy: INT's bank also stores INT's recording of line A whole, in memory (same media ID)."""
+    cooked = os.path.join(root, "BmGame", "CookedPCConsole")
+    a = {"INT": synth.wem(0.6, 400), "ITA": synth.wem(0.7, 420)}
+    b = {"INT": synth.wem(1.1, 600), "ITA": synth.wem(0.9, 640)}
+    # WEMs are stored in media-ID order: INT [A 0x10, B 0x77], ITA [A 0x20, B 0x77] -> here INT puts A first,
+    # ITA is written the other way round to show that positions do not matter
+    order = {"INT": [a["INT"], b["INT"]], "ITA": [b["ITA"], a["ITA"]]}
+    rows = {}
+    for L, ws in order.items():
+        offs = _offsets(ws)
+        media = [A[L], B] if L == "INT" else [B, A[L]]
+        rows[L] = list(zip(media, ws, offs))
+    wads = {"INT": ("English(US)", "28CCF006_LOC_INT.wad"), "ITA": ("Italian", "49D84887_LOC_ITA.wad")}
+    for L, (folder, name) in wads.items():
+        payloads = {ENTRY: synth.grouped(order[L])}
+        if L == "INT":
+            payloads[COPY_ENTRY] = a["INT"]            # the same recording stored a second time
+        synth.write_wad(os.path.join(cooked, folder, name), payloads)
+    # the localized export: language list, then per language its BKPK table(s) and its bank
+    lst = struct.pack("<I", 2) + synth._fstr("INT") + synth._fstr("ITA")
+    blob = bytes(4) + struct.pack("<HHiiii", 9, 0x84, 0, 0, len(lst), 0) + lst
+    bank_id = fnv1("Speech_Lines")
+    for L in ("INT", "ITA"):
+        blob += _table(ENTRY, rows[L])
+        mem = {A[L]: a[L]} if whole_copy and L == "INT" else {}
+        blob += synth.bank(bank_id, mem, name="Speech_Lines", events=[(EV_A, A[L]), (EV_B, B)]) + bytes(16)
+    # the copy entry's table sits in a second localized export (INT and ITA; ITA has no such entry: no rows)
+    blob2 = bytes(4) + struct.pack("<HHiiii", 9, 0x84, 0, 0, len(lst), 0) + lst
+    blob2 += _table(COPY_ENTRY, [(A["INT"], a["INT"], 0)]) + bytes(32) + _table(COPY_ENTRY, [(0x999, b"x" * 8, 0)])
+    synth.ue_package(os.path.join(cooked, "Lines.upk"),
+                     [("AkEvent", EV_A, bytes(8)), ("AkEvent", EV_B, bytes(8)),
+                      ("AkBank", "Speech_Lines", blob), ("AkBank", "Speech_Lines_Copy", blob2)])
+    with open(os.path.join(cooked, "CookedBanks.txt"), "w", newline="\r\n") as f:
+        f.write("Speech_Lines\n")
+    os.makedirs(os.path.join(root, "Binaries", "Win32"), exist_ok=True)
+    open(os.path.join(root, "Binaries", "Win32", "BatmanAC.exe"), "wb").write(b"MZ")
+    return a, b
+
+
+def test_tables_take_the_language_their_export_declares(tmp_path):
+    root = str(tmp_path / "game")
+    a, b = make(root)
+    sc = banks.scan(Game(root).cooked)
+    assert set(sc.packs[ENTRY]) == {"INT", "ITA"}
+    # the shared media ID has a different position in each language: nothing is merged
+    assert sc.packs[ENTRY]["INT"][B] != sc.packs[ENTRY]["ITA"][B]
+    assert sc.packs[ENTRY]["INT"][B][0] == synth._a(len(a["INT"]), 0x800)
+    assert sc.packs[ENTRY]["ITA"][B][0] == 0
+    assert B in sc.lang_media["INT"] and B in sc.lang_media["ITA"] and A["ITA"] in sc.lang_media["ITA"]
+
+
+def test_lines_are_one_to_one_across_languages(tmp_path):
+    root = str(tmp_path / "game")
+    make(root)
+    game = Game(root)
+    lines = build_lines(game, banks.scan(game.cooked))
+    assert not [i for i in lines.issues if i[0] != "row without WEM"]       # (the dummy ITA copy-table row)
+    la, lb = lines.line_of["INT:%08X:0" % ENTRY], lines.line_of["INT:%08X:1" % ENTRY]
+    assert la != lb
+    assert lines.line_of["ITA:%08X:1" % ENTRY] == la                        # A: position 1 in ITA
+    assert lines.line_of["ITA:%08X:0" % ENTRY] == lb                        # B: same media ID, position 0
+    assert lines.members[la] == {"INT": "INT:%08X:0" % ENTRY, "ITA": "ITA:%08X:1" % ENTRY}
+    assert lines.copy_of == {"INT:%08X" % COPY_ENTRY: "INT:%08X:0" % ENTRY}
+    assert lines.line_of["INT:%08X" % COPY_ENTRY] == la
+    assert lines.stats["lines_in_every_language"] == 2
+
+
+def test_names_and_copies_through_the_workspace(tmp_path):
+    root = str(tmp_path / "game")
+    a, _ = make(root)
+    ws = Workspace(root, str(tmp_path / "store"), ["INT"])
+    s = ws.catalog.by_key["INT:%08X:0" % ENTRY]
+    ita = Workspace(root, str(tmp_path / "store_ita"), ["ITA"]).catalog.by_key["ITA:%08X:1" % ENTRY]
+    assert s.name == ita.name == EV_A and s.line == ita.line                # one name, one line, any position
+    copy = ws.catalog.by_key["INT:%08X" % COPY_ENTRY]
+    assert copy.line == s.line and [c.key for c in ws.copies(s)] == [copy.key]
+    wad = ws.wad_path(s)
+    before = hashlib.sha1(open(wad, "rb").read()).hexdigest()
+    new = synth.wem(0.3, 900)
+    ws.apply([(s, new, {})])                                                # written into both places
+    assert ws.current_bytes(s) == new and ws.current_bytes(copy) == new
+    assert ws.state(s) == MODIFIED and ws.state(copy) == MODIFIED
+    ws.revert([s])                                                          # and reverted together
+    assert ws.state(s) == ORIGINAL and ws.state(copy) == ORIGINAL and not ws.store.entries
+    assert hashlib.sha1(open(wad, "rb").read()).hexdigest() == before
+    assert ws.current_bytes(copy) == a["INT"]
+
+
+def test_whole_recording_in_a_bank_is_written_with_its_line(tmp_path):
+    """A streamed recording can also be stored whole in a bank (same media ID, e.g. the game-over taunts): like
+    a prefetch head, that copy belongs to the WAD line and is written with it; it is not a sound of its own."""
+    root = str(tmp_path / "game")
+    a, _ = make(root, whole_copy=True)
+    ws = Workspace(root, str(tmp_path / "store"), ["INT", "BANK"])
+    assert "BANK:%08X" % A["INT"] not in ws.catalog.by_key
+    s = ws.catalog.by_key["INT:%08X:0" % ENTRY]
+    (l,) = ws.locations(s)
+    assert l.kind == banks.PREFETCH_HEAD and l.size == len(a["INT"]) and l.file == "Lines.upk"
+    pkg = os.path.join(ws.game.cooked, "Lines.upk")
+    before = hashlib.sha1(open(pkg, "rb").read()).hexdigest()
+    new = synth.fitting_wem(ws, s, 0.3, 900)
+    ws.apply([(s, new, {})])
+    assert banks.open_file(pkg).read(l.offset, l.size) == fill(new, l.size)     # filled exactly, not padded
+    assert ws.current_bytes(s) == new
+    ws.revert([s])
+    assert hashlib.sha1(open(pkg, "rb").read()).hexdigest() == before

tests/test_texts.py

@@ -0,0 +1,146 @@
+"""Subtitles (dialogue objects in packages) and menu text (localization files): parse, apply, revert."""
+import hashlib
+import os
+
+import pytest
+
+import synth
+from ackit import loctext, subtitles
+from ackit.game import Game
+from ackit.texts import MODIFIED, ORIGINAL, TextConflictError
+from ackit.workspace import Workspace
+
+SUB_KEY = "SUB:INT:%s:0" % synth.SPEECH_EVENT
+
+
+def _sha(p):
+    return hashlib.sha1(open(p, "rb").read()).hexdigest()
+
+
+def _ws(root, tmp_path):
+    return Workspace(root, store_root=str(tmp_path / "store"), archives=["INT", "SFX", "BANK"])
+
+
+def _files(root):
+    d = os.path.join(root, "BmGame")
+    return {os.path.join(a, f): _sha(os.path.join(a, f)) for a, _, fs in os.walk(d) for f in fs}
+
+
+# --- formats ------------------------------------------------------------------------------------
+def test_fstring_encoding():
+    b = subtitles.encode_fstring("Hi", 10, "a")
+    assert b == b"\x06\x00\x00\x00Hi\0\0\0\0" and subtitles.read_fstring(b, 0)[0] == "Hi"
+    w = subtitles.encode_fstring("Привет", 4 + 2 * 8, "w")
+    assert subtitles.read_fstring(w, 0) == ("Привет", 20, "w")
+    with pytest.raises(subtitles.SubtitleError):
+        subtitles.encode_fstring("too long", 10, "a")
+    with pytest.raises(subtitles.SubtitleError):
+        subtitles.encode_fstring("→", 10, "a")                      # not in cp1252
+
+
+def test_menu_parse_and_write():
+    fname, text = synth.MENU["INT"]
+    raw = b"\xff\xfe" + text.encode("utf-16-le")
+    es = {e.key: e for e in loctext.parse(raw, "INT", fname)}
+    assert es["INT:FRONTEND.int:Menu:Start"].value == "Press any key"
+    assert es["INT:FRONTEND.int:Menu:Quit"].quoted and es["INT:FRONTEND.int:Tips:Tip#2"].value == "Glide over the city"
+    assert loctext.write(raw, "INT", fname, {}) == raw
+    new = loctext.write(raw, "INT", fname, {"INT:FRONTEND.int:Menu:Quit": "Leave", "INT:FRONTEND.int:Tips:Tip#2": "Fly"})
+    assert new.decode("utf-16") == text.replace('"Quit game"', '"Leave"').replace("Glide over the city", "Fly")
+    with pytest.raises(loctext.TextError):
+        loctext.write(raw, "INT", fname, {"INT:FRONTEND.int:Menu:Start": "two\nlines"})
+
+
+# --- subtitles through the workspace ------------------------------------------------------------
+def test_subtitle_catalog_and_link(synth_game, tmp_path):
+    root, _ = synth_game
+    ws = _ws(root, tmp_path)
+    lines = ws.texts.subtitles("INT")
+    assert [(s.speech, s.line, s.text) for s in lines] == [(synth.GRUNT_EVENT, 0, "Hey you!"),
+                                                          (synth.SPEECH_EVENT, 0, "Scanner line two."),
+                                                          (synth.SPEECH_EVENT, 1, "Second part.")]
+    # a line kept in memory: each language's subtitle links to that language's BANK sound
+    grunt = ws.catalog.by_key["BANK:%08X" % synth.GRUNT_MEDIA["INT"]]
+    assert [s.text for s in ws.texts.for_sound(grunt)] == ["Hey you!"]
+    ita = {s.speech: s.sound for s in ws.texts.subtitles("ITA")}
+    assert ita[synth.GRUNT_EVENT] == "BANK:%08X" % synth.GRUNT_MEDIA["ITA"]
+    line = ws.catalog.by_key["INT:%08X:1" % synth.SPEECH_ENTRY]
+    assert [s.text for s in ws.texts.for_sound(line)] == ["Scanner line two.", "Second part."]
+    assert ita[synth.SPEECH_EVENT] == line.key                        # no ITA speech WAD here: INT line
+    assert ws.texts.subtitles("RUS")[0].enc == "w"
+    assert sorted(os.path.basename(p) for p in ws.texts.files(SUB_KEY)) == ["Synth_Dialogue.upk", "Synth_Dialogue2.upk"]
+    assert ws.texts.limit(SUB_KEY) == len("Scanner line two.")
+
+
+def test_subtitle_apply_revert_every_copy(synth_game, tmp_path):
+    root, _ = synth_game
+    ws = _ws(root, tmp_path)
+    before = _files(root)
+    ws.texts.apply({SUB_KEY: "Line two.", "SUB:RUS:%s:0" % synth.SPEECH_EVENT: "Две"})
+    assert ws.texts.current(SUB_KEY) == "Line two." and ws.texts.state(SUB_KEY) == MODIFIED
+    for p in ws.texts.files(SUB_KEY):                                # every copy
+        ws2 = _ws(root, tmp_path / "other")
+        assert ws2.texts.current(SUB_KEY) == "Line two."
+    with pytest.raises(subtitles.SubtitleError):
+        ws.texts.apply({SUB_KEY: "This line is far too long for the room"})
+    ws.texts.revert([SUB_KEY])
+    assert ws.texts.current(SUB_KEY) == "Scanner line two." and ws.texts.current("SUB:RUS:%s:0" % synth.SPEECH_EVENT) == "Две"
+    ws.texts.revert_all()
+    assert _files(root) == before and ws.texts.state(SUB_KEY) == ORIGINAL
+
+
+def test_subtitle_and_sound_in_one_package(synth_game, tmp_path):
+    root, _ = synth_game
+    ws = _ws(root, tmp_path)
+    before = _files(root)
+    s = ws.catalog.by_key["BANK:%08X" % synth.DLG_MEDIA]
+    ws.apply([(s, synth.fitting_wem(ws, s, 0.1, 1900), {})])
+    ws.texts.apply({SUB_KEY: "Both."})
+    assert ws.texts.current(SUB_KEY) == "Both." and ws.state(s) == "modified"
+    ws.revert([s])
+    assert ws.texts.current(SUB_KEY) == "Both." and ws.state(s) == "original"
+    ws.texts.revert_all()
+    assert _files(root) == before
+
+
+# --- menu text through the workspace ------------------------------------------------------------
+def test_menu_apply_revert(synth_game, tmp_path):
+    root, _ = synth_game
+    ws = _ws(root, tmp_path)
+    before = _files(root)
+    assert "INT" in ws.texts.menu_languages()
+    keys = {e.key: e for e in ws.texts.menu("INT")}
+    assert "INT:Version.int:ID:Unreal" not in keys
+    k = "MENU:INT:FRONTEND.int:Menu:Start"
+    ws.texts.apply({k: "Press Enter"})
+    assert ws.texts.current(k) == "Press Enter" and ws.texts.state(k) == MODIFIED
+    assert {e.key: e.value for e in ws.texts.menu("INT")}["INT:FRONTEND.int:Menu:Start"] == "Press Enter"
+    ws.texts.revert([k])
+    assert _files(root) == before
+
+
+def test_menu_changed_underneath(synth_game, tmp_path):
+    root, _ = synth_game
+    ws = _ws(root, tmp_path)
+    k = "MENU:INT:FRONTEND.int:Tips:Tip#2"
+    ws.texts.apply({k: "Glide"})
+    path = ws.texts.files(k)[0]
+    raw = open(path, "rb").read()
+    open(path, "wb").write(loctext.write(raw, "INT", "FRONTEND.int", {"INT:FRONTEND.int:Menu:Start": "Other tool"}))
+    with pytest.raises(TextConflictError):
+        ws.texts.apply({k: "Soar"})
+    ws.texts.apply({k: "Soar"}, force=True)
+    assert ws.texts.current(k) == "Soar"
+    ws.texts.revert_all()
+    assert open(path, "rb").read() == open(ws.store.full_copy(ws.game.rel(path)), "rb").read()
+
+
+# --- real game (read-only) ----------------------------------------------------------------------
+def test_real_subtitles_and_menu(real_game_dir, tmp_path):
+    ws = Workspace(Game(real_game_dir), store_root=str(tmp_path / "store"), archives=["ITA"])
+    lines = ws.texts.subtitles("ITA")
+    assert len(lines) > 9000
+    s = next(x for x in lines if x.sound)
+    assert ws.texts.current(subtitles.key("ITA", s.speech, s.line)) == s.text
+    menu = ws.texts.menu("ITA")
+    assert len(menu) > 7000 and all(e.lang == "ITA" for e in menu)

tests/test_workspace.py

@@ -30,7 +30,7 @@ def test_apply_in_place_and_revert_byte_identical(synth_game):
     root, originals = synth_game
     before = files(root)
     ws = ws_for(root)
-    s = ws.sound("CHR_RadioScanner_2")
+    s = ws.sound("Speech_Synth_Radio_02")
     x, sr = synth.tone(1.2, 24000, 900)
     wem, info = ws.prepare(s, x, sr)
     assert len(wem) <= ws.limit(s)
@@ -74,7 +74,7 @@ def test_size_limit_enforced(synth_game):
 def test_reapply_uses_original_limit(synth_game):
     root, _ = synth_game
     ws = ws_for(root)
-    s = ws.sound("CHR_RadioScanner_2")
+    s = ws.sound("Speech_Synth_Radio_02")
     lim = ws.limit(s)
     ws.apply([(s, synth.wem(0.3, 1000), {})])        # much smaller
     assert ws.limit(s) == lim                          # still the original's size
@@ -119,10 +119,44 @@ def test_full_backup_taken_once(synth_game):
     assert rel in restored and ws.state(s) == W.ORIGINAL
 
 
+def test_mod_from_another_language_matched_by_line(synth_game, tmp_path):
+    """A voice line from another language is found through the game's line identity (its WAD position there
+    can differ); a voice line from a package without line identities is never guessed by name."""
+    import json
+    import zipfile
+    root, _ = synth_game
+    ws = ws_for(root, store_root=str(tmp_path / "store"))
+    s = ws.catalog.by_key["INT:%08X:1" % synth.SPEECH_ENTRY]
+    assert s.line and s.name == synth.SPEECH_EVENT          # the game's line, named after its event
+    ws.apply([(s, synth.wem(0.3, 1000), {})])
+    pkg = str(tmp_path / "other_lang.acmod")
+    export_mod(ws, pkg, ws.modified_keys(), "t")
+    ws.revert_all()
+
+    def rewrite(change):
+        with zipfile.ZipFile(pkg) as z:
+            files = {n: z.read(n) for n in z.namelist()}
+        man = json.loads(files["manifest.json"])
+        change(man["items"][0])
+        files["manifest.json"] = json.dumps(man).encode()
+        with zipfile.ZipFile(pkg, "w") as z:
+            for n, b in files.items():
+                z.writestr(n, b)
+
+    def other_language(it):                               # as if made on the ITA game, at another position
+        it["archive"], it["target"], it["name"] = "ITA", "%08X:2" % synth.SPEECH_ENTRY, "Speech_Synth_Radio_02"
+    rewrite(other_language)
+    _, items = match_mod(ws, pkg)
+    assert items[0].how == "same line" and items[0].sound is s
+    rewrite(lambda it: it.pop("line"))                     # an older package: names alone are no proof
+    _, items = match_mod(ws, pkg)
+    assert items[0].how == "missing" and "older version" in items[0].note
+
+
 def test_mod_export_import_roundtrip(synth_game, synth_template, tmp_path):
     root, _ = synth_game
     ws = ws_for(root)
-    a = ws.sound("CHR_RadioScanner_2")
+    a = ws.sound("Speech_Synth_Radio_02")
     b = ws.catalog.by_key["SFX:%08X/%08X" % (synth.PACK_ID, synth.MEDIA[0])]
     wa, wb = synth.wem(1.0, 820), synth.wem(2.5, 410, 48000, 2)
     ws.apply([(a, wa, {}), (b, wb, {})])
@@ -136,7 +170,7 @@ def test_mod_export_import_roundtrip(synth_game, synth_template, tmp_path):
     ws2 = ws_for(other, store_root=str(tmp_path / "store2"))
     _, items = match_mod(ws2, pkg)
     assert [i.how for i in items] == ["exact", "exact"]
-    res = import_mod(ws2, pkg)
+    res, _ = import_mod(ws2, pkg)
     assert all(i.note == "exact copy" for i in res)
     assert ws2.current_bytes(ws2.catalog.by_key[a.key]) == wa
     assert ws2.current_bytes(ws2.catalog.by_key[b.key]) == wb

tools/build_catalog.py

@@ -4,9 +4,15 @@
 
 Outputs
   data/catalog/characters.csv   name -> verified speaker, from data/catalog/source/<lang>/*.txt
+  (speech sounds carry the game's line identity and one name in every language: ackit/lines.py)
   data/catalog/<ARCHIVE>.csv    every sound of one archive (schema: data/catalog/README.md)
-  data/catalog/index.json       WAD fingerprints the CSVs were built from + counts
-Run it only on clean game files (`ackit status` must say nothing is modified).
+  data/catalog/BANK.csv         in-memory sounds (soundbanks in SFX/*.bnk and the .upk packages)
+  data/catalog/BANK_locations.csv.gz   every stored copy of them + the prefetch heads of speech lines
+  data/catalog/SUB_<LANG>.csv   subtitles of the speech lines (one file per language)
+  data/catalog/SUB_locations.csv.gz    every stored copy of each subtitle string
+  data/catalog/index.json       fingerprints the CSVs were built from + counts
+Run it only on clean game files (`ackit status` must say nothing is modified). It stops before writing
+anything if a sound differs from the current catalog (a modified game); after a game update use --accept-changes.
 """
 import argparse
 import collections
@@ -15,10 +21,12 @@ import hashlib
 import json
 import os
 import sys
+import time
 
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-from ackit import settings  # noqa: E402
-from ackit.catalog import scan_archive, write_csv, hash_names  # noqa: E402
+from ackit import banks, settings, subtitles  # noqa: E402
+from ackit.catalog import (apply_lines, character_of, read_csv, scan_archive, scan_text,  # noqa: E402
+                           write_csv)
 from ackit.classify import USER_LABELS  # noqa: E402
 from ackit.game import Game, default_game_dir, wad_fingerprint  # noqa: E402
 from ackit.inject import read_target  # noqa: E402
@@ -45,17 +53,28 @@ def read_user_lists(src_dir):
     return out
 
 
-def build_characters(sounds, src_root):
-    """Map the user's wemscan-style names onto catalog names (name or alias)."""
-    by_alias = collections.defaultdict(list)
-    for s in sounds:
-        for a in s.names():
-            by_alias[a.lower()].append(s)
-    rows, unmatched, ambiguous = {}, [], []
-    for lang in sorted(os.listdir(src_root)):
-        d = os.path.join(src_root, lang)
+def build_characters(game, src_root, final):
+    """Map the curated lists onto the catalog. Each list folder (source/<lang>/) names sounds of THAT language's
+    WAD by position (wemscan names: NAME = single entry, NAME_i = i-th WEM of the entry). Positions differ
+    between languages, so a listed recording is found by its position in that language and the label goes to
+    its line (`final`: {key: Sound} with line identities and names), i.e. to the same recording in every
+    language; a recording without a line keeps the label in that language only.
+    -> ({(name, language or ''): set(characters)}, unmatched names, ambiguous names, skipped folders)"""
+    rows = collections.defaultdict(set)
+    unmatched, ambiguous, skipped = [], [], []
+    for folder in sorted(os.listdir(src_root)):
+        d = os.path.join(src_root, folder)
         if not os.path.isdir(d):
             continue
+        lang = folder.upper()
+        if lang not in game.archives:
+            skipped.append(folder)          # no speech WAD of that language: positions cannot be resolved
+            continue
+        ref = scan_archive(lang, game.wad(lang), characters={})      # position names of this language
+        by_alias = collections.defaultdict(list)
+        for s in ref:
+            for a in s.names():
+                by_alias[a.lower()].append(s)
         for uname, who in read_user_lists(d).items():
             hits = list({id(s): s for s in by_alias.get(uname, [])}.values())
             if not hits:
@@ -64,9 +83,27 @@ def build_characters(sounds, src_root):
             if len(hits) > 1:
                 ambiguous.append(uname)  # wemscan name collision: both candidates get the label
             for s in hits:
-                cur = rows.setdefault(s.name, set())
-                cur |= who
-    return rows, unmatched, ambiguous
+                f = final[s.key]
+                rows[(f.name, "" if f.line else lang)] |= who
+    return rows, unmatched, ambiguous, skipped
+
+
+def changed_since_catalog(game, out_dir, bank_snd):
+    """Keys whose original checksum in the existing catalog differs from the game files now."""
+    out = []
+    for arch, path in game.archives.items():
+        p = os.path.join(out_dir, "%s.csv" % arch)
+        if not os.path.exists(p):
+            continue
+        wad = Wad(path)
+        for s in read_csv(p):
+            if s.sha1 and s.sha1 != file_sha1(path, s.target, wad):
+                out.append(s.key)
+    p = os.path.join(out_dir, "%s.csv" % banks.ARCHIVE)
+    if os.path.exists(p):
+        old = {s.key: s.sha1 for s in read_csv(p)}
+        out += [s.key for s in bank_snd if old.get(s.key) and s.sha1 and old[s.key] != s.sha1]
+    return out
 
 
 def file_sha1(path, target, wad):
@@ -76,28 +113,61 @@ def file_sha1(path, target, wad):
 def main():
     ap = argparse.ArgumentParser()
     ap.add_argument("--game")
+    ap.add_argument("--accept-changes", action="store_true",
+                    help="the game was updated: accept sounds whose bytes differ from the current catalog")
     a = ap.parse_args()
     game = Game(a.game or default_game_dir(settings.load()))
     out_dir = catalog_dir()
     src_root = os.path.join(out_dir, "source")
-    hash_names()
 
-    # characters from the curated lists (names come from the language they were made for: ITA)
-    ref_lang = "ITA" if "ITA" in game.archives else game.languages[0]
-    ref = scan_archive(ref_lang, game.wad(ref_lang), characters={})
-    chars, unmatched, ambiguous = build_characters(ref, src_root)
+    # 1. every package once: in-memory sounds, prefetch heads, subtitles, events (a few minutes)
+    t0 = time.time()
+    bank_snd, locs, sc, texts, sublocs, sub_issues, lines = scan_text(
+        game, progress=lambda k, n: print("  banks + subtitles: %d/%d files" % (k, n), flush=True)
+        if k % 500 == 0 else None, characters={})
+    st = lines.stats
+    print("packages scanned (%.0fs); %d recordings in %d lines, %d of them in every language (%s); "
+          "%d recordings without a line; %d relation issues"
+          % (time.time() - t0, st["recordings"], st["lines"], st["lines_in_every_language"], "/".join(lines.langs),
+             len(st["without_line"]), len(lines.issues)))
+    for k, d in lines.issues[:10]:
+        print("  issue:", k, d)
+
+    # safety: the catalog records ORIGINAL bytes. A sound that differs from the current catalog means a modified
+    # game (an applied change, another tool) unless the game itself was updated -> stop before writing anything.
+    changed = changed_since_catalog(game, out_dir, bank_snd)
+    if changed and not a.accept_changes:
+        print("STOP: %d sound(s) differ from the current catalog, e.g. %s." % (len(changed), changed[:5]))
+        print("Revert all changes (ackit status / ackit revert --all) and run again; after a game update use "
+              "--accept-changes.")
+        sys.exit(2)
+
+    # 2. every speech sound with its line identity and name, then the curated speaker lists through them
+    wad_sounds = {}
+    for arch, path in game.archives.items():
+        sounds = scan_archive(arch, path, characters={})
+        wad_sounds[arch] = apply_lines(sounds, lines, lines.names) if arch in lines.langs else sounds
+    final = {s.key: s for ss in wad_sounds.values() for s in ss}
+    rows, unmatched, ambiguous, skipped = build_characters(game, src_root, final)
     with open(os.path.join(out_dir, "characters.csv"), "w", encoding="utf-8", newline="") as f:
         w = csv.writer(f, lineterminator="\n")
-        w.writerow(["name", "character"])
-        for n in sorted(chars, key=str.lower):
-            w.writerow([n, "; ".join(sorted(chars[n]))])
-    print("characters.csv: %d sounds labelled; %d unmatched names %s; %d ambiguous %s"
-          % (len(chars), len(unmatched), unmatched[:3], len(ambiguous), ambiguous[:3]))
-    charmap = {n.lower(): v for n, v in ((n, "; ".join(sorted(c))) for n, c in chars.items())}
+        w.writerow(["name", "character", "language"])
+        for (n, lg) in sorted(rows, key=lambda r: (r[0].lower(), r[1])):
+            w.writerow([n, "; ".join(sorted(rows[(n, lg)])), lg])
+    print("characters.csv: %d sounds labelled (%d by line name, %d language-only); %d unmatched names %s; "
+          "%d ambiguous %s; skipped list folders without a speech WAD: %s"
+          % (len(rows), sum(1 for _, lg in rows if not lg), sum(1 for _, lg in rows if lg),
+             len(unmatched), unmatched[:3], len(ambiguous), ambiguous[:3], skipped))
+
+    def charmap(lang):
+        return {n.lower(): "; ".join(sorted(c)) for (n, lg), c in rows.items() if not lg or lg == lang}
 
     index = {"built_from": "unmodified game files", "archives": {}}
     for arch, path in game.archives.items():
-        sounds = scan_archive(arch, path, characters=charmap)
+        cm = charmap(arch)
+        sounds = wad_sounds[arch]
+        for s in sounds:
+            s.character = character_of(s, cm)
         wad = Wad(path)
         for s in sounds:
             s.sha1 = file_sha1(path, s.target, wad)
@@ -107,6 +177,53 @@ def main():
                                    "sounds": len(sounds), "groups": dict(g),
                                    "with_character": sum(1 for s in sounds if s.character)}
         print("%s.csv: %d sounds %s" % (arch, len(sounds), dict(g)))
+
+    # in-memory sounds (scanned in step 1); localized speech lines carry the game's line names too
+    sounds = bank_snd
+    cm = charmap("")                  # only labels valid in every language (game line names)
+    for s in sounds:
+        s.character = character_of(s, cm)
+    write_csv(os.path.join(out_dir, "%s.csv" % banks.ARCHIVE), sounds)
+    banks.write_locations(os.path.join(out_dir, banks.LOCATIONS), locs)
+    g = collections.Counter(s.group for s in sounds)
+    heads = [l for l in locs if l.kind == banks.PREFETCH_HEAD]
+    played = {m for ms in banks.event_media(sc).values() for m in ms}       # reached by a named event
+    by_event = sum(1 for s in sounds if (s.line not in lines.unnamed if s.line else int(s.target, 16) in played))
+    index["archives"][banks.ARCHIVE] = {
+        "fingerprint": banks.game_fingerprint(game.cooked), "sounds": len(sounds), "groups": dict(g),
+        "named_by_event": by_event, "named_by_bank": len(sounds) - by_event,
+        "copies": sum(1 for l in locs if l.kind == banks.MEMORY), "files": len(sc.files),
+        # bank copies of WAD recordings: prefetch heads and whole in-memory copies (catalog._whole_recordings)
+        "prefetch_heads": {"recordings": len({l.media for l in heads}), "copies": len(heads),
+                           "unmapped": len({l.media for l in heads if not l.sound})},
+        "issues": ["%s: %s" % i for i in sc.issues]}
+    print("%s.csv: %d sounds %s, %d copies in %d files, %d heads of WAD recordings"
+          % (banks.ARCHIVE, len(sounds), dict(g), len(locs) - len(heads), len(sc.files), len(heads)))
+
+    # subtitles (dialogue objects in the packages), one CSV per language + every stored copy
+    for f in os.listdir(out_dir):
+        if f.startswith("SUB_") and f.endswith(".csv"):
+            os.remove(os.path.join(out_dir, f))
+    for lang, rows in texts.items():
+        subtitles.write_texts(os.path.join(out_dir, "SUB_%s.csv" % lang), rows)
+    subtitles.write_locations(os.path.join(out_dir, subtitles.LOCATIONS), sublocs)
+    any_lang = next(iter(texts.values()), [])
+    index["archives"]["SUB"] = {
+        "fingerprint": banks.game_fingerprint(game.cooked),
+        "lines": {lang: len(rows) for lang, rows in sorted(texts.items())},
+        "speech_lines": len({s.speech for s in any_lang}),
+        "linked": {lang: sum(1 for s in rows if s.sound) for lang, rows in sorted(texts.items())},
+        "copies": len(sublocs), **sc.subtitle_objects, "issues": ["%s: %s" % i for i in sub_issues]}
+    wad_lines = {lid for lid, per in lines.members.items() if any(not k.startswith("BANK:") for k in per.values())}
+    index["archives"]["LINES"] = {
+        "languages": lines.langs, "recordings": st["recordings"], "lines": st["lines"],
+        "wad_lines": len(wad_lines), "wad_lines_named_by_event": len(wad_lines - lines.unnamed),
+        "wad_lines_named_after_int_recording": len(wad_lines & lines.unnamed),
+        "lines_in_every_language": st["lines_in_every_language"],
+        "lines_missing_a_language": st["lines_missing_a_language"], "without_line": st["without_line"],
+        "issues": ["%s: %s" % i for i in lines.issues]}
+    print("SUB_*.csv: %d languages, %d subtitle strings, %d copies"
+          % (len(texts), sum(len(r) for r in texts.values()), len(sublocs)))
     # flatten for catalog.load_archive
     flat = {a: v for a, v in index["archives"].items()}
     json.dump(dict(flat, _info=index["built_from"]), open(os.path.join(out_dir, "index.json"), "w", newline="\n"), indent=1)

tools/check_docs.py

@@ -0,0 +1,557 @@
+"""Check every figure the documentation states against the data it describes (read-only).
+
+    python tools/check_docs.py [--game DIR] [--scan] [--wems]
+
+The figures are recomputed, never copied: from the shipped catalog (`data/catalog/`, built from the original
+game files) and, when the game is installed, its menu files; with --scan also from a fresh scan of every
+package (bank survey: blobs, source kinds, bank names; ~3 minutes) and from `ackit verify --catalog`'s facts
+(relations re-derived from the game; ~6 minutes more); with --wems from rebuilding every WEM of the speech and
+SFX WADs from the format rules (~10 minutes). Each claim is the exact text a document must contain
+(line breaks and repeated spaces ignored). Exit code 1 when a document states anything else.
+History logs (NOTES.md, findings.md, docs/history/) record what was true on their date and are not checked.
+"""
+import argparse
+import collections
+import csv
+import glob
+import gzip
+import os
+import re
+import sys
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, ROOT)
+
+from ackit import banks, settings, subtitles  # noqa: E402
+from ackit.catalog import read_csv  # noqa: E402
+from ackit.game import Game, default_game_dir  # noqa: E402
+from ackit.paths import catalog_dir  # noqa: E402
+
+VOICES = ["INT", "FRA", "ITA", "DEU", "ESN"]
+CLAIMS = []          # (document, text, what it is checked against)
+
+
+def n(x):
+    return "{:,}".format(x)
+
+
+def rng(values, fmt=n):
+    lo, hi = min(values), max(values)
+    return fmt(lo) if lo == hi else "%s-%s" % (fmt(lo), fmt(hi))
+
+
+def pct(a, b):
+    return int(round(100.0 * a / b))
+
+
+def claim(doc, text, why=""):
+    CLAIMS.append((doc, text, why))
+
+
+# --- figures from the shipped catalog --------------------------------------------------------------
+def catalog_claims(cat):
+    rows = {a: read_csv(os.path.join(cat, a + ".csv")) for a in VOICES + ["SFX", "BANK"]}
+    bank, sfx = rows["BANK"], rows["SFX"]
+    wad_n = {len(rows[L]) for L in VOICES}
+    assert len(wad_n) == 1, "WADs differ in size: %s" % wad_n
+    W = wad_n.pop()
+    claim("REPORT.md", "(5 languages x %s), the SFX WAD (%s)" % (n(W), n(len(sfx))), "catalog rows")
+    claim("REPORT.md", "the **%s in-memory sounds**" % n(len(bank)), "BANK.csv rows")
+    claim("docs/formats/banks.md", "so there are **%s in-memory sounds**" % n(len(bank)), "BANK.csv rows")
+
+    # languages of the in-memory sounds
+    neutral = [s for s in bank if not s.language]
+    single = collections.Counter(s.language for s in bank if s.language and ";" not in s.language)
+    multi = [s for s in bank if ";" in s.language]
+    multi_c = collections.Counter(s.language for s in multi)
+    claim("REPORT.md", "%s shared by all languages, ~%s localized speech lines per voice language, %d shared by "
+          "several" % (n(len(neutral)), n(int(round(sum(single.values()) / len(single), -1))), len(multi)))
+    claim("docs/formats/banks.md", "%s sounds are language-neutral, %s belong to one voice language, %d to several"
+          % (n(len(neutral)), rng(single.values()), len(multi)))
+    a4 = multi_c.get("ESN;FRA;INT;ITA", 0)
+    a3 = multi_c.get("FRA;INT;ITA", 0)
+    claim("docs/formats/banks.md", "(%d shared by INT/FRA/ITA/ESN while DEU has its own, %d by INT/FRA/ITA)" % (a4, a3))
+    claim("docs/formats/banks.md", "everywhere; %d of these are voice lines" % sum(1 for s in neutral if s.line))
+    claim("docs/formats/banks.md", "stored once for every language (%d sounds)"
+          % sum(1 for s in neutral if s.bank.startswith("Speech")))
+    claim("docs/formats/banks.md", "lines** (about %s per language" % n(int(round(sum(single.values()) / len(single), -1))))
+    ch = collections.Counter(s.channels for s in bank)
+    claim("docs/formats/banks.md", "of all %s in-memory sounds %s are mono, %s stereo, %d have 3-6 channels"
+          % (n(len(bank)), n(ch[1]), n(ch[2]), sum(v for k, v in ch.items() if k >= 3)))
+
+    # copies and heads
+    with gzip.open(os.path.join(cat, "BANK_locations.csv.gz"), "rt", newline="") as f:
+        loc = list(csv.DictReader(f))
+    m_rows = [r for r in loc if r["kind"] == "m"]
+    p_rows = [r for r in loc if r["kind"] == "p"]
+    keys = {s.key: s for L in VOICES for s in rows[L]}
+    whole = [r for r in p_rows if int(r["size"]) >= keys[r["sound"]].size]
+    pref = [r for r in p_rows if r not in whole]
+    whole_rec = {r["sound"] for r in whole}
+    pref_rec = {r["sound"] for r in pref}
+    pref_lines = {keys[k].line for k in pref_rec}
+    all_lines = {keys[r["sound"]].line for r in p_rows}
+    per_lang_pref = collections.Counter(k.split(":")[0] for k in pref_rec)
+    per_lang_whole = collections.Counter(k.split(":")[0] for k in whole_rec)
+    claim("REPORT.md", "copy (%s copies). %s WAD voice recordings (%d lines) also have bank copies (%d prefetch heads, %d "
+          "whole copies; %s copies)" % (n(len(m_rows)), n(len(pref_rec | whole_rec)), len(all_lines), len(pref_rec),
+                                        len(whole_rec), n(len(p_rows))))
+    claim("docs/formats/banks.md", "%d of them are **prefetch heads** of streamed WAD sounds and %d are **whole copies**"
+          % (len({r["media"] for r in pref}), len({r["media"] for r in whole})))
+    claim("docs/formats/banks.md", "%d WAD sounds have such heads (%s copies)" % (len(pref_rec), n(len(pref))))
+    claim("docs/formats/banks.md", "%d more WAD recordings (%s per language" % (len(whole_rec), rng(per_lang_whole.values())))
+    claim("docs/formats/banks.md", "byte-identical to the WAD WEM (%d copies)" % len(whole))
+    claim("docs/formats/banks.md", "Heads and whole copies together: %d recordings, %s copies"
+          % (len(pref_rec | whole_rec), n(len(p_rows))))
+    claim("docs/formats/banks.md", "the %d lines with a prefetch head (%d recordings over the 5 languages"
+          % (len(pref_lines), len(pref_rec)))
+    claim("docs/formats/banks.md", "A WAD recording that a bank also stores whole (%d)" % len(whole_rec))
+    claim("docs/decisions.md", "%d voice recordings (%d lines, %s recordings per language) have their first bytes"
+          % (len(pref_rec), len(pref_lines), rng(per_lang_pref.values(), str)))
+    claim("docs/decisions.md", "and %d more are stored whole in banks" % len(whole_rec))
+    places = len(loc)
+    media_all = {r["media"] for r in loc}
+    loose = {r["media"] for r in loc if r["file"].startswith("SFX/")}
+    pk = {r["media"] for r in loc if not r["file"].startswith("SFX/")}
+    claim("docs/formats/banks.md", "The two sets share %d media IDs: %s unique media IDs in %s places"
+          % (len(loose & pk), n(len(media_all)), n(places)))
+    claim("docs/formats/banks.md", "+ `TST_Prep_Media`) | %d | %s |"
+          % (len({r["file"] for r in loc if r["file"].startswith("SFX/")}), n(len(loose))))
+    claim("docs/formats/banks.md", "packages) | 3,340 | %s |" % n(len(pk)))
+    claim("REPORT.md", "in %d loose `.bnk` files and %d `.upk` packages"
+          % (len({r["file"] for r in loc if r["file"].startswith("SFX/")}), len({r["file"] for r in loc if r["file"].endswith(".upk")})))
+    assert len(media_all) == len(bank) + len(pref_rec | whole_rec)
+
+    # lines
+    lines_all = {s.line for L in VOICES for s in rows[L] if s.line} | {s.line for s in bank if s.line}
+    wad_lines = {L: {s.line for s in rows[L] if s.line} for L in VOICES}
+    assert len({len(v) for v in wad_lines.values()}) == 1
+    WL = len(wad_lines["INT"])
+    copies = {L: sum(1 for s in rows[L] if s.line) - len(wad_lines[L]) for L in VOICES}
+    recs = sum(len(wad_lines[L]) for L in VOICES) + len({(s.target) for s in bank if s.line})
+    for doc, text in [("REPORT.md", "%s lines (WAD + in-memory) with exactly one recording in each of the 5 voice languages"),
+                      ("docs/decisions.md", "recording per language (%s lines, all 5 languages)"),
+                      ("docs/formats/banks.md", "recordings that are sources of the same Sound objects: %s lines")]:
+        claim(doc, text % n(len(lines_all)), "distinct line ids")
+    claim("docs/formats/banks.md", "in each of the 5 voice languages (%s recordings)" % n(recs))
+    claim("REPORT.md", "recordings stored twice (%s per language) are written together" % rng(copies.values(), str))
+    claim("docs/formats/banks.md", "- %s recordings per language are stored twice" % rng(copies.values(), str).capitalize())
+    claim("docs/categories.md", "Coverage of the %s WAD voice lines" % n(WL))
+    # recordings the game shares between languages (the same bytes as the INT recording of the line)
+    intl = {s.line: s.sha1 for s in rows["INT"] if s.line}
+    same = {L: [s for s in {x.line: x for x in rows[L] if x.line}.values() if intl.get(s.line) == s.sha1]
+            for L in ("FRA", "ITA", "DEU", "ESN")}
+    ita = collections.Counter(s.category for s in same["ITA"])
+    shared = {s.target: s.language for s in bank if ";" in s.language}
+    claim("docs/formats/banks.md", "in each voice WAD %s lines are byte-identical to the INT recording (ITA %d: efforts "
+          "and emotes %d, cinematics %d, boss fights %d, story %d, radio/TV %d, character lines %d), %d in-memory voice "
+          "lines are stored once for every language (efforts, radio chatter, crowds) and %d DLC intro lines"
+          % (rng([len(v) for v in same.values()]), len(same["ITA"]), ita["Efforts & emotes"], ita["Cinematics"],
+             ita["Boss fights"], ita["Story missions"], ita["Radio & TV"], ita["Character lines"],
+             sum(1 for s in bank if s.line and not s.language), len(shared)))
+    claim("docs/decisions.md", "of the %s WAD voice lines" % n(WL))
+
+    # positions differ between languages
+    it = {s.target: s.line for s in rows["INT"] if ":" in s.target}
+    diff = []
+    for L in ["FRA", "ITA", "DEU", "ESN"]:
+        other = {s.target: s.line for s in rows[L] if ":" in s.target}
+        diff.append(100.0 * sum(1 for t in it if other.get(t) != it[t]) / len(it))
+    span = "%.1f-%.1f %%" % (min(diff), max(diff))
+    claim("docs/formats/wad.md", "(%s of the WEMs in multi-variation entries, compared with INT" % span)
+    claim("docs/decisions.md", "%s of the WEMs in multi-variation entries are at another position than in INT" % span)
+    pos = {}
+    for L in VOICES:
+        s = next(s for s in rows[L] if s.name == "Speech_TAPE_JOKER_1_04")
+        pos[L] = int(s.target.split(":")[1])
+    claim("docs/formats/wad.md", "`Speech_TAPE_JOKER_1_04` is variation %d in INT and ITA, %d in FRA, %d in DEU, %d in ESN"
+          % (pos["INT"], pos["FRA"], pos["DEU"], pos["ESN"]) if pos["INT"] == pos["ITA"] else "(INT != ITA)")
+    ents = collections.Counter(s.target.split(":")[0] for s in rows["INT"])
+    claim("docs/formats/wad.md", "(every language: %s WEMs in %s entries" % (n(W), n(len(ents))))
+    radio = [s for s in rows["INT"] if s.aliases and any(a.startswith("CHR_RadioScanner#") for a in s.names())
+             or s.name.startswith("CHR_RadioScanner#")]
+    claim("docs/formats/wad.md", "e.g. `CHR_RadioScanner` holds %d" % len({s.target for s in radio}))
+
+    # SFX WAD
+    packs = collections.Counter(s.target.split("/")[0] for s in sfx if "/" in s.target)
+    grouped = [s for s in sfx if "/" not in s.target]
+    claim("docs/formats/wad.md", "**SFX WAD**: %d grouped ambience/effect entries (%d WEMs) plus %d **stream packs**"
+          % (len({s.target.split(":")[0] for s in grouped}), len(grouped), len(packs)))
+    claim("docs/formats/wad.md", "`EDF036D6` (Music, %d WEMs" % packs["EDF036D6"])
+    claim("docs/formats/wad.md", "`E8F980BE` (Music_PDLC, %d), `57572DAC` (Global, %d), `FCA4F778` (%d)"
+          % (packs["E8F980BE"], packs["57572DAC"], packs["FCA4F778"]))
+
+    # categories
+    cats = [collections.Counter(s.category for s in rows[L]) for L in VOICES]
+    assert all(c == cats[0] for c in cats), "category counts differ between languages"
+    order = ["Story missions", "Predator chatter", "Character lines", "Open-world chatter", "Side missions",
+             "Combat barks", "Efforts & emotes", "Interview tapes", "Riddler", "Radio & TV", "Gameplay & hints", "DLC",
+             "Boss fights", "Cinematics", "Reactions", "Interrogations"]
+    g = collections.Counter(s.group for s in sfx)
+    bc = collections.Counter(s.category for s in bank).most_common(8)
+    claim("docs/categories.md", "Counts per voice language (identical in all 5): " + ", ".join(
+        "%s %d" % (c, cats[0][c]) for c in order) + ". SFX archive: " + ", ".join(
+        "%s %d" % (k, g[k]) for k in ["Music", "Ambience", "SFX", "Unknown"]) + ". BANK (all languages): " + ", ".join(
+        "%s %d" % kv for kv in bc) + ", ...")
+
+    # characters (by line)
+    first = {}
+    for s in rows["INT"]:
+        first.setdefault(s.line, s)
+    wc = sum(1 for s in first.values() if s.character)
+    inf = sum(1 for s in first.values() if not s.character and s.characters_inferred)
+    nei = [s for s in first.values() if not s.character and not s.characters_inferred]
+    top = collections.Counter(s.category for s in nei).most_common(4)
+    assert all(sum(1 for s in rows[L] if s.character) == sum(1 for s in rows["INT"] if s.character) for L in VOICES)
+    claim("docs/categories.md", "(every voice language, the same lines): %s with a verified speaker, %s more with inferred "
+          "characters, %s with neither (%s, ...)" % (n(wc), n(inf), n(len(nei)), ", ".join(
+              "%s %d" % (c.lower().replace("radio & tv", "radio & TV"), k) for c, k in top)))
+    claim("docs/categories.md", "A hint for the ~%s lines without a verified speaker" % n(int(round(WL - wc, -2))))
+    claim("REPORT.md", "speaker labels (%s in every language)" % n(wc))
+
+    # subtitles
+    langs = sorted(f[4:7] for f in os.listdir(cat) if f.startswith("SUB_") and f.endswith(".csv"))
+    texts = {L: subtitles.read_texts(os.path.join(cat, "SUB_%s.csv" % L), L) for L in langs}
+    total = sum(len(t) for t in texts.values())
+    linked = sum(1 for t in texts.values() for s in t if s.sound)
+    i_tot, i_l = len(texts["INT"]), sum(1 for s in texts["INT"] if s.sound)
+    for doc in ("REPORT.md", "docs/formats/subtitles.md"):
+        claim(doc, "%s of %s strings in %d languages (INT %s" % (n(linked), n(total), len(langs), n(i_l))
+              if doc == "REPORT.md" else "Linked: %s of %s strings in the %d languages (INT %s / %s)"
+              % (n(linked), n(total), len(langs), n(i_l), n(i_tot)))
+    claim("REPORT.md", "(INT %s of %s)" % (n(i_l), n(i_tot)))
+    unl = {L: {s.speech for s in t if not s.sound} for L, t in texts.items()}
+    unl_n = {len(v) for v in unl.values()}
+    assert len(unl_n) == 1, unl_n
+    un_strings = [sum(1 for s in t if not s.sound) for t in texts.values()]
+    claim("REPORT.md", "%d speech names (%s strings per language, INT %d" % (unl_n.pop(), rng(un_strings),
+                                                                             i_tot - i_l))
+    claim("docs/formats/subtitles.md", "(%d speech names, %s strings per language" % (len(unl["INT"]), rng(un_strings)))
+    two = {s.speech for s in texts["INT"] if ";" in s.sound}
+    claim("docs/formats/subtitles.md", "(%d subtitled events" % len(two))
+    one = []
+    for L, t in texts.items():
+        c = collections.Counter(s.speech for s in t)
+        one.append(100.0 * sum(1 for v in c.values() if v == 1) / len(c))
+    claim("docs/formats/subtitles.md", "%s subtitle strings per language: %d-%d %% of the subtitles have one line (JPN: all)"
+          % (rng([len(t) for t in texts.values()]), int(min(one)), int(round(max(x for x in one if x < 100)))))
+    enc = {L: collections.Counter(s.enc for s in t) for L, t in texts.items()}
+    pol = pct(enc["POL"]["w"], len(texts["POL"]))
+    west = [enc[L]["w"] for L in langs if L not in ("RUS", "JPN", "KOR", "POL")]
+    claim("docs/formats/subtitles.md", "%d %% of POL are UTF-16; the other languages have %s UTF-16 strings each"
+          % (pol, rng(west)))
+    with gzip.open(os.path.join(cat, subtitles.LOCATIONS), "rt", newline="") as f:
+        sublocs = sum(1 for _ in csv.DictReader(f))
+    claim("REPORT.md", "Edited in place in every copy (%s copies)" % n(sublocs))
+    claim("docs/formats/subtitles.md", "%s stored copies" % n(sublocs))
+    claim("docs/user-guide.md", "About %d subtitles per language have no sound" % int(round(sum(un_strings) / len(un_strings), -1)))
+
+    # subtitle coverage of the INT voice lines
+    lk = {k for s in texts["INT"] for k in s.sound.split(";") if k}
+    cov, tot = collections.Counter(), collections.Counter()
+    for s in rows["INT"] + [b for b in bank if b.line and "INT" in b.language.split(";")]:
+        if s.line:
+            tot[s.category] += 1
+            cov[s.category] += s.key in lk
+    p = {c: pct(cov[c], tot[c]) for c in tot}
+    claim("REPORT.md", "few for combat barks (%d %%), cinematics and boss fights (%d %%;"
+          % (p["Combat barks"], p["Cinematics"]) if p["Cinematics"] == p["Boss fights"] else "(cinematics != boss)")
+    hi = [p[c] for c in ("Story missions", "Predator chatter", "Character lines", "Open-world chatter")]
+    claim("REPORT.md", "%d-%d %% for story, predator, character and open-world lines" % (min(hi), max(hi)))
+    claim("REPORT.md", "(INT voice lines with a subtitle): none for efforts/grunts" if p["Efforts & emotes"] == 0 else "!")
+
+    # what the editor loads (Italian on the test PC: the smoke test)
+    ita = len(rows["ITA"]) + len(sfx) + sum(1 for s in bank if s.in_language("ITA"))
+    claim("REPORT.md", "%s sounds loaded, encode + inject, %s subtitles" % (n(ita), n(len(texts["ITA"]))))
+    claim("README.md", "~%s sounds of one language" % n(int(round(ita, -3))))
+    return rows
+
+
+# --- figures recorded by the catalog build (index.json: they need the full scan) ---------------------
+def index_claims(cat):
+    import json
+    ix = json.load(open(os.path.join(cat, "index.json"), encoding="utf-8"))
+    b, ln, sub = ix["BANK"], ix["LINES"], ix["SUB"]
+    claim("REPORT.md", "Names: %s BANK sounds are named after the Wwise event that plays them (`Play_...`, `Speech_...`), "
+          "the other %s after their bank" % (n(b["named_by_event"]), n(b["named_by_bank"])))
+    claim("REPORT.md", "**Unnamed sounds**: %s BANK sounds are reached by no event" % n(b["named_by_bank"]))
+    claim("docs/decisions.md", "sounds (%s of %s, voice lines included); the other %s are numbered per bank"
+          % (n(b["named_by_event"]), n(b["sounds"]), n(b["named_by_bank"])))
+    claim("docs/decisions.md", "the most specific speech event that plays it (%s of the %s WAD voice lines)"
+          % (n(ln["wad_lines_named_by_event"]), n(ln["wad_lines"])))
+    claim("docs/decisions.md", "the other %s keep the name of their English (INT) recording's position"
+          % n(ln["wad_lines_named_after_int_recording"]))
+    claim("REPORT.md", "- %s voice lines are played by no event whose name the game stores"
+          % n(ln["wad_lines_named_after_int_recording"]))
+    claim("docs/user-guide.md", "About %d %% of the lines are named by no game event"
+          % pct(ln["wad_lines_named_after_int_recording"], ln["wad_lines"]))
+    claim("docs/formats/subtitles.md", "%s dialogue objects in the packages for %s speech lines"
+          % (n(sub["dialogue_objects"]), n(sub["dialogue_lines"])))
+    claim("docs/formats/subtitles.md", "The sizes add up exactly in all %s objects" % n(sub["dialogue_objects"]))
+    claim("docs/formats/subtitles.md", "plus %d scripted subtitles" % sub["scripted_objects"])
+    claim("docs/formats/subtitles.md", "%d more subtitles (radio and encryption-card lines) are `RSubtitle` objects"
+          % sub["scripted_objects"])
+    claim("REPORT.md", "(%s speech lines x %d languages) and" % (n(sub["dialogue_lines"]), len(sub["lines"])))
+    claim("REPORT.md", "in %d scripted `RSubtitle` objects" % sub["scripted_objects"])
+    claim("docs/formats/banks.md", "(in %s of the 2,650 packages)" % n(b["files"] - 67))   # 67 loose .bnk with banks
+
+
+# --- the game's own files ----------------------------------------------------------------------------
+def menu_claims(game):
+    from ackit import loctext
+    root = os.path.join(game.root, "BmGame", "Localization")
+    counts = {L: len(loctext.load(root, L)) for L in loctext.languages(root)}
+    files = {L: len(loctext.files(root, L)) for L in loctext.languages(root)}
+    approx = n(int(round(sum(counts.values()) / len(counts), -3)))
+    claim("docs/formats/text.md", "~%d files and ~%s strings each" % (collections.Counter(files.values()).most_common(1)[0][0], approx))
+    claim("REPORT.md", "~%s strings per language" % approx)
+    claim("docs/user-guide.md", "of the selected language (about\n%s)" % approx)
+    claim("REPORT.md", "%s menu texts loaded" % n(counts["ITA"]))
+
+
+def scan_claims(game):
+    """Bank survey from a fresh scan of every package, and the relation facts of `ackit verify --catalog`."""
+    from ackit.bnk import find_banks, Bank, BankError, IN_MEMORY, STREAMED, PREFETCH
+    blobs = collections.Counter()
+    media_sets = collections.defaultdict(set)
+    kinds = collections.Counter()
+    in_pkg = set()
+
+    def visit(rel, pkg, start, data):
+        for bk in find_banks(data):
+            blobs[bk.id] += 1
+            media_sets[bk.id].add(frozenset(m.id for m in bk.media()))
+            in_pkg.add(bk.id)
+            own = {m.id for m in bk.media()}
+            for s in bk.sources():
+                if s.stream == IN_MEMORY:
+                    kinds["memory own" if s.media in own else "memory other"] += 1
+                elif s.stream == STREAMED:
+                    kinds["streamed"] += 1
+                elif s.stream == PREFETCH:
+                    kinds["prefetch"] += 1
+
+    sc = banks.scan(game.cooked, visit=visit)
+    for path in glob.glob(os.path.join(game.cooked, "SFX", "*.bnk")):     # loose banks: source kinds too
+        try:
+            bk = Bank(open(path, "rb").read(), 0)
+        except BankError:
+            continue
+        own = {m.id for m in bk.media()}
+        for s in bk.sources():
+            if s.stream == IN_MEMORY:
+                kinds["memory own" if s.media in own else "memory other"] += 1
+            elif s.stream == STREAMED:
+                kinds["streamed"] += 1
+            elif s.stream == PREFETCH:
+                kinds["prefetch"] += 1
+    names = {}
+    cooked_list = [l.strip() for l in open(os.path.join(game.cooked, "CookedBanks.txt"), encoding="latin-1") if l.strip()]
+    from ackit.bnk import fnv1
+    for nm in cooked_list:
+        names[fnv1(nm)] = nm
+    loose_ids = {int(os.path.splitext(os.path.basename(p))[0], 16) for p in glob.glob(os.path.join(game.cooked, "SFX", "*.bnk"))
+                 if re.fullmatch(r"[0-9A-Fa-f]{8}", os.path.splitext(os.path.basename(p))[0])}
+    embedded = [nm for h, nm in names.items() if h in in_pkg]
+    loose = sorted(nm for h, nm in names.items() if h in loose_ids and h not in in_pkg)
+    nowhere = sorted(nm for h, nm in names.items() if h not in in_pkg and h not in loose_ids)
+    claim("docs/formats/banks.md", "lists all %s cooked bank names" % n(len(cooked_list)))
+    both = sorted(nm for h, nm in names.items() if h in loose_ids and h in in_pkg)
+    claim("docs/formats/banks.md", "%s of them are embedded in packages (%s also as loose `.bnk` files), %d are only loose "
+          "`.bnk` files (%s); %s were found nowhere"
+          % (n(len(embedded)), ", ".join("`%s`" % x for x in both), len(loose), ", ".join("`%s`" % x for x in loose),
+             ", ".join("`%s`" % x for x in nowhere[:-1]) + " and `%s`" % nowhere[-1]))
+    diff_sets = sum(1 for b, v in media_sets.items() if len(v) > 1)
+    claim("docs/formats/banks.md", "Bank blobs in packages: %s for %s bank IDs" % (n(sum(blobs.values())), n(len(blobs))))
+    by_name = {names.get(b, ""): c for b, c in blobs.items()}
+    claim("docs/formats/banks.md", "`CHR_CatwomanFoleySuit` %d times, `CHR_BatmanFoleyBelt` %d). %d bank IDs"
+          % (by_name.get("CHR_CatwomanFoleySuit", 0), by_name.get("CHR_BatmanFoleyBelt", 0), diff_sets))
+    claim("docs/formats/banks.md", "| banks embedded in `.upk` packages (in %s of the %s packages) | %s |"
+          % (n(sum(1 for f in sc.files if f.lower().endswith(".upk"))), n(len(glob.glob(os.path.join(game.cooked, "*.upk")))),
+             n(len(blobs))))
+    claim("docs/formats/banks.md", "Source kinds over all bank copies: %s in-memory sources of the bank's own media; %s "
+          "in-memory sources with bank id 0" % (n(kinds["memory own"]), n(kinds["memory other"])))
+    claim("docs/formats/banks.md", "(loaded by another bank); %s streamed sources; %s **prefetch** sources"
+          % (n(kinds["streamed"]), n(kinds["prefetch"])))
+    refs = {h for h, t in sc.packs.items() if t}
+    claim("docs/formats/banks.md", "the BKPK tables in the packages reference %s WAD entries" % n(len(refs)))
+    per_bank = collections.defaultdict(set)
+    for m, e in sc.media.items():
+        for b in e["banks"]:
+            per_bank[b].add(m)
+    big = sorted(per_bank.items(), key=lambda kv: -len(kv[1]))[:7]
+    nm = [(sc.bank_names.get(b, "%08X" % b), len(v)) for b, v in big]
+    claim("docs/formats/banks.md", "Largest: `%s` %d WEMs, " % nm[0] + ", ".join("`%s` %d" % x for x in nm[1:]))
+    pkm = {l.media for l in sc.locations if not l.file.startswith("SFX/")}
+    pre = collections.Counter()
+    for m in pkm:
+        for p_ in {sc.bank_names.get(b, "%08X" % b).split("_")[0] for b in sc.media[m]["banks"]}:
+            pre[p_] += 1
+    top = [k for k, v in pre.most_common() if v >= 30]
+    text = ", ".join("`%s` %s" % (k, n(pre[k])) if k != "CHR" else "`CHR`\n(character foley, combat, bosses) %s" % n(pre[k])
+                     for k in top)
+    claim("docs/formats/banks.md", "(a media held by banks of two prefixes counts for both): " + text + ", others < 30")
+    # media IDs used by several languages' WAD tables (each at its own position)
+    langs_of = collections.defaultdict(set)
+    for h, tables in sc.packs.items():
+        for L, t in tables.items():
+            if L:
+                for m in t:
+                    langs_of[m].add(L)
+    claim("docs/formats/banks.md", "%d media IDs are used by several\n  languages" % sum(1 for v in langs_of.values() if len(v) > 1))
+
+    # relations, re-derived from the game by the catalog check
+    from ackit.verify import catalogcheck
+    rep = catalogcheck.run(game, log=lambda *a: None)
+    facts = "\n".join(rep.facts)
+    print("verify --catalog:", "catalog OK" if rep.ok else "%d issue(s)" % len(rep.issues))
+
+    def fact(pattern):
+        m = re.search(pattern, facts)
+        if not m:
+            raise SystemExit("fact not found: " + pattern)
+        return [int(x) for x in m.groups()]
+    src = fact(r"own language's bank: (\d+) of (\d+)")
+    claim("REPORT.md", "own language (%s / %s)" % (n(src[0]), n(src[1])))
+    claim("docs/formats/banks.md", "of its own language (%s of %s)" % (n(src[0]), n(src[1])))
+    hd = fact(r"whole bank copies: (\d+) of (\d+)")
+    claim("REPORT.md", "is the start of its recording (%s / %s)" % (n(hd[0]), n(hd[1])))
+    claim("docs/formats/banks.md", "line gives it (%s of %s)" % (n(hd[0]), n(hd[1])) if True else "")
+    rec, lns, full = fact(r"(\d+) recordings \(WAD .*?\), (\d+) lines, (\d+) of them")
+    claim("docs/formats/banks.md", "(%s recordings)" % n(rec))
+    ln15 = fact(r"(\d+) of (\d+) lines within a factor 1.5")
+    claim("docs/formats/banks.md", "(%s of %s\n  within a factor 1.5" % (n(ln15[0]), n(ln15[1])))
+    sp = fact(r"speakers: (\d+) labelled voice lines agree .*?, (\d+) listed as Unknown .*?, (\d+) differ")
+    claim("REPORT.md", "%s agree, %d differ" % (n(sp[0]), sp[2]))
+    claim("docs/categories.md", "%s labelled\n  lines agree" % n(sp[0]))
+    claim("docs/categories.md", "%d listed\n  as Unknown are named there, %d differ" % (sp[1], sp[2]))
+    if not rep.ok:
+        claim("(game)", "catalog OK", "ackit verify --catalog")
+
+
+def _rebuild_language(lang):
+    """Keys of the WEMs of one speech WAD that the builder does not reproduce byte for byte."""
+    from ackit.inject import read_target
+    from ackit.verify.selfcheck import rebuild_identical
+    from ackit.wad import Wad
+    game = Game(default_game_dir(settings.load()))
+    path = game.wad(lang)
+    w = Wad(path)
+    from ackit.wem import Wem
+    bad, count, extra = [], 0, collections.Counter()
+    for s in read_csv(os.path.join(catalog_dir(), lang + ".csv")):
+        count += 1
+        r = read_target(path, s.target, w)
+        tags = {c.tag for c in Wem.parse(r).chunks} - {b"fmt ", b"data"}
+        if tags:
+            extra["/".join(sorted(t.decode() for t in tags))] += 1
+        try:
+            ok = rebuild_identical(r)
+        except Exception:  # noqa: BLE001 - a WEM the builder cannot parse is a difference too
+            ok = False
+        if not ok:
+            bad.append(s.key)
+    return count, bad, extra
+
+
+def wem_claims():
+    """Every WEM of the 5 speech WADs and of the SFX WAD rebuilt from the format rules (docs/formats/wem.md;
+    ~10 minutes)."""
+    import multiprocessing
+    from ackit.inject import read_target
+    from ackit.wem import Wem
+    with multiprocessing.Pool(len(VOICES) + 1) as p:
+        res = p.map(_rebuild_language, VOICES + ["SFX"])
+    extra = collections.Counter()
+    for _, _, e in res:
+        extra.update(e)
+    sfx_count, sfx_bad, _ = res.pop()
+    game = Game(default_game_dir(settings.load()))
+    sfx = {s.key: s for s in read_csv(os.path.join(catalog_dir(), "SFX.csv"))}
+    kinds = collections.Counter()
+    per = []                                # samples per seek-table entry of the music tracks
+    from ackit.verify.selfcheck import rebuild_identical
+    for k in sfx_bad:
+        s = sfx[k]
+        r = read_target(game.wad("SFX"), s.target)
+        m = Wem.parse(r)
+        try:
+            rebuild_identical(r)
+            err = False
+        except ValueError:                  # "inconsistent total_frames": more samples than the packets hold
+            err = True
+        if any(c.tag == b"smpl" for c in m.chunks):
+            kinds["smpl"] += 1
+        elif err:
+            kinds["frames"] += 1
+        elif s.group == "Music" and m.vorb["seek_table_size"]:
+            kinds["seek"] += 1
+            per.append(m.vorb["total_frames"] / (m.vorb["seek_table_size"] // 4))
+        else:
+            kinds["other"] += 1
+    claim("REPORT.md", "%d SFX originals (music seek tables and sample counts, looping ambience)" % len(sfx_bad))
+    claim("docs/formats/wem.md", "In SFX %s of %s are identical; the others are %d music tracks whose seek table follows "
+          "another placement (one entry per %s samples), %d music tracks whose sample count exceeds what their "
+          "packets hold, and %d 4-channel ambience WEMs with a `smpl` (loop) chunk%s"
+          % (n(sfx_count - len(sfx_bad)), n(sfx_count), kinds["seek"], rng([int(x) for x in per]), kinds["frames"],
+             kinds["smpl"], "" if not kinds["other"] else " (+%d other)" % kinds["other"]))
+    total = sum(c for c, _, _ in res)
+    bad = [k for _, b, _ in res for k in b]
+    claim("docs/formats/wem.md", "`RIFF` size `WAVE`, then `fmt ` (0x42 bytes), optional %s, `data`" % ", ".join(
+        "%s (%s %s)" % (" + ".join("`%s`" % t for t in k.split("/")), n(v), "WEM" if v == 1 else "WEMs")
+        for k, v in sorted(extra.items())))
+    exceptions = ", ".join("%s `%s`" % tuple(k.split(":", 1)) for k in bad)       # e.g. INT `FE9963E8:74`
+    claim("docs/formats/wem.md", "Checked on every WEM of the 5 speech WADs: %s of %s are identical; the exception (%s)"
+          % (n(total - len(bad)), n(total), exceptions))
+
+
+def test_claims():
+    import subprocess
+    out = subprocess.run([sys.executable, "-m", "pytest", "--collect-only", "-q", "-p", "no:cacheprovider"],
+                         cwd=ROOT, capture_output=True, text=True).stdout
+    m = re.search(r"(\d+) tests? collected", out)
+    claim("REPORT.md", "**Tests**: %s automated tests" % (m.group(1) if m else "?"), "pytest --collect-only")
+
+
+def norm(t):
+    return re.sub(r"\s+", " ", t).strip()
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--game")
+    ap.add_argument("--scan", action="store_true", help="also scan every package and run the catalog check")
+    ap.add_argument("--wems", action="store_true", help="also rebuild every speech and SFX WEM from the rules")
+    a = ap.parse_args()
+    cat = catalog_dir()
+    catalog_claims(cat)
+    index_claims(cat)
+    test_claims()
+    gd = a.game or default_game_dir(settings.load())
+    game = Game(gd) if gd else None
+    if game:
+        menu_claims(game)
+    if a.scan:
+        if not game:
+            raise SystemExit("--scan needs the game")
+        scan_claims(game)
+    if a.wems:
+        if not game:
+            raise SystemExit("--wems needs the game")
+        wem_claims()
+    docs = {}
+    bad = 0
+    for doc, text, why in CLAIMS:
+        if doc not in docs:
+            path = os.path.join(ROOT, doc)
+            docs[doc] = norm(open(path, encoding="utf-8").read()) if os.path.exists(path) else ""
+        if norm(text) not in docs[doc]:
+            bad += 1
+            print("MISMATCH %s: expected \"%s\"%s" % (doc, norm(text), (" (" + why + ")") if why else ""))
+    print("%d claims checked in %d documents%s: %s" % (len(CLAIMS), len(docs), "" if a.scan else " (without --scan)",
+                                                    "all match" if not bad else "%d mismatch(es)" % bad))
+    return 1 if bad else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tools/collect_licenses.py

@@ -80,8 +80,6 @@ def components():
          "https://github.com/pyinstaller/pyinstaller", dist_file("pyinstaller", "COPYING.txt")),
         ("Wwise codebook data (ackit/data/game_codebooks.bin)", "-", "derived from libvorbis codebooks (BSD-3-Clause)",
          "https://github.com/xiph/vorbis", "(see libvorbis above)"),
-        ("Community hash list (ackit/data/hashlist.txt)", "-", "no license stated (list of Wwise ID -> name pairs)",
-         "from the 'Batman_bnk' tool shipped with AbdAshes' Batman Arkham Audio Guide", ""),
     ]
 
 

tools/e2e_editor.py

@@ -29,9 +29,6 @@ def sha(p):
 
 def main():
     from PySide6.QtWidgets import QApplication
-    from ackit import audio
-    from ackit.inject import pack_bnk_path, parse_target
-    from ackit.verify import ingame
     from app.mainwindow import MainWindow
     from app.workers import Runner
 
@@ -45,6 +42,21 @@ def main():
     ws = w.ws
     assert ws, "game not loaded"
     ok = True
+    try:
+        ok = run(w, ws, errors)
+    finally:                                   # never leave a change behind, whatever failed
+        ws.revert_all()
+        ws.texts.revert_all()
+    print("E2E EDITOR %s" % ("PASS" if ok else "FAIL"))
+    w.close()
+    return 0 if ok else 1
+
+
+def run(w, ws, errors):
+    from ackit import audio
+    from ackit.inject import pack_bnk_path, parse_target
+    from ackit.verify import ingame
+    ok = True
     with tempfile.TemporaryDirectory() as td:
         # --- music (stream pack, resized) ---
         s = ws.sound("Music_393BC53A")
@@ -71,10 +83,11 @@ def main():
         ok &= same
 
         # --- speech line (in place) ---
-        t = ws.sound("Speech_TAPE_JOKER_1_4")
+        t = ws.sound("Speech_TAPE_JOKER_1#4")         # 4th WEM of that entry in this language's WAD
         twad = ws.wad_path(t)
         tb = sha(twad)
-        w.select_name("Speech_TAPE_JOKER_1_4") or (w.clear_filters(), w.select_name("Speech_TAPE_JOKER_1_4"))
+        w.clear_filters()
+        assert w.select_name(t.name)
         y, ysr = audio.decode_wem(ws.original_bytes(t))
         line = os.path.join(td, "line.wav")
         audio.save_wav(line, y[::-1], ysr)  # reversed speech, same length
@@ -86,9 +99,7 @@ def main():
         same2 = sha(twad) == tb
         print("speech line applied (WAD changed: %s), reverted byte-identical: %s" % (changed, same2))
         ok &= changed and same2
-    print("E2E EDITOR %s" % ("PASS" if ok else "FAIL"))
-    w.close()
-    return 0 if ok else 1
+    return ok
 
 
 if __name__ == "__main__":

tools/e2e_features.py

@@ -0,0 +1,257 @@
+"""Real-game write round trips for the mission-3 features (no game launch; reverts everything itself).
+
+    python tools/e2e_features.py [--game DIR] [--lang ITA,INT,... | --lang all]
+
+Runs the checks for each language (default: the game's language; `all` = every subtitle/menu language). A
+language without its own voices (ESM, POL, POR, RUS, JPN, KOR) uses the INT voices, like the game and the editor.
+For every check: the SHA-1 of each touched file is taken, the change is applied through Workspace (the code path
+of the editor and the CLI), read back from the game files, reverted, and the files must be byte-identical again.
+  1. in-memory sound with the most copies (every copy written; once, it is shared by all languages)
+  2. localized in-memory speech line of the language's voices
+  3. speech line of the language's voices with prefetch heads (WAD + every head copy), and one that a bank
+     also stores whole (WAD + every whole copy)
+  4. subtitle with several copies (RDialogueEvent), and a scripted subtitle (RSubtitle, Startup.upk)
+  5. menu text
+  6. sound + subtitle in the same package, reverted one at a time
+  7. mod package made with the language's voices and imported into another voice language (INT; FRA for INT):
+     matched by the game's line identity, so the same line (not the same WAD position) is replaced; a text key
+     names its language, so the edited subtitle stays that language's subtitle (the other language's is untouched)
+Uses temporary originals stores (the user's store and settings are not touched), deleted at the end when every
+change was reverted (kept, with a warning, otherwise). Whatever happens (a failed check, an exception), every
+change is reverted before the tool exits; a sound that is not in its original state is never written.
+"""
+import argparse
+import hashlib
+import os
+import shutil
+import sys
+import tempfile
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, ROOT)
+
+import numpy as np  # noqa: E402
+
+from ackit import banks, settings, subtitles  # noqa: E402
+from ackit.game import Game, default_game_dir  # noqa: E402
+from ackit.modpack import export_mod, import_mod, match_mod  # noqa: E402
+from ackit.wem import fill  # noqa: E402
+from ackit.workspace import Workspace, MODIFIED, ORIGINAL  # noqa: E402
+
+RESULTS = []
+SPACES = []          # every Workspace created: reverted in main()'s finally block
+
+
+def sha(p):
+    h = hashlib.sha1()
+    with open(p, "rb") as f:
+        for c in iter(lambda: f.read(1 << 22), b""):
+            h.update(c)
+    return h.hexdigest()
+
+
+def check(name, ok, detail=""):
+    RESULTS.append((name, bool(ok)))
+    print("%-62s %s %s" % (name, "PASS" if ok else "FAIL", detail))
+
+
+def tone(seconds, freq):
+    t = np.arange(int(seconds * 48000)) / 48000.0
+    return (0.3 * np.sin(2 * np.pi * freq * t)).astype(np.float32)[:, None], 48000
+
+
+def snapshot(paths):
+    return {p: sha(p) for p in sorted(set(paths))}
+
+
+def copies_of(ws, s):
+    out = []
+    for l in ws.locations(s):
+        b = banks.open_file(ws.bank_path(l.file)).read(l.offset, l.size)
+        out.append((l, b))
+    return out
+
+
+def sound_round_trip(ws, s, label):
+    if ws.state(s) != ORIGINAL:
+        check(label, False, "(%s is not original: %s; nothing written)" % (s.name, ws.state(s)))
+        return
+    wem, _ = ws.prepare(s, *tone(min(max(s.seconds, 0.3), 1.0), 700))
+    files = ws.files(s)
+    before = snapshot(files)
+    ws.apply([(s, wem, {"source": "e2e"})])
+    ok = ws.state(s) == MODIFIED and ws.current_bytes(s) == wem
+    orig_size = ws.store.get(s.key)["orig_size"]
+    for l, b in copies_of(ws, s):
+        if l.kind == banks.MEMORY or l.size >= orig_size:     # a whole WEM in memory: fills its slot exactly
+            ok &= b == fill(wem, l.size)
+        else:                                  # prefetch head: the start of the WAD recording (zero-padded)
+            full = wem + bytes(max(0, orig_size - len(wem)))
+            ok &= b == (full + bytes(l.size))[:l.size]
+    ws.revert([s])
+    ok &= ws.state(s) == ORIGINAL and snapshot(files) == before and not ws.store.entries
+    check(label, ok, "(%s, %d files, %d copies/heads)" % (s.name, len(files), len(ws.locations(s))))
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--game")
+    ap.add_argument("--lang", help="comma-separated languages, or 'all' (default: the game's language)")
+    a = ap.parse_args()
+    game = Game(a.game or default_game_dir(settings.load()))
+    td = tempfile.mkdtemp()
+    try:
+        first = True
+        for lang in languages(game, a.lang, td):
+            voices = lang if lang in game.languages else "INT"
+            ws = Workspace(game, os.path.join(td, "store_" + lang), [voices, "SFX", "BANK"])
+            SPACES.append(ws)
+            print("\n== %s (voices %s): %d sounds" % (lang, voices, len(ws.catalog.sounds)))
+            run_checks(game, lang, voices, td, ws, first)
+            first = False
+    finally:
+        clean = True
+        for w in SPACES:                      # never leave a change behind
+            w.revert_all()
+            w.texts.revert_all()
+            if w.store.entries or w.texts.records():
+                clean = False
+                print("WARNING: could not revert everything; store: %s" % w.store.root)
+        if clean:                             # the stores hold full copies of game files (GBs): drop them
+            shutil.rmtree(td, ignore_errors=True)
+    print("\n%d/%d PASS" % (sum(ok for _, ok in RESULTS), len(RESULTS)))
+    return 0 if RESULTS and all(ok for _, ok in RESULTS) else 1
+
+
+def languages(game, arg, td):
+    if not arg:
+        return [game.language() or "INT"]
+    if arg.lower() != "all":
+        return [x.strip().upper() for x in arg.split(",") if x.strip()]
+    tx = Workspace(game, os.path.join(td, "store_list"), [game.languages[0]]).texts
+    return sorted(set(game.languages) | set(tx.subtitle_languages()) | set(tx.menu_languages()))
+
+
+def run_checks(game, lang, voices, td, ws, first):
+    bank = [s for s in ws.catalog.sounds if s.archive == "BANK" and s.size >= 2500]
+    if first:
+        sound_round_trip(ws, max((s for s in bank if not s.language), key=lambda s: len(ws.locations(s))),
+                         "1 in-memory sound, every copy")
+    loc = [s for s in bank if s.language == voices]
+    sound_round_trip(ws, max(loc, key=lambda s: len(ws.locations(s))),
+                     "2 %s localized in-memory speech line (%s)" % (lang, voices))
+    heads = [ws.catalog.by_key[k] for k in ws.catalog.heads if k in ws.catalog.by_key
+             and ws.catalog.by_key[k].archive == voices]
+    sound_round_trip(ws, max(heads, key=lambda s: len(ws.locations(s))),
+                     "3a %s speech line with prefetch heads" % lang)
+    whole = [s for s in heads if any(l.size >= s.size for l in ws.locations(s))]
+    if whole:
+        sound_round_trip(ws, max(whole, key=lambda s: len(ws.locations(s))),
+                         "3b %s speech line also stored whole in a bank" % lang)
+    else:
+        check("3b %s speech line also stored whole in a bank" % lang, False, "none found")
+
+    tx = ws.texts
+    subs = tx.subtitles(lang)
+
+    def text_round_trip(key, new, label):
+        files = tx.files(key)
+        before = snapshot(files)
+        tx.apply({key: new})
+        ok = tx.current(key) == new and tx.state(key) == MODIFIED
+        for l in tx.subtitle_locations(key) if key.startswith("SUB:") else []:
+            b = banks.open_file(ws.bank_path(l.file)).read(l.pos, l.size)
+            ok &= subtitles.read_fstring(b, 0)[0] == new
+        tx.revert([key])
+        ok &= snapshot(files) == before and not tx.records()
+        check(label, ok, "(%s, %d files)" % (key[:60], len(files)))
+
+    multi = max((s for s in subs if s.room >= 8), key=lambda s: len(tx.subtitle_locations(subtitles.key(lang, s.speech, s.line))))
+    text_round_trip(subtitles.key(lang, multi.speech, multi.line), "E2E test.", "4a %s subtitle with several copies" % lang)
+    rsub = next((s for s in subs if s.room >= 10 and
+                 {l.file for l in tx.subtitle_locations(subtitles.key(lang, s.speech, s.line))} == {"Startup.upk"}), None)
+    if rsub:
+        text_round_trip(subtitles.key(lang, rsub.speech, rsub.line), "E2E radio.", "4b %s scripted subtitle (RSubtitle)" % lang)
+    else:
+        check("4b %s scripted subtitle (RSubtitle)" % lang, False, "no scripted subtitle found in Startup.upk")
+    e = next(x for x in tx.menu(lang) if x.file.upper().startswith("FRONTEND"))
+    text_round_trip("MENU:" + e.key, "E2E " + e.value, "5 %s menu text" % lang)
+
+    # 6. sound + subtitle in one package
+    sub_files = {}
+    for s in subs:
+        if s.room < len("Shared package."):
+            continue
+        for l in tx.subtitle_locations(subtitles.key(lang, s.speech, s.line)):
+            sub_files.setdefault(l.file, subtitles.key(lang, s.speech, s.line))
+    pair = next(((s, sub_files[l.file]) for s in bank if not s.language for l in ws.locations(s)
+                 if l.file in sub_files and len(ws.locations(s)) <= 3), None)
+    if pair:
+        s, key = pair
+        files = set(ws.files(s)) | set(tx.files(key))
+        before = snapshot(files)
+        wem, _ = ws.prepare(s, *tone(0.3, 900))
+        ws.apply([(s, wem, {})])
+        tx.apply({key: "Shared package."})
+        ok = ws.state(s) == MODIFIED and tx.current(key) == "Shared package."
+        ws.revert([s])
+        ok &= ws.state(s) == ORIGINAL and tx.current(key) == "Shared package."
+        tx.revert([key])
+        ok &= snapshot(files) == before
+        check("6 %s sound + subtitle in one package, reverted separately" % lang, ok, "(%s + %s)" % (s.name, key[:40]))
+    else:
+        check("6 %s sound + subtitle in one package" % lang, False, "no such package found")
+
+    # 7. cross-language mod: the same game line, not the same WAD position
+    from ackit.catalog import read_csv
+    from ackit.paths import catalog_dir
+    other = "FRA" if voices == "INT" else "INT"
+    other_target = {s.name: s.target for s in read_csv(os.path.join(catalog_dir(), other + ".csv"))}
+    other_subs = {(s.speech, s.line) for s in tx.subtitles(other)}
+    by_sound = {}
+    for s in subs:
+        for k in s.sounds:
+            by_sound.setdefault(k, s)
+    line = next((s for s in ws.catalog.sounds if s.archive == voices and ":" in s.target
+                 and s.key in by_sound and by_sound[s.key].room >= 9
+                 and (by_sound[s.key].speech, by_sound[s.key].line) in other_subs   # the line exists there too
+                 and other_target.get(s.name, s.target) != s.target), None)      # at another position there
+    if line is None:
+        check("7 mod %s -> %s" % (lang, other), False, "no line found at another position in %s" % other)
+        return
+    sub = by_sound[line.key]
+    sub_key = subtitles.key(lang, sub.speech, sub.line)
+    if ws.state(line) != ORIGINAL or tx.state(sub_key) != ORIGINAL:
+        check("7 mod %s -> %s" % (lang, other), False, "(%s is not original; nothing written)" % line.name)
+        return
+    wem, _ = ws.prepare(line, *tone(1.0, 500))
+    ws.apply([(line, wem, {})])
+    tx.apply({sub_key: "Mod text."})
+    pkg = os.path.join(td, "e2e_%s.acmod" % lang)
+    export_mod(ws, pkg, ws.modified_keys(), "e2e", texts=tx.modified())
+    ws.revert_all()
+    tx.revert_all()
+    wo = Workspace(game, os.path.join(td, "store_%s_%s" % (lang, other)), [other, "SFX", "BANK"])
+    SPACES.append(wo)
+    _, items = match_mod(wo, pkg)
+    target = items[0].sound
+    same_line = target is not None and target.name == line.name
+    moved = target is not None and target.target != line.target         # positions differ between languages
+    other_sub = subtitles.key(other, sub.speech, sub.line)
+    files = set(wo.files(target)) if target else set()
+    files |= set(wo.texts.files(other_sub)) | set(wo.texts.files(sub_key))
+    before = snapshot(files)
+    other_text = wo.texts.current(other_sub)
+    res, texts = import_mod(wo, pkg)
+    ok = same_line and wo.state(target) == MODIFIED
+    ok &= [k for k, _, p in texts if not p] == [sub_key] and wo.texts.current(sub_key) == "Mod text."
+    ok &= wo.texts.current(other_sub) == other_text
+    wo.revert_all()
+    wo.texts.revert_all()
+    ok &= snapshot(files) == before
+    check("7 mod %s -> %s: same game line%s" % (lang, other, " (another WAD position)" if moved else ""), ok,
+          "(%s: %s:%s -> %s:%s)" % (line.name, voices, line.target, other, target.target if target else "-"))
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tools/screenshots.py

@@ -1,6 +1,9 @@
 """Render the user-guide screenshots headless (docs/img/*.png). Read-only: never applies anything.
 
-    python tools/screenshots.py [--game DIR]
+    python tools/screenshots.py [--game DIR] [--lang INT]
+
+The app starts in the game's language as always; the screenshots then switch to --lang (default INT, the
+game's English) the way a user does, through the Language list.
 """
 import argparse
 import os
@@ -18,6 +21,7 @@ os.environ["ACKIT_NO_AUDIO"] = "1"
 def main():
     ap = argparse.ArgumentParser()
     ap.add_argument("--game")
+    ap.add_argument("--lang", default="INT", help="language shown in the screenshots (default: INT)")
     a = ap.parse_args()
     from PySide6.QtWidgets import QApplication
     from ackit import audio
@@ -38,17 +42,22 @@ def main():
         w = MainWindow(settings=st, runner=Runner(sync=True), game_dir=a.game)
         w.resize(1500, 950)
         w.show()
-        app.processEvents()               # the game loads here
+        app.processEvents()               # the game loads here (in the game's language)
+        if a.lang and w.ws and w.language != a.lang.upper():
+            w.cmb_lang.setCurrentText(a.lang.upper())          # what choosing it in the Language list does
+            w.load_game(w.ws.game.root, a.lang.upper())
+            app.processEvents()
         # status bar: show the store where users have it, not this temporary folder (no local user names)
         w.lbl_game.setText(w.lbl_game.text().replace(os.path.join(td, "data"), r"%APPDATA%\UnheardCity"))
         # 1: browser filtered by character
         w.cmb_char.setCurrentText("Joker")
         w._filters_changed()
-        w.search.setText("cinematic")
+        w.search.setText("")
         w._filter_text()
         w.fit_editor()
-        target = next(w.proxy.sound_at(w.proxy.index(r, 1)) for r in range(w.proxy.rowCount())
-                      if w.proxy.sound_at(w.proxy.index(r, 1)).seconds > 4)
+        rows = [w.proxy.sound_at(w.proxy.index(r, 1)) for r in range(w.proxy.rowCount())]
+        with_sub = [s for s in rows if w.ws.texts.for_sound(s, w.language)]       # show the subtitle column too
+        target = max(with_sub or rows, key=lambda s: min(s.seconds, 6.0))
         w.select_name(target.name)
         app.processEvents()
         # a replacement loaded for display only (not applied)
@@ -63,7 +72,8 @@ def main():
         w.editor.sp_a.setValue(0.2)
         app.processEvents()
         w.grab().save(os.path.join(out, "main.png"))
-        w.editor.grab().save(os.path.join(out, "editor.png"))
+        from PySide6.QtCore import QPoint, QRect          # the editor area, cut from the window (full styling)
+        w.grab(QRect(w.scroll.mapTo(w, QPoint(0, 0)), w.scroll.size())).save(os.path.join(out, "editor.png"))
         # 2: category browsing
         w.clear_filters()
         w.cmb_group.setCurrentText("Speech")