Version 3.1.0: voice lines linked across all languages from the game data

REPORT.md

@@ -6,22 +6,29 @@ editor Unheard City (mission 2).
 ## What works (verified)
 - **Every sound of the game is cataloged and editable**: the speech WADs (5 languages x 10,816), the SFX WAD (651)
   and, new, the **15,404 in-memory sounds** stored inside soundbanks (archive `BANK`: footsteps, foley, combat,
-  weapons, UI, efforts; 8,098 shared by all languages + ~1,460 localized speech lines per voice language) in loose `.bnk` files and in 1,499 `.upk` packages. ackit reads and patches the
+  weapons, UI, efforts; 8,098 shared by all languages + ~1,460 localized speech lines per voice language) in 65
+  loose `.bnk` files and 975 `.upk` packages. ackit reads and patches the
   LZO-compressed UE3 packages itself (`ackit/upk.py`, `lzo.py`; no external tools). A replacement goes into every
   copy (93,910 copies); 396 speech lines with prefetch heads in banks get their heads rewritten too.
   Names: 6,456 BANK sounds are named after their Wwise event (`Play_...`), the rest after their bank.
 - **Catalog check**: `ackit verify --catalog` compares the catalog with a fresh scan of the game (missing, extra,
-  duplicate, size/checksum, category, character, language, bank copies, subtitle copies/texts): **catalog OK**
-  on the full game. It found and fixed 2 duplicate names per language (variation shown as `NAME#i`).
+  duplicate, size/checksum, category, character, language, bank copies, subtitle copies/texts) and re-derives
+  every relation from the game files (lines, names, subtitle links, prefetch heads, copies): **catalog OK** on
+  the full game.
 - **Subtitles**: found in `RDialogueEvent` objects inside the packages (9,016 speech lines x 11 languages) and
   in 41 scripted `RSubtitle` objects (radio, encryption cards, Riddler; linked through `RSubtitleLookup`),
-  parsed, cataloged (`SUB_<LANG>.csv`) and linked to their speech sound through the soundbanks (~99 % of the
-  strings). Edited in place in every copy.
-- **Names true to the game**: the position of a variation inside a WAD entry differs between languages (75 % of
-  the lines), so speech sounds are named after the game's line (its speech event) - the same in every language
-  and the same as the subtitle. The curated ITA speaker lists are mapped through these names; checked against
-  the speaker field of the game's own dialogue objects: 955 / 955 agree. In-memory speech is per language too
-  (one bank copy per language): the sound list shows only the selected language.
+  parsed, cataloged (`SUB_<LANG>.csv`) and linked to their voice recordings through the soundbanks: 103,304 of
+  103,658 strings in 11 languages (ITA 9,346 of 9,377), each on the same line in every language; an event that
+  plays two versions of a line links both. Edited in place in every copy (168,688 copies).
+- **One recording per language, from the game data alone** (`docs/formats/banks.md`, *Lines*): every voice
+  recording is tied to its line by the game's Wwise Sound objects (same IDs in every language), and each
+  WAD recording to its media by its own language's BKPK table (10,816 / 10,816 in each language). Result:
+  12,434 lines (WAD + in-memory) with exactly one recording in each of the 5 voice languages, no gap and no
+  conflict; recordings stored twice (5 per language) are written together. Independent checks: every WAD
+  recording is a source in a bank of its own language (54,055 / 54,055), every prefetch head is the start of
+  its recording (2,101 / 2,101). Each line has one name in every language and carries its identity (`line`):
+  speaker labels (3,366 in every language), subtitles and mod packages (another language: the same line)
+  follow it. Speaker labels vs the game's dialogue speaker field: 2,766 agree, 19 differ (categories.md).
 - **Menu text**: the localization files (`BmGame\Localization\<LANG>\*.int`, ~7,000 strings per language),
   edited value by value; everything else in the files stays byte-identical.
 - **Editor**: one *Language* setting (default: the game's language) for voices, subtitles and menu text
@@ -33,15 +40,21 @@ editor Unheard City (mission 2).
 - **Safety**: every changed file (WAD, bank, package, localization file) is rebuilt from a verified original copy
   in the store + the edits applied now; reverting the last edit restores it byte-for-byte; files changed by
   another tool or Steam are detected (whole-file SHA-1) and not overwritten without confirmation.
-- **Tests**: 121 automated tests (unit, synthetic game with packages/banks/dialogues/localization, real-game
-  read-only, offscreen GUI) pass. `ackit verify` PASS (incl. 300/300 in-memory WEMs rebuilt byte-identically).
-- **Real game** (2026-10-09), all reverting and checking byte-identity themselves:
-  `ackit verify --ingame` PASS (music heard, ncc 1.0); `--banks` PASS (77 in-memory sounds of the front-end
-  packages replaced in 54 files: the game loads them, no crash); `--texts` PASS (12 menu texts found in the
-  game's memory, 12 subtitles patched in packages, no crash); `tools/e2e_editor.py` PASS. Game files original
-  at the end (`ackit status`).
-- **Package**: `dist/UnheardCity-3.0.0-windows-amd64.zip` (86 MB, catalogs included). With `PATH` reduced to
-  the Windows folders the packaged exe passes its smoke test (26,871 sounds loaded, encode + inject, 9,352
+- **Tests**: 127 automated tests (unit, synthetic game with packages/banks/dialogues/localization, a
+  two-language game for the lines, real-game read-only, offscreen GUI) pass. `ackit verify` PASS (incl.
+  300/300 in-memory WEMs rebuilt byte-identically).
+- **Real game** (2026-10-10), all reverting and checking byte-identity themselves:
+  `tools/e2e_features.py --lang all` 78/78 PASS in all 11 languages (every copy of an in-memory sound, each
+  language's localized line, prefetch heads, a multi-copy subtitle, a scripted subtitle, a menu text, sound +
+  subtitle in one package, a mod moved to another voice language and matched to the same line at another
+  position); `ackit verify --ingame` PASS (music heard, ncc 1.0);
+  `--banks` PASS (77 in-memory sounds of the front-end packages replaced in 54 files: the game loads them, 68
+  probes found in its memory, no crash); `--texts` PASS (12/12 menu texts and 23/23 subtitles of Startup.upk
+  found in the game's memory, no crash); `tools/e2e_editor.py` PASS. Game files original at the end
+  (`ackit status`). The catalog builder refuses to run on a modified game, so the catalog always describes the
+  original files.
+- **Package**: `dist/UnheardCity-3.1.0-windows-amd64.zip` (86 MB, catalogs included). With `PATH` reduced to
+  the Windows folders the packaged exe passes its smoke test (21,025 ITA sounds loaded, encode + inject, 9,377
   subtitles and 7,020 menu texts loaded, an in-memory sound decoded from a package) and the GUI starts and
   quits cleanly. The installer is unchanged since mission 2 (install/upgrade/uninstall verified then).
 
@@ -54,19 +67,25 @@ editor Unheard City (mission 2).
   text's length ("room"); cp1252 subtitles accept cp1252 characters only. Lifting these limits means moving
   objects inside the packages (new export offsets/sizes, added chunks): designed but not done, because it could
   not be verified automatically in the game (decisions 4c).
-- **Unlinked subtitles**: ~1 % of the subtitle strings have no event path to a sound (unused or placeholder
-  lines); they are listed and editable but not linked.
+- **Unlinked subtitles**: 29 speech names (~32 strings per language: Riddler trophy texts, Arkham City Stories,
+  radio reports and tapes) have no event in any bank; they are listed and editable but not linked.
 - **Subtitle coverage is uneven in the game itself**: no subtitles exist for efforts/grunts, few for combat barks
   (11 %) and cutscene lines (10 %; most cutscene lines are not played through speech events and have no dialogue
   object). No other subtitle storage was found in the packages (all classes with subtitle data checked).
-- ~200 speech sounds per language are played by no event and keep a position name (`NAME_i`); their speaker
-  labels apply to ITA only.
+- 659 voice lines are played by no event whose name the game stores; they are named after their English
+  recording's position (the same name in every language). Mod packages from versions before 3.1 carry no line
+  identity: their voice lines are not moved to another language (listed instead of guessed).
+- Speaker labels: 2 names of the curated lists match two recordings each (wemscan name collision), both are
+  labelled; 19 labels differ from the game's dialogue speaker field (to be checked by listening).
 - **Unnamed sounds**: ~9,000 BANK sounds have no event name and are named `<Bank>_<n>`.
 - Two copies of media `1C898731` differ; its checksum is left empty. Channel order for 4-6 channel sounds is
   unverified. M4A/AAC/WMA input needs an ffmpeg on the system.
-- Another game build is scanned at first start (WADs: seconds; banks and subtitles: a few minutes, cached).
+- Another game build is scanned at first start (WADs: seconds; banks, lines and subtitles: a few minutes,
+  cached); it gets the same line identities, but no original checksums.
 
 ## Next steps
 1. Gameplay check (a person): a replaced footstep/foley sound and an edited subtitle in a mission.
 2. Optional: lift the subtitle length limit by relocating dialogue objects (needs that gameplay check).
-3. Extend the curated speaker lists (`data/catalog/source/<lang>/*.txt`) and rebuild the catalog.
+3. Listen to the 19 lines whose speaker label differs from the game's dialogue speaker field
+   (`docs/categories.md`) and fix the curated lists if needed; extend them (`data/catalog/source/<lang>/*.txt`)
+   and rebuild the catalog.