UnheardCity/NOTES.md

# NOTES — Arkham City audio toolkit & editor

## Current status / Next steps (rewrite as you go)
- Mission 1 (2026-10-07): DONE. Format reversed, encoder + injector, verified in-game (no exe patch).
- Mission 2 Part 1 (2026-10-08): repository cleanup DONE — tag `pre-cleanup` holds the old state.
  New layout (README "Where things are"); `ackit` refactored: portable encoder (libsndfile, no ffmpeg/vgmstream),
  `Game` (Steam/Linux/Proton discovery), catalog from `data/catalog/*.csv` (user's character lists + rules),
  originals store + `Workspace` (apply/revert/conflicts), mod packages, CLI. 63 tests pass; `ackit verify` PASS;
  `ackit verify --ingame` PASS (E14: ncc 1.0, revert byte-identical).
- Part 2 (2026-10-08): Unheard City DONE (app/, PySide6): browser (search + group/category/character/
  archive/tag/modified/favorites filters, sortable, multi-select, notes/tags/favorites), editor (original/installed/
  replacement waveforms, A/B, loop, volume, file or recording, trim/gain/normalize/match loudness, apply/revert,
  conflict + changed-underneath handling), mod export/import, settings. 72 tests incl. offscreen GUI tests.
  Packaging: `python tools/build_app.py` -> dist/UnheardCity (self-contained, licenses); packaged exe passes
  `--smoke-test` with PATH = C:\Windows only. Real e2e through the editor code path (tools/e2e_editor.py): PASS
  (music heard in game ncc 1.0, reverts byte-identical, speech line apply/revert byte-identical).
- Mission 3 (2026-10-09, brief: `docs/history/mission3_audio_text_prompt.md`): full-game audio, then subtitle
  and menu text editing. Plan (one commit per stage; a stage = code + tests + docs + this status):
  1. [x] `ackit/lzo.py`, `ackit/upk.py` (UE3 package summary, compressed chunks, block-level read/patch),
     `ackit/bnk.py` (bank parsing: DIDX/DATA/HIRC sources). Docs: formats/upk.md, banks.md. Findings: pure-Python
     LZO ~47 MB/s (== decompress.exe on every block); 15,800 media IDs in 96,011 copies, 396 are prefetch heads
     of WAD sounds (2,101 copies) -> 15,404 in-memory sounds; copies identical except 1C898731.
  2. [x] Catalog archive `BANK` (15,404 sounds, 6,416 named by events) + `BANK_locations.csv.gz` (93,910 copies,
     2,101 prefetch heads of 396 speech lines); `ackit verify --catalog` (full: catalog OK; fixed 2 duplicate
     names per language: variation shown as NAME#i); categories for bank prefixes.
  3. [x] Workspace writes BANK sounds into every copy and keeps speech-line prefetch heads in sync (bank files
     rebuilt from a mandatory full original copy); editor shows bank + copies. `ackit verify` PASS (300/300
     BANK WEMs rebuilt identically); `ackit verify --ingame --banks` PASS (loaded, no crash, revert identical;
     audible check not possible without input: findings 18).
  4. [x] Subtitles = RDialogueEvent objects in packages (9,016 lines x 11 languages, 168k copies; parser
     `ackit/subtitles.py`, catalog SUB_<LANG>.csv + SUB_locations.csv.gz, 92 % linked to speech sounds via
     events); menu text = Localization/<LANG>/*.int (`ackit/loctext.py`, ~7,000 strings/language).
     `Workspace.texts` (ackit/texts.py): states, apply/revert, conflicts, mod packages; CLI `ackit text`.
     Subtitles in place (NUL-padded, limit = original length); decision + reasons in decisions.md 4c.
  5. [x] Editor: tabs Sounds / Subtitles / Menu text (app/textpanel.py: search, filters, edit with room
     counter, apply/revert/revert all, find & replace on the filtered rows, go to sound); the sound editor
     shows and edits the sound's subtitle (SubtitleBox); texts in mod export/import and the changed-outside
     banner; smoke test loads texts. 15 GUI tests.
  6. [ ] Final e2e (audio + text), packaged build, REPORT.md.
- OPEN: channel order for >2 channels unverified; speech and in-memory sounds *heard* in-game (need gameplay;
  posted keys do not drive the menu).

## Environment facts
- Game: Steam AppID 200260 "Batman: Arkham City GOTY", `C:\Program Files (x86)\Steam\steamapps\common\Batman Arkham
  City GOTY`, exe `Binaries\Win32\BatmanAC.exe` (32-bit, SteamStub). buildid 3718300. Game language: Italian
  (`Documents\WB Games\...\UserEngine.ini` is UTF-16, `Language=Ita`).
- Steam must be running and logged in for the in-game test (`steam.exe -silent`, wait for
  HKCU\Software\Valve\Steam\ActiveProcess ActiveUser != 0). appmanifest AutoUpdateBehavior=0 (no updates expected).
- History: the ITA WAD in the game folder was once the user's modded copy; the clean original came from
  `%USERPROFILE%\Desktop\ArkhamCityModding\Audio\49D84887_LOC_ITA\WAD\` (sha1 1238172C...). The game is clean now
  (fingerprints match the old `_backup/`, deleted on 2026-10-08).
- Dev tools: `.venv` (Python 3.14.6, PySide6 6.10.3, soundfile 0.14.0 / libsndfile 1.2.2, soxr 1.1.0, numpy,
  pytest, PyInstaller 6.22.3, soundcard 0.4.6). git at `C:\Program Files\Git\cmd`. WinDbg Store (path changes on
  update; `ackit/verify/debugger.py: find_cdb`). No WSL distro, no Docker.
- User data (this PC): `%APPDATA%\UnheardCity\` (settings.json, stores\<id>\), cache `%LOCALAPPDATA%\UnheardCity\cache`.
- The user's own material: `%USERPROFILE%\Desktop\ArkhamCityModding` (old mod work, Ghidra project, dump);
  their third-party downloads were moved from `tools/`, `repositories/` to `external/` (git-ignored).
- Ghidra project BAC: `.text` Ghidra address = runtime + 0xC00 (docs use runtime addresses).

## Where to look
- Findings, failed hypotheses, in-game experiment log E0-E14: `docs/reverse-engineering/findings.md`.
- Formats: `docs/formats/`. Decoder addresses: `docs/reverse-engineering/decoder.md`.
- Decisions (stack, deps, licenses, originals store, size rule): `docs/decisions.md`.

## Session log
- 2026-10-08: user added `my_catalog_for_ita/` (per-character lists, wemscan names) -> moved to
  `data/catalog/source/ita/`; 3364 names labelled (2 wemscan name collisions label both candidates; one line had a
  `.wem.wav` typo). Built catalogs for ITA/INT/FRA/DEU/ESN/SFX (identical names and counts across languages).
  Assumption: speakers are the same in every language, so ITA labels apply to all.
- 2026-10-08: libsndfile's Vorbis writer crashed (C stack overflow) on a 30 s x 6 ch single write -> block writes.
- 2026-10-08: wad_fingerprint made structural (IDs + non-pack sizes) so music edits don't invalidate the catalog;
  scan mode never borrows checksums from the shipped CSV (other builds / synthetic tests).
- 2026-10-08: editor async loading looked stuck: Job objects were garbage-collected before their queued result
  signal arrived -> Runner keeps references until done. Qt 6.10 deprecates invalidateFilter/invalidateRowsFilter ->
  SoundFilter.refilter uses begin/endFilterChange when available.
- 2026-10-08: search haystack must not contain the variation index (":2" matched "2"); IDs are matched separately
  (words of >= 6 chars against the target id).
- Verification commands: `pytest -q`; `ackit verify`; `ackit verify --ingame`; `python tools/e2e_editor.py`
  (needs Steam logged in); packaged: `dist/UnheardCity/UnheardCity.exe --smoke-test --report f.txt`.
- 2026-10-08: UI restyle on request: vector icon set (app/icons.py, no emoji), "Arkham" theme (app/theme.py: QSS +
  palette, translucent panels over a wallpaper backdrop).
- 2026-10-08: user rejected the procedural icon/backdrop and supplied images (~/Downloads/Arkham City:
  bac_logo_full.png, wallpaper.jpg). tools/make_assets.py builds git-ignored app/assets/ (icon = logo bat over a
  red waveform; wallpaper copied); app/art.py, make_icon.py and the old icon files removed. Settings ->
  Background toggle (background_enabled). Character filter is now a plain combobox like the others.
- 2026-10-08: theme selector and custom background image removed (one look); the wallpaper is painted only in the
  filter panel (theme.Sidebar: scaled to panel height, left part shown).
- 2026-10-08: build simplified to one offline command: build_app.py runs make_assets when app/assets is missing
  (--art FOLDER), and license texts live in tools/licenses/ (www.gnu.org was down; no more downloads).
- 2026-10-08: neutral dark theme (no blue), editor header icon removed, new icon: flat BAC bat with echolocation
  arcs. build_app.py also regenerates app/assets when tools/make_assets.py is newer than the icon.
- 2026-10-08: background strength default 10 %; obsolete settings keys (theme, background_image) dropped on load.
  Spinners (app/spinner.py): waveform overlay while loading, editor action row, status bar for global jobs.
  Replacement files now decode in a background job.
- 2026-10-08: accent is now dark red (#b0242c; icons.ACCENT / theme.ACCENT, was AMBER). Icon: white rounded tile,
  red waveform bars, black logo bat on top. Filter panel opens at its maximum width (320 px, can only shrink).
  Sidebar veil alpha follows the background strength (at 10 % the wallpaper stays visible).
- 2026-10-09: editor renamed "Unheard City" (exe/zip UnheardCity, launcher unheard-city, user data %APPDATA%\UnheardCity).
  Icon: 9 wider/taller bars (5 at <=24 px), bat 78% of the tile. Panel background #0c0c0c.
- 2026-10-09: editor panel opens fitted to its contents (MainWindow.fit_editor; refits on layout changes until
  the divider is dragged). Windows installer: tools/installer (Install.cmd -> install.ps1/uninstall.ps1, per-user,
  HKCU uninstall entry), copied into the build by build_app.py; install/upgrade/uninstall cycle verified.
  User chose no new software (no Inno Setup).
- 2026-10-09: app/assets is committed (user: public artwork and generated content, not Warner Bros. pictures).
  build_app.py regenerates it only when missing or with --art; after changing make_assets.py run it by hand.
- 2026-10-09: SFX coverage check (read-only): the catalog covers every WAD sound, but ~15,800 in-memory WEMs in
  soundbanks are missing (docs/formats/banks.md). Packages were unpacked with Gildor's decompress.exe from
  `%USERPROFILE%\Desktop\ArkhamCityModding\Audio\decompress\` (outside the repo). User: document now, implement
  editing later.
- 2026-10-09 (mission 3): BANK = one row per media ID, written into every copy in place (assumption: the game
  may load any copy; a partial write would be inconsistent). Package writes store changed blocks as LZO literal
  runs (no compressor needed). Bank-file conflicts use the whole-file SHA-1 (the cheap guid+chunk-table
  fingerprint missed same-size edits in tests). Name collisions in the WAD catalogs (2 per language) are shown as
  NAME#i. `soundcard` comes from the `verify` extra (was missing from the venv).
- 2026-10-09 (mission 3, stage 4): subtitle links by name were wrong for most multi-variation entries (WEMs are
  stored in media-ID order, not _NN order): links come only from events. Numbered event names (Speech_X_04)
  live in export tables, not in name tables: the bank scan now collects export names (+40 named BANK sounds,
  links 71 % -> 92 %). Version*.int (build IDs) are not treated as UI text.