Version 3.4.0: Edit tab (edit list, large subtitle, replacements with effects and presets), Sounds tab only browses, ackit fx (pedalboard + own reverb) and replace --fx, first Linux build, catalog without a mod package

docs/history/mission4_dubbing_plan.md

@@ -0,0 +1,284 @@
+# Plan: dubbing voice lines, with the game's voice effects
+
+Status (2026-10-27): stage 0 done (section 9); the scope changed after it (section 10: manual effect controls,
+no automatic profiles); stages 1-2 of section 10 done (effects engine, Dubbing tab). The user's answers are in section 8. Sections 1-7 are the original proposal, written
+from the code, the docs and the shipped catalog CSVs; the steps marked **(game)** read the game files.
+The game is also installed on the Linux machine (Steam Flatpak, mounted into the podman container at
+`/home/gora/.var/app/com.valvesoftware.Steam/.local/share/Steam/steamapps/common/Batman Arkham City GOTY`;
+the container's HOME differs, so it is passed with `--game`), so the read-only stages can run there; in-game
+checks still need the Windows PC.
+Sections 3-7 name modules and files that were proposed but not built after the scope change (`ackit/fxdetect.py`,
+`ackit/dub.py`, `data/catalog/FX.csv`, `fx_profiles.json`, `docs/formats/effects.md`); what was built is in
+sections 10-11 and `NOTES.md`: `ackit/fx.py`, `app/edit.py`, `app/editor.py`, `app/effects.py`, `app/preview.py`.
+
+## 1. The problem
+A dub replaces a character's lines with new recordings. Some original lines are not dry: they sound like a
+telephone or radio (band-limited EQ), a PA announcement over the city (echo, several speakers), a room (reverb),
+a suit or a machine (Mr. Freeze). A dry dub next to them sounds wrong. Two questions:
+
+1. **Where is the effect applied?** This decides everything:
+   - **Baked into the WEM** (applied in the studio, or by Wwise when the bank was built, *rendered* effect):
+     the dry dub loses it, so the dub must get the effect before it is written.
+   - **Applied by the game at run time** (a Wwise effect on the Sound, on a parent container or on a bus, an
+     environmental/reverb zone, attenuation, obstruction): the game also applies it to the replacement. The
+     dub must stay **dry**, or it gets the effect twice.
+
+   The user heard effects "in game and here": the ones heard *here* (the editor plays the WEM) are baked; the ones
+   heard only in game are run-time and need nothing.
+2. **Which effects exist?** Unknown, so they have to be found in the game data (section 3).
+
+## 2. What the catalog already shows (INT, measured on the shipped `data/catalog/INT.csv`)
+Cheap signals that are already in the catalog (`rate`, `channels`) and point to baked effects:
+
+| signal | lines | groups (examples) | likely effect |
+|--------|-------|-------------------|---------------|
+| 3 or more channels | 858 WAD speech rows (mostly 5 ch, then 4 and 3 ch), plus 86 in-memory 5 ch | Predator chatter 251, story 211, Riddler 90: `PRED_RIDDLER`, `PRED_J`, `PRED_STRANGE`, `PRED_MAYOR`, `RIDDLER_R`, `GAME_OVER`, `COURT_B`, `UNDER_S` | PA / hacked-speaker broadcasts: one voice spread over several channels, probably with delays (echo) between them |
+| 32,000 Hz | 110 (102 are `CHAR_FREEZE`) | Mr. Freeze | suit voice filter |
+| 11,025 / 12,000 Hz | 91 | `CHR_RadioScanner_*` | police radio (band-limited) |
+| 6,000 Hz | 25 | `PSO_CarRadio_Chatter_*` | car radio |
+| category *Radio & TV* (`Speech_RADIO`, `Speech_TV`, `Speech_CAMERAS`, `DIA_`) | see `categories.md` | Vicki Vale reports, TV, cameras | radio/TV EQ, maybe noise |
+
+These are hypotheses until someone listens and the metadata scan (3a) confirms them. Two things follow already:
+- A mono dub of a 5-channel line is copied to every channel today (`audio.to_channels`): the spatial echo between
+  the channels is lost. Multichannel lines need a per-channel layout (gain + delay per channel) in their effect.
+- Low-rate lines limit the band themselves (6 kHz = nothing above 3 kHz), so part of the "radio" sound is just
+  the encode rate; the rest (low cut, mid boost, distortion, noise) has to be added.
+
+## 3. Finding every effect the game uses
+Three independent sources, from exact to fuzzy. Every result goes into the catalog, so the editor can say
+for each line: *baked: Radio EQ (from metadata)*, *baked?: echo, 3 channels (analysis, 0.8)*, *game adds: room
+reverb (run time, record dry)*.
+
+### 3a. Wwise metadata (exact, where it exists) **(game)**
+The banks (version 56, Wwise 2011.1) describe effects in HIRC; `ackit/bnk.py` reads only the Sound sources today.
+To add (`ackit/bnk.py` + a survey tool, like the bank survey of mission 3):
+- Parse the **node base parameters** of Sounds, containers, actor-mixers and buses: the effect list (slot,
+  effect object ID, and a **"rendered"** flag in this bank version, per wwiser's v56 layout, to confirm on the
+  game's banks), the parent ID (effects are inherited unless the child overrides them), the output bus.
+- Parse the **effect objects** (HIRC effect / share-set type; the type number for v56 to confirm): plugin ID
+  (Audiokinetic Parametric EQ, Delay, RoomVerb, Matrix Reverb, Peak Limiter, Compressor, Expander, Pitch
+  Shifter, Flanger, Guitar Distortion, Tremolo, ...; plugin ID = company + type + number) and its **parameters**
+  (EQ band frequencies/gains/Q, delay time and feedback, reverb time ...). The `FXPR` chunk the parser already
+  recognizes and `ENVS` in `Init.bnk` (environments, i.e. run-time reverb zones) belong here too.
+- Walk each line's Sound object up the parent chain to the master bus: list of `(effect, plugin, params,
+  where: sound/parent/bus, rendered or run time)`.
+- Line identity helps: Sound objects are the same in every language (`banks.md` *Lines*), so a line's chain is
+  the same in all 5 languages.
+
+Expected result: run-time effects (bus reverbs, environments) are fully known, with parameters. Rendered
+effects are known **with their exact parameters**, which is the best case for a dub. Effects applied in the
+studio before the files reached Wwise leave **no metadata at all**: likely many of the radio/PA lines. Those need 3b.
+
+### 3b. Signal analysis of the WEMs (no metadata needed) **(game, ~minutes over all speech)**
+`ackit/fxdetect.py` (numpy only), per recording, a small feature vector:
+- **Band limits**: frequency below which / above which 1 % of the energy lies (telephone ~300-3,400 Hz), steepness.
+- **Effect EQ curve**: long-term average spectrum (LTAS) of the line divided by the LTAS of the **same speaker's
+  dry lines** (speaker from `characters.csv` / the line's speaker). The voice cancels out and what is left is the
+  effect's frequency response: directly usable as the dub's EQ.
+- **Reverb**: energy decay after speech offsets (Schroeder-style decay fit -> RT60 estimate), direct-to-
+  reverberant ratio, tail length after the last word.
+- **Echo / delay**: peaks in the autocorrelation of the onset envelope, or in the cepstrum, at lags 30-800 ms;
+  for multichannel lines the cross-correlation **between channels** (delay and gain per channel).
+- **Distortion / noise**: crest factor, clipping, spectral flatness of the gaps between words (static, hiss),
+  harmonics added to a narrow band.
+- **Modulation / pitch**: envelope modulation 2-20 Hz (tremolo), pitch and formant shift against the speaker's
+  dry lines (masks, Freeze, Clayface, synthesized voices).
+
+Then **cluster** the vectors (e.g. k-means on normalized features, or simple thresholds first) and **listen to a
+few lines per cluster** (the user names it: "Radio scanner", "Freeze suit", "Strange PA", "Riddler hacked
+speakers", "Interview tape", ...). The names and cluster labels go into a profile table (3d). Cross-language
+check: the 5 recordings of one line must land in the same cluster; disagreements are lines to listen to.
+
+### 3c. Comparing with the game's output (optional, run-time effects only) **(game)**
+The in-game harness can already play sounds and record the loopback (`ackit/verify/ingame.py`, `loopback.py`).
+For a sample of lines with a run-time effect, the same features computed on the game's output vs on the WEM
+show what the game adds. **Never save raw captures** (CLAUDE.md: they can contain voice chat): compute the
+features in memory and keep only the numbers. Useful to confirm, not needed for the dub itself.
+
+### 3d. What gets stored
+- `data/catalog/FX.csv`: per line (`line` identity, language-independent where the metadata says so): `profile`,
+  `baked` (yes / no / unknown), `source` (metadata / analysis / manual), `confidence`, `runtime` (what the game
+  adds, for the "record dry" hint). Built by `tools/build_catalog.py` from the original files only, like the rest.
+- `data/catalog/fx_profiles.json`: the profiles (section 4), with where each parameter came from.
+- `docs/formats/effects.md` (format of the HIRC effect data, survey figures) and claims in `tools/check_docs.py`
+  for every figure; `ackit verify --catalog` re-derives the metadata part.
+- Manual labels of the user live in user data (like notes/tags), never in `source/`.
+
+## 4. Re-creating the effects on a dub
+An **effect profile** = a chain of simple, deterministic processors with parameters (data, not code), rendered by
+`ackit/fx.py`:
+
+| block | parameters | source of the numbers |
+|-------|------------|-----------------------|
+| high-pass / low-pass / peaking / shelf (biquads) | freq, gain, Q | metadata EQ, or fitted to the 3b EQ curve |
+| matched EQ (FIR) | the 3b curve itself | analysis (generic fallback for any baked EQ) |
+| delay / echo | time, feedback, mix, per-channel delays/gains | metadata, or 3b autocorrelation / inter-channel lags |
+| reverb | RT60, pre-delay, damping, mix | metadata (RoomVerb), or 3b decay fit; convolution with a generated IR (filtered, exponentially decaying noise) keeps it deterministic |
+| distortion / bit-crush / noise bed | drive, bits, noise level and colour | 3b, tuned by ear |
+| compressor / limiter | threshold, ratio | metadata, or by ear |
+| channel layout | per-channel gain + delay (replaces the plain mono copy for 3-5 channel lines) | 3b inter-channel analysis |
+
+The rendered dub then goes through the existing chain: loudness matched to the original, encoded at the
+original's rate/channels, fitted to the size limit (`Workspace.prepare`). A reverb or echo tail makes the audio
+longer: the editor's length warning and size fitting already cover that, but a dub UI should show the room.
+
+**DSP dependency** (user: "just use some libraries", no own DSP, no VST hosting):
+- Effects: **pedalboard** (Spotify, GPL-3.0, compatible with this project's GPL-3.0): ready filters (high/low-
+  pass, peak, shelf), reverb, delay, chorus, phaser, distortion, bit-crush, GSM compression (telephone),
+  compressor/limiter, pitch shift. A profile is a list of pedalboard plugins with parameters; the per-channel
+  layout (gain + delay per channel) is a few numpy lines. To check before stage 1: wheels for the project's Python
+  (3.14) on Windows and Linux, and that PyInstaller bundles it.
+- Analysis (3b): numpy (FFT, correlations), scipy only if a step really needs it.
+- Decision goes into `docs/decisions.md` like the other dependencies, licenses into `THIRD_PARTY_LICENSES.txt`.
+
+## 5. Where to build it: new app or the existing editor?
+
+| | Separate dubbing app on `ackit` | Inside Unheard City (new **Dubbing** tab) |
+|-|-------------------------------|------------------------------------------|
+| Write safety | `Workspace.lock` is a `threading.RLock`: it serializes writes **inside one process** only. Two apps open on the same game and store can write one package at the same time (the lost-edit bug of 2026-10-13, across processes). Needs a cross-process file lock + "changed outside" handling in both apps. | Same process, same lock, same store, same changed-outside banner: nothing new. |
+| Reuse | Re-implements catalog loading (background jobs, spinners), language setting, line identity, subtitles, waveform, player/recorder, encode/fit, apply/revert, mod export/import, settings, packaging. | All of that exists (`app/editor.py`, `audioio.py`, `waveform.py`, `textpanel.py`, `modpack.py`). |
+| Users | Two programs, two installs, two settings files, two originals stores to keep apart. | One program; a dub is just another kind of change (and a mod package). |
+| Complexity | The editor stays as it is. | `mainwindow.py` (774 lines) and `editor.py` (590) must not grow into one blob: the tab is its own package. |
+| UX | Free to design a session-based workflow. | Same freedom inside a tab; the Sounds tab stays for one-off replacements. |
+
+**Recommendation: build it into Unheard City, with the engine in `ackit`.**
+- `ackit/fx.py` (profiles + rendering), `ackit/fxdetect.py` (features, clustering), `ackit/dub.py` (dub
+  sessions/takes: files, states; no Qt), CLI `ackit fx survey | scan | render` and `ackit dub ...`: all headless,
+  testable, usable from scripts.
+- `app/dubbing/` (own package): the **Dubbing** tab.
+- The **Sounds** editor gets one small row: *Effect: [profile ▾] (detected: Radio EQ, metadata)* and a *game
+  adds ... record dry* hint, so single replacements benefit too.
+- A separate program makes sense only later, if voice actors record **without the game** (remote dubbing): a
+  small recorder that reads a script export and writes a take pack (WAVs + JSON), which Unheard City imports.
+  It never touches game files, so the lock problem does not arise.
+
+## 6. The Dubbing tab (workflow)
+1. **Session**: language + character (or a filter: category, a list of lines). Lists the character's lines via the
+   line identity; shows the script (subtitle of the loaded language, the INT text as reference), the original's
+   length, the effect profile and baked/run-time status, and a state per line: *to do / recorded / approved /
+   applied*. Sessions and takes are stored in user data (`%APPDATA%\UnheardCity\dubs\...`), never in the game.
+2. **Record**: play the original as a cue, count-in, record, auto-trim silence, several takes per line, pick one;
+   length vs the original (and optional time-stretch to fit, if pedalboard or a WSOLA stretch is added).
+3. **Process**: clean-up (high-pass, gate), loudness match to the original (exists), the effect profile
+   (suggested from FX.csv, editable, A/B against the original with and without the effect).
+4. **Apply** the approved lines in one `Workspace.apply` call (it takes a list); **export** the session as a mod
+   package (`modpack.export_mod`, with edited subtitles if the dub's wording changed, within their room).
+5. Non-destructive: raw takes + parameters are kept, so a changed profile re-renders every line.
+
+Warnings the tab must show (from the existing data):
+- Lines the game shares between languages (167 in-memory voice lines stored once for every language; 616-651 WAD
+  lines per language identical to INT, `banks.md` *Lines*): dubbing one changes it for every language that uses it.
+- Size limit (speech in place, never larger than the original WEM): long takes plus reverb tails get squeezed or
+  refused; show the room.
+- Subtitle room (NUL-padded in place).
+- Lines without a speaker label or with a disputed one (19 listed in `categories.md`).
+
+## 7. Stages (one commit each: code + tests + docs + NOTES status)
+0. **Evidence (game)**: HIRC effect survey (3a) and a first feature scan (3b) over all speech; listen to samples
+   per cluster with the user; write `docs/formats/effects.md` with figures (+ `check_docs` claims). Outcome: how
+   many lines are baked vs run time, and the list of profiles. **This decides the size of everything after it.**
+1. `ackit/fx.py` on pedalboard: profiles JSON, render; tests with synthetic signals (an EQ's
+   measured response, a delay's lag, an RT60 measured back from the rendered reverb).
+2. `ackit/fxdetect.py` + `FX.csv` / `fx_profiles.json` in `build_catalog.py`; `verify --catalog` checks; CLI.
+   Test: render a profile onto a dry synthetic voice, detect it back.
+3. Sounds editor: effect row + run-time hint; GUI tests.
+4. `ackit/dub.py` + Dubbing tab (sessions, takes, script, batch apply, mod export); GUI tests.
+5. Real-game round trips (`tools/e2e_features.py` / `e2e_editor.py` cases for a dubbed line with an effect,
+   reverted in `finally`), the user listens in game, packaging (`build_app.py`, licenses), version 3.4.0 or 4.0.0.
+
+## 8. Decisions (user, 2026-10-27)
+- **All characters** can be dubbed, from the existing catalogs (no per-character priority): every effect group
+  found in stage 0 needs a profile.
+- **Approximate effects** are enough; profiles are tuned later by ear (the Riddler lines' effect is complex).
+- **Libraries, not own DSP**: pedalboard for effects (section 4); no VST hosting, no time stretching for now.
+- **Everyone has the game**: no separate recorder or take packs; everything lives in Unheard City (section 5).
+- Still open, out of scope unless asked: audio inside video files (pre-rendered movies carry their own sound
+  track, not in the WADs or banks; to check whether the game has any with speech).
+
+## 9. Stage 0 results (2026-10-27, INT, read-only)
+Measured with scratch scripts (HIRC survey of every bank, signal features of 12,430 INT speech recordings),
+then the user listened to 77 candidate groups (`external/fx_samples/INDEX.md`, git-ignored). Figures here are a
+record of that day, not re-checked by `tools/check_docs.py`.
+
+**Metadata (Wwise, v56).** Layouts from wwiser: node base params = u8 override-parent-fx, u8 count, [u8 bypass,
+count x {u8 slot, u32 fx id, u8 share-set, u8 rendered}], u32 output bus, u32 parent; a Sound has them after its
+source (+ u8 source bits); a bus (v56) has u8 is-environmental at payload offset 33, ducks of 17 bytes, then its
+effects. Effect objects are HIRC types 0x12 (share-set) / 0x13 (custom): u32 plugin, u32 size, parameters.
+- Run time (the game adds them; the files are clean, so a dub is recorded **dry**): McDSP FutzBox share-set
+  `4272216A` (in 1,125 speech banks, on 464 actor-mixer slots, RTPC-driven) -> 2,858 speech rows; Parametric EQ
+  on a bus (30 rows) or on the sound (3); the ML1 limiter on the top bus (everything); 84 RoomVerb + 6 Matrix
+  Reverb share-sets in `Init` = the environments the game switches per area.
+- Rendered (baked by Wwise): 33 speech rows (car radio 25, radio scanner 3, museum thug-to-Penguin 4, Steel HQ
+  1); the slot's fx id is zeroed, so the effect type is lost.
+- Everything else that is baked was made in the studio and has no metadata.
+
+**Channels.** Speech WEMs with 2-5 channels carry the WAVE channel mask: 2 ch `0x3` (L R), 3 ch `0x7` (L R C),
+4 ch `0x33` (L R SL SR), 5 ch `0x37` (L R C SL SR); ackit decodes and encodes them in that order. PA lines put
+the dry voice in C and an echo 160-173 ms later in the surrounds (Harley/HQ 121 ms; Freeze's quad rears 73 ms);
+Riddler, Ivy and Ra's 5-channel lines are processed differently per channel.
+
+**What the user heard** (per group; "no effects" groups omitted: thug/Ninja/Catwoman/JR efforts, combat barks,
+Riddler `RIDDLER_R`, DLC `PDLC_PDLC`/`PDLC_EMOTES`/Robin/Harley, Outcast, Penguin `MUSEUM_B`, Steel/Church Harley,
+Bully hobo, Talia `UNDER_B`, `CH_ZERO` Strange, Strange/Black Mask game-over):
+
+| effect heard | groups |
+|--------------|--------|
+| radio EQ | radio scanner, Alfred/Oracle/Penguin/Strange/Ra's comms, copter, cameras, Gordon radio, car radio |
+| phone EQ | Penguin PRED, Zsasz calls, Two-Face court, Joker (PRED, theatre), Strange/Mayor/Harley PA, Riddler (Enigma, cameos), Vicki Vale radio scan, Penguin exhibits |
+| phone-mic EQ / TV-speaker EQ | interview tapes (Vicki Vale, Joker), Strange on TV, Joker church message, `UNDER_C` TV |
+| radio noise | copter, car radio, radio scanner (Vicki), Vicki Vale radio scan, map radio |
+| distortion (slight, speaker) | Penguin exhibits, radio scanner (Vicki), `UNDER_C` TV |
+| echo (PA delay) | Strange/Mayor/Harley/Joker/Two-Face PA and court lines, Catwoman side mission (Strange) |
+| very short delay (width) | Riddler cameos, radio scanner (Riddler) |
+| reverb (small/short room, slight wide room) | Riddler PRED, Harley 5 ch, Strange TV, Joker church, Talia sequence, Harley DLC, game-over lines (barely) |
+| reverse reverb (occasional) | Riddler (PRED, cameos, radio scanner), Poison Ivy |
+| glitches (parts repeat at another pitch), small repeats | Riddler (PRED, cameos, Enigma, radio scanner) |
+| pitch shift, fixed, down | Clayface |
+| pitch shift over time + robot + pan over time | Mr. Freeze |
+| octaves overlaid (down), formant shift, stereo width | Ra's al Ghul (giant, trials), Poison Ivy |
+| formant shift, unintelligible voice | car radio |
+
+## 10. Scope after stage 0 (user, 2026-10-27)
+Matching effects automatically per line is too complex: the Dubbing section gets **manual controls for every
+effect found** (section 9), which the user sets by ear. Dropped: automatic profiles, `FX.csv` /
+`fx_profiles.json`, detection in the catalog build (stage 2 of section 7). Kept: pedalboard, the Dubbing tab in
+Unheard City, everything else of sections 5-6.
+
+Effects rack (`ackit/fx.py`, one table of blocks and parameters that the UI is generated from; every block off
+by default; fixed order; seeded randomness, so a setting always renders the same audio):
+
+| # | block | controls | built on |
+|---|-------|----------|----------|
+| 1 | Pitch | shift (semitones), formant shift, drift depth + rate (pitch over time) | `pedalboard.time_stretch` (pitch curve, formants) |
+| 2 | Octave layers | two layers: interval + level | `time_stretch` pitch, mixed |
+| 3 | Robot | ring modulator frequency + mix, metallic comb (delay, feedback) | numpy |
+| 4 | Glitch | events per 10 s, length, repeats, pitch of the repeats, seed | numpy + `time_stretch` |
+| 5 | Voice band (EQ) | low cut, high cut, slope, mid boost (freq, gain, Q) | `HighpassFilter`, `LowpassFilter`, `PeakFilter` |
+| 6 | Distortion | drive, clip, bit depth, phone codec (GSM) | `Distortion`, `Clipping`, `Bitcrush`, `GSMFullRateCompressor` |
+| 7 | Noise | level, colour (band), crackle | numpy |
+| 8 | Echo | time, feedback, mix | `Delay` |
+| 9 | Reverb | room size, damping, wet, width; reverse reverb amount + how often | `Reverb` (reversed for the reverse swell) |
+| 10 | Channels (2-5 ch lines) | width (L/R delay), auto-pan rate + depth, surround echo delay + level, dry voice in C | numpy, channel order of section 9 |
+
+Then the existing chain: loudness match / normalize / gain, encode within the size limit.
+
+Stages from here (one commit each; 1 and 2 done on 2026-10-27, see NOTES.md):
+1. `ackit/fx.py` + tests (synthetic signals: each block measured back) + pedalboard dependency
+   (`pyproject.toml`, `docs/decisions.md`, licenses).
+2. Dubbing tab (`app/dubbing/`): line list (character/category/search filters, state per line), the existing
+   sound editor embedded (record/load, trim, A/B, subtitle, apply/revert), the effects rack between the take and
+   the loudness step, settings saved per line in user data; GUI tests; user guide.
+3. Real-game round trip of a dubbed line with effects, packaging (PyInstaller + pedalboard), version bump.
+
+## 11. UI revision (user, 2026-10-27, after trying stages 1-2)
+- The subtitle box was too small for dubbing, and the Sounds tab and the Dubbing tab both had an editor:
+  browsing and editing are now separate. **Sounds** only browses (list + a compact preview with **Edit** / **Add
+  to edit list**); **Dubbing** became **Edit**, for every kind of sound, with an edit list instead of a second
+  browser, the subtitle as a large **script**, and Previous / Next.
+- The effects panel was "raw": sliders with readable values (log scale for frequencies and times), **looks**
+  for the voices heard in the game (section 9) and **preset buttons** per effect that set the sliders in place.
+- The reverb (pedalboard's Freeverb) sounded poor: replaced by a generated room response (decay, pre-delay,
+  reflections, damping, width) with presets (small room to church, tiled cell, outdoors).
+- Second round: the browser's player keeps Play/Stop together (editing actions apart on the right); the Edit tab
+  fits the window without scrolling (effects as a chain with one settings area); the app and docs use generic
+  terms (replacement, effects, subtitle), since the Edit tab is for every kind of sound, not only dubbing.