# Plan: dubbing voice lines, with the game's voice effects

Status (2026-10-27): stage 0 done (section 9); the scope changed after it (section 10: manual effect controls,
no automatic profiles); stages 1-2 of section 10 done (effects engine, Dubbing tab). The user's answers are in section 8. Sections 1-7 are the original proposal, written
from the code, the docs and the shipped catalog CSVs; the steps marked **(game)** read the game files.
The game is also installed on the Linux machine (Steam Flatpak, mounted into the podman container at
`/home/gora/.var/app/com.valvesoftware.Steam/.local/share/Steam/steamapps/common/Batman Arkham City GOTY`;
the container's HOME differs, so it is passed with `--game`), so the read-only stages can run there; in-game
checks still need the Windows PC.
Sections 3-7 name modules and files that were proposed but not built after the scope change (`ackit/fxdetect.py`,
`ackit/dub.py`, `data/catalog/FX.csv`, `fx_profiles.json`, `docs/formats/effects.md`); what was built is in
sections 10-11 and `NOTES.md`: `ackit/fx.py`, `app/edit.py`, `app/editor.py`, `app/effects.py`, `app/preview.py`.

## 1. The problem
A dub replaces a character's lines with new recordings. Some original lines are not dry: they sound like a
telephone or radio (band-limited EQ), a PA announcement over the city (echo, several speakers), a room (reverb),
a suit or a machine (Mr. Freeze). A dry dub next to them sounds wrong. Two questions:

1. **Where is the effect applied?** This decides everything:
   - **Baked into the WEM** (applied in the studio, or by Wwise when the bank was built, *rendered* effect):
     the dry dub loses it, so the dub must get the effect before it is written.
   - **Applied by the game at run time** (a Wwise effect on the Sound, on a parent container or on a bus, an
     environmental/reverb zone, attenuation, obstruction): the game also applies it to the replacement. The
     dub must stay **dry**, or it gets the effect twice.

   The user heard effects "in game and here": the ones heard *here* (the editor plays the WEM) are baked; the ones
   heard only in game are run-time and need nothing.
2. **Which effects exist?** Unknown, so they have to be found in the game data (section 3).

## 2. What the catalog already shows (INT, measured on the shipped `data/catalog/INT.csv`)
Cheap signals that are already in the catalog (`rate`, `channels`) and point to baked effects:

| signal | lines | groups (examples) | likely effect |
|--------|-------|-------------------|---------------|
| 3 or more channels | 858 WAD speech rows (mostly 5 ch, then 4 and 3 ch), plus 86 in-memory 5 ch | Predator chatter 251, story 211, Riddler 90: `PRED_RIDDLER`, `PRED_J`, `PRED_STRANGE`, `PRED_MAYOR`, `RIDDLER_R`, `GAME_OVER`, `COURT_B`, `UNDER_S` | PA / hacked-speaker broadcasts: one voice spread over several channels, probably with delays (echo) between them |
| 32,000 Hz | 110 (102 are `CHAR_FREEZE`) | Mr. Freeze | suit voice filter |
| 11,025 / 12,000 Hz | 91 | `CHR_RadioScanner_*` | police radio (band-limited) |
| 6,000 Hz | 25 | `PSO_CarRadio_Chatter_*` | car radio |
| category *Radio & TV* (`Speech_RADIO`, `Speech_TV`, `Speech_CAMERAS`, `DIA_`) | see `categories.md` | Vicki Vale reports, TV, cameras | radio/TV EQ, maybe noise |

These are hypotheses until someone listens and the metadata scan (3a) confirms them. Two things follow already:
- A mono dub of a 5-channel line is copied to every channel today (`audio.to_channels`): the spatial echo between
  the channels is lost. Multichannel lines need a per-channel layout (gain + delay per channel) in their effect.
- Low-rate lines limit the band themselves (6 kHz = nothing above 3 kHz), so part of the "radio" sound is just
  the encode rate; the rest (low cut, mid boost, distortion, noise) has to be added.

## 3. Finding every effect the game uses
Three independent sources, from exact to fuzzy. Every result goes into the catalog, so the editor can say
for each line: *baked: Radio EQ (from metadata)*, *baked?: echo, 3 channels (analysis, 0.8)*, *game adds: room
reverb (run time, record dry)*.

### 3a. Wwise metadata (exact, where it exists) **(game)**
The banks (version 56, Wwise 2011.1) describe effects in HIRC; `ackit/bnk.py` reads only the Sound sources today.
To add (`ackit/bnk.py` + a survey tool, like the bank survey of mission 3):
- Parse the **node base parameters** of Sounds, containers, actor-mixers and buses: the effect list (slot,
  effect object ID, and a **"rendered"** flag in this bank version, per wwiser's v56 layout, to confirm on the
  game's banks), the parent ID (effects are inherited unless the child overrides them), the output bus.
- Parse the **effect objects** (HIRC effect / share-set type; the type number for v56 to confirm): plugin ID
  (Audiokinetic Parametric EQ, Delay, RoomVerb, Matrix Reverb, Peak Limiter, Compressor, Expander, Pitch
  Shifter, Flanger, Guitar Distortion, Tremolo, ...; plugin ID = company + type + number) and its **parameters**
  (EQ band frequencies/gains/Q, delay time and feedback, reverb time ...). The `FXPR` chunk the parser already
  recognizes and `ENVS` in `Init.bnk` (environments, i.e. run-time reverb zones) belong here too.
- Walk each line's Sound object up the parent chain to the master bus: list of `(effect, plugin, params,
  where: sound/parent/bus, rendered or run time)`.
- Line identity helps: Sound objects are the same in every language (`banks.md` *Lines*), so a line's chain is
  the same in all 5 languages.

Expected result: run-time effects (bus reverbs, environments) are fully known, with parameters. Rendered
effects are known **with their exact parameters**, which is the best case for a dub. Effects applied in the
studio before the files reached Wwise leave **no metadata at all**: likely many of the radio/PA lines. Those need 3b.

### 3b. Signal analysis of the WEMs (no metadata needed) **(game, ~minutes over all speech)**
`ackit/fxdetect.py` (numpy only), per recording, a small feature vector:
- **Band limits**: frequency below which / above which 1 % of the energy lies (telephone ~300-3,400 Hz), steepness.
- **Effect EQ curve**: long-term average spectrum (LTAS) of the line divided by the LTAS of the **same speaker's
  dry lines** (speaker from `characters.csv` / the line's speaker). The voice cancels out and what is left is the
  effect's frequency response: directly usable as the dub's EQ.
- **Reverb**: energy decay after speech offsets (Schroeder-style decay fit -> RT60 estimate), direct-to-
  reverberant ratio, tail length after the last word.
- **Echo / delay**: peaks in the autocorrelation of the onset envelope, or in the cepstrum, at lags 30-800 ms;
  for multichannel lines the cross-correlation **between channels** (delay and gain per channel).
- **Distortion / noise**: crest factor, clipping, spectral flatness of the gaps between words (static, hiss),
  harmonics added to a narrow band.
- **Modulation / pitch**: envelope modulation 2-20 Hz (tremolo), pitch and formant shift against the speaker's
  dry lines (masks, Freeze, Clayface, synthesized voices).

Then **cluster** the vectors (e.g. k-means on normalized features, or simple thresholds first) and **listen to a
few lines per cluster** (the user names it: "Radio scanner", "Freeze suit", "Strange PA", "Riddler hacked
speakers", "Interview tape", ...). The names and cluster labels go into a profile table (3d). Cross-language
check: the 5 recordings of one line must land in the same cluster; disagreements are lines to listen to.

### 3c. Comparing with the game's output (optional, run-time effects only) **(game)**
The in-game harness can already play sounds and record the loopback (`ackit/verify/ingame.py`, `loopback.py`).
For a sample of lines with a run-time effect, the same features computed on the game's output vs on the WEM
show what the game adds. **Never save raw captures** (CLAUDE.md: they can contain voice chat): compute the
features in memory and keep only the numbers. Useful to confirm, not needed for the dub itself.

### 3d. What gets stored
- `data/catalog/FX.csv`: per line (`line` identity, language-independent where the metadata says so): `profile`,
  `baked` (yes / no / unknown), `source` (metadata / analysis / manual), `confidence`, `runtime` (what the game
  adds, for the "record dry" hint). Built by `tools/build_catalog.py` from the original files only, like the rest.
- `data/catalog/fx_profiles.json`: the profiles (section 4), with where each parameter came from.
- `docs/formats/effects.md` (format of the HIRC effect data, survey figures) and claims in `tools/check_docs.py`
  for every figure; `ackit verify --catalog` re-derives the metadata part.
- Manual labels of the user live in user data (like notes/tags), never in `source/`.

## 4. Re-creating the effects on a dub
An **effect profile** = a chain of simple, deterministic processors with parameters (data, not code), rendered by
`ackit/fx.py`:

| block | parameters | source of the numbers |
|-------|------------|-----------------------|
| high-pass / low-pass / peaking / shelf (biquads) | freq, gain, Q | metadata EQ, or fitted to the 3b EQ curve |
| matched EQ (FIR) | the 3b curve itself | analysis (generic fallback for any baked EQ) |
| delay / echo | time, feedback, mix, per-channel delays/gains | metadata, or 3b autocorrelation / inter-channel lags |
| reverb | RT60, pre-delay, damping, mix | metadata (RoomVerb), or 3b decay fit; convolution with a generated IR (filtered, exponentially decaying noise) keeps it deterministic |
| distortion / bit-crush / noise bed | drive, bits, noise level and colour | 3b, tuned by ear |
| compressor / limiter | threshold, ratio | metadata, or by ear |
| channel layout | per-channel gain + delay (replaces the plain mono copy for 3-5 channel lines) | 3b inter-channel analysis |

The rendered dub then goes through the existing chain: loudness matched to the original, encoded at the
original's rate/channels, fitted to the size limit (`Workspace.prepare`). A reverb or echo tail makes the audio
longer: the editor's length warning and size fitting already cover that, but a dub UI should show the room.

**DSP dependency** (user: "just use some libraries", no own DSP, no VST hosting):
- Effects: **pedalboard** (Spotify, GPL-3.0, compatible with this project's GPL-3.0): ready filters (high/low-
  pass, peak, shelf), reverb, delay, chorus, phaser, distortion, bit-crush, GSM compression (telephone),
  compressor/limiter, pitch shift. A profile is a list of pedalboard plugins with parameters; the per-channel
  layout (gain + delay per channel) is a few numpy lines. To check before stage 1: wheels for the project's Python
  (3.14) on Windows and Linux, and that PyInstaller bundles it.
- Analysis (3b): numpy (FFT, correlations), scipy only if a step really needs it.
- Decision goes into `docs/decisions.md` like the other dependencies, licenses into `THIRD_PARTY_LICENSES.txt`.

## 5. Where to build it: new app or the existing editor?

| | Separate dubbing app on `ackit` | Inside Unheard City (new **Dubbing** tab) |
|-|-------------------------------|------------------------------------------|
| Write safety | `Workspace.lock` is a `threading.RLock`: it serializes writes **inside one process** only. Two apps open on the same game and store can write one package at the same time (the lost-edit bug of 2026-10-13, across processes). Needs a cross-process file lock + "changed outside" handling in both apps. | Same process, same lock, same store, same changed-outside banner: nothing new. |
| Reuse | Re-implements catalog loading (background jobs, spinners), language setting, line identity, subtitles, waveform, player/recorder, encode/fit, apply/revert, mod export/import, settings, packaging. | All of that exists (`app/editor.py`, `audioio.py`, `waveform.py`, `textpanel.py`, `modpack.py`). |
| Users | Two programs, two installs, two settings files, two originals stores to keep apart. | One program; a dub is just another kind of change (and a mod package). |
| Complexity | The editor stays as it is. | `mainwindow.py` (774 lines) and `editor.py` (590) must not grow into one blob: the tab is its own package. |
| UX | Free to design a session-based workflow. | Same freedom inside a tab; the Sounds tab stays for one-off replacements. |

**Recommendation: build it into Unheard City, with the engine in `ackit`.**
- `ackit/fx.py` (profiles + rendering), `ackit/fxdetect.py` (features, clustering), `ackit/dub.py` (dub
  sessions/takes: files, states; no Qt), CLI `ackit fx survey | scan | render` and `ackit dub ...`: all headless,
  testable, usable from scripts.
- `app/dubbing/` (own package): the **Dubbing** tab.
- The **Sounds** editor gets one small row: *Effect: [profile ▾] (detected: Radio EQ, metadata)* and a *game
  adds ... record dry* hint, so single replacements benefit too.
- A separate program makes sense only later, if voice actors record **without the game** (remote dubbing): a
  small recorder that reads a script export and writes a take pack (WAVs + JSON), which Unheard City imports.
  It never touches game files, so the lock problem does not arise.

## 6. The Dubbing tab (workflow)
1. **Session**: language + character (or a filter: category, a list of lines). Lists the character's lines via the
   line identity; shows the script (subtitle of the loaded language, the INT text as reference), the original's
   length, the effect profile and baked/run-time status, and a state per line: *to do / recorded / approved /
   applied*. Sessions and takes are stored in user data (`%APPDATA%\UnheardCity\dubs\...`), never in the game.
2. **Record**: play the original as a cue, count-in, record, auto-trim silence, several takes per line, pick one;
   length vs the original (and optional time-stretch to fit, if pedalboard or a WSOLA stretch is added).
3. **Process**: clean-up (high-pass, gate), loudness match to the original (exists), the effect profile
   (suggested from FX.csv, editable, A/B against the original with and without the effect).
4. **Apply** the approved lines in one `Workspace.apply` call (it takes a list); **export** the session as a mod
   package (`modpack.export_mod`, with edited subtitles if the dub's wording changed, within their room).
5. Non-destructive: raw takes + parameters are kept, so a changed profile re-renders every line.

Warnings the tab must show (from the existing data):
- Lines the game shares between languages (167 in-memory voice lines stored once for every language; 616-651 WAD
  lines per language identical to INT, `banks.md` *Lines*): dubbing one changes it for every language that uses it.
- Size limit (speech in place, never larger than the original WEM): long takes plus reverb tails get squeezed or
  refused; show the room.
- Subtitle room (NUL-padded in place).
- Lines without a speaker label or with a disputed one (19 listed in `categories.md`).

## 7. Stages (one commit each: code + tests + docs + NOTES status)
0. **Evidence (game)**: HIRC effect survey (3a) and a first feature scan (3b) over all speech; listen to samples
   per cluster with the user; write `docs/formats/effects.md` with figures (+ `check_docs` claims). Outcome: how
   many lines are baked vs run time, and the list of profiles. **This decides the size of everything after it.**
1. `ackit/fx.py` on pedalboard: profiles JSON, render; tests with synthetic signals (an EQ's
   measured response, a delay's lag, an RT60 measured back from the rendered reverb).
2. `ackit/fxdetect.py` + `FX.csv` / `fx_profiles.json` in `build_catalog.py`; `verify --catalog` checks; CLI.
   Test: render a profile onto a dry synthetic voice, detect it back.
3. Sounds editor: effect row + run-time hint; GUI tests.
4. `ackit/dub.py` + Dubbing tab (sessions, takes, script, batch apply, mod export); GUI tests.
5. Real-game round trips (`tools/e2e_features.py` / `e2e_editor.py` cases for a dubbed line with an effect,
   reverted in `finally`), the user listens in game, packaging (`build_app.py`, licenses), version 3.4.0 or 4.0.0.

## 8. Decisions (user, 2026-10-27)
- **All characters** can be dubbed, from the existing catalogs (no per-character priority): every effect group
  found in stage 0 needs a profile.
- **Approximate effects** are enough; profiles are tuned later by ear (the Riddler lines' effect is complex).
- **Libraries, not own DSP**: pedalboard for effects (section 4); no VST hosting, no time stretching for now.
- **Everyone has the game**: no separate recorder or take packs; everything lives in Unheard City (section 5).
- Still open, out of scope unless asked: audio inside video files (pre-rendered movies carry their own sound
  track, not in the WADs or banks; to check whether the game has any with speech).

## 9. Stage 0 results (2026-10-27, INT, read-only)
Measured with scratch scripts (HIRC survey of every bank, signal features of 12,430 INT speech recordings),
then the user listened to 77 candidate groups (`external/fx_samples/INDEX.md`, git-ignored). Figures here are a
record of that day, not re-checked by `tools/check_docs.py`.

**Metadata (Wwise, v56).** Layouts from wwiser: node base params = u8 override-parent-fx, u8 count, [u8 bypass,
count x {u8 slot, u32 fx id, u8 share-set, u8 rendered}], u32 output bus, u32 parent; a Sound has them after its
source (+ u8 source bits); a bus (v56) has u8 is-environmental at payload offset 33, ducks of 17 bytes, then its
effects. Effect objects are HIRC types 0x12 (share-set) / 0x13 (custom): u32 plugin, u32 size, parameters.
- Run time (the game adds them; the files are clean, so a dub is recorded **dry**): McDSP FutzBox share-set
  `4272216A` (in 1,125 speech banks, on 464 actor-mixer slots, RTPC-driven) -> 2,858 speech rows; Parametric EQ
  on a bus (30 rows) or on the sound (3); the ML1 limiter on the top bus (everything); 84 RoomVerb + 6 Matrix
  Reverb share-sets in `Init` = the environments the game switches per area.
- Rendered (baked by Wwise): 33 speech rows (car radio 25, radio scanner 3, museum thug-to-Penguin 4, Steel HQ
  1); the slot's fx id is zeroed, so the effect type is lost.
- Everything else that is baked was made in the studio and has no metadata.

**Channels.** Speech WEMs with 2-5 channels carry the WAVE channel mask: 2 ch `0x3` (L R), 3 ch `0x7` (L R C),
4 ch `0x33` (L R SL SR), 5 ch `0x37` (L R C SL SR); ackit decodes and encodes them in that order. PA lines put
the dry voice in C and an echo 160-173 ms later in the surrounds (Harley/HQ 121 ms; Freeze's quad rears 73 ms);
Riddler, Ivy and Ra's 5-channel lines are processed differently per channel.

**What the user heard** (per group; "no effects" groups omitted: thug/Ninja/Catwoman/JR efforts, combat barks,
Riddler `RIDDLER_R`, DLC `PDLC_PDLC`/`PDLC_EMOTES`/Robin/Harley, Outcast, Penguin `MUSEUM_B`, Steel/Church Harley,
Bully hobo, Talia `UNDER_B`, `CH_ZERO` Strange, Strange/Black Mask game-over):

| effect heard | groups |
|--------------|--------|
| radio EQ | radio scanner, Alfred/Oracle/Penguin/Strange/Ra's comms, copter, cameras, Gordon radio, car radio |
| phone EQ | Penguin PRED, Zsasz calls, Two-Face court, Joker (PRED, theatre), Strange/Mayor/Harley PA, Riddler (Enigma, cameos), Vicki Vale radio scan, Penguin exhibits |
| phone-mic EQ / TV-speaker EQ | interview tapes (Vicki Vale, Joker), Strange on TV, Joker church message, `UNDER_C` TV |
| radio noise | copter, car radio, radio scanner (Vicki), Vicki Vale radio scan, map radio |
| distortion (slight, speaker) | Penguin exhibits, radio scanner (Vicki), `UNDER_C` TV |
| echo (PA delay) | Strange/Mayor/Harley/Joker/Two-Face PA and court lines, Catwoman side mission (Strange) |
| very short delay (width) | Riddler cameos, radio scanner (Riddler) |
| reverb (small/short room, slight wide room) | Riddler PRED, Harley 5 ch, Strange TV, Joker church, Talia sequence, Harley DLC, game-over lines (barely) |
| reverse reverb (occasional) | Riddler (PRED, cameos, radio scanner), Poison Ivy |
| glitches (parts repeat at another pitch), small repeats | Riddler (PRED, cameos, Enigma, radio scanner) |
| pitch shift, fixed, down | Clayface |
| pitch shift over time + robot + pan over time | Mr. Freeze |
| octaves overlaid (down), formant shift, stereo width | Ra's al Ghul (giant, trials), Poison Ivy |
| formant shift, unintelligible voice | car radio |

## 10. Scope after stage 0 (user, 2026-10-27)
Matching effects automatically per line is too complex: the Dubbing section gets **manual controls for every
effect found** (section 9), which the user sets by ear. Dropped: automatic profiles, `FX.csv` /
`fx_profiles.json`, detection in the catalog build (stage 2 of section 7). Kept: pedalboard, the Dubbing tab in
Unheard City, everything else of sections 5-6.

Effects rack (`ackit/fx.py`, one table of blocks and parameters that the UI is generated from; every block off
by default; fixed order; seeded randomness, so a setting always renders the same audio):

| # | block | controls | built on |
|---|-------|----------|----------|
| 1 | Pitch | shift (semitones), formant shift, drift depth + rate (pitch over time) | `pedalboard.time_stretch` (pitch curve, formants) |
| 2 | Octave layers | two layers: interval + level | `time_stretch` pitch, mixed |
| 3 | Robot | ring modulator frequency + mix, metallic comb (delay, feedback) | numpy |
| 4 | Glitch | events per 10 s, length, repeats, pitch of the repeats, seed | numpy + `time_stretch` |
| 5 | Voice band (EQ) | low cut, high cut, slope, mid boost (freq, gain, Q) | `HighpassFilter`, `LowpassFilter`, `PeakFilter` |
| 6 | Distortion | drive, clip, bit depth, phone codec (GSM) | `Distortion`, `Clipping`, `Bitcrush`, `GSMFullRateCompressor` |
| 7 | Noise | level, colour (band), crackle | numpy |
| 8 | Echo | time, feedback, mix | `Delay` |
| 9 | Reverb | room size, damping, wet, width; reverse reverb amount + how often | `Reverb` (reversed for the reverse swell) |
| 10 | Channels (2-5 ch lines) | width (L/R delay), auto-pan rate + depth, surround echo delay + level, dry voice in C | numpy, channel order of section 9 |

Then the existing chain: loudness match / normalize / gain, encode within the size limit.

Stages from here (one commit each; 1 and 2 done on 2026-10-27, see NOTES.md):
1. `ackit/fx.py` + tests (synthetic signals: each block measured back) + pedalboard dependency
   (`pyproject.toml`, `docs/decisions.md`, licenses).
2. Dubbing tab (`app/dubbing/`): line list (character/category/search filters, state per line), the existing
   sound editor embedded (record/load, trim, A/B, subtitle, apply/revert), the effects rack between the take and
   the loudness step, settings saved per line in user data; GUI tests; user guide.
3. Real-game round trip of a dubbed line with effects, packaging (PyInstaller + pedalboard), version bump.

## 11. UI revision (user, 2026-10-27, after trying stages 1-2)
- The subtitle box was too small for dubbing, and the Sounds tab and the Dubbing tab both had an editor:
  browsing and editing are now separate. **Sounds** only browses (list + a compact preview with **Edit** / **Add
  to edit list**); **Dubbing** became **Edit**, for every kind of sound, with an edit list instead of a second
  browser, the subtitle as a large **script**, and Previous / Next.
- The effects panel was "raw": sliders with readable values (log scale for frequencies and times), **looks**
  for the voices heard in the game (section 9) and **preset buttons** per effect that set the sliders in place.
- The reverb (pedalboard's Freeverb) sounded poor: replaced by a generated room response (decay, pre-delay,
  reflections, damping, width) with presets (small room to church, tiled cell, outdoors).
- Second round: the browser's player keeps Play/Stop together (editing actions apart on the right); the Edit tab
  fits the window without scrolling (effects as a chain with one settings area); the app and docs use generic
  terms (replacement, effects, subtitle), since the Edit tab is for every kind of sound, not only dubbing.