UnheardCity/docs/history/mission4_dubbing_plan.md

Plan: dubbing voice lines, with the game's voice effects

Status (2026-10-27): stage 0 done (section 9); the scope changed after it (section 10: manual effect controls, no automatic profiles); stages 1-2 of section 10 done (effects engine, Dubbing tab). The user's answers are in section 8. Sections 1-7 are the original proposal, written from the code, the docs and the shipped catalog CSVs; the steps marked (game) read the game files. The game is also installed on the Linux machine (Steam Flatpak, mounted into the podman container at /home/gora/.var/app/com.valvesoftware.Steam/.local/share/Steam/steamapps/common/Batman Arkham City GOTY; the container's HOME differs, so it is passed with --game), so the read-only stages can run there; in-game checks still need the Windows PC. Sections 3-7 name modules and files that were proposed but not built after the scope change (ackit/fxdetect.py, ackit/dub.py, data/catalog/FX.csv, fx_profiles.json, docs/formats/effects.md); what was built is in sections 10-11 and NOTES.md: ackit/fx.py, app/edit.py, app/editor.py, app/effects.py, app/preview.py.

1. The problem

A dub replaces a character's lines with new recordings. Some original lines are not dry: they sound like a telephone or radio (band-limited EQ), a PA announcement over the city (echo, several speakers), a room (reverb), a suit or a machine (Mr. Freeze). A dry dub next to them sounds wrong. Two questions:

1. Where is the effect applied? This decides everything:

   • Baked into the WEM (applied in the studio, or by Wwise when the bank was built, rendered effect): the dry dub loses it, so the dub must get the effect before it is written.
   • Applied by the game at run time (a Wwise effect on the Sound, on a parent container or on a bus, an environmental/reverb zone, attenuation, obstruction): the game also applies it to the replacement. The dub must stay dry, or it gets the effect twice.

   The user heard effects "in game and here": the ones heard here (the editor plays the WEM) are baked; the ones heard only in game are run-time and need nothing.

2. Which effects exist? Unknown, so they have to be found in the game data (section 3).

2. What the catalog already shows (INT, measured on the shipped data/catalog/INT.csv)

Cheap signals that are already in the catalog (rate, channels) and point to baked effects:

signal	lines	groups (examples)	likely effect
3 or more channels	858 WAD speech rows (mostly 5 ch, then 4 and 3 ch), plus 86 in-memory 5 ch	Predator chatter 251, story 211, Riddler 90: PRED_RIDDLER, PRED_J, PRED_STRANGE, PRED_MAYOR, RIDDLER_R, GAME_OVER, COURT_B, UNDER_S	PA / hacked-speaker broadcasts: one voice spread over several channels, probably with delays (echo) between them
32,000 Hz	110 (102 are CHAR_FREEZE)	Mr. Freeze	suit voice filter
11,025 / 12,000 Hz	91	CHR_RadioScanner_*	police radio (band-limited)
6,000 Hz	25	PSO_CarRadio_Chatter_*	car radio
category Radio & TV (Speech_RADIO, Speech_TV, Speech_CAMERAS, DIA_)	see categories.md	Vicki Vale reports, TV, cameras	radio/TV EQ, maybe noise

These are hypotheses until someone listens and the metadata scan (3a) confirms them. Two things follow already:

• A mono dub of a 5-channel line is copied to every channel today (audio.to_channels): the spatial echo between the channels is lost. Multichannel lines need a per-channel layout (gain + delay per channel) in their effect.
• Low-rate lines limit the band themselves (6 kHz = nothing above 3 kHz), so part of the "radio" sound is just the encode rate; the rest (low cut, mid boost, distortion, noise) has to be added.

3. Finding every effect the game uses

Three independent sources, from exact to fuzzy. Every result goes into the catalog, so the editor can say for each line: baked: Radio EQ (from metadata), baked?: echo, 3 channels (analysis, 0.8), game adds: room reverb (run time, record dry).

3a. Wwise metadata (exact, where it exists) (game)

The banks (version 56, Wwise 2011.1) describe effects in HIRC; ackit/bnk.py reads only the Sound sources today. To add (ackit/bnk.py + a survey tool, like the bank survey of mission 3):

• Parse the node base parameters of Sounds, containers, actor-mixers and buses: the effect list (slot, effect object ID, and a "rendered" flag in this bank version, per wwiser's v56 layout, to confirm on the game's banks), the parent ID (effects are inherited unless the child overrides them), the output bus.
• Parse the effect objects (HIRC effect / share-set type; the type number for v56 to confirm): plugin ID (Audiokinetic Parametric EQ, Delay, RoomVerb, Matrix Reverb, Peak Limiter, Compressor, Expander, Pitch Shifter, Flanger, Guitar Distortion, Tremolo, ...; plugin ID = company + type + number) and its parameters (EQ band frequencies/gains/Q, delay time and feedback, reverb time ...). The FXPR chunk the parser already recognizes and ENVS in Init.bnk (environments, i.e. run-time reverb zones) belong here too.
• Walk each line's Sound object up the parent chain to the master bus: list of (effect, plugin, params, where: sound/parent/bus, rendered or run time).
• Line identity helps: Sound objects are the same in every language (banks.md Lines), so a line's chain is the same in all 5 languages.

Expected result: run-time effects (bus reverbs, environments) are fully known, with parameters. Rendered effects are known with their exact parameters, which is the best case for a dub. Effects applied in the studio before the files reached Wwise leave no metadata at all: likely many of the radio/PA lines. Those need 3b.

3b. Signal analysis of the WEMs (no metadata needed) (game, ~minutes over all speech)

ackit/fxdetect.py (numpy only), per recording, a small feature vector:

• Band limits: frequency below which / above which 1 % of the energy lies (telephone ~300-3,400 Hz), steepness.
• Effect EQ curve: long-term average spectrum (LTAS) of the line divided by the LTAS of the same speaker's dry lines (speaker from characters.csv / the line's speaker). The voice cancels out and what is left is the effect's frequency response: directly usable as the dub's EQ.
• Reverb: energy decay after speech offsets (Schroeder-style decay fit -> RT60 estimate), direct-to- reverberant ratio, tail length after the last word.
• Echo / delay: peaks in the autocorrelation of the onset envelope, or in the cepstrum, at lags 30-800 ms; for multichannel lines the cross-correlation between channels (delay and gain per channel).
• Distortion / noise: crest factor, clipping, spectral flatness of the gaps between words (static, hiss), harmonics added to a narrow band.
• Modulation / pitch: envelope modulation 2-20 Hz (tremolo), pitch and formant shift against the speaker's dry lines (masks, Freeze, Clayface, synthesized voices).

Then cluster the vectors (e.g. k-means on normalized features, or simple thresholds first) and listen to a few lines per cluster (the user names it: "Radio scanner", "Freeze suit", "Strange PA", "Riddler hacked speakers", "Interview tape", ...). The names and cluster labels go into a profile table (3d). Cross-language check: the 5 recordings of one line must land in the same cluster; disagreements are lines to listen to.

3c. Comparing with the game's output (optional, run-time effects only) (game)

The in-game harness can already play sounds and record the loopback (ackit/verify/ingame.py, loopback.py). For a sample of lines with a run-time effect, the same features computed on the game's output vs on the WEM show what the game adds. Never save raw captures (CLAUDE.md: they can contain voice chat): compute the features in memory and keep only the numbers. Useful to confirm, not needed for the dub itself.

3d. What gets stored

• data/catalog/FX.csv: per line (line identity, language-independent where the metadata says so): profile, baked (yes / no / unknown), source (metadata / analysis / manual), confidence, runtime (what the game adds, for the "record dry" hint). Built by tools/build_catalog.py from the original files only, like the rest.
• data/catalog/fx_profiles.json: the profiles (section 4), with where each parameter came from.
• docs/formats/effects.md (format of the HIRC effect data, survey figures) and claims in tools/check_docs.py for every figure; ackit verify --catalog re-derives the metadata part.
• Manual labels of the user live in user data (like notes/tags), never in source/.

4. Re-creating the effects on a dub

An effect profile = a chain of simple, deterministic processors with parameters (data, not code), rendered by ackit/fx.py:

block	parameters	source of the numbers
high-pass / low-pass / peaking / shelf (biquads)	freq, gain, Q	metadata EQ, or fitted to the 3b EQ curve
matched EQ (FIR)	the 3b curve itself	analysis (generic fallback for any baked EQ)
delay / echo	time, feedback, mix, per-channel delays/gains	metadata, or 3b autocorrelation / inter-channel lags
reverb	RT60, pre-delay, damping, mix	metadata (RoomVerb), or 3b decay fit; convolution with a generated IR (filtered, exponentially decaying noise) keeps it deterministic
distortion / bit-crush / noise bed	drive, bits, noise level and colour	3b, tuned by ear
compressor / limiter	threshold, ratio	metadata, or by ear
channel layout	per-channel gain + delay (replaces the plain mono copy for 3-5 channel lines)	3b inter-channel analysis

The rendered dub then goes through the existing chain: loudness matched to the original, encoded at the original's rate/channels, fitted to the size limit (Workspace.prepare). A reverb or echo tail makes the audio longer: the editor's length warning and size fitting already cover that, but a dub UI should show the room.

DSP dependency (user: "just use some libraries", no own DSP, no VST hosting):

• Effects: pedalboard (Spotify, GPL-3.0, compatible with this project's GPL-3.0): ready filters (high/low- pass, peak, shelf), reverb, delay, chorus, phaser, distortion, bit-crush, GSM compression (telephone), compressor/limiter, pitch shift. A profile is a list of pedalboard plugins with parameters; the per-channel layout (gain + delay per channel) is a few numpy lines. To check before stage 1: wheels for the project's Python (3.14) on Windows and Linux, and that PyInstaller bundles it.
• Analysis (3b): numpy (FFT, correlations), scipy only if a step really needs it.
• Decision goes into docs/decisions.md like the other dependencies, licenses into THIRD_PARTY_LICENSES.txt.

5. Where to build it: new app or the existing editor?

	Separate dubbing app on ackit	Inside Unheard City (new Dubbing tab)
Write safety	Workspace.lock is a threading.RLock: it serializes writes inside one process only. Two apps open on the same game and store can write one package at the same time (the lost-edit bug of 2026-10-13, across processes). Needs a cross-process file lock + "changed outside" handling in both apps.	Same process, same lock, same store, same changed-outside banner: nothing new.
Reuse	Re-implements catalog loading (background jobs, spinners), language setting, line identity, subtitles, waveform, player/recorder, encode/fit, apply/revert, mod export/import, settings, packaging.	All of that exists (app/editor.py, audioio.py, waveform.py, textpanel.py, modpack.py).
Users	Two programs, two installs, two settings files, two originals stores to keep apart.	One program; a dub is just another kind of change (and a mod package).
Complexity	The editor stays as it is.	mainwindow.py (774 lines) and editor.py (590) must not grow into one blob: the tab is its own package.
UX	Free to design a session-based workflow.	Same freedom inside a tab; the Sounds tab stays for one-off replacements.

Recommendation: build it into Unheard City, with the engine in ackit.

• ackit/fx.py (profiles + rendering), ackit/fxdetect.py (features, clustering), ackit/dub.py (dub sessions/takes: files, states; no Qt), CLI ackit fx survey | scan | render and ackit dub ...: all headless, testable, usable from scripts.
• app/dubbing/ (own package): the Dubbing tab.
• The Sounds editor gets one small row: Effect: [profile ▾] (detected: Radio EQ, metadata) and a game adds ... record dry hint, so single replacements benefit too.
• A separate program makes sense only later, if voice actors record without the game (remote dubbing): a small recorder that reads a script export and writes a take pack (WAVs + JSON), which Unheard City imports. It never touches game files, so the lock problem does not arise.

6. The Dubbing tab (workflow)

1. Session: language + character (or a filter: category, a list of lines). Lists the character's lines via the line identity; shows the script (subtitle of the loaded language, the INT text as reference), the original's length, the effect profile and baked/run-time status, and a state per line: to do / recorded / approved / applied. Sessions and takes are stored in user data (%APPDATA%\UnheardCity\dubs\...), never in the game.
2. Record: play the original as a cue, count-in, record, auto-trim silence, several takes per line, pick one; length vs the original (and optional time-stretch to fit, if pedalboard or a WSOLA stretch is added).
3. Process: clean-up (high-pass, gate), loudness match to the original (exists), the effect profile (suggested from FX.csv, editable, A/B against the original with and without the effect).
4. Apply the approved lines in one Workspace.apply call (it takes a list); export the session as a mod package (modpack.export_mod, with edited subtitles if the dub's wording changed, within their room).
5. Non-destructive: raw takes + parameters are kept, so a changed profile re-renders every line.

Warnings the tab must show (from the existing data):

• Lines the game shares between languages (167 in-memory voice lines stored once for every language; 616-651 WAD lines per language identical to INT, banks.md Lines): dubbing one changes it for every language that uses it.
• Size limit (speech in place, never larger than the original WEM): long takes plus reverb tails get squeezed or refused; show the room.
• Subtitle room (NUL-padded in place).
• Lines without a speaker label or with a disputed one (19 listed in categories.md).

7. Stages (one commit each: code + tests + docs + NOTES status)

0. Evidence (game): HIRC effect survey (3a) and a first feature scan (3b) over all speech; listen to samples per cluster with the user; write docs/formats/effects.md with figures (+ check_docs claims). Outcome: how many lines are baked vs run time, and the list of profiles. This decides the size of everything after it.
1. ackit/fx.py on pedalboard: profiles JSON, render; tests with synthetic signals (an EQ's measured response, a delay's lag, an RT60 measured back from the rendered reverb).
2. ackit/fxdetect.py + FX.csv / fx_profiles.json in build_catalog.py; verify --catalog checks; CLI. Test: render a profile onto a dry synthetic voice, detect it back.
3. Sounds editor: effect row + run-time hint; GUI tests.
4. ackit/dub.py + Dubbing tab (sessions, takes, script, batch apply, mod export); GUI tests.
5. Real-game round trips (tools/e2e_features.py / e2e_editor.py cases for a dubbed line with an effect, reverted in finally), the user listens in game, packaging (build_app.py, licenses), version 3.4.0 or 4.0.0.

8. Decisions (user, 2026-10-27)

• All characters can be dubbed, from the existing catalogs (no per-character priority): every effect group found in stage 0 needs a profile.
• Approximate effects are enough; profiles are tuned later by ear (the Riddler lines' effect is complex).
• Libraries, not own DSP: pedalboard for effects (section 4); no VST hosting, no time stretching for now.
• Everyone has the game: no separate recorder or take packs; everything lives in Unheard City (section 5).
• Still open, out of scope unless asked: audio inside video files (pre-rendered movies carry their own sound track, not in the WADs or banks; to check whether the game has any with speech).

9. Stage 0 results (2026-10-27, INT, read-only)

Measured with scratch scripts (HIRC survey of every bank, signal features of 12,430 INT speech recordings), then the user listened to 77 candidate groups (external/fx_samples/INDEX.md, git-ignored). Figures here are a record of that day, not re-checked by tools/check_docs.py.

Metadata (Wwise, v56). Layouts from wwiser: node base params = u8 override-parent-fx, u8 count, [u8 bypass, count x {u8 slot, u32 fx id, u8 share-set, u8 rendered}], u32 output bus, u32 parent; a Sound has them after its source (+ u8 source bits); a bus (v56) has u8 is-environmental at payload offset 33, ducks of 17 bytes, then its effects. Effect objects are HIRC types 0x12 (share-set) / 0x13 (custom): u32 plugin, u32 size, parameters.

• Run time (the game adds them; the files are clean, so a dub is recorded dry): McDSP FutzBox share-set 4272216A (in 1,125 speech banks, on 464 actor-mixer slots, RTPC-driven) -> 2,858 speech rows; Parametric EQ on a bus (30 rows) or on the sound (3); the ML1 limiter on the top bus (everything); 84 RoomVerb + 6 Matrix Reverb share-sets in Init = the environments the game switches per area.
• Rendered (baked by Wwise): 33 speech rows (car radio 25, radio scanner 3, museum thug-to-Penguin 4, Steel HQ 1); the slot's fx id is zeroed, so the effect type is lost.
• Everything else that is baked was made in the studio and has no metadata.

Channels. Speech WEMs with 2-5 channels carry the WAVE channel mask: 2 ch 0x3 (L R), 3 ch 0x7 (L R C), 4 ch 0x33 (L R SL SR), 5 ch 0x37 (L R C SL SR); ackit decodes and encodes them in that order. PA lines put the dry voice in C and an echo 160-173 ms later in the surrounds (Harley/HQ 121 ms; Freeze's quad rears 73 ms); Riddler, Ivy and Ra's 5-channel lines are processed differently per channel.

What the user heard (per group; "no effects" groups omitted: thug/Ninja/Catwoman/JR efforts, combat barks, Riddler RIDDLER_R, DLC PDLC_PDLC/PDLC_EMOTES/Robin/Harley, Outcast, Penguin MUSEUM_B, Steel/Church Harley, Bully hobo, Talia UNDER_B, CH_ZERO Strange, Strange/Black Mask game-over):

effect heard	groups
radio EQ	radio scanner, Alfred/Oracle/Penguin/Strange/Ra's comms, copter, cameras, Gordon radio, car radio
phone EQ	Penguin PRED, Zsasz calls, Two-Face court, Joker (PRED, theatre), Strange/Mayor/Harley PA, Riddler (Enigma, cameos), Vicki Vale radio scan, Penguin exhibits
phone-mic EQ / TV-speaker EQ	interview tapes (Vicki Vale, Joker), Strange on TV, Joker church message, UNDER_C TV
radio noise	copter, car radio, radio scanner (Vicki), Vicki Vale radio scan, map radio
distortion (slight, speaker)	Penguin exhibits, radio scanner (Vicki), UNDER_C TV
echo (PA delay)	Strange/Mayor/Harley/Joker/Two-Face PA and court lines, Catwoman side mission (Strange)
very short delay (width)	Riddler cameos, radio scanner (Riddler)
reverb (small/short room, slight wide room)	Riddler PRED, Harley 5 ch, Strange TV, Joker church, Talia sequence, Harley DLC, game-over lines (barely)
reverse reverb (occasional)	Riddler (PRED, cameos, radio scanner), Poison Ivy
glitches (parts repeat at another pitch), small repeats	Riddler (PRED, cameos, Enigma, radio scanner)
pitch shift, fixed, down	Clayface
pitch shift over time + robot + pan over time	Mr. Freeze
octaves overlaid (down), formant shift, stereo width	Ra's al Ghul (giant, trials), Poison Ivy
formant shift, unintelligible voice	car radio

10. Scope after stage 0 (user, 2026-10-27)

Matching effects automatically per line is too complex: the Dubbing section gets manual controls for every effect found (section 9), which the user sets by ear. Dropped: automatic profiles, FX.csv / fx_profiles.json, detection in the catalog build (stage 2 of section 7). Kept: pedalboard, the Dubbing tab in Unheard City, everything else of sections 5-6.

Effects rack (ackit/fx.py, one table of blocks and parameters that the UI is generated from; every block off by default; fixed order; seeded randomness, so a setting always renders the same audio):

#	block	controls	built on
1	Pitch	shift (semitones), formant shift, drift depth + rate (pitch over time)	pedalboard.time_stretch (pitch curve, formants)
2	Octave layers	two layers: interval + level	time_stretch pitch, mixed
3	Robot	ring modulator frequency + mix, metallic comb (delay, feedback)	numpy
4	Glitch	events per 10 s, length, repeats, pitch of the repeats, seed	numpy + time_stretch
5	Voice band (EQ)	low cut, high cut, slope, mid boost (freq, gain, Q)	HighpassFilter, LowpassFilter, PeakFilter
6	Distortion	drive, clip, bit depth, phone codec (GSM)	Distortion, Clipping, Bitcrush, GSMFullRateCompressor
7	Noise	level, colour (band), crackle	numpy
8	Echo	time, feedback, mix	Delay
9	Reverb	room size, damping, wet, width; reverse reverb amount + how often	Reverb (reversed for the reverse swell)
10	Channels (2-5 ch lines)	width (L/R delay), auto-pan rate + depth, surround echo delay + level, dry voice in C	numpy, channel order of section 9

Then the existing chain: loudness match / normalize / gain, encode within the size limit.

Stages from here (one commit each; 1 and 2 done on 2026-10-27, see NOTES.md):

1. ackit/fx.py + tests (synthetic signals: each block measured back) + pedalboard dependency (pyproject.toml, docs/decisions.md, licenses).
2. Dubbing tab (app/dubbing/): line list (character/category/search filters, state per line), the existing sound editor embedded (record/load, trim, A/B, subtitle, apply/revert), the effects rack between the take and the loudness step, settings saved per line in user data; GUI tests; user guide.
3. Real-game round trip of a dubbed line with effects, packaging (PyInstaller + pedalboard), version bump.

11. UI revision (user, 2026-10-27, after trying stages 1-2)

• The subtitle box was too small for dubbing, and the Sounds tab and the Dubbing tab both had an editor: browsing and editing are now separate. Sounds only browses (list + a compact preview with Edit / Add to edit list); Dubbing became Edit, for every kind of sound, with an edit list instead of a second browser, the subtitle as a large script, and Previous / Next.
• The effects panel was "raw": sliders with readable values (log scale for frequencies and times), looks for the voices heard in the game (section 9) and preset buttons per effect that set the sliders in place.
• The reverb (pedalboard's Freeverb) sounded poor: replaced by a generated room response (decay, pre-delay, reflections, damping, width) with presets (small room to church, tiled cell, outdoors).
• Second round: the browser's player keeps Play/Stop together (editing actions apart on the right); the Edit tab fits the window without scrolling (effects as a chain with one settings area); the app and docs use generic terms (replacement, effects, subtitle), since the Edit tab is for every kind of sound, not only dubbing.