Initial commit

docs/history/mission1_prompt.md

@@ -0,0 +1,92 @@
+# MISSION: Arkham City audio modding toolchain (autonomous)
+
+You are working fully autonomously. I will not answer questions or intervene. When something is ambiguous, pick the most reasonable option, record the assumption in `NOTES.md`, and continue. Never stop to ask me for permission or clarification. Stop only when a success criterion below is met, when a stop condition is hit, or when usage limits force a pause (see "Limits").
+
+## Goal
+
+Make it possible to replace audio in **Batman: Arkham City (PC)** with custom audio in a simple, repeatable way. Deliverables:
+
+1. **Encoder tool**: converts an arbitrary WAV/OGG/MP3 into a WEM file the game actually accepts and plays (correct RIFF/Wwise chunks, Vorbis stream and codebooks as the game's Wwise version expects).
+2. **Injector tool**: replaces a given WEM inside the game's WAD container, tracking file sizes and alignment, and updating all offset/size tables so the container stays valid. Must be reversible (restore from backup).
+3. **A short README** documenting the WAD structure, WEM format findings, and usage, so a non-expert can replace a sound with one command.
+
+Success = a custom sound I supply plays correctly in-game in place of the original, with no crash, verified by your harness (see "Verification").
+
+## What is already known (do not redo this work)
+
+- Game was built with **Audiokinetic Wwise 2011.1.0**, which is unavailable. The closest available is **2011.2.2** (Internet Archive). Audio is a custom Wwise RIFF (`RIFF/WAVE`, `fmt`, `data`, etc.) containing Vorbis with **stripped headers and packed codebooks**.
+- Differences between 2011.1.0 and 2011.2.x exist, including **which Vorbis codebooks the encoder uses**. Header layout differences are also possible and not fully mapped.
+- Existing public tools (unpackers/repackers) work for extracting and repacking, but the resulting audio is incompatible: the game refuses to play it or crashes. Alternative codecs (PCM, etc.) were already tried and do not work.
+- I already made a **patch that stops the crash**: a code cave that patches the variable holding the number of bytes to allocate on a **bump allocator**. Re-derive details from my patch/notes in the repo if present; do not assume it is complete.
+- The game executable has **DRM/packing**: many functions are missing when the original exe is analyzed statically. A **WinDbg memory dump of the running executable** exposes them. Use the dumped exe for Ghidra analysis; use the live process for dynamic analysis.
+- Useful references: **vgmstream** and **ww2ogg** (Wwise RIFF parsing, packed codebook handling, codebook files such as `packed_codebooks.bin` / aoTuV variants), **revorb**, and Wwise 2011.2.2 itself as a reference encoder.
+
+Search the filesystem first for my existing patches, dumps, notes, and tools. Treat anything you find as input data, not as ground truth; verify before relying on it.
+
+## Two approaches (pursue in this order, switch when evidence says so)
+
+**A. Make the files compatible with the game.** Reverse engineer the game's WEM loader/Vorbis decoder to find exactly what it requires (header fields, codebook set/IDs, setup-packet format, packet/granule layout, block sizes, seek table, etc.). Then build an encoder that produces that format, likely by encoding with Wwise 2011.2.2 or libvorbis and rewriting headers/codebooks to match the game's expectations. Compare original game WEMs against Wwise 2011.2.2 output to find the differences.
+
+**B. Patch the game binary** to accept more encodings (for example, make the decoder accept the 2011.2.x codebooks or headers, or add a path that supports another codec). Use this if A proves infeasible, or as a complement if it is simpler.
+
+Decide by evidence. Document the reasoning in `NOTES.md`.
+
+## Environment and tools
+
+- Native Windows. Work in a git repo; commit often with meaningful messages.
+- Ghidra via MCP (static analysis, on the dumped exe).
+- **Steam-only game.** The game is installed through **Steam** and can only be run through Steam (Steam DRM/ownership check). Steam must be running and logged in. Launching the game exe directly may fail or exit; do not rely on it. Consequences you must handle:
+  - Find the install folder (`steamapps\common\...`) and the game's `appmanifest_*.acf` to get the real App ID and exe name; do not guess them.
+  - Launch the game via Steam (for example `steam://rungameid/<appid>` or `steam.exe -applaunch <appid>`), or via a Steam **launch option** that wraps the game with the debugger (`<debugger command> %command%`), so the Steam environment is inherited. Test which approach works and record it in `NOTES.md`. Use whichever allows attaching early enough to break during startup and audio loading.
+  - **Never use Steam's "Verify integrity of game files"** and make sure automatic updates are off for this game, or Steam will overwrite the modified files. Your restore script, not Steam, is the way to restore.
+  - Keep the backup of the game folder outside the Steam library, and write the restore script to work while Steam is running.
+- **Debugger (dynamic analysis): WinDbg is the baseline.** WinDbg is installed and is the default for dynamic analysis. First detect how it is installed (Store version `windbgx.exe`, install folder, any WinDbg MCP tool in your tool list) and record it in `NOTES.md`. Prefer a WinDbg MCP tool if one is connected; otherwise drive WinDbg from the command line with a startup command string, a log file, and debugger scripts (verify the switches actually work with the installed version before building the harness around them). It must run unattended: log to a file, set breakpoints/commands at startup, and terminate the debuggee and debugger on its own at the end. Attaching to the Steam-launched process (by name or PID) is a valid approach. Prefer scripted debugging with log files over interactive tool calls when it is cheaper for context.
+- **You are free to improve the workflow.** Any tool, library, script, or technique that speeds up or de-risks the modding process is welcome: other debuggers or tracing tools, API/file-access monitors (for example Process Monitor or Frida-style hooks), in-process hooks or injected DLLs that log or call the game's audio functions directly, emulation or unit-testing of the decoder in isolation, custom Ghidra scripts, format-fuzzing harnesses, or anything else you judge useful. Install what you need, keep installs minimal, and record each addition and why in `NOTES.md`.
+- Python for tooling. Available/installable: vgmstream-cli, ww2ogg, revorb, ffmpeg, libvorbis/oggenc, and Wwise 2011.2.2. Check what is installed; if a tool is missing and the network allows, install it; otherwise write a minimal replacement.
+- You may debug the running game autonomously: launch it, attach debuggers, set breakpoints, read memory, and kill it. Always work on a copy-restorable game directory.
+
+## Rules of engagement
+
+- **Back up first.** Before modifying anything in the game directory or WAD, create a verified backup and a one-command restore script. Never edit the only copy of anything. Every test run starts from a clean restore plus exactly one change.
+- Keep the game's original files byte-identical except for the intended injections.
+- Change one variable at a time when testing hypotheses, and log the result.
+- Prefer small, falsifiable experiments over long speculation. State a hypothesis, test it, record the outcome.
+- Do not spend effort on unrelated goals (graphics, gameplay, other codecs) unless needed.
+
+## Verification (build this first)
+
+Before deep reverse engineering, build an **automated test harness** script that: restores a clean game copy, injects a given WEM, launches the game through Steam with WinDbg attached and logging, triggers or loads the target sound in a deterministic way (for example, a save or a scripted entry point that plays it early), detects crash / hang / clean exit, captures debugger logs and exception info, and prints a PASS/FAIL summary.
+
+Make "does it play" machine-checkable without a human listening. For example: breakpoint on the decoder's init/decode functions and log return values or error codes, check that decoded sample counts match expectations, and optionally capture audio output (loopback) and compare against the source with a similarity metric. Choose whatever is most reliable and document it. Also use vgmstream/ffmpeg decoding as an offline sanity check, remembering that offline success does not imply the game accepts the file.
+
+## Suggested phases
+
+1. Inventory: find existing work, tools, dumps, and notes. Set up the repo, `CLAUDE.md`, `NOTES.md`, backups, and restore script.
+2. Reverse the **WAD container**: header, file table, offsets, sizes, alignment, compression/encryption if any. Write a parser and a round-trip test (unpack, repack, byte-identical output).
+3. Build the test harness.
+4. Reverse the **WEM loading and Vorbis decoding path** in the dumped exe (and live process). Document exact format requirements.
+5. Compare original game WEMs with Wwise 2011.2.2 output and map every difference. Test hypotheses with the harness.
+6. Implement the encoder (approach A) or binary patch (approach B), whichever the evidence supports.
+7. Implement the injector with size/alignment/offset handling, including the case where the new file is larger or smaller than the original.
+8. Final end-to-end test with a custom sound, then write the README.
+
+## Memory and context management
+
+- `NOTES.md` is your persistent memory. After every meaningful finding or failed experiment, append: what you tried, the result, and the conclusion. Keep a short "Current status / Next steps" section at the top, rewritten as you go.
+- Record addresses, struct layouts, function names, and format specs in a `docs/` folder as you discover them, so you never have to rediscover them after a context reset.
+- At the start of every session or after a context reset, read `NOTES.md` first and continue from there.
+- Use subagents for bulk or mechanical work (large searches, log triage, batch experiments) to keep the main context small.
+
+## Limits and efficiency
+
+I am on a limited Claude Pro quota. Be economical: avoid dumping huge outputs into context (filter, grep, summarize logs), do not re-read large files unnecessarily, and delegate mechanical work to cheaper subagents. If you hit a usage limit, write a clear status in `NOTES.md` so work can resume cleanly.
+
+## Stop conditions
+
+- **Success:** the end-to-end test passes and the deliverables exist.
+- **Dead end:** if approach A and B have each had at least five distinct, well-documented failed hypotheses, stop and write a `REPORT.md` that lists what was proven, what was ruled out, the best remaining hypotheses, and the exact next experiments.
+- If any action risks something irreversible outside the project directory and the game directory, do not do it.
+
+## Output at the end
+
+Summarize in a few lines: what works, how to use the tools, and what remains. Everything else lives in the repo.

docs/history/mission2_editor_prompt.md

@@ -0,0 +1,109 @@
+# MISSION: Clean up the repo, then build the Arkham City Audio Editor (autonomous)
+
+You are working fully autonomously, exactly as in the previous mission. I will not answer questions or intervene. When something is ambiguous, pick the most reasonable option, record the assumption and reasoning in `NOTES.md`, and continue. Never stop to ask for permission or clarification. Stop only when the success criteria below are met, when a stop condition is hit, or when usage limits force a pause.
+
+## Context
+
+The previous mission succeeded: you reverse engineered the WAD container and the WEM format of **Batman: Arkham City (PC, Steam)**, built an encoder and an injector (the "ackit" tooling, in Python), and patched the game so modified audio plays. All of that is in this repository, together with many experiment scripts, notes, and session-resume markdown files.
+
+Start by reading `NOTES.md`, `CLAUDE.md`, and the docs to re-learn the current state. Do not redo reverse engineering. The Steam rules still apply: the game runs only through Steam, never use "Verify integrity of game files", keep auto-updates off for this game, keep backups outside the Steam library, and keep game files byte-identical except for intended changes. WinDbg and your other tools remain available if you need to verify something.
+
+## Part 1: Repository cleanup (do this first, then commit)
+
+Goal: a small, clear repository where a newcomer can find everything.
+
+1. **Safety net first.** Before deleting anything, commit the current state and create a git tag (for example `pre-cleanup`) so every deleted file is recoverable from history. Do not rewrite history.
+2. **Delete** every script and file that is not needed any more: one-off experiments, scratch analysis scripts, dead ends, duplicated tools, temporary outputs, large generated artifacts. Judge by whether the file is part of the final working pipeline or documents it.
+3. **Keep:**
+   - `ackit` (the final toolkit), with only the code that matters: encoding, injection, WAD/WEM parsing, patching, and testing/verification. Merge duplicates, remove leftover and unused code, give modules clear names, and add short docstrings. Keep the tests, make sure they pass, and add tests where core behavior is uncovered (WAD round-trip, WEM encode/decode check, inject/revert).
+   - **All documentation you wrote**: format specs, address maps, reverse-engineering notes, findings, failed hypotheses worth remembering. Move them into a tidy `docs/` structure, but do not delete or condense away information.
+   - **The markdown files used to resume sessions** (`NOTES.md`, `CLAUDE.md`, and similar). Keep them, update them to describe the new layout, and keep a short "Current status / Next steps" section.
+4. **Simplify usage.** Provide one clear entry point for the toolkit, for example a single CLI (`ackit <command>`) with a few well-named subcommands (such as `info`, `list`, `extract`, `encode`, `inject`, `revert`, `verify`) and helpful `--help` text. Replace the multi-step manual procedures with one-command workflows where possible. Write a short, clear top-level `README.md`: what it is, install, quick start with 2-3 examples, and where things live.
+5. **Reorganize the layout** so each kind of thing has an obvious place. Suggested (adapt if you find better):
+   ```
+   ackit/            Python package (core library + CLI)
+   tests/            automated tests
+   data/catalog/     CSV catalogs (one per language/WAD, see below)
+   docs/             format specs, reverse-engineering notes, how-tos
+   app/              the editor application (Part 2)
+   tools/            build/packaging helper scripts only
+   NOTES.md, CLAUDE.md, README.md
+   ```
+6. **Catalog.** My curated catalog of audio files (by character, mainly for the ITA language, though the idea applies to other language WADs too) is the source for names/characters. You had started converting it into CSV files. Locate the original catalog files and your CSV work. Finish converting everything into well-structured CSVs under `data/catalog/`, with a documented schema (see Part 2) and a short `data/catalog/README.md` explaining columns and how to add other languages. Do not lose or overwrite my original catalog files; keep them in a clearly named `data/catalog/source/` folder (unless they are too large or clearly not redistributable, in that case document where they are).
+7. Add a `.gitignore` so game assets, extracted audio, dumps, and build outputs never get committed. Do not commit any Arkham City game audio.
+8. Run the test suite and one real end-to-end check (inject a custom sound, confirm it plays in game through the existing harness if still available, and revert) to prove the cleanup did not break anything. Commit.
+
+## Part 2: The Audio Editor application
+
+Build a simple yet powerful desktop editor for the game's audio, working on **Windows and Linux** (on Linux the game runs through Steam/Proton, so the editor must accept any game folder path and handle Proton library paths; the editor itself does not need to launch the game).
+
+### Technology decision (you decide, document it in `docs/decisions.md`)
+
+Existing tools are Python. Pick the stack that best balances: working on Windows and Linux, easy installation for non-technical users, good audio playback/recording, reuse of `ackit`, and reasonable effort. Candidates: Python with a Qt GUI (PySide6) packaged with PyInstaller/Nuitka (bundles its own Python, no separate install), a web UI served locally and shown in a window, or a rewrite in another language (for example Rust/Tauri or C#). Prefer reusing `ackit` unchanged over rewriting it, unless there is a strong reason. Requirements for the decision:
+- Distribution must not require the user to install Python or other runtimes (bundle what is needed), or else provide a clearly simple one-step installer.
+- The encoder must work **without Wwise** at runtime and on Linux as well as Windows. If any part of the encoding pipeline depends on a Windows-only tool, replace that part with portable code, or ship the required pre-generated data (such as codebook data) with the app. Verify on both platforms as far as you can (you can test Linux behavior under WSL or a Linux container if available).
+- Audio I/O must be portable (for example `sounddevice`/PortAudio, `miniaudio`, or Qt Multimedia), with device selection.
+- **Bundle external tooling in the distribution.** If the app or `ackit` needs any external tool or binary (for example ffmpeg/ffprobe for decoding and converting, vgmstream-cli for decoding game WEMs, revorb, ww2ogg, oggenc/libvorbis, PortAudio or other native libraries), include it in the final dist build for each platform, so the user installs nothing else. Rules:
+  - Prefer removing a dependency when a small portable library does the job (for example a Python decoder instead of a full ffmpeg), to keep the package small. Otherwise bundle the binary.
+  - Ship per-platform binaries (Windows and Linux, x86-64 at least) in a `vendor/` or `bin/` folder inside the package, and resolve them at runtime relative to the app (never rely on `PATH`). Fall back to a system copy only if the bundled one is missing.
+  - Download them from official/trusted sources in a build script (`tools/fetch_vendor.*`), pin versions, and verify checksums. Do not commit large binaries to git; fetch them at build time.
+  - **Licenses:** check each tool's license (for example ffmpeg LGPL/GPL build variants; prefer an LGPL build without GPL-only components if sufficient). Include a `THIRD_PARTY_LICENSES` file with each license text, version, and source URL, and satisfy redistribution terms (for example, point to the source). Record the decisions in `docs/decisions.md`.
+  - Test the packaged build on a machine/container **without** these tools on `PATH` to prove it is self-contained.
+
+### Feature 1: Browser
+
+- Load a game folder (and its WADs), remember the last path, and show a table of all audio entries with fast loading even for tens of thousands of entries (lazy loading, indexing/caching in a user cache folder; do not decode everything up front).
+- **Search** by name/ID, plus **filters** for category, character, language/WAD, and any other useful field. Combine them freely, with instant results as you type.
+- **Categories:** you decide a sensible category scheme (for example Speech, Music, SFX, Ambience, UI, Combat, Cinematic...). Derive them from the catalog and from the data itself (WAD/folder names, Wwise bank/event names, file size/duration heuristics, naming patterns), and document the scheme and the rules in `docs/`. Entries that cannot be classified go into an "Unknown" category and must remain searchable.
+- **Characters** come from my catalog CSVs. Entries not in the catalog still appear, with empty character.
+- Add other columns/filters you judge useful (duration, size, sample rate, channels, modified/original status, catalog notes or transcript text if present in the catalog, favorites/tags). Sorting on every column, multi-select, and a "modified only" filter.
+- Quick preview playback directly from the list (play/stop, keyboard-driven navigation), with a visible indicator for entries modified by the user.
+- Let the user add their own notes/tags per entry, stored in a user data file (not in the game folder).
+
+### Feature 2: Single-file editor
+
+Opening an entry shows an editor view with:
+- **Listen**: play original and current (modified) audio, waveform display, seek, volume, loop, and A/B compare between original and replacement.
+- **Substitute**: replace the audio from a file (WAV, OGG, MP3, FLAC, and whatever else ffmpeg or the chosen libs decode easily). Handle sample-rate/channel conversion, loudness/volume adjustment, and trimming. Warn clearly if the replacement is much longer/larger than the original, and make sure the injector handles any size correctly (offsets and alignment already solved in ackit; reuse it).
+- **Record**: record in the moment from a user-selected **audio input device** (device list, level meter, record/stop/re-record, preview before accepting, optional trim and normalize). Test recording logic with a virtual or file-based input where no hardware exists.
+- **Apply / Revert**: applying encodes with ackit and injects into the game WAD (with the safety rules from before: back up before first write, atomic/safe writes, never leave the WAD in a half-written state). **Revert** must restore the original audio of that entry, and there must also be "Revert all". Decide where originals live and justify it in `docs/decisions.md`. A good default: the first time an entry is modified, store its original bytes (with hash and location metadata) in an `originals` store; revert re-injects from it, so no full-WAD copy is required. Consider also keeping a pristine backup of the whole WAD as a fallback. Choose the location (user data folder vs. next to the game files) by what is most convenient and portable, for example a per-game-folder store with a configurable path, and make it survive app updates. Detect when the game files changed underneath (for example after a Steam update or verification) and warn the user safely instead of corrupting anything.
+- **Export/import mods:** let the user export their changes as a portable package (replacement audio plus a manifest) and import it on another machine or share it, so mods are not tied to one WAD state. Keep it simple.
+
+### UX and quality bar
+
+- Simple first: the primary flow (search, select, listen, replace/record, apply) must be obvious without reading docs. Power features stay available but unobtrusive.
+- Clear messages for errors (wrong game version, missing patch, unsupported audio, WAD locked by a running game, Steam path issues).
+- Do not add telemetry or network access, except where strictly needed (none expected).
+- Keep dependencies reasonable and pinned; keep the app responsive (long operations in background threads with progress and cancel).
+
+## Verification (build it as you go)
+
+- Unit/integration tests for the library; headless tests for the app logic (search/filter index, catalog loading, apply/revert, mod export/import). Use fixtures or synthetic WADs where real game data cannot be committed.
+- Automated GUI smoke tests (for example headless/offscreen Qt or a browser driver) that open the app, search, open an entry, replace audio from a file, apply, and revert, checking byte-level restoration of the original.
+- At least one **real end-to-end test against the actual game** using the existing harness: apply a replacement through the editor code path, confirm the game still starts and plays it, then revert and confirm the WAD is byte-identical to the original.
+- Build the distributable packages for the platform you are on, and verify as far as possible that the Linux package starts and works (container or WSL). Document exactly how to build for the other platform.
+
+## Deliverables
+
+1. Cleaned repository with the structure above, passing tests, and updated `NOTES.md`/`CLAUDE.md`/docs.
+2. Simplified `ackit` CLI and top-level `README.md`.
+3. Catalog CSVs and schema documentation.
+4. The editor app source in `app/`, with build scripts in `tools/`, plus a self-contained packaged build for the current platform (bundled external tools included, with `THIRD_PARTY_LICENSES`) and instructions to build for Windows and Linux.
+5. `docs/decisions.md` (stack, category scheme, originals store, trade-offs) and a user guide with a few screenshots (headless screenshots are fine).
+
+## Working rules (unchanged from before)
+
+- `NOTES.md` is your persistent memory: log progress, decisions, and failed attempts; keep "Current status / Next steps" at the top. After a context reset or a usage pause, read it first and continue.
+- Commit often with meaningful messages. Back up before touching game files, and test on restorable copies.
+- Be economical with quota: filter and summarize outputs, avoid re-reading large files, delegate mechanical or bulk work to subagents, and work in stages (cleanup first, then library changes, then browser, then editor, then packaging) so partial progress is always usable.
+- Order of priority if time or quota is short: (1) cleanup, (2) browser with search and filters, (3) single-file editor with substitute and revert, (4) recording, (5) mod export/import, (6) packaging polish. Never sacrifice the revert safety for features.
+
+## Stop conditions
+
+- **Success:** all deliverables exist and the verification tests pass, including the real game end-to-end test.
+- **Blocked:** if a requirement cannot be met (for example, the encoder truly needs a Windows-only component), implement the best alternative, document the limitation clearly in `docs/decisions.md` and `README.md`, and continue with the rest. Write `REPORT.md` at the end with what works, what does not, and the next steps.
+- Do not take irreversible actions outside the project and game folders.
+
+## Output at the end
+
+A few lines: what was cleaned up, what the app can do, how to run and build it, and known limitations. Everything else lives in the repo.