Initial commit

docs/formats/wad.md

@@ -0,0 +1,70 @@
+# Arkham City WAD / BKPK container format
+
+All values little-endian. Verified by byte-identical round trips of every shipped WAD
+(INT/ITA/FRA/DEU/ESN LOC WADs and `SFX\17705D3E.wad`) and every BKPK pack.
+
+## WWAD (`*.wad`)
+
+| off  | type | meaning |
+|------|------|---------|
+| 0x00 | 4s   | `WWAD` |
+| 0x04 | u32  | version `0x01020000` |
+| 0x08 | u32  | entry count N |
+| 0x0C | u32  | table offset (always `0x1C`) |
+| 0x10 | u32  | table size (`12*N`) |
+| 0x14 | u32  | data start (header area size; `0x8000` LOC, `0x1000` SFX) |
+| 0x18 | u32  | data size (`file size - data start`) |
+| 0x1C | N x {u32 hash, u32 size, u32 offset} | |
+
+Rules:
+- Table sorted by `hash` interpreted as **signed** int32. Payloads stored in table order.
+- Each payload starts at the next **0x1000-aligned** offset after the previous one.
+- `size` = payload length rounded up to **0x800**; padding is zero.
+- File ends 0x1000-aligned; header area after the table is zero.
+- `hash` = Wwise media/file ID (names: `hashlist.txt`, e.g. `CHR_RadioScanner`, `AMB_Steel_B3`, `Music`).
+
+Payload kinds:
+- **LOC WADs**: each entry is a group of WEMs, the variations of one line (ITA: 10816 WEMs in
+  2464 entries; e.g. `CHR_RadioScanner` holds 91). Each WEM starts at the next 0x800 boundary after
+  the previous one (zero padding); 3 entries pack two WEMs back to back. Addressed `HASH:i`.
+- **SFX WAD**: 79 grouped ambience/effect entries (260 WEMs) plus 4 **stream packs** (below).
+
+Where the game gets the inner offsets: the entry IDs appear next to `BKPK` headers inside the
+LZO-compressed UE3 packages (e.g. `BmGame.upk`, `OW.upk`, `Church_B1_*.upk`). Those embedded BKPK
+tables give each WEM's offset **and size**. ackit does not rewrite `.upk` files, so these WEMs are
+replaced **in place, no larger than the original** (zero-filled to the original size), and the WAD
+layout never changes. The user's earlier quickbms-reimport mod did the same and was loaded by the game.
+Only the 4 SFX stream packs, whose BKPK index is a loose `.bnk`, can change size.
+
+## BKPK stream packs (`SFX\<ID>.bnk` + SFX WAD entry `<ID>`)
+
+`EDF036D6` (Music, 314 WEMs, incl. the intro/front-end music `393BC53A`), `E8F980BE` (Music_PDLC, 75),
+`57572DAC` (Global, 1), `FCA4F778` (1). The `.bnk` file is an index + an embedded soundbank; the WEM
+data lives in the SFX WAD entry with the same ID.
+
+| off  | meaning |
+|------|---------|
+| 0x00 | `BKPK` |
+| 0x04 | version `0x01020000` |
+| 0x08 | pack id (== WAD entry hash) |
+| 0x10 | media count N |
+| 0x18 | table size `12*N` |
+| 0x1C | table offset `0x48` |
+| 0x20 | total stream size (== WAD entry size) |
+| 0x2C | end of table (`0x48 + 12*N`) |
+| 0x38 | embedded bank size |
+| 0x3C | embedded bank offset (0x800 or 0x1000; a normal BKHD/HIRC/STID bank) |
+| 0x48 | N x {u32 media id, u32 size, u32 offset in the WAD entry} |
+
+Rules: table sorted by media id; each media starts at the next **0x800-aligned** offset; total =
+end rounded up to 0x800. Padding is mostly zero (a few non-zero bytes in `EDF036D6`; the tool keeps
+original bytes where nothing moved).
+
+Replacing a pack media (`ackit` target `EDF036D6/393BC53A`) rewrites the pack table (`.bnk`), the
+WAD entry payload, and the WAD table/offsets. Verified in-game (NOTES.md, E2).
+
+## Regular soundbanks (`SFX\*.bnk`, bank version 56)
+
+BKHD / DIDX {id, offset, size} / DATA / HIRC / STID. In-memory WEMs (DATA) are 16-byte aligned.
+`ackit/bnk.py` can read/rebuild them, but most front-end sounds come from banks embedded in UE3
+packages (`.upk`), which this toolkit does not modify.

docs/formats/wem.md

@@ -0,0 +1,82 @@
+# WEM format accepted by Arkham City (Wwise 2011.1, Vorbis)
+
+Derived from the game's WEMs; `ackit verify` / `tests/test_realgame.py` rebuild them byte-for-byte from these rules
+alone (all 10816 ITA WEMs; in SFX only the 4 looping music tracks with loop points differ).
+
+## RIFF layout
+`RIFF` size `WAVE`, then `fmt ` (0x42 bytes), optional `cue `/`LIST` (37 files), `data`.
+Odd-sized `data` chunks are **not** padded.
+
+### `fmt ` (0x42)
+| off | type | value |
+|-----|------|-------|
+| 0x00 | u16 | 0xFFFF (Wwise Vorbis) |
+| 0x02 | u16 | channels (1,2,3,4,5,6 seen) |
+| 0x04 | u32 | sample rate |
+| 0x08 | u32 | avg bytes/s = `(data_len - seek_size) * rate // total_frames` |
+| 0x0C | u16,u16 | 0, 0 |
+| 0x10 | u16 | cbSize 0x30 |
+| 0x12 | u16 | valid bits (16 or 24 = source bit depth, informational) |
+| 0x14 | u32 | channel mask (mono 0x4, stereo 0x3, quad 0x33, 5.0 0x37, 5.1 0x3F) |
+| 0x18 | 42 B | AkVorbisInfo (below) |
+
+### AkVorbisInfo (fmt+0x18)
+| off | type | field | rule |
+|-----|------|-------|------|
+| 0x00 | u32 | dwTotalPCMFrames | sample count |
+| 0x04 | u32 | dwLoopStartPacketOffset | `2 + setup_len` (relative to end of seek table) |
+| 0x08 | u32 | dwLoopEndPacketOffset | `data_len - seek_size` |
+| 0x0C | u16 | uLoopBeginExtra | 0 |
+| 0x0E | u16 | uLoopEndExtra | = uLastGranuleExtra |
+| 0x10 | u32 | dwSeekTableSize | bytes (4 per entry) |
+| 0x14 | u32 | dwVorbisDataOffset | `seek_size + 2 + setup_len` |
+| 0x18 | u16 | uMaxPacketSize | max audio packet payload |
+| 0x1A | u16 | uLastGranuleExtra | `sum(packet samples) - total_frames` (< blocksize) |
+| 0x1C | u32 | dwDecodeAllocSize | see "decode alloc" — **must be right** (too small = NULL write crash) |
+| 0x20 | u32 | dwDecodeX64AllocSize | `alloc + 8*nbooks + 40 + 52*nfloors` (fit; unused by 32-bit game) |
+| 0x24 | u32 | uHashCodebook | **FNV-1 32** of the stripped setup bytes; codec-setup cache key |
+| 0x28 | u8,u8 | uBlockSizes | log2 short, log2 long |
+
+## `data` chunk
+```
+[seek table: N x {u16 granule delta, u16 byte delta}]
+[u16 setup_len][stripped setup packet]
+[u16 len][audio packet] ...
+```
+- Seek points: walk packets; point at the first packet whose *end* granule >= previous point + interval
+  (8192 for SFX/speech; music uses 1024 or other per-sound values). Byte offsets are relative to the
+  setup start (end of seek table).
+- No identification/comment headers (rebuilt from fmt).
+
+## Stripped setup ("external codebooks")
+Standard Vorbis setup with: no `\x05vorbis` header; codebook count (8b), then each codebook as a
+**10-bit ID** into the built-in library; no time-domain section; floor type field removed (always 1);
+residue type 2 bits; mapping type removed; mode window/transform type removed. See
+`ackit/wwvorbis.py:transcode_setup` (bidirectional).
+
+Codebook library: the game's table (runtime `0x0158E748`, 598 books) is byte-identical to ww2ogg's
+standard `packed_codebooks.bin` (`ackit/data/game_codebooks.bin`). **Wwise 2011.2.x uses the aoTuV 6.03
+library instead** — same IDs, different books — which is why files from the newer encoder decode to
+garbage/silence in this game.
+
+Stock libvorbis (1.3.7, in ffmpeg and in the libsndfile wheels ackit uses) setups map onto this library for all tested rates with 1/2/4/5
+channels, and for 6 channels up to 32 kHz (its 5.1 books at 44.1/48 kHz are newer than the game).
+At 24 kHz q>=5 needs a book not in the library. The encoder probes and picks a compatible quality.
+
+## Audio packets
+- Blocksizes differ (e.g. 256/2048): "modified" packets — packet-type bit and the long-window
+  prev/next flags are removed (mode number stays first).
+- Equal blocksizes (512/512, 1024/1024): standard Vorbis packets (type bit kept).
+- 2-byte size header, no granule.
+
+## Decode alloc (dwDecodeAllocSize)
+`0x30 + parser allocations + 4*nbooks + 8`, every allocation rounded up to 4 bytes. Parser
+allocations (Tremor-lowmem derived, `ackit/allocsize.py`):
+- codebooks: `nbooks*0x3C` + per book: decode table (`used<2` or nodeb==4: `used*8+4`, else
+  `align4((used*(leafw+1)-2)*nodeb)`) + for dec_type 2 the quantized values (`align4(quantvals)` or
+  `align4(2*quantvals)`).
+- floors: `nfloors*0x24` + per floor `a4(parts) + a4(11*(maxclass+1)) + a4(2*(posts+2)) + a4(posts+2) + 2*a4(posts)`.
+- residues: `nres*0x1C` + per residue `a4(ncls) + 8*ncls`.
+- mappings: `nmaps*0x14` + `a4(2*coupling_steps)` + `a4(channels)` if submaps>1 + `a4(2*submaps)`.
+- modes: `a4(2*nmodes)`.
+Exact on all 24 setups the game ships; in-game test E10 used a new setup with computed values.