Read and patch UE3 packages and soundbanks

docs/formats/banks.md

@@ -12,8 +12,10 @@ Survey of the installed game (buildid 3718300) on 2026-10-09, read-only.
 | loose `SFX\*.bnk` (Batman/Catwoman footsteps, 32 surfaces each, + `TST_Prep_Media`) | 65 | 1,117 | 0 |
 | banks embedded in `.upk` packages (in 1,432 of the 2,650 packages) | 3,340 | 15,641 (~276 MB) | 0 |
 
-The two sets share 119 media IDs: about **15,800 unique sounds** in total. The WADs, by comparison, are fully
-covered: the BKPK tables in the packages reference 2,543 WAD entries, all of them in the catalog.
+The two sets share 119 media IDs: 15,800 unique media IDs in 96,011 places (bank copies). 396 of them are
+**prefetch heads** of streamed WAD sounds (below), so there are **15,404 in-memory sounds**. All copies of a
+media ID are byte-identical except `1C898731` (two variants). The WADs, by comparison, are fully covered: the
+BKPK tables in the packages reference 2,543 WAD entries, all of them in the catalog.
 
 Embedded WEMs by bank name prefix: `Speech` 7,872, `CHR` (character foley, combat, bosses) 3,896, `PSO` 781,
 `WPN` 664, `NPC` 475, `OBJ` 439, `SEQ` 265, `GUI` 208, `PSA` 203, `MTN` 196, `PX` 195, `DVC` 100, `GEN` 80,
@@ -40,14 +42,17 @@ HIRC Sound object (type 2) payload: u32 object id, u32 plugin id (`0x00040001` =
 (0 = in memory, 1 = streamed, 2 = prefetch), u32 media id, u32 bank id of the media; for in-memory sources
 then u32 offset, u32 size, where offset = `DATA` payload position (from the start of the bank) + DIDX offset
 and size = DIDX size (checked on every in-memory source of the footstep banks and of `BmGame.upk`).
-A few sources use media of another bank. Every streamed media ID found is in a BKPK table, i.e. a WAD sound
-(`wad.md`).
+Source kinds over all bank copies: 115,861 in-memory sources of the bank's own media; 2,233 in-memory sources
+with bank id 0 whose media is not in the same bank (loaded by another bank); 133,075 streamed sources;
+2,101 **prefetch** sources. Every streamed or prefetched media ID is in a BKPK table, i.e. a WAD sound
+(`wad.md`). A prefetch source's media is also in the bank's DIDX: that in-memory copy is the **head** of the
+streamed WEM (its size = the source's size), which the game plays while the rest is read from the WAD. 396
+WAD sounds have such heads (2,101 copies); replacing one of them must rewrite the heads too.
 
 ## Banks inside packages
-- The packages are fully LZO-compressed (package version 805, licensee 101). For the survey they were unpacked
-  with Gildor's `decompress.exe` (gildor.org; used from the user's old modding folder, not part of this
-  repository) and searched for `BKHD` (banks) and `BKPK` (WAD indexes). `UnrealX.upk` (512 bytes) is not a
-  readable package (wrong tag, probably encrypted).
+- The packages are LZO-compressed UE3 packages (`upk.md`); ackit decompresses them itself and searches the
+  uncompressed data for `BKHD` (banks) and `BKPK` (WAD indexes). (The first survey used Gildor's
+  decompress.exe; `ackit/upk.py` gives identical data.)
 - Bank blobs in packages: 41,798 for 3,340 bank IDs. Many banks are copied into several packages (e.g.
   `CHR_CatwomanFoleySuit` 31 times, `CHR_BatmanFoleyBelt` 26). 302 bank IDs occur as several blobs with
   *different* media subsets (often one WEM each, several per package), so a bank's media set is the union

docs/formats/upk.md

@@ -0,0 +1,38 @@
+# UE3 packages (`*.upk`)
+
+Arkham City's 2,650 packages in `BmGame\CookedPCConsole` are Unreal Engine 3 packages, version 805,
+licensee 101. 2,647 are fully LZO-compressed, 2 are stored uncompressed, and `UnrealX.upk` (512 bytes)
+is not a readable package. ackit reads and patches them with `ackit/upk.py` and `ackit/lzo.py` (no external
+tools). Soundbanks inside them: `banks.md`.
+
+## Summary (file start, little-endian)
+
+| field | notes |
+|-------|-------|
+| u32 tag `0x9E2A83C1`, u16 version 805, u16 licensee 101, i32 TotalHeaderSize | |
+| FString FolderName | i32 length incl. NUL, then bytes (`None`) |
+| 12 x i32 | PackageFlags, NameCount, NameOffset, ExportCount, ExportOffset, ImportCount, ImportOffset, DependsOffset, ImportExportGuidsOffset, ImportGuidsCount, ExportGuidsCount, ThumbnailTableOffset |
+| 16 bytes Guid | |
+| i32 N + N x {ExportCount, NameCount, NetObjectCount} | generations (N = 1) |
+| u32 EngineVersion 7748, u32 CookerVersion, u32 CompressionFlags | 2 = LZO, 0 = none |
+| i32 C + C x {i32 UncompressedOffset, UncompressedSize, CompressedOffset, CompressedSize} | chunk table |
+| u32 PackageSource, ... | not used by ackit |
+
+All offsets in the summary (names, exports, ...) refer to the **uncompressed** package. The uncompressed
+package = the summary without the chunk table, followed by the chunks' data at their UncompressedOffset
+(the first chunk starts right after the summary, at NameOffset).
+
+## Compressed chunks
+Each chunk (at CompressedOffset): `u32 tag, u32 block size (0x20000), u32 compressed sum, u32 uncompressed
+sum`, then one `{u32 compressed size, u32 uncompressed size}` per block, then the blocks. Every block is an
+independent LZO1X stream. In all game packages the chunks follow each other without gaps, start right
+after the summary and end at the end of the file (checked for every package).
+
+## How ackit reads and writes them
+- Positions of sounds are offsets in the uncompressed package, the same as in a decompressed copy
+  (e.g. Gildor's decompress.exe output). Reading decompresses only the blocks that are needed.
+- Writing (`Package.patched`): every changed block is stored again as a single LZO literal run (a valid
+  LZO1X stream without matches, `lzo.compress_stored`), the chunk headers of the changed chunks and the
+  CompressedOffset/CompressedSize of all following chunks are updated; every other byte is copied, so a
+  package without changes is rebuilt byte-for-byte. Replacements never change the uncompressed layout
+  (sounds are replaced in place), so names, exports and their offsets stay valid.