Read and patch UE3 packages and soundbanks

docs/formats/upk.md

@@ -0,0 +1,38 @@
+# UE3 packages (`*.upk`)
+
+Arkham City's 2,650 packages in `BmGame\CookedPCConsole` are Unreal Engine 3 packages, version 805,
+licensee 101. 2,647 are fully LZO-compressed, 2 are stored uncompressed, and `UnrealX.upk` (512 bytes)
+is not a readable package. ackit reads and patches them with `ackit/upk.py` and `ackit/lzo.py` (no external
+tools). Soundbanks inside them: `banks.md`.
+
+## Summary (file start, little-endian)
+
+| field | notes |
+|-------|-------|
+| u32 tag `0x9E2A83C1`, u16 version 805, u16 licensee 101, i32 TotalHeaderSize | |
+| FString FolderName | i32 length incl. NUL, then bytes (`None`) |
+| 12 x i32 | PackageFlags, NameCount, NameOffset, ExportCount, ExportOffset, ImportCount, ImportOffset, DependsOffset, ImportExportGuidsOffset, ImportGuidsCount, ExportGuidsCount, ThumbnailTableOffset |
+| 16 bytes Guid | |
+| i32 N + N x {ExportCount, NameCount, NetObjectCount} | generations (N = 1) |
+| u32 EngineVersion 7748, u32 CookerVersion, u32 CompressionFlags | 2 = LZO, 0 = none |
+| i32 C + C x {i32 UncompressedOffset, UncompressedSize, CompressedOffset, CompressedSize} | chunk table |
+| u32 PackageSource, ... | not used by ackit |
+
+All offsets in the summary (names, exports, ...) refer to the **uncompressed** package. The uncompressed
+package = the summary without the chunk table, followed by the chunks' data at their UncompressedOffset
+(the first chunk starts right after the summary, at NameOffset).
+
+## Compressed chunks
+Each chunk (at CompressedOffset): `u32 tag, u32 block size (0x20000), u32 compressed sum, u32 uncompressed
+sum`, then one `{u32 compressed size, u32 uncompressed size}` per block, then the blocks. Every block is an
+independent LZO1X stream. In all game packages the chunks follow each other without gaps, start right
+after the summary and end at the end of the file (checked for every package).
+
+## How ackit reads and writes them
+- Positions of sounds are offsets in the uncompressed package, the same as in a decompressed copy
+  (e.g. Gildor's decompress.exe output). Reading decompresses only the blocks that are needed.
+- Writing (`Package.patched`): every changed block is stored again as a single LZO literal run (a valid
+  LZO1X stream without matches, `lzo.compress_stored`), the chunk headers of the changed chunks and the
+  CompressedOffset/CompressedSize of all following chunks are updated; every other byte is copied, so a
+  package without changes is rebuilt byte-for-byte. Replacements never change the uncompressed layout
+  (sounds are replaced in place), so names, exports and their offsets stay valid.