Full correctness check against the docs: safer writes, reverts and texts

Safety
- Writes are serialized (Workspace.lock): two background jobs rebuilding one
  package (a sound and its subtitle applied side by side) lost an edit.
- A store copies a bank file or package as "original" only if it matches the
  shipped catalog (Workspace.unoriginal_files: in-memory sounds, bank copies of
  voice lines, subtitles); the editor refuses to switch stores while changes
  are applied. A second store could otherwise resurrect a reverted sound.
- Mod import in the editor asks before overwriting files changed outside the
  tool (it forced silently).
- Revert all in the editor reverts texts too; the uninstaller counts text
  edits before offering to delete the store.
- ackit restore-backup: the full-backup restore decisions.md described; it
  keeps the sounds of WADs without a full copy recorded.
- build_catalog also stops when a subtitle differs from the catalog.

Languages and texts
- Game.language() returns languages without voices (JPN, RUS, ...) and
  Game.voice_archive() their INT voices; ackit text --lang JPN works.
- Menu lookups parse one cached file (find & replace: minutes -> ms).
- Subtitles decode cp1252 as Windows does: the UTF-8-in-cp1252 line
  Speech_TAPE_J_Answer_1_00 (KOR/POL/POR/RUS) is written back byte-for-byte;
  SUB_KOR/SUB_RUS rebuilt from the original game (only those rows changed).
- Mods exported while another language is loaded keep name and line.

Other fixes
- Down-mix folds extra channels in at -3 dB (a 5.1 centre voice was dropped).
- e2e_editor no longer overwrites the user's settings.json; kill_game waits
  for the game to exit before reverts.
- Recorder handles Int32/UInt8 input and split frames; package block cache
  locked; conflict messages, CLI help examples, stale docstrings and doc
  references fixed; tests no longer read this PC's game language.

Docs (decisions, user guide, README, subtitles.md, banks.md, catalog README,
REPORT, NOTES, CLAUDE.md) updated; check_docs: 102 claims match. 148 tests;
e2e_features 9/9, e2e_editor and verify PASS; game original afterwards.
This commit is contained in:
2026-10-13 21:34:13 +02:00
parent a6ef3b9458
commit 1ccaa8b398
41 changed files with 846 additions and 186 deletions

View File

@@ -12,7 +12,8 @@ Outputs
data/catalog/SUB_locations.csv.gz every stored copy of each subtitle string
data/catalog/index.json fingerprints the CSVs were built from + counts
Run it only on clean game files (`ackit status` must say nothing is modified). It stops before writing
anything if a sound differs from the current catalog (a modified game); after a game update use --accept-changes.
anything if a sound or a subtitle differs from the current catalog (a modified game); after a game update use
--accept-changes. (Menu text has no catalog.)
"""
import argparse
import collections
@@ -88,9 +89,16 @@ def build_characters(game, src_root, final):
return rows, unmatched, ambiguous, skipped
def changed_since_catalog(game, out_dir, bank_snd):
"""Keys whose original checksum in the existing catalog differs from the game files now."""
def changed_since_catalog(game, out_dir, bank_snd, texts=None):
"""Keys whose original in the existing catalog differs from the game files now: sounds (checksums) and
subtitles (texts: {lang: [SubLine]} scanned now)."""
out = []
for lang, rows in (texts or {}).items():
p = os.path.join(out_dir, "SUB_%s.csv" % lang)
if os.path.exists(p):
old = {(s.speech, s.line): s.text for s in subtitles.read_texts(p, lang)}
out += [subtitles.key(lang, s.speech, s.line) for s in rows
if (s.speech, s.line) in old and old[(s.speech, s.line)] != s.text]
for arch, path in game.archives.items():
p = os.path.join(out_dir, "%s.csv" % arch)
if not os.path.exists(p):
@@ -133,11 +141,12 @@ def main():
for k, d in lines.issues[:10]:
print(" issue:", k, d)
# safety: the catalog records ORIGINAL bytes. A sound that differs from the current catalog means a modified
# game (an applied change, another tool) unless the game itself was updated -> stop before writing anything.
changed = changed_since_catalog(game, out_dir, bank_snd)
# safety: the catalog records ORIGINAL bytes and texts. A sound or subtitle that differs from the current
# catalog means a modified game (an applied change, another tool) unless the game itself was updated -> stop
# before writing anything.
changed = changed_since_catalog(game, out_dir, bank_snd, texts)
if changed and not a.accept_changes:
print("STOP: %d sound(s) differ from the current catalog, e.g. %s." % (len(changed), changed[:5]))
print("STOP: %d sound(s) or subtitle(s) differ from the current catalog, e.g. %s." % (len(changed), changed[:5]))
print("Revert all changes (ackit status / ackit revert --all) and run again; after a game update use "
"--accept-changes.")
sys.exit(2)
@@ -193,7 +202,7 @@ def main():
"fingerprint": banks.game_fingerprint(game.cooked), "sounds": len(sounds), "groups": dict(g),
"named_by_event": by_event, "named_by_bank": len(sounds) - by_event,
"copies": sum(1 for l in locs if l.kind == banks.MEMORY), "files": len(sc.files),
# bank copies of WAD recordings: prefetch heads and whole in-memory copies (catalog._whole_recordings)
# bank copies of WAD recordings: prefetch heads and whole in-memory copies (catalog.whole_recordings)
"prefetch_heads": {"recordings": len({l.media for l in heads}), "copies": len(heads),
"unmapped": len({l.media for l in heads if not l.sound})},
"issues": ["%s: %s" % i for i in sc.issues]}

View File

@@ -240,6 +240,16 @@ def catalog_claims(cat):
west = [enc[L]["w"] for L in langs if L not in ("RUS", "JPN", "KOR", "POL")]
claim("docs/formats/subtitles.md", "%d %% of POL are UTF-16; the other languages have %s UTF-16 strings each"
% (pol, rng(west)))
utf8 = [] # cp1252 strings whose bytes are UTF-8 text (the game's data)
for L, t in sorted(texts.items()):
for s in t:
if s.enc == "a":
u = subtitles._encode_cp1252(s.text).decode("utf-8", "replace")
multi = sum(1 for c in u if ord(c) > 0x7F and c != "\ufffd") # valid multi-byte characters
if multi and u.count("\ufffd") <= multi // 4: # (one corrupt byte tolerated)
utf8.append((L, s.speech))
claim("docs/formats/subtitles.md", "%d cp1252 strings hold UTF-8 bytes instead (the same line in %s: `%s`"
% (len(utf8), ", ".join(L for L, _ in utf8), "`, `".join(sorted({sp for _, sp in utf8}))))
with gzip.open(os.path.join(cat, subtitles.LOCATIONS), "rt", newline="") as f:
sublocs = sum(1 for _ in csv.DictReader(f))
claim("REPORT.md", "Edited in place in every copy (%s copies)" % n(sublocs))
@@ -505,6 +515,21 @@ def wem_claims():
% (n(total - len(bad)), n(total), exceptions))
def originality_claims(game):
"""Workspace.unoriginal_files on every file with sounds or subtitles of the (original) game: no false alarm."""
import tempfile
from ackit import subtitles
from ackit.workspace import Workspace
with tempfile.TemporaryDirectory() as td:
ws = Workspace(game, store_root=td, archives=[game.voice_archive("INT"), "SFX", "BANK"])
rels = {l.file for ls in ws.catalog.locations.values() for l in ls}
rels |= {l.file for ls in subtitles.read_locations(os.path.join(catalog_dir(), subtitles.LOCATIONS)).values()
for l in ls}
bad = ws.unoriginal_files([ws.bank_path(r) for r in rels])
claim("REPORT.md", "checked on all %s files with sounds or subtitles: %s" % (
n(len(rels)), "no false alarm" if not bad else "%d false alarms" % len(bad)), "Workspace.unoriginal_files")
def test_claims():
import subprocess
out = subprocess.run([sys.executable, "-m", "pytest", "--collect-only", "-q", "-p", "no:cacheprovider"],
@@ -535,6 +560,7 @@ def main():
if not game:
raise SystemExit("--scan needs the game")
scan_claims(game)
originality_claims(game)
if a.wems:
if not game:
raise SystemExit("--wems needs the game")

View File

@@ -5,6 +5,8 @@
3. start the game under cdb + speaker loopback: the probe must be heard, no crash, setup parsed;
4. press Revert (EditorPanel.revert) and check the WAD + pack index are byte-identical to before;
5. also apply/revert a speech line in place and check byte identity.
Uses the user's settings and originals store (an interrupted run shows up in `ackit status`); the settings file
is never written.
python tools/e2e_editor.py
"""
@@ -29,11 +31,15 @@ def sha(p):
def main():
from PySide6.QtWidgets import QApplication
from ackit import settings as settings_mod
from app.mainwindow import MainWindow
from app.workers import Runner
app = QApplication([])
w = MainWindow(settings={"full_backup": True}, runner=Runner(sync=True))
# the user's settings (same game folder and originals store as `ackit status`), never saved: the window
# stores its settings on load and on close, which would overwrite the user's file with this test's ones
w = MainWindow(settings=dict(settings_mod.load(), full_backup=True), runner=Runner(sync=True))
w.save_settings = lambda: None
w.confirm = lambda *a: True
w.editor.confirm = lambda *a: True
errors = []

View File

@@ -14,10 +14,14 @@ $Data = @((Join-Path $env:APPDATA "UnheardCity"), (Join-Path $env:LOCALAPPDATA "
function Say($msg) { if (-not $Quiet) { Write-Host $msg } }
function Modified-Count {
# sounds still changed in the game, according to the originals stores
# sounds and texts (subtitles, menu text) still changed in the game, according to the originals stores
$n = 0
Get-ChildItem -Path (Join-Path $Data[0] "stores\*\store.json") -ErrorAction SilentlyContinue | ForEach-Object {
try { $n += @((Get-Content -Raw -LiteralPath $_.FullName | ConvertFrom-Json).entries.PSObject.Properties).Count } catch {}
try {
$s = Get-Content -Raw -LiteralPath $_.FullName | ConvertFrom-Json
$n += @($s.entries.PSObject.Properties).Count
if ($s.texts) { $n += @($s.texts.PSObject.Properties).Count }
} catch {}
}
return $n
}
@@ -26,8 +30,8 @@ try {
if (Get-Process -Name "UnheardCity" -ErrorAction SilentlyContinue) { throw "$Name is running. Close it first." }
$mods = Modified-Count
if ($mods -gt 0 -and -not $Quiet) {
Write-Host "$mods sound(s) are still modified in the game. To restore the originals, open $Name and use"
Write-Host "'Revert all' BEFORE uninstalling (your originals store is kept either way)."
Write-Host "$mods sound(s) or text(s) are still modified in the game. To restore the originals, open $Name and"
Write-Host "use 'Revert all' BEFORE uninstalling (your originals store is kept either way)."
if ((Read-Host "Uninstall anyway? [y/N]") -notmatch '^(y|yes)$') { exit 1 }
}
if (-not $Quiet -and -not $RemoveUserData -and $mods -eq 0 -and (Test-Path $Data[0])) {