Full correctness check against the docs: safer writes, reverts and texts
Safety - Writes are serialized (Workspace.lock): two background jobs rebuilding one package (a sound and its subtitle applied side by side) lost an edit. - A store copies a bank file or package as "original" only if it matches the shipped catalog (Workspace.unoriginal_files: in-memory sounds, bank copies of voice lines, subtitles); the editor refuses to switch stores while changes are applied. A second store could otherwise resurrect a reverted sound. - Mod import in the editor asks before overwriting files changed outside the tool (it forced silently). - Revert all in the editor reverts texts too; the uninstaller counts text edits before offering to delete the store. - ackit restore-backup: the full-backup restore decisions.md described; it keeps the sounds of WADs without a full copy recorded. - build_catalog also stops when a subtitle differs from the catalog. Languages and texts - Game.language() returns languages without voices (JPN, RUS, ...) and Game.voice_archive() their INT voices; ackit text --lang JPN works. - Menu lookups parse one cached file (find & replace: minutes -> ms). - Subtitles decode cp1252 as Windows does: the UTF-8-in-cp1252 line Speech_TAPE_J_Answer_1_00 (KOR/POL/POR/RUS) is written back byte-for-byte; SUB_KOR/SUB_RUS rebuilt from the original game (only those rows changed). - Mods exported while another language is loaded keep name and line. Other fixes - Down-mix folds extra channels in at -3 dB (a 5.1 centre voice was dropped). - e2e_editor no longer overwrites the user's settings.json; kill_game waits for the game to exit before reverts. - Recorder handles Int32/UInt8 input and split frames; package block cache locked; conflict messages, CLI help examples, stale docstrings and doc references fixed; tests no longer read this PC's game language. Docs (decisions, user guide, README, subtitles.md, banks.md, catalog README, REPORT, NOTES, CLAUDE.md) updated; check_docs: 102 claims match. 148 tests; e2e_features 9/9, e2e_editor and verify PASS; game original afterwards.
This commit is contained in:
@@ -12,7 +12,8 @@ Outputs
|
||||
data/catalog/SUB_locations.csv.gz every stored copy of each subtitle string
|
||||
data/catalog/index.json fingerprints the CSVs were built from + counts
|
||||
Run it only on clean game files (`ackit status` must say nothing is modified). It stops before writing
|
||||
anything if a sound differs from the current catalog (a modified game); after a game update use --accept-changes.
|
||||
anything if a sound or a subtitle differs from the current catalog (a modified game); after a game update use
|
||||
--accept-changes. (Menu text has no catalog.)
|
||||
"""
|
||||
import argparse
|
||||
import collections
|
||||
@@ -88,9 +89,16 @@ def build_characters(game, src_root, final):
|
||||
return rows, unmatched, ambiguous, skipped
|
||||
|
||||
|
||||
def changed_since_catalog(game, out_dir, bank_snd):
|
||||
"""Keys whose original checksum in the existing catalog differs from the game files now."""
|
||||
def changed_since_catalog(game, out_dir, bank_snd, texts=None):
|
||||
"""Keys whose original in the existing catalog differs from the game files now: sounds (checksums) and
|
||||
subtitles (texts: {lang: [SubLine]} scanned now)."""
|
||||
out = []
|
||||
for lang, rows in (texts or {}).items():
|
||||
p = os.path.join(out_dir, "SUB_%s.csv" % lang)
|
||||
if os.path.exists(p):
|
||||
old = {(s.speech, s.line): s.text for s in subtitles.read_texts(p, lang)}
|
||||
out += [subtitles.key(lang, s.speech, s.line) for s in rows
|
||||
if (s.speech, s.line) in old and old[(s.speech, s.line)] != s.text]
|
||||
for arch, path in game.archives.items():
|
||||
p = os.path.join(out_dir, "%s.csv" % arch)
|
||||
if not os.path.exists(p):
|
||||
@@ -133,11 +141,12 @@ def main():
|
||||
for k, d in lines.issues[:10]:
|
||||
print(" issue:", k, d)
|
||||
|
||||
# safety: the catalog records ORIGINAL bytes. A sound that differs from the current catalog means a modified
|
||||
# game (an applied change, another tool) unless the game itself was updated -> stop before writing anything.
|
||||
changed = changed_since_catalog(game, out_dir, bank_snd)
|
||||
# safety: the catalog records ORIGINAL bytes and texts. A sound or subtitle that differs from the current
|
||||
# catalog means a modified game (an applied change, another tool) unless the game itself was updated -> stop
|
||||
# before writing anything.
|
||||
changed = changed_since_catalog(game, out_dir, bank_snd, texts)
|
||||
if changed and not a.accept_changes:
|
||||
print("STOP: %d sound(s) differ from the current catalog, e.g. %s." % (len(changed), changed[:5]))
|
||||
print("STOP: %d sound(s) or subtitle(s) differ from the current catalog, e.g. %s." % (len(changed), changed[:5]))
|
||||
print("Revert all changes (ackit status / ackit revert --all) and run again; after a game update use "
|
||||
"--accept-changes.")
|
||||
sys.exit(2)
|
||||
@@ -193,7 +202,7 @@ def main():
|
||||
"fingerprint": banks.game_fingerprint(game.cooked), "sounds": len(sounds), "groups": dict(g),
|
||||
"named_by_event": by_event, "named_by_bank": len(sounds) - by_event,
|
||||
"copies": sum(1 for l in locs if l.kind == banks.MEMORY), "files": len(sc.files),
|
||||
# bank copies of WAD recordings: prefetch heads and whole in-memory copies (catalog._whole_recordings)
|
||||
# bank copies of WAD recordings: prefetch heads and whole in-memory copies (catalog.whole_recordings)
|
||||
"prefetch_heads": {"recordings": len({l.media for l in heads}), "copies": len(heads),
|
||||
"unmapped": len({l.media for l in heads if not l.sound})},
|
||||
"issues": ["%s: %s" % i for i in sc.issues]}
|
||||
|
||||
@@ -240,6 +240,16 @@ def catalog_claims(cat):
|
||||
west = [enc[L]["w"] for L in langs if L not in ("RUS", "JPN", "KOR", "POL")]
|
||||
claim("docs/formats/subtitles.md", "%d %% of POL are UTF-16; the other languages have %s UTF-16 strings each"
|
||||
% (pol, rng(west)))
|
||||
utf8 = [] # cp1252 strings whose bytes are UTF-8 text (the game's data)
|
||||
for L, t in sorted(texts.items()):
|
||||
for s in t:
|
||||
if s.enc == "a":
|
||||
u = subtitles._encode_cp1252(s.text).decode("utf-8", "replace")
|
||||
multi = sum(1 for c in u if ord(c) > 0x7F and c != "\ufffd") # valid multi-byte characters
|
||||
if multi and u.count("\ufffd") <= multi // 4: # (one corrupt byte tolerated)
|
||||
utf8.append((L, s.speech))
|
||||
claim("docs/formats/subtitles.md", "%d cp1252 strings hold UTF-8 bytes instead (the same line in %s: `%s`"
|
||||
% (len(utf8), ", ".join(L for L, _ in utf8), "`, `".join(sorted({sp for _, sp in utf8}))))
|
||||
with gzip.open(os.path.join(cat, subtitles.LOCATIONS), "rt", newline="") as f:
|
||||
sublocs = sum(1 for _ in csv.DictReader(f))
|
||||
claim("REPORT.md", "Edited in place in every copy (%s copies)" % n(sublocs))
|
||||
@@ -505,6 +515,21 @@ def wem_claims():
|
||||
% (n(total - len(bad)), n(total), exceptions))
|
||||
|
||||
|
||||
def originality_claims(game):
|
||||
"""Workspace.unoriginal_files on every file with sounds or subtitles of the (original) game: no false alarm."""
|
||||
import tempfile
|
||||
from ackit import subtitles
|
||||
from ackit.workspace import Workspace
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
ws = Workspace(game, store_root=td, archives=[game.voice_archive("INT"), "SFX", "BANK"])
|
||||
rels = {l.file for ls in ws.catalog.locations.values() for l in ls}
|
||||
rels |= {l.file for ls in subtitles.read_locations(os.path.join(catalog_dir(), subtitles.LOCATIONS)).values()
|
||||
for l in ls}
|
||||
bad = ws.unoriginal_files([ws.bank_path(r) for r in rels])
|
||||
claim("REPORT.md", "checked on all %s files with sounds or subtitles: %s" % (
|
||||
n(len(rels)), "no false alarm" if not bad else "%d false alarms" % len(bad)), "Workspace.unoriginal_files")
|
||||
|
||||
|
||||
def test_claims():
|
||||
import subprocess
|
||||
out = subprocess.run([sys.executable, "-m", "pytest", "--collect-only", "-q", "-p", "no:cacheprovider"],
|
||||
@@ -535,6 +560,7 @@ def main():
|
||||
if not game:
|
||||
raise SystemExit("--scan needs the game")
|
||||
scan_claims(game)
|
||||
originality_claims(game)
|
||||
if a.wems:
|
||||
if not game:
|
||||
raise SystemExit("--wems needs the game")
|
||||
|
||||
@@ -5,6 +5,8 @@
|
||||
3. start the game under cdb + speaker loopback: the probe must be heard, no crash, setup parsed;
|
||||
4. press Revert (EditorPanel.revert) and check the WAD + pack index are byte-identical to before;
|
||||
5. also apply/revert a speech line in place and check byte identity.
|
||||
Uses the user's settings and originals store (an interrupted run shows up in `ackit status`); the settings file
|
||||
is never written.
|
||||
|
||||
python tools/e2e_editor.py
|
||||
"""
|
||||
@@ -29,11 +31,15 @@ def sha(p):
|
||||
|
||||
def main():
|
||||
from PySide6.QtWidgets import QApplication
|
||||
from ackit import settings as settings_mod
|
||||
from app.mainwindow import MainWindow
|
||||
from app.workers import Runner
|
||||
|
||||
app = QApplication([])
|
||||
w = MainWindow(settings={"full_backup": True}, runner=Runner(sync=True))
|
||||
# the user's settings (same game folder and originals store as `ackit status`), never saved: the window
|
||||
# stores its settings on load and on close, which would overwrite the user's file with this test's ones
|
||||
w = MainWindow(settings=dict(settings_mod.load(), full_backup=True), runner=Runner(sync=True))
|
||||
w.save_settings = lambda: None
|
||||
w.confirm = lambda *a: True
|
||||
w.editor.confirm = lambda *a: True
|
||||
errors = []
|
||||
|
||||
@@ -14,10 +14,14 @@ $Data = @((Join-Path $env:APPDATA "UnheardCity"), (Join-Path $env:LOCALAPPDATA "
|
||||
function Say($msg) { if (-not $Quiet) { Write-Host $msg } }
|
||||
|
||||
function Modified-Count {
|
||||
# sounds still changed in the game, according to the originals stores
|
||||
# sounds and texts (subtitles, menu text) still changed in the game, according to the originals stores
|
||||
$n = 0
|
||||
Get-ChildItem -Path (Join-Path $Data[0] "stores\*\store.json") -ErrorAction SilentlyContinue | ForEach-Object {
|
||||
try { $n += @((Get-Content -Raw -LiteralPath $_.FullName | ConvertFrom-Json).entries.PSObject.Properties).Count } catch {}
|
||||
try {
|
||||
$s = Get-Content -Raw -LiteralPath $_.FullName | ConvertFrom-Json
|
||||
$n += @($s.entries.PSObject.Properties).Count
|
||||
if ($s.texts) { $n += @($s.texts.PSObject.Properties).Count }
|
||||
} catch {}
|
||||
}
|
||||
return $n
|
||||
}
|
||||
@@ -26,8 +30,8 @@ try {
|
||||
if (Get-Process -Name "UnheardCity" -ErrorAction SilentlyContinue) { throw "$Name is running. Close it first." }
|
||||
$mods = Modified-Count
|
||||
if ($mods -gt 0 -and -not $Quiet) {
|
||||
Write-Host "$mods sound(s) are still modified in the game. To restore the originals, open $Name and use"
|
||||
Write-Host "'Revert all' BEFORE uninstalling (your originals store is kept either way)."
|
||||
Write-Host "$mods sound(s) or text(s) are still modified in the game. To restore the originals, open $Name and"
|
||||
Write-Host "use 'Revert all' BEFORE uninstalling (your originals store is kept either way)."
|
||||
if ((Read-Host "Uninstall anyway? [y/N]") -notmatch '^(y|yes)$') { exit 1 }
|
||||
}
|
||||
if (-not $Quiet -and -not $RemoveUserData -and $mods -eq 0 -and (Test-Path $Data[0])) {
|
||||
|
||||
Reference in New Issue
Block a user