build a symbolicated, diffable decompilation of every GCP build

The project had one symbolicated build per TA type, so cross-version comparison was
impossible: there was nothing to compare against. This adds the two pieces that were
missing and proves them on GCP as a pilot.

`fw/mkelf_ta.py` wraps each carved TA body as an ELF IDA can load. The format is read
back off the twelve existing reference ELFs rather than assumed, and the control is
strict: regenerating all fourteen of them is byte-identical, so the generator is not
inventing a layout of its own.

`fw/ta_re.py` ports the reference symbols onto every build and emits whole-program
pseudocode. Names are applied in tiers -- EXACT bytes, MNEM identical mnemonics, FUZZY
similarity above 0.90 -- and anything below the floor is left as sub_*, never guessed. A
signature shared by two differently named functions is treated as ambiguous and skipped
rather than resolved arbitrarily. The gate that makes the rest trustworthy is porting the
reference onto its own build: 112/117 EXACT, and a type whose self-port fails is reported
void instead of emitted.

The output is shaped for diffing, which cost two iterations to get right. Addresses are
omitted and functions ordered by name so the same function lines up across versions;
named and unnamed functions go to separate files, because one function named in build A
and anonymous in B reordered everything after it and turned a 48-function change into
2900 diff lines. Adjacent builds now differ by 1026 lines, consecutive ones by 261.

Chained porting is included and honestly oversold: seeding each build from its nearest
already-named neighbour moved names from MNEM into EXACT, which is a real confidence gain,
but total coverage did not move. The 22.0.0.62-77 branch stays at 52% named because those
builds genuinely diverge -- the function count jumps 119 to 126 -- not because the
reference was too distant. FUZZY names are deliberately not fed back into the chain, so a
single bad guess cannot propagate and look more confident at each hop.

A scare worth recording as resolved: all 78 TA bodies carry a module header whose own
metadata (end == base + length, 78/78) implies the project's ELF base is 0x100 too high,
which would have made every recorded TA address wrong. It is not. The trampoline at
header+0x10 must yield Thumb-bit entry pointers at +0xA0/+0xB0, and across all 78 builds
that resolves only in the 0x400100 frame. The existing convention is now confirmed by
evidence that discriminates between the two frames, unlike the emulator's verify(), which
is self-consistent because its expected values came from harnesses in the same frame.

1 files changed, 433 insertions, 0 deletions

diff --git a/fw/ta_re.py b/fw/ta_re.py
new file mode 100644
index 0000000..d342898
--- /dev/null
+++ b/fw/ta_re.py
@@ -0,0 +1,433 @@
+#!/usr/bin/env python3
+"""Symbolicate and decompile every build of a TA type, in a form that diffs cleanly.
+
+Why this exists: the project had one symbolicated build per TA type, so cross-version
+comparison was impossible -- there was nothing to compare against. `mkelf_ta.py` produced an
+ELF per corpus build; this ports the reference symbols onto each one and emits whole-program
+pseudocode.
+
+## Porting tiers (lifted from psp_out/phase2_port_symbols.py, which proved the approach)
+
+    EXACT  identical function bytes
+    MNEM   identical mnemonic sequence, operands differ (the usual cross-version case:
+           a relinked literal pool or branch displacement)
+    FUZZY  positional mnemonic similarity >= THRESHOLD
+    (none) below threshold -> LEFT AS sub_*, never guessed
+
+A reference signature shared by two or more *differently named* functions is **ambiguous** and
+is not applied at all, rather than picking one arbitrarily.
+
+## The self-validation gate
+
+Porting the reference onto its own build must yield ~100% EXACT. Anything else means the
+matcher, the base address or the body extraction is broken, and the run for that type is
+**void** -- reported as such, not silently emitted. This is the control that catches a
+rebasing error, which would otherwise mislabel every function in every build.
+
+## Why the output is shaped the way it is
+
+Addresses shift between builds, so including them would make every diff hunk noise. Hence:
+  * `decompile(include_addresses=False)`;
+  * functions ordered by **name**, not address, so the same function lines up across versions;
+  * unnamed `sub_*` functions collected at the end, ordered by size then body hash, since
+    their numbering is meaningless across builds;
+  * each function carries its porting tier, so a FUZZY name is never mistaken for a certainty.
+
+usage:
+  ./ta_re.py --type gcp [--limit N] [--outdir fw/decomp] [--validate-only]
+  ./ta_re.py --list
+"""
+from __future__ import annotations
+
+import argparse
+import glob
+import hashlib
+import json
+import os
+import re
+import struct
+import sys
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+REPO = HERE.parent
+sys.path.insert(0, str(Path.home() / "dev" / "ida-tui-maybe"))
+
+ELF_DIR = REPO / "psp_out" / "versions_win" / "elf_all"
+THRESHOLD = 0.90
+
+# Reference build per type: the one that already carries hand-made symbols.
+# 12 Windows types -> fw/libdumps/<t>.json (ea/size/name/hash, richest source).
+LIBDUMP_REF = {
+    "auc": "16.0.0.9", "csu": "42.0.0.6", "fp": "19.0.0.8", "gcp": "22.0.4.50",
+    "merit": "24.0.4.42", "oc": "50.0.0.1", "opm": "25.0.4.48",
+    "powerplay": "34.0.0.20", "rap": "7.0.2.19", "tmz": "29.0.0.4",
+    "wspe": "30.0.4.27", "xgmi": "32.0.0.20",
+}
+
+
+def elf_body(path: str) -> tuple[bytes, int]:
+    raw = Path(path).read_bytes()
+    ph = struct.unpack("<I", raw[28:32])[0]
+    off, va, _pa, fsz = struct.unpack("<IIII", raw[ph + 4:ph + 20])
+    return raw[off:off + fsz], va
+
+
+def mnems(body: bytes, va: int, start: int, size: int) -> tuple[str, ...]:
+    from capstone import Cs, CS_ARCH_ARM, CS_MODE_THUMB
+    md = Cs(CS_ARCH_ARM, CS_MODE_THUMB)
+    off = start - va
+    if off < 0 or off + size > len(body):
+        return ()
+    return tuple(i.mnemonic for i in md.disasm(body[off:off + size], start))
+
+
+def sim(a: tuple, b: tuple) -> float:
+    if not a or not b:
+        return 0.0
+    n, m = min(len(a), len(b)), max(len(a), len(b))
+    return sum(1 for i in range(n) if a[i] == b[i]) / m
+
+
+def vtuple(ver: str) -> tuple[int, ...]:
+    return tuple(int(x) for x in ver.split("."))
+
+
+def vdist(a: str, b: str) -> int:
+    """Version distance, major-weighted, so 22.0.4.50 is near 22.0.4.49 and far from 22.0.0.53."""
+    ta_, tb = vtuple(a), vtuple(b)
+    n = max(len(ta_), len(tb))
+    ta_ = ta_ + (0,) * (n - len(ta_))
+    tb = tb + (0,) * (n - len(tb))
+    w = [10 ** (3 * (n - 1 - i)) for i in range(n)]
+    return sum(wi * abs(x - y) for wi, x, y in zip(w, ta_, tb))
+
+
+POOL_TIERS = ("EXACT", "MNEM")
+
+
+def funcs_as_reference(funcs, assigned, body, va):
+    """Turn a just-processed build into reference material for its neighbours.
+
+    Only EXACT and MNEM names are contributed. A FUZZY name is a similarity judgement, and
+    feeding one back as a seed would let a single bad guess propagate down the whole chain
+    and look more confident at every hop.
+    """
+    out = []
+    for fn in funcs:
+        nm, tier = assigned.get(fn["addr"], (None, None))
+        if not nm or tier not in POOL_TIERS:
+            continue
+        ea, size = int(fn["addr"], 16), fn["size"]
+        off = ea - va
+        if off < 0 or size <= 0 or off + size > len(body):
+            continue
+        b = body[off:off + size]
+        out.append({"name": nm, "ea": ea, "size": size, "bytes": b,
+                    "hash": hashlib.sha256(b).hexdigest(),
+                    "mnems": mnems(body, va, ea, size)})
+    return out
+
+
+def load_reference(ta: str):
+    """-> (ref_elf_path, [ {name,ea,size,bytes,hash,mnems} ])"""
+    ver = LIBDUMP_REF.get(ta)
+    if ver is None:
+        raise SystemExit(f"no reference build registered for type {ta!r}")
+    cands = glob.glob(str(ELF_DIR / f"{ta}_{ver}_*.elf"))
+    if not cands:
+        raise SystemExit(f"reference ELF missing: {ta}_{ver}_*.elf (run mkelf_ta.py --all)")
+    refelf = cands[0]
+    body, va = elf_body(refelf)
+    lib = json.loads((HERE / "libdumps" / f"{ta}.json").read_text())
+    out = []
+    for f in lib:
+        name, size, ea = f.get("name"), f.get("size", 0), f.get("ea")
+        if not name or size <= 0 or ea is None or name.startswith("sub_"):
+            continue
+        off = ea - va
+        if off < 0 or off + size > len(body):
+            continue
+        b = body[off:off + size]
+        out.append({
+            "name": name, "ea": ea, "size": size, "bytes": b,
+            "hash": hashlib.sha256(b).hexdigest(),
+            "mnems": mnems(body, va, ea, size),
+        })
+    return refelf, out
+
+
+def build_index(ref):
+    """Signature -> name, with signatures shared by different names marked ambiguous."""
+    by_hash, by_mnem = {}, {}
+    for r in ref:
+        by_hash.setdefault(r["hash"], set()).add(r["name"])
+        if r["mnems"]:
+            by_mnem.setdefault(r["mnems"], set()).add(r["name"])
+    amb = {"hash": {k for k, v in by_hash.items() if len(v) > 1},
+           "mnem": {k for k, v in by_mnem.items() if len(v) > 1}}
+    return by_hash, by_mnem, amb
+
+
+def match_build(ref, by_hash, by_mnem, amb, tgt_funcs, body, va):
+    """-> {addr_hex: (name, tier)}, stats"""
+    assigned: dict[str, tuple[str, str]] = {}
+    used: set[str] = set()
+    stats = {"EXACT": 0, "MNEM": 0, "FUZZY": 0, "none": 0, "ambiguous": 0}
+    pending = []
+    for fn in tgt_funcs:
+        ea, size = int(fn["addr"], 16), fn["size"]
+        off = ea - va
+        if off < 0 or size <= 0 or off + size > len(body):
+            stats["none"] += 1
+            continue
+        b = body[off:off + size]
+        h = hashlib.sha256(b).hexdigest()
+        if h in amb["hash"]:
+            stats["ambiguous"] += 1
+            continue
+        names = by_hash.get(h)
+        if names:
+            n = next(iter(names))
+            if n not in used:
+                assigned[fn["addr"]] = (n, "EXACT"); used.add(n); stats["EXACT"] += 1
+                continue
+        pending.append((fn, b, ea, size))
+
+    for fn, b, ea, size in pending:
+        mm = mnems(body, va, ea, size)
+        if mm and mm in amb["mnem"]:
+            stats["ambiguous"] += 1
+            continue
+        names = by_mnem.get(mm) if mm else None
+        if names:
+            n = next(iter(names))
+            if n not in used:
+                assigned[fn["addr"]] = (n, "MNEM"); used.add(n); stats["MNEM"] += 1
+                continue
+        best, score = None, 0.0
+        for r in ref:
+            if r["name"] in used or not r["mnems"] or not mm:
+                continue
+            s = sim(mm, r["mnems"])
+            if s > score:
+                best, score = r["name"], s
+        if best and score >= THRESHOLD:
+            assigned[fn["addr"]] = (best, "FUZZY"); used.add(best); stats["FUZZY"] += 1
+        else:
+            stats["none"] += 1
+    return assigned, stats
+
+
+def decompile_all(client, remote_ops, funcs, assigned):
+    rows = []
+    for fn in funcs:
+        name, tier = assigned.get(fn["addr"], (None, "none"))
+        try:
+            r = client.call(remote_ops.decompile, addr=fn["addr"], include_addresses=False)
+            code = r.get("code") if isinstance(r, dict) else None
+            if code is None:
+                code = "/* decompile failed: %s */" % (
+                    (r or {}).get("error", "no code returned"))
+        except Exception as exc:
+            code = "/* decompile failed: %s */" % exc
+        rows.append({"addr": fn["addr"], "size": fn["size"], "name": name,
+                     "tier": tier, "code": code,
+                     "fallback": fn["name"]})
+    return rows
+
+
+REF_TOKEN = re.compile(r"_(?:[0-9A-Fa-f]{6,8})(?:__h\d+)?$|_[0-9A-Fa-f]{8}_")
+
+
+def _banner(fh, meta, nrows, nnamed, nanon, kind):
+    fh.write("/* %s %s  (%s) -- %s\n" % (meta["type"], meta["version"], meta["hash8"], kind))
+    fh.write(" * generated by fw/ta_re.py -- do not edit\n")
+    fh.write(" * reference: %s\n" % meta["reference"])
+    fh.write(" * functions: %d total in build, %d named, %d anonymous\n"
+             % (nrows, nnamed, nanon))
+    fh.write(" * porting tiers: %s\n" % json.dumps(meta["stats"], sort_keys=True))
+    fh.write(" *\n"
+             " * Addresses are omitted and functions are ordered by NAME so that the same\n"
+             " * function lines up across versions. See the .index.json sidecar for\n"
+             " * name -> address.\n"
+             " *\n"
+             " * CAUTION on ported names: a name is only an identity claim at tier=EXACT.\n"
+             " * At tier=MNEM/FUZZY the bytes differ from the reference, so any ADDRESS or\n"
+             " * CONTENT HASH embedded in the name (e.g. *_4025F8, gcp_lib_<hash8>_*) still\n"
+             " * refers to the REFERENCE build and does not describe this one.\n */\n\n")
+
+
+def emit_c(path: Path, meta: dict, rows: list):
+    """Named functions and anonymous ones go to separate files.
+
+    Mixing them defeats the purpose: a function named in build A but anonymous in build B
+    reorders everything after it, so a two-build comparison came out as ~2900 diff lines when
+    only ~48 functions had actually changed. The named file is the one worth diffing.
+    """
+    named = [r for r in rows if r["name"]]
+    anon = [r for r in rows if not r["name"]]
+    named.sort(key=lambda r: r["name"])
+    # sub_* numbering is build-specific and meaningless across versions; order by shape
+    anon.sort(key=lambda r: (r["size"], hashlib.sha256(r["code"].encode()).hexdigest()))
+    with path.open("w") as fh:
+        _banner(fh, meta, len(rows), len(named), len(anon), "named functions")
+        for r in named:
+            flag = "" if r["tier"] == "EXACT" or not REF_TOKEN.search(r["name"]) \
+                   else "  REF-DERIVED-NAME"
+            fh.write("/* ==== %s  [tier=%s size=%d]%s ==== */\n"
+                     % (r["name"], r["tier"], r["size"], flag))
+            fh.write(r["code"].rstrip() + "\n\n")
+    apath = path.with_suffix(".anon.c")
+    with apath.open("w") as fh:
+        _banner(fh, meta, len(rows), len(named), len(anon), "UNNAMED functions")
+        fh.write("/* These did not match the reference above the %.2f similarity floor and are\n"
+                 " * deliberately left unnamed rather than guessed. Ordered by size then body\n"
+                 " * hash, because sub_* numbering is not stable across builds.\n */\n\n"
+                 % THRESHOLD)
+        for r in anon:
+            fh.write("/* ==== ANON size=%d ==== */\n" % r["size"])
+            fh.write(r["code"].rstrip() + "\n\n")
+    idx = {"type": meta["type"], "version": meta["version"], "hash8": meta["hash8"],
+           "stats": meta["stats"],
+           "functions": [{"name": r["name"], "addr": r["addr"], "size": r["size"],
+                          "tier": r["tier"], "ida_name": r["fallback"]} for r in rows]}
+    path.with_suffix(".index.json").write_text(json.dumps(idx, indent=1, sort_keys=True))
+
+
+def run_type(ta: str, outdir: Path, limit: int | None, validate_only: bool) -> int:
+    from idatui import remote_ops
+    from idatui.nexus_client import NexusClient
+
+    refelf, seed_ref = load_reference(ta)
+    _bh, _bm, _amb = build_index(seed_ref)
+    print("type %s: reference %s, %d named reference functions "
+          "(%d ambiguous byte sigs, %d ambiguous mnem sigs)"
+          % (ta, os.path.basename(refelf), len(seed_ref),
+             len(_amb["hash"]), len(_amb["mnem"])), flush=True)
+
+    builds = sorted(glob.glob(str(ELF_DIR / f"{ta}_*.elf")))
+    vers = {}
+    for b in builds:
+        m = re.match(r"^(.+?)_([0-9.]+)_([0-9a-f]{8})\.elf$", os.path.basename(b))
+        if m:
+            vers[b] = m.group(2)
+
+    # CHAINED ORDER: grow outward from the reference, always taking the unprocessed build
+    # closest to something already named. Porting every build from the single distant
+    # reference left the 22.0.0.62-77 branch at 52% named; adjacent builds share far more
+    # exact bytes, so each one is seeded from its nearest already-processed neighbour.
+    order = [refelf]
+    remaining = [b for b in builds if b != refelf and b in vers]
+    while remaining:
+        nxt = min(remaining, key=lambda b: min(vdist(vers[b], vers[d]) for d in order))
+        order.append(nxt)
+        remaining.remove(nxt)
+    if limit:
+        order = order[:limit + 1]
+
+    outdir.mkdir(parents=True, exist_ok=True)
+    gate_ok = None
+    summary = []
+    # reference pool: the seed libdump plus every build processed so far
+    pool = [{"ver": vers.get(refelf, LIBDUMP_REF[ta]), "funcs": seed_ref,
+             "src": os.path.basename(refelf)}]
+    for path in order:
+        name = os.path.basename(path)
+        m = re.match(r"^(.+?)_([0-9.]+)_([0-9a-f]{8})\.elf$", name)
+        if not m:
+            print("  SKIP unparsable name %s" % name); continue
+        _t, ver, h8 = m.groups()
+        # Union of every source processed so far, nearest version first. The union is what
+        # lifts coverage: a function unchanged since an old build matches EXACT against that
+        # build even when the reference has diverged.
+        srcs = sorted(pool, key=lambda s: vdist(ver, s["ver"]))
+        ref = [r for s in srcs for r in s["funcs"]]
+        by_hash, by_mnem, amb = build_index(ref)
+        nearest = srcs[0]["src"] if srcs else "-"
+        body, va = elf_body(path)
+        client = NexusClient(path)
+        try:
+            client.connect(progress=None)
+            funcs, off = [], 0
+            while True:
+                r = client.call(remote_ops.list_funcs,
+                                queries=[{"count": 500, "offset": off}])
+                blk = r["result"][0]
+                funcs.extend(blk["data"])
+                off = blk["next_offset"]
+                if off >= blk["total"]:
+                    break
+            assigned, stats = match_build(ref, by_hash, by_mnem, amb, funcs, body, va)
+            is_ref = (path == refelf)
+            if is_ref:
+                exact_rate = stats["EXACT"] / max(1, len(ref))
+                gate_ok = exact_rate >= 0.95
+                print("  GATE self-port %s: EXACT %d/%d (%.0f%%) -> %s"
+                      % (ver, stats["EXACT"], len(ref), 100 * exact_rate,
+                         "PASS" if gate_ok else "FAIL -- type is VOID"), flush=True)
+                if not gate_ok:
+                    return 2
+                if validate_only:
+                    return 0
+            if assigned:
+                items = [{"addr": a, "name": n} for a, (n, _t2) in assigned.items()]
+                rr = client.call(remote_ops.rename,
+                                 batch={"func": items, "allow_overwrite": False})
+                errs = sum(1 for cat in (rr.get("result") or [])
+                           for row in (cat.get("func") or []) if row.get("error"))
+                if errs:
+                    print("    %d rename errors" % errs, flush=True)
+            rows = decompile_all(client, remote_ops, funcs, assigned)
+            meta = {"type": ta, "version": ver, "hash8": h8, "stats": stats,
+                    "reference": os.path.basename(refelf)}
+            emit_c(outdir / f"{ta}_{ver}_{h8}.c", meta, rows)
+            try:
+                client.save_database()
+            except Exception:
+                pass
+            nf = sum(1 for r in rows if r["code"].startswith("/* decompile failed"))
+            print("  %-34s funcs=%-4d named=%-4d E=%-3d M=%-3d F=%-3d none=%-3d amb=%-3d "
+                  "decomp_fail=%d" % (f"{ver}_{h8}", len(funcs), len(assigned),
+                                      stats["EXACT"], stats["MNEM"], stats["FUZZY"],
+                                      stats["none"], stats["ambiguous"], nf), flush=True)
+            pool.append({"ver": ver, "src": name,
+                         "funcs": funcs_as_reference(funcs, assigned, body, va)})
+            summary.append({"version": ver, "hash8": h8, "funcs": len(funcs),
+                            "nearest_source": nearest,
+                            "named": len(assigned), "stats": stats, "decomp_fail": nf})
+        finally:
+            try:
+                client.close()
+            except Exception:
+                pass
+    (outdir / f"{ta}.summary.json").write_text(json.dumps(summary, indent=1))
+    return 0
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--type", help="TA type, e.g. gcp")
+    ap.add_argument("--list", action="store_true")
+    ap.add_argument("--limit", type=int, default=None,
+                    help="process only the first N non-reference builds")
+    ap.add_argument("--outdir", default=str(HERE / "decomp"))
+    ap.add_argument("--validate-only", action="store_true",
+                    help="run the self-port gate and stop")
+    a = ap.parse_args()
+    if a.list:
+        seen: dict[str, int] = {}
+        for p in sorted(glob.glob(str(ELF_DIR / "*.elf"))):
+            t = os.path.basename(p).split("_")[0]
+            seen[t] = seen.get(t, 0) + 1
+        for k in sorted(seen):
+            print("  %-10s %2d builds%s" % (k, seen[k],
+                  "" if k in LIBDUMP_REF else "   (no libdump reference)"))
+        return 0
+    if not a.type:
+        ap.error("--type or --list required")
+    return run_type(a.type, Path(a.outdir), a.limit, a.validate_only)
+
+
+if __name__ == "__main__":
+    sys.exit(main())