diff options
| author | blasty <blasty@local> | 2026-07-25 11:26:11 +0200 |
|---|---|---|
| committer | blasty <blasty@local> | 2026-07-25 11:26:11 +0200 |
| commit | 54e5aab945d6c27edcaf7900c5d328384143fab2 (patch) | |
| tree | c69046f7af4f93b75ed15c5e24ba969d64a3abdf | |
| parent | split: propagate pure scrolls (wheel/scrollbar) to the companion pane (diff) | |
| download | ida-tui-54e5aab945d6c27edcaf7900c5d328384143fab2.tar.gz ida-tui-54e5aab945d6c27edcaf7900c5d328384143fab2.tar.xz ida-tui-54e5aab945d6c27edcaf7900c5d328384143fab2.zip | |
projects: project model + binary staging (phase 1a)
First slice of multi-binary projects (docs/PROJECTS.md): the on-disk model, with
no runtime wiring yet.
idatui/project.py (stdlib-only, like domain/worker):
* Project.load/create/save — an explicit JSON project file listing binaries;
paths resolve relative to it, labels default to the basename and are
disambiguated on collision (they name files).
* A sidecar dir beside the project file (<stem>.idatui.d/) holds bin/ (staged
binaries), their .i64 + scratch, and idx/ for phase 2. IDA opens the STAGED
file, so nothing lands in the source tree — today targets/ carries ~244MB of
IDA litter around ~13MB of binaries, much of it stale wedge files.
* stage() copies rather than hardlinks. A hardlink is free but makes source and
staged one inode, so an in-place rebuild (cp over the path truncates instead of
replacing) would silently swap the bytes under an analysed DB with nothing to
detect it. The unit test caught exactly that. A copy also leaves the sidecar
self-contained once the sources are gone.
* Re-staging a changed source drops its now-stale DB; sweep_scratch() clears the
unpacked working files a hard-killed worker leaves behind (never the .i64).
tests/test_project.py: 27 checks, pure stdlib (no IDA/textual/worker), <1s.
docs/PROJECTS.md: the full design — the one-worker-per-DB constraint with
measured costs (bash worker = 126MB RSS/117MB PSS; libcrypto's DB is 72MB, so
residency is budgeted by MEMORY, not a worker count), the switch between
"switching needs a live worker" and "searching doesn't (cached index)", and
phases 1-4.
| -rw-r--r-- | docs/PROJECTS.md | 143 | ||||
| -rw-r--r-- | idatui/project.py | 258 | ||||
| -rw-r--r-- | tests/test_project.py | 158 |
3 files changed, 559 insertions, 0 deletions
diff --git a/docs/PROJECTS.md b/docs/PROJECTS.md new file mode 100644 index 0000000..d1e2806 --- /dev/null +++ b/docs/PROJECTS.md @@ -0,0 +1,143 @@ +# Projects — grouping many binaries under one target + +RE'ing a real target means several binaries at once: a daemon, the libraries it +links, maybe a loader. IDA has no notion of this (one IDB, one window); Ghidra +does. A **project** groups binaries so you can switch between them instantly, +search across all of them, and (later) follow calls from one into another. + +## The constraint that shapes everything + +`idatui/worker.py` is `serve(sock, binpath)` — **one worker process holds exactly +one database** (idalib is main-thread-only and single-DB). So N binaries = N +worker processes, each with the analyzed DB resident. + +Measured cost (this box, `targets/`): + +| binary | size | `.i64` | worker RSS / PSS | +|--------|------|--------|------------------| +| `bash` | 1.2 MB | 14 MB | **126 MB / 117 MB** | +| `libcrypto.so.3` | 5.9 MB | 72 MB | several hundred MB (DB working set dominates) | + +PSS ≈ RSS, so workers share almost nothing — the Nth worker costs roughly its +full footprint. A ~120 MB baseline (Python + the idalib kernel) dominates for +small binaries; the DB working set dominates for big ones. + +**Therefore residency is budgeted by memory, not by a fixed worker count** — a +count is the wrong knob when one project holds both a 50 KB helper and a 6 MB +crypto library. + +## The key split + +Two capabilities that feel like one, but aren't: + +1. **Switching** to a binary needs a *live worker*. +2. **Searching across** binaries does *not* — if a per-binary index (functions, + strings, imports/exports) is cached on disk. + +That split is the unlock: project-wide search stays instant across every binary, +including ones never opened this session, and only *jumping* to a hit costs a +worker spawn. + +## Layout + +The project file is explicit (`--project router-fw.json`); everything IDA +generates lives in a sidecar dir beside it, so **source trees stay pristine**: + +``` +router-fw.json # the project file +router-fw.idatui.d/ + bin/httpd # staged copy of the source binary + bin/httpd.i64 # IDA's DB + scratch land here automatically + idx/httpd.json # cached index (phase 2) +``` + +Binaries are **staged** into `bin/` and IDA opens the staged file, so the `.i64` +and all the `.id0/.id1/.id2/.nam/.til` scratch are created there rather than next +to the original. (Today they pile up beside the source: `targets/` holds ~244 MB +of IDA litter around ~13 MB of binaries, much of it stale wedge files from +hard-killed workers.) A source whose size/mtime no longer matches the staged copy +is re-staged, and its now-stale DB dropped. + +Staging **copies** rather than hardlinks. A hardlink is free, but source and +staged would be one inode: an in-place rebuild (`cp newbuild /path/bin` +truncates rather than replacing) would silently swap the bytes under an already +analysed database, with nothing to detect it. A copy costs a fraction of the +`.i64` it will grow, and leaves the sidecar self-contained — the project still +opens after the sources go away (an unmounted firmware image, a cleaned build +tree). *The unit test caught this: with hardlinks an in-place rewrite never went +stale.* + +```json +{ + "name": "router-fw", + "binaries": [ + {"path": "/abs/sbin/httpd", "label": "httpd"}, + {"path": "../lib/libauth.so"} + ], + "memory_pct": 25 +} +``` + +Relative paths resolve against the project file. `label` defaults to the +basename and must be unique (it names the staged file). + +## Runtime + +- **`WorkerPool`** — one `WorkerClient` per binary, spawned lazily on first + switch, kept resident until the memory budget is exceeded, then LRU-evicted. + Eviction **saves the DB first**, so returning to a binary is a DB load, not a + re-analysis. Binaries can be pinned to stay resident. +- **`BinaryState`** — per binary: `client, program, nav, cur, func_index, + pref/active/split, filter`. Switching snapshots the current state and restores + the target's. `_after_reconnect` already does exactly this swap (client + + program, reload the index, re-open the entry) — switching reuses that seam. +- **Clean shutdown** — the worker currently does `close_database(save=False)` and + is hard-killed on exit, which is why wedge files accumulate. Projects need + save-on-evict and an orderly close anyway, so that gets fixed here. + +## UI + +- **Binary switcher** — a palette (mirroring `SymbolPalette`/`StringsPalette`) + listing each binary with resident/cold state and function count; Enter + switches. +- **Status bar** shows the active binary. +- **Scope toggle** — the existing symbol (`Ctrl+N`) and strings (`"`) palettes + gain a this-binary ↔ whole-project toggle rather than new keys, so there's one + mental model. Project-scope hits are prefixed with the binary label; Enter + switches binary *and* jumps. + +## Phases + +**Phase 1 — project model + switching.** Project file, staging dir, +`WorkerPool` (lazy spawn, budget eviction, save-on-evict, clean shutdown), +`BinaryState` save/restore, switcher palette, active binary in the status bar, +`--project` CLI. One active binary; no cross-binary search yet. + +**Phase 2 — index cache + project-wide search.** Per-binary index (functions, +strings) persisted after first open, keyed by source size+mtime. Scope toggle in +the symbol/strings palettes, working for never-opened binaries. + +**Phase 3 — cross-binary linking.** Import/export index; "who in the project +calls this export"; follow an import stub in A into its implementation in B. +Subsumes the standing PLT/import-stub backlog item — today following a libc call +dead-ends on `extrn X:near`. + +**Phase 4 — polish.** Background pre-warm, nav-history policy (per-binary to +start; a global stack is a later question), RPC/`drive` verbs (`binaries`, +`switch`), project-level persistence. + +## Decisions + +- Residency is a **memory budget** (default 25% of RAM, `memory_pct`), not a + worker count; pinning supported. +- **Nav history is per-binary** to start. +- Search scope is a **toggle inside the existing palettes**. +- The project file is **explicit** (`--project`), not auto-discovered. +- IDA artifacts are **centralized** in the sidecar dir via staging. + +## Open questions + +- Re-staging a changed source drops its DB (renames included) — right call, but + it should warn loudly. +- Should `Esc`/back ever cross binaries (global history)? Deferred. +- Pre-warm policy: analyze the whole project up front, or purely on demand? diff --git a/idatui/project.py b/idatui/project.py new file mode 100644 index 0000000..2d417df --- /dev/null +++ b/idatui/project.py @@ -0,0 +1,258 @@ +"""Multi-binary projects: group the binaries of one target under a project file. + +A project is an explicit JSON file plus a sidecar directory beside it:: + + router-fw.json # the project file + router-fw.idatui.d/ + bin/httpd # hardlink (or copy) of the source binary + bin/httpd.i64 # IDA's DB + scratch land here automatically + idx/httpd.json # cached index (phase 2) + +Binaries are **staged** into ``bin/`` and IDA opens the staged file, so the +``.i64`` and every ``.id0/.id1/.id2/.nam/.til`` scratch file are created inside +the sidecar instead of next to the original. Source trees stay pristine. + +Staging **copies** rather than hardlinks. A hardlink would be free, but source +and staged would be one inode: an in-place rebuild (``cp newbuild /path/bin`` +truncates instead of replacing) would silently swap the bytes under an already +analysed database, with nothing to detect it. A copy costs a fraction of the +``.i64`` it will grow, decouples the two completely, and leaves the sidecar +self-contained — the project still opens after the sources go away (an unmounted +firmware image, a cleaned build tree). + +A source whose size/mtime no longer matches the staged copy is re-staged, and its +now-stale database is dropped (the DB describes the old bytes). + +stdlib-only, like the domain/worker layers — the TUI is the only Textual consumer. +""" +from __future__ import annotations + +import json +import os +import shutil +from dataclasses import dataclass + +SIDECAR_SUFFIX = ".idatui.d" +DEFAULT_MEMORY_PCT = 25 + +#: The database plus the working files IDA unpacks beside it while it's open. +DB_SUFFIXES = (".i64", ".idb", ".id0", ".id1", ".id2", ".nam", ".til") +#: Just the unpacked scratch — safe to delete when no worker holds the DB. +SCRATCH_SUFFIXES = (".id0", ".id1", ".id2", ".nam", ".til") + + +class ProjectError(Exception): + """A malformed project file, or a binary that can't be staged.""" + + +@dataclass(frozen=True) +class BinaryRef: + """One binary in a project: where it came from, and where IDA works on it.""" + + label: str # unique within the project; names the staged file + source: str # absolute path to the original binary + staged: str # absolute path IDA actually opens (inside the sidecar) + + @property + def db(self) -> str: + """The database IDA creates for the staged file.""" + return self.staged + ".i64" + + +def _stat_key(path: str) -> tuple[int, int] | None: + """(size, mtime) identity used to spot a source that changed under us.""" + try: + st = os.stat(path) + except OSError: + return None + return (st.st_size, int(st.st_mtime)) + + +def _unlink(path: str) -> bool: + try: + os.remove(path) + return True + except OSError: + return False + + +class Project: + """A set of binaries analysed together, with all IDA artifacts corralled.""" + + def __init__(self, path: str, name: str, entries: list[dict], + memory_pct: int = DEFAULT_MEMORY_PCT) -> None: + self.path = os.path.abspath(os.path.expanduser(path)) + self.name = name + self.memory_pct = memory_pct + self._entries = entries # raw, as written to the file + self._refs = self._build_refs() + + # -- construction ------------------------------------------------------ # + @classmethod + def load(cls, path: str) -> "Project": + path = os.path.abspath(os.path.expanduser(path)) + try: + with open(path) as f: + raw = json.load(f) + except OSError as e: + raise ProjectError(f"cannot read project {path}: {e}") from e + except ValueError as e: + raise ProjectError(f"malformed project {path}: {e}") from e + if not isinstance(raw, dict): + raise ProjectError(f"malformed project {path}: expected an object") + entries = raw.get("binaries") + if not isinstance(entries, list) or not entries: + raise ProjectError(f"project {path} lists no binaries") + norm: list[dict] = [] + for e in entries: + if isinstance(e, str): + e = {"path": e} + if not isinstance(e, dict) or not e.get("path"): + raise ProjectError(f"project {path}: bad binary entry {e!r}") + norm.append({k: e[k] for k in ("path", "label") if e.get(k)}) + name = raw.get("name") or os.path.splitext(os.path.basename(path))[0] + try: + pct = int(raw.get("memory_pct", DEFAULT_MEMORY_PCT)) + except (TypeError, ValueError): + pct = DEFAULT_MEMORY_PCT + return cls(path, str(name), norm, max(1, min(pct, 90))) + + @classmethod + def create(cls, path: str, binaries: list[str], name: str | None = None, + memory_pct: int = DEFAULT_MEMORY_PCT) -> "Project": + """Write a new project file listing ``binaries`` (an ad-hoc project).""" + if not binaries: + raise ProjectError("a project needs at least one binary") + entries = [{"path": os.path.abspath(os.path.expanduser(b))} + for b in binaries] + path = os.path.abspath(os.path.expanduser(path)) + proj = cls(path, name or os.path.splitext(os.path.basename(path))[0], + entries, memory_pct) + proj.save() + return proj + + def save(self) -> None: + data = {"name": self.name, "memory_pct": self.memory_pct, + "binaries": self._entries} + tmp = self.path + ".tmp" + os.makedirs(os.path.dirname(self.path) or ".", exist_ok=True) + with open(tmp, "w") as f: + json.dump(data, f, indent=2) + f.write("\n") + os.replace(tmp, self.path) + + # -- layout ------------------------------------------------------------ # + @property + def sidecar(self) -> str: + """The directory holding staged binaries, databases and indexes.""" + return os.path.splitext(self.path)[0] + SIDECAR_SUFFIX + + @property + def bin_dir(self) -> str: + return os.path.join(self.sidecar, "bin") + + @property + def index_dir(self) -> str: + return os.path.join(self.sidecar, "idx") + + def index_path(self, ref: BinaryRef) -> str: + """Where the cached per-binary index lives (phase 2).""" + return os.path.join(self.index_dir, ref.label + ".json") + + # -- binaries ---------------------------------------------------------- # + def _build_refs(self) -> tuple[BinaryRef, ...]: + base = os.path.dirname(self.path) + refs: list[BinaryRef] = [] + used: set[str] = set() + for e in self._entries: + src = os.path.expanduser(str(e["path"])) + if not os.path.isabs(src): # relative to the project file + src = os.path.join(base, src) + src = os.path.abspath(src) + label = str(e.get("label") or os.path.basename(src)) or "binary" + label = label.replace("/", "_").replace(os.sep, "_") + if label in used: # labels name files; keep them unique + stable + n = 2 + while f"{label}_{n}" in used: + n += 1 + label = f"{label}_{n}" + used.add(label) + refs.append(BinaryRef(label=label, source=src, + staged=os.path.join(self.bin_dir, label))) + return tuple(refs) + + @property + def refs(self) -> tuple[BinaryRef, ...]: + return self._refs + + def by_label(self, label: str) -> BinaryRef | None: + return next((r for r in self._refs if r.label == label), None) + + def add(self, binary: str, label: str | None = None) -> BinaryRef: + entry = {"path": os.path.abspath(os.path.expanduser(binary))} + if label: + entry["label"] = label + self._entries.append(entry) + self._refs = self._build_refs() + return self._refs[-1] + + def remove(self, label: str) -> bool: + ref = self.by_label(label) + if ref is None: + return False + i = self._refs.index(ref) + del self._entries[i] + self._refs = self._build_refs() + return True + + # -- staging ----------------------------------------------------------- # + def is_stale(self, ref: BinaryRef) -> bool: + """True when the staged file is missing or no longer matches its source.""" + staged = _stat_key(ref.staged) + return staged is None or staged != _stat_key(ref.source) + + def stage(self, ref: BinaryRef) -> str: + """Ensure ``ref`` is staged in the sidecar; returns the staged path. + + Re-staging a changed source drops its database: the DB describes the old + bytes, so keeping it would silently mismatch the disassembly (any renames + in it are lost, which is why callers should say so out loud). + """ + if not os.path.isfile(ref.source): + raise ProjectError(f"no such binary: {ref.source}") + if not self.is_stale(ref): + return ref.staged + os.makedirs(self.bin_dir, exist_ok=True) + tmp = ref.staged + ".staging" + _unlink(tmp) + # copy2 (not link): keeps size+mtime so freshness compares cleanly, while + # leaving the staged bytes immune to an in-place rewrite of the source. + shutil.copy2(ref.source, tmp) + os.replace(tmp, ref.staged) + for suf in DB_SUFFIXES: # the old DB describes the old bytes + _unlink(ref.staged + suf) + return ref.staged + + def stage_all(self, progress=None) -> list[BinaryRef]: + out = [] + for ref in self._refs: + if progress is not None: + progress(f"staging {ref.label}\u2026") + self.stage(ref) + out.append(ref) + return out + + def sweep_scratch(self, ref: BinaryRef) -> int: + """Delete IDA's unpacked working files (never the ``.i64``) for ``ref``. + + A hard-killed worker leaves them behind and the database then refuses to + reopen. Only safe when no worker holds it. + """ + return sum(1 for suf in SCRATCH_SUFFIXES if _unlink(ref.staged + suf)) + + def has_db(self, ref: BinaryRef) -> bool: + """True once the binary has been analysed and saved at least once.""" + return os.path.isfile(ref.db) + + def __repr__(self) -> str: # pragma: no cover - debug aid + return f"<Project {self.name!r} {len(self._refs)} binaries {self.path}>" diff --git a/tests/test_project.py b/tests/test_project.py new file mode 100644 index 0000000..1f93d4a --- /dev/null +++ b/tests/test_project.py @@ -0,0 +1,158 @@ +#!/usr/bin/env python3 +"""Unit tests for idatui.project (the multi-binary project model + staging). + +Pure stdlib: no IDA, no textual, no worker — runs anywhere in under a second. + + python tests/test_project.py +""" +import json +import os +import sys +import tempfile + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from idatui.project import Project, ProjectError, SIDECAR_SUFFIX # noqa: E402 + +PASS = FAIL = 0 + + +def check(name, cond, detail=""): + global PASS, FAIL + if cond: + PASS += 1 + print(f" ok {name}") + else: + FAIL += 1 + print(f" FAIL {name} {detail}") + + +def _bin(path, content=b"\x7fELF fake binary"): + with open(path, "wb") as f: + f.write(content) + return path + + +def main() -> int: + with tempfile.TemporaryDirectory() as tmp: + src = os.path.join(tmp, "src") + os.makedirs(src) + httpd = _bin(os.path.join(src, "httpd")) + libauth = _bin(os.path.join(src, "libauth.so"), b"\x7fELF lib") + pfile = os.path.join(tmp, "router-fw.json") + + # -- create / load round-trip ------------------------------------- # + proj = Project.create(pfile, [httpd, libauth], name="router-fw") + check("create() writes the project file", os.path.isfile(pfile)) + proj = Project.load(pfile) + check("load() round-trips name + binaries", + proj.name == "router-fw" and len(proj.refs) == 2, + f"name={proj.name} n={len(proj.refs)}") + check("labels default to the basename", + [r.label for r in proj.refs] == ["httpd", "libauth.so"], + f"{[r.label for r in proj.refs]}") + + # -- layout -------------------------------------------------------- # + check("sidecar sits beside the project file", + proj.sidecar == os.path.join(tmp, "router-fw" + SIDECAR_SUFFIX), + proj.sidecar) + ref = proj.by_label("httpd") + check("staged path lives in the sidecar, not the source tree", + ref.staged.startswith(proj.bin_dir) and src not in ref.staged, + ref.staged) + check("db path hangs off the staged file", ref.db == ref.staged + ".i64") + + # -- staging: hardlink, freshness ---------------------------------- # + check("a binary starts out stale (not yet staged)", proj.is_stale(ref)) + proj.stage(ref) + check("stage() materialises the binary in the sidecar", + os.path.isfile(ref.staged)) + check("stage() copies (distinct inode) so the source can't be mutated " + "through it", + os.stat(ref.staged).st_ino != os.stat(ref.source).st_ino + and open(ref.staged, "rb").read() == open(ref.source, "rb").read()) + check("a staged binary is no longer stale", not proj.is_stale(ref)) + check("source tree stays clean (no IDA artifacts beside it)", + sorted(os.listdir(src)) == ["httpd", "libauth.so"], + f"{sorted(os.listdir(src))}") + + # -- a changed source re-stages and drops the stale DB ------------- # + open(ref.db, "wb").write(b"fake i64") + open(ref.staged + ".id0", "wb").write(b"scratch") + check("has_db() sees the analysed database", proj.has_db(ref)) + # rewritten IN PLACE (same inode) — the case a hardlink would miss + _bin(ref.source, b"\x7fELF fake binary v2 (longer)") + os.utime(ref.source, (1, 1)) + check("a source rebuilt in place goes stale", proj.is_stale(ref)) + proj.stage(ref) + check("re-staging refreshes the staged bytes", + open(ref.staged, "rb").read().endswith(b"v2 (longer)")) + check("re-staging drops the now-stale database", not proj.has_db(ref)) + check("re-staging drops the stale scratch too", + not os.path.exists(ref.staged + ".id0")) + + # -- scratch sweep keeps the DB ------------------------------------ # + open(ref.db, "wb").write(b"fake i64") + for suf in (".id0", ".id1", ".nam"): + open(ref.staged + suf, "wb").write(b"x") + n = proj.sweep_scratch(ref) + check("sweep_scratch() removes the unpacked working files", n == 3, f"n={n}") + check("sweep_scratch() never touches the .i64", proj.has_db(ref)) + + # -- relative paths + label collisions ----------------------------- # + sub = os.path.join(tmp, "other") + os.makedirs(sub) + dup = _bin(os.path.join(sub, "httpd"), b"\x7fELF other httpd") + with open(pfile, "w") as f: + json.dump({"name": "p", "binaries": [ + {"path": "src/httpd"}, # relative to the project file + {"path": dup}, # same basename -> collision + {"path": libauth, "label": "auth"}, + ]}, f) + proj2 = Project.load(pfile) + check("relative paths resolve against the project file", + proj2.refs[0].source == httpd, proj2.refs[0].source) + check("colliding labels are disambiguated", + [r.label for r in proj2.refs] == ["httpd", "httpd_2", "auth"], + f"{[r.label for r in proj2.refs]}") + check("explicit labels are honoured", proj2.by_label("auth") is not None) + proj2.stage_all() + check("stage_all() stages every binary to a distinct file", + len({r.staged for r in proj2.refs}) == 3 + and all(os.path.isfile(r.staged) for r in proj2.refs)) + + # -- add / remove ---------------------------------------------------- # + extra = _bin(os.path.join(src, "extra")) + proj2.add(extra, label="extra") + check("add() appends a binary", proj2.by_label("extra") is not None) + check("remove() drops one", proj2.remove("extra") + and proj2.by_label("extra") is None) + + # -- bad input ------------------------------------------------------- # + bad = os.path.join(tmp, "bad.json") + with open(bad, "w") as f: + f.write("{not json") + try: + Project.load(bad) + check("malformed JSON raises ProjectError", False, "no raise") + except ProjectError: + check("malformed JSON raises ProjectError", True) + with open(bad, "w") as f: + json.dump({"binaries": []}, f) + try: + Project.load(bad) + check("an empty project raises ProjectError", False, "no raise") + except ProjectError: + check("an empty project raises ProjectError", True) + missing = proj2.add(os.path.join(tmp, "nope")) + try: + proj2.stage(missing) + check("staging a missing binary raises ProjectError", False, "no raise") + except ProjectError: + check("staging a missing binary raises ProjectError", True) + + print(f"\n{PASS} passed, {FAIL} failed") + return 1 if FAIL else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) |
