diff options
| author | blasty <blasty@local> | 2026-07-25 23:03:10 +0200 |
|---|---|---|
| committer | blasty <blasty@local> | 2026-07-25 23:03:10 +0200 |
| commit | b8d9aa4fce78f4523cffaba8b46095d2dfb66310 (patch) | |
| tree | 97e73ec6b027a19cbeafe026aaf8b90bad0773d1 | |
| parent | projects phase 3: follow an import into the binary that implements it (diff) | |
| download | ida-tui-b8d9aa4fce78f4523cffaba8b46095d2dfb66310.tar.gz ida-tui-b8d9aa4fce78f4523cffaba8b46095d2dfb66310.tar.xz ida-tui-b8d9aa4fce78f4523cffaba8b46095d2dfb66310.zip | |
projects phase 4: cross-binary back, linkage-guided pre-warm, project verbs
Three things, all following from phase 3 making cross-binary jumps ordinary.
**A cross-binary jump was a one-way door.** Nav history is per-binary, so
arriving in another binary — a project search hit, or now following an import
into the library that implements it — landed you in an empty history with nothing
to take you back. _switch_then_goto records the binary it came FROM, and
action_back falls through to that hop once local history is spent: Esc walks back
through the function you were in, then the binary you were in. Manual Ctrl+O
switching records nothing, because that isn't navigation.
**Pre-warm follows the linkage graph, not list order.** _prewarm_provider warms
the binary providing the most of this one's imports — where a follow is most
likely to go, so its startup is paid before you ask for it. "Next in the list"
would have been arbitrary; phase 3 gave us something better to ask.
pool.prewarm() refuses rather than making room. Evicting a binary the user
visited to speculatively load one they haven't is a straight downgrade, and it
throws away that binary's caches as well; at a tight budget pre-warm just does
nothing. The cost of a worker that doesn't exist yet can only be estimated, so it
uses the largest resident one (same program, different database) — and if that
estimate proves wrong, the speculative worker is the one evicted, never a chosen
one.
**Driving a project.** pane spawn --project FILE [--open BIN]; `binaries` lists
the inventory (active / resident / indexed / where Esc returns to) and `switch
{binary,addr?}` makes another active — with an address it takes the search-hit
path, so it records a hop. state gains `binary` and `hops`, which it should have
had the moment project mode existed.
Verified on real sessions: drive binaries/switch against an echo+cat project
pane; Esc crossing back from a switch; and prewarm on echo+libc picking libc
(provider of echo's imports) and warming it after an evict.
tests: +5 pool (prewarm warms, no-ops when resident, refuses at budget, evicts
nothing when refusing, ignores unknown labels) and +4 project UI (jump records
the hop, Esc crosses back, hop consumed). Confirmed the Esc-back checks fail with
the branch removed. 195/0 scenarios, 27/0 project UI, 36/0 index, 27/0 pool,
33/0 project.
Left open: project-level persistence across sessions.
| -rw-r--r-- | docs/PROJECTS.md | 33 | ||||
| -rw-r--r-- | idatui/app.py | 52 | ||||
| -rw-r--r-- | idatui/drive.py | 26 | ||||
| -rw-r--r-- | idatui/pane.py | 32 | ||||
| -rw-r--r-- | idatui/pool.py | 31 | ||||
| -rw-r--r-- | idatui/rpc.py | 40 | ||||
| -rw-r--r-- | tests/test_pool.py | 29 | ||||
| -rw-r--r-- | tests/test_project_ui.py | 33 |
8 files changed, 264 insertions, 12 deletions
diff --git a/docs/PROJECTS.md b/docs/PROJECTS.md index e0484ad..e5359e0 100644 --- a/docs/PROJECTS.md +++ b/docs/PROJECTS.md @@ -197,9 +197,36 @@ original fix — recognise the thunk and present it as an import ("strcmp — imported, provider not in project") rather than surfacing a decompiler error. Phase 3 covers the valuable half; stub *presentation* stays worth doing. -**Phase 4 — polish.** Background pre-warm, nav-history policy (per-binary to -start; a global stack is a later question), RPC/`drive` verbs (`binaries`, -`switch`), project-level persistence. +**Phase 4 — polish (mostly done).** + +*Nav history is per-binary, with a cross-binary way back.* Each binary keeps its +own stack in `BinaryState`; switching restores it (addresses outlive the worker, +so it survives eviction). But phase 3 made a cross-binary jump an ordinary +action, and arriving in a binary whose history is empty made it a one-way door. +`_switch_then_goto` now records the binary it came FROM, and `action_back` falls +through to that hop once local history is spent. So Esc walks back through the +function you were in, then the binary you were in. Manual switching (Ctrl+O) +records nothing — that's not navigation. + +*Pre-warm follows the linkage graph, not list order.* When a binary finishes +indexing, `_prewarm_provider` warms the binary that provides the most of its +imports — where a follow is most likely to take you, so its startup is paid +before you ask. `WorkerPool.prewarm()` refuses rather than evicting: spending a +binary you visited on one you haven't is a straight downgrade, and it would throw +away that binary's caches too. At a tight budget pre-warm simply does nothing. It +estimates the cost of a not-yet-spawned worker from the largest resident one, +which is the only evidence available; if the estimate proves wrong, the +speculative worker is the one that goes. + +*Driving a project.* `pane spawn --project FILE [--open BIN]` opens a project +pane. `binaries` lists the inventory (active / resident / indexed count / where +Esc returns to) and `switch {binary,addr?}` makes another one active — with an +address it goes through the same path as a search hit, so it records a hop. The +state snapshot now carries `binary` and `hops`. + +Still open: project-level persistence (remember the active binary and each +binary's position across sessions — today a fresh launch auto-lands). + ## Decisions diff --git a/idatui/app.py b/idatui/app.py index 39a4b94..cef58f1 100644 --- a/idatui/app.py +++ b/idatui/app.py @@ -3170,6 +3170,7 @@ class IdaTui(App): self._states: dict[str, BinaryState] = {} self._pending_restore = None # entry to reopen after a switch self._goto_after_switch = None # cross-binary search hit to land on + self._hops: list[str] = [] # binaries a navigation crossed FROM self._nav_seq = 0 # bumped per navigation; drops stale ones # None = teardown wasn't an explicit quit (crash/kill): save defensively. # False = the user chose discard, or we already saved on the way out. @@ -3486,6 +3487,41 @@ class IdaTui(App): self.app.call_from_thread(self._auto_land) self._index_binary() # project mode: keep the cross-binary index fresh + @work(thread=True, exclusive=True, group="prewarm") + def _prewarm_provider(self) -> None: + """Warm the binary this one leans on hardest, once we're idle. + + Not "the next in the list" — phase 3 tells us something better. The + binary providing the most of this one's imports is where a follow is + most likely to take you, so paying its startup now is the switch you + would otherwise wait for. Refuses to evict anything (see pool.prewarm), + so at a tight budget this simply does nothing. + """ + if self._pool is None or self._index is None or self._binary is None: + return + try: + imps, _ = self.program.linkage() + except Exception: # noqa: BLE001 + return + if not imps: + return + from collections import Counter + votes: Counter = Counter() + for name in {i.name for i in imps}: + for h in self._index.providers(name, exclude=self._binary): + votes[h.binary] += 1 + resident = set(self._pool.resident()) + cand = next((b for b, _ in votes.most_common() if b not in resident), None) + if cand is None: + return + n = votes[cand] + try: + if self._pool.prewarm(cand): + self.app.call_from_thread( + self._status, f"pre-warmed {cand} (provides {n} imports)") + except Exception: # noqa: BLE001 -- speculative work must never surface + pass + @work(thread=True, exclusive=True, group="index") def _index_binary(self) -> None: """Fold this binary's symbols + strings into the project index, so it can @@ -3516,6 +3552,7 @@ class IdaTui(App): return self.app.call_from_thread( self._status, f"indexed {self._binary}: {n} symbols, strings + linkage") + self._prewarm_provider() # -- initial landing --------------------------------------------------- # #: function names tried (in order) as the startup landing spot @@ -3695,7 +3732,15 @@ class IdaTui(App): def _switch_then_goto(self, binary: str, addr: int) -> None: """A project-wide hit in another binary: switch to it, then jump there - once its index has loaded.""" + once its index has loaded. + + Records the hop so Esc can come back. Nav history is per-binary, so + without this a cross-binary jump — a project search hit, or following an + import into the library that implements it — is a one-way door: you + arrive in a binary whose history is empty and nothing takes you back. + """ + if self._binary is not None and binary != self._binary: + self._hops.append(self._binary) self._goto_after_switch = addr self._switch_binary(binary) @@ -4117,6 +4162,11 @@ class IdaTui(App): dest = self._nav[-1] self._status(f"\u25c2 back to {dest.name}\u2026") self._open_entry(dest, push=False) + elif self._hops: + # Local history is spent, but we got here from another binary. + label = self._hops.pop() + self._status(f"\u25c2 back to {label}\u2026") + self._switch_binary(label) # _states restores its nav and position elif self.query_one("#left", FunctionsPanel).display: table.focus() else: diff --git a/idatui/drive.py b/idatui/drive.py index 6c96335..b6fa641 100644 --- a/idatui/drive.py +++ b/idatui/drive.py @@ -165,6 +165,31 @@ def cmd_names(c, args): or "(no match)" +def cmd_binaries(c, args): + """Project inventory: which binaries, which is active, which have a live + worker, how much of each is indexed.""" + r = c.call("binaries") + out = [] + for b in r["binaries"]: + mark = "*" if b["active"] else ("~" if b["resident"] else " ") + out.append(f" {mark} {b['label']:<28} indexed={b['indexed']:<7} {b['source']}") + if r.get("hops"): + out.append(f" (Esc returns to: {' <- '.join(r['hops'])})") + return "\n".join(out) + "\n * active ~ worker resident" + + +def cmd_switch(c, args): + if not args: + raise SystemExit("usage: switch <binary> [addr|name]") + p = {"binary": args[0]} + if len(args) > 1: + a = args[1] + p["addr"] = a if a.lower().startswith("0x") else c.call("resolve", name=a)["ea"] + r = c.call("switch", **p) + fn = (r.get("function") or {}).get("name") + return f" now on {r.get('binary') or args[0]}" + (f" @ {fn}" if fn else "") + + def _rename_one(c, old, new): c.call("goto", target=old, delay_ms=0) st = c.call("rename", name=new, word=old, delay_ms=0) @@ -232,6 +257,7 @@ COMMANDS = { "callees": cmd_callees, "callers": cmd_callers, "names": cmd_names, "rename": cmd_rename, "mv": cmd_mv, "note": cmd_note, "retype": cmd_retype, "save": cmd_save, "screen": cmd_screen, "raw": cmd_raw, + "binaries": cmd_binaries, "switch": cmd_switch, } diff --git a/idatui/pane.py b/idatui/pane.py index e34b822..53afef3 100644 --- a/idatui/pane.py +++ b/idatui/pane.py @@ -131,15 +131,20 @@ def spawn(args) -> int: if not os.environ.get("TMUX"): print("error: not inside tmux (spawn creates a tmux pane)", file=sys.stderr) return 2 - if not args.open: - print("error: pass --open <binary>", file=sys.stderr) + if not args.open and not getattr(args, "project", None): + print("error: pass --open <binary> or --project <file>", file=sys.stderr) return 2 sock = args.sock or os.path.join(_sockdir(), f"idatui-{secrets.token_hex(3)}.sock") - target = os.path.abspath(os.path.expanduser(args.open)) - if not os.path.exists(target): + project = (os.path.abspath(os.path.expanduser(args.project)) + if getattr(args, "project", None) else None) + target = os.path.abspath(os.path.expanduser(args.open)) if args.open else None + if target is not None and not os.path.exists(target): print(f"error: no such binary: {target}", file=sys.stderr) return 2 + if project is not None and target is None and not os.path.exists(project): + print(f"error: no such project: {project}", file=sys.stderr) + return 2 # Reap workers leaked by previously-stopped/crashed panes so we don't spawn # into a full IDA_MCP_MAX_WORKERS (which makes the new TUI hang forever, @@ -151,7 +156,15 @@ def spawn(args) -> int: # the command the pane runs: the launcher spawns a private idalib worker for # this binary and becomes the TUI, so kill-pane tears the whole thing down. - inner = [args.python, "-m", "idatui.launch", target, "--rpc", sock] + if project is not None: + # launch takes: --project FILE [binaries...]; extra binaries are added to + # the project (and a missing project file is created from them). + inner = [args.python, "-m", "idatui.launch", "--project", project] + if target is not None: + inner.append(target) + inner += ["--rpc", sock] + else: + inner = [args.python, "-m", "idatui.launch", target, "--rpc", sock] cmd = f"cd {REPO!r} && exec " + " ".join(_q(a) for a in inner) split = ["split-window", "-v" if args.vertical else "-h", @@ -166,8 +179,8 @@ def spawn(args) -> int: split.append(cmd) pane = _tmux(*split) - row = {"sock": sock, "pane": pane, "target": target, - "kind": "open", "started": time.time()} + row = {"sock": sock, "pane": pane, "target": project or target, + "kind": "project" if project else "open", "started": time.time()} reg = [r for r in _load_registry() if r.get("sock") != sock] reg.append(row) _save_registry(reg) @@ -301,8 +314,11 @@ def main(argv: list[str]) -> int: sub = p.add_subparsers(dest="cmd", required=True) sp = sub.add_parser("spawn", help="open a TUI pane and wait until ready") - sp.add_argument("--open", metavar="PATH", required=True, + sp.add_argument("--open", metavar="PATH", help="binary to open (its dir must be writable)") + sp.add_argument("--project", metavar="FILE", + help="project file to open instead of a single binary; " + "any --open paths are added to it (created if absent)") sp.add_argument("--sock", help="RPC socket path (default: auto in $XDG_RUNTIME_DIR)") sp.add_argument("--python", default=DEFAULT_PY, help=f"python for the TUI ({DEFAULT_PY})") sp.add_argument("--vertical", action="store_true", help="split vertically (stacked)") diff --git a/idatui/pool.py b/idatui/pool.py index df852f8..25ea879 100644 --- a/idatui/pool.py +++ b/idatui/pool.py @@ -129,6 +129,37 @@ class WorkerPool: self._enforce_budget(protect=label) return client + def prewarm(self, label: str, progress=None) -> bool: + """Spawn a worker for ``label`` only if it fits the budget AS IT STANDS. + + Pre-warming must never cost residency: evicting a binary the user + actually visited to speculatively load one they haven't is a straight + downgrade, and the eviction would also throw away that binary's caches. + So this refuses rather than making room, and returns False. + + The cost of a worker that doesn't exist yet can only be estimated; the + largest resident one is the best evidence available (they are all the + same program with a different database). With nothing resident we have + no evidence at all, so we allow one — that is the case where the budget + is certainly free. + """ + if label in self._clients: + return False + if self.project.by_label(label) is None: + return False + used = self.memory_mb() + est = max((self._mem(c) for c in self._clients.values()), default=0) + if used + est > self.budget_mb: + return False + self.get(label, progress=progress) + # get() enforces the budget protecting the NEW label; if that had to + # evict, our estimate was wrong and the speculative worker is the one + # that should go — never a binary the user chose. + if self.memory_mb() > self.budget_mb and label != self.active: + self.evict(label) + return False + return True + def _touch(self, label: str) -> None: if label in self._lru: self._lru.remove(label) diff --git a/idatui/rpc.py b/idatui/rpc.py index dd0c9c6..0921b89 100644 --- a/idatui/rpc.py +++ b/idatui/rpc.py @@ -72,6 +72,8 @@ METHODS = { "search": "{term,direction?=1} incremental search in the code view", "select": "{index?} choose the highlighted/nth item in the open modal", "save": "persist the .i64 (Ctrl+S)", + "binaries": "-> project binaries {label,active,resident,indexed} (project mode)", + "switch": "{binary,addr?} make another project binary active (addr also jumps)", "close": "dismiss a modal (Escape)", "move": "{dir,n?=1} fast movement (down/up/.../pagedown)", "cursor": "{line?,col?} set the code-pane cursor directly", @@ -173,7 +175,9 @@ def snapshot(app) -> dict[str, Any]: "cursor": _cursor_info(app, w), "status": st, "filter": app._filter_term, + "binary": app._binary, # None outside project mode "nav_depth": len(app._nav), + "hops": list(getattr(app, "_hops", [])), "dirty": bool(app._dirty), "modal": _modal_snapshot(app), **_readiness(app), @@ -526,6 +530,42 @@ class RpcServer: if method == "functions": return functions(app, params.get("filter"), int(params.get("limit", 50))) + # -- projects ------------------------------------------------------ # + if method == "binaries": + if app._project is None: + raise ValueError("not a project session (launch with --project)") + counts = app._index.counts() if app._index is not None else {} + resident = set(app._pool.resident()) if app._pool is not None else set() + return {"active": app._binary, "hops": list(app._hops), + "binaries": [{"label": r.label, "source": r.source, + "active": r.label == app._binary, + "resident": r.label in resident, + "indexed": int(counts.get(r.label, 0))} + for r in app._project.refs]} + + if method == "switch": + if app._project is None: + raise ValueError("not a project session (launch with --project)") + label = str(params.get("binary") or params.get("label") or "") + if app._project.by_label(label) is None: + have = ", ".join(r.label for r in app._project.refs) + raise ValueError(f"no such binary {label!r} (have: {have})") + addr = params.get("addr") + if label == app._binary and addr is None: + return snapshot(app) + if addr is None: + app._switch_binary(label) + else: + # Same path a project search hit takes, so it records a hop and + # Esc comes back here. + app._switch_then_goto(label, int(str(addr), 0) + if isinstance(addr, str) else int(addr)) + await settle(app, lambda: app._binary == label + and app._func_index is not None + and app._func_index.complete, + timeout=float(params.get("timeout", 300.0))) + return snapshot(app) + # -- structured introspection (heavy: run off the UI loop) -------- # loop = asyncio.get_running_loop() if method == "pseudocode": diff --git a/tests/test_pool.py b/tests/test_pool.py index d4347d9..5c6e2c4 100644 --- a/tests/test_pool.py +++ b/tests/test_pool.py @@ -155,6 +155,35 @@ def main() -> int: check("default budget is derived, not a fixed worker count", pool3.budget_mb >= 256, pool3.budget_mb) + # -- prewarm: speculative, and never at the cost of a real binary ------ # + with tempfile.TemporaryDirectory() as tmp: + proj = _mkproject(tmp, n=3) + made2 = {} + + def spawn2(ref, ttl): + c = FakeClient(ref, mem=100) + made2[ref.label] = c + return c + + pool = WorkerPool(proj, budget_mb=250, spawn=spawn2, + mem_fn=lambda c: c.mem) + labels = [r.label for r in proj.refs] + a, b, c_ = labels[0], labels[1], labels[2] + pool.get(a) + pool.set_active(a) + check("prewarm warms a binary when the budget has room", + pool.prewarm(b) is True and b in pool.resident(), f"{pool.resident()}") + check("prewarm is a no-op for something already resident", + pool.prewarm(b) is False) + # 2 x 100MB resident, estimate 100 more -> 300 > 250: must refuse + check("prewarm refuses rather than making room", + pool.prewarm(c_) is False and c_ not in pool.resident(), + f"resident={pool.resident()} mem={pool.memory_mb()}/{pool.budget_mb}") + check("refusing to prewarm evicts nothing", + set(pool.resident()) == {a, b}, f"{pool.resident()}") + check("prewarm ignores a label outside the project", + pool.prewarm("nope") is False) + print(f"\n{PASS} passed, {FAIL} failed") return 1 if FAIL else 0 diff --git a/tests/test_project_ui.py b/tests/test_project_ui.py index 731811d..47e511a 100644 --- a/tests/test_project_ui.py +++ b/tests/test_project_ui.py @@ -168,6 +168,39 @@ async def run(bins): await pilot.press("escape") await pilot.pause(0.2) + # -- a cross-binary jump is not a one-way door ----------------- # + # Nav history is per-binary, so arriving in another binary lands you + # in an empty history. Esc must fall through to the binary you came + # from, or a project search hit (and, since phase 3, following an + # import) strands you. + here, there = app._binary, (first if app._binary == second else second) + hops0 = len(app._hops) + target = app._index.search("main", limit=200) + tgt = next((h for h in target if h.binary == there), None) + if tgt is None: + check("cross-binary jump records a hop", False, "no hit in the other binary") + else: + app._switch_then_goto(tgt.binary, tgt.addr) + jumped = await settle(lambda: app._binary == there + and app._func_index is not None + and app._func_index.complete, 180) + check("a project hit switches to the other binary", jumped, + f"binary={app._binary} want={there}") + check("the jump records where it came from", + len(app._hops) == hops0 + 1 and app._hops[-1] == here, + f"hops={app._hops}") + # spend the local history first, then Esc must cross back + for _ in range(6): + if not app._hops or app._binary != there: + break + await pilot.press("escape") + await pilot.pause(0.6) + returned = await settle(lambda: app._binary == here, 180) + check("Esc crosses back to the binary the jump came from", + returned, f"binary={app._binary} want={here} hops={app._hops}") + check("the hop is consumed, not repeated", + not app._hops, f"hops={app._hops}") + # -- and the same toggle for strings --------------------------- # from idatui.app import StringsPalette await pilot.press("quotation_mark") |
