From c9208de05d8583b677117fe43c9d3567e89eb2ce Mon Sep 17 00:00:00 2001 From: blasty Date: Fri, 7 Aug 2026 12:39:54 +0200 Subject: Rebase MISTER EXO's ida-codemode port onto the current tree Mechanical part of the port: the 27-file patch was cut against a base ~148 commits behind us, so it did not apply. Resolved 11 conflicts (all of them diff drift, not semantic clashes) and the three file deletions: - app.py: the patch re-inserted _do_rename/_do_name_addr/_seek_split etc. as "theirs" because our tree moved them to edit_ctl.py/trace_ctl.py. Kept ours and applied the real intent (WorkerClient->CodeModeClient, .call->.invoke, _open_worker_client->_open_database_client) at their current homes. - domain.py: kept Head as a NamedTuple -- the patch reverted it to a frozen dataclass, which the perf work measured at 2.9us vs 1.9us per row on a quarter-million-row walk. Dropped _fetch_output (no download_url under Code Mode) and its now-dead urllib/json imports. - pane.py: the patch's deletion swallowed our zellij support along with the worker-reaping block it meant to remove. Kept zellij, removed the reaping. - test_scenarios.py: the idb_save->save_database teardown hunk belongs to tests/_fixtures.py in our tree; applied it there and kept our pc_num_format scenario that the drift landed on. Three defects in the patch itself, fixed here: - It made "import idatui" hard-require ida_codemode, so every offline suite died at import -- including the pure ones (graph/index/trace) that are the house rule for "tests/run.py --fast". The import is now deferred and gated on the binding, which is also what lets the port's own contract tests inject a fake DatabaseHandle. - project.stage() inlined an ida_codemode.registry import and treated "library not installed" as "someone owns this database", which broke IDA-free project staging. Ownership lookup moved to codemode_client.database_owner(). - tests/test_codemode_client.py had no NEEDS_IDA marker, which tests/run.py rejects outright. Offline suite: 301 passed, 0 failed. Against master's 344 the whole delta is accounted for: -40 worker_client (module deleted), -18 launch sweep checks (behaviour deliberately removed) +3 guarding that it stays removed, +2 pool (GUI-save semantics), +13 new codemode_client contract tests. NOT yet done, and the port is not functional without it: the adapter is missing five operations our tree grew since the patch's base (flowchart, op_format, pc_nums, pc_num_format, survey_binary) and its "heads" predates back-walking and digest/expect. --- docs/PROJECTS.md | 41 +++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 20 deletions(-) (limited to 'docs/PROJECTS.md') diff --git a/docs/PROJECTS.md b/docs/PROJECTS.md index fe6c2a8..0efee31 100644 --- a/docs/PROJECTS.md +++ b/docs/PROJECTS.md @@ -7,9 +7,11 @@ search across all of them, and (later) follow calls from one into another. ## The constraint that shapes everything -`idatui/worker.py` is `serve(sock, binpath)` — **one worker process holds exactly -one database** (idalib is main-thread-only and single-DB). So N binaries = N -worker processes, each with the analyzed DB resident. +IDA still exposes one active database per GUI/idalib process. Code Mode makes +those instances discoverable and shareable: each project entry retains one +`DatabaseHandle` lease, which may target a registered GUI or a managed idalib +worker. N resident project databases can therefore mean up to N processes, but +ida-tui no longer owns or terminates them. Measured cost (this box, `targets/`): @@ -30,13 +32,13 @@ crypto library. Two capabilities that feel like one, but aren't: -1. **Switching** to a binary needs a *live worker*. +1. **Switching** to a binary needs a *live Code Mode lease*. 2. **Searching across** binaries does *not* — if a per-binary index (functions, strings, imports/exports) is cached on disk. That split is the unlock: project-wide search stays instant across every binary, including ones never opened this session, and only *jumping* to a hit costs a -worker spawn. +Code Mode attach/open. ## Layout @@ -83,17 +85,16 @@ basename and must be unique (it names the staged file). ## Runtime -- **`WorkerPool`** — one `WorkerClient` per binary, spawned lazily on first - switch, kept resident until the memory budget is exceeded, then LRU-evicted. - Eviction **saves the DB first**, so returning to a binary is a DB load, not a - re-analysis. Binaries can be pinned to stay resident. +- **`DatabasePool`** — one `CodeModeClient` lease per resident binary, attached + lazily on first switch and LRU-released when the advisory memory budget is + exceeded. Eviction explicitly saves managed IDBs but never implicitly saves a + GUI. Closing a lease never kills a GUI or another client's managed worker; + Code Mode owns final worker shutdown. - **`BinaryState`** — per binary: `client, program, nav, cur, func_index, pref/active/split, filter`. Switching snapshots the current state and restores - the target's. `_after_reconnect` already does exactly this swap (client + - program, reload the index, re-open the entry) — switching reuses that seam. -- **Clean shutdown** — the worker currently does `close_database(save=False)` and - is hard-killed on exit, which is why wedge files accumulate. Projects need - save-on-evict and an orderly close anyway, so that gets fixed here. + the target's. `_after_reconnect` provides the client/program swap seam. +- **Clean shutdown** — release all leases. Managed idalib workers save/close on + their own main thread after the final lease; GUI sessions remain open. ## UI @@ -109,8 +110,8 @@ basename and must be unique (it names the staged file). ## Phases **Phase 1 — project model + switching. DONE.** Project file + staging -(`idatui/project.py`), `WorkerPool` with budget eviction / save-on-evict / -clean shutdown (`idatui/pool.py`), `BinaryState` snapshot+restore and the switch +(`idatui/project.py`), `DatabasePool` with budgeted lease release and +save-on-evict (`idatui/pool.py`), `BinaryState` snapshot+restore and the switch itself, the `Ctrl+O` switcher palette, the active binary in the status line, and `--project` (which creates the project when given binaries). One active binary; no cross-binary search yet. @@ -118,9 +119,9 @@ no cross-binary search yet. Project mode is **additive**: with no `--project` the app is byte-for-byte the single-binary tool it was, which is what keeps the 167-check pilot honest. Switching reuses the `_after_reconnect` shape — swap client+program, rebuild the -index, reopen the entry. A binary whose worker is still resident restores -instantly (its `Program` and index are still in memory); an evicted one comes -back with a fresh worker but keeps its nav history, since that is just addresses. +index, reopen the entry. A binary whose lease is still resident restores +instantly (its `Program` and index are still in memory); an evicted one attaches +again but keeps its nav history, since that is just addresses. **Phase 2 — index cache + project-wide search. (symbols done)** `idatui/index.py` keeps one **SQLite FTS5 trigram** index at @@ -222,7 +223,7 @@ records nothing — that's not navigation. *Pre-warm follows the linkage graph, not list order.* When a binary finishes indexing, `_prewarm_provider` warms the binary that provides the most of its imports — where a follow is most likely to take you, so its startup is paid -before you ask. `WorkerPool.prewarm()` refuses rather than evicting: spending a +before you ask. `DatabasePool.prewarm()` refuses rather than evicting: spending a binary you visited on one you haven't is a straight downgrade, and it would throw away that binary's caches too. At a tight budget pre-warm simply does nothing. It estimates the cost of a not-yet-spawned worker from the largest resident one, -- cgit v1.3.1-sl0p