diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/GRAPH_VIEW.md | 2 | ||||
| -rw-r--r-- | docs/NEXUS_UPSTREAM.md (renamed from docs/CODEMODE_UPSTREAM.md) | 167 | ||||
| -rw-r--r-- | docs/PAGING_FINDINGS.md | 14 | ||||
| -rw-r--r-- | docs/PROJECTS.md | 10 | ||||
| -rw-r--r-- | docs/SPLIT_VIEW.md | 8 |
5 files changed, 109 insertions, 92 deletions
diff --git a/docs/GRAPH_VIEW.md b/docs/GRAPH_VIEW.md index 3aced1f..0b24aa5 100644 --- a/docs/GRAPH_VIEW.md +++ b/docs/GRAPH_VIEW.md @@ -54,7 +54,7 @@ Growing a second disassembly renderer for graph mode would have been the real cost. The backend adds exactly one operation, `flowchart(addr)` in -`idatui/codemode_client.py`, which returns block ranges and typed edges — **not** +`idatui/nexus_client.py`, which returns block ranges and typed edges — **not** text. ## Two layout engines diff --git a/docs/CODEMODE_UPSTREAM.md b/docs/NEXUS_UPSTREAM.md index 9d3e09f..f4dcf91 100644 --- a/docs/CODEMODE_UPSTREAM.md +++ b/docs/NEXUS_UPSTREAM.md @@ -1,29 +1,30 @@ -# Findings from porting a real client to IDA Code Mode +# Findings from porting a real client to IDA Nexus -Notes for the `ida-codemode` maintainers, gathered while porting **ida-tui** (a +Notes for the `ida-nexus` maintainers, gathered while porting **ida-tui** (a Textual TUI frontend for IDA) from a private idalib worker to -`ida_codemode.DatabaseHandle`. +`ida_nexus.DatabaseHandle`. Everything below is measured, not inferred. Where we worked around something, the workaround is named so you can judge whether the library should make it unnecessary. -**Environment:** ida-codemode 0.3.1, IDA 9.4 (idalib), Linux, single managed +**Environment:** ida-nexus 0.3.1, IDA 9.4 (idalib), Linux, single managed worker backend, quiet box. Target for timings: `targets/echo` unless stated. -> **Status against 0.6.1 (upstream `439289f`) — every item re-checked.** +> **Status against the protocol-6 event-stream development tree, based on 0.6.1 +> (upstream `439289f`) — every item re-checked.** > > | item | verdict | > |---|---| > | 1 `timeout_trace` line tracing | ✅ **fixed in 0.3.2** — no `settrace` in the runtime at all | > | 2 `to_jsonable` on large results | ✅ **fixed in 0.3.2** — `dumps_json` C fast path | > | 3 2 ms `execute_sync` floor | ✅ **fixed in 0.3.2, 7.0x** — 2.055 ms → 0.294 ms | -> | 4 loader switches fatal on reopen | ✅ **fixed in 0.5.x** — see the caveat in §4 | -> | 5 IDB replaced under a live lease | ❌ open | -> | 6 no close-without-save | ❌ open | -> | 7 no change notification | ⚠️ partial — `DatabaseEventCallback` exists on `DatabaseManager`, but there is still no revision counter for an *external* caching client | +> | 4 loader switches fatal on reopen | **partial** — normal reopen fixed in 0.5.x; direct `.i64` paths remain [issue #36](https://github.com/HexRaysSA/ida-nexus/issues/36) | +> | 5 IDB replaced under a live lease | **stale** — out-of-band replacement is outside the supported lifecycle, as it is for the IDA GUI | +> | 6 close without save | **fixed in protocol 6** — the final managed-worker lease can choose `shutdown_database(save=False)` | +> | 7 no change notification | ✅ **fixed in protocol 6** — `DatabaseHandle.subscribe_idb_events()` streams revisioned, operation-attributed IDB changes | > | 8 package exports | ✅ **fixed in 0.5.x** — a real `__all__` on the package root | -> | 9 no `py.typed` / handle Protocol | ✅ **fixed in 0.5.x** — `ida_codemode/py.typed` ships | +> | 9 no `py.typed` / handle Protocol | ✅ **fixed in 0.5.x** — `ida_nexus/py.typed` ships | > > **0.5.x restructured the package**, which is why the old "these files are > byte-identical" re-check recipe no longer works: `client.py` → `handle.py`, @@ -31,16 +32,17 @@ worker backend, quiet box. Target for timings: `targets/echo` unless stated. > and the loader options moved into a frozen `DatabaseOpenOptions` dataclass. > Everything private is now underscore-prefixed, so the cheap re-check after an > upstream pull is simply: does anything we import still appear in -> `ida_codemode.__all__`? +> `ida_nexus.__all__`? > > 0.5.3 → 0.6.1 changed **nothing** we depend on: `__init__.py`, `handle.py`, > `instances.py`, `options.py`, `errors.py` and `models.py` are byte-identical > between those two releases. 0.6.1 only collapses the six console scripts into a -> single `ida-codemode` command. +> single `ida-nexus` command. > -> Both of our client-side workarounds re-measured at **0.99x and 0.97x** on 0.3.2 -> — i.e. nothing. The settrace strip has been deleted; `_PACK_EPILOGUE` is kept -> only for encoder determinism. Harness: `experiments/bench_pack_trace.py`. +> Both client-side workarounds re-measured at **0.99x and 0.97x** on 0.3.2 — +> i.e. nothing — and are deleted. Remote code is now ordinary typed Python, +> installed as content-addressed modules by ida-nexus. Harness: +> `experiments/bench_pack_trace.py`. **What the client does**, for scale: it renders a continuous disassembly listing, pseudocode, a CFG graph view and a hex view, paging over the database as the user @@ -108,10 +110,10 @@ and do the same, which is an argument for fixing it in the runtime. ## 2. `to_jsonable` dominates any large result -✅ **FIXED in 0.3.2**, via the first suggested fix below: `serialization.dumps_json` -calls `json.dumps(value, default=to_jsonable)`, so a JSON-safe result never enters -the Python walker. Our packing workaround now measures 0.97x and is retained only -to pin encoder settings, not for speed. +**FIXED in 0.3.2**, via the first suggested fix below: +`serialization.dumps_json` calls `json.dumps(value, default=to_jsonable)`, so a +JSON-safe result never enters the Python walker. Our packing workaround measured +0.97x and has been deleted. `execute_python` runs `to_jsonable()` over whatever the snippet returns. Our answers are already JSON-safe and they are big — a 200-row listing page is @@ -133,10 +135,10 @@ That is 114x, and it was 72% of the page's total cost before we changed it. envelope such as `{"__json__": "<...>"}`, or simply passing `str`/`bytes` through untouched. -**Our workaround:** snippets `json.dumps` inside the database process and return -one string, which the client parses. `to_jsonable` then walks a single scalar. -Cost went 66.2 ms → ~0.6 ms. It works, but every client with a large result set -has to discover and re-implement it. +**Retired workaround:** snippets used to `json.dumps` inside the database process +and return one string, which the client parsed. The typed remote API now owns +strict argument/result encoding, and ida-tui contains no generated script +strings or packing envelope. --- @@ -181,7 +183,7 @@ would let chatty clients amortise it without redesigning around it. --- -## 4. Loader switches on an existing database are a FATAL, not an error +## 4. Loader switches on an existing database are a FATAL, not an error — one edge remains Opening a target that already has an `.i64`, while passing spawn-only options, kills the worker: @@ -226,63 +228,80 @@ Our workaround is therefore deleted. **One narrow case remains**: the strip need `input_path != source`, so passing an `.i64` path *directly* together with load options (`ida-tui foo.i64 --processor arm`) still forwards the switches and still fatals. Our old guard keyed on "the target IDB exists" and so covered it. It is a -nonsense invocation and no idatui code path generates it — the project layer +nonsense invocation and no ida-tui code path generates it — the project layer always passes `output_database`, and `_needs_load_options` bails when an `.i64` -exists — but if this ever resurfaces as "worker exited with status 1", that is -where it comes from. +exists — but the library boundary should still reject or normalize it rather +than launch a known-fatal IDA command. Tracked upstream as +[issue #36](https://github.com/HexRaysSA/ida-nexus/issues/36). --- -## 5. Deleting or replacing an IDB under a live lease fails silently +## 5. Deleting or replacing an IDB under a live lease — STALE -A suite that did "delete the `.i64`, reopen the same path" (safe when it owned a -private worker) now races the previous worker's lease grace. The reopen produced -a handle that never became usable, with no error — just a database with no -listing, and every wait timing out. +The original suite deleted an `.i64` while a private worker still had it open, +then immediately reopened the same path. That ownership model no longer applies: +IDA Nexus databases are shared resources, and the IDA GUI itself does not survive +out-of-band replacement of its open database. Detecting arbitrary filesystem +replacement is therefore not part of the supported lifecycle. -**Suggested fixes** +The actionable lifecycle gaps that originally forced private-registry access are +fixed. `find_database_owner()` and `wait_database_released()` are public exports; +`DatabaseHandle.close(wait_for_database=True)` can wait for a final managed close; +a draining owner remains registered until the IDB is actually closed; and +`new_database=True` refuses to replace a live owner. -- Detect that the IDB backing a registered instance has been removed or replaced - and fail loudly (the registry already holds `idb_key`). -- Expose a **public** "wait until this database is released" primitive. We needed - one and ended up reaching into `registry.REGISTRY_DIR` and `FileLock` to build - it, which is not an API we should be depending on. -- Document the lease-grace window as part of the lifecycle contract. +ida-tui now uses the public owner/release API while recreating a database and no +longer reaches into registry locks. Owner loss is attach-only: ida-tui will +rediscover a replacement GUI or worker, but will never turn a +user-closing-the-GUI action into an implicit headless reopen. There is no +remaining upstream request in this section. --- -## 6. No close-without-save, and no rollback +## 6. Close without save — FIXED in protocol 6 + +`DatabaseHandle.shutdown_database(save=False)` can discard a managed idalib +worker when the requesting handle is its only active lease and no other operation +is running. The server rejects GUI databases and shared workers. -A managed worker saves when its final lease closes. A GUI handle leaves GUI state -as-is. Neither gives a client a way to say "discard what I did". +The coherent ownership model is the **final lease**, not necessarily the lease +that spawned the worker. Releasing a non-final lease makes no whole-database save +decision; responsibility transfers to the leases that remain. The final client +can save or discard the shared session. A client that needs its work to survive +regardless of that later decision must call `save_database()` before releasing +its lease. -ida-tui had a "discard & quit" that we could not port; it is now "leave as-is & -quit", and we cannot honestly promise the user their edits are not persisted. +This does not claim to provide per-client rollback. Discard applies to all +changes since the last database save, and attempting it while another lease is +active is correctly rejected. That is the same ref-counted lifetime model used +by other shared resources and requires no separate starter capability. -**Suggested fixes:** a close policy on a lease the client created -(`close(save=False)`), or a transaction/rollback API, or a documented -disposable-copy pattern that clients can follow. +The upstream gap is therefore closed. ida-tui now routes its discard action +through `shutdown_database(save=False)`: a final managed-worker lease discards, +while GUI-backed and still-shared sessions transfer finalization to their owner +or remaining leases. --- -## 7. No change notification for shared databases +## 7. No change notification for shared databases — FIXED in protocol 6 -The lease reports liveness, not mutations. If a GUI user or another Code Mode -client renames or retypes while we are attached, our materialised caches (name -generation, decompilation, listing pages) are silently stale. Our own edits -invalidate correctly; someone else's cannot. +`DatabaseHandle.subscribe_idb_events()` now returns a closeable iterator over +structured IDB changes. Each event carries a monotonic revision plus +`operation_id`/`operation_label` attribution and an opaque `origin_id`. +`DatabaseHandle.owns_event()` compares that origin with the handle's lease, so a +caching client does not need to generate, retain, or race operation IDs itself. -**Suggested fix — cheap and sufficient:** a monotonic database revision counter, -bumped on any mutating operation and exposed on `/health` (and ideally on the -lease event stream). Clients can then invalidate by comparing one integer. A full -change feed would be better but is much more work; the counter alone would make -shared editing safe for every caching client. +ida-tui keeps one subscription for its active database, asks the handle to drop +its own events, and batches peer events behind a 200 ms quiet period. One batch +invalidates the function, listing, decompiler, graph, strings, linkage, segment +and byte caches, then reloads the visible view in place. Closing or switching +databases closes the subscription, so the blocking event reader does not leak. --- ## 8. Package exports and API surface stability — FIXED in 0.5.x -`ida_codemode/__init__.py` used to export nothing, so a library consumer had to +`ida_nexus/__init__.py` used to export nothing, so a library consumer had to import from submodules, including things that were clearly internals (`FileLock`, `REGISTRY_DIR`, `canonical_path`, `idb_key`, `scan_instances`) that we only touched because no public equivalent existed. @@ -294,9 +313,9 @@ the package root, and mark the intended-public registry helpers explicitly. the internals moved behind an underscore: ```python -from ida_codemode import DatabaseHandle, DatabaseOpenOptions, DatabaseInstance -from ida_codemode import RemoteError, DatabaseBusyError, DatabaseDisconnectedError -from ida_codemode import discover_databases, find_database_owner, wait_database_released +from ida_nexus import DatabaseHandle, DatabaseOpenOptions, DatabaseInstance +from ida_nexus import RemoteError, DatabaseBusyError, DatabaseDisconnectedError +from ida_nexus import discover_databases, find_database_owner, wait_database_released ``` The two lock-poking helpers we had reimplemented client-side @@ -323,13 +342,13 @@ can be checked against the real signature and a typo is caught statically. (We added a test asserting our kwargs are a subset of `inspect.signature(DatabaseHandle.open).parameters`, which is a poor substitute.) -**0.5.x ships `ida_codemode/py.typed`**, and the 30 keyword-only options became a +**0.5.x ships `ida_nexus/py.typed`**, and the 30 keyword-only options became a frozen `DatabaseOpenOptions` dataclass — which is strictly better, because an invented option name is now a `TypeError` at construction rather than something a `**kwargs` fake swallows. Our subset test survives in two halves (`_open_kwargs_are_real` for `open()`, `_option_fields_are_real` for the dataclass fields), because the offline contract suite must keep running with no -`ida_codemode` installed at all and therefore still fakes both. +`ida_nexus` installed at all and therefore still fakes both. --- @@ -340,10 +359,10 @@ dataclass fields), because the offline contract suite must keep running with no | ~~1~~ | ~~`timeout_trace` line tracing~~ | ~~52x on IDA calls~~ | ✅ fixed in 0.3.2 | | ~~2~~ | ~~`to_jsonable` on large results~~ | ~~114x on serialisation~~ | ✅ fixed in 0.3.2 | | ~~3~~ | ~~2 ms `execute_sync` floor~~ | ~~shapes client design~~ | ✅ fixed in 0.3.2, 7.0x | -| 7 | no change/revision counter | correctness for shared editing | yes, cheap | -| ~~4~~ | ~~loader switches fatal on reopen~~ | ~~crashes, hard to diagnose~~ | ✅ fixed in 0.5.x (one edge, §4) | -| 5 | replaced/deleted IDB under lease | silent hang | yes | -| 6 | no close-without-save | a feature we had to drop | design question | +| ~~7~~ | ~~no change/revision counter~~ | ~~correctness for shared editing~~ | ✅ fixed in protocol 6 | +| 4 | direct `.i64` forwards loader-only options | fatal worker startup | [issue #36](https://github.com/HexRaysSA/ida-nexus/issues/36) | +| ~~5~~ | ~~replaced/deleted IDB under lease~~ | ~~out-of-contract filesystem mutation~~ | **stale** | +| ~~6~~ | ~~no close without save~~ | ~~could not discard a managed session~~ | **fixed in protocol 6: final lease decides** | | ~~8~~ | ~~package exports~~ | ~~forces internal imports~~ | ✅ fixed in 0.5.x | | ~~9~~ | ~~typed handle for fakes~~ | ~~catches a whole bug class~~ | ✅ fixed in 0.5.x (`py.typed` + options dataclass) | @@ -356,13 +375,11 @@ worth fixing centrally rather than leaving each client to rediscover. fixed upstream, and both client-side workarounds could be measured at parity and retired. That is the outcome this document was written for. -**What is left is entirely non-performance**: 0.5.x then fixed the API-surface -items (4, 8, 9) — the package root is a real public API, `py.typed` ships, and -the loader-switch fatal is handled in the resolver. What remains open is -lifecycle: **5** (replacing an IDB under a live lease), **6** (close without -save) and **7** — a monotonic revision counter on `/health`, still the cheapest -large win for any caching client, and still the one thing an *external* client -cannot build for itself. +**What is left is entirely non-performance.** Items 6 through 9 are fixed, and +item 5 is stale because out-of-band replacement is not a supported lifecycle for +either IDA Nexus or the IDA GUI. One narrow piece remains: **4**, normalize or +reject loader-only options when the source is itself an existing `.i64` +([issue #36](https://github.com/HexRaysSA/ida-nexus/issues/36)). Happy to supply the benchmark harness (it is backend-agnostic and runs against -both our old worker and Code Mode), or to test a patch. +both our old worker and IDA Nexus), or to test a patch. diff --git a/docs/PAGING_FINDINGS.md b/docs/PAGING_FINDINGS.md index bd6c38f..f343973 100644 --- a/docs/PAGING_FINDINGS.md +++ b/docs/PAGING_FINDINGS.md @@ -3,9 +3,9 @@ Measured against a real target: `libcrypto.so.3` (5.7 MB, **10,092 functions**, biggest function **52,120 instructions**). These constraints drive the domain / paging layer. The measurements below came from the former ida-pro-mcp tool -backend. The Code Mode port preserves the adapter response shapes and conservative +backend. The IDA Nexus port preserves the adapter response shapes and conservative page sizes, but executes enumeration through ida-domain; old server caps and RTT -numbers are historical rather than Code Mode constraints. +numbers are historical rather than IDA Nexus constraints. ## Response shape (list_* / *_query tools) @@ -93,17 +93,17 @@ disasm totals are **top-level** fields, not under `asm`: (correct). The pseudocode view must handle "decompilation failed" gracefully — fall back to the disassembly view or show an error panel. -Code Mode returns the complete execution result directly; ida-tui no longer +IDA Nexus returns the complete execution result directly; ida-tui no longer needs MCP structured-content/download-URL recovery for large pseudocode bodies. -## Code Mode lifecycle +## IDA Nexus lifecycle -`CodeModeClient` owns an authenticated SSE lease on a registered database: +`NexusClient` owns an authenticated SSE lease on a registered database: * A matching GUI is preferred and remains open when the TUI exits. -* Otherwise Code Mode reuses or starts a shared managed idalib worker. +* Otherwise IDA Nexus reuses or starts a shared managed idalib worker. * Releasing one lease never terminates another client's session. A managed - worker saves and exits after its final lease under Code Mode's grace policy. + worker saves and exits after its final lease under IDA Nexus's grace policy. * Lease loss surfaces as `IDAConnectionError`; reconnect performs discovery again and may bind a newly-created instance. It does not silently swap the handle underneath an operation. diff --git a/docs/PROJECTS.md b/docs/PROJECTS.md index 0efee31..52dbf83 100644 --- a/docs/PROJECTS.md +++ b/docs/PROJECTS.md @@ -7,7 +7,7 @@ search across all of them, and (later) follow calls from one into another. ## The constraint that shapes everything -IDA still exposes one active database per GUI/idalib process. Code Mode makes +IDA still exposes one active database per GUI/idalib process. IDA Nexus makes those instances discoverable and shareable: each project entry retains one `DatabaseHandle` lease, which may target a registered GUI or a managed idalib worker. N resident project databases can therefore mean up to N processes, but @@ -32,13 +32,13 @@ crypto library. Two capabilities that feel like one, but aren't: -1. **Switching** to a binary needs a *live Code Mode lease*. +1. **Switching** to a binary needs a *live IDA Nexus lease*. 2. **Searching across** binaries does *not* — if a per-binary index (functions, strings, imports/exports) is cached on disk. That split is the unlock: project-wide search stays instant across every binary, including ones never opened this session, and only *jumping* to a hit costs a -Code Mode attach/open. +IDA Nexus attach/open. ## Layout @@ -85,11 +85,11 @@ basename and must be unique (it names the staged file). ## Runtime -- **`DatabasePool`** — one `CodeModeClient` lease per resident binary, attached +- **`DatabasePool`** — one `NexusClient` lease per resident binary, attached lazily on first switch and LRU-released when the advisory memory budget is exceeded. Eviction explicitly saves managed IDBs but never implicitly saves a GUI. Closing a lease never kills a GUI or another client's managed worker; - Code Mode owns final worker shutdown. + IDA Nexus owns final worker shutdown. - **`BinaryState`** — per binary: `client, program, nav, cur, func_index, pref/active/split, filter`. Switching snapshots the current state and restores the target's. `_after_reconnect` provides the client/program swap seam. diff --git a/docs/SPLIT_VIEW.md b/docs/SPLIT_VIEW.md index c421656..6e9b938 100644 --- a/docs/SPLIT_VIEW.md +++ b/docs/SPLIT_VIEW.md @@ -32,7 +32,7 @@ known technique: The old ida-pro-mcp backend derived the per-line marker via `cfunc.get_line_item(line, col=0, …).get_ea()`. To get the **full set**, sweep every column of the line (`get_line_item(line, x, …).get_ea()` for `x` in -`0..len`) and collect distinct non-`BADADDR` EAs. The Code Mode adapter's +`0..len`) and collect distinct non-`BADADDR` EAs. The IDA Nexus adapter's `decomp_map(ea)` operation returns `[{line, primary_ea, eas:[…]}, …]`; invert for `ea → line`. @@ -78,14 +78,14 @@ decomp→listing uses `ListingModel.ensure_ea`. Tab re-links from the new driver Still single-ea per line (one instruction highlighted); the region comes in phase 3. -**Phase 3 — rich highlight. DONE.** The Code Mode `decomp_map` operation -(`idatui/codemode_client.py`) sweeps `cfunc.get_line_item` across every column of +**Phase 3 — rich highlight. DONE.** The IDA Nexus `decomp_map` operation +(`idatui/nexus_client.py`) sweeps `cfunc.get_line_item` across every column of each pseudocode line and collects the EAs from each item's `dstr()` (`'EA: desc'` — the same source as the `/*ea*/` marker, so it aligns). `Program.decomp_map(ea)` returns the per-line ea lists (cached by name-gen); the app loads it async into `_split_eamap` / `_split_ea2line` and `_sync_split` bands the **whole** instruction region of a C line (and uses the exact ea→line inverse for the reverse). Falls -back to the single marker until the map lands. Verified on a real Code Mode database +back to the single marker until the map lands. Verified on a real IDA Nexus database (alignment + multi-instruction region band). **Phase 4 — polish. DONE.** |