diff options
| author | user <user@clank> | 2026-08-14 00:26:27 +0200 |
|---|---|---|
| committer | user <user@clank> | 2026-08-14 00:26:27 +0200 |
| commit | bb5db57c72ac814cfb8019583e9cfd3c5fdea00e (patch) | |
| tree | 48b83020f76630ab11ff74b0505971d857bbc325 | |
| parent | Port to the ida-codemode 0.5.3+ public API (diff) | |
| download | ida-tui-bb5db57c72ac814cfb8019583e9cfd3c5fdea00e.tar.gz ida-tui-bb5db57c72ac814cfb8019583e9cfd3c5fdea00e.tar.xz ida-tui-bb5db57c72ac814cfb8019583e9cfd3c5fdea00e.zip | |
Re-check the upstream findings report against 0.6.1
Items 4, 8 and 9 are fixed (loader switches on reopen, package exports,
py.typed), so the report and the skill no longer claim 4-9 are all open. Item 7
is partial: DatabaseManager grew a DatabaseEventCallback, but there is still
nothing an external caching client can observe, which was the point. 5 and 6
are untouched.
The "are these files byte-identical" re-check recipe is removed: it named
client.py/registry.py/resolver.py, which 0.5.x renamed, so it would have
reported "unchanged" forever. The replacement is a diff of __all__ and the
DatabaseOpenOptions fields, both of which the contract suite now asserts
whenever the library is importable.
Also records two things that cost time here: the loader-switch strip needs
input_path != source, so an .i64 passed directly alongside load options still
fatals; and an editable checkout does not enforce its own dependency floors, so
a `git pull` to 0.6.1 leaves ida-domain/zeromcp silently too old.
| -rw-r--r-- | .agents/skills/idatui/SKILL.md | 54 | ||||
| -rw-r--r-- | docs/CODEMODE_UPSTREAM.md | 115 |
2 files changed, 116 insertions, 53 deletions
diff --git a/.agents/skills/idatui/SKILL.md b/.agents/skills/idatui/SKILL.md index 336b31a..ba28e56 100644 --- a/.agents/skills/idatui/SKILL.md +++ b/.agents/skills/idatui/SKILL.md @@ -11,7 +11,7 @@ compatibility: "Backend is the ida-codemode LIBRARY (a normal dependency; see py # idatui — IDA TUI frontend A minimal, tasteful, keyboard-first (mouse-capable) TUI for IDA Pro, driving -**IDA Code Mode** (`ida_codemode.client.DatabaseHandle`) — it takes a *lease* on a +**IDA Code Mode** (`ida_codemode.DatabaseHandle`) — it takes a *lease* on a database that is either an already-open IDA **GUI** session or a shared **managed idalib worker**. idatui never owns or terminates an IDA process. Repo: this repository. Built with **Textual**. This skill is the memory of how it @@ -184,12 +184,32 @@ it fails with `ModuleNotFoundError: idatui`. ## IDA Code Mode gotchas -- **`ida-codemode` is a normal dependency** (`pyproject.toml` pins `>=0.3.1`); these - notes are current as of **0.3.2**. If you install it **editable from a git checkout** - (handy for tracking upstream), remember that a `git pull` in that checkout swaps the - backend under the TUI **immediately, with no reinstall** — convenient, but it means an - upstream change can alter behaviour without anything in this repo changing. Run the - full gate after any such pull. +- **`ida-codemode` is a normal dependency** (`pyproject.toml` pins `>=0.5.3`, `uv.lock` + pins **0.6.1**); these notes are current as of **0.6.1**. If you install it **editable + from a git checkout** (handy for tracking upstream), remember that a `git pull` in that + checkout swaps the backend under the TUI **immediately, with no reinstall** — + convenient, but it means an upstream change can alter behaviour without anything in + this repo changing. Run the full gate after any such pull. A version bump can also move + the checkout's *dependency floors* without touching its code: 0.6.1 needs + `ida-domain>=0.5.1` and `zeromcp>=1.8.0`, and an editable install does not enforce + them — run `pip check` in `~/ida-venv` after a pull, or the failure surfaces much later + and much less legibly. +- **We import from the package ROOT, never from submodules.** 0.5.x made + `ida_codemode.__all__` the public API and hid the rest behind underscores + (`client.py` → `handle.py`, `registry.py` → `_registry.py` + `instances.py`, + `resolver.py` → `_resolver.py`). Anything importing `ida_codemode.client`, + `.registry` or `.resolver` is pre-0.5 and stale. Loader options are a frozen + `DatabaseOpenOptions` dataclass passed as `open(path, options=...)`, ownership + questions are `find_database_owner()`, and lock-release waiting is + `wait_database_released()` — we no longer poke `FileLock`/`REGISTRY_DIR` ourselves. +- **`codemode_client.py` binds every Code Mode name to `None` when the library is + absent**, so the pure suites run under stdlib `python3`. That makes `except + SomeCodeModeError:` a landmine: `except None` raises `TypeError: catching classes that + do not inherit from BaseException`, but ONLY if something inside the `try` actually + raises — so it hides until the day another line in that block fails. If you add a call + to a Code Mode symbol inside a `try`, make `tests/test_codemode_client.py` inject a + fake for it too (it fakes `DatabaseHandle`, `DatabaseOpenOptions` and + `DatabaseBusyError` for exactly this reason). - **Two big client-side workarounds existed and are now DELETED, because 0.3.2 fixed them upstream.** Don't re-add them, and don't trust older docs that describe them: - the runtime used to wrap every `execute_python` in `sys.settrace(timeout_trace)`, @@ -202,16 +222,16 @@ it fails with `ModuleNotFoundError: idatui`. (compact separators, `default=str`) — no longer load-bearing for performance. - Re-measure both with `PYTHONPATH=. ~/ida-venv/bin/python experiments/bench_pack_trace.py`. - **`docs/CODEMODE_UPSTREAM.md` is our findings report to the ida-codemode maintainers, - and every item is re-checked against 0.3.2.** Items **1, 2 and 3 are FIXED** (trace - hook, to_jsonable, the execute_sync floor). Items **4–9 are still open** — loader - switches fatal on reopen, replaced IDB under a live lease, no close-without-save, no - change notification, package exports, no `py.typed`/handle Protocol — and they are - open *by construction*: `client.py`, `registry.py`, `resolver.py`, `server.py`, - `database.py` and `worker.py` are byte-identical between 0.3.1 and 0.3.2, and those - items all live in those files. Read it before assuming a Code Mode behaviour is a bug - in our code. **Cheap way to re-check after any upstream pull:** - `for f in client registry resolver server database worker; do git diff --quiet OLD..HEAD -- ida_codemode/$f.py; done` - — if they're all unchanged, items 4–9 cannot have moved. + and every item is re-checked against 0.6.1.** Items **1, 2, 3 are FIXED** (trace hook, + to_jsonable, the execute_sync floor) and so are **4, 8, 9** (loader switches on reopen, + package exports, `py.typed`). Still open: **5** replaced IDB under a live lease, **6** + no close-without-save, **7** no change notification (0.6.1 has a + `DatabaseEventCallback` on `DatabaseManager`, but nothing an external caching client + can observe). Read it before assuming a Code Mode behaviour is a bug in our code. + The old "are these files byte-identical" re-check recipe is DEAD — those files were + renamed in 0.5.x. Re-check by diffing `ida_codemode/__all__` and the + `DatabaseOpenOptions` fields instead; `tests/test_codemode_client.py` asserts both + automatically whenever the library is importable. - **The old ~2ms per-operation floor is GONE in 0.3.2 (7.0x).** Same-box A/B: `execute_python("result = 1")` was **2.055ms** on 0.3.1 and is **0.294ms** on 0.3.2, about the cost of a bare HTTP GET — the `execute_sync` marshalling that was ~93% of diff --git a/docs/CODEMODE_UPSTREAM.md b/docs/CODEMODE_UPSTREAM.md index fe6426f..9d3e09f 100644 --- a/docs/CODEMODE_UPSTREAM.md +++ b/docs/CODEMODE_UPSTREAM.md @@ -2,7 +2,7 @@ Notes for the `ida-codemode` maintainers, gathered while porting **ida-tui** (a Textual TUI frontend for IDA) from a private idalib worker to -`ida_codemode.client.DatabaseHandle`. +`ida_codemode.DatabaseHandle`. Everything below is measured, not inferred. Where we worked around something, the workaround is named so you can judge whether the library should make it @@ -11,25 +11,32 @@ unnecessary. **Environment:** ida-codemode 0.3.1, IDA 9.4 (idalib), Linux, single managed worker backend, quiet box. Target for timings: `targets/echo` unless stated. -> **Status against 0.3.2 (upstream `93e8aad`) — every item re-checked.** +> **Status against 0.6.1 (upstream `439289f`) — every item re-checked.** > > | item | verdict | > |---|---| -> | 1 `timeout_trace` line tracing | ✅ **fixed** — no `settrace` in the runtime at all | -> | 2 `to_jsonable` on large results | ✅ **fixed** — `dumps_json` C fast path | -> | 3 2 ms `execute_sync` floor | ✅ **fixed, 7.0x** — 2.055 ms → 0.294 ms | -> | 4 loader switches fatal on reopen | ❌ open | +> | 1 `timeout_trace` line tracing | ✅ **fixed in 0.3.2** — no `settrace` in the runtime at all | +> | 2 `to_jsonable` on large results | ✅ **fixed in 0.3.2** — `dumps_json` C fast path | +> | 3 2 ms `execute_sync` floor | ✅ **fixed in 0.3.2, 7.0x** — 2.055 ms → 0.294 ms | +> | 4 loader switches fatal on reopen | ✅ **fixed in 0.5.x** — see the caveat in §4 | > | 5 IDB replaced under a live lease | ❌ open | > | 6 no close-without-save | ❌ open | -> | 7 no change notification | ❌ open | -> | 8 package exports | ❌ open | -> | 9 no `py.typed` / handle Protocol | ❌ open | +> | 7 no change notification | ⚠️ partial — `DatabaseEventCallback` exists on `DatabaseManager`, but there is still no revision counter for an *external* caching client | +> | 8 package exports | ✅ **fixed in 0.5.x** — a real `__all__` on the package root | +> | 9 no `py.typed` / handle Protocol | ✅ **fixed in 0.5.x** — `ida_codemode/py.typed` ships | > -> Items 4–9 are open **by construction**: `client.py`, `registry.py`, -> `resolver.py`, `server.py`, `database.py` and `worker.py` are byte-identical -> between 0.3.1 and 0.3.2 (`git diff --quiet 4195f21..HEAD -- <file>`), and every -> one of those items lives in those files. Only `runtime.py`, `http.py`, -> `serialization.py` (new) and `benchmark.py` (new) changed. +> **0.5.x restructured the package**, which is why the old "these files are +> byte-identical" re-check recipe no longer works: `client.py` → `handle.py`, +> `registry.py` → `_registry.py` + `instances.py`, `resolver.py` → `_resolver.py`, +> and the loader options moved into a frozen `DatabaseOpenOptions` dataclass. +> Everything private is now underscore-prefixed, so the cheap re-check after an +> upstream pull is simply: does anything we import still appear in +> `ida_codemode.__all__`? +> +> 0.5.3 → 0.6.1 changed **nothing** we depend on: `__init__.py`, `handle.py`, +> `instances.py`, `options.py`, `errors.py` and `models.py` are byte-identical +> between those two releases. 0.6.1 only collapses the six console scripts into a +> single `ida-codemode` command. > > Both of our client-side workarounds re-measured at **0.99x and 0.97x** on 0.3.2 > — i.e. nothing. The settrace strip has been deleted; `_PACK_EPILOGUE` is kept @@ -204,8 +211,25 @@ crash five minutes into a run. already exists on the worker's stderr; losing it turns a one-line fix into a bisect. -**Our workaround:** the client checks whether the expected IDB exists and drops -`processor`/`image_base`/`file_type` when it does. +**Our workaround:** the client checked whether the expected IDB exists and +dropped `processor`/`image_base`/`file_type` when it did. + +**FIXED in 0.5.x**, with exactly this fix, in `_resolver._build_worker_command`: + +```python +if input_path == expected_idb and input_path != source: + # Loader/import switches are baked into an existing IDB... + options = WorkerLaunchOptions() +``` + +Our workaround is therefore deleted. **One narrow case remains**: the strip needs +`input_path != source`, so passing an `.i64` path *directly* together with load +options (`ida-tui foo.i64 --processor arm`) still forwards the switches and still +fatals. Our old guard keyed on "the target IDB exists" and so covered it. It is a +nonsense invocation and no idatui code path generates it — the project layer +always passes `output_database`, and `_needs_load_options` bails when an `.i64` +exists — but if this ever resurfaces as "worker exited with status 1", that is +where it comes from. --- @@ -256,27 +280,35 @@ shared editing safe for every caching client. --- -## 8. Package exports and API surface stability +## 8. Package exports and API surface stability — FIXED in 0.5.x + +`ida_codemode/__init__.py` used to export nothing, so a library consumer had to +import from submodules, including things that were clearly internals (`FileLock`, +`REGISTRY_DIR`, `canonical_path`, `idb_key`, `scan_instances`) that we only +touched because no public equivalent existed. + +**Suggested fix was:** export `DatabaseHandle` and the public exception types from +the package root, and mark the intended-public registry helpers explicitly. -`ida_codemode/__init__.py` exports nothing, so a library consumer must import -from submodules: +**That is what 0.5.x did.** Everything we need is now on the package root, and +the internals moved behind an underscore: ```python -from ida_codemode.client import DatabaseHandle, ClientError, RemoteError, InstanceDisconnectedError -from ida_codemode.registry import REGISTRY_DIR, FileLock, RegistryEntry, canonical_path, idb_key, scan_instances -from ida_codemode.resolver import IdbBusy, expected_idb_path +from ida_codemode import DatabaseHandle, DatabaseOpenOptions, DatabaseInstance +from ida_codemode import RemoteError, DatabaseBusyError, DatabaseDisconnectedError +from ida_codemode import discover_databases, find_database_owner, wait_database_released ``` -Some of those are clearly internals (`FileLock`, `REGISTRY_DIR`) that we only -touch because no public equivalent exists (see §5). - -**Suggested fix:** export `DatabaseHandle` and the public exception types from the -package root, and mark the intended-public registry helpers explicitly. It also -makes "what is API and what is internal" answerable, which right now it is not. +The two lock-poking helpers we had reimplemented client-side +(`_wait_for_entry_release`) are now `wait_database_released()`, and our +registry-scanning ownership check is now `find_database_owner()`. Both are +deleted from our tree. Note `find_database_owner()` *raises* +`AmbiguousDatabaseError` where our scan silently took the first match — a +behaviour improvement, but callers need a handler. --- -## 9. A testing note: `DatabaseHandle.open()`'s 30 keyword-only options +## 9. A testing note: `DatabaseHandle.open()`'s 30 keyword-only options — FIXED in 0.5.x The port we started from called `open(..., loading_address=...)`. The real parameter is `image_base`. Every `connect()` would have raised `TypeError` on the @@ -291,6 +323,14 @@ can be checked against the real signature and a typo is caught statically. (We added a test asserting our kwargs are a subset of `inspect.signature(DatabaseHandle.open).parameters`, which is a poor substitute.) +**0.5.x ships `ida_codemode/py.typed`**, and the 30 keyword-only options became a +frozen `DatabaseOpenOptions` dataclass — which is strictly better, because an +invented option name is now a `TypeError` at construction rather than something a +`**kwargs` fake swallows. Our subset test survives in two halves +(`_open_kwargs_are_real` for `open()`, `_option_fields_are_real` for the +dataclass fields), because the offline contract suite must keep running with no +`ida_codemode` installed at all and therefore still fakes both. + --- ## Priority, from a client author's view @@ -301,11 +341,11 @@ added a test asserting our kwargs are a subset of | ~~2~~ | ~~`to_jsonable` on large results~~ | ~~114x on serialisation~~ | ✅ fixed in 0.3.2 | | ~~3~~ | ~~2 ms `execute_sync` floor~~ | ~~shapes client design~~ | ✅ fixed in 0.3.2, 7.0x | | 7 | no change/revision counter | correctness for shared editing | yes, cheap | -| 4 | loader switches fatal on reopen | crashes, hard to diagnose | yes | +| ~~4~~ | ~~loader switches fatal on reopen~~ | ~~crashes, hard to diagnose~~ | ✅ fixed in 0.5.x (one edge, §4) | | 5 | replaced/deleted IDB under lease | silent hang | yes | | 6 | no close-without-save | a feature we had to drop | design question | -| 8 | package exports | forces internal imports | yes, trivial | -| 9 | typed handle for fakes | catches a whole bug class | yes | +| ~~8~~ | ~~package exports~~ | ~~forces internal imports~~ | ✅ fixed in 0.5.x | +| ~~9~~ | ~~typed handle for fakes~~ | ~~catches a whole bug class~~ | ✅ fixed in 0.5.x (`py.typed` + options dataclass) | Items 1 and 2 together were the difference between "the port is 35x slower than the private worker it replaced" and "the port is within 2x, and faster on several @@ -316,10 +356,13 @@ worth fixing centrally rather than leaving each client to rediscover. fixed upstream, and both client-side workarounds could be measured at parity and retired. That is the outcome this document was written for. -**What is left is entirely non-performance**: items 4–9 are correctness, -lifecycle and API-surface items, and all six are untouched in 0.3.2 because the -files they live in are byte-identical to 0.3.1. Item 7 (a monotonic revision -counter on `/health`) remains the cheapest large win for any caching client. +**What is left is entirely non-performance**: 0.5.x then fixed the API-surface +items (4, 8, 9) — the package root is a real public API, `py.typed` ships, and +the loader-switch fatal is handled in the resolver. What remains open is +lifecycle: **5** (replacing an IDB under a live lease), **6** (close without +save) and **7** — a monotonic revision counter on `/health`, still the cheapest +large win for any caching client, and still the one thing an *external* client +cannot build for itself. Happy to supply the benchmark harness (it is backend-agnostic and runs against both our old worker and Code Mode), or to test a patch. |
