tui for headless ida https://git.sl0p.foo/ida-tui.git/atom/idatui/codemode_client.py?h=main 2026-08-07T21:16:30Z 2026-08-07T21:16:30Z blasty blasty@local 2026-08-07T21:16:30Z urn:sha1:41b3710d5b6f9be24430fd55c46c83f9ae3b8d83 `/` only ever searched the lines of the view you were in. This adds the search you actually need on a binary: over the entire database, either through the rendered disassembly or through the image. * **text** matches the line as displayed, whitespace-normalised, so `call cs:` finds `call cs:getenv_ptr` (IDA's column padding is not something anyone types). Smartcase; `regex` available over RPC. * **bytes** is IDA's own `find_bytes`, so the pattern language people already know works unchanged: hex pairs, `?` wildcards for a whole byte or one nibble (`48 8? ?? 24`), quoted literals (`"Hello", 0`). Commas, no separators (`488B05C3`) and ragged spacing all normalise. **Which mode you meant is guessed, and the guess is biased on purpose.** `dead`, `add`, `cafe` and `ff` are valid hex AND ordinary things to search for, so a bare hex-looking word stays TEXT; nobody types `48 8b ?? c3` meaning prose. `hex:`/`text:` prefixes and F2 override it. The subtle case is a *typo* in a byte pattern. `48 zz c3` first fell through to a text search and reported "no match" — indistinguishable from "those bytes are not in this binary", which is the most misleading answer a search can give. Now any query whose tokens are all byte-sized is treated as bytes, and a bad token is refused BY NAME. IDA does the same thing quietly (find_bytes answers a malformed pattern with zero hits and no error), so the validation lives in Program.search, not just in the UI. Enter searches, then Enter opens the highlighted hit; the title says which it will do, because a database-wide scan is far too slow to run on every keystroke like the other palettes. Navigation goes to the item head — a byte match can start mid-instruction — and the status names the exact address. Also: the `find` RPC verb and `drive find`, which is the one an agent wants (`drive find '48 8b ?? c3'`). idatui/search.py holds the classification and is pure, so the whole question of "what did they mean" is tested offline: tests/test_search.py, 35 checks, 0.1s. Pilot scenario db_search covers the UI end to end. Full suite: 890 passed, 0 failed, 51.3s. 2026-08-07T20:55:27Z blasty blasty@local 2026-08-07T20:55:27Z urn:sha1:2eb2a0a8cff586fffecfcb068c53b65e8f6f9839 The output of an RE session is what you worked out, and it was locked in a .i64 that only IDA can read. Ctrl+E (or `drive export`, or the `export` RPC verb) writes it out: your comments grouped by function with the line each annotates, the names and prototypes you set, the types you declared. **The hard part was provenance, and it needed a mechanism, not a filter.** A database does not record WHO wrote a comment or a name. IDA's analyzer sets `; switch 73 cases` and `; s1` with the same `set_cmt` a person uses, and the ELF loader sets `elf_gnu_hash_nbuckets` and `File class: 64-bit` the same way. Four probes, all negative: the FF_COMM flag is identical, `get_cmt` returns them all, `generate_disasm_line` tags every one of them COLOR_REGCMT (not COLOR_AUTOCMT), and they survive with auto-comments switched off. A first cut filtered by shape and produced a report whose first screen was ELF header trivia and `; jumptable ... case 99`. So idatui journals its own edits (idatui/journal.py) into a netnode in the database: it rides along in the .i64, it is still there next session, and the report is then exactly what was done here -- 2 findings out of a database carrying 693 other annotations. Recorded at the choke points in edit_ctl (rename, name-address, comment, retype) and in the struct editor; flushed on save, on export and on quit, so no edit pays a round trip. Without a journal (a database worked on in the IDA GUI, or predating this) the report falls back to filtering by shape -- dummy names, imports, loader segments, the analyzer's stereotyped switch/jumptable strings -- and says so in the document rather than claiming authorship it cannot prove. idatui/findings.py splits gather (needs IDA) from render (does not), so the formatting, grouping, sorting, escaping and the empty cases are tested offline: tests/test_findings.py, 32 checks, no worker, 0.1s. The pilot scenario covers the round trip that matters -- edit through the UI, export, find it in the file, and reload the journal from the .i64. Full suite: 842 passed, 0 failed, 51.2s. 2026-08-07T13:02:05Z blasty blasty@local 2026-08-07T13:02:05Z urn:sha1:55e9d9fdca4baafce728659d43614e203bfbea9a Two changes, both about work that was never ours to do, found by profiling the A/B benchmark rather than guessing. 1. Serialise inside the database process. Code Mode runs to_jsonable() over whatever a snippet returns, walking the entire structure to make it JSON-safe. Our answers are already JSON-safe and they are large: a 200-row listing page is ~10k small objects, and walking them cost 66ms of the page's 92ms -- 114x what json.dumps of the very same data costs (0.58ms). Snippets now return one pre-serialised string, so that walk is O(1) and the client parses a payload it was going to parse anyway. heads(200): 92ms -> 24.7ms. 2. Detach the runtime's trace hook while our snippet runs. ida_codemode.runtime wraps every execute_python in sys.settrace(timeout_trace) to enforce deadlines, and timeout_trace RETURNS ITSELF -- which switches on LINE tracing in every frame it sees. Every line of every function we call pays a Python-level callback. Measured here: ida_bytes.get_flags 0.106us untraced 5.49us traced 52x (plain idalib, no Code Mode: 0.119us -- i.e. untraced == native) heads(200 rows) 2.0ms untraced 20.2ms traced 10x That one hook was the entire residual gap against the old unix-socket worker. The snippet now detaches it and restores it in a finally. What that gives up, stated plainly: the deadline is no longer enforced for a pure-Python loop inside our snippet. The runtime's other cancellation path -- a threading.Timer calling ida_kernwin.set_cancelled() -- does not go through the trace and still fires, so a long IDA operation remains interruptible, and every operation here is bounded by its own count/limit argument. Set IDATUI_CODEMODE_TRACE=1 to keep the stock behaviour. Against the worker backend, same box, targets/echo (worker -> codemode): heads_200 2.65ms -> 5.85ms 2.2x (was 35x) heads_500 6.21ms -> 10.67ms 1.7x heads expect-hit 2.16ms -> 4.61ms 2.1x disasm_200 9.03ms -> 5.25ms 0.6x faster decompile_cold 162.62ms -> 30.66ms 0.2x faster decompile_warm 30.02ms -> 25.39ms 0.8x faster decomp_map 45.82ms -> 47.85ms 1.0x parity pc_nums 19.90ms -> 22.56ms 1.1x parity rename_func 254.08ms -> 255.54ms 1.0x parity connect 550.0ms -> 410.0ms 0.7x faster What is left is the transport floor: an empty execute_python round trip is 2.0ms, so trivial calls (data_type 0.07ms -> 2.63ms, force_recompile, a single xref query) look like 40x while being 2.5ms of wall clock. Reducing those needs fewer calls, not faster ones -- the digest/expect path already does that for the listing, which is where call volume actually is. Full suite: 788 passed, 0 failed, 115.3s (was 146.3s; the pilot alone went 80.9s -> 62.2s). 2026-08-07T12:32:56Z blasty blasty@local 2026-08-07T12:32:56Z urn:sha1:21971d0a9ce9b8a2fb1db2ec67303696cea9cbbb Three separate wastes, all of the same family: waiting on a guess instead of a signal, and paying for work that never had to be repeated. 1. The 64KB blob was built with os.urandom into a fresh TemporaryDirectory on every run. New bytes at a new path means the pristine-database cache can never apply, so full auto-analysis of 64KB of AArch64-decoded noise was paid every single run. It is now built from a seeded PRNG at a stable path (tests/.synthetic/, gitignored) and staged through the existing cache. Determinism is also a correctness fix: whether 64KB of chance bytes contains something IDA reads as a function is luck, and this suite asserts "and really has no functions". 2. `wait(lambda: lst.model is not old, ..., 30)` after commenting. The perf work made an item edit KEEP the listing's walk and re-render in place, so the model object is never replaced and this waited out its full 30s timeout on every run -- and then "commenting leaves the view where it was" passed vacuously, because nothing had happened at all. A test that burns 30s to check nothing is worse than no test. 3. Two `pause(2.0)`/`pause(2.5)` after a carve, replaced with settle() on a real condition. The second one deliberately has NO predicate: that spot is random data, so the carve may legitimately produce nothing, and "the row became code" would never hold -- gating on it cost another 30s timeout. What that check is about is the VIEW not moving, so the gate is "the app finished reacting". Fixing (1) exposed a real bug in the client, fixed here too: reopening a database that already exists while passing loader switches is FATAL in IDA -- FATAL ERROR: Switch '-b400' can be used only when loading a new file which kills the worker before it can report anything. Loader switches describe an IMPORT and are recorded in the database they produce, so they are now sent only when there is an import to describe. This was never reachable from the old suite (a fresh random blob never had a database to reopen), but it is reachable by any user who opens a raw blob with --ida-args twice. 30 passed, 0 failed. 2026-08-07T12:08:12Z blasty blasty@local 2026-08-07T12:08:12Z urn:sha1:ecc58d7725db6d4929ae3e299dea4f0202a81946 Benchmarking the port against master op-by-op (rather than only asking whether tests pass) turned up three real bugs, all in the most user-visible path: opening pseudocode. 1. decompile was doing decomp_map's job. It called the full per-column line map purely to fill in each line's /*0xEA*/ anchor. The tool ida-tui was written against takes ONE get_line_item at column 0 per line; the port took one per COLUMN, i.e. thousands of get_line_item+dstr() calls per function instead of one per line. Every pseudocode open cost the same as opening the split view. Carried the real implementation over: 1888ms -> 53ms. 2. decomp_map used the pre-optimisation line map. Ours memoises obj_id -> ea for the whole function (commit 853d90c: dstr() was 79% of the tool, and consecutive columns report the same ctree item), the port's did not. 1925ms -> 287ms. 3. _idatui_compact imported ida_pro_mcp on every call. Under Code Mode that package is not installed in the database process, so the import failed every time -- and a FAILED import is never cached, so each one re-searched the whole of sys.path: 422 failed imports per pc_nums call, which was most of its runtime. 1428ms -> 257ms. The same bug was a correctness bug hiding behind the perf bug: the fallback path collapsed whitespace INSIDE string literals, where the real function preserves it. Pseudocode columns are served in those coordinates, so on any line containing a string with two spaces, every literal's mark and every reformat would have been placed on the wrong column. It never fired on master because ida_pro_mcp is installed there. Now calls the byte-identical module-level shim directly, with the deviation from the extracted original documented in place. Narrow verification: decomp/split_view/opfmt/follow/comment/structs scenarios, 72 passed, 0 failed. Full gate running separately. 2026-08-07T11:50:42Z blasty blasty@local 2026-08-07T11:50:42Z urn:sha1:df4a42e25881531469a0a8399f457ecf58cab099 Found by tests/test_rawimage_rpc.py, which the earlier runs had not covered: every rename_many check failed with {"ok": 0, "failed": 2, "errors": [{"addr": null, "error": "list indices must be integers or slices, not str"}]} The port's rename read each category as a single edit (edit["addr"]), but the batch shape is {func: [{addr,name}, ...], data: [...], local/stack: [...]} -- a list per category, with a single dict accepted as shorthand. Indexing the list with "addr" raised, and because the whole category was one try block the error came back attached to addr=null, naming nothing. That is the entire point of the rename_many RPC verb: a firmware image arrives with hundreds of names from a loader map or an emulator's symbols.json, and applying them one at a time costs a navigation plus two prompt round trips each. Only the single-rename UI path worked. Now mirrors the real tool: one row per EDIT (addr/old/name plus a per-row error), a summary counting edits rather than categories, conflict detection before the write, and dry_run/allow_overwrite/stop_on_error. Renaming a function refreshes Hex-Rays' ctext, whose cache is per function and persisted in the .i64 -- without it the pseudocode keeps calling the old name forever while every other readback reports the new one. Clearing a label with an empty new name is kept as a real request (the scenarios revert with it) rather than being rejected as a missing argument. tests/test_rawimage_rpc.py: 14 passed/7 failed -> 21 passed, 0 failed. 2026-08-07T11:34:03Z blasty blasty@local 2026-08-07T11:34:03Z urn:sha1:ddeb7afea4553e73a7c1decd78fde968e40d186e This closes the five operations the port was missing and restores the listing's own tooling instead of a re-implementation of it. idatui/remote_tools.py is the port's IDAPython island: `heads` (the continuous listing) and `op_format`/`pc_nums`/`pc_num_format` (`o`/`O`), extracted verbatim from the BODY that server/patch_server.py used to inject. They are real, diffable source shipped to the database process as text, not string literals, because this is the most performance-tuned and behaviour-sensitive code in the project. Why carry `heads` over rather than keep the port's version: the port's rewrite emitted no per-operand extents ("ops"), so no keypress could show which literal it would reformat (opfmt_highlight had no two-operand row to find); it had no digest/`expect` support, so every page was re-sent after any edit; and its span walk was the per-character loop ours had already been rewritten out of. It also dropped struct-member expansion sizing and the func banner/label rows' exact shapes. The library is installed ONCE per database process (sys.modules, keyed by a hash of the source) and then called by name. Code Mode's execute_python builds a fresh namespace per call, so a library exec'd inline is rebuilt every time and its module-level caches thrown away -- the per-line render lru_cache in particular, which the perf work sized to 65536 entries. Installing it once took `heads` count=200 from 181ms to 92ms; the cache reports 211 hits on a second call where it previously reported none. (Extraction footgun recorded: ast FunctionDef.lineno points at `def`, not at the decorators, so a naive slice silently drops @lru_cache.) Also ported: flowchart, survey_binary, and the xref contract. Live pilot suite on targets/echo: 301 passed, 0 failed -- identical to master. Known, quantified, and NOT fixed here: Code Mode's transport is much slower than the unix-socket worker for the listing's paging. heads count=200 is 2.6ms on master vs 92ms here, count=500 is 6.3ms vs 214ms. Roughly half of that is to_jsonable + HTTP framing per call and is inherent to the architecture; the empty round trip alone is 2ms. The digest/`expect` path (unchanged pages) is the main mitigation and is restored. 2026-08-07T11:10:53Z blasty blasty@local 2026-08-07T11:10:53Z urn:sha1:3edb58cc0b93fdd90b68c6aacc8af91874142099 The pseudocode follow's address fallback broke: following a call landed on the NEXT LINE instead of the callee (decomp_nav's stale-name check, cur=0x20dd want=0x2060). The port's xref_query returned rows in raw IDA order, and at a call site IDA yields the ordinary-flow xref (fl_F, the next instruction) before the call xref (fl_CN), so 'first code xref' picked the fall-through. The tool ida-tui was written against sorts rows by the far-end address and dedups by default; sorted, 0x2060 precedes 0x210e and the follow is correct. That ordering is load-bearing, so it is now part of the port rather than an accident of the old implementation. Also fixed: the port attached 'fn' to ref.from_ea for both directions, where a from-xref must describe its TARGET (the xref dialog shows the wrong function otherwise), and the envelope was missing direction/addr/total/next_offset/resolved_addr. xref_types (ours, the kind badges in the xref dialog) is ported verbatim and deliberately stays UNsorted -- that dialog lists xrefs in IDA's own order. decomp_nav, follow_xrefs, xref_labels, decomp_follow_self: 15 passed, 0 failed. 2026-08-07T11:02:27Z blasty blasty@local 2026-08-07T11:02:27Z urn:sha1:5de318682c527976f3a298d38a0dc947c3a8c0dd Verified live now: ida-codemode 0.3.1 spawns a managed idalib worker on this box, so the pilot suite runs against the port. flowchart: the port simply does not have the operation, so domain.get_flowchart returned None and every graph key reported 'no control-flow graph for this function'. Ported ours onto ida_gdl (ida-domain exposes no basic-block or edge-kind surface). Blocks stay address RANGES, never text -- that is what lets graph boxes reuse the listing's own rows. Graph suite: 0 -> 50 passed. set_comments: the port set only the disassembly comment via db.comments.set_at(), so a comment never appeared in the pseudocode. A Hex-Rays comment is anchored to a ctree location and an anchor the ctree does not own is discarded as an orphan, so the itp slot must be searched until one sticks, and the entry ea is a function comment instead. Ported that logic back. survey_binary: added as the (caught) fallback domain.py expects behind file_regions, so the fallback path is real rather than always empty. 2026-08-07T10:56:27Z blasty blasty@local 2026-08-07T10:56:27Z urn:sha1:6e58e9e4ea72771619ded48b0ac2390a58d31cfb ida-codemode is now cloned at ../ida-codemode (0.3.1) and installed into ~/ida-venv, so the adapter can be checked against the library instead of against assumptions. First thing it found: connect() passed loading_address=, which DatabaseHandle.open() does not have. The real parameter is image_base, and it already wants the natural 16-byte-aligned address we compute, so this is a rename. Every connect would have died with TypeError on the first call. The port's own contract test could not catch it: its fake handle takes **kwargs, so any keyword at all looks accepted. The test now also validates the keywords we send against inspect.signature(DatabaseHandle.open) when the library is importable, and skips that one check when it is not. Offline suite: 302 passed with the library installed, 302 without it.