ida-tui.git/idatui/worker_client.py, branch pre-codemode

CORRECTNESS REPAIR, kept on its merits. The full suite (which the gate was NOT running) revealed that the worker-connect poll change made test_project_ui flaky: 5ms polling on a background thread through a cold auto-analysis starved the UI thread enough that the loading overlay was still up when the test pressed Ctrl+O. Poll now backs off to a 25ms cap (keeps the boot win, no busy-wait), the racy boot wait is fixed, and checks.sh runs tests/run.py in full (830 checks) instead of just the scenario suite.

2026-08-07T02:00:03Z

Result: {"status":"keep","total_ms":18608,"lg_boot_ms":708.5,"lg_decomp_ms":2454.9,"lg_graph_ms":1034.5,"lg_hex_ms":431.9,"lg_index_ms":95.1,"lg_listing_cold_ms":530.2,"lg_listing_warm_ms":413.4,"lg_nav_ms":7057.9,"lg_palette_ms":5,"lg_render_ms":214.2,"lg_search_ms":1408.9,"pure_graph_ms":213.3,"sm_boot_ms":433.6,"sm_decomp_ms":1292,"sm_graph_ms":754,"sm_hex_ms":433.4,"sm_index_ms":2.6,"sm_listing_cold_ms":260.4,"sm_listing_warm_ms":280.5,"sm_nav_ms":286.1,"sm_palette_ms":0.3,"sm_render_ms":251,"sm_search_ms":46.5,"fails":0}

Two independent constants: memoise the pygments token -> Rich style lookup (a decompilation uses ~18 distinct token types but each token walked up to nine 'token in ttype' hierarchy checks), and hold the worker-connect poll at 5ms for the first 5s instead of backing off geometrically from the first probe.

2026-08-07T00:35:07Z

Result: {"status":"keep","total_ms":18856.8,"lg_boot_ms":689.7,"lg_decomp_ms":2484.2,"lg_graph_ms":1120,"lg_hex_ms":700.1,"lg_index_ms":96.5,"lg_listing_cold_ms":425.6,"lg_listing_warm_ms":511.5,"lg_nav_ms":6590.7,"lg_palette_ms":4.8,"lg_render_ms":215.1,"lg_search_ms":2195.5,"pure_graph_ms":238.1,"sm_boot_ms":431.5,"sm_decomp_ms":667.2,"sm_graph_ms":686.9,"sm_hex_ms":569.8,"sm_index_ms":0,"sm_listing_cold_ms":258.1,"sm_listing_warm_ms":283.2,"sm_nav_ms":365.2,"sm_palette_ms":0.3,"sm_render_ms":243.7,"sm_search_ms":79,"fails":0}

worker_client: tests, and stop resurrecting a closed worker

2026-08-06T22:14:28Z

The layer between the app and idalib had no tests, which is awkward: it is where failures are silent. A worker that dies during startup, a socket that drops mid-call, two UI threads sharing one socket -- none of those look like bugs from outside, they look like the TUI hanging or showing stale data. None of it needs IDA. WorkerClient spawns whatever _WORKER_PY points at, so the suite points it at a fake speaking the same length-prefixed pickle and tells it to misbehave on demand: die at startup, never bind, drop the connection, fail a tool, take its time. 40 checks in the --fast tier. Two things the tests found: call() reconnects when _sock is None, which is what makes a dropped socket recoverable -- but it made an explicitly CLOSED client resurrect too, spawning a whole new idalib worker to serve one stray call (verified: pid 1066961 -> 1066962). close() runs on teardown and on binary-switch while @work threads are still in flight, so quitting during a decompile could leave a fresh process re-opening the .i64 we had just released, which is the wedging hazard. A closed client now refuses; connect() still revives it, which is all _reconnect needs (it builds a new client anyway). connect() polled on a flat 0.2s sleep, so every caller paid a fifth of a second even when the worker was ready in milliseconds -- a seeded .i64, a small binary. Backs off from 5ms instead. 786 checks, 144.6s; --fast is 297 in 3.3s.

loading: say how to read a headerless blob (processor, base)

2026-07-25T21:49:14Z

A raw firmware dump has no format to detect, so IDA fell back to x86 at address 0. It doesn't fail — it opens, analyses, and finds nothing. An AArch64 image loaded this way gave 0 functions; told the truth it gives 35. ida-tui fw.bin --processor arm --base 0x8000000 and per binary in a project, which is what a multi-image firmware actually needs: {"path": "app.bin", "processor": "arm", "base": "0x8000000"} idapro.open_database() already accepted IDA command-line switches; nothing was passing any. Plumbed BinaryRef -> WorkerPool -> WorkerClient -> worker argv, plus a load_args for the single-binary path that has no project ref. base is written the way people say it (0x8000000, int or string, any base). IDA's -b is in PARAGRAPHS — -b1000 loads at 0x10000 — so BinaryRef.load_args converts, and a base that isn't 16-byte aligned is refused rather than silently landing 16x off. ida_args passes anything else through. Two bugs found by testing the whole path rather than the happy one: * Project.load() whitelisted path/label when normalising entries, so the load options were dropped the first time a project was reopened — set a processor, come back tomorrow, it's gone. * Re-passing the switches to an EXISTING database makes IDA refuse the open (rc != 0, no functions). The .i64 already records how the image was loaded, so the worker skips them once a database exists. My first guard checked splitext(path) + ".i64" and never fired, because IDA names it ".i64" — keeping the extension. It checks both spellings now. Verified on a real AArch64 blob: fresh load 35 functions based at 0x8002440, reopen 35 again, and the CLI rejects an unaligned or non-numeric --base. tests: +6 project (options recorded, paragraph conversion, file round-trip, add() takes them, an ELF passes nothing, hex-string base). 39/0 project, 195/0 scenarios, 30/0 project UI, 36/0 index, 27/0 pool.

projects: worker pool with memory-budgeted residency (phase 1b)

2026-07-25T09:29:36Z

idatui/pool.py — WorkerPool keeps one live worker per project binary: * lazy spawn on first use; staging + a scratch sweep happen first, so a database wedged by a previously hard-killed worker reopens instead of crash-looping. * residency is bounded by a MEMORY BUDGET (default project.memory_pct of RAM), not a worker count — a count is the wrong knob when one project holds a 50KB helper and a 6MB crypto lib. Cost is measured per worker from /proc//smaps_rollup (PSS, which splits shared pages so summing means something). * over budget -> evict least-recently-used, never the active or pinned binary, and give up rather than thrash when nothing is evictable. Eviction calls idb_save first, so returning to a binary is a DB load, not a re-analysis. * status() feeds the switcher UI (resident/pinned/active/analysed/memory). worker_client: fix the wedge-file bug this depends on. close() sent __shutdown__ and then IMMEDIATELY SIGTERMed, killing the worker mid close_database() — which is what leaves the unpacked .id0/.id1/... behind and makes the .i64 refuse to reopen. Now it waits out a grace period (the worker returns from serve() on __shutdown__ and closes the DB in its finally) and only escalates if it's genuinely stuck. Also expose .pid for memory accounting. Verified: a pilot run that used to leave echo.id0/.id1/.id2/.nam/.til now leaves only echo.i64, and teardown is no slower (14 checks in 5s). tests/test_pool.py: 22 checks with an injected fake client — LRU order, budget eviction, active/pinned protection, save-before-close, thrash avoidance, status(), teardown. Pure stdlib, no idalib.

refactor: extract shared error hierarchy + Session into idatui/errors.py

2026-07-24T12:41:23Z

The IDAError/IDAConnectionError/IDAToolError/... exceptions and the Session dataclass were defined in client.py (the ida-pro-mcp HTTP client), but the idalib worker path (worker_client/domain/app) needs them without the HTTP transport. Move them to a transport-agnostic errors.py; client.py re-exports them so the deprecated mcp tooling and stress tests are unchanged (verified: errors.IDAToolError IS client.IDAToolError, so cross-module `except` still works). worker_client, domain (TYPE_CHECKING-guarded IDAClient hint), app, and __init__ now import the shared types from errors.py. This decouples the worker path from client.py at runtime -- the prerequisite for deleting the mcp transport.

worker: spawn under the IDA python (has ida_pro_mcp), not the TUI's

2026-07-23T23:47:05Z

Root cause of "ModuleNotFoundError: No module named 'ida_pro_mcp'": the worker was spawned with sys.executable — the TUI's python (~/ida-venv) which has idalib + textual but NOT ida_pro_mcp. The package split on this box: /usr/bin/python : idapro + ida_pro_mcp (the "IDA python") ~/ida-venv/python : idapro + textual (the "TUI python", runs the app) Fix: WorkerClient now auto-detects a python that can import ida_pro_mcp (IDATUI_WORKER_PYTHON override, else /usr/bin/python[3], else sys.executable) and runs worker.py as a SCRIPT rather than `-m idatui.worker`, so it doesn't import the textual-dependent idatui package __init__ under a python that has no textual. worker.py itself is pure stdlib at load; idapro/ida_pro_mcp are imported at runtime (both present in the IDA python). Verified: detection returns /usr/bin/python; worker.py loads clean there.

worker: surface the real startup failure (not just "code 1")

2026-07-23T23:42:43Z

The worker's stderr was swallowed by the TUI, so an open failure showed only "worker exited during startup (code 1)". Now: * WorkerClient captures the worker's stdout+stderr to /tmp/idatui-worker-*.log and, on a startup exit, surfaces the last meaningful line in the error (the worker prints a clean 'WORKER-FATAL: ...' marker; _log_tail prefers it). * worker.py wraps main() to print that marker + traceback before exiting 1, and gives an ACTIONABLE open error: "failed to open : the .i64 is likely held by a running ida-mcp worker (pkill -f idalib) or wedged (delete .id0/.id1/ .id2/.nam/.til)". Also calls ida_auto.auto_wait() after open to fully match ida-mcp's session manager (open_database + auto_wait). Root cause of the reported failure is almost certainly a leftover ida-mcp worker still holding bash's .i64 from earlier --backend mcp runs: idalib can't open a database another process has locked. Fix: pkill -f idalib, then retry --backend worker; the error message now says so instead of "code 1".

worker: idalib worker + WorkerClient (drop-in for IDAClient) — migration step 1

2026-07-23T23:29:06Z

First concrete step off the mcp HTTP transport. Instead of reimplementing ~25 tools, reuse ida-pro-mcp's tool *functions* verbatim and replace only the transport + process management: * idatui/worker.py — opens ONE database in-process on the main thread (as idalib requires), imports ida_pro_mcp (which registers every stock + our patched-in custom tool against MCP_SERVER), then serves MCP_SERVER.tools.methods[name] (**args) over a unix socket with length-prefixed pickle. Serial on the main thread (idalib is single-threaded; tools run inline through execute_sync). Session-management tools (idb_open/idb_save/server_health/idb_list) are shimmed since the worker *is* the single session. * idatui/worker_client.py — WorkerClient exposes the exact surface the app/domain use on the client (call/call_envelope/connect/set_db/resolve_db/list_sessions/ health/keepalive/close) and returns byte-identical payloads (the worker calls the same functions IDAClient.call ultimately hits). So domain.py and the app are UNCHANGED — you just construct a WorkerClient instead of an IDAClient. Calls are serialized under a lock over one socket; keepalive is a no-op (the worker is ours and never idles out). Not wired into the app yet — the mcp path is fully intact. Verified without idalib: pickle framing round-trips arbitrary payloads incl raw bytes; WorkerClient has full IDAClient surface; call_envelope produces the result.structuredContent shape domain.decompile() reads. The idalib E2E (experiments/worker_smoke.py drives the real domain.Program read path through the worker) is written but couldn't run here — this sandbox has degraded to reaping any idalib spawn; the underlying unix-socket protocol already ran clean in the inproc_spike bench (~50us/call), and the worker dispatches the same tool functions the HTTP path does, so shapes match by construction. Next: stand up progress reporting during analysis, then flip _connect/_reconnect to build a WorkerClient behind a flag and run the pilot suite against it.