| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
| |
NOT running) revealed that the worker-connect poll change made test_project_ui flaky: 5ms polling on a background thread through a cold auto-analysis starved the UI thread enough that the loading overlay was still up when the test pressed Ctrl+O. Poll now backs off to a 25ms cap (keeps the boot win, no busy-wait), the racy boot wait is fixed, and checks.sh runs tests/run.py in full (830 checks) instead of just the scenario suite.
Result: {"status":"keep","total_ms":18608,"lg_boot_ms":708.5,"lg_decomp_ms":2454.9,"lg_graph_ms":1034.5,"lg_hex_ms":431.9,"lg_index_ms":95.1,"lg_listing_cold_ms":530.2,"lg_listing_warm_ms":413.4,"lg_nav_ms":7057.9,"lg_palette_ms":5,"lg_render_ms":214.2,"lg_search_ms":1408.9,"pure_graph_ms":213.3,"sm_boot_ms":433.6,"sm_decomp_ms":1292,"sm_graph_ms":754,"sm_hex_ms":433.4,"sm_index_ms":2.6,"sm_listing_cold_ms":260.4,"sm_listing_warm_ms":280.5,"sm_nav_ms":286.1,"sm_palette_ms":0.3,"sm_render_ms":251,"sm_search_ms":46.5,"fails":0}
|
| |
|
|
|
|
| |
(a decompilation uses ~18 distinct token types but each token walked up to nine 'token in ttype' hierarchy checks), and hold the worker-connect poll at 5ms for the first 5s instead of backing off geometrically from the first probe.
Result: {"status":"keep","total_ms":18856.8,"lg_boot_ms":689.7,"lg_decomp_ms":2484.2,"lg_graph_ms":1120,"lg_hex_ms":700.1,"lg_index_ms":96.5,"lg_listing_cold_ms":425.6,"lg_listing_warm_ms":511.5,"lg_nav_ms":6590.7,"lg_palette_ms":4.8,"lg_render_ms":215.1,"lg_search_ms":2195.5,"pure_graph_ms":238.1,"sm_boot_ms":431.5,"sm_decomp_ms":667.2,"sm_graph_ms":686.9,"sm_hex_ms":569.8,"sm_index_ms":0,"sm_listing_cold_ms":258.1,"sm_listing_warm_ms":283.2,"sm_nav_ms":365.2,"sm_palette_ms":0.3,"sm_render_ms":243.7,"sm_search_ms":79,"fails":0}
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The layer between the app and idalib had no tests, which is awkward: it is
where failures are silent. A worker that dies during startup, a socket that
drops mid-call, two UI threads sharing one socket -- none of those look like
bugs from outside, they look like the TUI hanging or showing stale data.
None of it needs IDA. WorkerClient spawns whatever _WORKER_PY points at, so the
suite points it at a fake speaking the same length-prefixed pickle and tells it
to misbehave on demand: die at startup, never bind, drop the connection, fail a
tool, take its time. 40 checks in the --fast tier.
Two things the tests found:
call() reconnects when _sock is None, which is what makes a dropped socket
recoverable -- but it made an explicitly CLOSED client resurrect too, spawning a
whole new idalib worker to serve one stray call (verified: pid 1066961 ->
1066962). close() runs on teardown and on binary-switch while @work threads are
still in flight, so quitting during a decompile could leave a fresh process
re-opening the .i64 we had just released, which is the wedging hazard. A closed
client now refuses; connect() still revives it, which is all _reconnect needs
(it builds a new client anyway).
connect() polled on a flat 0.2s sleep, so every caller paid a fifth of a second
even when the worker was ready in milliseconds -- a seeded .i64, a small binary.
Backs off from 5ms instead.
786 checks, 144.6s; --fast is 297 in 3.3s.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A raw firmware dump has no format to detect, so IDA fell back to x86 at address
0. It doesn't fail — it opens, analyses, and finds nothing. An AArch64 image
loaded this way gave 0 functions; told the truth it gives 35.
ida-tui fw.bin --processor arm --base 0x8000000
and per binary in a project, which is what a multi-image firmware actually
needs:
{"path": "app.bin", "processor": "arm", "base": "0x8000000"}
idapro.open_database() already accepted IDA command-line switches; nothing was
passing any. Plumbed BinaryRef -> WorkerPool -> WorkerClient -> worker argv, plus
a load_args for the single-binary path that has no project ref.
base is written the way people say it (0x8000000, int or string, any base).
IDA's -b is in PARAGRAPHS — -b1000 loads at 0x10000 — so BinaryRef.load_args
converts, and a base that isn't 16-byte aligned is refused rather than silently
landing 16x off. ida_args passes anything else through.
Two bugs found by testing the whole path rather than the happy one:
* Project.load() whitelisted path/label when normalising entries, so the load
options were dropped the first time a project was reopened — set a processor,
come back tomorrow, it's gone.
* Re-passing the switches to an EXISTING database makes IDA refuse the open
(rc != 0, no functions). The .i64 already records how the image was loaded, so
the worker skips them once a database exists. My first guard checked
splitext(path) + ".i64" and never fired, because IDA names it "<file>.i64" —
keeping the extension. It checks both spellings now.
Verified on a real AArch64 blob: fresh load 35 functions based at 0x8002440,
reopen 35 again, and the CLI rejects an unaligned or non-numeric --base.
tests: +6 project (options recorded, paragraph conversion, file round-trip,
add() takes them, an ELF passes nothing, hex-string base). 39/0 project, 195/0
scenarios, 30/0 project UI, 36/0 index, 27/0 pool.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
idatui/pool.py — WorkerPool keeps one live worker per project binary:
* lazy spawn on first use; staging + a scratch sweep happen first, so a database
wedged by a previously hard-killed worker reopens instead of crash-looping.
* residency is bounded by a MEMORY BUDGET (default project.memory_pct of RAM),
not a worker count — a count is the wrong knob when one project holds a 50KB
helper and a 6MB crypto lib. Cost is measured per worker from
/proc/<pid>/smaps_rollup (PSS, which splits shared pages so summing means
something).
* over budget -> evict least-recently-used, never the active or pinned binary,
and give up rather than thrash when nothing is evictable. Eviction calls
idb_save first, so returning to a binary is a DB load, not a re-analysis.
* status() feeds the switcher UI (resident/pinned/active/analysed/memory).
worker_client: fix the wedge-file bug this depends on. close() sent
__shutdown__ and then IMMEDIATELY SIGTERMed, killing the worker mid
close_database() — which is what leaves the unpacked .id0/.id1/... behind and
makes the .i64 refuse to reopen. Now it waits out a grace period (the worker
returns from serve() on __shutdown__ and closes the DB in its finally) and only
escalates if it's genuinely stuck. Also expose .pid for memory accounting.
Verified: a pilot run that used to leave echo.id0/.id1/.id2/.nam/.til now leaves
only echo.i64, and teardown is no slower (14 checks in 5s).
tests/test_pool.py: 22 checks with an injected fake client — LRU order, budget
eviction, active/pinned protection, save-before-close, thrash avoidance,
status(), teardown. Pure stdlib, no idalib.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
The IDAError/IDAConnectionError/IDAToolError/... exceptions and the Session
dataclass were defined in client.py (the ida-pro-mcp HTTP client), but the idalib
worker path (worker_client/domain/app) needs them without the HTTP transport.
Move them to a transport-agnostic errors.py; client.py re-exports them so the
deprecated mcp tooling and stress tests are unchanged (verified:
errors.IDAToolError IS client.IDAToolError, so cross-module `except` still works).
worker_client, domain (TYPE_CHECKING-guarded IDAClient hint), app, and __init__
now import the shared types from errors.py. This decouples the worker path from
client.py at runtime -- the prerequisite for deleting the mcp transport.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Root cause of "ModuleNotFoundError: No module named 'ida_pro_mcp'": the worker was
spawned with sys.executable — the TUI's python (~/ida-venv) which has idalib +
textual but NOT ida_pro_mcp. The package split on this box:
/usr/bin/python : idapro + ida_pro_mcp (the "IDA python")
~/ida-venv/python : idapro + textual (the "TUI python", runs the app)
Fix: WorkerClient now auto-detects a python that can import ida_pro_mcp
(IDATUI_WORKER_PYTHON override, else /usr/bin/python[3], else sys.executable) and
runs worker.py as a SCRIPT rather than `-m idatui.worker`, so it doesn't import
the textual-dependent idatui package __init__ under a python that has no textual.
worker.py itself is pure stdlib at load; idapro/ida_pro_mcp are imported at
runtime (both present in the IDA python).
Verified: detection returns /usr/bin/python; worker.py loads clean there.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The worker's stderr was swallowed by the TUI, so an open failure showed only
"worker exited during startup (code 1)". Now:
* WorkerClient captures the worker's stdout+stderr to /tmp/idatui-worker-*.log
and, on a startup exit, surfaces the last meaningful line in the error (the
worker prints a clean 'WORKER-FATAL: ...' marker; _log_tail prefers it).
* worker.py wraps main() to print that marker + traceback before exiting 1, and
gives an ACTIONABLE open error: "failed to open <bin>: the .i64 is likely held
by a running ida-mcp worker (pkill -f idalib) or wedged (delete .id0/.id1/
.id2/.nam/.til)". Also calls ida_auto.auto_wait() after open to fully match
ida-mcp's session manager (open_database + auto_wait).
Root cause of the reported failure is almost certainly a leftover ida-mcp worker
still holding bash's .i64 from earlier --backend mcp runs: idalib can't open a
database another process has locked. Fix: pkill -f idalib, then retry
--backend worker; the error message now says so instead of "code 1".
|
|
|
step 1
First concrete step off the mcp HTTP transport. Instead of reimplementing ~25
tools, reuse ida-pro-mcp's tool *functions* verbatim and replace only the
transport + process management:
* idatui/worker.py — opens ONE database in-process on the main thread (as idalib
requires), imports ida_pro_mcp (which registers every stock + our patched-in
custom tool against MCP_SERVER), then serves MCP_SERVER.tools.methods[name]
(**args) over a unix socket with length-prefixed pickle. Serial on the main
thread (idalib is single-threaded; tools run inline through execute_sync).
Session-management tools (idb_open/idb_save/server_health/idb_list) are shimmed
since the worker *is* the single session.
* idatui/worker_client.py — WorkerClient exposes the exact surface the app/domain
use on the client (call/call_envelope/connect/set_db/resolve_db/list_sessions/
health/keepalive/close) and returns byte-identical payloads (the worker calls
the same functions IDAClient.call ultimately hits). So domain.py and the app
are UNCHANGED — you just construct a WorkerClient instead of an IDAClient.
Calls are serialized under a lock over one socket; keepalive is a no-op (the
worker is ours and never idles out).
Not wired into the app yet — the mcp path is fully intact.
Verified without idalib: pickle framing round-trips arbitrary payloads incl raw
bytes; WorkerClient has full IDAClient surface; call_envelope produces the
result.structuredContent shape domain.decompile() reads. The idalib E2E
(experiments/worker_smoke.py drives the real domain.Program read path through the
worker) is written but couldn't run here — this sandbox has degraded to reaping
any idalib spawn; the underlying unix-socket protocol already ran clean in the
inproc_spike bench (~50us/call), and the worker dispatches the same tool
functions the HTTP path does, so shapes match by construction.
Next: stand up progress reporting during analysis, then flip _connect/_reconnect
to build a WorkerClient behind a flag and run the pilot suite against it.
|
