# SPEED — ida-tui-maybe _Read this BEFORE running anything slow._ This repo does **not** use pytest for its suites; every suite is a standalone script with its own runner and its own tally line (`N passed, M failed` / `N checks, M failed`). `speedscan`'s generic pytest template is wrong here — these are the real commands. --- ## The two interpreters | python | has | use for | |---|---|---| | `python3` (system) | stdlib only | every **pure** suite. No IDA, no Code Mode. | | `~/ida-venv/bin/python` | textual + idapro + ida_codemode | anything marked `ida`, and the TUI itself | `tests/run.py --list` prints `pure` / `ida` per file. A `pure` file must keep running under system `python3` — that is a house rule, and it is why `idatui/codemode_client.py` defers its `ida_codemode` import instead of doing it at module top. --- ## Escalation ladder for THIS repo **Cheap gates (seconds, run these first)** ```bash python3 -c "import ast;ast.parse(open('idatui/codemode_client.py').read())" # ~0.05s syntax python3 tests/run.py --fast # every pure suite, 302 checks, 0.6s python3 tests/run.py graph -x # one suite by substring, fail-fast ``` **Narrow test — DEFAULT RUNG.** The pilot suite takes `--only `: ```bash ~/ida-venv/bin/python tests/test_scenarios.py /tmp/scratch_bin --only rename # ~2s ~/ida-venv/bin/python tests/test_scenarios.py --list # scenario names ~/ida-venv/bin/python tests/test_scenarios.py targets/echo --profile # where the time went ``` **`--profile` is how you find the next 20 seconds.** It reports, per scenario, seconds spent settling / in `wait()` / in keystrokes, and — the important one — any wait that **EXPIRED**, with its line number. An expired wait costs its whole timeout *and* means the check after it passed vacuously. Do not optimise this suite by guessing; run `--profile` and fix the top line. `--only` matches substrings, so `--only listing` runs listing_view + listing_name_addr + listing_make_string + listing_struct_expand. Name the scenario exactly to stay narrow. **Full gate — BACKGROUND ONLY, once, at the end** (see next section for the launch form): ```bash ~/ida-venv/bin/python tests/run.py # every suite, serial on purpose ``` --- ## Coverage map — which suite catches what The pilot suite is big and slow and it is **not** the whole story. Two backend bugs this port shipped were invisible to `test_scenarios.py` and only fell out of the smaller RPC/UI suites: | suite | ~time | catches | |---|---|---| | `test_scenarios.py` | 166s | the TUI end to end: nav, views, listing, graph, opfmt, rename-one | | `test_rawimage_rpc.py` | ~60s | **`rename_many` (batch rename), define/blob workflow, opfmt over RPC** | | `test_blob_ui.py` | ~40s | raw-image/blob UI | | `test_project_ui.py` | ~8s | multi-binary project UI | | `test_trace_rpc.py` / `test_trace_ui.py` | ~30s | Tenet trace integration | **If you touch a backend operation, grep for its callers and run the suite that owns them — not just the pilot.** `rename` is exercised one-edit-at-a-time by the pilot and as a *batch* only by `test_rawimage_rpc.py`, which is exactly where the list-vs-dict bug hid. --- ## Backgrounding in THIS harness — the gotcha that cost a turn `bgrun run ... ` **blocks the agent tool call** even though bgrun itself returns in ms. `bgrun` line 64 is `( "$@" >"$LOG" 2>&1; echo $? >"$EXIT" ) &` — the command's own stdio goes to the log, but the *backgrounded subshell* still holds the inherited stdout fd, and the harness's bash waits for EOF on that pipe, not for the parent to exit. **Always launch detached with all three fds redirected:** ```bash SK="$HOME/.the assistant/agent/skills/fast-feedback/scripts" timeout 20 setsid "$SK/bgrun" run fullgate -- ~/ida-venv/bin/python tests/run.py \ /tmp/bgrun.out 2>&1 ; cat /tmp/bgrun.out # returns in ~7ms timeout 20 "$SK/bgrun" check fullgate # non-blocking, safe ``` `bgrun check` / `bgrun list` do **not** need this treatment; only `run` does. **Never `pkill -f ` here.** The agent's own shell command line contains the pattern you are matching, so `pkill -f "tests/run.py"` kills the very shell issuing it and the rest of the command silently never runs. Kill by **pid** (`bgrun list`, `pgrep -af` first), then gate on it with `waitfor pid-gone `. **Killing a pilot leaves a Code Mode worker behind** for its `--lease-grace` (20s); it self-exits, and it is holding only that run's temp `.i64`, so it does not block a new run. Do not `kill -9` it — a hard-killed IDA wedges its database. --- ## Simulated keypresses cost 85ms each unless you patch Textual The single biggest cost in this repo's UI suites was **Textual's own key path**, not our code. `Pilot.press` → `App._press_keys` calls `wait_for_idle` *twice per key*, and that helper sleeps in 20ms granules until *process* time stops advancing — a CPU-load heuristic for "the state is predictable now", which takes more granules the busier the box is. Measured on this box: **84ms per keypress**, and the pilot presses enough keys for that to be **23s of its 43s**. `tests/_fixtures.py: fast_keys()` replaces it (call it at import; every UI suite does). Two halves, and the second is the point: - `textual.app.wait_for_idle` → a bare `await asyncio.sleep(0)`. - `Pilot.press` → send the keys, then **`settle(app)`** — pump drained, workers finished. Deleting the heuristic *without* this broke nine checks, so it was doing a job, badly; `settle` is strictly stronger and ~2ms. **What `settle` still cannot see — and what to do instead:** | driven by | example | gate | |---|---|---| | a worker | decompile, navigation, index load | `settle(app, pred)` | | a **timer** | the function filter's `set_timer(0.08)` debounce | `wait(lambda: rows < full)` | | a **frame** | `widget.region` / `size` / a repaint trace | `wait(lambda: inp.region.height >= 1)` | A settled app has not necessarily been laid out or painted. Every site in this repo that needs a frame or a debounce is commented as such — if a check that reads geometry or a filtered row count starts flaking, that is the reason, and the fix is a wait on the effect, never a longer sleep. The frame-dependent ones found this way, as a shopping list of what to suspect: `si.region` after the search prompt opens, `gv._minimap_rect()` / `gv.size` after the graph opens, `gv.render_line()` scraping box glyphs, and a `render_line` trace asserting a repaint happened at a restored scroll. All four passed for years on the 85ms-per-key sleep and failed within three runs without it — they were always races, just paid-for ones. ## The four ways a test here wastes minutes Every slow suite in this repo was slow for one of these, not for doing real work. Check them before optimising anything else. 1. **A wait on a signal that can no longer happen.** `wait(lambda: lst.model is not old, ..., 60)` was the idiom for "the edit landed". The perf work made edits KEEP the listing's walk and re-render in place, so the model object is never replaced: every one of those waits sat out its full timeout and the check afterwards passed **vacuously**. This alone was 30s in blob_ui and 4x60s in thumb_ui. → Gate on what the check is about (the row is code, the status says Thumb, the function is in the index), via `settle(app, pred)` from `idatui._sync`. 2. **A wait on a signal that is set too early.** `app._t` is assigned when the key is handled; the navigation it starts runs in a worker. Waiting on it and then reading the cursor is a race master won (single-digit ms backend) and Code Mode loses. → `settle()` gates on quiescence *and* the predicate, which is what you want. 3. **Regenerated fixtures.** `os.urandom` into a fresh `TemporaryDirectory` means new bytes at a new path every run, so the pristine-database cache can never apply and full auto-analysis is paid forever. → `_fixtures.synthetic(name, build)` writes deterministic bytes to a stable path; `staged()` then caches the analysis. Determinism is also correctness: "this blob has no functions" must not depend on luck. 4. **Deleting a database and reopening the same path.** Safe when the TUI owned a private worker; under Code Mode the previous phase's worker still holds the lease for its grace period, so the delete races a live owner and the reopen yields no listing. → Give each phase its own temp copy (`fresh_copy`). Never `os.remove` an `.i64` a suite is about to reopen. 5. **A flat `pause(d)` where a gate belongs.** `Ctx.pause` in the pilot *is* `settle` now (`d` is the upper bound, not the cost), which took 20.4s of fixed sleeping down to 2.8s across ~140 call sites. `Ctx.sleep(d)` is the escape hatch for the genuinely timer-driven; reach for it only after checking the table above. In the other suites the same conversion took `test_trace_ui` from 19.6s to 5.5s — it was 13.5s of `pilot.pause(1.0)`. **`settle(app, pred, timeout=...)`** (`idatui/_sync.py`) is the one true gate: it drains the message pump, waits for workers, and returns the moment `pred` holds. It is what the live RPC layer uses, so tests and driver agree on what "done" means. A bare `settle(app)` (no pred) means "the app finished reacting" — the right gate when the edit may legitimately do nothing (e.g. carving random bytes). ## Measured timings (2026-08-07, this box, warm; keypress work 2026-08-08) | command | time | notes | |---|---|---| | `python3 tests/run.py --fast` | **0.6s** | 302 checks, all pure suites | | `python3 tests/test_graph.py` | ~1s | 86 checks, pure layout engine | | `python3 tests/test_codemode_client.py` | ~0.1s | 14 checks, fakes the DatabaseHandle | | pilot boot (first scenario) | ~1.7s | opens the DB; seeded from `.pristine.i64` | | pilot `--only rename` | ~2s | 11 checks | | **pilot full (`test_scenarios.py`)** | **21.2s** | 57 scenarios, 313 checks (was 166s, then 62s) | | `test_rawimage_rpc.py` | 7.5s | 21 checks, owns batch rename | | `test_trace_ui.py` | **5.5s** | 39 checks — was 19.6s | | `test_trace_rpc.py` | 5.1s | 45 checks | | `test_project_ui.py` | **3.8s** | 30 checks — was 8.1s | | `test_thumb_ui.py` | **3.6s** | 20 checks — was 313s AND crashing | | `test_blob_ui.py` | **1.3s** | 30 checks — was 39.8s | | `tests/run.py` (everything) | **49s** | 800 checks. was ~9m20s, then 117s | IDA-suite total: ~9m21s → 117s (the four wastes below) → **49s** (keypresses and the pause→settle conversion). No check was removed to get there; the suite gained 12 and the two runs behind these numbers were 49.2s and 48.9s, 800/800 both. Where the pilot's remaining ~20s goes (`--profile`): 10.8s keystrokes (which now includes the real work each key triggers, since `press` settles), 2.9s settling, 2.1s waits, ~5s scenario bodies. The rest of the gate is dominated by per-suite IDA boot — 7 processes, each opening its own database. ### Backend performance (worker vs Code Mode), after the two transport fixes | op | worker | codemode | | |---|---|---|---| | `heads` 200 rows | 2.65ms | 5.85ms | 2.2x | | `heads` expect-hit | 2.16ms | 4.61ms | 2.1x | | `disasm` 200 | 9.03ms | 5.25ms | **0.6x** | | `decompile` cold | 162.6ms | 30.7ms | **0.2x** | | `decomp_map` | 45.8ms | 47.9ms | 1.0x | | empty round trip | ~0.07ms | **2.0ms** | the floor | Two things dominated and are fixed (see `idatui/codemode_client.py:_script`): `to_jsonable` walking every returned object (snippets now return one pre-serialised JSON string), and `sys.settrace` — the runtime installs a trace that returns itself, i.e. LINE tracing in every frame, which made `ida_bytes.get_flags` 52x slower than native. The snippet detaches it and restores it in a finally; `IDATUI_CODEMODE_TRACE=1` keeps the stock behaviour. **What is left is the 2ms round-trip floor, and it is NOT ours.** Measured against the same worker, same connection: | | cost | whose | |---|---|---| | `GET /health` (no execute_sync) | 0.165ms | HTTP transport | | `execute_python("result = 1")` | 2.025ms | + `ida_kernwin.execute_sync` | So HTTP is 7% of the floor and marshalling an operation onto IDA's main thread is 92%. The worker runs IDA's own `kernwin.serve()` dispatch loop, so that latency is inside IDA, not something Code Mode exposes a knob for. **Do not re-chase this by batching operations.** The call volume is already minimal, measured on `targets/bash`: | flow | wall | calls | |---|---|---| | open a listing | 3.2ms | 1 | | scroll 2000 rows | 87.9ms | 8 | | rename + re-render those rows | 36.2ms | 4 | | graph of a **1060-block** function | 171.0ms | 4 | | decompile a 17785-byte function | 10806ms | 1 | Block-coalescing, the page digest and the prefetch caches already collapse the bursts, so a batch endpoint would save single-digit milliseconds on flows that cost hundreds. And the big number is pure Hex-Rays: that same decompile is 10723ms on the worker backend (0.8% apart) — there is no transport in it at all. _Superseded note:_ **What is left is the 2ms round-trip floor.** A trivial op (`data_type`, `force_recompile`, one xref query) is ~2.5ms wall clock and looks like 40x against an in-process worker. That is fixed by making FEWER calls, not faster ones — which is what the digest/`expect` path does for the listing. Benchmark harness: `/tmp/cmport/bench.py` + `compare.py` (backend-agnostic; it picks whichever client the checked-out tree has, so it runs on master too). --- ## Known-flaky - _(fixed 2026-08-08)_ `test_scenarios.py --only listing_view` used to fail about one run in three: `listing shows data heads (not just code)` reported `{'label','sep','funchdr','code'}`, i.e. the model was read before its pages had materialised. The scenario waited on `c.lst.total > 0`, which is computed from the segment size and is true **before any row exists** — the textbook "wait on a signal that is set too early". It now waits for a materialised data row. Nothing else is known-flaky; if something starts flaking, check the timer/frame table above first. --- ## Slow traps - **Never run the pilot suite against `targets/echo` directly.** It edits the database. The suite already stages a scratch copy (`tests/_fixtures.py: staged()`), seeded from `.pristine.i64` which nothing writes back to. Pass a **copy** in `/tmp` when invoking by hand, or you are testing your own history. - **`tests/run.py` runs suites serially on purpose.** Running them 4-up took the suite from 153s to 296s and got three killed mid-analysis (idalib contends hard). Do not "optimise" it with parallelism. - **Cold analysis dominates a first run.** `.pristine.i64` turns ~30s of auto-analysis into a file copy; if it is missing or older than the binary it is rebuilt. A suite that suddenly takes minutes longer is usually rebuilding that cache. - **Under load, idalib gets SIGKILLed mid-analysis** and it looks like a hang or empty output. Check `uptime` before believing a failure. - **`experiments/*.py` need `PYTHONPATH=$PWD`** under `~/ida-venv/bin/python` (they are scripts, not a package entry point). ## Comparing against master (backend A/B) The port replaces the whole backend, so "is this a regression?" means running the same scenario on both. `git stash -u; git checkout master; ; git checkout -; git stash pop` works, but **use a different scratch binary per branch** (`/tmp/x_master`, `/tmp/x_port`) so the two runs never share a `.pristine.i64` or a live Code Mode instance.