SPEED — ida-tui-maybe

Read this BEFORE running anything slow. This repo does not use pytest for its suites; every suite is a standalone script with its own runner and its own tally line (N passed, M failed / N checks, M failed). speedscan's generic pytest template is wrong here — these are the real commands.

The two interpreters

tests/run.py --list prints pure / ida per file. A pure file must keep running under system python3 — that is a house rule, and it is why idatui/codemode_client.py defers its ida_codemode import instead of doing it at module top.

Escalation ladder for THIS repo

Cheap gates (seconds, run these first)

python3 -c "import ast;ast.parse(open('idatui/codemode_client.py').read())"   # ~0.05s syntax
python3 tests/run.py --fast                    # every pure suite, 302 checks, 0.6s
python3 tests/run.py graph -x                  # one suite by substring, fail-fast

Narrow test — DEFAULT RUNG. The pilot suite takes --only <substr,...>:

~/ida-venv/bin/python tests/test_scenarios.py /tmp/scratch_bin --only rename       # ~2s
~/ida-venv/bin/python tests/test_scenarios.py --list                              # scenario names
~/ida-venv/bin/python tests/test_scenarios.py targets/echo --profile              # where the time went

--profile is how you find the next 20 seconds. It reports, per scenario, seconds spent settling / in wait() / in keystrokes, and — the important one — any wait that EXPIRED, with its line number. An expired wait costs its whole timeout and means the check after it passed vacuously. Do not optimise this suite by guessing; run --profile and fix the top line. --only matches substrings, so --only listing runs listing_view + listing_name_addr + listing_make_string + listing_struct_expand. Name the scenario exactly to stay narrow.

Full gate — BACKGROUND ONLY, once, at the end (see next section for the launch form):

~/ida-venv/bin/python tests/run.py             # every suite, serial on purpose

Coverage map — which suite catches what

The pilot suite is big and slow and it is not the whole story. Two backend bugs this port shipped were invisible to test_scenarios.py and only fell out of the smaller RPC/UI suites:

If you touch a backend operation, grep for its callers and run the suite that owns them — not just the pilot. rename is exercised one-edit-at-a-time by the pilot and as a batch only by test_rawimage_rpc.py, which is exactly where the list-vs-dict bug hid.

Backgrounding in THIS harness — the gotcha that cost a turn

bgrun run ... blocks the agent tool call even though bgrun itself returns in ms. bgrun line 64 is ( "$@" >"$LOG" 2>&1; echo $? >"$EXIT" ) & — the command's own stdio goes to the log, but the backgrounded subshell still holds the inherited stdout fd, and the harness's bash waits for EOF on that pipe, not for the parent to exit.

Always launch detached with all three fds redirected:

SK="$HOME/.the assistant/agent/skills/fast-feedback/scripts"
timeout 20 setsid "$SK/bgrun" run fullgate -- ~/ida-venv/bin/python tests/run.py \
    </dev/null >/tmp/bgrun.out 2>&1 ; cat /tmp/bgrun.out      # returns in ~7ms
timeout 20 "$SK/bgrun" check fullgate                          # non-blocking, safe

bgrun check / bgrun list do not need this treatment; only run does.

Never pkill -f <pattern> here. The agent's own shell command line contains the pattern you are matching, so pkill -f "tests/run.py" kills the very shell issuing it and the rest of the command silently never runs. Kill by pid (bgrun list, pgrep -af first), then gate on it with waitfor pid-gone <pid>.

Killing a pilot leaves a Code Mode worker behind for its --lease-grace (20s); it self-exits, and it is holding only that run's temp .i64, so it does not block a new run. Do not kill -9 it — a hard-killed IDA wedges its database.

Simulated keypresses cost 85ms each unless you patch Textual

The single biggest cost in this repo's UI suites was Textual's own key path, not our code. Pilot.press → App._press_keys calls wait_for_idle twice per key, and that helper sleeps in 20ms granules until process time stops advancing — a CPU-load heuristic for "the state is predictable now", which takes more granules the busier the box is. Measured on this box: 84ms per keypress, and the pilot presses enough keys for that to be 23s of its 43s.

tests/_fixtures.py: fast_keys() replaces it (call it at import; every UI suite does). Two halves, and the second is the point:

• textual.app.wait_for_idle → a bare await asyncio.sleep(0).
• Pilot.press → send the keys, then settle(app) — pump drained, workers finished. Deleting the heuristic without this broke nine checks, so it was doing a job, badly; settle is strictly stronger and ~2ms.

What settle still cannot see — and what to do instead:

A settled app has not necessarily been laid out or painted. Every site in this repo that needs a frame or a debounce is commented as such — if a check that reads geometry or a filtered row count starts flaking, that is the reason, and the fix is a wait on the effect, never a longer sleep.

The frame-dependent ones found this way, as a shopping list of what to suspect: si.region after the search prompt opens, gv._minimap_rect() / gv.size after the graph opens, gv.render_line() scraping box glyphs, and a render_line trace asserting a repaint happened at a restored scroll. All four passed for years on the 85ms-per-key sleep and failed within three runs without it — they were always races, just paid-for ones.

The four ways a test here wastes minutes

Every slow suite in this repo was slow for one of these, not for doing real work. Check them before optimising anything else.

1. A wait on a signal that can no longer happen. wait(lambda: lst.model is not old, ..., 60) was the idiom for "the edit landed". The perf work made edits KEEP the listing's walk and re-render in place, so the model object is never replaced: every one of those waits sat out its full timeout and the check afterwards passed vacuously. This alone was 30s in blob_ui and 4x60s in thumb_ui. → Gate on what the check is about (the row is code, the status says Thumb, the function is in the index), via settle(app, pred) from idatui._sync.
2. A wait on a signal that is set too early. app._t is assigned when the key is handled; the navigation it starts runs in a worker. Waiting on it and then reading the cursor is a race master won (single-digit ms backend) and Code Mode loses. → settle() gates on quiescence and the predicate, which is what you want.
3. Regenerated fixtures. os.urandom into a fresh TemporaryDirectory means new bytes at a new path every run, so the pristine-database cache can never apply and full auto-analysis is paid forever. → _fixtures.synthetic(name, build) writes deterministic bytes to a stable path; staged() then caches the analysis. Determinism is also correctness: "this blob has no functions" must not depend on luck.

4. Deleting a database and reopening the same path. Safe when the TUI owned a private worker; under Code Mode the previous phase's worker still holds the lease for its grace period, so the delete races a live owner and the reopen yields no listing. → Give each phase its own temp copy (fresh_copy). Never os.remove an .i64 a suite is about to reopen.

5. A flat pause(d) where a gate belongs. Ctx.pause in the pilot is settle now (d is the upper bound, not the cost), which took 20.4s of fixed sleeping down to 2.8s across ~140 call sites. Ctx.sleep(d) is the escape hatch for the genuinely timer-driven; reach for it only after checking the table above. In the other suites the same conversion took test_trace_ui from 19.6s to 5.5s — it was 13.5s of pilot.pause(1.0).

settle(app, pred, timeout=...) (idatui/_sync.py) is the one true gate: it drains the message pump, waits for workers, and returns the moment pred holds. It is what the live RPC layer uses, so tests and driver agree on what "done" means. A bare settle(app) (no pred) means "the app finished reacting" — the right gate when the edit may legitimately do nothing (e.g. carving random bytes).

Measured timings (2026-08-07, this box, warm; keypress work 2026-08-08)

IDA-suite total: ~9m21s → 117s (the four wastes below) → 49s (keypresses and the pause→settle conversion). No check was removed to get there; the suite gained 12 and the two runs behind these numbers were 49.2s and 48.9s, 800/800 both.

Where the pilot's remaining ~20s goes (--profile): 10.8s keystrokes (which now includes the real work each key triggers, since press settles), 2.9s settling, 2.1s waits, ~5s scenario bodies. The rest of the gate is dominated by per-suite IDA boot — 7 processes, each opening its own database.

Backend performance (worker vs Code Mode), after the two transport fixes

Two things dominated and are fixed (see idatui/codemode_client.py:_script): to_jsonable walking every returned object (snippets now return one pre-serialised JSON string), and sys.settrace — the runtime installs a trace that returns itself, i.e. LINE tracing in every frame, which made ida_bytes.get_flags 52x slower than native. The snippet detaches it and restores it in a finally; IDATUI_CODEMODE_TRACE=1 keeps the stock behaviour.

What is left is the 2ms round-trip floor, and it is NOT ours. Measured against the same worker, same connection:

So HTTP is 7% of the floor and marshalling an operation onto IDA's main thread is 92%. The worker runs IDA's own kernwin.serve() dispatch loop, so that latency is inside IDA, not something Code Mode exposes a knob for.

Do not re-chase this by batching operations. The call volume is already minimal, measured on targets/bash:

Block-coalescing, the page digest and the prefetch caches already collapse the bursts, so a batch endpoint would save single-digit milliseconds on flows that cost hundreds. And the big number is pure Hex-Rays: that same decompile is 10723ms on the worker backend (0.8% apart) — there is no transport in it at all.

Superseded note: What is left is the 2ms round-trip floor. A trivial op (data_type, force_recompile, one xref query) is ~2.5ms wall clock and looks like 40x against an in-process worker. That is fixed by making FEWER calls, not faster ones — which is what the digest/expect path does for the listing.

Benchmark harness: /tmp/cmport/bench.py + compare.py (backend-agnostic; it picks whichever client the checked-out tree has, so it runs on master too).

Known-flaky

• (fixed 2026-08-08) test_scenarios.py --only listing_view used to fail about one run in three: listing shows data heads (not just code) reported {'label','sep','funchdr','code'}, i.e. the model was read before its pages had materialised. The scenario waited on c.lst.total > 0, which is computed from the segment size and is true before any row exists — the textbook "wait on a signal that is set too early". It now waits for a materialised data row. Nothing else is known-flaky; if something starts flaking, check the timer/frame table above first.

Slow traps

• Never run the pilot suite against targets/echo directly. It edits the database. The suite already stages a scratch copy (tests/_fixtures.py: staged()), seeded from <binary>.pristine.i64 which nothing writes back to. Pass a copy in /tmp when invoking by hand, or you are testing your own history.
• tests/run.py runs suites serially on purpose. Running them 4-up took the suite from 153s to 296s and got three killed mid-analysis (idalib contends hard). Do not "optimise" it with parallelism.
• Cold analysis dominates a first run. .pristine.i64 turns ~30s of auto-analysis into a file copy; if it is missing or older than the binary it is rebuilt. A suite that suddenly takes minutes longer is usually rebuilding that cache.
• Under load, idalib gets SIGKILLed mid-analysis and it looks like a hang or empty output. Check uptime before believing a failure.
• experiments/*.py need PYTHONPATH=$PWD under ~/ida-venv/bin/python (they are scripts, not a package entry point).

Comparing against master (backend A/B)

The port replaces the whole backend, so "is this a regression?" means running the same scenario on both. git stash -u; git checkout master; <run>; git checkout -; git stash pop works, but use a different scratch binary per branch (/tmp/x_master, /tmp/x_port) so the two runs never share a .pristine.i64 or a live Code Mode instance.