path: root/.fastfeedback/SPEED.md

# SPEED — ida-tui-maybe

_Read this BEFORE running anything slow._ This repo does **not** use pytest for its
suites; every suite is a standalone script with its own runner and its own tally line
(`N passed, M failed` / `N checks, M failed`). `speedscan`'s generic pytest template is
wrong here — these are the real commands.

---

## The two interpreters

| python | has | use for |
|---|---|---|
| `python3` (system) | stdlib only | every **pure** suite. No IDA, no Code Mode. |
| `~/ida-venv/bin/python` | textual + idapro + ida_codemode | anything marked `ida`, and the TUI itself |

`tests/run.py --list` prints `pure` / `ida` per file. A `pure` file must keep running
under system `python3` — that is a house rule, and it is why `idatui/codemode_client.py`
defers its `ida_codemode` import instead of doing it at module top.

---

## Escalation ladder for THIS repo

**Cheap gates (seconds, run these first)**
```bash
python3 -c "import ast;ast.parse(open('idatui/codemode_client.py').read())"   # ~0.05s syntax
python3 tests/run.py --fast                    # every pure suite, 302 checks, 0.6s
python3 tests/run.py graph -x                  # one suite by substring, fail-fast
```

**Narrow test — DEFAULT RUNG.** The pilot suite takes `--only <substr,...>`:
```bash
~/ida-venv/bin/python tests/test_scenarios.py /tmp/scratch_bin --only rename       # ~2s
~/ida-venv/bin/python tests/test_scenarios.py --list                              # scenario names
~/ida-venv/bin/python tests/test_scenarios.py targets/echo --profile              # where the time went
```

**`--profile` is how you find the next 20 seconds.** It reports, per scenario,
seconds spent settling / in `wait()` / in keystrokes, and — the important one —
any wait that **EXPIRED**, with its line number. An expired wait costs its whole
timeout *and* means the check after it passed vacuously. Do not optimise this
suite by guessing; run `--profile` and fix the top line.
`--only` matches substrings, so `--only listing` runs listing_view + listing_name_addr +
listing_make_string + listing_struct_expand. Name the scenario exactly to stay narrow.

**Full gate — BACKGROUND ONLY, once, at the end** (see next section for the launch form):
```bash
~/ida-venv/bin/python tests/run.py             # every suite, serial on purpose
```

---

## Coverage map — which suite catches what

The pilot suite is big and slow and it is **not** the whole story. Two backend bugs
this port shipped were invisible to `test_scenarios.py` and only fell out of the
smaller RPC/UI suites:

| suite | ~time | catches |
|---|---|---|
| `test_scenarios.py` | 166s | the TUI end to end: nav, views, listing, graph, opfmt, rename-one |
| `test_rawimage_rpc.py` | ~60s | **`rename_many` (batch rename), define/blob workflow, opfmt over RPC** |
| `test_blob_ui.py` | ~40s | raw-image/blob UI |
| `test_project_ui.py` | ~8s | multi-binary project UI |
| `test_trace_rpc.py` / `test_trace_ui.py` | ~30s | Tenet trace integration |

**If you touch a backend operation, grep for its callers and run the suite that owns
them — not just the pilot.** `rename` is exercised one-edit-at-a-time by the pilot and
as a *batch* only by `test_rawimage_rpc.py`, which is exactly where the list-vs-dict
bug hid.

---

## Backgrounding in THIS harness — the gotcha that cost a turn

`bgrun run ... ` **blocks the agent tool call** even though bgrun itself returns in ms.
`bgrun` line 64 is `( "$@" >"$LOG" 2>&1; echo $? >"$EXIT" ) &` — the command's own stdio
goes to the log, but the *backgrounded subshell* still holds the inherited stdout fd, and
the harness's bash waits for EOF on that pipe, not for the parent to exit.

**Always launch detached with all three fds redirected:**
```bash
SK="$HOME/.the assistant/agent/skills/fast-feedback/scripts"
timeout 20 setsid "$SK/bgrun" run fullgate -- ~/ida-venv/bin/python tests/run.py \
    </dev/null >/tmp/bgrun.out 2>&1 ; cat /tmp/bgrun.out      # returns in ~7ms
timeout 20 "$SK/bgrun" check fullgate                          # non-blocking, safe
```
`bgrun check` / `bgrun list` do **not** need this treatment; only `run` does.

**Never `pkill -f <pattern>` here.** The agent's own shell command line contains the
pattern you are matching, so `pkill -f "tests/run.py"` kills the very shell issuing it
and the rest of the command silently never runs. Kill by **pid** (`bgrun list`,
`pgrep -af` first), then gate on it with `waitfor pid-gone <pid>`.

**Killing a pilot leaves a Code Mode worker behind** for its `--lease-grace` (20s); it
self-exits, and it is holding only that run's temp `.i64`, so it does not block a new
run. Do not `kill -9` it — a hard-killed IDA wedges its database.

---

## Simulated keypresses cost 85ms each unless you patch Textual

The single biggest cost in this repo's UI suites was **Textual's own key path**,
not our code. `Pilot.press` → `App._press_keys` calls `wait_for_idle` *twice per
key*, and that helper sleeps in 20ms granules until *process* time stops
advancing — a CPU-load heuristic for "the state is predictable now", which takes
more granules the busier the box is. Measured on this box: **84ms per keypress**,
and the pilot presses enough keys for that to be **23s of its 43s**.

`tests/_fixtures.py: fast_keys()` replaces it (call it at import; every UI suite
does). Two halves, and the second is the point:

- `textual.app.wait_for_idle` → a bare `await asyncio.sleep(0)`.
- `Pilot.press` → send the keys, then **`settle(app)`** — pump drained, workers
  finished. Deleting the heuristic *without* this broke nine checks, so it was
  doing a job, badly; `settle` is strictly stronger and ~2ms.

**What `settle` still cannot see — and what to do instead:**

| driven by | example | gate |
|---|---|---|
| a worker | decompile, navigation, index load | `settle(app, pred)` |
| a **timer** | the function filter's `set_timer(0.08)` debounce | `wait(lambda: rows < full)` |
| a **frame** | `widget.region` / `size` / a repaint trace | `wait(lambda: inp.region.height >= 1)` |

A settled app has not necessarily been laid out or painted. Every site in this
repo that needs a frame or a debounce is commented as such — if a check that
reads geometry or a filtered row count starts flaking, that is the reason, and
the fix is a wait on the effect, never a longer sleep.

The frame-dependent ones found this way, as a shopping list of what to suspect:
`si.region` after the search prompt opens, `gv._minimap_rect()` / `gv.size` after
the graph opens, `gv.render_line()` scraping box glyphs, and a `render_line`
trace asserting a repaint happened at a restored scroll. All four passed for
years on the 85ms-per-key sleep and failed within three runs without it — they
were always races, just paid-for ones.

## The four ways a test here wastes minutes

Every slow suite in this repo was slow for one of these, not for doing real work.
Check them before optimising anything else.

1. **A wait on a signal that can no longer happen.** `wait(lambda: lst.model is not
   old, ..., 60)` was the idiom for "the edit landed". The perf work made edits KEEP
   the listing's walk and re-render in place, so the model object is never replaced:
   every one of those waits sat out its full timeout and the check afterwards passed
   **vacuously**. This alone was 30s in blob_ui and 4x60s in thumb_ui.
   → Gate on what the check is about (the row is code, the status says Thumb, the
   function is in the index), via `settle(app, pred)` from `idatui._sync`.
2. **A wait on a signal that is set too early.** `app._t` is assigned when the key is
   handled; the navigation it starts runs in a worker. Waiting on it and then reading
   the cursor is a race master won (single-digit ms backend) and Code Mode loses.
   → `settle()` gates on quiescence *and* the predicate, which is what you want.
3. **Regenerated fixtures.** `os.urandom` into a fresh `TemporaryDirectory` means new
   bytes at a new path every run, so the pristine-database cache can never apply and
   full auto-analysis is paid forever. → `_fixtures.synthetic(name, build)` writes
   deterministic bytes to a stable path; `staged()` then caches the analysis.
   Determinism is also correctness: "this blob has no functions" must not depend on luck.
4. **Deleting a database and reopening the same path.** Safe when the TUI owned a
   private worker; under Code Mode the previous phase's worker still holds the lease
   for its grace period, so the delete races a live owner and the reopen yields no
   listing. → Give each phase its own temp copy (`fresh_copy`). Never `os.remove` an
   `.i64` a suite is about to reopen.

5. **A flat `pause(d)` where a gate belongs.** `Ctx.pause` in the pilot *is*
   `settle` now (`d` is the upper bound, not the cost), which took 20.4s of
   fixed sleeping down to 2.8s across ~140 call sites. `Ctx.sleep(d)` is the
   escape hatch for the genuinely timer-driven; reach for it only after
   checking the table above. In the other suites the same conversion took
   `test_trace_ui` from 19.6s to 5.5s — it was 13.5s of `pilot.pause(1.0)`.

**`settle(app, pred, timeout=...)`** (`idatui/_sync.py`) is the one true gate: it drains
the message pump, waits for workers, and returns the moment `pred` holds. It is what the
live RPC layer uses, so tests and driver agree on what "done" means. A bare
`settle(app)` (no pred) means "the app finished reacting" — the right gate when the edit
may legitimately do nothing (e.g. carving random bytes).

## Measured timings (2026-08-07, this box, warm; keypress work 2026-08-08)

| command | time | notes |
|---|---|---|
| `python3 tests/run.py --fast` | **0.6s** | 302 checks, all pure suites |
| `python3 tests/test_graph.py` | ~1s | 86 checks, pure layout engine |
| `python3 tests/test_codemode_client.py` | ~0.1s | 14 checks, fakes the DatabaseHandle |
| pilot boot (first scenario) | ~1.7s | opens the DB; seeded from `.pristine.i64` |
| pilot `--only rename` | ~2s | 11 checks |
| **pilot full (`test_scenarios.py`)** | **21.2s** | 57 scenarios, 313 checks (was 166s, then 62s) |
| `test_rawimage_rpc.py` | 7.5s | 21 checks, owns batch rename |
| `test_trace_ui.py` | **5.5s** | 39 checks — was 19.6s |
| `test_trace_rpc.py` | 5.1s | 45 checks |
| `test_project_ui.py` | **3.8s** | 30 checks — was 8.1s |
| `test_thumb_ui.py` | **3.6s** | 20 checks — was 313s AND crashing |
| `test_blob_ui.py` | **1.3s** | 30 checks — was 39.8s |
| `tests/run.py` (everything) | **49s** | 800 checks. was ~9m20s, then 117s |

IDA-suite total: ~9m21s → 117s (the four wastes below) → **49s** (keypresses and
the pause→settle conversion). No check was removed to get there; the suite gained
12 and the two runs behind these numbers were 49.2s and 48.9s, 800/800 both.

Where the pilot's remaining ~20s goes (`--profile`): 10.8s keystrokes (which now
includes the real work each key triggers, since `press` settles), 2.9s settling,
2.1s waits, ~5s scenario bodies. The rest of the gate is dominated by per-suite
IDA boot — 7 processes, each opening its own database.

### Backend performance (worker vs Code Mode), after the two transport fixes

| op | worker | codemode | |
|---|---|---|---|
| `heads` 200 rows | 2.65ms | 5.85ms | 2.2x |
| `heads` expect-hit | 2.16ms | 4.61ms | 2.1x |
| `disasm` 200 | 9.03ms | 5.25ms | **0.6x** |
| `decompile` cold | 162.6ms | 30.7ms | **0.2x** |
| `decomp_map` | 45.8ms | 47.9ms | 1.0x |
| empty round trip | ~0.07ms | **2.0ms** | the floor |

Two things dominated and are fixed (see `idatui/codemode_client.py:_script`):
`to_jsonable` walking every returned object (snippets now return one
pre-serialised JSON string), and `sys.settrace` — the runtime installs a trace
that returns itself, i.e. LINE tracing in every frame, which made
`ida_bytes.get_flags` 52x slower than native. The snippet detaches it and
restores it in a finally; `IDATUI_CODEMODE_TRACE=1` keeps the stock behaviour.

**What is left is the 2ms round-trip floor, and it is NOT ours.** Measured against
the same worker, same connection:

| | cost | whose |
|---|---|---|
| `GET /health` (no execute_sync) | 0.165ms | HTTP transport |
| `execute_python("result = 1")` | 2.025ms | + `ida_kernwin.execute_sync` |

So HTTP is 7% of the floor and marshalling an operation onto IDA's main thread is
92%. The worker runs IDA's own `kernwin.serve()` dispatch loop, so that latency is
inside IDA, not something Code Mode exposes a knob for.

**Do not re-chase this by batching operations.** The call volume is already
minimal, measured on `targets/bash`:

| flow | wall | calls |
|---|---|---|
| open a listing | 3.2ms | 1 |
| scroll 2000 rows | 87.9ms | 8 |
| rename + re-render those rows | 36.2ms | 4 |
| graph of a **1060-block** function | 171.0ms | 4 |
| decompile a 17785-byte function | 10806ms | 1 |

Block-coalescing, the page digest and the prefetch caches already collapse the
bursts, so a batch endpoint would save single-digit milliseconds on flows that
cost hundreds. And the big number is pure Hex-Rays: that same decompile is
10723ms on the worker backend (0.8% apart) — there is no transport in it at all.


_Superseded note:_ **What is left is the 2ms round-trip floor.** A trivial op (`data_type`,
`force_recompile`, one xref query) is ~2.5ms wall clock and looks like 40x
against an in-process worker. That is fixed by making FEWER calls, not faster
ones — which is what the digest/`expect` path does for the listing.

Benchmark harness: `/tmp/cmport/bench.py` + `compare.py` (backend-agnostic; it
picks whichever client the checked-out tree has, so it runs on master too).

---

## Known-flaky

- _(fixed 2026-08-08)_ `test_scenarios.py --only listing_view` used to fail about one
  run in three: `listing shows data heads (not just code)` reported
  `{'label','sep','funchdr','code'}`, i.e. the model was read before its pages had
  materialised. The scenario waited on `c.lst.total > 0`, which is computed from the
  segment size and is true **before any row exists** — the textbook "wait on a signal
  that is set too early". It now waits for a materialised data row. Nothing else is
  known-flaky; if something starts flaking, check the timer/frame table above first.

---

## Slow traps

- **Never run the pilot suite against `targets/echo` directly.** It edits the database.
  The suite already stages a scratch copy (`tests/_fixtures.py: staged()`), seeded from
  `<binary>.pristine.i64` which nothing writes back to. Pass a **copy** in `/tmp` when
  invoking by hand, or you are testing your own history.
- **`tests/run.py` runs suites serially on purpose.** Running them 4-up took the suite
  from 153s to 296s and got three killed mid-analysis (idalib contends hard). Do not
  "optimise" it with parallelism.
- **Cold analysis dominates a first run.** `.pristine.i64` turns ~30s of auto-analysis
  into a file copy; if it is missing or older than the binary it is rebuilt. A suite that
  suddenly takes minutes longer is usually rebuilding that cache.
- **Under load, idalib gets SIGKILLed mid-analysis** and it looks like a hang or empty
  output. Check `uptime` before believing a failure.
- **`experiments/*.py` need `PYTHONPATH=$PWD`** under `~/ida-venv/bin/python` (they are
  scripts, not a package entry point).

## Comparing against master (backend A/B)

The port replaces the whole backend, so "is this a regression?" means running the same
scenario on both. `git stash -u; git checkout master; <run>; git checkout -; git stash pop`
works, but **use a different scratch binary per branch** (`/tmp/x_master`, `/tmp/x_port`)
so the two runs never share a `.pristine.i64` or a live Code Mode instance.