autoresearch: final playbook update - budget, floors, dead ends

2 files changed, 36 insertions, 8 deletions

diff --git a/.auto/log.jsonl b/.auto/log.jsonl
index cf6fadd..a392e8f 100644
--- a/.auto/log.jsonl
+++ b/.auto/log.jsonl
@@ -28,3 +28,4 @@
 {"run":23,"commit":"98b3b98","metric":17944.4,"metrics":{"lg_boot_ms":703.7,"lg_decomp_ms":2453.7,"lg_graph_ms":1000,"lg_hex_ms":478,"lg_index_ms":96,"lg_listing_cold_ms":453,"lg_listing_warm_ms":411.2,"lg_nav_ms":6555.9,"lg_palette_ms":4.9,"lg_render_ms":218.4,"lg_search_ms":1308.8,"pure_graph_ms":212.3,"sm_boot_ms":432.4,"sm_decomp_ms":1267.8,"sm_graph_ms":742.7,"sm_hex_ms":440,"sm_index_ms":2.5,"sm_listing_cold_ms":264.3,"sm_listing_warm_ms":289.5,"sm_nav_ms":305.4,"sm_palette_ms":0.3,"sm_render_ms":258.6,"sm_search_ms":45,"fails":0},"status":"keep","description":"Three targeted cuts: the graph's transposition pass counts keep and swap in one pass over the neighbour pairs (was four _pair_cross calls); the barycentre median answers degree 1 and 2 without sorting; and the search body is built from windowed model reads instead of one locked row lookup per line.","timestamp":1786066135637,"segment":4,"confidence":29.46391752577091,"asi":{"hypothesis":"clear the last measurable constants in the layout engine and the search body build","gains":"total 18516 -> 17944 (-3.1%); lg_search 1475 -> 1309; pure_graph 242 -> 212; corpus layout 236 -> 204ms standalone","layout_change":"_swap_delta returns (keep, swap) from one pass over the neighbour pairs. _pair_cross was called four times per candidate swap -- 353k calls over the corpus -- and each pair was compared twice, once per direction. median() answers |neighbours| of 1 or 2 arithmetically; sorted() was called 67k times, almost always on a list of one or two.","layout_equivalence":"node geometry (id, x, y, w, h) is IDENTICAL for all 128 corpus functions; tests/test_graph.py passes 470 checks including the no-edge-inside-a-box invariant.","PRE_EXISTING_BUG_FOUND":"idatui/graph.py edge ROUTING is non-deterministic: running the UNCHANGED engine twice on the same input gives different painting.vruns for 71 of 128 functions. Node placement is stable; only the routing moves. That is why a naive old-vs-new painting diff is useless here -- old-vs-old fails it too. Logged in .auto/ideas.md; worth fixing on its own merits (a graph should not redraw differently when you reopen it).","search_change":"SearchMixin gained a _search_line_texts(start, count) hook; ListingView serves it from model.window(), so building the joined body takes the model lock and bisects its row table once per 4096-row chunk instead of once per row.","verified":".auto/check_search.py 140 prefixes 0 mismatches; 301 scenarios pass","next_action_hint":"budget: lg_nav 6556 (37%), decomp 3722, graph 1743, search 1354, listing 1418, boot 1136, hex 918, render 477, pure_graph 212. Everything except nav and Hex-Rays is now within ~2x of the Textual compositor's own per-frame cost."}}
 {"run":24,"commit":"8218b91","metric":18608,"metrics":{"lg_boot_ms":708.5,"lg_decomp_ms":2454.9,"lg_graph_ms":1034.5,"lg_hex_ms":431.9,"lg_index_ms":95.1,"lg_listing_cold_ms":530.2,"lg_listing_warm_ms":413.4,"lg_nav_ms":7057.9,"lg_palette_ms":5,"lg_render_ms":214.2,"lg_search_ms":1408.9,"pure_graph_ms":213.3,"sm_boot_ms":433.6,"sm_decomp_ms":1292,"sm_graph_ms":754,"sm_hex_ms":433.4,"sm_index_ms":2.6,"sm_listing_cold_ms":260.4,"sm_listing_warm_ms":280.5,"sm_nav_ms":286.1,"sm_palette_ms":0.3,"sm_render_ms":251,"sm_search_ms":46.5,"fails":0},"status":"keep","description":"CORRECTNESS REPAIR, kept on its merits. The full suite (which the gate was NOT running) revealed that the worker-connect poll change made test_project_ui flaky: 5ms polling on a background thread through a cold auto-analysis starved the UI thread enough that the loading overlay was still up when the test pressed Ctrl+O. Poll now backs off to a 25ms cap (keeps the boot win, no busy-wait), the racy boot wait is fixed, and checks.sh runs tests/run.py in full (830 checks) instead of just the scenario suite.","timestamp":1786068002977,"segment":4,"confidence":12.426086956521708,"asi":{"honesty_note":"total_ms 17944 -> 18608 is WORSE, and I am keeping it anyway. The whole delta is lg_nav (6556 -> 7058), a phase nothing here touches and whose characterised spread is +/-500ms; boot, the phase this actually affects, is unchanged (sm_boot 432 -> 434, lg_boot 704 -> 709). Discarding would auto-revert a regression repair and a gate fix, which is the wrong trade whatever the number says. Re-running next to confirm the drift.","the_regression":"tests/test_project_ui.py went from 4/4 passing on the pre-autoresearch code to 1/3 on the branch. Bisected to idatui/worker_client.py: reverting it alone gave 3/3, reverting idatui/app.py alone gave 1/3.","root_cause":"experiment #13 held the connect poll at 5ms for the first 5 seconds. That poll runs on a background thread while the UI thread draws; 200 wakeups a second through a cold auto-analysis (targets/cat has no prebuilt .i64) cost enough GIL time to delay the app's own startup. The loading overlay is a ModalScreen and was still up when the test pressed Ctrl+O, so the key was swallowed and the switcher never opened.","fix":"cap the geometric backoff at 25ms instead. Bounded overshoot on a fast open (the 351ms -> 273ms boot win survives) and 40 probes/sec during a real analysis, which is nothing.","test_repair_and_its_three_proofs":["bisect: worker_client.py flips it, app.py does not","race reproduced outside the suite: the same steps print screen_stack ['Screen','LoadingScreen'] on the fast backend and ['Screen'] on the slow one","repaired test passes 5/5 on the fast backend and 3/3 on the pre-autoresearch backend"],"test_edit_scope":"the boot wait now also requires the loading overlay to be gone; two later waits require an empty screen stack. No assertion changed. Same class of fix as graph_minimap in v1 #6.","GATE_GAP_CLOSED":"checks.sh ran only tests/run.py --fast + the scenario suite. test_project_ui, blob_ui, thumb_ui, trace_ui, trace_rpc, rawimage_rpc were never run -- 830 checks now run, not 645. Verified the failure path end to end by injecting a deliberate failing check: the gate names the file, re-runs it alone, and exits 1.","lesson":"'poll faster' is not free in a GIL runtime when the poller shares a process with a UI thread. And a gate that runs a subset of the suite will eventually let something through -- it took 13 experiments here."}}
 {"run":25,"commit":"8218b91","metric":18746.2,"metrics":{"lg_boot_ms":713.2,"lg_decomp_ms":2575.4,"lg_graph_ms":1010.9,"lg_hex_ms":457.3,"lg_index_ms":97.3,"lg_listing_cold_ms":561.7,"lg_listing_warm_ms":411.8,"lg_nav_ms":6936.8,"lg_palette_ms":4.7,"lg_render_ms":225.9,"lg_search_ms":1442.5,"pure_graph_ms":212.9,"sm_boot_ms":433.3,"sm_decomp_ms":1304.2,"sm_graph_ms":710.8,"sm_hex_ms":456.7,"sm_index_ms":2.3,"sm_listing_cold_ms":263.9,"sm_listing_warm_ms":285.7,"sm_nav_ms":334.7,"sm_palette_ms":0.3,"sm_render_ms":257.2,"sm_search_ms":46.5,"fails":0},"status":"discard","description":"Confirmation re-run of #24 with no code change. 18608 -> 18746, i.e. the elevated lg_nav is not drift within a run — the BOX got busier (load average 0.62 at session start, 1.55 now). The direct paging microbenchmark is unchanged at 24.4 us/row, so the code is the same speed; the wall clock is not.","timestamp":1786068203628,"segment":4,"confidence":7.8732782369147385,"asi":{"hypothesis":"is the +500ms on lg_nav since #23 a code cost or the environment?","answer":"environment. /tmp/domain_break.py (a direct cold walk of bash's .text through the worker, no UI) reads 20.31us/row in the heads call + 4.04us/row client-side -- identical to the reading taken before the correctness fix. Meanwhile the box's load average went 0.62 -> 1.55 during the session.","consequence":"totals measured from here on are NOT comparable with those from the first half of the session. lg_nav is 224k sequential worker round trips and is the phase most exposed to a co-tenant; the idatui skill already warns that idalib is reap-prone under load. Compare against a same-session baseline, or use the phase-level microbenchmarks (/tmp/domain_break.py, /tmp/spanbench2.py, .auto/diff_spans.py) which are far less exposed.","no_code_change":"nothing to revert; logged as discard because the metric did not improve","next_action_hint":"if the loop continues on a loaded box, prefer changes whose effect is verifiable in a microbenchmark rather than in total_ms"}}
+{"run":26,"commit":"7683ca3","metric":17829.9,"metrics":{"lg_boot_ms":724.4,"lg_decomp_ms":2353.5,"lg_graph_ms":1041,"lg_hex_ms":434.9,"lg_index_ms":95.5,"lg_listing_cold_ms":545.6,"lg_listing_warm_ms":407.9,"lg_nav_ms":6922,"lg_palette_ms":4.8,"lg_render_ms":218.1,"lg_search_ms":884.3,"pure_graph_ms":213.6,"sm_boot_ms":432,"sm_decomp_ms":1250.2,"sm_graph_ms":687.4,"sm_hex_ms":457.5,"sm_index_ms":2.6,"sm_listing_cold_ms":265.4,"sm_listing_warm_ms":288.8,"sm_nav_ms":303.2,"sm_palette_ms":0.3,"sm_render_ms":253.9,"sm_search_ms":43,"fails":0},"status":"keep","description":"Highlight ranges are computed per line on demand instead of for every match. Searching one character over bash matches 177k lines at 310k places, and all but the forty on screen were built and thrown away. _MatchRanges keeps the line SET eagerly and works out the offsets when a line is painted or the cursor lands on it; the blob scan now also skips to the next line after a hit.","timestamp":1786068588431,"segment":4,"confidence":4.5317040951122864,"asi":{"hypothesis":"profiling the search phase directly showed the per-match range building, not the scanning, was the cost","microbench_before_after_ms_per_term":{"note":"targets/bash, 228659 rows, _compute_matches only","m":[139.7,41.6],"mo":[153.9,49.2],"mov":[55.4,47],"c":[329.6,66.1],"ca":[51.5,47.3],"cal":[32.9,30],"call":[29.2,26.8],"haystack_build":[458,459]},"equivalence":"the microbenchmark materialises every range afterwards and prints the totals: 116736 / 76271 / 76070 / 309827 / 25222 / 16312 / 15722 -- identical to the eager version, term for term. Plus .auto/check_search.py (140 prefixes, listing + pseudocode, 0 mismatches) and the full 830-check suite.","api_shape":"_MatchRanges quacks as the dict it replaced for the subset anything uses (in / get / [] / items / len). tests/test_scenarios.py reads dis._ranges.get(cursor) and dis._ranges[cursor][0][0] directly, so that had to keep working -- it does, untouched.","gains":"total 18608 -> 17830 on a box that has got busier since the earlier runs; lg_search 1409 -> 884 (-37%), sm_search 46 -> 43","whats_left_in_search":"the haystack build, 459ms: one _line_plain pass over 228k rows plus a 12.3MB join and lower(). That is the floor unless the plain text is cached with the rows.","next_action_hint":"budget: lg_nav 6922 (39%, and inflated by machine load), decomp 3604, graph 1728, search 927, listing 1508, boot 1156, hex 892, render 472, pure_graph 214."}}
diff --git a/.auto/prompt.md b/.auto/prompt.md
index 4c3bbef..e41e95d 100644
--- a/.auto/prompt.md
+++ b/.auto/prompt.md
@@ -128,10 +128,15 @@ v1 bench baseline `total_ms` ≈ 46 600 → 18 900 after thirteen experiments
    196 k listing lines of bash are only 53 k distinct, 26.7 → 16.5 µs/row.
 4. *(v1 #7, −9%)* **Incremental search narrows instead of rescanning** — typing
    a character can only remove lines.
-5. *(v1 #9/#10/#13)* Constants: `bytes.hex(" ")` for the opcode column (12×),
-   `bisect` imported at module scope, the deferred `refresh(layout=True)` only
-   when a scroll actually clamped, a memoised pygments token→style lookup, and
-   a worker-connect poll that doesn't back off before the worker can be ready.
+5. *(v5 #6)* **Highlight ranges are computed per line on demand.** Searching
+   one character over bash matches 177 k lines at 310 k places; all but the
+   forty on screen were built and thrown away.
+6. *(v1 #9/#10/#13, v2 #3, v5 #3)* Constants: `bytes.hex(" ")` for the opcode
+   column (12×), `bisect` imported at module scope, the deferred
+   `refresh(layout=True)` only when a scroll actually clamped, a memoised
+   pygments token→style lookup, `_CellRow` writing by slice, `HexView` emitting
+   style runs instead of a Segment per byte, `Head` as a `NamedTuple`, and the
+   graph's transposition counting keep and swap in one pass.
 
 **Dead ends / things not to re-try**
 
@@ -147,12 +152,34 @@ v1 bench baseline `total_ms` ≈ 46 600 → 18 900 after thirteen experiments
   model-scoped one is unsafe (`id()` is reused once the page dicts die).
 - Growing the line cache past 16 384 does nothing for a *cold* sweep (16.5
   µs/row at both 16 k and 64 k); it only helps a second sweep (10.6 vs 16.4).
+- Merging same-kind adjacent spans in `_idatui_spans`: only 2.5% fewer spans on
+  20 k real lines. Not worth a wire-format change.
+- Applying an already-decompiled function inline instead of via a `@work`
+  thread: a Textual thread spawn plus its `call_from_thread` is worth well under
+  1% of an F5. Measured flat; reverted for complexity.
+- Deferring the "decompiling…" loading cover until ~120 ms (worth ~9 ms per F5)
+  is blocked: a scenario asserts F5 raises it **synchronously**, guarding a real
+  past regression. That is an assertion, not setup, so it stands.
+- Polling the worker socket faster than ~25 ms is actively harmful: the poll
+  runs on a background thread and starves the UI thread through a cold
+  auto-analysis (see v5 #4).
 
-**Where the time is now (17 944 ms on the v5 bench)**
+**Where the time is now (17 830 ms on the v5 bench)**
 
-`lg_nav` 6 556 (37%) · decomp lg+sm 3 722 · graph lg+sm 1 743 · search lg+sm
-1 354 · listing lg+sm 1 418 · boot lg+sm 1 136 · hex lg+sm 918 · render 477 ·
-pure_graph 212.
+`lg_nav` 6 922 (39%) · decomp lg+sm 3 604 · graph lg+sm 1 728 · listing lg+sm
+1 508 · boot lg+sm 1 156 · search lg+sm 927 · hex lg+sm 892 · render 472 ·
+pure_graph 214.
+
+**Four of those are at a floor that is not ours to move:**
+
+- `nav` — `generate_disasm_line` is 5.9 µs of the ~16.5 µs/row the worker
+  spends, and the walk is inherently linear.
+- `decomp` — the raw `decompile` tool is 1 711 ms cold for echo's twelve largest
+  functions and 218 ms warm; pickling the result is 0.1 ms. It is Hex-Rays.
+- `search` — what remains is the 459 ms haystack build, one `_line_plain` pass
+  over 228 k rows.
+- `listing`/`hex`/`graph`/`render` — mostly Textual's own compositing, ~6 ms per
+  full-screen frame. Our `render_line` is ~1.8 ms of a ~10 ms hex frame.
 
 `lg_nav` is a linear walk of the segment at its floor: ~16.5 µs/row in the
 worker (of which `generate_disasm_line` is 5.9 µs and IDA's) plus ~7 µs/row on