Re-apply #5 (lru_cache on the per-line render + Heads built with their opcode bytes already attached) with the graph_minimap scenario's racy SETUP made deterministic: clear _graph_sticky before the second navigation so Space is known to be entering the graph, not leaving it. No assertion changed.

Result: {"status":"keep","total_ms":22980.2,"lg_boot_ms":738.2,"lg_decomp_ms":2401.8,"lg_graph_ms":944.1,"lg_hex_ms":920.6,"lg_index_ms":75.2,"lg_listing_cold_ms":538.5,"lg_listing_warm_ms":411.1,"lg_nav_ms":6813.9,"lg_palette_ms":4.9,"lg_render_ms":221.8,"lg_search_ms":5630.1,"pure_graph_ms":240.7,"sm_boot_ms":537.5,"sm_decomp_ms":595.1,"sm_graph_ms":715.7,"sm_hex_ms":858.8,"sm_index_ms":0,"sm_listing_cold_ms":263.3,"sm_listing_warm_ms":265.3,"sm_nav_ms":335.2,"sm_palette_ms":0.3,"sm_render_ms":271.4,"sm_search_ms":196.5,"fails":0}

1 files changed, 2 insertions, 0 deletions

diff --git a/.auto/log.jsonl b/.auto/log.jsonl
index 55322e0..58754b0 100644
--- a/.auto/log.jsonl
+++ b/.auto/log.jsonl
@@ -2,3 +2,5 @@
 {"run":1,"commit":"2910c93","metric":46572.1,"metrics":{"lg_boot_ms":863.5,"lg_decomp_ms":2881.3,"lg_graph_ms":951.7,"lg_hex_ms":900.9,"lg_index_ms":219.3,"lg_listing_cold_ms":434.4,"lg_listing_warm_ms":552.3,"lg_nav_ms":29106.9,"lg_palette_ms":4.8,"lg_render_ms":231.2,"lg_search_ms":5590.1,"pure_graph_ms":236.2,"sm_boot_ms":539.4,"sm_decomp_ms":621.4,"sm_graph_ms":702,"sm_hex_ms":850.5,"sm_index_ms":0,"sm_listing_cold_ms":270.8,"sm_listing_warm_ms":270.6,"sm_nav_ms":875.1,"sm_palette_ms":0.3,"sm_render_ms":272.3,"sm_search_ms":197.1,"fails":0},"status":"checks_failed","description":"Baseline run of the new bench harness. Benchmark clean (fails=0) but the pilot scenario suite reported 300 passed / 1 failed with ZERO code changes -> flaky, and checks.sh printed the tail instead of the FAIL line so the name is unknown.","timestamp":1786058161497,"segment":0,"confidence":null,"asi":{"hypothesis":"establish a baseline for total_ms","bottleneck":"lg_nav_ms=29107 is 62% of total_ms; lg_nav_worst_ms=28274 is ONE cold jump to a high address in bash. ListingModel.ensure_ea walks the segment forward in 500-head pages from seg_start, so landing near the end of a 224k-row listing costs ~450 sequential worker round trips.","second_bottleneck":"lg_search_ms=5590 (91783 hits over the whole segment)","cheap_phases":"palette/index/render/pure_graph are all <600ms; not where the time is","rollback_reason":"checks.sh flagged 1 scenario failure with no code change (flake)","next_action_hint":"make checks.sh print the FAIL line name on non-zero exit, re-run baseline, then attack ListingModel address->row lookup (needs a backend primitive in server/patch_server.py: heads walking anchored at an address, or a segment head-index built in one call)"}}
 {"run":2,"commit":"a3f3400","metric":46685.5,"metrics":{"lg_boot_ms":885.7,"lg_decomp_ms":2871.8,"lg_graph_ms":930.9,"lg_hex_ms":947.3,"lg_index_ms":207.8,"lg_listing_cold_ms":564.3,"lg_listing_warm_ms":454.7,"lg_nav_ms":29018.5,"lg_palette_ms":4.6,"lg_render_ms":223.6,"lg_search_ms":5448.7,"pure_graph_ms":535.4,"sm_boot_ms":537.9,"sm_decomp_ms":643.8,"sm_graph_ms":680.4,"sm_hex_ms":854.9,"sm_index_ms":0,"sm_listing_cold_ms":265.5,"sm_listing_warm_ms":263.7,"sm_nav_ms":887.2,"sm_palette_ms":0.3,"sm_render_ms":264.1,"sm_search_ms":194.5,"fails":0},"status":"discard","description":"Baseline re-run with the fixed checks gate. Checks pass; total_ms reproduces to within 0.24% of run #1 (46572 -> 46686), so the noise floor is ~115ms on a 46.6s metric.","timestamp":1786058370297,"segment":0,"confidence":null,"asi":{"hypothesis":"confirm the baseline is reproducible and the checks gate is green","noise_floor_ms":115,"reproducibility":"run1 46572 / run2 46686 -> 0.24% spread; pure_graph_ms is the jumpiest single phase (236 -> 535, it is CPU-only and gets descheduled)","checks":"flaky-scenario retry logic works; suite green on a clean tree","next_action_hint":"attack ListingModel.ensure_ea / the heads tool: 29s of 46.6s is one cold address->row walk over bash"}}
 {"run":3,"commit":"93240e2","metric":26923.9,"metrics":{"lg_boot_ms":762.2,"lg_decomp_ms":2631.7,"lg_graph_ms":941.8,"lg_hex_ms":1052,"lg_index_ms":67.2,"lg_listing_cold_ms":440.6,"lg_listing_warm_ms":530.5,"lg_nav_ms":10598.3,"lg_palette_ms":4.6,"lg_render_ms":227.8,"lg_search_ms":5265.5,"pure_graph_ms":237.9,"sm_boot_ms":535.3,"sm_decomp_ms":631.8,"sm_graph_ms":702.1,"sm_hex_ms":841.1,"sm_index_ms":0,"sm_listing_cold_ms":268.2,"sm_listing_warm_ms":269.4,"sm_nav_ms":443.6,"sm_palette_ms":0.3,"sm_render_ms":277.8,"sm_search_ms":194.4,"fails":0},"status":"keep","description":"Stop ida-pro-mcp installing a sys.setprofile hook around every tool call. Its deadline mechanism profiles every python call/return so a pure-python tool loop can be interrupted; our tools are call-heavy, so it taxed the whole backend 3.3x. Worker now sets IDA_MCP_TOOL_TIMEOUT_SEC=0 and arms the deadline itself with one polling watchdog thread + ida_kernwin.set_cancelled() (the half that actually frees the IDA main thread). Also rewrote _idatui_spans to jump between colour tags instead of walking characters (byte-identical over 258k real lines).","timestamp":1786059117542,"segment":0,"confidence":173.264550264548,"asi":{"hypothesis":"the heads tool is not IDA-bound; the ida-pro-mcp sync wrapper's sys.setprofile deadline is the tax","evidence":"in-process A/B on targets/bash: heads(count=500,annotate) 92.2us/row with IDA_MCP_TOOL_TIMEOUT_SEC unset vs 27.5us/row with it 0; domain ListingModel._load_next_page 116 -> 37 us/row","where_it_lives":"site-packages/ida_pro_mcp/ida_mcp/sync.py sync_wrapper() installs profilefunc via sys.setprofile whenever timeout>0; default _DEFAULT_TOOL_TIMEOUT_SEC=60","what_we_kept":"the native half: threading watchdog + ida_kernwin.set_cancelled(), which is what decompile/auto_wait/find_bytes actually poll. Only the interruption of a pure-python loop inside a tool body is gone, and every idatui tool loop is bounded by its count argument (<=2000).","watchdog_design":"polling at 0.25s instead of signalling, because arm/disarm is on the path of EVERY worker call (a scroll is dozens); arming is two attribute writes + clr_cancelled()","spans_rewrite":"_idatui_spans now re.finditer's over the 3 control chars instead of walking characters. Verified byte-identical vs HEAD over 258k real disasm lines across echo/ls_ttl/bash/libcrypto (.auto/diff_spans.py). Standalone it measured 12.5 -> 13.6 us/line i.e. slightly SLOWER, so it is probably a small net loss that the setprofile win is hiding -- A/B it on its own next.","gains":"total 46572 -> 26924 (-42%); lg_nav 29107 -> 10598; lg_nav_worst 28274 -> 10170; sm_nav_worst 656 -> 224; lg_index 219 -> 67","work_unchanged":"nav_rows/decomp_ok/graph_ok/search_hits/render_cells all identical -> the speedup is not less work","next_action_hint":"1) A/B the _idatui_spans rewrite alone (suspect regression). 2) lg_nav is still 10.6s: ListingModel.ensure_ea is O(distance) at ~34us/row; a skeleton (ea,size,kind only) walk or a bigger PAGE would cut it further. 3) lg_search_ms 5265 is now the #2 term."}}
+{"run":4,"commit":"b6a95ec","metric":26050.2,"metrics":{"lg_boot_ms":727.4,"lg_decomp_ms":2618.1,"lg_graph_ms":935.1,"lg_hex_ms":954.2,"lg_index_ms":67.4,"lg_listing_cold_ms":459.3,"lg_listing_warm_ms":545.7,"lg_nav_ms":9257.8,"lg_palette_ms":4.7,"lg_render_ms":238.8,"lg_search_ms":5472.5,"pure_graph_ms":530.7,"sm_boot_ms":534.6,"sm_decomp_ms":635.8,"sm_graph_ms":748.1,"sm_hex_ms":858.7,"sm_index_ms":0,"sm_listing_cold_ms":268.7,"sm_listing_warm_ms":269.2,"sm_nav_ms":441.6,"sm_palette_ms":0.3,"sm_render_ms":281.7,"sm_search_ms":200,"fails":0},"status":"keep","description":"_idatui_spans: one capturing re.split over the tag pairs instead of finditer+char-slicing, and collapse whitespace with ' '.join(txt.split()) instead of a regex sub. 13.15 -> 11.07 us/line (the previous finditer attempt was 14.4, i.e. SLOWER than the original char loop it replaced).","timestamp":1786059375617,"segment":0,"confidence":2.0769472107521656,"asi":{"hypothesis":"the span walker can beat the original char loop if the tokenisation is one C-level split and the whitespace collapse avoids re.sub","microbench_us_per_line":{"original_char_loop":13.15,"finditer_attempt":14.42,"re.split_version":11.07},"lesson":"re.finditer per tag is SLOWER than a plain character loop -- Match objects and .start() calls cost more than the ~54 trivial loop iterations they replace. A single capturing re.split that hands back [text, tag, text, ...] is what actually wins.","lesson2":"re.sub for whitespace collapse cost ~1us per call at ~6.5 calls/line; ' '.join(txt.split()) splits on exactly str.isspace() and is far cheaper. Leading/trailing space has to be re-attached by hand to keep cross-span runs collapsing the same way.","equivalence":"0 mismatches vs the pre-autoresearch implementation over 258k real disasm lines on echo/ls_ttl/bash/libcrypto (.auto/diff_spans.py --ref 2b0ae8d)","gains":"total 26924 -> 26050 (-3.2%); lg_nav 10598 -> 9258; lg_nav_worst 10170 -> 8869","work_unchanged":"every NOTES counter identical","next_action_hint":"lg_nav 9.3s and lg_search 5.5s are now the top two. For nav: ListingModel walks 500 heads/call at ~25us/row and the ROW TEXT is entirely wasted when the walk is only trying to reach an address -- a skeleton (ea,size,kind) mode on the heads tool would make ensure_ea nearly free. For search: _compute_matches/_line_plain over 224k rows."}}
+{"run":5,"commit":"b6a95ec","metric":23259.6,"metrics":{"lg_boot_ms":752.7,"lg_decomp_ms":2475.3,"lg_graph_ms":963,"lg_hex_ms":950.4,"lg_index_ms":70.8,"lg_listing_cold_ms":533.2,"lg_listing_warm_ms":406.1,"lg_nav_ms":6862.6,"lg_palette_ms":4.7,"lg_render_ms":228.9,"lg_search_ms":5401.5,"pure_graph_ms":510.5,"sm_boot_ms":538.8,"sm_decomp_ms":666.7,"sm_graph_ms":715.7,"sm_hex_ms":856.8,"sm_index_ms":0,"sm_listing_cold_ms":260.3,"sm_listing_warm_ms":262.5,"sm_nav_ms":335.5,"sm_palette_ms":0.3,"sm_render_ms":270.8,"sm_search_ms":192.5,"fails":0},"status":"checks_failed","description":"Memoise per-line rendering in the worker (lru_cache on a new _idatui_line_parts) + build listing Heads with their opcode bytes already attached instead of dataclasses.replace-ing them in. total 26050 -> 23260, lg_nav 9258 -> 6863. Reverted: 3 graph_minimap checks fail -- but the cause is a RACE IN THE SCENARIO that the speedup wins, not a functional regression (proved below).","timestamp":1786060327739,"segment":0,"confidence":5.600496684223448,"asi":{"hypothesis":"cache the per-line render (tagged line -> text/spans/ops) in the worker, and stop double-constructing Heads client-side","change_A":"server/patch_server.py: new _idatui_line_parts(line) = (text, spans, ops), functools.lru_cache(16384). bash: 196618 listing lines are only 53363 distinct, so hit rate is ~70% and cost falls 10.4 -> 3.9 us/line. Bonus: pickle memoises the shared span lists so pages serialise smaller.","change_B":"idatui/domain.py: ListingModel._build_page reads the code extent FIRST and passes raw into Head.from_raw, replacing _attach_opcode_bytes' dataclasses.replace (which re-ran __init__ per code head). from_raw now uses tuple(map(tuple,...)) instead of a coercing genexpr.","measured":"cold ListingModel paging 35.7 -> 25.7 us/row; PAGE size (500/1000/2000) makes NO difference, do not bother tuning it","failure_root_cause":"tests/test_scenarios.py graph_minimap. _open_graph() leaves _graph_sticky=True; the scenario then does c.open(big,'listing') and presses space expecting to ENTER the graph. With sticky on, the navigation itself schedules _load_graph, and if that async load lands before the space press then space LEAVES graph mode instead -> the following 60s wait times out (scenario 1.9s -> 65.5s) and every minimap click lands on a hidden widget.","proof":"/tmp/mmrace.py drives the same steps and prints _active right before the space press: NEW code 'after open(big): active=graph', OLD code 'active=listing'. Bisected: stashing idatui/domain.py alone still fails, stashing server/patch_server.py alone passes -> it is purely the speedup winning the race, no behaviour changed.","equivalence_evidence":"diff_spans.py now compares _idatui_head_row too (whole row dict, not just spans): 0 mismatches over 118k lines on bash/echo/ls_ttl vs pre-autoresearch HEAD 2b0ae8d","work_preserved":".auto/wip-headcache.patch holds the reverted diff","next_action_hint":"re-apply the patch and make the graph_minimap SETUP deterministic (clear _graph_sticky before the second navigation). Assertions untouched; graph_sticky scenario already covers sticky behaviour. Record the amended tests/ rule in .auto/prompt.md."}}