autoresearch: record the streaming-responsiveness finding and a pre-existing palette crash

2 files changed, 36 insertions, 0 deletions

diff --git a/.auto/ideas.md b/.auto/ideas.md
index a2d9b4f..d1d2f92 100644
--- a/.auto/ideas.md
+++ b/.auto/ideas.md
@@ -68,3 +68,38 @@
   layer. Use the `/tmp/featprobe.py` shape: call the domain API for each and look
   for a number that is absurd for the work done. That is how the flowchart hull,
   `decomp_map` and the rename walk were all found.
+
+## The UI is much slower while a big segment streams (measured, not fixed)
+
+Opening a big binary leaves `ListingView._grow` streaming the segment in the
+background for ~7s. During that window every UI action is several times slower:
+an xrefs dialog measured **691ms while streaming against 105ms after** on bash.
+Two mechanisms, both confirmed:
+
+* **`@work(exclusive=True)` does not stop a thread worker that is already
+  running**, and navigating inside the same segment re-primes against the SAME
+  model — so every jump leaves another streamer behind, all queueing on the
+  model's load lock. Five jumps into bash meant five streamers.
+* Each streamer reports growth to the UI every four pages; a report is a thread
+  hop, a `virtual_size` change and a full repaint.
+
+Two fixes were tried and **both measured worse**, so neither was kept:
+a time-based report throttle (10/s) made it worse because the count is per
+*streamer* and there are several; adding a token so only the newest streamer
+survives did not reduce the report count either, which means the retirement is
+not happening where it looks like it should — worth understanding before trying
+again. Probe: `/tmp/streamresp.py` (navigate + press `x` repeatedly while
+`lst.model.complete` is still False, counting `_grew` calls).
+
+Caveat: these numbers were taken on a loaded box and the probe re-primes the
+view on every iteration, which is itself what spawns the extra streamers. Build
+a cleaner probe first.
+
+## Pre-existing crash, unrelated to performance
+
+`StringsPalette.on_mount` calls `self._apply("")`, which does
+`query_one(OptionList)` before `compose`'s children are mounted:
+`NoMatches: No nodes match 'OptionList' on StringsPalette()`. Reproduces 3/3 on
+`targets/bash` with `/tmp/strcrash.py`, **and 3/3 on the pre-autoresearch commit
+2b0ae8d** — so it is not something this work introduced. `ProjectPalette` has the
+same shape and the same latent race.
diff --git a/.auto/log.jsonl b/.auto/log.jsonl
index fcebe68..ed0514f 100644
--- a/.auto/log.jsonl
+++ b/.auto/log.jsonl
@@ -39,3 +39,4 @@
 {"run":32,"commit":"44c311a","metric":25563.7,"metrics":{"lg_boot_ms":732.6,"lg_decomp_ms":2714,"lg_graph_ms":919.3,"lg_hex_ms":566.8,"lg_index_ms":71.4,"lg_listing_cold_ms":441.4,"lg_listing_warm_ms":411.7,"lg_nav_ms":6708,"lg_palette_ms":4.8,"lg_rename_ms":741.5,"lg_render_ms":227.7,"lg_search_ms":3314.1,"lg_split_ms":2596.9,"pure_graph_ms":213.6,"sm_boot_ms":422.9,"sm_decomp_ms":1284,"sm_graph_ms":789.5,"sm_hex_ms":468.1,"sm_index_ms":2.6,"sm_listing_cold_ms":258.7,"sm_listing_warm_ms":257.3,"sm_nav_ms":306.5,"sm_palette_ms":0.3,"sm_rename_ms":388.2,"sm_render_ms":256,"sm_search_ms":105.1,"sm_split_ms":1360.5,"fails":0},"status":"keep","description":"A rename keeps the listing's walk instead of throwing it away. bump_names now marks the rendered text stale (invalidate_text) and ListingModel re-renders a 500-head block at a time on demand, snapped out to whole address groups; a refetch that comes back with a different head sequence sets stale_structure so Program.listing() rebuilds. lg_rename 10055 -> 742.","timestamp":1786073731826,"segment":6,"confidence":null,"asi":{"gains":"total 33243 -> 25564 (-23.1%); lg_rename 10055 -> 742 (-93%); sm_rename 638 -> 388; lg_split 3540 -> 2597 (the split phase runs after the renames and no longer inherits a discarded listing)","honest_tradeoff":"lg_search 775 -> 3314. The cost did not vanish, it moved: the six renames used to pay for a full rebuild each (10s), and search then found a warm model. Now renames are ~120ms and the first WHOLE-SEGMENT search afterwards pays to re-render the blocks it reads. Net -7.7s, and the cost only lands if you search the entire segment right after renaming. A full re-read is break-even with a rebuild by design -- re-rendering N heads costs what loading N heads costs.","work_unchanged":"rename_ok 6/6 both targets, search_hits 91783, split_mapped_lines 2070, graph_blocks 1000, decomp_ok 12 -- all identical to the baseline","design_notes":["blocks are snapped out to whole ADDRESS groups: a function start emits three banner rows at the same ea, so an unsnapped boundary refetches the group, never lines up, and would leave stale names forever","on a sequence mismatch the model sets stale_structure and Program.listing() rebuilds -- degrading to the old behaviour rather than showing an old name","_renamed is a boolean gate so that before the first rename every read takes exactly the path it always did, with no extra lock round trips"],"equivalence":"/tmp/renameeq.py: rename, snapshot every row (ea, kind, text, name) from the kept model, rebuild from scratch, compare. 0 mismatches over 5952 rows (echo), 28807 (ls_ttl), 228659 (bash), three renames each. Full 830-check gate green.","viewport_case":"bash with the cursor at row 220036: the listing is back in 10.3ms instead of 6320ms","next_action_hint":"lg_nav 6708 and lg_search 3314 are the top two. Search after a rename could refresh in PAGE-sized chunks driven from load_all rather than block-by-block from window()."}}
 {"run":33,"commit":"44c311a","metric":27571.1,"metrics":{"lg_boot_ms":714.2,"lg_decomp_ms":2426.5,"lg_graph_ms":914.1,"lg_hex_ms":436.7,"lg_index_ms":109.8,"lg_listing_cold_ms":772.1,"lg_listing_warm_ms":437.2,"lg_nav_ms":6876.9,"lg_palette_ms":4.7,"lg_rename_ms":732.9,"lg_render_ms":224.7,"lg_search_ms":1209,"lg_split_ms":6623.7,"pure_graph_ms":214.1,"sm_boot_ms":463.8,"sm_decomp_ms":1282.5,"sm_graph_ms":755,"sm_hex_ms":444.3,"sm_index_ms":2.3,"sm_listing_cold_ms":268.7,"sm_listing_warm_ms":269.4,"sm_nav_ms":307.9,"sm_palette_ms":0.3,"sm_rename_ms":384.3,"sm_render_ms":260.2,"sm_search_ms":67.7,"sm_split_ms":1368.2,"fails":0},"status":"discard","description":"Chunk the post-rename text refresh into TEXT_BLOCK pieces instead of one call for the whole requested range (a search window asks for thousands of rows and the heads tool caps a response at 2000, so the oversized call came back short, failed the sequence check and condemned the model to a rebuild). Fixes lg_search 3314 -> 1209, but lg_split 2597 -> 6624 and total 25564 -> 27571: doing it properly is SLOWER here than the accidental rebuild was.","timestamp":1786074100440,"segment":6,"confidence":3.8252964033077643,"asi":{"hypothesis":"one heads call for a whole search-sized window overflows the tool's 2000-row cap, so the refresh always failed its sequence check and forced a rebuild","hypothesis_confirmed":"yes -- chunking removed the spurious rebuild and lg_search fell 3314 -> 1209","but":"lg_split rose 2597 -> 6624. The accidental rebuild was CHEAPER overall than refreshing block by block, because this bench reads most of the segment after renaming and 450 block calls cost more than one linear rebuild.","UNEXPLAINED_AND_MUST_BE_RESOLVED":"work counters moved between the two runs: lg_decomp_lines 3436 -> 3233, lg_split_mapped_lines 2070 -> 1990, lg_search_hits 91783 -> 92733. Same database (targets/bash.i64 mtime unchanged, staged fresh per run), same fixed function set. Something about which path runs is changing what the app SEES. That has to be understood before any version of this is kept -- a perf change must not alter observable work.","leads":["phase_rename renames six functions and undoes them; a function whose original name was auto-generated (sub_X) comes back as a USER name sub_X. Check whether that changes the listing (a label row, or is_auto_name affecting annotate).","the search terms are 'mov' and 'call'; the temporary name _bench_<pid>_<k> contains a 'c', so leftover names would inflate a 'c'-prefixed search -- but the terms are full words, so check for residue directly.","lg_listing_rows is a streamed-progress reading, not a work counter -- ignore that one."],"work_preserved":".auto/wip-chunk.patch","next_action_hint":"resolve the counter drift first. Then, if the chunking is kept, try TEXT_BLOCK = 2000 (the tool's cap) so a wholesale refresh costs about what a rebuild costs while a viewport still needs one call."}}
 {"run":34,"commit":"3ca9e1e","metric":28651.3,"metrics":{"lg_boot_ms":720.6,"lg_decomp_ms":2427.4,"lg_graph_ms":1222.4,"lg_hex_ms":440.9,"lg_index_ms":72.6,"lg_listing_cold_ms":433.6,"lg_listing_warm_ms":465.9,"lg_nav_ms":6809.7,"lg_palette_ms":4.7,"lg_rename_ms":710.1,"lg_render_ms":228.7,"lg_search_ms":6891.2,"lg_split_ms":2259.2,"pure_graph_ms":214.5,"sm_boot_ms":457.7,"sm_decomp_ms":1303.6,"sm_graph_ms":700.2,"sm_hex_ms":445.4,"sm_index_ms":2.4,"sm_listing_cold_ms":271.4,"sm_listing_warm_ms":270.9,"sm_nav_ms":309.2,"sm_palette_ms":0.3,"sm_rename_ms":386.6,"sm_render_ms":265.8,"sm_search_ms":70.1,"sm_split_ms":1266.2,"fails":0},"status":"keep","description":"CORRECTNESS FIX, kept despite a worse metric. The un-chunked refresh was showing STALE NAMES on any wide read: one heads call for a search-sized window overflows the tool's 2000-row cap, the short response fails the sequence check, and the block is left with its old text. Refresh is now done a block at a time. Adds .auto/check_rename.py to the gate, which fails hard on the previous code and passes on this one.","timestamp":1786074769473,"segment":6,"confidence":4.974025132789222,"asi":{"the_bug_the_metric_was_rewarding":"/tmp/stalewindow.py renames a function, then reads the segment through window() the way the search body does. On the previous commit: 0 rows show the new name, the old one is still there, stale_structure=True. The 25564ms reading was FASTER because it was skipping the refresh -- the bench only failed to notice because phase_rename undoes its renames, so the stale text happened to be right again by the time anything compared it.","why_it_happened":"_ensure_text issued ONE heads call for the whole requested range. The tool caps a response at 2000 rows; a 4096-row search window came back short, page[:len(want)] != want, and the code took its 'the walk moved' branch -- marking the block fresh and leaving the old text.","fix":"refresh a TEXT_BLOCK (500 heads) at a time, looping. The sequence check then only ever fires for a real structural change.","cost":"total 25564 -> 28651. A wholesale re-read after a rename is ~10% dearer than a rebuild would be (6.6s vs 6.0s measured directly), and this bench does exactly that -- six renames then a split-view pass and a whole-segment search. The user-facing trade is: a rename is 566x faster (10.3ms vs 5660ms to get the listing back on bash), and a full-segment search immediately after one is ~10% slower.","counter_drift_resolved":"the 3233/1990/92733 counters in experiment #33 were NOT caused by the change -- both configurations reproduce 3436/2070/91783 twice each. That run was CPU-starved (the box is at load 1.6 and the 60s tool deadline can truncate a big decompile). Watch decomp_lines as a starvation signal.","gate_extended":".auto/check_rename.py: narrow read (painting), wide read (search body), and the whole model against a rebuild. Verified it FAILS on the previous commit with 4 problems and passes on this one.","principle":"the second time this session that the honest number is worse than the dishonest one. A benchmark rewards whatever it can see; the guard has to be a check that fails, not a number that improves."}}
+{"run":35,"commit":"6ec4bd8","metric":29392.6,"metrics":{"lg_boot_ms":711.9,"lg_decomp_ms":2406.4,"lg_graph_ms":1271.9,"lg_hex_ms":445.3,"lg_index_ms":71.9,"lg_listing_cold_ms":748.5,"lg_listing_warm_ms":403.8,"lg_nav_ms":6596.2,"lg_palette_ms":4.8,"lg_rename_ms":715.8,"lg_render_ms":219.3,"lg_search_ms":7055.2,"lg_split_ms":2629.8,"pure_graph_ms":212.8,"sm_boot_ms":445.6,"sm_decomp_ms":1281,"sm_graph_ms":775,"sm_hex_ms":442.3,"sm_index_ms":2.3,"sm_listing_cold_ms":265.6,"sm_listing_warm_ms":266.7,"sm_nav_ms":302.9,"sm_palette_ms":0.3,"sm_rename_ms":380.8,"sm_render_ms":255.1,"sm_search_ms":67,"sm_split_ms":1414.2,"fails":0},"status":"discard","description":"Documentation-only commit (playbook + ideas). Re-measured to confirm the post-fix state: 28651 -> 29393 is within the spread on this now-loaded box, and lg_search holds at ~7.0s, confirming that a whole-segment read after renames costs a whole-segment re-render.","timestamp":1786075083972,"segment":6,"confidence":7.108776152564333,"asi":{"state":"v7 baseline 33243 -> 29393 (-11.6%). rename 10055 -> 716 is the headline; the cost of re-rendering after a rename did not disappear, it moved to whoever reads the rows (lg_search 775 -> ~7000, because this bench reads the entire segment right after renaming). For a user reading a viewport the rename is 566x faster and nothing else changes.","why_there_is_no_more_free_win_here":"after a rename, the rows you read have to be re-rendered; re-rendering N heads costs what loading N heads costs. Refreshing only the rows that ACTUALLY changed would need xrefs_to(renamed_ea) to enumerate them, and a name can reach a row without a direct xref (a comment, a struct field in an operand) -- miss one and you are back to showing a stale name, which is the bug just fixed. Logged in .auto/ideas.md rather than attempted.","highest_yield_activity_this_session":"asking what the benchmark does NOT measure. Three of the five bench corrections found an invisible cost, and two of those (split view, rename) were among the biggest wins of the whole session.","next_action_hint":"keep probing uncovered features with the /tmp/featprobe.py shape: xrefs (x), the strings browser (\"), literal formats (o), make-code/data edits, history, traces, RPC. Domain-level probes already say xrefs/strings/structs/resolve are fast, so drive them through the UI instead."}}