| Commit message (Collapse) | Author | Files | Lines |
|---|
|
Fixes the bug found while building M3. Navigations run in workers and are applied
when they land; the trace's OPENING seek goes to t=0, which for a normal binary
is _start, and that navigation is slow. It arrived after later seeks and won,
leaving the cursor and _cur on _start while the trace's pc was elsewhere — and it
never settled, measured stable for 3+ seconds. Anything cursor-based done just
after a seek (`>` asks about the address under the cursor) then acted on the
wrong address.
The decompiler path has had a staleness guard since 756589a; the listing path
never got one. It has one now (_open_at_if_current), and a seek bumps _nav_seq so
older in-flight navigations are dropped.
Verified both directions on the exact reproduction: seek to the first execution
of a repeated instruction, seek to the second, wait — cursor stays put with the
guard, and with the guard removed it drifts to 0x34d0 (_start) exactly as
reported.
Scope, deliberately narrow. I first bumped _nav_seq in _goto_ea for EVERY
navigation, which is the more general rule, and a full run then failed
follow_xrefs — a follow can be dropped by whatever navigates next. That check has
flaked before so it is not proof, but the mechanism is real and my evidence is
only about seeks, so the bump lives in _seek. TODO records what would justify the
general version and what test it needs.
tests: +1 trace UI (39) — seek, seek again, wait 3s, and the cursor is still on
the instruction the trace is at. Two consecutive full runs 212/0 after narrowing.
|
|
M3. Stepping walks time; seeking jumps to the next time THIS thing was touched,
which is what makes a trace more than a very long single-step log.
`>` / `<` — next/previous execution of whatever the focused view addresses. One
pair of keys, two questions, because what's on screen already says which:
* listing: the instruction under the cursor. "When else did this run?"
* pseudocode: the whole C line, as the union of its instructions' executions. A
line is not one address, and falling back to its single /*ea*/ marker would
answer a narrower question — usually none at all, since most lines have no
marker.
* hex: the byte under the cursor, via memory_accesses.
It says where you landed ("execution of 0x3160: 2 of 2 @ t=320") and, at either
end, that you're AT the end rather than silently doing nothing — a key that does
nothing is indistinguishable from a broken one.
`W` — the registers with the instruction that set each to its current value, and
the distance back. Enter seeks to that write, f seeks forward. Backward is the
direction people want: you notice a bad value after it has been used. This is
the question a trace exists to answer and it was already in the model
(last_write/next_write), untested in anger until now.
tests: +13 trace UI (38) — > and < move between the two executions of a
repeated instruction, the status names which execution it is, both edges report
instead of moving, W opens, and choosing a register lands on an instruction that
REALLY wrote it (checked against the trace's own changed-set, not just the
timestamp matching).
Two things the tests taught me, both recorded:
* focus() does not make a view active outside split mode — Tab does. My first
seek test pressed > while _active was still "decomp", so it asked the
pseudocode about a line with no instructions.
* TODO gets a new entry: a stray late navigation to the entry function arrives
after a seek and wins, leaving the cursor on 'start' while the pc is
elsewhere. Same shape as the stale-decomp-result bug fixed in 756589a, which
got a sequence guard the listing path never did.
212/0 scenarios, 35/0 model, 12/0 differential.
|
|
Finishing the thing the last commit only worked around. The split view and the
trace path each kept their own per-line instruction map of the same
pseudocode — fetched separately, indexed separately, and keyed differently: the
split one on _cur (the function the CURSOR is in), the trace one on the function
the DECOMPILER has loaded. Those are not the same thing, which is precisely how
the two ended up describing different functions and why I spent a commit
chasing a "sparse decomp_map" that was never sparse.
_apply_split_map now indexes once and both read it. Keyed on the decompiler's
loaded function, and no longer conditional on split being on — the old guard
dropped the result whenever _cur had moved while the fetch was in flight, which
during stepping is almost always.
Measured after: three decomp_map fetches across 28 steps (two for main, one for
the function stepped into), the split map and the trace map are literally the
same object, and both describe what is on screen.
Process note, because this is the second time: my first attempt at this edit
SILENTLY DID NOTHING — the pattern didn't match (a duplicated comment line I'd
mangled), the old method stayed, and the new caller hit its `not self._split`
guard, so the painting tests went from passing to "0 lines". Same failure mode
as the key bindings that never got added. Structural edits now assert that the
anchor was found and that the replacement is present before writing.
212/0 scenarios, 26/0 trace UI, 30/0 project UI, 20/0 split view.
|
|
I blamed decomp_map in the last commit. It was innocent: called directly it
returns 769 lines, 475 with addresses, for exactly the function I said it
returned four for. The four-line map belonged to a PLT stub the decompiler had
momentarily switched to, and I sampled mid-bounce.
The actual fault: _seek_split decided "has execution left the decompiled
function?" from _split_range, which is maintained by a guarded async path
(_apply_split_map drops its result if _cur moved while in flight) and therefore
lags during stepping. A stale range made every step look like a function change,
so the decompiler bounced main -> stub -> main, each bounce paying a synchronous
769-line map fetch on the UI thread.
Now the decision comes from the map the trail painting already holds, keyed to
what the decompiler currently HAS loaded. The bouncing is gone — three map
fetches across twelve steps instead of one per step — and the pseudocode cursor
follows every instruction the decompiler attributes to a line, including across
a call into another function.
What it does NOT do: guess. Roughly half of a function's instructions have no
line attributed, and the obvious fallback (nearest mapped address at or before
the pc) is unsound — C lines are not monotonic in address, and it put an
instruction early in main on line 708, "sub_2040();", near the end. The cursor
waits instead; the trail still marks where you are.
tests: +1 trace UI (26) — over ~28 steps, every instruction that IS mapped is
followed by the pseudocode cursor. 212/0 scenarios.
TODO corrected: the entry blaming decomp_map now says what actually happened,
including that _split_ea2line/_split_range are still fed by the laggy path and
remain a latent issue for the split view's own sync.
|
|
Normal navigation moves one pane and gives the companion a band, never a cursor
— that rule exists so the two can't chase each other. A trace step isn't
navigation: time is a single global position and both panes are showing the same
instant, so the cursor belongs on it in both.
_seek_split places the listing cursor on the current instruction, then hands off
to the existing _sync_split so the companion still gets its band and align() at
the driver's screen row. The anchoring machinery is used, not bypassed.
PARTIAL, and the shortfall is worth stating plainly: the LISTING cursor tracks
the pc reliably (tested over consecutive steps). The PSEUDOCODE cursor only
follows when decomp_map covers that address, and for cat's main it covers almost
nothing — four entries for a 700-line function. That is not something this
commit introduced and not something I could fix responsibly without
understanding it; TODO has what I measured, including that dec.goto(96) left the
cursor at 0 in the same run, which may or may not be the same bug.
One real fix along the way: _place_decomp_at prefers the map the trail painting
keeps (keyed to the decompiler's currently loaded function) over the split
view's _split_ea2line. The latter is refreshed by a guarded async path that
drops its result if _cur moved while in flight, and a burst of steps moves _cur
constantly — so during stepping it is frequently a map of the function you just
left.
tests: +2 trace UI (25) — stepping in split moves the listing cursor onto the pc
for six consecutive steps, and the trail marks it 'now' in both panes. 212/0
scenarios.
|
|
The suite edits the database — defines code, undefines items, renames, comments —
and IDA saves all of it. Running that against targets/echo.i64 meant every run
inherited the last one's damage.
That cost real time twice. decomp_follow_self "started failing" with no code
change, and stayed failing until the .i64 was deleted; an edit-position check
looked flaky about one run in three and I nearly reported it as an async race.
Both were the database drifting. A suite whose result depends on its own history
cannot be trusted to accuse the code — and it had been quietly laundering bad
conclusions for however long.
Now the suite copies the binary into a temp dir and seeds it from a golden
database (<target>.pristine.i64) that nothing ever writes back to. Every run
starts from identical bytes; the tracked target is never opened.
The golden copy is built once, on first run, by analysing and saving before any
scenario runs — so it costs one analysis rather than one per run. Rebuilt
automatically if the binary is newer.
Verified: two consecutive full runs both 209/0; targets/echo.i64 no longer
exists after a run; a deliberately corrupted targets/echo.i64 is ignored
completely (11/0 with junk in place, and the junk untouched afterwards); no temp
directories leak.
The other suites were already clean for the same reason, by different means:
test_blob_ui builds a throwaway binary, test_project_ui stages copies, and
test_thumb_ui deletes the .i64 before each phase because the T flag and the
segment's bitness are saved in it.
|
|
|
|
The listing showed the mnemonic bright and every operand in one body colour.
IDA already classifies each token, for every processor it supports:
generate_disasm_line() emits \x01<tag>text\x02<tag> and the tag says what the
text IS. We were calling tag_remove() and throwing that away.
So: no lexer. A pygments asm lexer would be a worse guess and would need one
dialect per architecture — this is arch-correct for free, including the ARM/MIPS
blobs the loader work just made openable.
lea rcx, function; "usage"
insn reg punct name cmt
_idatui_spans() parses the tags into [[kind, text], ...], heads rows carry
"spans", Head.spans holds them, and _span_segments() renders them with a
fallback to the old mnemonic/rest split for older workers.
Palette rule: NEUTRALS for the machine (mnemonic brightest — it's the column you
scan; registers at body weight because they're most of the text), HUES only where
they mean something (numbers, strings, symbols), structure recedes so commas and
brackets stop competing with operands.
Two things that fail SILENTLY and are now encoded:
* The constants are SCOLOR_DATNAME / SCOLOR_CODNAME. There is no SCOLOR_DNAME —
a wrong guess leaves the tag unmapped, symbols render as plain body text, and
nothing tells you why. Probed the live IDA to get the real names.
* Spans must be whitespace-collapsed exactly as `text` is, walking characters
rather than per span, because a run of IDA's column padding straddles span
boundaries. A row only gets spans when they reconstruct `text` exactly, so a
mismatch degrades to the old rendering instead of corrupting the line.
The reason this was parked yesterday was NOT a bug in it. listing_view's
"undefining a data head yields an unknown run" waits for
`index_of_ea(dea) >= 0` — but dea is the head it just undefined, so it is in the
OLD model too and the predicate passes instantly, asserting against pre-edit
rows. It only ever passed because the model swap won the race; spans made pages
3x bigger, the swap lost, and the check accused working code. It now waits for
the model to be REPLACED.
Cost measured on libcrypto: 95KB per 500-row page, 50ms; model ensure(2000)
228ms. Acceptable for what it buys.
tests: new asm_highlight scenario (+7) — >90% of code rows carry spans, insn/reg/
punct present, every span kind has a style, spans reconstruct the row text
exactly, mnemonic is the first span. 202/0 scenarios, 26/0 blob, 30/0 project UI.
TODO: DisasmView appears to be dead code (never instantiated; Ctx.dis returns
ListingView), which is why this only needed doing once.
|
|
IDA already classifies every disassembly token for every processor it supports —
generate_disasm_line emits \x01<tag>text\x02<tag> and the tag says what the text
IS — so this needs no lexer at all, and certainly not a pygments asm lexer, which
would be a worse guess and need one dialect per architecture.
Working implementation is in docs/wip-asm-spans.patch and verified on echo:
"lea rcx, function; \"usage\"" tokenises as insn/text/reg/punct/text/name/cmt,
with sane coverage over 400 rows.
Not merged: it breaks listing_view's "undefining a data head yields an unknown
run" (reports kind=data). Bisected to this change on a fresh database — 10/0 with
it stashed, 9/1 with it applied. The span code doesn't touch `kind`, so the cause
is not yet understood, and I would rather park a working-but-unexplained change
than merge one that makes a real check lie.
TODO records the two non-obvious findings so the next attempt doesn't repeat
them: the constants are SCOLOR_DATNAME/SCOLOR_CODNAME (no SCOLOR_DNAME, and a
wrong guess fails silently as plain body text), and spans must be
whitespace-collapsed by walking characters, not per span, because a run of
spaces straddles span boundaries.
|
|
Two bugs in two days had the same shape — an edit rebuilds the model and
whatever ran last wins. The status message got clobbered by the reload's own
status write; the scroll position got recomputed from a row index that no longer
meant the same thing. Both were patched by hand. A third was coming.
ViewAnchor makes it one thing: where you are looking, in ADDRESSES, plus the
message the rebuild must not eat. _anchor() captures it on the UI thread before
the edit; _anchor_rows() resolves it against the rebuilt model; _edit_done()
settles the aftermath. The edit paths (_do_edit_item, _do_make_data) now hand one
object through instead of threading positions and messages separately.
Addresses, not indices, because an edit can change how many rows an item takes:
four undefined byte rows collapse into one instruction row, undefining does the
reverse. An index means a different place afterwards.
_reload_active_code deliberately does NOT use it. Renames and comments don't
change row structure, and the model that path rebuilds is constructed empty —
index_of_ea returns -1 until pages load, so an anchor would resolve to nothing
while costing an extra model build on the UI thread. Wrote that out and reverted
it rather than leave an abstraction applied where it does nothing.
Also fixed in passing: _do_make_data had the same latent bug (no scroll
preservation at all) and now goes through the same path.
The bigger find is in TODO. The scenario suite mutates targets/echo.i64 and
SAVES it, so a scenario that undefines an instruction breaks later runs
permanently — decomp_follow_self had been failing on a polluted database, not on
any code change. It also made an edit-position check look flaky one run in
three, which I nearly wrote up as a race. Coverage for this lives in
test_blob_ui.py instead, which builds a throwaway binary and can mutate freely.
tests: +1 blob UI (commenting leaves the view where it was), alongside the
carve checks. 26/0 blob, 202/0 scenarios (on a fresh .i64), 30/0 project UI.
|
|
Opening a binary now defaults to our own idalib worker; the ida-pro-mcp HTTP
supervisor path is deprecated (kept only for --db/attach and --backend mcp).
* launch.py: --backend default resolves to worker for a fresh binary open, mcp
for the attach modes (--db / bare `ida-tui`, which have no worker equivalent);
explicit --backend or IDATUI_BACKEND still wins. Logs a deprecation notice when
the mcp path is used.
* Deprecation markers on client.py and server/patch_server.py; the ida-tui shell
header and README now describe the worker as primary and note
$IDATUI_WORKER_PYTHON. TODO tracks the removal checklist.
No code deleted yet — the mcp fallback stays until the worker is proven on a box
where idalib can spawn (pilot against --backend worker is the gate). Backend
resolution matrix verified: `ida-tui bash`->worker, bare/`--db`->mcp, explicit
flag/env honored.
|
|
The function disasm view is now a filtered listing: DisasmModel sources its
lines from the heads walker bounded to [func.start, func.end) instead of the
disasm tool, so both code views render from the one listing mechanism.
DisasmView UI (opcode bytes, o-toggle, Tab, rename) preserved -> no test churn.
total() stays on disasm include_total: paging heads for the count hit the MCP
response-size limit (count=1000 truncated -> wrong count) and corrupted prime.
include_total is one fast exact call; for a code function it equals the heads
row count backing the lines. _fetch_block uses count=257 (safe).
Verified: disasm-heavy scenarios 56/0; full suite 127/1 (1 = flaky filter).
M4 complete.
|
|
struct-expand/unify deferred
|
|
Press 'd' on a listing head to define typed data at that address via a prompt
(the ".data type definitions" backlog item). Mirrors the retype prompt flow:
MakeDataRequested -> #makedata Input (prefilled with a size-appropriate default
type: unsigned __int8/16/32/64 or char[N]) -> _do_make_data worker ->
Program.make_data -> bump_items -> reopen the listing in place. Accepts any C
type IDA's SetType understands: int, char[16], my_struct, T *arr[4], ...
Pilot: listing_view gains a deterministic sub-test — undefine a data head to
synthesize an unknown run, assert it renders as `unknown`, then 'd' char[4]
over it and assert it becomes a `data` head. 118 pass / 1 pre-existing flaky
(filter); rpc_smoke 29/0.
|
|
The real disassembly-listing view. ListingView is a line-virtualized
ScrollView (reusing ColumnCursor/SearchMixin/NavMixin) backed by
ListingModel, rendering code, data (db/dw/dd, strings, jump tables) and
undefined bytes interleaved with per-kind styling. Non-function regions now
open in it instead of the M0 DisasmModel stopgap.
Integration (IdaTui): a new `listing` active mode. _open_entry routes region
entries to ListingView; _show_active/compose add it; follow/xrefs/comment and
the c/p/u edit verbs handle it (define_func upgrades a region straight to the
function view); backslash reaches hex and returns via _code_mode (so leaving
hex from a region lands back on the listing, not a func view); Tab explains
there's no pseudocode for a region. rpc._active_widget learns `listing`.
Server fix (the important one): the `heads` walker now steps by get_item_end
instead of next_head. next_head SKIPS undefined bytes, so after undefining a
function its start address wasn't even a listing line and the cursor snapped to
the next defined item; stepping by item-end renders undefined bytes as `db ?`
lines (IDA-accurate) and makes navigation land exactly on any address —
essential for "go to unmarked bytes and hit c". Needs a supervisor restart.
Tests: region_define rewritten to assert the ListingView path + segment-wide
listing + p-upgrade; new listing_view scenario (data-segment listing renders
data heads, cursor reports head ea, backslash->hex->back). Pilot 115 pass / 1
pre-existing flaky (filter); rpc_smoke 29/0; test_domain 27/0.
|
|
Navigating to an address not inside a function no longer refuses ("no
function contains X"): _do_navigate now opens a region view (flat listing
anchored at the EA, forced to disasm since there's no pseudocode). The
server's no-function disasm already walks heads to the segment end, so
DisasmModel is reused as-is; NavEntry gains is_region.
Adds the IDA c/p/u structure-edit verbs on the disasm view:
c = define_code (undefine-first, since create_insn won't carve a live item)
p = define_func
u = undefine
wired via EditItemRequested -> _do_edit_item worker -> Program.define_code/
define_func/undefine. define_func upgrades the region to a real function
view in place.
New Program.bump_items() invalidates the item/function structure caches
(disasm blocks, decomp, function indices + name gen) — broader than
bump_names(), which only covers renames.
Pilot: new `region_define` scenario undefines a small function, navigates
to the bare region via the 'g' prompt (asserts it opens, not refused),
then 'p' recreates the function and upgrades the view; restores the IDB in
a finally. 107 passed / 1 pre-existing flaky (filter, fails on baseline).
|
|
|
|
Fetch per-instruction opcode bytes alongside the disasm listing and show
them in a column between the address and the mnemonic. Instruction length
comes from consecutive addresses (variable-length safe: x86 movabs shows
its full 10 bytes); the block over-fetches one instruction for the last
line's boundary, falling back to the function end for the final insn.
The column pads to the widest instruction across the whole function:
_fetch_block tracks a running max, and on load a background scan_bytes()
settles a stable width so padding doesn't jump as blocks stream in.
_op_field is shared by the rendered strip, _line_plain, and search _fmt
so cursor/match offsets stay aligned. 'o' toggles the column.
|
|
- Visibility: #search/#rename/#status were dock:bottom like the Footer and got
placed on the SAME row (y=39) as the Footer, which drew over them, so the
typed text was invisible. Drop the docking; they now sit in normal flow just
above the Footer (y=38). Verified: input region is above the footer.
- Stale names on 'back': a rename only invalidated the current function, so
popping back to a caller (cached earlier) showed the old name. Add
Program.bump_names(): a rename bumps a name-generation, clears all disasm block
caches (disasm names are live), and decompilation is generation-checked and
force-recompiled lazily on next view. _after_rename now invalidates globally.
- verified: rename a callee, 'back' to the caller shows the new name.
- pilot suite 51/51.
|
|
Restoring only the cursor re-derived the viewport via scroll-into-view, so back
landed on the right line but scrolled differently. NavEntry now stores scroll_y
(disasm) and dec_scroll_y/dec_scroll_x (pseudocode); a single _save_current_pos()
snapshots cursor+scroll of both views right before navigating away, and load/show
restore the exact scroll. Disasm defers scroll_to via call_after_refresh (its
virtual_size is only set in _on_primed, so an immediate scroll_to clamps to 0).
Also routes rename's reload through the same save/restore path.
pilot suite 46/46.
|
