| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
"EMPTY BL" in the status bar is triskel's own bracket-list assertion from
its SESE pass, and it turned out to be the mild version of the problem.
Triskel's graph root is whichever node was created FIRST, and every
analysis walks out from it. Anything unreachable from that node is
undefined behaviour. We were:
- creating nodes in id order, so the root was the lowest-numbered
block rather than the entry, and
- splitting only WEAKLY connected components, which says nothing about
reachability.
A 7-block CFG whose entry has no successors -- IDA hands those out for
thunks and for dead code it could not resolve -- SEGFAULTS the
interpreter. That is unsurvivable: it takes the session down and there
is no exception to fall back from.
Now the entry is created first, orphan blocks are attached to it with
phantom edges that steer placement but are never drawn (one edge usually
adopts a whole orphan subgraph, attached at a node no other orphan
reaches), and reachability is asserted in python BEFORE crossing into
C++. This replaces the component splitting entirely: one layout instead
of N stacked side by side, and triskel gets to place the orphans.
The reproducer is now a test (t_unreachable_entry). Remaining fallbacks
on the ls corpus are 8/1200 layouts, all the upstream box-overlap
defect, all but one on 300-500 block functions.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
"engine triskel -> native+triskel-failed" was unreadable and, worse,
gave no reason: the cause went to a logger a TUI user never sees.
Dumped all 400 functions of bin/ls (echo's 128 were not enough) and swept
them at three zoom levels: 6 of 1200 layouts fell back, all of them my
own _verify tripping over a detour that could not be placed.
- the detour jumped to the nearest side of the FIRST box in the way,
which in a dense layout is usually inside the next box along. It now
collects every box the run passes and picks the nearest genuinely
free line.
- it skipped the first and last segments because they carry the port
and the arrowhead. But that is exactly where the failures were:
triskel is happy to park a block directly above its successor and
drive the final approach straight through it. Those segments may now
move ALONG their own box's border, which is free almost every time.
- repairs are swept to a fixed point: moving one segment stretches its
neighbours, which can push those into a box.
0/1200 fallbacks after that. Then the bigger corpus turned up a second,
genuinely upstream defect: superimposing SESE regions can leave two
blocks a couple of columns into each other (2 of ls's 400 functions,
in float space, before rounding). Cosmetic in a PNG; here the boxes are
made of text, so one block's disassembly overwrites another's. _verify
now checks it and falls back, which is the right trade.
Reporting, so this is never mute again:
- stats["engine_error"] carries the reason, the status line shows it,
and the label is "native (triskel failed)".
- the corpus test asserts fallbacks are rare AND explained, rather
than asserting they never happen.
Also lowered AUTO_TRISKEL_MAX_BLOCKS 250 -> 180. Layout runs on every
zoom keypress and triskel knees hard past ~175 blocks (174: 66ms,
233: 489ms, 329: 555ms). The old cap allowed a 489ms stall. The corpus
timing check now measures only sizes `auto` can actually reach, plus a
5s ceiling so nothing blows up quadratically when forced.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`e` in graph mode cycles auto -> native -> triskel, and `auto` prefers
triskel where it is installed and the function is at most 250 blocks.
Why: our layered engine draws wide-and-short pictures with a lot of
crossings on anything branchy. Triskel splits the CFG into Single-Entry
Single-Exit regions first and lays each out on its own, which on the
128-function corpus means fewer crossings on 12 functions, equal on 9,
worse on 3 -- and the wins are the hairballs (sub_5CA0 41 -> 6,
sub_2C90 32 -> 7, sub_2C00 12 -> 0). It also routes loop edges around
the side of the graph the way IDA does, which was a known gap here.
It is not free: ~2x slower at 87 blocks, 10x at 424, hence the cap.
The library needed a fork (~/dev/triskel, branch idatui) before it could
be used from Python at all -- its get_waypoints() threw on every
published version, an empty graph segfaulted the interpreter, and its
spacing constants were pixels baked in at compile time. Making those
settable is what makes this integration cheap: we hand it CELLS, so
its output is integral and two edge lanes can never round onto the same
row. The feared quantisation problem measured out backwards -- cells
claimed by more than one edge: native 131, triskel 35.
Not trusted with degenerate input, all handled before the call:
self-loops and disconnected components make it throw, and one corpus
edge comes back routed through a block, which we detour and re-verify.
A triskel failure is never fatal; it falls back to native.
Two things the second engine flushed out of the existing code:
- the canvas was sized from boxes alone, which is exact only because
native's dummy nodes reserve the space. Triskel routes outside that
bounding box and the edges were being clipped.
- arrowhead placement read e.back, conflating "this is a loop edge"
(style) with "this polyline runs against control flow" (geometry).
Now Edge.flipped, which is also a latent fix for residual-cycle edges
whose succ/pred were being reported backwards.
tests/test_graph.py runs its whole suite once per available engine
(943 checks); new graph_engine scenario covers the live toggle.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reported as "the splash logo stopped rendering". It had not stopped: the
splash asks for the artwork's NATURAL size and shows nothing when that
does not fit, and the artwork needs 31 rows plus 10 of box chrome. A pane
in a split zellij window is 31 rows — one row short of the 41 it wanted —
so the logo silently disappeared. Traced with $IDATUI_KITTY_LOG in the
real session:
compose: supported=True app.size=Size(width=159, height=31)
cells=60x23 fits=False
The terminal scales an image into whatever cell box it is placed in
(`c=`/`r=` on the placement), so there was never a reason for
all-or-nothing. `logo_cells(max_rows)` now fits the art to the room left
after the box's furniture, and the same number reserves the cells and
sizes the placement, so a resize needs no relayout. In that same 31-row
pane it now draws 55x21 instead of nothing.
Two things fixed on the way:
* The chrome constant was one row optimistic (`rows + 9` where the box
measures 10: border 2, padding 2, art margin 1, title 1, note 1+1,
help 1+1). At exactly the old threshold the help line was clipped off
the bottom.
* `_fits` conflated "is the terminal big enough" with "is the artwork the
right size", which is what made the image path inherit the block art's
all-or-nothing behaviour. The block art genuinely cannot scale (it is
half-block cells, 26 rows) and still falls back to the text splash;
the image no longer does.
`splash_scaling` pins it at 31, 30 and 44 rows: the logo is drawn, it is
scaled to the room, the box is never clipped, and a big pane still gets
the natural size.
905 passed, 0 failed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
SearchPalette opened pinned to the top of the screen: the CSS named the
screens that centre (`SymbolPalette, StringsPalette, ProjectPalette, …`)
and a new dialog is not on a list nobody remembers to edit. The comment
sitting above that rule — "every #pal-box palette centres, not just the
symbol one" — was the *first* time this happened.
`ModalScreen { align: center middle; }` matches subclasses, so every
dialog inherits it and the next one is centred for free; the eight
per-screen rules that only repeated it are gone. Textual's own Ctrl+P
CommandPalette is a ModalScreen too and wants its stock top alignment, so
it opts out in one visible line rather than by omission.
The `modal_centering` scenario checks both halves: that centring is
expressed as a rule, and that it actually reaches a dialog's laid-out
region (above/below and left/right within a cell).
894 passed, 0 failed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`/` only ever searched the lines of the view you were in. This adds the
search you actually need on a binary: over the entire database, either
through the rendered disassembly or through the image.
* **text** matches the line as displayed, whitespace-normalised, so
`call cs:` finds `call cs:getenv_ptr` (IDA's column padding is not
something anyone types). Smartcase; `regex` available over RPC.
* **bytes** is IDA's own `find_bytes`, so the pattern language people
already know works unchanged: hex pairs, `?` wildcards for a whole byte
or one nibble (`48 8? ?? 24`), quoted literals (`"Hello", 0`). Commas,
no separators (`488B05C3`) and ragged spacing all normalise.
**Which mode you meant is guessed, and the guess is biased on purpose.**
`dead`, `add`, `cafe` and `ff` are valid hex AND ordinary things to search
for, so a bare hex-looking word stays TEXT; nobody types `48 8b ?? c3`
meaning prose. `hex:`/`text:` prefixes and F2 override it.
The subtle case is a *typo* in a byte pattern. `48 zz c3` first fell
through to a text search and reported "no match" — indistinguishable from
"those bytes are not in this binary", which is the most misleading answer
a search can give. Now any query whose tokens are all byte-sized is
treated as bytes, and a bad token is refused BY NAME. IDA does the same
thing quietly (find_bytes answers a malformed pattern with zero hits and
no error), so the validation lives in Program.search, not just in the UI.
Enter searches, then Enter opens the highlighted hit; the title says which
it will do, because a database-wide scan is far too slow to run on every
keystroke like the other palettes. Navigation goes to the item head — a
byte match can start mid-instruction — and the status names the exact
address.
Also: the `find` RPC verb and `drive find`, which is the one an agent
wants (`drive find '48 8b ?? c3'`).
idatui/search.py holds the classification and is pure, so the whole
question of "what did they mean" is tested offline: tests/test_search.py,
35 checks, 0.1s. Pilot scenario db_search covers the UI end to end.
Full suite: 890 passed, 0 failed, 51.3s.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The output of an RE session is what you worked out, and it was locked in a
.i64 that only IDA can read. Ctrl+E (or `drive export`, or the `export` RPC
verb) writes it out: your comments grouped by function with the line each
annotates, the names and prototypes you set, the types you declared.
**The hard part was provenance, and it needed a mechanism, not a filter.**
A database does not record WHO wrote a comment or a name. IDA's analyzer
sets `; switch 73 cases` and `; s1` with the same `set_cmt` a person uses,
and the ELF loader sets `elf_gnu_hash_nbuckets` and `File class: 64-bit`
the same way. Four probes, all negative: the FF_COMM flag is identical,
`get_cmt` returns them all, `generate_disasm_line` tags every one of them
COLOR_REGCMT (not COLOR_AUTOCMT), and they survive with auto-comments
switched off. A first cut filtered by shape and produced a report whose
first screen was ELF header trivia and `; jumptable ... case 99`.
So idatui journals its own edits (idatui/journal.py) into a netnode in the
database: it rides along in the .i64, it is still there next session, and
the report is then exactly what was done here -- 2 findings out of a
database carrying 693 other annotations. Recorded at the choke points in
edit_ctl (rename, name-address, comment, retype) and in the struct editor;
flushed on save, on export and on quit, so no edit pays a round trip.
Without a journal (a database worked on in the IDA GUI, or predating this)
the report falls back to filtering by shape -- dummy names, imports, loader
segments, the analyzer's stereotyped switch/jumptable strings -- and says
so in the document rather than claiming authorship it cannot prove.
idatui/findings.py splits gather (needs IDA) from render (does not), so the
formatting, grouping, sorting, escaping and the empty cases are tested
offline: tests/test_findings.py, 32 checks, no worker, 0.1s. The pilot
scenario covers the round trip that matters -- edit through the UI, export,
find it in the file, and reload the journal from the .i64.
Full suite: 842 passed, 0 failed, 51.2s.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The suite spent its time in two kinds of guess.
**Flat pauses.** ~140 `pause(d)` calls were 20.4s of the pilot's 62s, and
`test_trace_ui` was 13.5s of `pilot.pause(1.0)` out of 19.6s. `Ctx.pause` is
now `settle` (`d` is the upper bound, not the cost) and the other suites'
sleeps became gates on the thing the check is about. `Ctx.sleep` stays for
what a timer really drives.
**Textual's keypress path.** `Pilot.press` calls `wait_for_idle` twice per
key, which sleeps in 20ms granules until process time stops advancing --
84ms per keypress here, 23s of the pilot's 43s. `_fixtures.fast_keys()`
replaces it with the gate the suites already use: send the keys, then
settle. Deleting the heuristic *without* that broke nine checks, so it was
doing a job, badly.
Four checks turned out to be riding on those sleeps: they read geometry or
a repaint (`si.region`, `gv._minimap_rect()`, glyphs off `gv.render_line`,
a repaint trace), and a settled app has not necessarily been laid out or
painted. They now wait for the frame. The debounced function filter
(`set_timer(0.08)`) likewise waits for its effect.
Also fixed two waits on signals that never arrive: the comment wait in
`rename` carried a `dec.loaded_ea == app._cur.ea` conjunct that cost 9s of
timeout and then let the check pass vacuously, and `listing_view` -- the
one entry under "Known-flaky" -- waited on `lst.total`, which is true
before a single row exists.
`--profile` reports, per scenario, seconds settling / waiting / pressing,
and names any wait that expired with its line number. It is how the above
was found and how the next 20s should be.
Verified: 4 full `tests/run.py` runs, 800 passed each, 49.0-49.2s
(was 117.4s); 4 consecutive pilot runs, 313 passed each, 21.2s (was 63.7s).
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
"the code view follows the trace" failed on the port (None vs 0x2ae4) and
passed on master, but it was not a trace regression: the test pressed "]",
waited for `app._t == 1`, and then read the listing cursor. `app._t` is
assigned the moment the key is handled -- the navigation it starts runs in a
worker -- so the wait was satisfied before the view had moved, and the check
read a cursor that had no address yet.
Master won that race because its backend answers in single-digit milliseconds.
The Code Mode backend is slower, so the race became a reliable failure. The
gate is now the condition the check is about (the cursor is on the trace's ip),
for both the forward and the backward step.
39 passed, 0 failed -- the same tally as master.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It was not slow because ARM analysis is slow. It was slow because it fought
Code Mode's ownership model and then waited out timeouts for the result.
The suite deleted <fixture>.i64 and REOPENED THE SAME PATH before each of its
four phases. That was safe when the TUI owned a private worker that died with
it. Under Code Mode the database is leased, and the previous phase's managed
worker can still hold it through its lease grace -- so the delete raced a live
owner, the next open never produced a listing, and the suite died on
`lst.model.index_of_ea(0)` with model=None after burning minutes in waits whose
results were never checked. Each phase now gets its own temp copy: separate
paths cannot collide and nothing waits for anyone to let go. This is the same
hazard docs/CODEMODE_PORT.md flags -- sweeping files that another client may
own -- showing up in the test suite rather than in the app.
Also replaced four `wait(lambda: lst.model is not <old>, ..., 60)` gates. An
item edit now keeps the listing's walk instead of rebuilding it, so the model
object is never replaced and each of those sat out its full 60 seconds while
the suite still reported success. They now settle() on the signal the checks
actually read: the status line announcing Thumb/ARM/64-bit, and the function
appearing in the index.
20 passed, 0 failed (it never reached a tally before).
Verified separately that the four ARM operations the port carried over do work
against a live database: set_thumb, thumb_scan, define_code_run, define_func_run.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Three separate wastes, all of the same family: waiting on a guess instead of a
signal, and paying for work that never had to be repeated.
1. The 64KB blob was built with os.urandom into a fresh TemporaryDirectory on
every run. New bytes at a new path means the pristine-database cache can
never apply, so full auto-analysis of 64KB of AArch64-decoded noise was paid
every single run. It is now built from a seeded PRNG at a stable path
(tests/.synthetic/, gitignored) and staged through the existing cache.
Determinism is also a correctness fix: whether 64KB of chance bytes contains
something IDA reads as a function is luck, and this suite asserts "and really
has no functions".
2. `wait(lambda: lst.model is not old, ..., 30)` after commenting. The perf work
made an item edit KEEP the listing's walk and re-render in place, so the
model object is never replaced and this waited out its full 30s timeout on
every run -- and then "commenting leaves the view where it was" passed
vacuously, because nothing had happened at all. A test that burns 30s to
check nothing is worse than no test.
3. Two `pause(2.0)`/`pause(2.5)` after a carve, replaced with settle() on a real
condition. The second one deliberately has NO predicate: that spot is random
data, so the carve may legitimately produce nothing, and "the row became
code" would never hold -- gating on it cost another 30s timeout. What that
check is about is the VIEW not moving, so the gate is "the app finished
reacting".
Fixing (1) exposed a real bug in the client, fixed here too: reopening a
database that already exists while passing loader switches is FATAL in IDA --
FATAL ERROR: Switch '-b400' can be used only when loading a new file
which kills the worker before it can report anything. Loader switches describe
an IMPORT and are recorded in the database they produce, so they are now sent
only when there is an import to describe. This was never reachable from the old
suite (a fresh random blob never had a database to reopen), but it is reachable
by any user who opens a raw blob with --ida-args twice.
30 passed, 0 failed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
signature
ida-codemode is now cloned at ../ida-codemode (0.3.1) and installed into
~/ida-venv, so the adapter can be checked against the library instead of
against assumptions.
First thing it found: connect() passed loading_address=, which
DatabaseHandle.open() does not have. The real parameter is image_base, and it
already wants the natural 16-byte-aligned address we compute, so this is a
rename. Every connect would have died with TypeError on the first call.
The port's own contract test could not catch it: its fake handle takes
**kwargs, so any keyword at all looks accepted. The test now also validates
the keywords we send against inspect.signature(DatabaseHandle.open) when the
library is importable, and skips that one check when it is not.
Offline suite: 302 passed with the library installed, 302 without it.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Mechanical part of the port: the 27-file patch was cut against a base ~148
commits behind us, so it did not apply. Resolved 11 conflicts (all of them
diff drift, not semantic clashes) and the three file deletions:
- app.py: the patch re-inserted _do_rename/_do_name_addr/_seek_split etc. as
"theirs" because our tree moved them to edit_ctl.py/trace_ctl.py. Kept ours
and applied the real intent (WorkerClient->CodeModeClient, .call->.invoke,
_open_worker_client->_open_database_client) at their current homes.
- domain.py: kept Head as a NamedTuple -- the patch reverted it to a frozen
dataclass, which the perf work measured at 2.9us vs 1.9us per row on a
quarter-million-row walk. Dropped _fetch_output (no download_url under Code
Mode) and its now-dead urllib/json imports.
- pane.py: the patch's deletion swallowed our zellij support along with the
worker-reaping block it meant to remove. Kept zellij, removed the reaping.
- test_scenarios.py: the idb_save->save_database teardown hunk belongs to
tests/_fixtures.py in our tree; applied it there and kept our pc_num_format
scenario that the drift landed on.
Three defects in the patch itself, fixed here:
- It made "import idatui" hard-require ida_codemode, so every offline suite
died at import -- including the pure ones (graph/index/trace) that are the
house rule for "tests/run.py --fast". The import is now deferred and gated
on the binding, which is also what lets the port's own contract tests
inject a fake DatabaseHandle.
- project.stage() inlined an ida_codemode.registry import and treated "library
not installed" as "someone owns this database", which broke IDA-free project
staging. Ownership lookup moved to codemode_client.database_owner().
- tests/test_codemode_client.py had no NEEDS_IDA marker, which tests/run.py
rejects outright.
Offline suite: 301 passed, 0 failed. Against master's 344 the whole delta is
accounted for: -40 worker_client (module deleted), -18 launch sweep checks
(behaviour deliberately removed) +3 guarding that it stays removed, +2 pool
(GUI-save semantics), +13 new codemode_client contract tests.
NOT yet done, and the port is not functional without it: the adapter is
missing five operations our tree grew since the patch's base (flowchart,
op_format, pc_nums, pc_num_format, survey_binary) and its "heads" predates
back-walking and digest/expect.
|
| |
|
|
|
|
| |
NOT running) revealed that the worker-connect poll change made test_project_ui flaky: 5ms polling on a background thread through a cold auto-analysis starved the UI thread enough that the loading overlay was still up when the test pressed Ctrl+O. Poll now backs off to a 25ms cap (keeps the boot win, no busy-wait), the racy boot wait is fixed, and checks.sh runs tests/run.py in full (830 checks) instead of just the scenario suite.
Result: {"status":"keep","total_ms":18608,"lg_boot_ms":708.5,"lg_decomp_ms":2454.9,"lg_graph_ms":1034.5,"lg_hex_ms":431.9,"lg_index_ms":95.1,"lg_listing_cold_ms":530.2,"lg_listing_warm_ms":413.4,"lg_nav_ms":7057.9,"lg_palette_ms":5,"lg_render_ms":214.2,"lg_search_ms":1408.9,"pure_graph_ms":213.3,"sm_boot_ms":433.6,"sm_decomp_ms":1292,"sm_graph_ms":754,"sm_hex_ms":433.4,"sm_index_ms":2.6,"sm_listing_cold_ms":260.4,"sm_listing_warm_ms":280.5,"sm_nav_ms":286.1,"sm_palette_ms":0.3,"sm_render_ms":251,"sm_search_ms":46.5,"fails":0}
|
| |
|
|
|
|
| |
opcode bytes already attached) with the graph_minimap scenario's racy SETUP made deterministic: clear _graph_sticky before the second navigation so Space is known to be entering the graph, not leaving it. No assertion changed.
Result: {"status":"keep","total_ms":22980.2,"lg_boot_ms":738.2,"lg_decomp_ms":2401.8,"lg_graph_ms":944.1,"lg_hex_ms":920.6,"lg_index_ms":75.2,"lg_listing_cold_ms":538.5,"lg_listing_warm_ms":411.1,"lg_nav_ms":6813.9,"lg_palette_ms":4.9,"lg_render_ms":221.8,"lg_search_ms":5630.1,"pure_graph_ms":240.7,"sm_boot_ms":537.5,"sm_decomp_ms":595.1,"sm_graph_ms":715.7,"sm_hex_ms":858.8,"sm_index_ms":0,"sm_listing_cold_ms":263.3,"sm_listing_warm_ms":265.3,"sm_nav_ms":335.2,"sm_palette_ms":0.3,"sm_render_ms":271.4,"sm_search_ms":196.5,"fails":0}
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A TUI must not die because one background load failed, so this codebase catches
broadly -- ~50 `except Exception` sites, two dozen resolving to `pass`. Right
policy, one bad consequence: with 44 `@work(thread=True)` workers, a failure in
a background load leaves no trace whatsoever. The view stays empty and there is
nothing to read afterwards, because the app owns the screen.
kittygfx already solved this for itself with $IDATUI_KITTY_LOG. idatui/diag.py
is the same idea for everything else: $IDATUI_LOG writes every swallowed error
plus its traceback to a file, and the last 50 are kept in memory regardless so a
driver can ask a live app what went wrong. Unset, it costs an environ lookup.
Wired in where losing the error changes a DECISION rather than just a pixel:
* rename: a resolve() that throws renames as DATA instead of as a function.
* name: a function_of() that throws means we never learn the address is a
function start, so the index keeps the old name and every readback says the
rename didn't happen.
* retype: a resolve() that throws retypes the ENCLOSING function instead.
* decompile: a failed full-body fetch silently returns CLIPPED pseudocode.
* trail: a failed decomp_map stops the pseudocode being painted, silently.
Deliberately NOT wired into the query_one guards -- a modal owning the screen is
normal and constant, and logging it would bury the real entries in noise.
New RPC verb `diag {n?, clear?}`, documented in docs/RPC.md: the answer to "the
verb reported success and the pane shows nothing".
Also a flake, same shape as the others: follow_xrefs waited on the nav depth but
asserted on _cur, and a follow pushes the source entry BEFORE opening the
target -- so the check could run in between and see the function it jumped
from. About one run in ten. It waits on the postcondition it asserts now; three
clean full runs since.
833 checks; --fast is 344 in 3.5s.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
_sweep_locks removes the scratch IDA unpacks beside a .i64 (.id0/.id1/.id2/
.nam/.til) when an open fails, keyed on both the full name and the stem. It
never touched the .i64, which is the dangerous one everybody thinks of.
It did delete the input. '.til' is an unpacked-DB suffix AND the extension of an
IDA type library, so 'ida-tui mylib.til' swept its own argument out of
existence -- irreversibly, on a path that runs automatically. Same for anything
named *.id0/*.id1/*.id2/*.nam. Now the sweep skips whatever it was asked to
open, compared as an absolute path so a relative argument is covered too.
tests/test_launch.py pins the whole contract: what it takes, what it must never
take (the .i64, the input, the neighbours), and what it reports. Pure, in the
--fast tier. It is the right shape of test for code whose failure mode is
deleting the wrong file.
Also: _load_args parsed the base with bare int(), which raises on the
'0x8000000' string a project file writes. Unreachable from our own CLI (which
int()s first) but the asymmetry with project._as_addr was a trap, so both go
through the same parser now.
813 checks; --fast is 324 in 3.4s.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The layer between the app and idalib had no tests, which is awkward: it is
where failures are silent. A worker that dies during startup, a socket that
drops mid-call, two UI threads sharing one socket -- none of those look like
bugs from outside, they look like the TUI hanging or showing stale data.
None of it needs IDA. WorkerClient spawns whatever _WORKER_PY points at, so the
suite points it at a fake speaking the same length-prefixed pickle and tells it
to misbehave on demand: die at startup, never bind, drop the connection, fail a
tool, take its time. 40 checks in the --fast tier.
Two things the tests found:
call() reconnects when _sock is None, which is what makes a dropped socket
recoverable -- but it made an explicitly CLOSED client resurrect too, spawning a
whole new idalib worker to serve one stray call (verified: pid 1066961 ->
1066962). close() runs on teardown and on binary-switch while @work threads are
still in flight, so quitting during a decompile could leave a fresh process
re-opening the .i64 we had just released, which is the wedging hazard. A closed
client now refuses; connect() still revives it, which is all _reconnect needs
(it builds a new client anyway).
connect() polled on a flat 0.2s sleep, so every caller paid a fifth of a second
even when the worker was ready in milliseconds -- a seeded .i64, a small binary.
Backs off from 5ms instead.
786 checks, 144.6s; --fast is 297 in 3.3s.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
_active was a bare string with 49 comparisons across four modules and a fifth
value nobody meant to keep. "disasm" was assigned on exactly one path -- a
decompile that failed with nowhere to return to -- and named the same widget as
"listing". Four sites understood it; five compared against "listing" alone and
silently took the wrong branch:
* Tab out of a failed decompile set "listing" instead of "decomp", so the
first press appeared to do nothing.
* rpc.py carried a workaround for a mode change that never arrived, keyed on
being ALREADY in the ghost state -- so it fired in the rare case and not in
the common one. Now keyed on LISTING, which is the case that happens.
* drive.py asked the socket to show it "disasm", a value the app will now
never report, and would have toggled twice and given up.
ViewMode is a StrEnum on purpose: _active goes straight to drivers as
cursor.kind and the pilot compares it to plain strings, so members being strings
keeps every payload and comparison working. What it buys is one place that says
which modes exist, and an AttributeError instead of silence on a typo.
Read it through is_listing/is_decomp/is_hex/is_graph/in_code rather than ==.
The bare comparisons are what let the ghost hide, and they are what the next
mode would have to hunt down -- adding "graph" already cost one crash that way
(_active_code_view returning None when a prompt closed).
view_modes_all_handled walks the enum and asks the app the questions it asks
itself. Verified it bites: adding a fifth unhandled member fails it twice.
746 checks, 142.3s.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The split resync loop (f898350) had no test. Two attempts at one were worthless
and are not in this commit: a scroll-based guard passed with the bug
reintroduced, and a constructed anchor -- inside the loaded function, outside
its mapped span -- skipped, because on this target the map covers the whole
function. The real trigger is the race window while the decomp map lags the
decompiler re-pointing, which is tedious to force but wide open in split_view's
own flow.
So split_view counts lookup_funcs across its body and bounds it. Verified both
ways, which is the only reason it's worth having: 29,227 calls with the bug put
back, under 500 with the fix. The bound is loose because the bug was three
orders of magnitude out, not a near miss.
733 checks, 139s.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
all_funcs() forced a full load of the function index only when it was EMPTY, so
a partially streamed index -- non-empty but incomplete, which is exactly the
state during boot and after any bump_items() -- came back truncated. Every
fixture picked through find_func/biggest therefore depended on how far streaming
had got by the time a scenario asked.
That is the graph_minimap flake: on an unlucky run find_func(size > 0x300)
picked a much larger function than usual, whose graph never finished inside the
scenario's own 60s wait. Three failures and 65 seconds, one run in several, with
no code change to blame. Three consecutive clean runs at 1.7s since.
CORRECTION to f898350, which said a range cache for function_of 'broke
graph_minimap'. It did not. The failure happened in the run after I added the
cache and I attributed it without checking; it recurred with the cache long
gone. The cache is still not here, but for the honest reason: with the resync
loop fixed, function_of is down to 340 calls and 1.4s across the whole suite,
so caching it is not worth the invalidation surface.
Suite 195.7s -> 138.4s, 732 checks.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
test_trace_ui spent 18.8 of its 35.2 seconds in flat pilot.pause() calls placed
to let an async seek land. Two loops were most of it: 6 iterations at 0.5s and
28 at 0.3s, 11.4s of sleeping to check that a step moves the cursor.
They are condition waits now. The questions are unchanged -- does the listing
cursor reach the pc, does the pseudocode cursor follow -- but they cost what
they cost instead of a fixed budget. The second loop settles on something that
does NOT presuppose the answer (the listing cursor arriving, and the trail map
belonging to the loaded function): waiting on 'is this pc mapped' would have
burned the timeout on every unmapped instruction, about half of them, and come
out slower than the sleep it replaced.
35.2s -> 20.9s, 39 checks, stable over repeated runs.
tests/_fixtures.py collects the staging both this suite and test_scenarios need
-- scratch copy, seeded from a golden .i64 nothing writes back to -- which was
private to test_scenarios. Worth saying plainly: on targets/echo the seeding is
worth 0.19s, not the analysis time I assumed when I went looking. It is shared
for the deduplication and for whatever gets pointed at a bigger binary.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
_split_range is the min/max of the decomp_map's addresses, which does not cover
every address in the function -- Hex-Rays doesn't attribute them all. An anchor
inside the loaded function but outside that span therefore asked _sync_split for
a resync, _apply_resync found the function already decompiled, called
_sync_split again, and it asked again. One thread worker and one lookup_funcs
round trip per iteration, for as long as the cursor sat there.
Measured in the pilot: 23,888 function_of calls in one scenario across FOUR
distinct addresses, 21,156 of them for 0x2060 alone. In the live app that is an
idle split view pegging the worker.
_sync_split grows a resync flag; the one caller that is itself the resync passes
resync=False, so the branch can be entered at most once per chain.
While measuring, three scenarios waited on "fail" appearing in the status --
the app says "cannot decompile". decomp_fallback burned its full 25s timeout and
then passed a check on _active == "listing", which was already true before Tab
was pressed: it asserted nothing, slowly. Now waits for the real text and checks
that the fallback actually said something.
scenarios 115.8s -> 74.9s, suite 195.7s -> 153.3s, 732 checks green.
Not included: a range cache for function_of. It broke graph_minimap (the graph
stopped loading at all -- the 65s was that scenario's own 60s wait timing out)
and with the loop gone it buys little. Left out rather than shipped
half-understood.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fourteen test files, each its own __main__, and no way to run them but from
memory -- so in practice you ran the one you were working on and hoped. Worse,
nothing said which files need a licensed IDA and a real worker (minutes) and
which are pure stdlib (milliseconds), so the cheap ones nobody ran either.
tests/run.py runs the lot and prints one table. --fast selects only the suites
that need nothing, which is 257 checks in half a second under any python3 --
that's the one you run between edits.
The classification lives in the test files, not in a table here that would rot
the first time someone adds a test: each declares NEEDS_IDA at module scope and
run.py reads it with ast (it can't import them -- they run their suite at
import). A file without the marker is a hard error rather than a silent guess.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
F1 is swallowed before it ever reaches us on at least one setup here -- the
app's own binding fires when the key is injected directly, zellij has no F1
binding of its own, and every common F1 encoding written straight into the pane
(SS3 ESC O P, CSI ESC [11~, CSI-u ESC [1;1P) opens it. So the key is being eaten
by something upstream, which is not ours to fix, and a cheatsheet reachable only
through a function key is fragile anyway: terminals and multiplexers claim them
routinely.
H opens and closes it too. '?' stays with the incremental search, which is what
it has always done in the code views.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Clicking the minimap panned to the exact coordinate under the pointer and moved
the cursor only if a block happened to sit there. Since one minimap cell covers
many canvas cells, "there" was almost always padding: you got a jump into empty
space and the cursor stayed behind, so you had to click a block afterwards to
actually go anywhere.
Blocks cover a few percent of a laid-out graph -- 4.6% of an 87-block function,
0.8% of a 424-block one -- and the rest is the space that keeps edges apart. So
coordinates are the wrong thing to navigate by here. The minimap now snaps to
the nearest block and takes the cursor with it, and a drag scrubs from block to
block. Distance is measured with the column halved, because cells are twice as
tall as they are wide and otherwise "nearest" is not what looks nearest.
A drag-pan or ctrl+d/pageup that ends with no block on screen at all now eases
to the nearest one too, since an empty screen leaves nothing to navigate back
by. It only fires when nothing is visible, so a deliberate pan is never fought.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Click it to jump the view to that part of the graph, drag to scrub. If the
point you clicked is over a block the cursor lands in it, so the keyboard
carries on from where you pointed instead of snapping back.
This also fixes a real bug rather than only adding a feature. The minimap
FLOATS over the canvas -- it is pinned to the viewport, not drawn into the
graph -- so a click on it was being translated into canvas coordinates and
dropping the cursor into whatever block happened to lie underneath. It has to
be hit-tested before the canvas, which is what on_click now does.
_minimap_rect() is the one source of truth for where it is: the renderer and
the hit-test both take the position from it, so the two-column inset that
keeps it clear of the ScrollView's scrollbar can't drift between them.
|
| |
|
|
|
|
|
|
| |
Opening, navigation and edge-following, the three zoom levels, the drawing
actually reaching the screen (a layout that is right but paints nothing
looks fine from the outside), clicking a block, renaming from inside one,
and the mode surviving a navigation. The help test now derives its group
list from _HELP instead of hardcoding it, so adding a card isn't a failure.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Textbook Sugiyama, the same shape IDA's own graph uses: break cycles,
longest-path layering, dummy nodes, median/transposition ordering,
priority x-coords, then port-and-channel edge routing. Pure python -- no
IDA, no Textual, no I/O -- so it is tested offline in milliseconds with no
worker, which is the whole reason the hard part is kept out of the UI.
Dummy nodes are what make routing tractable: a long edge occupies real
horizontal space, so no edge ever has to cross a box. The tests assert
exactly that over a 128-function corpus, and it holds at 0.
Two things cost real time to find. A self-loop never drains its own
in-degree, so it deadlocks the ranking and collapses the graph into three
layers, 280 columns wide -- they are dropped from the layout and drawn as
a marker. And crossing minimisation is the entire runtime: recounting
globally per candidate swap is O(n^3) and took 20.4s on a 424-block
function, against 152ms for Fenwick inversion counting plus a local
O(deg*deg) swap delta.
The result is not a painted canvas -- that function is ~13M cells. It is
an index: per-row runs, bucketed vertical intervals, and point marks,
queried one row at a time.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Pilot scenarios for the listing and the pseudocode, for the mark moving
between operands, for a refusal not being swallowed by the previous
success, and for the cursor staying on its literal across a reflow. Plus
experiments/opfmt_tools.py, which runs the real injected tool sources
against a live database with the decorators stubbed -- faster than the
pilot and the right place for the IDA-side edge cases.
Also fixes two pre-existing bugs the work surfaced, both of which made
edits happen off screen: cursor_on searched from row 0 of the whole
segment and never scrolled, so a driver's word= edit landed in an
unrelated function while reporting success; and the cursor verb didn't
scroll either. Both now go through rpc.place_cursor.
|
| |
|
|
|
|
|
|
|
| |
The multiplexer is auto-detected ($ZELLIJ then $TMUX) and every pane
command works the same under both. Pane ids are self-identifying, so a
mixed set of tmux and zellij panes can be tracked at once. zellij has no
-l, so --size is ignored there, it always focuses a new pane (--detached
is emulated), and it leaves an EXITED husk behind that stop/reap now
clear. The pane tests skip on neither multiplexer rather than on no TMUX.
|
| |
|
|
|
|
|
|
|
|
| |
Hex-Rays caches per function and does not notice that a *callee* was renamed;
worse, that cache is persisted in the .i64, so a bulk import left pseudocode
calling sub_98C0 forever while the listing and every readback said memset --
the exact readback disagreement a driver cannot detect. Batch now calls
force_recompile before bumping the local caches.
Test extended: decompile, rename via rename_many, read the pseudocode back.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Opening a headerless blob was the one workflow that fell out of the driving
surface entirely, and each gap hid the next:
- `pane spawn` couldn't pass --processor/--base/--ida-args, so the pane came up
"ready" with zero functions (x86 at 0) and the only way through was to
hand-write a project file. It now forwards them to idatui.launch.
- c/p/t/T (code, function, ARM<->Thumb, vector scan) existed as listing
bindings with no verb, so a driver had to guess raw keys -- and raw keys are
swallowed by whatever modal happens to be up. `define {kind,target?}` goes
through the app's own edit worker and reports what IDA actually did.
- every name went through the typed rename prompt: a navigation (listing page +
decompile) plus two prompt round-trips each. A 427-symbol map took tens of
minutes of driving. `rename_many {items|file}` hands IDA's rename tool the
whole list in one call (371 symbols in 3s) and refreshes the caches and the
function table once.
drive gains `define <kind> [target...]` and `syms <file.json>`.
Verified live against a real pane (tests/test_rawimage_rpc.py, 13 checks:
spawn load options, define thumb/func + unknown-kind rejection, rename_many
from a file and inline, with resolve/functions readback).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The existing trace suites (test_trace.py, test_trace_ui.py,
test_trace_vs_tenet.py) cover the model layer, the UI via the Textual pilot,
and differential correctness against Tenet's reference. None of them exercise
the path an agent actually takes: the trace RPC verb driven over the unix
socket through rpcclient.
This one spawns a real tmux pane with --trace, records a trace with the QEMU
tracer (falling back to /tmp/echotrace.0.log if the tracer isn't built), and
drives every trace operation through the RPC socket, validating the JSON
responses:
seek (absolute, percentage, string, edge-clamping)
step (forward, backward, multi-step, clamp at bounds)
step over (finds a call via SP drop, verifies it lands after the return)
goto (by name, by hex address, error on unexecuted)
changed registers in the response
cursor tracking (ea follows the trace pc)
response shape (trace key is a superset of snapshot)
interaction with non-trace verbs (pseudocode, state, view, goto)
trace position independence from navigation
All 45 checks pass against the echo binary with a 226-instruction trace.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes the bug found while building M3. Navigations run in workers and are applied
when they land; the trace's OPENING seek goes to t=0, which for a normal binary
is _start, and that navigation is slow. It arrived after later seeks and won,
leaving the cursor and _cur on _start while the trace's pc was elsewhere — and it
never settled, measured stable for 3+ seconds. Anything cursor-based done just
after a seek (`>` asks about the address under the cursor) then acted on the
wrong address.
The decompiler path has had a staleness guard since 756589a; the listing path
never got one. It has one now (_open_at_if_current), and a seek bumps _nav_seq so
older in-flight navigations are dropped.
Verified both directions on the exact reproduction: seek to the first execution
of a repeated instruction, seek to the second, wait — cursor stays put with the
guard, and with the guard removed it drifts to 0x34d0 (_start) exactly as
reported.
Scope, deliberately narrow. I first bumped _nav_seq in _goto_ea for EVERY
navigation, which is the more general rule, and a full run then failed
follow_xrefs — a follow can be dropped by whatever navigates next. That check has
flaked before so it is not proof, but the mechanism is real and my evidence is
only about seeks, so the bump lives in _seek. TODO records what would justify the
general version and what test it needs.
tests: +1 trace UI (39) — seek, seek again, wait 3s, and the cursor is still on
the instruction the trace is at. Two consecutive full runs 212/0 after narrowing.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
M3. Stepping walks time; seeking jumps to the next time THIS thing was touched,
which is what makes a trace more than a very long single-step log.
`>` / `<` — next/previous execution of whatever the focused view addresses. One
pair of keys, two questions, because what's on screen already says which:
* listing: the instruction under the cursor. "When else did this run?"
* pseudocode: the whole C line, as the union of its instructions' executions. A
line is not one address, and falling back to its single /*ea*/ marker would
answer a narrower question — usually none at all, since most lines have no
marker.
* hex: the byte under the cursor, via memory_accesses.
It says where you landed ("execution of 0x3160: 2 of 2 @ t=320") and, at either
end, that you're AT the end rather than silently doing nothing — a key that does
nothing is indistinguishable from a broken one.
`W` — the registers with the instruction that set each to its current value, and
the distance back. Enter seeks to that write, f seeks forward. Backward is the
direction people want: you notice a bad value after it has been used. This is
the question a trace exists to answer and it was already in the model
(last_write/next_write), untested in anger until now.
tests: +13 trace UI (38) — > and < move between the two executions of a
repeated instruction, the status names which execution it is, both edges report
instead of moving, W opens, and choosing a register lands on an instruction that
REALLY wrote it (checked against the trace's own changed-set, not just the
timestamp matching).
Two things the tests taught me, both recorded:
* focus() does not make a view active outside split mode — Tab does. My first
seek test pressed > while _active was still "decomp", so it asked the
pseudocode about a line with no instructions.
* TODO gets a new entry: a stray late navigation to the entry function arrives
after a seek and wins, leaving the cursor on 'start' while the pc is
elsewhere. Same shape as the stale-decomp-result bug fixed in 756589a, which
got a sequence guard the listing path never did.
212/0 scenarios, 35/0 model, 12/0 differential.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
M2. Trace.memory(addr, length, idx) reconstructs what memory held at a moment,
returning the bytes AND a per-byte "known" mask. The mask is the point: a trace
knows what it observed and nothing else, so a byte nobody read or wrote is
genuinely unknown and must not be drawn as zero. That distinction is the whole
reason to read memory from a trace instead of the database — the database has
the file's bytes, the trace has what was actually there.
Reads count as evidence, not just writes: an instruction reading a byte reveals
what it held then.
Indexed by ADDRESS (sorted once, bisect per query), because the question is
"what was in this window at time t" and the accesses that matter are the few
touching that window, not the tens of thousands in the trace.
Where the memory actually is: measured, 0% of accesses in either real trace fall
inside the image — every one is stack or heap. So the primary view is the STACK,
in the dock, anchored at SP:
stack (rsp)
▸7ffff6f99470 ????????????????
7ffff6f99478 00007ffff6fb0b00
7ffff6f99488 00007ffff6fa94e5
The hex view overlays trace bytes on the file's contents (green = the trace saw
this byte at this timestamp, grey = still the file's idea). Correct, and it will
matter for a program that writes globals, but on these traces it shows nothing —
which is why the stack pane is the deliverable and not a nice-to-have.
One bug the work surfaced: MemOp.addr was having the image slide applied to it,
which is nonsense for a stack address — it produced -0xc838. The slide relocates
the IMAGE; stack and heap have no database counterpart. Memory op addresses now
stay in trace space, and memory_raw() queries there, while memory() takes
database addresses for the hex view.
tests: +9 model (35) covering the known-mask, reads-as-evidence, partial
coverage and the writers/accessors queries; +1 differential (12) checking
reconstructed memory state against Tenet's own get_memory at sampled timestamps;
+5 UI (30) for the stack pane — present, anchored at SP, marks unseen bytes,
follows time. 212/0 scenarios.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I blamed decomp_map in the last commit. It was innocent: called directly it
returns 769 lines, 475 with addresses, for exactly the function I said it
returned four for. The four-line map belonged to a PLT stub the decompiler had
momentarily switched to, and I sampled mid-bounce.
The actual fault: _seek_split decided "has execution left the decompiled
function?" from _split_range, which is maintained by a guarded async path
(_apply_split_map drops its result if _cur moved while in flight) and therefore
lags during stepping. A stale range made every step look like a function change,
so the decompiler bounced main -> stub -> main, each bounce paying a synchronous
769-line map fetch on the UI thread.
Now the decision comes from the map the trail painting already holds, keyed to
what the decompiler currently HAS loaded. The bouncing is gone — three map
fetches across twelve steps instead of one per step — and the pseudocode cursor
follows every instruction the decompiler attributes to a line, including across
a call into another function.
What it does NOT do: guess. Roughly half of a function's instructions have no
line attributed, and the obvious fallback (nearest mapped address at or before
the pc) is unsound — C lines are not monotonic in address, and it put an
instruction early in main on line 708, "sub_2040();", near the end. The cursor
waits instead; the trail still marks where you are.
tests: +1 trace UI (26) — over ~28 steps, every instruction that IS mapped is
followed by the pseudocode cursor. 212/0 scenarios.
TODO corrected: the entry blaming decomp_map now says what actually happened,
including that _split_ea2line/_split_range are still fed by the laggy path and
remain a latent issue for the split view's own sync.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Normal navigation moves one pane and gives the companion a band, never a cursor
— that rule exists so the two can't chase each other. A trace step isn't
navigation: time is a single global position and both panes are showing the same
instant, so the cursor belongs on it in both.
_seek_split places the listing cursor on the current instruction, then hands off
to the existing _sync_split so the companion still gets its band and align() at
the driver's screen row. The anchoring machinery is used, not bypassed.
PARTIAL, and the shortfall is worth stating plainly: the LISTING cursor tracks
the pc reliably (tested over consecutive steps). The PSEUDOCODE cursor only
follows when decomp_map covers that address, and for cat's main it covers almost
nothing — four entries for a 700-line function. That is not something this
commit introduced and not something I could fix responsibly without
understanding it; TODO has what I measured, including that dec.goto(96) left the
cursor at 0 in the same run, which may or may not be the same bug.
One real fix along the way: _place_decomp_at prefers the map the trail painting
keeps (keyed to the decompiler's currently loaded function) over the split
view's _split_ea2line. The latter is refreshed by a guarded async path that
drops its result if _cur moved while in flight, and a burst of steps moves _cur
constantly — so during stepping it is frequently a map of the function you just
left.
tests: +2 trace UI (25) — stepping in split moves the listing cursor onto the pc
for six consecutive steps, and the trail marks it 'now' in both panes. 212/0
scenarios.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Caught while demoing this in a live pane, not by a test: press Tab to read the
pseudocode, press ] once, and you're back in the disassembly.
_seek() follows the trace by navigating to the new PC, and navigating to an
ADDRESS opens the listing unless the decompiler is explicitly preferred. So
every step out of C dropped you out of C — the painting work of the last commit
was unusable in the view it was built for, from the first keypress.
_seek now passes prefer_decomp=(self._active == "decomp"), the same thing the
xref handler already does for the same reason.
Worth noting what it looks like when it works: stepping in pseudocode follows
execution INTO a callee and the view switches to that function's C, which is
what you want and what makes the decompiler painting worth having.
tests: +2 trace UI (23) asserting the view survives a step in both directions.
212/0 scenarios.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The status bar said "0x2490" without saying what it belonged to. Obvious once
there are two panes open, or after switching binaries in a project — which
already prefixed its label, so single-binary sessions were the odd ones out.
[cat] .text @ 0x472b [listing] (c code · p func · u undefine · Enter follow)
Uses the opened file's basename, not _module(): that one asks the worker over
RPC and this runs on every status write. Kept in step when the path changes
(project switch, reload).
Three messages already carried the module name themselves and would have read
"[echo] echo — 128 functions"; they don't say it twice now.
tests: +3 scenarios (212) — the bar names the file, keeps naming it as you move
(the idle status is not the only writer), and doesn't say it twice.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Both code views now show where you came from and where you're going: the
instruction you're on ('now'), the ~96 steps behind it ('past', warm) and the
~96 ahead ('future', cool).
A trail, not all of history. Painting every address the trace ever touched says
almost nothing on a loop-heavy program; the last and next few dozen steps say
how you GOT here. Where an address appears on both sides — a loop body, which is
most of them — the nearer side wins, because that's the one explaining the step
you just took or are about to.
**The pseudocode is painted too**, which is the reason to build this here rather
than use Tenet. A trace records instructions, so that's what Tenet paints. We
already have decomp_map from the split-view work, saying which instructions each
C line covers, so the same trail lands on the decompilation:
line 46 now | v3 = getenv("POSIXLY_CORRECT");
line 47 future | v4 = (__int64)*a2;
line 49 future | if ( v3 )
A C line covers many instructions, so it takes the strongest kind present: now
beats past beats future — if the instruction you're standing on belongs to this
line, this line is where you are.
Two things kept cheap: the trail is recomputed per SEEK rather than per repaint
(~200 lookups, and repaints vastly outnumber steps), and decomp_map is cached
per function because it's an RPC and stepping is interactive.
The colours sit deliberately under the code palette — the trail says "you came
through here", the text still has to read as code.
tests: +8 UI (21) — the listing carries now/past/future and it reaches the
screen; pseudocode is painted; exactly ONE C line is 'now' and it is the line
covering the current instruction (not merely some executed line, which is the
mistake this check exists to catch). 209/0 scenarios, 27/0 model, 10/0 diff.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
M0 of the trace viewer. --trace FILE loads a Tenet trace beside the binary and
docks a pane on the right: where you are in time, the register state there, and
a timeline.
Docked rather than modal on purpose. A trace turns every other view into "state
at time T", so time and registers are context you read WHILE looking at code,
not something you open and dismiss.
The registers the current instruction WROTE are highlighted. That difference is
the entire reason a delta trace is readable, and it's free — the trace already
says which registers each line changed.
] / [ step one instruction. } / { step over, by following the stack pointer: a
call pushes, so the callee runs with SP below where we started, and stepping
until SP comes back up lands after the return. That's cheaper and more portable
than recognising call instructions per architecture, and it degrades correctly —
on an instruction that calls nothing, SP is already >= the start and it's one
step. Verified on a real call: t=13 -> 18, past 5 instructions, where a plain
step gives 14.
The load waits for the function index because rebasing needs the database's
addresses: our echo trace runs at 0x7ffff6faa000 and the same code sits at
0x2000 in the database. Rebased -0x7ffff6fa8000, 12 functions touched.
Register values stay as the trace recorded them (they're machine state) while
everything else on screen is in database addresses, so the header shows both —
"pc 0x2aed (trace 0x7ffff6faaaed)" — rather than leaving the two to be puzzled
over side by side.
tests: test_trace_ui.py (13) records its own trace with the QEMU tracer and
drives the real UI — loads, rebases onto real functions, the dock renders, ] and
[ step and the code view follows, and } steps OVER a call found in that trace
rather than at a hardcoded index. Skips with a message if the tracer isn't
built. 209/0 scenarios.
One thing worth recording: my first attempt to add the key bindings SILENTLY did
nothing — the pattern contained a literal \\u2026 where the file has a real
ellipsis, so the replace matched nothing and the bindings never appeared. The
action worked when called directly, which made it look like a key-routing
problem. Assert on the replacement, not on the diff looking plausible.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
First slice of the execution-trace viewer: parse, index and query a Tenet trace.
No UI yet — this is the layer everything else stands on, and its shape decides
whether decompiler painting is cheap later.
A Tenet trace is a line-per-instruction delta log: registers that changed, the
PC every line, and each memory access WITH its bytes. That's enough to
reconstruct any register or address at any point in time, in either direction.
Our own reader, not a port. The reference (tenet-original) packs traces into
segments with compressed address/mask tables, which earns its keep for its Qt
timeline; we need different queries and would rather own ~400 lines than inherit
3700.
Indexed around the query the UI actually asks, which the reference answers one
address at a time: WHICH TIMESTAMPS EXECUTED THIS SET OF ADDRESSES. A listing
row is one address, but a pseudocode line covers many (we already have
decomp_map for that), so by_ip maps address -> timestamps and hits() takes a
set. Painting a pseudocode line will be one call, not one per instruction.
Registers are stored as per-register change points, so a value at time t is a
bisect, and "which instruction set this register?" (last_write) is the same
lookup — that being the question a trace explorer exists to answer.
Rebasing is not optional: our echo trace runs at 0x7ffff6faa000 while the
database has that code at 0x2490. Page offsets survive relocation, so the low 12
bits of an instruction address are invariant; bucket the database's addresses by
those bits and take the slide the most trace addresses agree on. Verified
against a real IDB: slide -0x7ffff6fa8000, and it picks out the 12 functions the
trace actually entered (main, start, ...) from 128.
Performance: 176k instructions parse in 350ms (~500k lines/s), so a 10M-line
trace is ~20s and wants a progress callback, which load() takes.
FOUND A BUG IN THE REFERENCE while building the differential test. A register
written on the LAST line of a 65535-line segment is missing from the next
segment's base state, so Tenet returns a stale value until that register is
written again — measured: wrong for all 179 timestamps of one such window. It
survives in the reference because it only shows when the register isn't
rewritten immediately.
That changed how the test works. Rather than "must agree with the reference",
it ARBITRATES with the raw text when they differ: if the text backs us it's
reported and allowed, if the text backs them it fails. Blanket agreement would
have made us copy their bug to stay green.
tests: test_trace.py (27, pure stdlib) covers reconstruction, the set queries,
rebasing (including that a lone agreeing address is not enough, and that
matching is on page offsets rather than addresses looking plausible) and
malformed input; test_trace_vs_tenet.py (10) diffs against the reference on real
traces.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The suite edits the database — defines code, undefines items, renames, comments —
and IDA saves all of it. Running that against targets/echo.i64 meant every run
inherited the last one's damage.
That cost real time twice. decomp_follow_self "started failing" with no code
change, and stayed failing until the .i64 was deleted; an edit-position check
looked flaky about one run in three and I nearly reported it as an async race.
Both were the database drifting. A suite whose result depends on its own history
cannot be trusted to accuse the code — and it had been quietly laundering bad
conclusions for however long.
Now the suite copies the binary into a temp dir and seeds it from a golden
database (<target>.pristine.i64) that nothing ever writes back to. Every run
starts from identical bytes; the tracked target is never opened.
The golden copy is built once, on first run, by analysing and saving before any
scenario runs — so it costs one analysis rather than one per run. Rebuilt
automatically if the binary is newer.
Verified: two consecutive full runs both 209/0; targets/echo.i64 no longer
exists after a run; a deliberately corrupted targets/echo.i64 is ignored
completely (11/0 with junk in place, and the junk untouched afterwards); no temp
directories leak.
The other suites were already clean for the same reason, by different means:
test_blob_ui builds a throwaway binary, test_project_ui stages copies, and
test_thumb_ui deletes the .i64 before each phase because the T flag and the
segment's bitness are saved in it.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An ARM function pointer carries the mode in bit 0: odd means Thumb. A Cortex-M
vector table is therefore a list of Thumb entry points, and IDA won't follow them
on a headerless image because nothing tells it those words are pointers at all.
Shift+T scans forward from the cursor and marks them.
0 functions -> 3 Thumb entries found, 3 disassembled
A word only counts when it is odd, lands in a loaded segment, and its target is
executable and not already data. The even words in a vector table — the initial
stack pointer — fail the first test, which is the point: marking a data word as
code corrupts the listing, so a false positive costs more than a miss. The
fixture includes an even in-range word and an odd OUT-of-range word to keep that
honest.
A note on how this started: I recommended this feature, then probed
experiments/fibonacci.bin for the signal and found ZERO odd in-range pointers —
it's a flat code blob, not a firmware image. Rather than build a detector I
couldn't test, I wrote experiments/cortexm.bin: a real vector table pointing at
small self-contained Thumb handlers. The first version of that fixture aimed its
handlers into the middle of copied code, so two "entries" were really inside one
function — the tool was right and the fixture was wrong, which is worth stating
because I nearly filed it as a bug.
Function creation goes through one _idatui_add_func helper now, shared with
define_func_run: add_func(ea) alone fails on freshly-marked code (IDA can't find
the end), and the scan hit exactly the same wall `p` did.
Status precedence, fixed properly this time. An action's result kept being
overwritten by the reload it triggered — cursor moved, filter re-applied,
functions re-counted. I patched that at FIVE separate call sites before
admitting it's one problem. _status(text, priority=True) now marks a result: it
holds the bar for 8s or until the next keypress, and routine chatter can't
outrank it. The per-site special cases are gone.
tests: +4 thumb (20) — a bare vector table gives IDA nothing, scanning finds
exactly the three handlers, the non-pointer words are ignored, and the result
survives both the reload and the reindex. 209/0 scenarios, 30/0 blob, 30/0
project UI.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The 64-bit failure now reported Hex-Rays verbatim — "only 64-bit functions can
be decompiled in the current database" — with the actionable half appended after
it. A status bar cuts off the end, so the user got a perfect description of their
problem and nothing about what to do, which is the same dead end as before with
extra words.
before: sub_0: cannot decompile — only 64-bit functions can be decompiled in
the current database — Ctrl+L and pick arm:ARMv7-A (125 chars)
after: sub_0: cannot decompile — this database is 64-bit — Ctrl+L, pick
arm:ARMv7-A (76 chars)
For this one failure the instruction IS the whole message: it can't be fixed in
place (bitness is decided at load), so describing the database serves nobody.
Other Hex-Rays reasons still pass through verbatim — they're usually about the
function, and there the description is the useful part.
Verified the whole path in a live pane on experiments/fibonacci.bin: the load
dialog now shows arm vs arm:ARMv7-A/M/v6-M/v5TE with their bitness spelled out,
picking arm:ARMv7-A gives 54 functions, and sub_0 decompiles:
void __fastcall __noreturn sub_0(int a1) { ... v2 = sub_E3C(a1, 0); ... }
tests: thumb (16) now asserts the message names the fix rather than quoting
Hex-Rays, and that it fits under 110 chars — the truncation is what made the
last version useless, so it's worth a check. 209/0 scenarios, 30/0 blob.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
"I can't decompile my freshly defined function" — and the app's answer was to
flash and go back to the listing. It knew the reason and threw it away twice.
The plain decompile tool reports "Decompilation failed at 0x0 (address: 0x0)".
Hex-Rays itself fills in a hexrays_failure_t with the actual sentence, and in
this case it is the whole answer: "only 64-bit functions can be decompiled in
the current database". Nobody can guess that from a flash, and it is not fixable
in place — the database's bitness is set at load — so without the message there
is no way forward at all.
New decomp_error tool returns Hex-Rays' own description; _load_decomp asks for it
in the same worker when a decompile fails, and the status now reads:
sub_0: cannot decompile — only 64-bit functions can be decompiled in the
current database — Ctrl+L and pick arm:ARMv7-A
Then it got thrown away a second time, by the reload. Falling back to the
listing reopens it, and the reload writes its own status afterwards — the same
clobber that has now bitten four times. The fix this round is the last one:
_flash is SHOWN by idle status writes but no longer CONSUMED by them, because a
reload emits several (prime, then cursor) and consuming on the first meant the
second erased the message. It clears on the next keypress instead — when the
user has actually moved on.
(That also fixes a self-inflicted "status: None": the old code read _flash back
after something else had already consumed it.)
tests: +2 thumb (15) — a failed decompile says why in Hex-Rays' words, and the
reason survives the view reloading under it. 209/0 scenarios, 30/0 blob, 30/0
project UI.
|
