| Commit message (Collapse) | Author | Files | Lines |
|---|
|
Fixes the bug found while building M3. Navigations run in workers and are applied
when they land; the trace's OPENING seek goes to t=0, which for a normal binary
is _start, and that navigation is slow. It arrived after later seeks and won,
leaving the cursor and _cur on _start while the trace's pc was elsewhere — and it
never settled, measured stable for 3+ seconds. Anything cursor-based done just
after a seek (`>` asks about the address under the cursor) then acted on the
wrong address.
The decompiler path has had a staleness guard since 756589a; the listing path
never got one. It has one now (_open_at_if_current), and a seek bumps _nav_seq so
older in-flight navigations are dropped.
Verified both directions on the exact reproduction: seek to the first execution
of a repeated instruction, seek to the second, wait — cursor stays put with the
guard, and with the guard removed it drifts to 0x34d0 (_start) exactly as
reported.
Scope, deliberately narrow. I first bumped _nav_seq in _goto_ea for EVERY
navigation, which is the more general rule, and a full run then failed
follow_xrefs — a follow can be dropped by whatever navigates next. That check has
flaked before so it is not proof, but the mechanism is real and my evidence is
only about seeks, so the bump lives in _seek. TODO records what would justify the
general version and what test it needs.
tests: +1 trace UI (39) — seek, seek again, wait 3s, and the cursor is still on
the instruction the trace is at. Two consecutive full runs 212/0 after narrowing.
|
|
M3. Stepping walks time; seeking jumps to the next time THIS thing was touched,
which is what makes a trace more than a very long single-step log.
`>` / `<` — next/previous execution of whatever the focused view addresses. One
pair of keys, two questions, because what's on screen already says which:
* listing: the instruction under the cursor. "When else did this run?"
* pseudocode: the whole C line, as the union of its instructions' executions. A
line is not one address, and falling back to its single /*ea*/ marker would
answer a narrower question — usually none at all, since most lines have no
marker.
* hex: the byte under the cursor, via memory_accesses.
It says where you landed ("execution of 0x3160: 2 of 2 @ t=320") and, at either
end, that you're AT the end rather than silently doing nothing — a key that does
nothing is indistinguishable from a broken one.
`W` — the registers with the instruction that set each to its current value, and
the distance back. Enter seeks to that write, f seeks forward. Backward is the
direction people want: you notice a bad value after it has been used. This is
the question a trace exists to answer and it was already in the model
(last_write/next_write), untested in anger until now.
tests: +13 trace UI (38) — > and < move between the two executions of a
repeated instruction, the status names which execution it is, both edges report
instead of moving, W opens, and choosing a register lands on an instruction that
REALLY wrote it (checked against the trace's own changed-set, not just the
timestamp matching).
Two things the tests taught me, both recorded:
* focus() does not make a view active outside split mode — Tab does. My first
seek test pressed > while _active was still "decomp", so it asked the
pseudocode about a line with no instructions.
* TODO gets a new entry: a stray late navigation to the entry function arrives
after a seek and wins, leaving the cursor on 'start' while the pc is
elsewhere. Same shape as the stale-decomp-result bug fixed in 756589a, which
got a sequence guard the listing path never did.
212/0 scenarios, 35/0 model, 12/0 differential.
|
|
M2. Trace.memory(addr, length, idx) reconstructs what memory held at a moment,
returning the bytes AND a per-byte "known" mask. The mask is the point: a trace
knows what it observed and nothing else, so a byte nobody read or wrote is
genuinely unknown and must not be drawn as zero. That distinction is the whole
reason to read memory from a trace instead of the database — the database has
the file's bytes, the trace has what was actually there.
Reads count as evidence, not just writes: an instruction reading a byte reveals
what it held then.
Indexed by ADDRESS (sorted once, bisect per query), because the question is
"what was in this window at time t" and the accesses that matter are the few
touching that window, not the tens of thousands in the trace.
Where the memory actually is: measured, 0% of accesses in either real trace fall
inside the image — every one is stack or heap. So the primary view is the STACK,
in the dock, anchored at SP:
stack (rsp)
▸7ffff6f99470 ????????????????
7ffff6f99478 00007ffff6fb0b00
7ffff6f99488 00007ffff6fa94e5
The hex view overlays trace bytes on the file's contents (green = the trace saw
this byte at this timestamp, grey = still the file's idea). Correct, and it will
matter for a program that writes globals, but on these traces it shows nothing —
which is why the stack pane is the deliverable and not a nice-to-have.
One bug the work surfaced: MemOp.addr was having the image slide applied to it,
which is nonsense for a stack address — it produced -0xc838. The slide relocates
the IMAGE; stack and heap have no database counterpart. Memory op addresses now
stay in trace space, and memory_raw() queries there, while memory() takes
database addresses for the hex view.
tests: +9 model (35) covering the known-mask, reads-as-evidence, partial
coverage and the writers/accessors queries; +1 differential (12) checking
reconstructed memory state against Tenet's own get_memory at sampled timestamps;
+5 UI (30) for the stack pane — present, anchored at SP, marks unseen bytes,
follows time. 212/0 scenarios.
|
|
Finishing the thing the last commit only worked around. The split view and the
trace path each kept their own per-line instruction map of the same
pseudocode — fetched separately, indexed separately, and keyed differently: the
split one on _cur (the function the CURSOR is in), the trace one on the function
the DECOMPILER has loaded. Those are not the same thing, which is precisely how
the two ended up describing different functions and why I spent a commit
chasing a "sparse decomp_map" that was never sparse.
_apply_split_map now indexes once and both read it. Keyed on the decompiler's
loaded function, and no longer conditional on split being on — the old guard
dropped the result whenever _cur had moved while the fetch was in flight, which
during stepping is almost always.
Measured after: three decomp_map fetches across 28 steps (two for main, one for
the function stepped into), the split map and the trace map are literally the
same object, and both describe what is on screen.
Process note, because this is the second time: my first attempt at this edit
SILENTLY DID NOTHING — the pattern didn't match (a duplicated comment line I'd
mangled), the old method stayed, and the new caller hit its `not self._split`
guard, so the painting tests went from passing to "0 lines". Same failure mode
as the key bindings that never got added. Structural edits now assert that the
anchor was found and that the replacement is present before writing.
212/0 scenarios, 26/0 trace UI, 30/0 project UI, 20/0 split view.
|
|
I blamed decomp_map in the last commit. It was innocent: called directly it
returns 769 lines, 475 with addresses, for exactly the function I said it
returned four for. The four-line map belonged to a PLT stub the decompiler had
momentarily switched to, and I sampled mid-bounce.
The actual fault: _seek_split decided "has execution left the decompiled
function?" from _split_range, which is maintained by a guarded async path
(_apply_split_map drops its result if _cur moved while in flight) and therefore
lags during stepping. A stale range made every step look like a function change,
so the decompiler bounced main -> stub -> main, each bounce paying a synchronous
769-line map fetch on the UI thread.
Now the decision comes from the map the trail painting already holds, keyed to
what the decompiler currently HAS loaded. The bouncing is gone — three map
fetches across twelve steps instead of one per step — and the pseudocode cursor
follows every instruction the decompiler attributes to a line, including across
a call into another function.
What it does NOT do: guess. Roughly half of a function's instructions have no
line attributed, and the obvious fallback (nearest mapped address at or before
the pc) is unsound — C lines are not monotonic in address, and it put an
instruction early in main on line 708, "sub_2040();", near the end. The cursor
waits instead; the trail still marks where you are.
tests: +1 trace UI (26) — over ~28 steps, every instruction that IS mapped is
followed by the pseudocode cursor. 212/0 scenarios.
TODO corrected: the entry blaming decomp_map now says what actually happened,
including that _split_ea2line/_split_range are still fed by the laggy path and
remain a latent issue for the split view's own sync.
|
|
Normal navigation moves one pane and gives the companion a band, never a cursor
— that rule exists so the two can't chase each other. A trace step isn't
navigation: time is a single global position and both panes are showing the same
instant, so the cursor belongs on it in both.
_seek_split places the listing cursor on the current instruction, then hands off
to the existing _sync_split so the companion still gets its band and align() at
the driver's screen row. The anchoring machinery is used, not bypassed.
PARTIAL, and the shortfall is worth stating plainly: the LISTING cursor tracks
the pc reliably (tested over consecutive steps). The PSEUDOCODE cursor only
follows when decomp_map covers that address, and for cat's main it covers almost
nothing — four entries for a 700-line function. That is not something this
commit introduced and not something I could fix responsibly without
understanding it; TODO has what I measured, including that dec.goto(96) left the
cursor at 0 in the same run, which may or may not be the same bug.
One real fix along the way: _place_decomp_at prefers the map the trail painting
keeps (keyed to the decompiler's currently loaded function) over the split
view's _split_ea2line. The latter is refreshed by a guarded async path that
drops its result if _cur moved while in flight, and a burst of steps moves _cur
constantly — so during stepping it is frequently a map of the function you just
left.
tests: +2 trace UI (25) — stepping in split moves the listing cursor onto the pc
for six consecutive steps, and the trail marks it 'now' in both panes. 212/0
scenarios.
|
|
Caught while demoing this in a live pane, not by a test: press Tab to read the
pseudocode, press ] once, and you're back in the disassembly.
_seek() follows the trace by navigating to the new PC, and navigating to an
ADDRESS opens the listing unless the decompiler is explicitly preferred. So
every step out of C dropped you out of C — the painting work of the last commit
was unusable in the view it was built for, from the first keypress.
_seek now passes prefer_decomp=(self._active == "decomp"), the same thing the
xref handler already does for the same reason.
Worth noting what it looks like when it works: stepping in pseudocode follows
execution INTO a callee and the view switches to that function's C, which is
what you want and what makes the decompiler painting worth having.
tests: +2 trace UI (23) asserting the view survives a step in both directions.
212/0 scenarios.
|
|
The status bar said "0x2490" without saying what it belonged to. Obvious once
there are two panes open, or after switching binaries in a project — which
already prefixed its label, so single-binary sessions were the odd ones out.
[cat] .text @ 0x472b [listing] (c code · p func · u undefine · Enter follow)
Uses the opened file's basename, not _module(): that one asks the worker over
RPC and this runs on every status write. Kept in step when the path changes
(project switch, reload).
Three messages already carried the module name themselves and would have read
"[echo] echo — 128 functions"; they don't say it twice now.
tests: +3 scenarios (212) — the bar names the file, keeps naming it as you move
(the idle status is not the only writer), and doesn't say it twice.
|
|
Driving the trace viewer needed the same treatment as everything else: seeking by
hand through a few hundred keypresses to reach an interesting timestamp is not a
way to test it.
trace {seek: 120} absolute timestamp
trace {seek: "!50"} halfway through, like Tenet's timestamp shell
trace {goto: "main"} first execution of a name or 0xADDR
trace {step: 20} relative, negative goes back
trace {step: 5, over: true}
Returns the usual state snapshot plus {idx, length, pc, changed} so a driver can
see where it landed and what that instruction wrote.
pane spawn --trace FILE passes it through to the launcher, so a trace pane is one
command.
|
|
Both code views now show where you came from and where you're going: the
instruction you're on ('now'), the ~96 steps behind it ('past', warm) and the
~96 ahead ('future', cool).
A trail, not all of history. Painting every address the trace ever touched says
almost nothing on a loop-heavy program; the last and next few dozen steps say
how you GOT here. Where an address appears on both sides — a loop body, which is
most of them — the nearer side wins, because that's the one explaining the step
you just took or are about to.
**The pseudocode is painted too**, which is the reason to build this here rather
than use Tenet. A trace records instructions, so that's what Tenet paints. We
already have decomp_map from the split-view work, saying which instructions each
C line covers, so the same trail lands on the decompilation:
line 46 now | v3 = getenv("POSIXLY_CORRECT");
line 47 future | v4 = (__int64)*a2;
line 49 future | if ( v3 )
A C line covers many instructions, so it takes the strongest kind present: now
beats past beats future — if the instruction you're standing on belongs to this
line, this line is where you are.
Two things kept cheap: the trail is recomputed per SEEK rather than per repaint
(~200 lookups, and repaints vastly outnumber steps), and decomp_map is cached
per function because it's an RPC and stepping is interactive.
The colours sit deliberately under the code palette — the trail says "you came
through here", the text still has to read as code.
tests: +8 UI (21) — the listing carries now/past/future and it reaches the
screen; pseudocode is painted; exactly ONE C line is 'now' and it is the line
covering the current instruction (not merely some executed line, which is the
mistake this check exists to catch). 209/0 scenarios, 27/0 model, 10/0 diff.
|
|
M0 of the trace viewer. --trace FILE loads a Tenet trace beside the binary and
docks a pane on the right: where you are in time, the register state there, and
a timeline.
Docked rather than modal on purpose. A trace turns every other view into "state
at time T", so time and registers are context you read WHILE looking at code,
not something you open and dismiss.
The registers the current instruction WROTE are highlighted. That difference is
the entire reason a delta trace is readable, and it's free — the trace already
says which registers each line changed.
] / [ step one instruction. } / { step over, by following the stack pointer: a
call pushes, so the callee runs with SP below where we started, and stepping
until SP comes back up lands after the return. That's cheaper and more portable
than recognising call instructions per architecture, and it degrades correctly —
on an instruction that calls nothing, SP is already >= the start and it's one
step. Verified on a real call: t=13 -> 18, past 5 instructions, where a plain
step gives 14.
The load waits for the function index because rebasing needs the database's
addresses: our echo trace runs at 0x7ffff6faa000 and the same code sits at
0x2000 in the database. Rebased -0x7ffff6fa8000, 12 functions touched.
Register values stay as the trace recorded them (they're machine state) while
everything else on screen is in database addresses, so the header shows both —
"pc 0x2aed (trace 0x7ffff6faaaed)" — rather than leaving the two to be puzzled
over side by side.
tests: test_trace_ui.py (13) records its own trace with the QEMU tracer and
drives the real UI — loads, rebases onto real functions, the dock renders, ] and
[ step and the code view follows, and } steps OVER a call found in that trace
rather than at a hardcoded index. Skips with a message if the tracer isn't
built. 209/0 scenarios.
One thing worth recording: my first attempt to add the key bindings SILENTLY did
nothing — the pattern contained a literal \\u2026 where the file has a real
ellipsis, so the replace matched nothing and the bindings never appeared. The
action worked when called directly, which made it look like a key-routing
problem. Assert on the replacement, not on the diff looking plausible.
|
|
First slice of the execution-trace viewer: parse, index and query a Tenet trace.
No UI yet — this is the layer everything else stands on, and its shape decides
whether decompiler painting is cheap later.
A Tenet trace is a line-per-instruction delta log: registers that changed, the
PC every line, and each memory access WITH its bytes. That's enough to
reconstruct any register or address at any point in time, in either direction.
Our own reader, not a port. The reference (tenet-original) packs traces into
segments with compressed address/mask tables, which earns its keep for its Qt
timeline; we need different queries and would rather own ~400 lines than inherit
3700.
Indexed around the query the UI actually asks, which the reference answers one
address at a time: WHICH TIMESTAMPS EXECUTED THIS SET OF ADDRESSES. A listing
row is one address, but a pseudocode line covers many (we already have
decomp_map for that), so by_ip maps address -> timestamps and hits() takes a
set. Painting a pseudocode line will be one call, not one per instruction.
Registers are stored as per-register change points, so a value at time t is a
bisect, and "which instruction set this register?" (last_write) is the same
lookup — that being the question a trace explorer exists to answer.
Rebasing is not optional: our echo trace runs at 0x7ffff6faa000 while the
database has that code at 0x2490. Page offsets survive relocation, so the low 12
bits of an instruction address are invariant; bucket the database's addresses by
those bits and take the slide the most trace addresses agree on. Verified
against a real IDB: slide -0x7ffff6fa8000, and it picks out the 12 functions the
trace actually entered (main, start, ...) from 128.
Performance: 176k instructions parse in 350ms (~500k lines/s), so a 10M-line
trace is ~20s and wants a progress callback, which load() takes.
FOUND A BUG IN THE REFERENCE while building the differential test. A register
written on the LAST line of a 65535-line segment is missing from the next
segment's base state, so Tenet returns a stale value until that register is
written again — measured: wrong for all 179 timestamps of one such window. It
survives in the reference because it only shows when the register isn't
rewritten immediately.
That changed how the test works. Rather than "must agree with the reference",
it ARBITRATES with the raw text when they differ: if the text backs us it's
reported and allowed, if the text backs them it fails. Blanket agreement would
have made us copy their bug to stay green.
tests: test_trace.py (27, pure stdlib) covers reconstruction, the set queries,
rebasing (including that a lone agreeing address is not enough, and that
matching is on page offsets rather than addresses looking plausible) and
malformed input; test_trace_vs_tenet.py (10) diffs against the reference on real
traces.
|
|
An ARM function pointer carries the mode in bit 0: odd means Thumb. A Cortex-M
vector table is therefore a list of Thumb entry points, and IDA won't follow them
on a headerless image because nothing tells it those words are pointers at all.
Shift+T scans forward from the cursor and marks them.
0 functions -> 3 Thumb entries found, 3 disassembled
A word only counts when it is odd, lands in a loaded segment, and its target is
executable and not already data. The even words in a vector table — the initial
stack pointer — fail the first test, which is the point: marking a data word as
code corrupts the listing, so a false positive costs more than a miss. The
fixture includes an even in-range word and an odd OUT-of-range word to keep that
honest.
A note on how this started: I recommended this feature, then probed
experiments/fibonacci.bin for the signal and found ZERO odd in-range pointers —
it's a flat code blob, not a firmware image. Rather than build a detector I
couldn't test, I wrote experiments/cortexm.bin: a real vector table pointing at
small self-contained Thumb handlers. The first version of that fixture aimed its
handlers into the middle of copied code, so two "entries" were really inside one
function — the tool was right and the fixture was wrong, which is worth stating
because I nearly filed it as a bug.
Function creation goes through one _idatui_add_func helper now, shared with
define_func_run: add_func(ea) alone fails on freshly-marked code (IDA can't find
the end), and the scan hit exactly the same wall `p` did.
Status precedence, fixed properly this time. An action's result kept being
overwritten by the reload it triggered — cursor moved, filter re-applied,
functions re-counted. I patched that at FIVE separate call sites before
admitting it's one problem. _status(text, priority=True) now marks a result: it
holds the bar for 8s or until the next keypress, and routine chatter can't
outrank it. The per-site special cases are gone.
tests: +4 thumb (20) — a bare vector table gives IDA nothing, scanning finds
exactly the three handlers, the non-pointer words are ignored, and the result
survives both the reload and the reindex. 209/0 scenarios, 30/0 blob, 30/0
project UI.
|
|
The 64-bit failure now reported Hex-Rays verbatim — "only 64-bit functions can
be decompiled in the current database" — with the actionable half appended after
it. A status bar cuts off the end, so the user got a perfect description of their
problem and nothing about what to do, which is the same dead end as before with
extra words.
before: sub_0: cannot decompile — only 64-bit functions can be decompiled in
the current database — Ctrl+L and pick arm:ARMv7-A (125 chars)
after: sub_0: cannot decompile — this database is 64-bit — Ctrl+L, pick
arm:ARMv7-A (76 chars)
For this one failure the instruction IS the whole message: it can't be fixed in
place (bitness is decided at load), so describing the database serves nobody.
Other Hex-Rays reasons still pass through verbatim — they're usually about the
function, and there the description is the useful part.
Verified the whole path in a live pane on experiments/fibonacci.bin: the load
dialog now shows arm vs arm:ARMv7-A/M/v6-M/v5TE with their bitness spelled out,
picking arm:ARMv7-A gives 54 functions, and sub_0 decompiles:
void __fastcall __noreturn sub_0(int a1) { ... v2 = sub_E3C(a1, 0); ... }
tests: thumb (16) now asserts the message names the fix rather than quoting
Hex-Rays, and that it fits under 110 chars — the truncation is what made the
last version useless, so it's worth a check. 209/0 scenarios, 30/0 blob.
|
|
"I can't decompile my freshly defined function" — and the app's answer was to
flash and go back to the listing. It knew the reason and threw it away twice.
The plain decompile tool reports "Decompilation failed at 0x0 (address: 0x0)".
Hex-Rays itself fills in a hexrays_failure_t with the actual sentence, and in
this case it is the whole answer: "only 64-bit functions can be decompiled in
the current database". Nobody can guess that from a flash, and it is not fixable
in place — the database's bitness is set at load — so without the message there
is no way forward at all.
New decomp_error tool returns Hex-Rays' own description; _load_decomp asks for it
in the same worker when a decompile fails, and the status now reads:
sub_0: cannot decompile — only 64-bit functions can be decompiled in the
current database — Ctrl+L and pick arm:ARMv7-A
Then it got thrown away a second time, by the reload. Falling back to the
listing reopens it, and the reload writes its own status afterwards — the same
clobber that has now bitten four times. The fix this round is the last one:
_flash is SHOWN by idle status writes but no longer CONSUMED by them, because a
reload emits several (prime, then cursor) and consuming on the first meant the
second erased the message. It clears on the next keypress instead — when the
user has actually moved on.
(That also fixes a self-inflicted "status: None": the old code read _flash back
after something else had already consumed it.)
tests: +2 thumb (15) — a failed decompile says why in Hex-Rays' words, and the
reason survives the view reloading under it. 209/0 scenarios, 30/0 blob, 30/0
project UI.
|
|
"even when I define functions with `p` I still get 'no functions: wrong
processor/base?'". Two bugs, and the second is the worse one.
The hint was LATCHED at load and only cleared on a reload, so it went on telling
you the image was described wrongly long after you'd proved otherwise. It is now
derived: the moment the index has a function, it stops being true.
But the index never had one. Nothing rebuilt _func_index after an edit, so `p`
gave you a function the rest of the app could not see — the names pane didn't
list it and Ctrl+N couldn't find it. The hint was just the visible symptom of
that.
_edit_done now reindexes when the edit changed which functions exist (`p` and
`u`; carving code doesn't, and a full walk after every `c` would be waste). It
uses its own worker rather than _load_functions(), which is the BOOT path — that
one clears the table, streams progress and auto-lands, which would yank the view
off the function you just made.
Verified on a blob with no functions: carve, `p`, and the status reads "created
function 0x4040–0x404c", the index reports 1, and the hint is gone.
Also: the reload confirmation said "1 functions". It counts now.
tests: +4 blob UI (30) — no functions and the hint says so, `p` creates one the
index can see, the stale hint is gone, the status names it. The Ctrl+L check
became wrong in the good way and now asserts the confirmation counts what would
be lost, since by then there IS something to lose. 209/0 scenarios, 13/0 thumb,
30/0 project UI.
|
|
Reported as "after c a few times and p at the entry point, Tab just flashes and
nothing decompiles". Three separate things, found by following it down:
**1. `p` failed on hand-carved code.** ida_funcs.add_func(ea) asks IDA to find
the function's end and on carved code it often can't — a run ending in a tail
call, or whose last instruction isn't recognised as a return, fails with no
reason given. add_func(ea, end) with an explicit end succeeds. define_func_run
tries IDA's way first, then falls back to the end of the contiguous instruction
run, and says which it used.
**2. The database was 64-bit, so Hex-Rays refused it regardless.** Bare `-parm`
gives an AArch64 database. Ask Hex-Rays for the failure object rather than
reading None as "dunno" and it says exactly what's wrong: "only 64-bit functions
can be decompiled in the current database". So the disassembly looked right and
F5 could never work.
That is decided at LOAD and cannot be corrected — inf_set_app_bitness(32)
afterwards makes the decompiler INTERR 50735. The fix is at the load dialog:
arm:ARMv7-A (most firmware), arm:ARMv7-M / arm:ARMv6-M (Cortex-M, Thumb only)
and arm:ARMv5TE now sit alongside 64-bit `arm`, labelled with their bitness.
With arm:ARMv7-A, experiments/fibonacci.bin decompiles:
void __fastcall __noreturn sub_0(int a1) { int v2; v2 = sub_E3C(a1, 0); ... }
— and IDA's own auto-analysis finds 54 Thumb functions on load, versus none as
plain `arm`.
**3. `t` was silently building an undecompilable state.** It forced the SEGMENT
to 32-bit in a 64-bit database, which produces correct-looking disassembly that
F5 will never touch. It now says so and names the fix (Ctrl+L, arm:ARMv7-A)
rather than leaving you to discover it.
tools/verify_procs.py now reports each processor's resulting bitness, since that
is the reason the variants exist — and it compares against the base module name,
because a variant reports "ARM".
tests: test_thumb_ui.py +5 (13 total) — a 64-bit database warns and names the
fix, a 32-bit one finds functions by itself, Tab decompiles a Thumb function and
the result reads like C. test_formats.py +2 (34) pinning that a 32-bit variant is
offered and the ARM labels state their bitness. 209/0 scenarios, 26/0 blob, 30/0
project UI.
|
|
`c` could not carve Thumb code. Thumb isn't a property of the bytes — it's a
mode the CPU is in — so a raw image gives IDA nothing to detect: at a Thumb entry
point it decodes 16-bit instructions as 32-bit ARM and produces confident
nonsense. experiments/fibonacci.bin starts with `08 b5` = push {r3,lr}, which
IDA reads as SVCLT 0xBF00.
`t` on the listing switches the mode at the cursor and disassembles in it:
Thumb @ 0x0 (segment set to 32-bit; Thumb needs ARM32) — 10 instructions
0x0 PUSH {R3,LR} 0x2 MOVS R1, #0 0x4 MOV R4, R0 0x6 BL unk_E3C
Setting the T segment register is only half of it. Thumb does not exist in
AArch64, and a headerless blob loaded with -parm comes up 64-bit, so T alone
changes nothing and looks broken — I watched exactly that happen while probing
the API. Asking for Thumb IS asking for ARM32, so set_thumb forces the segment
to 32-bit and says so rather than doing it silently.
It also has to del_items over the range first: the bytes are currently decoded
in the old mode, and leaving that item defined pins the wrong instruction length
so the new mode has nothing to apply to.
Implemented as a `thumb` kind in the existing edit-item flow, so it inherits the
shared reload — same cache bump, same ViewAnchor restore, same status flash. It
switches AND disassembles, because flipping T and leaving the bytes undefined
shows you nothing and reading the code was the point.
tests: new tests/test_thumb_ui.py (8) driving the real Thumb binary — `c` alone
does NOT produce the prologue, `t` does, the instructions are 16-bit wide (in ARM
mode those three rows would be one 4-byte instruction), the run continues, the
status explains the 32-bit forcing, and `t` toggles back. Deletes the .i64 first,
because T and the segment's addressing mode are saved in it and a stale database
would answer the question for us.
209/0 scenarios, 26/0 blob, 8/0 thumb.
|
|
DisasmView was the function-scoped code view from before the unification. The
app hasn't instantiated it since — compose() yields only ListingView, and even
the test harness's Ctx.dis returns ListingView with a comment saying so. Its
CursorMoved messages had no handler, so every one it posted went nowhere.
304 lines of rendering, search, cursor and navigation logic that never ran. It
also cost real time this week: it made assembly highlighting look like a job
that needed doing twice, and its isinstance branches in the follow and xrefs
handlers were unreachable twins of the ListingView branch directly below them,
which is exactly the kind of thing you read carefully before realising it can't
execute.
Gone with it: the dead branches (folded into the ListingView ones, keeping the
fall-through-edge comment that was worth keeping), its CSS rule, and the
comments that pointed at it as though it were a live alternative.
DisasmModel STAYS — the domain still uses it to index a function's instructions
(_do_edit_item resolves a row within a function that way). Only the widget was
dead.
209/0 scenarios, 26/0 blob, 30/0 project UI, 36/0 index, 32/0 formats, 39/0
project, 27/0 pool. Smoke-tested a live pane afterwards: the listing renders,
highlighted, and `drive where` answers.
|
|
The listing showed the mnemonic bright and every operand in one body colour.
IDA already classifies each token, for every processor it supports:
generate_disasm_line() emits \x01<tag>text\x02<tag> and the tag says what the
text IS. We were calling tag_remove() and throwing that away.
So: no lexer. A pygments asm lexer would be a worse guess and would need one
dialect per architecture — this is arch-correct for free, including the ARM/MIPS
blobs the loader work just made openable.
lea rcx, function; "usage"
insn reg punct name cmt
_idatui_spans() parses the tags into [[kind, text], ...], heads rows carry
"spans", Head.spans holds them, and _span_segments() renders them with a
fallback to the old mnemonic/rest split for older workers.
Palette rule: NEUTRALS for the machine (mnemonic brightest — it's the column you
scan; registers at body weight because they're most of the text), HUES only where
they mean something (numbers, strings, symbols), structure recedes so commas and
brackets stop competing with operands.
Two things that fail SILENTLY and are now encoded:
* The constants are SCOLOR_DATNAME / SCOLOR_CODNAME. There is no SCOLOR_DNAME —
a wrong guess leaves the tag unmapped, symbols render as plain body text, and
nothing tells you why. Probed the live IDA to get the real names.
* Spans must be whitespace-collapsed exactly as `text` is, walking characters
rather than per span, because a run of IDA's column padding straddles span
boundaries. A row only gets spans when they reconstruct `text` exactly, so a
mismatch degrades to the old rendering instead of corrupting the line.
The reason this was parked yesterday was NOT a bug in it. listing_view's
"undefining a data head yields an unknown run" waits for
`index_of_ea(dea) >= 0` — but dea is the head it just undefined, so it is in the
OLD model too and the predicate passes instantly, asserting against pre-edit
rows. It only ever passed because the model swap won the race; spans made pages
3x bigger, the swap lost, and the check accused working code. It now waits for
the model to be REPLACED.
Cost measured on libcrypto: 95KB per 500-row page, 50ms; model ensure(2000)
228ms. Acceptable for what it buys.
tests: new asm_highlight scenario (+7) — >90% of code rows carry spans, insn/reg/
punct present, every span kind has a style, spans reconstruct the row text
exactly, mnemonic is the first span. 202/0 scenarios, 26/0 blob, 30/0 project UI.
TODO: DisasmView appears to be dead code (never instantiated; Ctx.dis returns
ListingView), which is why this only needed doing once.
|
|
Two bugs in two days had the same shape — an edit rebuilds the model and
whatever ran last wins. The status message got clobbered by the reload's own
status write; the scroll position got recomputed from a row index that no longer
meant the same thing. Both were patched by hand. A third was coming.
ViewAnchor makes it one thing: where you are looking, in ADDRESSES, plus the
message the rebuild must not eat. _anchor() captures it on the UI thread before
the edit; _anchor_rows() resolves it against the rebuilt model; _edit_done()
settles the aftermath. The edit paths (_do_edit_item, _do_make_data) now hand one
object through instead of threading positions and messages separately.
Addresses, not indices, because an edit can change how many rows an item takes:
four undefined byte rows collapse into one instruction row, undefining does the
reverse. An index means a different place afterwards.
_reload_active_code deliberately does NOT use it. Renames and comments don't
change row structure, and the model that path rebuilds is constructed empty —
index_of_ea returns -1 until pages load, so an anchor would resolve to nothing
while costing an extra model build on the UI thread. Wrote that out and reverted
it rather than leave an abstraction applied where it does nothing.
Also fixed in passing: _do_make_data had the same latent bug (no scroll
preservation at all) and now goes through the same path.
The bigger find is in TODO. The scenario suite mutates targets/echo.i64 and
SAVES it, so a scenario that undefines an instruction breaks later runs
permanently — decomp_follow_self had been failing on a polluted database, not on
any code change. It also made an edit-position check look flaky one run in
three, which I nearly wrote up as a race. Coverage for this lives in
test_blob_ui.py instead, which builds a throwaway binary and can mutate freely.
tests: +1 blob UI (commenting leaves the view where it was), alongside the
carve checks. 26/0 blob, 202/0 scenarios (on a fresh .i64), 30/0 project UI.
|
|
Pressing `c` jumped the scroll position. Defining an item reloads the view, and
the reload only carried the cursor's row index — the viewport was recomputed
from scratch, so you landed somewhere else and lost your place mid-carve.
Row indices are the wrong thing to remember across this reload anyway: carving
COLLAPSES rows (four undefined byte rows become one instruction row), so the row
that was at the top is a different address afterwards. The anchor has to be the
top visible ADDRESS, resolved back to a row after the model is rebuilt.
on_edit_item_requested captures it before the edit, _do_edit_item resolves it
against the new model, and _open_at grew a scroll_y so the entry can carry it.
Verified: cursor at 0x4800 with the top of the screen at 0x47da, press `c`, and
both are unchanged afterwards.
tests: +3 blob UI (scrolled far enough to have something to lose, top address
unchanged, cursor address unchanged). 25/0 blob, 202/0 scenarios.
|
|
One instruction per keypress means pressing `c` once per opcode for the length
of a routine, which on a raw image is the whole job. IDA's `c` runs; ours now
does too.
New define_code_run tool: create instructions consecutively and report why it
stopped — 'undecodable' (bytes aren't an instruction), 'flow' (control flow ends
here), 'defined' (ran into existing code/data), 'segment' or 'limit'. It loops
inside the worker; from the client this would be one round trip per instruction,
minutes on a real image.
Stops AT a ret rather than past it: beyond the end of a routine the bytes are
usually padding or data, and running on turns a clean carve into something you
have to undo by hand. Stopping at already-defined items is the same principle —
undefining someone's existing work to keep a speculative run going isn't a trade
the user asked for.
The ret test is ida_idp.is_ret_insn, NOT canonical features: on AArch64
insn.get_canon_feature() returns 0 for RET, so a CF_STOP check silently never
fires and the run walks straight through the end of the function. Verified
against a live IDA before relying on it.
`c` on something already defined now says "already defined @ addr" instead of
claiming the instruction failed to be created — count==0 from a run means two
very different things.
Also: the result message survives the reload. Defining an item rebuilds the view,
and the reload's own cursor handler had the last word, so every edit reported
itself as "ROM @ 0x4040 [listing]". A one-shot _flash is handed to whichever
status write lands first after the edit. (Third time this clobber pattern has
turned up: split view, the no-functions hint, now this.)
Verified: nop/nop/nop/ret at 0x4040 -> "defined 4 instructions (0x4040–0x4050) —
control flow ends here", with 0x4050 left as an undefined byte. Starting on
existing code -> no-op. Random bytes -> stops at the first that won't decode.
tests: +4 blob UI (runs to the end of flow, stops at the ret, doesn't touch the
junk after it, and the status reports it). 22/0 blob, 202/0 scenarios, 30/0
project UI.
|
|
Loading a blob and pressing `c` at 0 disassembles one instruction; everything
after it collapsed into a single row — "db 2044 dup(?)" — with no cursor
position anywhere inside it. There was no way to start a second instruction
stream at an arbitrary offset, which is most of what carving a firmware image
IS. In IDA every undefined byte is its own line and you just put the cursor on
one.
The collapse existed for a real reason (see the comment in the heads tool): a
.bss or a fresh blob would otherwise be millions of one-byte rows, and this
model materialises what it walks. Expanding physically would also make
`g <far address>` walk every byte in between.
So the run stays ONE physical head and PRESENTS as N logical rows. _row_at is a
prefix sum over heads, _phys() maps a row back to (head, byte offset), and the
text for an interior row is synthesised on demand — "db 4Ah", the actual value,
because the byte values are the whole point when you're looking for a stream.
Memory is unchanged (libcrypto: 1 head for its 80-byte .bss, 61MB RSS), and
index_of_ea into the middle of a run is 0.01ms via bisect.
Now: cursor on any byte, `c`, and you get an instruction; the bytes before it
stay individually addressable.
Two bugs found on the way:
* IDAToolError takes (tool, message) and five call sites in domain.py passed one
string. Every one of those error paths raised TypeError INSTEAD of the real
error — "define code @ 0x4020: Failed to create instruction" reached the user
as "IDAToolError.__init__() missing 1 required positional argument". Fixed all
five; the message that finally came through is what identified the next issue.
* Searching now walks one row per undefined byte, so _index_for_search is capped
at 400k lines and says when it truncated, rather than grinding through a
multi-megabyte blob nobody wants to text-search.
tests: test_blob_ui.py +8 — a run presents one row per byte, each is a single
addressable byte showing its value, an interior address resolves to its own row,
`c` on a chosen byte carves there, the carved row spans the instruction, and
neighbouring bytes stay addressable. Uses PLANTED A64 instructions, because
whether random bytes decode is chance and a test that depends on chance is
worthless on the run where it fails. 19/0 blob, 202/0 scenarios, 30/0 project UI.
|
|
Pick a random .bin, say ARM at 0x4000, and you got two empty panes and
"functions still loading…" — which was a lie; loading had finished. _auto_land
falls back to the symbol picker when there's no entry function, and the picker
answers an empty index with that message. Nothing ever opened.
Zero functions is not a corner case. It's exactly what a real firmware image
looks like when it's described wrongly, and IDA has no complaint of its own to
make about it, so this was the last silent-wrong-answer in the blob path.
Now:
* Land in the listing at the start of the image. The bytes exist even when no
code was recognised, so there is always something to show.
* Say so, in the status bar, for as long as it stays true — an image with no
functions is a property of the database, not an event, and writing it once
meant the next status write erased it (the same clobber that bit the split
view's "decompiling…").
* Ctrl+L re-asks. The .i64 has the old processor and base baked in and takes
precedence over any switches, so reloading means deleting it; the confirmation
says what that costs, and when there are no functions it says nothing is lost.
Verified with real keys on a random 64K blob: ARM @ 0x4000 lands showing
"db 65536 dup(?)" with the hint in the status; Ctrl+L -> confirm -> dialog ->
metapc reloads at 0. The hint survives scrolling.
tests: new tests/test_blob_ui.py (11) driving a real random blob end to end —
lands, has rows, right base, honest status, hint survives navigation, Ctrl+L
offers the reload and declining leaves the binary open. 202/0 scenarios, 30/0
project UI.
(Harness note for future me: tmux send-keys reads "0x4000" as a hex KEY CODE and
sends U+4000. Use send-keys -l for literal text.)
|
|
bad answer
Three bugs, reported together, with one shared root: you couldn't get to the
address field, so the address went into the processor filter, so IDA got a
nonsense processor name and refused to open — and the app dead-ended with a
misleading error.
**Tab never reached any modal.** Binding("tab,shift+tab", "toggle_view",
priority=True) is an APP binding, and priority bindings run before the focus
chain. Nothing in any dialog in this app could ever be tabbed to; the load
dialog is just where it finally mattered. action_toggle_view now hands the key
back when a modal is up, which fixes it everywhere.
**...and DOM order was the wrong tab order anyway.** focus_next() stopped at the
processor list, which is arrow-driven and has nothing to type. LoadOptionsScreen
overrides it to cycle the two fields you actually type into.
**...and the dialog outgrew the terminal.** With the palette's default
max-height the 21-row list pushed the address field and help line off the bottom
of the screen. Nothing errors — the field simply isn't there, which reads as
"Tab does nothing". Capped per-dialog.
**Project mode never asked.** _should_ask_load_options bailed on
`self._project is not None` with the comment "project mode carries per-binary
options already" — true only if someone had already filled them in. A raw blob
added to a project got the silent x86-at-0 treatment the dialog exists to
prevent. Now asked at boot AND on switching to an undescribed binary, and the
answer is written back to the project entry (Project.set_load), so it is asked
once per binary, not once per run.
**A rejected answer dead-ended.** Getting a processor wrong is an ordinary
mistake; it left an empty app with "connect failed: worker exited (code 1)" and
a message blaming a locked .i64. The worker now names the real suspect when load
switches were in play, and the app re-opens the dialog instead of giving up.
Verified with real keys in a tmux pane, which is the only way any of this shows
up: Tab -> address field -> 0x8000000 -> Enter -> 35 functions at 0x80039AC; a
bogus processor -> "those load options were rejected — try again" with the
dialog back; project mode -> asks, loads at the right base, and the answer is in
the project file.
tests: +3 scenarios (Tab moves focus under a modal, lands on the address field,
cycles back). 202/0 scenarios, 39/0 project, 32/0 formats, 30/0 project UI.
docs/TEXTUAL_NOTES.md gets the priority-binding and clipped-modal traps.
|
|
The curated list was written from the procs/ directory listing. Two of the
twenty names were wrong, and wrong here is not a soft failure: IDA REFUSES to
open the database (rc=4) with nothing useful said. The load dialog would have
handed people a dead end from inside the UI that exists to rescue them — the
same silent-failure class the dialog was built to kill.
h8 -> h8300 (h8.so is the module FILENAME, not a processor name)
sparc -> sparcb / sparcl (and SPARC has endianness variants, like MIPS/PPC)
Also probed the aliases people reach for first: arm64, aarch64, mips, m68k are
all invalid. 'arm' covers AArch64 (verified: an AArch64 blob analyses to 35
functions under -parm), so those names now live in the human labels, where the
filter still finds them — typing "arm64" finds ARM, "m68k" finds 68k, "mips"
finds both endiannesses.
tools/verify_procs.py does the check: open a scratch blob with -p<name>, read
back inf_get_procname(), compare. Fresh temp dir per name, because once a
database exists IDA ignores the load switches and every name after the first
would "pass". 21/21 verified.
tests: +11 formats, including the verified-set guard (adding a processor without
re-running the script fails on purpose) and checks that the rejected aliases are
NOT offered but ARE still findable by typing them. 32/0 formats, 199/0 scenarios.
|
|
Last commit let you SAY how to load a blob. This one notices when you should
have. Interactive IDA pops a dialog when no loader matches; we silently loaded
as x86 at 0 and analysed to nothing, so the flag only helped people who already
knew they needed it — which is exactly the people who don't need help.
formats.sniff() recognises the formats IDA definitely handles (ELF, PE, Mach-O,
dex, wasm, ar, COFF, Intel HEX, S-records). Anything else gets
LoadOptionsScreen: a filterable processor list with human labels, a load-address
field, Enter to accept, Esc to load it the way IDA would have anyway.
Deliberate asymmetry: the sniff only claims formats it is sure about. A false
"unknown" costs one dismissible dialog; a false "known" is the silent wrong
answer this exists to kill. Esc is always an escape hatch.
The list offers 20 processors, not IDA's 73 — most of the rest are museum
pieces, and a name typed into the filter that matches nothing is taken literally
so nothing is actually unreachable. Endianness is spelled out (arm vs armb)
because getting it backwards is the most common route to zero functions.
Asked only when nobody has answered yet: not with --processor, not in project
mode (entries carry their own), and not when a database exists — the .i64
already records how the image was loaded.
Textual trap worth recording: the screen stored the file size in self._size,
which is Widget's own backing field for outer_size. Assigning an int to it
crashes layout with "'int' object has no attribute 'region'" from deep inside
_set_dirty, nowhere near the cause. Same family as the _render collision.
The paragraph conversion now lives in exactly one place (formats.load_args);
BinaryRef and launch both call it.
Verified on a real AArch64 blob: dialog appears, filtering to "arm" leaves two
entries, base 0x8000000 accepted -> "-parm -b800000" -> 35 functions at
0x8002440. An ELF never asks, and neither does a blob that already has a .i64.
tests: new tests/test_formats.py (21) and a load_options scenario asserting the
dialog stays out of the way for a recognised binary. 199/0 scenarios, 39/0
project, 30/0 project UI, 36/0 index, 27/0 pool, 21/0 formats.
|
|
A raw firmware dump has no format to detect, so IDA fell back to x86 at address
0. It doesn't fail — it opens, analyses, and finds nothing. An AArch64 image
loaded this way gave 0 functions; told the truth it gives 35.
ida-tui fw.bin --processor arm --base 0x8000000
and per binary in a project, which is what a multi-image firmware actually
needs:
{"path": "app.bin", "processor": "arm", "base": "0x8000000"}
idapro.open_database() already accepted IDA command-line switches; nothing was
passing any. Plumbed BinaryRef -> WorkerPool -> WorkerClient -> worker argv, plus
a load_args for the single-binary path that has no project ref.
base is written the way people say it (0x8000000, int or string, any base).
IDA's -b is in PARAGRAPHS — -b1000 loads at 0x10000 — so BinaryRef.load_args
converts, and a base that isn't 16-byte aligned is refused rather than silently
landing 16x off. ida_args passes anything else through.
Two bugs found by testing the whole path rather than the happy one:
* Project.load() whitelisted path/label when normalising entries, so the load
options were dropped the first time a project was reopened — set a processor,
come back tomorrow, it's gone.
* Re-passing the switches to an EXISTING database makes IDA refuse the open
(rc != 0, no functions). The .i64 already records how the image was loaded, so
the worker skips them once a database exists. My first guard checked
splitext(path) + ".i64" and never fired, because IDA names it "<file>.i64" —
keeping the extension. It checks both spellings now.
Verified on a real AArch64 blob: fresh load 35 functions based at 0x8002440,
reopen 35 again, and the CLI rejects an unaligned or non-numeric --base.
tests: +6 project (options recorded, paragraph conversion, file round-trip,
add() takes them, an ELF passes nothing, hex-string base). 39/0 project, 195/0
scenarios, 30/0 project UI, 36/0 index, 27/0 pool.
|
|
xrefs_to only ever sees the current database, so an exported function looks
unused from the inside even when the rest of the project calls it. The import
side of the phase-3 linkage index already knew better; nothing surfaced it.
_foreign_importers appends those callers to the xrefs dialog. From libc's
strrchr, with echo in the project:
0000C318 import [echo] strrchr
Read from the on-disk index, so a caller appears whether or not its worker is
resident. Choosing one carries a (binary, addr) payload instead of a bare
address; _on_xref_chosen routes that through _switch_then_goto — the same path a
project search hit takes — so it records a hop and Esc comes back.
Only fires for a symbol this binary actually EXPORTS. A local name that happens
to collide with some other binary's import is not a caller of ours, and without
that check every common name (main, read, error) would sprout fictional callers.
Names are compared after link_name(), so ELF versioning doesn't hide the match.
Verified on a real echo+libc project: the dialog lists echo's call site, and
selecting it switches to echo, lands on 0xc318 and leaves hops=['libc.so.6'].
tests: +3 project UI — a symbol we don't export gets no cross-binary callers,
the (binary, addr) payload jumps to the other binary, and it records the hop.
195/0 scenarios, 30/0 project UI.
|
|
Three things, all following from phase 3 making cross-binary jumps ordinary.
**A cross-binary jump was a one-way door.** Nav history is per-binary, so
arriving in another binary — a project search hit, or now following an import
into the library that implements it — landed you in an empty history with nothing
to take you back. _switch_then_goto records the binary it came FROM, and
action_back falls through to that hop once local history is spent: Esc walks back
through the function you were in, then the binary you were in. Manual Ctrl+O
switching records nothing, because that isn't navigation.
**Pre-warm follows the linkage graph, not list order.** _prewarm_provider warms
the binary providing the most of this one's imports — where a follow is most
likely to go, so its startup is paid before you ask for it. "Next in the list"
would have been arbitrary; phase 3 gave us something better to ask.
pool.prewarm() refuses rather than making room. Evicting a binary the user
visited to speculatively load one they haven't is a straight downgrade, and it
throws away that binary's caches as well; at a tight budget pre-warm just does
nothing. The cost of a worker that doesn't exist yet can only be estimated, so it
uses the largest resident one (same program, different database) — and if that
estimate proves wrong, the speculative worker is the one evicted, never a chosen
one.
**Driving a project.** pane spawn --project FILE [--open BIN]; `binaries` lists
the inventory (active / resident / indexed / where Esc returns to) and `switch
{binary,addr?}` makes another active — with an address it takes the search-hit
path, so it records a hop. state gains `binary` and `hops`, which it should have
had the moment project mode existed.
Verified on real sessions: drive binaries/switch against an echo+cat project
pane; Esc crossing back from a switch; and prewarm on echo+libc picking libc
(provider of echo's imports) and warming it after an evict.
tests: +5 pool (prewarm warms, no-ops when resident, refuses at budget, evicts
nothing when refusing, ignores unknown labels) and +4 project UI (jump records
the hop, Esc crosses back, hop consumed). Confirmed the Esc-back checks fail with
the branch removed. 195/0 scenarios, 27/0 project UI, 36/0 index, 27/0 pool,
33/0 project.
Left open: project-level persistence across sessions.
|
|
Following a call to strcmp reached the PLT/extern entry and stopped there —
Hex-Rays has nothing to decompile, because the code lives in a library this
binary only references. With the other binary open in the same project we
already had everything needed to cross that gap; we just weren't indexing it.
Index each binary's imports and exports (KIND_IMPORT / KIND_EXPORT) alongside its
functions and strings. On a follow, _import_stub asks whether the target address
is one of this binary's import stubs; if so _cross_binary_impl asks the index who
exports that name, and we switch there instead of landing on the thunk.
Verified end to end on a real echo + libc project: Enter on `strrchr(a1, 47)` in
echo's pseudocode switches to libc.so.6 and lands on strrchr at 0xaf960.
Three things it turns on:
* ELF symbol versioning. The importer sees strrchr@@GLIBC_2.2.5 while the
provider may export any of three spellings, so raw names resolve almost
nothing. domain.link_name() cuts at the first '@'; Linkage.raw keeps what IDA
reported, which is what the listing shows.
* Exact match, not substring — ProjectIndex.exact(), so `read` doesn't bind to
pread/read_line/thread_start. It also answers below the 3-char trigram floor,
and plenty of real exports are that short.
* Resolution reads the on-disk index, so a provider resolves while its worker is
evicted. That's what the index was for.
When nothing in the project provides the symbol _follow_import declines and the
normal navigation runs: landing on the stub is still the honest answer, and a
single-binary session is unchanged. The PLT-stub PRESENTATION item stays open —
an unprovided import should say "imported, provider not in project" rather than
show a decompiler error.
server/patch_server.py gains list_linkage (idautils.Entries + enum_import_names);
a worker without it degrades to no linkage rather than failing.
tests: index join +8 (exact vs substring, short names, exclude-self, reverse
join, kind isolation, forget unresolves) and link_name +4. 36/0 index, 195/0
scenarios, 23/0 project UI, 33/0 project, 22/0 pool.
|
|
The app carried three overlapping ideas of the current pane: _active, _pref, and
Textual focus. 190e28b tied focus to _active in split; this removes _pref, which
turns out never to have been a variable at all.
_pref was assigned "listing" in __init__ and "listing" on a project binary
switch. Nothing else ever wrote it. But _code_view() branched on it:
return self.query_one(DecompView if self._pref == "decomp" else ListingView)
so it always returned the listing, whatever you were reading. Its two callers put
focus back after the goto prompt closes — so cancelling `g` while in the
pseudocode focused the HIDDEN listing, and the pane you were looking at stopped
answering the keyboard. Arrows did nothing until you clicked. (It also explains
why routing follow through _code_view() earlier made Enter a dead key: the helper
had been quietly lying the whole time.)
_code_view() now returns the active code pane. _pref is gone from the app,
BinaryState and the RPC snapshot keeps "pref" for wire compat, sourced from
_code_mode() — the one place that answers "which code view do we return to from
hex", and a constant by design in the unified layout.
Verified on a live pane both ways: g then Esc in pseudocode, then two Downs.
Fixed, the cursor moves 0 -> 2; with the old lookup restored it sits at 0 with
focus=ListingView. Same check added to the view_toggle scenario, and it fails
without the fix. 195/0, project UI 23/23.
|
|
Walking back with Esc ended in a dead keypress: nav_depth dropped but the screen
didn't change. Measured on a live pane, four Escs after one follow:
Esc #1 -> sub_3500 L7 (where we followed from) good
Esc #2 -> main L54 decomp (where we followed from) good
Esc #3 -> main L344 listing good
Esc #4 -> main L344 listing -- nav 2 -> 1, nothing moved
The app auto-lands on main at startup, and opening main again from Ctrl+N
appended a second, identical entry. Every "navigate to where you already are"
did this; the extra Esc it bought is invisible except that it does nothing, which
is exactly what "back is broken" feels like from the keyboard.
Both push sites now go through _push_nav, which replaces the top entry instead of
appending when the target is the same (ea, view, line) — the newer entry's
metadata still wins. _same_spot compares dec_cursor for pseudocode and cursor for
the listing.
Note what is NOT a duplicate: the first follow after Tab-to-pseudocode still
pushes twice (nav 1 -> 3). That's deliberate — Tab leaves _cur transient and off
the stack, so the follow records the pseudocode position first, which is why Esc
#1 above returns to main L54 rather than dropping you into the listing. Verified
that subsequent follows push exactly one each.
tests: the palette scenario re-opens the function it is already on and asserts
nav depth is unchanged. Fails without the fix (nav 2 -> 3). 194/0.
|
|
Ctrl+N found nothing on libcrypto. Typing "PEM_read_bio" returned 0 of 10093
functions while the backend resolved the very same name to 0x1d6290.
_fuzzy lowercased the NAME but not the QUERY, then walked the query's characters
through the lowered name. One capital letter and the subsequence walk fails at
the first character, so the match is not merely worse — it is None, and the
palette shows nothing at all.
Invisible on the test binary because C symbols there are lowercase (main, strlen,
error) and every existing palette check typed a lowercase query. Fatal on any
library that capitalises: OpenSSL, most SDKs, Windows binaries. The paging index
was the obvious suspect and was innocent — all_loaded() had all 10093.
Verified on a live libcrypto pane: "PEM_read_bio" now returns 34 hits, exact
match ranked first.
tests: the palette scenario now types "MAIN" and expects "main". Confirmed it
fails without the fix (results=[]) and passes with it. 193/0.
|
|
highlight.py was still carrying VS Code Dark+ (#c586c0 keywords, #4ec9b0 types,
#dcdcaa identifiers) — an entirely different colour system from the listing,
which moved to a measured palette in d1cfab2. Reading a function in split view
meant reading two unrelated themes side by side, with the same thing coloured
differently in each pane.
Same palette in both panes now, so one hue means one thing everywhere: a string
is #9ece6a and a symbol #7aa2f7 whether you're in disassembly or pseudocode.
Contrast against the app background #12161c, signal 7:1+ and structure 3-4:1:
keyword (control) #e8ecf2 15.3 operator/body #c3cad3 11.0
string #9ece6a 9.9 number #d8a657 8.2
type/builtin #93aee0 8.1 name #7aa2f7 7.2
comment #7c8b9e 5.2 punctuation #626c7a 3.4
Control keywords take the brightest NEUTRAL rather than a hue, mirroring the
mnemonic column: they're the skeleton you scan for, and a hue there would claim a
meaning the rest of the palette already assigns. Punctuation drops from grey70 to
3.4:1 — it was competing with the code it delimits.
Suite 192/0.
|
|
Travelling the history stack in split mode showed no feedback at all, so Esc
still read as a no-op whenever the target needed decompiling — the exact thing
cc44f5c was supposed to fix. That commit set the message in action_back, but the
split branch of _show_active ends with _status_for_cur("split"), which
unconditionally wrote the idle status straight over it. The spinner overlay was
already being raised correctly; only the words were lost.
The split branch now keeps the in-flight message while it has just kicked off a
decompile, and on_descendant_focus (added in 190e28b, and firing exactly when
focus lands on the restored pane) no longer restatuses while dec.loading.
Caveat on verification: I could not observe this on targets/echo. Going back
always lands on a function the decompiler has already cached, so the busy branch
completes without a visible frame — sampling the status bar every 60ms across the
whole Esc showed no intermediate state. What's verified here is that the busy
branch sets the message and that nothing downstream clobbers it; the visible
effect needs a target slow enough to decompile, i.e. a real binary.
Suite 192/0.
|
|
Reported as "Esc in the decompiler needs two presses". It was never Esc.
In split mode Tab sets _active AND focus together, but focus also moves on its
own — a click, or a pane focusing itself after a load — and _active was left
naming the pane you are NOT in. Everything downstream trusts _active, so with the
pseudocode on screen and the listing still "active", Enter resolved the word
under the LISTING's cursor: it followed something unrelated (typically the outer
function itself, landing on line 0 col 19 — its own name in the prototype) and
pushed history for it. The first Esc was then spent undoing that bogus entry and
looked like a no-op; the second did what you meant.
on_descendant_focus now syncs _active to the focused code view while split, the
same pair of steps Tab already does (set _active, re-link the band, restatus).
Verified on a live pane driven through tmux, which is the only harness in this
session that told the truth — the in-process pilot never exercised follow-then-Esc
in split, which is why four earlier attempts "passed" while the bug sat there:
before: Enter on sub_3500 -> fn main, L0 (never left the function)
after: Enter on sub_3500 -> fn sub_3500, L0 then Esc -> main L54, one press
Suite 192/0.
Known remainder, deliberately not fixed here: a follow still pushes two entries
(nav 2 -> 4), so the stack is deeper than it should be. Landing is correct at
every step, so it isn't visible as the reported bug. The obvious dedupe in
_open_decomp_entry (skip appending src when the top entry is the same place) is
inert — the duplicate comes from somewhere else, and I'd rather leave it open
than commit a fix I can't show working.
|
|
a terminal"
This reverts commit baf599e143dbf23f33886ba03a60c5d9f3c19078.
|
|
Esc-to-go-back needed two presses. It looked like the decomp pane was swallowing
the first one, but the RPC state dump before/after a single Esc settles it:
nav_depth 3 -> 3 action_back never ran
cursor.col 19 -> 20 +1 column, i.e. `right`
modal null nothing was up to eat the key
The keypress arrived as a right-arrow. \x1b is both Escape and the lead byte of
every arrow/function-key sequence (ESC [ C is right), so a terminal or tmux that
merges a lone Esc with what follows delivers something else entirely. Nothing in
the nav path touches cursor_x and there is no widget-level escape binding on the
code views, so this was never application logic — I spent two commits looking in
the wrong place before asking for the dump.
Bind back to "escape,backspace". Backspace is unambiguous, so going back never
depends on a byte the terminal can reinterpret; Esc still works wherever it's
delivered intact. tmux users generally want `set -sg escape-time 10` regardless.
docs/TEXTUAL_NOTES.md records the symptom and, more usefully, the triage: check
nav_depth in the state dump first — if the action never ran, it's the input
layer, not the logic.
Verified: backspace in the pseudocode pane takes nav 3 -> 2 and reloads the
previous function. Suite 192/0.
|
|
Going back to a function that isn't currently decompiled has to decompile it
first, and until that lands both panes still show where you were — so Esc read as
a no-op and people press it again.
action_back now sets a status the moment it pops ("◂ back to <name>…"), which
_show_active then upgrades to "<name> — decompiling…" with the spinner overlay
up. Verified: immediately after Esc, with no pause at all, the status says
decompiling and dec.loading is True; it settles on the target a moment later.
Suite 192/0.
|
|
Esc in the pseudocode view appeared to jump back and then not update until you
pressed something else. It wasn't a repaint problem (my previous commit guessed
that and fixed a different, real, but unrelated staleness): the Esc WAS being
undone.
_do_navigate decompiles on a worker thread and hands the result to
_open_decomp_entry, which applied it unconditionally. Decompiling a large
function takes a moment, so pressing Esc while it's in flight meant the late
result landed afterwards and re-applied the forward navigation — putting you back
where you'd just left. The next keypress then made things look "correct" again,
which is exactly the reported symptom.
Add a navigation sequence number: _do_navigate captures it when it starts,
action_back bumps it, and _open_decomp_entry drops a result whose sequence no
longer matches.
Verified both ways on a slow target (sub_3720): Esc pressed 20ms into the
navigation now stays back at main:60, and removing just the guard reproduces the
bug exactly (cursor=0, loaded=0x3720 — the function being left). Suite 192/0,
project UI 23/23.
|
|
Esc in the pseudocode view navigated correctly but the pane kept showing the old
cursor line until you pressed something else. The listing view was fine because
its path goes through load(), which relayouts and repaints.
DecompView.goto() sets `cursor`, which is reactive(repaint=False), and then
leaves redrawing to _apply_scroll — but that only repaints via
call_after_refresh, which needs a refresh to actually happen. Jump somewhere
inside the viewport already on screen (exactly what Esc-back within a function
does) and the scroll doesn't change, so nothing asked the widget to redraw: the
old highlight stayed and the status line had already updated, which is why it
looked like the jump half-worked. _move() doesn't hit this because it repaints
the old and new rows via _refresh_lines.
goto() now refreshes explicitly.
Verification note: a render_line-level probe CANNOT see this bug — render_line
reads self.cursor live, so it always reports the new position. The staleness is
in Textual's compositor cache, which only a repaint request clears. Verified by
spying on the widget's refresh instead: a same-viewport goto now requests one
(it requested none before). Suite 192/0.
|
|
`--project fw.json a.elf b.elf` where those are already listed appended them
again, so the project grew a second copy on every launch — each duplicate then
staged its own file, opened its own worker and got its own index entries.
Dedupe on the RESOLVED SOURCE PATH, which is the only identity that's actually
correct here: two different foo.elf from different directories are different
binaries and must both be accepted (the label disambiguator already gives them
foo.elf and foo.elf_2), while ./a.elf, /abs/a.elf and a symlink to it are all the
same file and must collapse to one entry.
* Project.by_source(path) — lookup by realpath.
* Project.add() returns the existing entry instead of appending a duplicate.
* Project.create() drops repeats on one command line too.
* launch.py reports what it did: "added N binary(ies)" / "N already in the
project (matched by path) — left alone", and only rewrites the file when
something actually changed.
Not deduped in _build_refs on load: remove() maps refs to entries by index, so
collapsing there would desync them, and a hand-edited duplicate still works
(labels disambiguate).
tests/test_project.py +8 checks: re-add is a no-op, so are a relative spelling,
a messy ../ path and a symlink; a same-named file from another directory IS
added and gets a distinct label; create() drops repeats. 33/33. Verified on the
real CLI: re-running the reported command leaves the project at 3 entries.
|
|
The old palette had the hierarchy INVERTED. Measured against the background:
address grey58 = 6.0:1 and opcode bytes grey50 = 4.6:1, versus mnemonic cyan
3.8:1 and label yellow 4.3:1 — the structural noise you never read was louder
than the mnemonics and names that are the entire point. The pseudocode gutter
(grey37, 2.8:1) sat below the 3:1 accessibility floor. Half the colours were
terminal-palette names (cyan/yellow/white/greyNN), so the scheme rendered
differently on every user's terminal and could never match our hex theme. And
amber meant five different things (label, function header, data, member, match).
Principles now applied:
* frequency is inverse to intensity — mnemonics are on every line, so they get
the brightest NEUTRAL rather than a hue; colour is spent on what you scan for
* one hue, one meaning — amber is attention/match and nothing else
* structure recedes: addresses/opcodes/gutters 3.2-3.9:1, never above body
* all hex, no terminal-palette names
* foreground encodes kind, background encodes state (match/cursor/link)
* never hue alone — bold/italic carries the same distinction for red-green
colour-blind users
Result, all measured vs #12161c:
mnemonic #e8ecf2 15.3 body #c3cad3 11.0 strings #9ece6a 9.9
data/num #d8a657 8.2 member #93aee0 8.1 names #7aa2f7 7.2
comment #7c8b9e 5.2 address #6b7684 3.9 gutter #626c7a 3.4
opcodes #5e6875 3.2
Signal band 7.2-15.3:1, structure band 3.2-3.9:1 — separated, nothing below 3:1,
and the old 2.8:1 gutter failure is fixed.
Verified by rendering a real listing: opcode bytes dimmest, addresses dim,
mnemonics brightest, names in blue. Suite 192/0.
Not touched: highlight.py maps Pygments tokens for the pseudocode view and still
uses its own colours — worth bringing into the same scheme next.
|
|
Nobody picked that orange — it's #ffa62b from Textual's default textual-dark,
which sets accent AND warning to the same value, so every border, title bar and
the quit dialog were all the same neon. Meanwhile the code views already use a
coherent muted palette (amber #b58900 for matches and the spinner, #6a9955 green
for comments, #264f78 blue for the word highlight), so the chrome was the only
thing shouting.
Register an "idatui" theme in the same family: desaturated blue-greys for
background/surface/panel, amber (#b58900 — the exact match highlight colour) as
the accent, a burnt orange (#c9762f) for warning so it reads as care rather than
as another accent, blue (#5aa0d6) reserved for focus, and the reds/greens we
already used for failure and comments.
Title bars were color:$text on the accent, which was passable on bright orange
and muddy on amber; they're now dark-on-amber and bold, which reads as
intentional.
Suite 192/0, project UI 23/23.
|
|
Five sections stacked vertically ran ~45 rows, so F1 scrolled on any terminal
under ~50 lines. Each section is now its own bordered card (title in the border),
and the cards flow into as many columns as the width allows.
Textual CSS has no media queries, so the column count is computed in compose from
the real app width. Sizing off the WIDEST section would let one long row (Move's
"Ctrl+Home / Ctrl+End") inflate every column and cost a column that would
otherwise fit, so _columns() measures the actual layout instead: chunk the
sections, sum the per-chunk maxima, take the most columns that fit. Keys are
right-aligned per card rather than globally, so a card of short keys stays narrow.
200x50 -> 3 cols 140x44 -> 2 cols 100x36 -> 1 col (scrolls)
160x44 -> 3 cols 120x40 -> 2 cols
The scroll container stays, so a very small terminal degrades to scrolling
instead of clipping — but at any normal size nothing scrolls. The box hugs its
content (width:auto all the way down; VerticalScroll needed width:auto too or it
filled to max-width) and stays centred.
Tried a real CSS Grid first: it collapsed to zero height at narrow widths, and
grid sizing from on_mount read a stale app width. Explicit columns are
predictable. Also shortened one description that was pushing the widest card out.
help scenario now asserts the five cards exist and that the content fits without
a scrollbar. Suite 192/0.
|
|
Completes phase 2 — the index already carried strings (KIND_STRING), the palette
just didn't offer the toggle. StringsPalette now mirrors SymbolPalette: F2 flips
between this binary and the project, project rows are prefixed with their binary,
and choosing a literal in another binary switches to it and jumps.
Ranking matches the symbol side: the trigram index guarantees the match, so
ordering is earliest-match then shortest, with a (binary, addr) tiebreak — the
same tie that crashed symbol search when two binaries shared a name, avoided here
by construction. Project scope caps at 60 like symbols.
_results is now (binary, addr, text) tuples in both palettes, so _on_string_chosen
takes the same (binary, addr) choice and routes through _switch_then_goto.
test_project_ui.py: local scope stays single-binary, F2 spans >=2 binaries
(23 checks). Suite 191/0.
|
|
It always highlighted row 0, so switching away and back meant hunting for your
current position in the list. Preselect the entry marked active instead; if a
filter excludes it, fall back to the first row as before.
Locked in test_project_ui.py (21 checks).
|
|
Reported: arrowing through project-scope results is sluggish, and switching back
to this-binary scope is instantly fast again.
Everything I can measure here says the palette is fine, so this is a
hypothesis-driven mitigation, not a proven fix. Ruled out by measurement:
* per-option render + highlight move: 0.80 ms (local) vs 0.82 ms (project) at
200 options each — identical and fast
* wrapping from the binary-name prefix: none, 1.00 lines/option in both
* option count driving render cost: flat from 20 to 400 options
* an app-level OptionHighlighted handler doing work: none exists
* scroll animation: Textual's scroll_to_highlight already passes
animate=False, immediate=True
The synthetic pilot harness has a ~62 ms floor per press (Textual awaits a full
refresh cycle), so end-to-end key timing there is meaningless.
What remains plausible is simply LIST LENGTH in a real terminal: project scope
saturates its cap because every binary contributes, while a local filter usually
returns a handful — which matches the symptom exactly. So project scope now caps
at 60 rather than 200 (and fetches 3x that from the index). 60 is plenty for a
search palette; typing narrows further.
|
