| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Three separate wastes, all of the same family: waiting on a guess instead of a
signal, and paying for work that never had to be repeated.
1. The 64KB blob was built with os.urandom into a fresh TemporaryDirectory on
every run. New bytes at a new path means the pristine-database cache can
never apply, so full auto-analysis of 64KB of AArch64-decoded noise was paid
every single run. It is now built from a seeded PRNG at a stable path
(tests/.synthetic/, gitignored) and staged through the existing cache.
Determinism is also a correctness fix: whether 64KB of chance bytes contains
something IDA reads as a function is luck, and this suite asserts "and really
has no functions".
2. `wait(lambda: lst.model is not old, ..., 30)` after commenting. The perf work
made an item edit KEEP the listing's walk and re-render in place, so the
model object is never replaced and this waited out its full 30s timeout on
every run -- and then "commenting leaves the view where it was" passed
vacuously, because nothing had happened at all. A test that burns 30s to
check nothing is worse than no test.
3. Two `pause(2.0)`/`pause(2.5)` after a carve, replaced with settle() on a real
condition. The second one deliberately has NO predicate: that spot is random
data, so the carve may legitimately produce nothing, and "the row became
code" would never hold -- gating on it cost another 30s timeout. What that
check is about is the VIEW not moving, so the gate is "the app finished
reacting".
Fixing (1) exposed a real bug in the client, fixed here too: reopening a
database that already exists while passing loader switches is FATAL in IDA --
FATAL ERROR: Switch '-b400' can be used only when loading a new file
which kills the worker before it can report anything. Loader switches describe
an IMPORT and are recorded in the database they produce, so they are now sent
only when there is an import to describe. This was never reachable from the old
suite (a fresh random blob never had a database to reopen), but it is reachable
by any user who opens a raw blob with --ida-args twice.
30 passed, 0 failed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Benchmarking the port against master op-by-op (rather than only asking whether
tests pass) turned up three real bugs, all in the most user-visible path:
opening pseudocode.
1. decompile was doing decomp_map's job. It called the full per-column line
map purely to fill in each line's /*0xEA*/ anchor. The tool ida-tui was
written against takes ONE get_line_item at column 0 per line; the port took
one per COLUMN, i.e. thousands of get_line_item+dstr() calls per function
instead of one per line. Every pseudocode open cost the same as opening the
split view. Carried the real implementation over: 1888ms -> 53ms.
2. decomp_map used the pre-optimisation line map. Ours memoises obj_id -> ea
for the whole function (commit 853d90c: dstr() was 79% of the tool, and
consecutive columns report the same ctree item), the port's did not.
1925ms -> 287ms.
3. _idatui_compact imported ida_pro_mcp on every call. Under Code Mode that
package is not installed in the database process, so the import failed every
time -- and a FAILED import is never cached, so each one re-searched the
whole of sys.path: 422 failed imports per pc_nums call, which was most of
its runtime. 1428ms -> 257ms.
The same bug was a correctness bug hiding behind the perf bug: the fallback
path collapsed whitespace INSIDE string literals, where the real function
preserves it. Pseudocode columns are served in those coordinates, so on any
line containing a string with two spaces, every literal's mark and every
reformat would have been placed on the wrong column. It never fired on
master because ida_pro_mcp is installed there. Now calls the byte-identical
module-level shim directly, with the deviation from the extracted original
documented in place.
Narrow verification: decomp/split_view/opfmt/follow/comment/structs scenarios,
72 passed, 0 failed. Full gate running separately.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Found by tests/test_rawimage_rpc.py, which the earlier runs had not covered:
every rename_many check failed with
{"ok": 0, "failed": 2, "errors": [{"addr": null,
"error": "list indices must be integers or slices, not str"}]}
The port's rename read each category as a single edit (edit["addr"]), but the
batch shape is {func: [{addr,name}, ...], data: [...], local/stack: [...]} --
a list per category, with a single dict accepted as shorthand. Indexing the
list with "addr" raised, and because the whole category was one try block the
error came back attached to addr=null, naming nothing.
That is the entire point of the rename_many RPC verb: a firmware image arrives
with hundreds of names from a loader map or an emulator's symbols.json, and
applying them one at a time costs a navigation plus two prompt round trips
each. Only the single-rename UI path worked.
Now mirrors the real tool: one row per EDIT (addr/old/name plus a per-row
error), a summary counting edits rather than categories, conflict detection
before the write, and dry_run/allow_overwrite/stop_on_error. Renaming a
function refreshes Hex-Rays' ctext, whose cache is per function and persisted
in the .i64 -- without it the pseudocode keeps calling the old name forever
while every other readback reports the new one. Clearing a label with an empty
new name is kept as a real request (the scenarios revert with it) rather than
being rejected as a missing argument.
tests/test_rawimage_rpc.py: 14 passed/7 failed -> 21 passed, 0 failed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
them
This closes the five operations the port was missing and restores the listing's
own tooling instead of a re-implementation of it.
idatui/remote_tools.py is the port's IDAPython island: `heads` (the continuous
listing) and `op_format`/`pc_nums`/`pc_num_format` (`o`/`O`), extracted verbatim
from the BODY that server/patch_server.py used to inject. They are real, diffable
source shipped to the database process as text, not string literals, because this
is the most performance-tuned and behaviour-sensitive code in the project.
Why carry `heads` over rather than keep the port's version: the port's rewrite
emitted no per-operand extents ("ops"), so no keypress could show which literal
it would reformat (opfmt_highlight had no two-operand row to find); it had no
digest/`expect` support, so every page was re-sent after any edit; and its span
walk was the per-character loop ours had already been rewritten out of. It also
dropped struct-member expansion sizing and the func banner/label rows' exact
shapes.
The library is installed ONCE per database process (sys.modules, keyed by a hash
of the source) and then called by name. Code Mode's execute_python builds a fresh
namespace per call, so a library exec'd inline is rebuilt every time and its
module-level caches thrown away -- the per-line render lru_cache in particular,
which the perf work sized to 65536 entries. Installing it once took `heads`
count=200 from 181ms to 92ms; the cache reports 211 hits on a second call where
it previously reported none. (Extraction footgun recorded: ast FunctionDef.lineno
points at `def`, not at the decorators, so a naive slice silently drops
@lru_cache.)
Also ported: flowchart, survey_binary, and the xref contract.
Live pilot suite on targets/echo: 301 passed, 0 failed -- identical to master.
Known, quantified, and NOT fixed here: Code Mode's transport is much slower than
the unix-socket worker for the listing's paging. heads count=200 is 2.6ms on
master vs 92ms here, count=500 is 6.3ms vs 214ms. Roughly half of that is
to_jsonable + HTTP framing per call and is inherent to the architecture; the
empty round trip alone is 2ms. The digest/`expect` path (unchanged pages) is the
main mitigation and is restored.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The pseudocode follow's address fallback broke: following a call landed on the
NEXT LINE instead of the callee (decomp_nav's stale-name check, cur=0x20dd
want=0x2060). The port's xref_query returned rows in raw IDA order, and at a
call site IDA yields the ordinary-flow xref (fl_F, the next instruction) before
the call xref (fl_CN), so 'first code xref' picked the fall-through.
The tool ida-tui was written against sorts rows by the far-end address and
dedups by default; sorted, 0x2060 precedes 0x210e and the follow is correct.
That ordering is load-bearing, so it is now part of the port rather than an
accident of the old implementation. Also fixed: the port attached 'fn' to
ref.from_ea for both directions, where a from-xref must describe its TARGET
(the xref dialog shows the wrong function otherwise), and the envelope was
missing direction/addr/total/next_offset/resolved_addr.
xref_types (ours, the kind badges in the xref dialog) is ported verbatim and
deliberately stays UNsorted -- that dialog lists xrefs in IDA's own order.
decomp_nav, follow_xrefs, xref_labels, decomp_follow_self: 15 passed, 0 failed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Verified live now: ida-codemode 0.3.1 spawns a managed idalib worker on this
box, so the pilot suite runs against the port.
flowchart: the port simply does not have the operation, so domain.get_flowchart
returned None and every graph key reported 'no control-flow graph for this
function'. Ported ours onto ida_gdl (ida-domain exposes no basic-block or
edge-kind surface). Blocks stay address RANGES, never text -- that is what lets
graph boxes reuse the listing's own rows. Graph suite: 0 -> 50 passed.
set_comments: the port set only the disassembly comment via
db.comments.set_at(), so a comment never appeared in the pseudocode. A Hex-Rays
comment is anchored to a ctree location and an anchor the ctree does not own is
discarded as an orphan, so the itp slot must be searched until one sticks, and
the entry ea is a function comment instead. Ported that logic back.
survey_binary: added as the (caught) fallback domain.py expects behind
file_regions, so the fallback path is real rather than always empty.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
signature
ida-codemode is now cloned at ../ida-codemode (0.3.1) and installed into
~/ida-venv, so the adapter can be checked against the library instead of
against assumptions.
First thing it found: connect() passed loading_address=, which
DatabaseHandle.open() does not have. The real parameter is image_base, and it
already wants the natural 16-byte-aligned address we compute, so this is a
rename. Every connect would have died with TypeError on the first call.
The port's own contract test could not catch it: its fake handle takes
**kwargs, so any keyword at all looks accepted. The test now also validates
the keywords we send against inspect.signature(DatabaseHandle.open) when the
library is importable, and skips that one check when it is not.
Offline suite: 302 passed with the library installed, 302 without it.
|
|
|
Mechanical part of the port: the 27-file patch was cut against a base ~148
commits behind us, so it did not apply. Resolved 11 conflicts (all of them
diff drift, not semantic clashes) and the three file deletions:
- app.py: the patch re-inserted _do_rename/_do_name_addr/_seek_split etc. as
"theirs" because our tree moved them to edit_ctl.py/trace_ctl.py. Kept ours
and applied the real intent (WorkerClient->CodeModeClient, .call->.invoke,
_open_worker_client->_open_database_client) at their current homes.
- domain.py: kept Head as a NamedTuple -- the patch reverted it to a frozen
dataclass, which the perf work measured at 2.9us vs 1.9us per row on a
quarter-million-row walk. Dropped _fetch_output (no download_url under Code
Mode) and its now-dead urllib/json imports.
- pane.py: the patch's deletion swallowed our zellij support along with the
worker-reaping block it meant to remove. Kept zellij, removed the reaping.
- test_scenarios.py: the idb_save->save_database teardown hunk belongs to
tests/_fixtures.py in our tree; applied it there and kept our pc_num_format
scenario that the drift landed on.
Three defects in the patch itself, fixed here:
- It made "import idatui" hard-require ida_codemode, so every offline suite
died at import -- including the pure ones (graph/index/trace) that are the
house rule for "tests/run.py --fast". The import is now deferred and gated
on the binding, which is also what lets the port's own contract tests
inject a fake DatabaseHandle.
- project.stage() inlined an ida_codemode.registry import and treated "library
not installed" as "someone owns this database", which broke IDA-free project
staging. Ownership lookup moved to codemode_client.database_owner().
- tests/test_codemode_client.py had no NEEDS_IDA marker, which tests/run.py
rejects outright.
Offline suite: 301 passed, 0 failed. Against master's 344 the whole delta is
accounted for: -40 worker_client (module deleted), -18 launch sweep checks
(behaviour deliberately removed) +3 guarding that it stays removed, +2 pool
(GUI-save semantics), +13 new codemode_client contract tests.
NOT yet done, and the port is not functional without it: the adapter is
missing five operations our tree grew since the patch's base (flowchart,
op_format, pc_nums, pc_num_format, survey_binary) and its "heads" predates
back-walking and digest/expect.
|
