| Commit message (Collapse) | Author | Age |
|---|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The listing used to learn its shape by fetching it. Even after skeleton pages
that was 457 round trips and 227k rows for a 1.2MB bash, to end up knowing
how many rows there are and where each one is.
segment_index(detail=True) now returns exactly that -- every row's address,
kind and size as packed arrays, plus the page boundaries -- from one walk that
builds no rows and renders no text. ListingModel.build_from_index() decodes it
straight into _heads/_head_eas/_row_at/_by_ea/_page_*, marks every row
_SKELETON_GEN, and declares itself complete. _grow has nothing left to stream.
Nothing else in the model changed, because a row without text is a state it
already had: the FIRST read of a page materialises it through the same
_ensure_text/_ensure_page path a rename uses. That is why this is a ~90 line
change to a core view rather than a rewrite.
bash boot: 911 calls / 9.26s -> 4 calls / 1.25s 7.4x
whole census (boot + 9 UI actions): 933 calls -> 30
Two things had to be exactly right, and both are tested rather than argued:
* the ROW COUNT, or the scrollbar lies. Verified equal to a fully streamed
model, and every row's ea/kind/size equal too, 228,659 of them, zero
mismatches.
* the PAGE BOUNDARIES, or _ensure_page refetches a page that does not line
up, fails its structure check and triggers a full rebuild. heads() pages on
PHYSICAL rows; anchoring every N LOGICAL rows looks identical (the two only
diverge once a segment holds an undefined run) and would have been a
lurking bug on .bss. Anchors now carry [logical_row, ea, head_index] taken
at the real boundary, and are asserted equal to the streamer's own.
Transport note: the packed arrays are base64, not raw bytes. _PACK_EPILOGUE
serialises with json.dumps(default=str), which turns bytes into their repr --
2.97MB arrived as 11.26MB of unparseable text before that was spotted.
The new test builds both models back to back and compares every internal
array. An earlier version compared against the app's long-lived model and was
off by one row, because scenarios before it rename and define things: that
model describes the database at boot, not now.
Full gate: 1063 passed, twice.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The listing streams a whole segment for one reason -- to know how many rows
it has, so the scrollbar and paging are right. Even as skeletons that is 458
round trips and 227k rows for a 1.2MB bash, none of which is displayed.
segment_index walks the same items and counts what heads() WOULD emit,
building none of them, and returns the total plus [row, ea] anchors every 500
rows. Measured on bash, same process and database:
segment_index : 228,659 rows, 1 call, 501ms
streaming : 228,659 rows, 458 calls, 1836ms 3.7x
Exactness is the whole point, so it mirrors _rows_for's arithmetic rather
than approximating it: 3 banner rows at a function start, a label row for a
named code head that is not one, the head row, struct members for data, 2
footer rows at a function end, and an undefined run counted as its byte
length because the client presents one collapsed row as that many logical
rows. A count that is off by a handful means the scrollbar lies and a seek
lands on the wrong row, so the test compares against a fully streamed model
in the same process rather than against a tolerance, and checks that every
anchor names the address of the row it claims.
Getting that right took a false alarm worth recording: the count first looked
35 rows short of a model built by the pilot, which turned out to be a
DIFFERENT DATABASE (tests run on a pristine scratch copy). Against the same
database it matches exactly, head for head, with zero differing addresses.
Nothing consumes this yet. Spending it means teaching ListingModel to hold
sparse pages seeked through the anchors instead of one dense array grown from
the segment start, which is a real change to the core view and wants its own
run at it.
Full gate: 1054 passed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The listing walk asked ida_bytes.is_code/is_data ~6 times per head (the row's
kind, plus twice more via _is_unknown_f from _advance and _rows_for). Each is
a python wrapper around a C call; the mask compare is the same question with
no call at all.
Equivalence was proven, not assumed: checked against is_code/is_data over all
four classes x 16,020 synthetic flag values, and over every distinct flag in
a real database (125 across 207,191 heads). Zero mismatches. Output is
byte-identical over 3003 rows spanning all five row kinds.
(The digest DOES differ run to run, which looked alarming until it turned out
the same unmodified code differs too: hash() is seed-randomised per process
and the digest is documented as per-process. Cross-process digest comparison
proves nothing.)
Worth 5.6% on a skeleton page (3.37 -> 3.18ms) and 2.7% on a full one -- the
~5% the profile predicted, not the 1.5-2x I guessed out loud. End to end it
does NOT move bash's boot (3117 -> 3140ms, noise), because at 455 x 3.18ms
the backend is now only ~1.45s of a 3.14s boot: the other ~54% is transport,
Head construction and model bookkeeping on the client.
Only the two hot sites changed. The four in the operand-format paths are not
per-head and read better as is_code/is_data.
Full gate: 1050 passed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
ListingView._grow streams the entire segment in the background for one
reason: to learn how many rows it has, so the scrollbar and paging are
right. It did that by rendering every row in full -- 227,500 rows of a
1.2MB bash, 911 backend calls, 9.3 seconds -- essentially none of which is
ever looked at.
generate_disasm_line is 22x the cost of the walk around it, so heads() gains
text=False: a SKELETON page with the same rows at the same addresses with the
same kinds and sizes, and no rendered text. Measured identical structurally
(rows, addresses, kinds, sizes and cursor all match a real page) which is
what makes one swappable for the other later. It also skips the digest
(nothing to go stale) and lets the client skip the bulk opcode read, so a
page costs ONE round trip instead of two.
Client side is deliberately tiny, because the machinery already existed: a
skeleton page is just a page whose text is stale. It is marked with a
sentinel generation no _text_gen can equal, and the FIRST read of it goes
through the same _ensure_text/_ensure_page path a rename uses -- which
already refetches a page by address, verifies the structure still lines up
and splices it in. Two staleness gates learn to fire for _skeleton as well
as _renamed; that is the whole integration.
bash boot: 911 calls / 9.26s -> 456 calls / 3.12s, 3.0x. The trade is that a
page you actually display is fetched twice (3.3ms + 9.4ms vs 9.4ms), paid
only for what is shown. _prime still loads real pages, so the viewport you
land on is never a skeleton.
The failure mode is BLANK ROWS, not an exception, and nothing in the suite
scrolled far enough to see one: _prime renders the first ~1000 rows for real,
so a test that pages down a few screens passes against a completely broken
implementation. The new scenario reads deep rows through both the model and
the render path, and asserts materialising changes neither the row count nor
the walk. Verified by reverting the two gates: it fails with text=''.
Full gate: 1050 passed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
_idatui_head_row and _idatui_line_parts were each defined TWICE at module
scope. Python keeps the last, so the first 74 lines were dead -- but the two
copies of _idatui_line_parts were not equivalent: only the live one carries
@lru_cache(maxsize=_IDATUI_LINE_CACHE).
That makes the duplication worse than clutter. Anyone tidying it up by
deleting the second copy would keep a byte-identical function and silently
lose the most important cache in the backend: 10.4us -> 3.9us per listing
row, ~2.7x on every row of every page. A comment now marks the spot and says
which copy to keep.
Found while profiling the digest; an AST duplicate-check reported the two as
IDENTICAL because ast.get_source_segment on a FunctionDef excludes the
decorator list. Compare decorators explicitly.
Verified before deleting: the block defines only names redefined below, and
nothing above it referenced them. Behaviour unchanged -- heads(500) still
9.3ms, digest still stable, and the expect/short-circuit contract still
omits rows on a match and returns them on a miss.
Full gate: 1042 passed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
_idatui_head_row ran 'import ida_bytes/ida_lines/ida_name' on every LISTING
ROW -- 25,500 sys.modules lookups per 500-row page. 50 such imports across
the file, all pointless: this module is never imported by the client
(codemode_client reads it as TEXT and installs it in the database process),
so the no-IDA house rule that forces function-local imports elsewhere does
not apply here.
Measured honestly, it is worth about 3%: heads(500) best-of-40 goes
9.37ms -> 9.07ms, median 9.73 -> 9.65. A microbenchmark of the bare import
predicted 3.2ms/page and was 10x optimistic -- the A/B is what counts, and
it is a small win, not the big one.
ida_hexrays deliberately stays function-local: it is licence-dependent, and
hoisting it would break the whole remote library for someone without the
decompiler rather than failing only when they decompile. Verified by AST
that every ida_* reference still resolves.
Adds experiments/profile_remote.py, which ships cProfile INTO the database
process -- the only way to see the split between IDA's own calls and our
python, which no client-side timer can show.
Full gate: 1042 passed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Benchmarking the port against master op-by-op (rather than only asking whether
tests pass) turned up three real bugs, all in the most user-visible path:
opening pseudocode.
1. decompile was doing decomp_map's job. It called the full per-column line
map purely to fill in each line's /*0xEA*/ anchor. The tool ida-tui was
written against takes ONE get_line_item at column 0 per line; the port took
one per COLUMN, i.e. thousands of get_line_item+dstr() calls per function
instead of one per line. Every pseudocode open cost the same as opening the
split view. Carried the real implementation over: 1888ms -> 53ms.
2. decomp_map used the pre-optimisation line map. Ours memoises obj_id -> ea
for the whole function (commit 853d90c: dstr() was 79% of the tool, and
consecutive columns report the same ctree item), the port's did not.
1925ms -> 287ms.
3. _idatui_compact imported ida_pro_mcp on every call. Under Code Mode that
package is not installed in the database process, so the import failed every
time -- and a FAILED import is never cached, so each one re-searched the
whole of sys.path: 422 failed imports per pc_nums call, which was most of
its runtime. 1428ms -> 257ms.
The same bug was a correctness bug hiding behind the perf bug: the fallback
path collapsed whitespace INSIDE string literals, where the real function
preserves it. Pseudocode columns are served in those coordinates, so on any
line containing a string with two spaces, every literal's mark and every
reformat would have been placed on the wrong column. It never fired on
master because ida_pro_mcp is installed there. Now calls the byte-identical
module-level shim directly, with the deviation from the extracted original
documented in place.
Narrow verification: decomp/split_view/opfmt/follow/comment/structs scenarios,
72 passed, 0 failed. Full gate running separately.
|
|
|
them
This closes the five operations the port was missing and restores the listing's
own tooling instead of a re-implementation of it.
idatui/remote_tools.py is the port's IDAPython island: `heads` (the continuous
listing) and `op_format`/`pc_nums`/`pc_num_format` (`o`/`O`), extracted verbatim
from the BODY that server/patch_server.py used to inject. They are real, diffable
source shipped to the database process as text, not string literals, because this
is the most performance-tuned and behaviour-sensitive code in the project.
Why carry `heads` over rather than keep the port's version: the port's rewrite
emitted no per-operand extents ("ops"), so no keypress could show which literal
it would reformat (opfmt_highlight had no two-operand row to find); it had no
digest/`expect` support, so every page was re-sent after any edit; and its span
walk was the per-character loop ours had already been rewritten out of. It also
dropped struct-member expansion sizing and the func banner/label rows' exact
shapes.
The library is installed ONCE per database process (sys.modules, keyed by a hash
of the source) and then called by name. Code Mode's execute_python builds a fresh
namespace per call, so a library exec'd inline is rebuilt every time and its
module-level caches thrown away -- the per-line render lru_cache in particular,
which the perf work sized to 65536 entries. Installing it once took `heads`
count=200 from 181ms to 92ms; the cache reports 211 hits on a second call where
it previously reported none. (Extraction footgun recorded: ast FunctionDef.lineno
points at `def`, not at the decorators, so a naive slice silently drops
@lru_cache.)
Also ported: flowchart, survey_binary, and the xref contract.
Live pilot suite on targets/echo: 301 passed, 0 failed -- identical to master.
Known, quantified, and NOT fixed here: Code Mode's transport is much slower than
the unix-socket worker for the listing's paging. heads count=200 is 2.6ms on
master vs 92ms here, count=500 is 6.3ms vs 214ms. Roughly half of that is
to_jsonable + HTTP framing per call and is inherent to the architecture; the
empty round trip alone is 2ms. The digest/`expect` path (unchanged pages) is the
main mitigation and is restored.
|
