aboutsummaryrefslogtreecommitdiffstats
path: root/docs/PROJECTS.md (follow)
Commit message (Collapse)AuthorAgeFilesLines
* arm: find Thumb entry points from a vector table (Shift+T)blasty3 hours1-0/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | An ARM function pointer carries the mode in bit 0: odd means Thumb. A Cortex-M vector table is therefore a list of Thumb entry points, and IDA won't follow them on a headerless image because nothing tells it those words are pointers at all. Shift+T scans forward from the cursor and marks them. 0 functions -> 3 Thumb entries found, 3 disassembled A word only counts when it is odd, lands in a loaded segment, and its target is executable and not already data. The even words in a vector table — the initial stack pointer — fail the first test, which is the point: marking a data word as code corrupts the listing, so a false positive costs more than a miss. The fixture includes an even in-range word and an odd OUT-of-range word to keep that honest. A note on how this started: I recommended this feature, then probed experiments/fibonacci.bin for the signal and found ZERO odd in-range pointers — it's a flat code blob, not a firmware image. Rather than build a detector I couldn't test, I wrote experiments/cortexm.bin: a real vector table pointing at small self-contained Thumb handlers. The first version of that fixture aimed its handlers into the middle of copied code, so two "entries" were really inside one function — the tool was right and the fixture was wrong, which is worth stating because I nearly filed it as a bug. Function creation goes through one _idatui_add_func helper now, shared with define_func_run: add_func(ea) alone fails on freshly-marked code (IDA can't find the end), and the scan hit exactly the same wall `p` did. Status precedence, fixed properly this time. An action's result kept being overwritten by the reload it triggered — cursor moved, filter re-applied, functions re-counted. I patched that at FIVE separate call sites before admitting it's one problem. _status(text, priority=True) now marks a result: it holds the bar for 8s or until the next keypress, and routine chatter can't outrank it. The per-site special cases are gone. tests: +4 thumb (20) — a bare vector table gives IDA nothing, scanning finds exactly the three handlers, the non-pointer words are ignored, and the result survives both the reload and the reindex. 209/0 scenarios, 30/0 blob, 30/0 project UI.
* arm: offer 32-bit ARM at load, and fix `p` on carved codeblasty9 hours1-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Reported as "after c a few times and p at the entry point, Tab just flashes and nothing decompiles". Three separate things, found by following it down: **1. `p` failed on hand-carved code.** ida_funcs.add_func(ea) asks IDA to find the function's end and on carved code it often can't — a run ending in a tail call, or whose last instruction isn't recognised as a return, fails with no reason given. add_func(ea, end) with an explicit end succeeds. define_func_run tries IDA's way first, then falls back to the end of the contiguous instruction run, and says which it used. **2. The database was 64-bit, so Hex-Rays refused it regardless.** Bare `-parm` gives an AArch64 database. Ask Hex-Rays for the failure object rather than reading None as "dunno" and it says exactly what's wrong: "only 64-bit functions can be decompiled in the current database". So the disassembly looked right and F5 could never work. That is decided at LOAD and cannot be corrected — inf_set_app_bitness(32) afterwards makes the decompiler INTERR 50735. The fix is at the load dialog: arm:ARMv7-A (most firmware), arm:ARMv7-M / arm:ARMv6-M (Cortex-M, Thumb only) and arm:ARMv5TE now sit alongside 64-bit `arm`, labelled with their bitness. With arm:ARMv7-A, experiments/fibonacci.bin decompiles: void __fastcall __noreturn sub_0(int a1) { int v2; v2 = sub_E3C(a1, 0); ... } — and IDA's own auto-analysis finds 54 Thumb functions on load, versus none as plain `arm`. **3. `t` was silently building an undecompilable state.** It forced the SEGMENT to 32-bit in a 64-bit database, which produces correct-looking disassembly that F5 will never touch. It now says so and names the fix (Ctrl+L, arm:ARMv7-A) rather than leaving you to discover it. tools/verify_procs.py now reports each processor's resulting bitness, since that is the reason the variants exist — and it compares against the base module name, because a variant reports "ARM". tests: test_thumb_ui.py +5 (13 total) — a 64-bit database warns and names the fix, a 32-bit one finds functions by itself, Tab decompiles a Thumb function and the result reads like C. test_formats.py +2 (34) pinning that a 32-bit variant is offered and the ARM labels state their bitness. 209/0 scenarios, 26/0 blob, 30/0 project UI.
* arm: switch ARM/Thumb decoding with `t`blasty9 hours1-0/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | `c` could not carve Thumb code. Thumb isn't a property of the bytes — it's a mode the CPU is in — so a raw image gives IDA nothing to detect: at a Thumb entry point it decodes 16-bit instructions as 32-bit ARM and produces confident nonsense. experiments/fibonacci.bin starts with `08 b5` = push {r3,lr}, which IDA reads as SVCLT 0xBF00. `t` on the listing switches the mode at the cursor and disassembles in it: Thumb @ 0x0 (segment set to 32-bit; Thumb needs ARM32) — 10 instructions 0x0 PUSH {R3,LR} 0x2 MOVS R1, #0 0x4 MOV R4, R0 0x6 BL unk_E3C Setting the T segment register is only half of it. Thumb does not exist in AArch64, and a headerless blob loaded with -parm comes up 64-bit, so T alone changes nothing and looks broken — I watched exactly that happen while probing the API. Asking for Thumb IS asking for ARM32, so set_thumb forces the segment to 32-bit and says so rather than doing it silently. It also has to del_items over the range first: the bytes are currently decoded in the old mode, and leaving that item defined pins the wrong instruction length so the new mode has nothing to apply to. Implemented as a `thumb` kind in the existing edit-item flow, so it inherits the shared reload — same cache bump, same ViewAnchor restore, same status flash. It switches AND disassembles, because flipping T and leaving the bytes undefined shows you nothing and reading the code was the point. tests: new tests/test_thumb_ui.py (8) driving the real Thumb binary — `c` alone does NOT produce the prologue, `t` does, the instructions are 16-bit wide (in ARM mode those three rows would be one 4-byte instruction), the run continues, the status explains the 32-bit forcing, and `t` toggles back. Deletes the .i64 first, because T and the segment's addressing mode are saved in it and a stale database would answer the question for us. 209/0 scenarios, 26/0 blob, 8/0 thumb.
* loading: a blob that analyses to nothing is still openableblasty15 hours1-0/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pick a random .bin, say ARM at 0x4000, and you got two empty panes and "functions still loading…" — which was a lie; loading had finished. _auto_land falls back to the symbol picker when there's no entry function, and the picker answers an empty index with that message. Nothing ever opened. Zero functions is not a corner case. It's exactly what a real firmware image looks like when it's described wrongly, and IDA has no complaint of its own to make about it, so this was the last silent-wrong-answer in the blob path. Now: * Land in the listing at the start of the image. The bytes exist even when no code was recognised, so there is always something to show. * Say so, in the status bar, for as long as it stays true — an image with no functions is a property of the database, not an event, and writing it once meant the next status write erased it (the same clobber that bit the split view's "decompiling…"). * Ctrl+L re-asks. The .i64 has the old processor and base baked in and takes precedence over any switches, so reloading means deleting it; the confirmation says what that costs, and when there are no functions it says nothing is lost. Verified with real keys on a random 64K blob: ARM @ 0x4000 lands showing "db 65536 dup(?)" with the hint in the status; Ctrl+L -> confirm -> dialog -> metapc reloads at 0. The hint survives scrolling. tests: new tests/test_blob_ui.py (11) driving a real random blob end to end — lands, has rows, right base, honest status, hint survives navigation, Ctrl+L offers the reload and declining leaves the binary open. 202/0 scenarios, 30/0 project UI. (Harness note for future me: tmux send-keys reads "0x4000" as a hex KEY CODE and sends U+4000. Use send-keys -l for literal text.)
* formats: verify every offered processor name against a real IDAblasty24 hours1-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | The curated list was written from the procs/ directory listing. Two of the twenty names were wrong, and wrong here is not a soft failure: IDA REFUSES to open the database (rc=4) with nothing useful said. The load dialog would have handed people a dead end from inside the UI that exists to rescue them — the same silent-failure class the dialog was built to kill. h8 -> h8300 (h8.so is the module FILENAME, not a processor name) sparc -> sparcb / sparcl (and SPARC has endianness variants, like MIPS/PPC) Also probed the aliases people reach for first: arm64, aarch64, mips, m68k are all invalid. 'arm' covers AArch64 (verified: an AArch64 blob analyses to 35 functions under -parm), so those names now live in the human labels, where the filter still finds them — typing "arm64" finds ARM, "m68k" finds 68k, "mips" finds both endiannesses. tools/verify_procs.py does the check: open a scratch blob with -p<name>, read back inf_get_procname(), compare. Fresh temp dir per name, because once a database exists IDA ignores the load switches and every name after the first would "pass". 21/21 verified. tests: +11 formats, including the verified-set guard (adding a processor without re-running the script fails on purpose) and checks that the rejected aliases are NOT offered but ARE still findable by typing them. 32/0 formats, 199/0 scenarios.
* loading: ask how to load an unrecognised file, like IDA doesblasty24 hours1-0/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Last commit let you SAY how to load a blob. This one notices when you should have. Interactive IDA pops a dialog when no loader matches; we silently loaded as x86 at 0 and analysed to nothing, so the flag only helped people who already knew they needed it — which is exactly the people who don't need help. formats.sniff() recognises the formats IDA definitely handles (ELF, PE, Mach-O, dex, wasm, ar, COFF, Intel HEX, S-records). Anything else gets LoadOptionsScreen: a filterable processor list with human labels, a load-address field, Enter to accept, Esc to load it the way IDA would have anyway. Deliberate asymmetry: the sniff only claims formats it is sure about. A false "unknown" costs one dismissible dialog; a false "known" is the silent wrong answer this exists to kill. Esc is always an escape hatch. The list offers 20 processors, not IDA's 73 — most of the rest are museum pieces, and a name typed into the filter that matches nothing is taken literally so nothing is actually unreachable. Endianness is spelled out (arm vs armb) because getting it backwards is the most common route to zero functions. Asked only when nobody has answered yet: not with --processor, not in project mode (entries carry their own), and not when a database exists — the .i64 already records how the image was loaded. Textual trap worth recording: the screen stored the file size in self._size, which is Widget's own backing field for outer_size. Assigning an int to it crashes layout with "'int' object has no attribute 'region'" from deep inside _set_dirty, nowhere near the cause. Same family as the _render collision. The paragraph conversion now lives in exactly one place (formats.load_args); BinaryRef and launch both call it. Verified on a real AArch64 blob: dialog appears, filtering to "arm" leaves two entries, base 0x8000000 accepted -> "-parm -b800000" -> 35 functions at 0x8002440. An ELF never asks, and neither does a blob that already has a .i64. tests: new tests/test_formats.py (21) and a load_options scenario asserting the dialog stays out of the way for a recognised binary. 199/0 scenarios, 39/0 project, 30/0 project UI, 36/0 index, 27/0 pool, 21/0 formats.
* loading: say how to read a headerless blob (processor, base)blasty24 hours1-0/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A raw firmware dump has no format to detect, so IDA fell back to x86 at address 0. It doesn't fail — it opens, analyses, and finds nothing. An AArch64 image loaded this way gave 0 functions; told the truth it gives 35. ida-tui fw.bin --processor arm --base 0x8000000 and per binary in a project, which is what a multi-image firmware actually needs: {"path": "app.bin", "processor": "arm", "base": "0x8000000"} idapro.open_database() already accepted IDA command-line switches; nothing was passing any. Plumbed BinaryRef -> WorkerPool -> WorkerClient -> worker argv, plus a load_args for the single-binary path that has no project ref. base is written the way people say it (0x8000000, int or string, any base). IDA's -b is in PARAGRAPHS — -b1000 loads at 0x10000 — so BinaryRef.load_args converts, and a base that isn't 16-byte aligned is refused rather than silently landing 16x off. ida_args passes anything else through. Two bugs found by testing the whole path rather than the happy one: * Project.load() whitelisted path/label when normalising entries, so the load options were dropped the first time a project was reopened — set a processor, come back tomorrow, it's gone. * Re-passing the switches to an EXISTING database makes IDA refuse the open (rc != 0, no functions). The .i64 already records how the image was loaded, so the worker skips them once a database exists. My first guard checked splitext(path) + ".i64" and never fired, because IDA names it "<file>.i64" — keeping the extension. It checks both spellings now. Verified on a real AArch64 blob: fresh load 35 functions based at 0x8002440, reopen 35 again, and the CLI rejects an unaligned or non-numeric --base. tests: +6 project (options recorded, paragraph conversion, file round-trip, add() takes them, an ELF passes nothing, hex-string base). 39/0 project, 195/0 scenarios, 30/0 project UI, 36/0 index, 27/0 pool.
* xrefs: show project binaries that import this exportblasty25 hours1-0/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | xrefs_to only ever sees the current database, so an exported function looks unused from the inside even when the rest of the project calls it. The import side of the phase-3 linkage index already knew better; nothing surfaced it. _foreign_importers appends those callers to the xrefs dialog. From libc's strrchr, with echo in the project: 0000C318 import [echo] strrchr Read from the on-disk index, so a caller appears whether or not its worker is resident. Choosing one carries a (binary, addr) payload instead of a bare address; _on_xref_chosen routes that through _switch_then_goto — the same path a project search hit takes — so it records a hop and Esc comes back. Only fires for a symbol this binary actually EXPORTS. A local name that happens to collide with some other binary's import is not a caller of ours, and without that check every common name (main, read, error) would sprout fictional callers. Names are compared after link_name(), so ELF versioning doesn't hide the match. Verified on a real echo+libc project: the dialog lists echo's call site, and selecting it switches to echo, lands on 0xc318 and leaves hops=['libc.so.6']. tests: +3 project UI — a symbol we don't export gets no cross-binary callers, the (binary, addr) payload jumps to the other binary, and it records the hop. 195/0 scenarios, 30/0 project UI.
* projects phase 4: cross-binary back, linkage-guided pre-warm, project verbsblasty25 hours1-3/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Three things, all following from phase 3 making cross-binary jumps ordinary. **A cross-binary jump was a one-way door.** Nav history is per-binary, so arriving in another binary — a project search hit, or now following an import into the library that implements it — landed you in an empty history with nothing to take you back. _switch_then_goto records the binary it came FROM, and action_back falls through to that hop once local history is spent: Esc walks back through the function you were in, then the binary you were in. Manual Ctrl+O switching records nothing, because that isn't navigation. **Pre-warm follows the linkage graph, not list order.** _prewarm_provider warms the binary providing the most of this one's imports — where a follow is most likely to go, so its startup is paid before you ask for it. "Next in the list" would have been arbitrary; phase 3 gave us something better to ask. pool.prewarm() refuses rather than making room. Evicting a binary the user visited to speculatively load one they haven't is a straight downgrade, and it throws away that binary's caches as well; at a tight budget pre-warm just does nothing. The cost of a worker that doesn't exist yet can only be estimated, so it uses the largest resident one (same program, different database) — and if that estimate proves wrong, the speculative worker is the one evicted, never a chosen one. **Driving a project.** pane spawn --project FILE [--open BIN]; `binaries` lists the inventory (active / resident / indexed / where Esc returns to) and `switch {binary,addr?}` makes another active — with an address it takes the search-hit path, so it records a hop. state gains `binary` and `hops`, which it should have had the moment project mode existed. Verified on real sessions: drive binaries/switch against an echo+cat project pane; Esc crossing back from a switch; and prewarm on echo+libc picking libc (provider of echo's imports) and warming it after an evict. tests: +5 pool (prewarm warms, no-ops when resident, refuses at budget, evicts nothing when refusing, ignores unknown labels) and +4 project UI (jump records the hop, Esc crosses back, hop consumed). Confirmed the Esc-back checks fail with the branch removed. 195/0 scenarios, 27/0 project UI, 36/0 index, 27/0 pool, 33/0 project. Left open: project-level persistence across sessions.
* projects phase 3: follow an import into the binary that implements itblasty26 hours1-15/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Following a call to strcmp reached the PLT/extern entry and stopped there — Hex-Rays has nothing to decompile, because the code lives in a library this binary only references. With the other binary open in the same project we already had everything needed to cross that gap; we just weren't indexing it. Index each binary's imports and exports (KIND_IMPORT / KIND_EXPORT) alongside its functions and strings. On a follow, _import_stub asks whether the target address is one of this binary's import stubs; if so _cross_binary_impl asks the index who exports that name, and we switch there instead of landing on the thunk. Verified end to end on a real echo + libc project: Enter on `strrchr(a1, 47)` in echo's pseudocode switches to libc.so.6 and lands on strrchr at 0xaf960. Three things it turns on: * ELF symbol versioning. The importer sees strrchr@@GLIBC_2.2.5 while the provider may export any of three spellings, so raw names resolve almost nothing. domain.link_name() cuts at the first '@'; Linkage.raw keeps what IDA reported, which is what the listing shows. * Exact match, not substring — ProjectIndex.exact(), so `read` doesn't bind to pread/read_line/thread_start. It also answers below the 3-char trigram floor, and plenty of real exports are that short. * Resolution reads the on-disk index, so a provider resolves while its worker is evicted. That's what the index was for. When nothing in the project provides the symbol _follow_import declines and the normal navigation runs: landing on the stub is still the honest answer, and a single-binary session is unchanged. The PLT-stub PRESENTATION item stays open — an unprovided import should say "imported, provider not in project" rather than show a decompiler error. server/patch_server.py gains list_linkage (idautils.Entries + enum_import_names); a worker without it degrades to no linkage rather than failing. tests: index join +8 (exact vs substring, short names, exclude-self, reverse join, kind isolation, forget unresolves) and link_name +4. 36/0 index, 195/0 scenarios, 23/0 project UI, 33/0 project, 22/0 pool.
* docs: refresh PROJECTS.md phase 2/3 to match what shippedblasty32 hours1-8/+28
| | | | | | | | | | | | | | * Phase 2: strings has the F2 toggle now, so drop the "still needs it" note. Record the 60-row project cap and the reason for it, and the ranking rule — including why the (binary, addr) tiebreak is load-bearing (a name shared by two binaries ties on every other field and Python then compares hit objects, raising TypeError; that shipped as a crash and was fixed). * Phase 3: spell out precisely how it relates to the PLT/import-stub backlog item. An export index turns "follow strcmp -> extrn strcmp:near, dead end" into a real jump into libc WHEN that library is in the project. It doesn't retire the item: a single-binary session, or a provider outside the project, still lands on a stub and still wants it recognised and presented as an import rather than a decompiler error.
* projects: project-wide symbol search over a SQLite FTS5 index (phase 2)blasty34 hours1-3/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | idatui/index.py — one on-disk index (<sidecar>/idx/project.db) over every binary in a project, so search works for binaries whose worker isn't running. Indexing choice, measured rather than guessed: * SQLite FTS5 with the TRIGRAM tokenizer — stdlib, no dependency (nothing else was installed and nothing is needed), and unlike a prefix index it matches arbitrary substrings, which is what symbol names and string bodies need. * 300k-entry corpus: 1.9 ms per query vs 11.8 ms for a Python scan and 28.9 ms for plain LIKE; 0.2 ms per incremental insert. * Size was the stated worry and turned out not to bite: bash contributes 5.9k entries / 0.15MB of text, libcrypto.so.3 30.7k / 0.52MB. At ~5.7x the text a 20-binary project is ~12-23MB — against .i64 files already in the sidecar (libcrypto's alone is 72MB), roughly 1% of what the project already costs. The reason to be on disk is residency, not size. * Trigram can't answer queries under 3 chars and returns nothing rather than erroring, so search() falls back to LIKE — otherwise incremental typing would look broken until the third keystroke. Wiring: after a binary's functions load, its symbols + strings are folded into the index (skipped when the source's size/mtime is unchanged). Ctrl+N gains a scope toggle on F2 — not ctrl+a, which the focused Input binds to "home" so it never reaches the palette. Project scope narrows via the index then ranks with the existing _fuzzy, keeping the same feel; hits are prefixed with their binary, and choosing one elsewhere switches binary and jumps to it. Also fixes another instance of the Textual-markup trap: the palette titles ate "[project]" as a style tag (same class of bug as the status bar), so the pal titles are markup=False now. tests/test_index.py: 24 stdlib checks — substring/case-insensitive matching, kind filter, the <3 char fallback, multi-binary search, per-binary incremental reindex, staleness, forget, persistence. Suite 191/0. Strings (") still needs the same scope toggle; the index already carries them.
* projects: switch between a project's binaries in the TUI (phase 1c)blasty36 hours1-4/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Wires the project model + worker pool into the app. Project mode is ADDITIVE — without --project the app is exactly the single-binary tool it was, which is what keeps the 167-check pilot meaningful. * IdaTui(project=...) builds a WorkerPool and opens the project's first binary; _open_worker_client asks the pool instead of spawning directly. * BinaryState snapshots what a switch leaves behind (program, func_index, nav, cur, view prefs, filter). Switching reuses the _after_reconnect shape: swap client+program, rebuild the index, reopen the entry. A still-resident binary restores instantly (Program + index are in memory); an evicted one gets a fresh worker but keeps its nav history, which is just addresses. * ProjectPalette (Ctrl+O, + a "Switch binary…" palette command): the project's binaries with resident/analysed/pinned/active state and memory, filterable. * launch.py --project FILE, creating the project when binaries are also given; stages everything up front so the source tree is never written to. Two bugs found while testing: * _did_auto_land is app-wide, but landing is per-binary: after the first binary landed, a cold switch never landed at all AND left the switch overlay up forever. Reset it per switch. * PRE-EXISTING: the status Static had Textual markup enabled, so a single-word bracket marker parses as a style tag and is silently eaten — [listing] and [pseudocode] have never actually rendered (only [split · listing] survived, because the · makes it an invalid tag). Status is plain text with brackets and symbol names, so markup=False. tests/test_project_ui.py: end-to-end pilot on two real binaries (18 checks) — boot, switcher contents, switch, per-binary index, both workers resident, switch back with state intact, return-to-where-you-were, and the promise that the source tree stays pristine while every artifact lands in the sidecar. Full single-binary suite unchanged at 167/2 (the standing flakes).
* projects: project model + binary staging (phase 1a)blasty37 hours1-0/+143
First slice of multi-binary projects (docs/PROJECTS.md): the on-disk model, with no runtime wiring yet. idatui/project.py (stdlib-only, like domain/worker): * Project.load/create/save — an explicit JSON project file listing binaries; paths resolve relative to it, labels default to the basename and are disambiguated on collision (they name files). * A sidecar dir beside the project file (<stem>.idatui.d/) holds bin/ (staged binaries), their .i64 + scratch, and idx/ for phase 2. IDA opens the STAGED file, so nothing lands in the source tree — today targets/ carries ~244MB of IDA litter around ~13MB of binaries, much of it stale wedge files. * stage() copies rather than hardlinks. A hardlink is free but makes source and staged one inode, so an in-place rebuild (cp over the path truncates instead of replacing) would silently swap the bytes under an analysed DB with nothing to detect it. The unit test caught exactly that. A copy also leaves the sidecar self-contained once the sources are gone. * Re-staging a changed source drops its now-stale DB; sweep_scratch() clears the unpacked working files a hard-killed worker leaves behind (never the .i64). tests/test_project.py: 27 checks, pure stdlib (no IDA/textual/worker), <1s. docs/PROJECTS.md: the full design — the one-worker-per-DB constraint with measured costs (bash worker = 126MB RSS/117MB PSS; libcrypto's DB is 72MB, so residency is budgeted by MEMORY, not a worker count), the switch between "switching needs a live worker" and "searching doesn't (cached index)", and phases 1-4.