Projects — grouping many binaries under one target

RE'ing a real target means several binaries at once: a daemon, the libraries it links, maybe a loader. IDA has no notion of this (one IDB, one window); Ghidra does. A project groups binaries so you can switch between them instantly, search across all of them, and (later) follow calls from one into another.

The constraint that shapes everything

idatui/worker.py is serve(sock, binpath) — one worker process holds exactly one database (idalib is main-thread-only and single-DB). So N binaries = N worker processes, each with the analyzed DB resident.

Measured cost (this box, targets/):

PSS ≈ RSS, so workers share almost nothing — the Nth worker costs roughly its full footprint. A ~120 MB baseline (Python + the idalib kernel) dominates for small binaries; the DB working set dominates for big ones.

Therefore residency is budgeted by memory, not by a fixed worker count — a count is the wrong knob when one project holds both a 50 KB helper and a 6 MB crypto library.

The key split

Two capabilities that feel like one, but aren't:

1. Switching to a binary needs a live worker.
2. Searching across binaries does not — if a per-binary index (functions, strings, imports/exports) is cached on disk.

That split is the unlock: project-wide search stays instant across every binary, including ones never opened this session, and only jumping to a hit costs a worker spawn.

Layout

The project file is explicit (--project router-fw.json); everything IDA generates lives in a sidecar dir beside it, so source trees stay pristine:

router-fw.json               # the project file
router-fw.idatui.d/
  bin/httpd                  # staged copy of the source binary
  bin/httpd.i64              # IDA's DB + scratch land here automatically
  idx/httpd.json             # cached index (phase 2)

Binaries are staged into bin/ and IDA opens the staged file, so the .i64 and all the .id0/.id1/.id2/.nam/.til scratch are created there rather than next to the original. (Today they pile up beside the source: targets/ holds ~244 MB of IDA litter around ~13 MB of binaries, much of it stale wedge files from hard-killed workers.) A source whose size/mtime no longer matches the staged copy is re-staged, and its now-stale DB dropped.

Staging copies rather than hardlinks. A hardlink is free, but source and staged would be one inode: an in-place rebuild (cp newbuild /path/bin truncates rather than replacing) would silently swap the bytes under an already analysed database, with nothing to detect it. A copy costs a fraction of the .i64 it will grow, and leaves the sidecar self-contained — the project still opens after the sources go away (an unmounted firmware image, a cleaned build tree). The unit test caught this: with hardlinks an in-place rewrite never went stale.

{
  "name": "router-fw",
  "binaries": [
    {"path": "/abs/sbin/httpd", "label": "httpd"},
    {"path": "../lib/libauth.so"}
  ],
  "memory_pct": 25
}

Relative paths resolve against the project file. label defaults to the basename and must be unique (it names the staged file).

Runtime

• WorkerPool — one WorkerClient per binary, spawned lazily on first switch, kept resident until the memory budget is exceeded, then LRU-evicted. Eviction saves the DB first, so returning to a binary is a DB load, not a re-analysis. Binaries can be pinned to stay resident.
• BinaryState — per binary: client, program, nav, cur, func_index, pref/active/split, filter. Switching snapshots the current state and restores the target's. _after_reconnect already does exactly this swap (client + program, reload the index, re-open the entry) — switching reuses that seam.
• Clean shutdown — the worker currently does close_database(save=False) and is hard-killed on exit, which is why wedge files accumulate. Projects need save-on-evict and an orderly close anyway, so that gets fixed here.

UI

• Binary switcher — a palette (mirroring SymbolPalette/StringsPalette) listing each binary with resident/cold state and function count; Enter switches.
• Status bar shows the active binary.
• Scope toggle — the existing symbol (Ctrl+N) and strings (") palettes gain a this-binary ↔ whole-project toggle rather than new keys, so there's one mental model. Project-scope hits are prefixed with the binary label; Enter switches binary and jumps.

Phases

Phase 1 — project model + switching. DONE. Project file + staging (idatui/project.py), WorkerPool with budget eviction / save-on-evict / clean shutdown (idatui/pool.py), BinaryState snapshot+restore and the switch itself, the Ctrl+O switcher palette, the active binary in the status line, and --project (which creates the project when given binaries). One active binary; no cross-binary search yet.

Project mode is additive: with no --project the app is byte-for-byte the single-binary tool it was, which is what keeps the 167-check pilot honest. Switching reuses the _after_reconnect shape — swap client+program, rebuild the index, reopen the entry. A binary whose worker is still resident restores instantly (its Program and index are still in memory); an evicted one comes back with a fresh worker but keeps its nav history, since that is just addresses.

Phase 2 — index cache + project-wide search. (symbols done) idatui/index.py keeps one SQLite FTS5 trigram index at <sidecar>/idx/project.db, populated per binary after its functions load and re-done only when the source's size/mtime changes.

Why that and not a library: nothing needed installing (FTS5 + the trigram tokenizer are stdlib), and trigram indexes arbitrary substrings, which is what symbol names and string bodies need. Measured on 300k entries: 1.9 ms per query vs 11.8 ms for a Python scan and 28.9 ms for plain LIKE; 0.2 ms per incremental insert.

Size turned out to be a non-issue. Real binaries: bash = 5.9k entries / 0.15 MB of text, libcrypto.so.3 = 30.7k / 0.52 MB. The index runs ~5.7x the text, so a 20-binary project lands around 12-23 MB — next to the .i64 files already in the sidecar (libcrypto's alone is 72 MB) that is ~1% of what the project already costs. The reason to keep it on disk is residency, not size: search has to work for binaries whose worker isn't running.

Querying is two-stage: the trigram index narrows across every binary, then the existing _fuzzy ranks what's left, so project scope keeps the same fuzzy-subsequence feel as local scope. Queries shorter than 3 characters fall back to LIKE — a trigram index silently returns nothing below that, which would make incremental typing look broken until the third keystroke.

Both palettes gain a scope toggle on F2 (not ctrl+a: the focused Input binds that to home) — Ctrl+N for symbols and " for strings. Project-scope hits are prefixed with their binary, and choosing one in another binary switches to it and jumps. Project scope caps at 60 rows; it saturates any cap because every binary contributes, and a long list is what makes arrowing feel sluggish.

Ranking, both palettes: the index already guarantees the match, so ordering only sorts — earliest match, then shortest, then (binary, addr). That last tiebreak is load-bearing: without it, a name shared by two binaries (main, textdomain, most of libc) ties on every other field and Python falls through to comparing the hit objects, which raises TypeError.

Phase 3 — cross-binary linking (done). Each binary's imports and exports go into the project index alongside its functions and strings (KIND_IMPORT, KIND_EXPORT), and following an import resolves it to the binary that provides it.

Following a call to strcmp used to reach the PLT/extern entry and stop: Hex-Rays has nothing to decompile because the code lives in a library this binary only references. Now _follow_import checks whether the target address is one of this binary's import stubs, asks the index who exports that name, and switches there. From an echo + libc project, Enter on strrchr(a1, 47) lands on strrchr at 0xaf960 in libc.

Three things this depends on:

ELF symbol versioning. The importer sees strrchr@@GLIBC_2.2.5; the provider may export strrchr, strrchr@GLIBC_2.2.5, or the versioned spelling. Comparing raw names resolves almost nothing, so domain.link_name() cuts at the first @ and both sides meet on the bare symbol. Linkage.raw keeps what IDA reported, which is what the listing shows.

Exact match, not substring. The join uses ProjectIndex.exact(), not search(): an import must bind to its own name, or read picks up pread, read_line and thread_start. Exact match also works below the 3-character trigram floor, which matters — plenty of real exports are one or two characters.

Resolution reads the index, not a worker. A provider resolves while its worker is evicted; that is the reason the index is on disk. The switch itself then pages the provider in through the normal pool path.

The reverse direction is in the xrefs dialog. xrefs_to only ever sees the current database, so an exported function looks unused from the inside even when half the project calls it. _foreign_importers appends the project binaries that IMPORT the symbol — read from the on-disk index, so a caller shows up whether or not its worker is resident — and choosing one carries a (binary, addr) payload through the same switch path as a search hit, hop included. From libc's strrchr: 0000C318 import [echo] strrchr.

It only fires for a symbol this binary actually exports. A local name that happens to collide with another binary's import is not a caller of ours.

When nothing in the project provides the symbol, _follow_import declines and the local navigation proceeds — landing on the stub is still the honest answer, and a single-binary session behaves exactly as before.

This does not retire the PLT/import-stub backlog item. An import whose provider isn't in the project still lands on a stub, and that case wants the original fix — recognise the thunk and present it as an import ("strcmp — imported, provider not in project") rather than surfacing a decompiler error. Phase 3 covers the valuable half; stub presentation stays worth doing.

Phase 4 — polish (mostly done).

Nav history is per-binary, with a cross-binary way back. Each binary keeps its own stack in BinaryState; switching restores it (addresses outlive the worker, so it survives eviction). But phase 3 made a cross-binary jump an ordinary action, and arriving in a binary whose history is empty made it a one-way door. _switch_then_goto now records the binary it came FROM, and action_back falls through to that hop once local history is spent. So Esc walks back through the function you were in, then the binary you were in. Manual switching (Ctrl+O) records nothing — that's not navigation.

Pre-warm follows the linkage graph, not list order. When a binary finishes indexing, _prewarm_provider warms the binary that provides the most of its imports — where a follow is most likely to take you, so its startup is paid before you ask. WorkerPool.prewarm() refuses rather than evicting: spending a binary you visited on one you haven't is a straight downgrade, and it would throw away that binary's caches too. At a tight budget pre-warm simply does nothing. It estimates the cost of a not-yet-spawned worker from the largest resident one, which is the only evidence available; if the estimate proves wrong, the speculative worker is the one that goes.

Driving a project. pane spawn --project FILE [--open BIN] opens a project pane. binaries lists the inventory (active / resident / indexed count / where Esc returns to) and switch {binary,addr?} makes another one active — with an address it goes through the same path as a search hit, so it records a hop. The state snapshot now carries binary and hops.

Still open: project-level persistence (remember the active binary and each binary's position across sessions — today a fresh launch auto-lands).

Headerless blobs

An ELF/PE/Mach-O says what it is, so IDA figures it out. A raw firmware dump says nothing, and IDA falls back to x86 at address 0 — which on an ARM image analyses to zero functions. It doesn't fail; it just quietly gives you nothing.

Open one without saying anything and idatui asks, the way interactive IDA does when no loader matches:

┌ unrecognised file — processor? (20) ────────────────┐
│ fw.bin  (32,768 bytes) — no loader matched; without │
│ a processor IDA assumes x86 at 0                    │
│ arm                                                 │
│   arm          ARM / AArch64 — little-endian        │
│   armb         ARM — big-endian                     │
│ load address, e.g. 0x8000000 (blank = 0)            │
│ Enter accept · Tab base address · Esc load as IDA   │
└─────────────────────────────────────────────────────┘

Every name in that list is verified against a real IDA by tools/verify_procs.py (open a scratch blob with -p<name>, read back inf_get_procname()). A wrong name is rejected — rc=4, nothing useful said — which would be a dead end handed out from inside the dialog meant to rescue you. Two of the first twenty were wrong: h8 and sparc are module filenames (procs/h8.so), not processor names; the real ones are h8300 and sparcb/sparcl. The aliases people reach for first — arm64, aarch64, mips, m68k — are all invalid too, so they live in the human labels where the filter still finds them.

formats.sniff() decides whether to ask, and only recognises formats IDA definitely handles (ELF, PE, Mach-O, dex, wasm, ar, COFF, Intel HEX, S-records). Being cautious the wrong way costs one dismissible dialog; being cautious the other way is the silent-nothing case we're trying to kill. Esc loads it the way IDA would have, because the sniff can be wrong.

A text container is only accepted when the whole header is printable — a raw image whose first byte happens to be : is far likelier than Intel HEX.

The dialog is skipped when the answer already exists: options on the command line, options in the project, or a database that already records them.

Or say it up front:

ida-tui fw.bin --processor arm --base 0x8000000

or per binary in a project, which is where it belongs for a multi-image firmware:

{"binaries": [
   {"path": "bootloader.bin", "processor": "arm",  "base": "0x0"},
   {"path": "app.bin",        "processor": "arm",  "base": "0x8000000"},
   {"path": "dsp.bin",        "processor": "mipsb", "base": "0x10000000"}
]}

base is written the way you'd say it (0x8000000, any base, int or string). IDA's own -b switch is in paragraphs — -b1000 loads at 0x10000 — so the conversion happens in BinaryRef.load_args, and a base that isn't 16-byte aligned is rejected rather than silently landing somewhere else. ida_args passes anything else through untouched.

ARM/Thumb

Thumb is not a property of the bytes — it's a mode the CPU is in — so a raw image gives IDA nothing to detect. At a Thumb entry point it decodes 16-bit instructions as 32-bit ARM and produces confident nonsense: push {r3,lr} (08 b5) reads as SVCLT 0xBF00, and c either refuses or carves garbage.

t on the listing switches the mode at the cursor and disassembles in it:

Thumb @ 0x0 (segment set to 32-bit; Thumb needs ARM32) — 10 instructions

It sets IDA's T segment register, and forces the segment to 32-bit when turning Thumb on. That second part isn't optional: Thumb doesn't exist in AArch64, and a headerless blob loaded with -parm comes up 64-bit, so setting T alone changes nothing and looks broken. Asking for Thumb is asking for ARM32.

t again toggles back — the mode is a guess, and guesses get revised. Both states are saved in the .i64.

Pick a 32-bit processor at load, or none of this decompiles. Bare arm gives a 64-BIT database (AArch64), and that is decided at load time and cannot be changed afterwards — setting the bitness post-hoc makes the decompiler INTERR. In a 64-bit database:

• Thumb doesn't exist, so t sets a segment flag that means nothing; and
• Hex-Rays refuses a 32-bit function outright — "only 64-bit functions can be decompiled in the current database" — so the disassembly looks right and F5 silently produces nothing.

So the list offers arm:ARMv7-A (most firmware), arm:ARMv7-M / arm:ARMv6-M (Cortex-M, Thumb only) and arm:ARMv5TE alongside 64-bit arm. t warns when it notices the database is 64-bit and points at Ctrl+L.

Worth knowing: a correctly-chosen 32-bit ARM database also lets IDA's own auto-analysis find Thumb functions — experiments/fibonacci.bin yields 54 functions on load with arm:ARMv7-A and none with arm.

Thumb entry points from a vector table

Shift+T scans forward from the cursor for odd pointers — an ARM function pointer carries the mode in bit 0, so a Cortex-M vector table is a list of Thumb entry points. IDA won't follow them on a headerless image, because nothing tells it those words are pointers at all:

3 Thumb entries found, 3 disassembled

A word only counts when it is odd, lands inside a loaded segment, and its target is executable and not already data. The even words in a vector table (the initial stack pointer) fail the first test, which is the point — marking a data word as code corrupts the listing, so a false positive costs more than a miss.

When analysis finds nothing

Zero functions is what a raw image described wrongly looks like — right file, wrong architecture, and IDA has no complaint to make about it. The app used to fall through to the symbol picker, which had nothing to show, so you got empty panes and a status reading "functions still loading…" long after loading had finished.

Now it opens the listing at the start of the image (the bytes are there even when no code was recognised) and the status bar carries the diagnosis for as long as it's true:

seg000 @ 0x0  [listing]  — no functions: wrong processor/base? Ctrl+L to reload

Ctrl+L re-asks. The database IDA already built has the old processor and base baked into it and wins over any switches, so reloading means deleting it — hence the confirmation, which tells you what you'd lose (and, in this case, that you'd lose nothing).

Two things worth knowing:

• The options apply to the first open only. Afterwards the .i64 records how the image was loaded, and passing the switches again makes IDA refuse to open it — so the worker skips them once a database exists.
• Changing the processor or base of a binary you've already analysed does nothing, because the database wins. Delete its .i64 from the sidecar to reload with new options.

Decisions

• Residency is a memory budget (default 25% of RAM, memory_pct), not a worker count; pinning supported.
• Nav history is per-binary to start.
• Search scope is a toggle inside the existing palettes.
• The project file is explicit (--project), not auto-discovered.
• IDA artifacts are centralized in the sidecar dir via staging.

Open questions

• Re-staging a changed source drops its DB (renames included) — right call, but it should warn loudly.
• Should Esc/back ever cross binaries (global history)? Deferred.
• Pre-warm policy: analyze the whole project up front, or purely on demand?