# Projects — grouping many binaries under one target

RE'ing a real target means several binaries at once: a daemon, the libraries it
links, maybe a loader. IDA has no notion of this (one IDB, one window); Ghidra
does. A **project** groups binaries so you can switch between them instantly,
search across all of them, and (later) follow calls from one into another.

## The constraint that shapes everything

`idatui/worker.py` is `serve(sock, binpath)` — **one worker process holds exactly
one database** (idalib is main-thread-only and single-DB). So N binaries = N
worker processes, each with the analyzed DB resident.

Measured cost (this box, `targets/`):

| binary | size | `.i64` | worker RSS / PSS |
|--------|------|--------|------------------|
| `bash` | 1.2 MB | 14 MB | **126 MB / 117 MB** |
| `libcrypto.so.3` | 5.9 MB | 72 MB | several hundred MB (DB working set dominates) |

PSS ≈ RSS, so workers share almost nothing — the Nth worker costs roughly its
full footprint. A ~120 MB baseline (Python + the idalib kernel) dominates for
small binaries; the DB working set dominates for big ones.

**Therefore residency is budgeted by memory, not by a fixed worker count** — a
count is the wrong knob when one project holds both a 50 KB helper and a 6 MB
crypto library.

## The key split

Two capabilities that feel like one, but aren't:

1. **Switching** to a binary needs a *live worker*.
2. **Searching across** binaries does *not* — if a per-binary index (functions,
   strings, imports/exports) is cached on disk.

That split is the unlock: project-wide search stays instant across every binary,
including ones never opened this session, and only *jumping* to a hit costs a
worker spawn.

## Layout

The project file is explicit (`--project router-fw.json`); everything IDA
generates lives in a sidecar dir beside it, so **source trees stay pristine**:

```
router-fw.json               # the project file
router-fw.idatui.d/
  bin/httpd                  # staged copy of the source binary
  bin/httpd.i64              # IDA's DB + scratch land here automatically
  idx/httpd.json             # cached index (phase 2)
```

Binaries are **staged** into `bin/` and IDA opens the staged file, so the `.i64`
and all the `.id0/.id1/.id2/.nam/.til` scratch are created there rather than next
to the original. (Today they pile up beside the source: `targets/` holds ~244 MB
of IDA litter around ~13 MB of binaries, much of it stale wedge files from
hard-killed workers.) A source whose size/mtime no longer matches the staged copy
is re-staged, and its now-stale DB dropped.

Staging **copies** rather than hardlinks. A hardlink is free, but source and
staged would be one inode: an in-place rebuild (`cp newbuild /path/bin`
truncates rather than replacing) would silently swap the bytes under an already
analysed database, with nothing to detect it. A copy costs a fraction of the
`.i64` it will grow, and leaves the sidecar self-contained — the project still
opens after the sources go away (an unmounted firmware image, a cleaned build
tree). *The unit test caught this: with hardlinks an in-place rewrite never went
stale.*

```json
{
  "name": "router-fw",
  "binaries": [
    {"path": "/abs/sbin/httpd", "label": "httpd"},
    {"path": "../lib/libauth.so"}
  ],
  "memory_pct": 25
}
```

Relative paths resolve against the project file. `label` defaults to the
basename and must be unique (it names the staged file).

## Runtime

- **`WorkerPool`** — one `WorkerClient` per binary, spawned lazily on first
  switch, kept resident until the memory budget is exceeded, then LRU-evicted.
  Eviction **saves the DB first**, so returning to a binary is a DB load, not a
  re-analysis. Binaries can be pinned to stay resident.
- **`BinaryState`** — per binary: `client, program, nav, cur, func_index,
  pref/active/split, filter`. Switching snapshots the current state and restores
  the target's. `_after_reconnect` already does exactly this swap (client +
  program, reload the index, re-open the entry) — switching reuses that seam.
- **Clean shutdown** — the worker currently does `close_database(save=False)` and
  is hard-killed on exit, which is why wedge files accumulate. Projects need
  save-on-evict and an orderly close anyway, so that gets fixed here.

## UI

- **Binary switcher** — a palette (mirroring `SymbolPalette`/`StringsPalette`)
  listing each binary with resident/cold state and function count; Enter
  switches.
- **Status bar** shows the active binary.
- **Scope toggle** — the existing symbol (`Ctrl+N`) and strings (`"`) palettes
  gain a this-binary ↔ whole-project toggle rather than new keys, so there's one
  mental model. Project-scope hits are prefixed with the binary label; Enter
  switches binary *and* jumps.

## Phases

**Phase 1 — project model + switching. DONE.** Project file + staging
(`idatui/project.py`), `WorkerPool` with budget eviction / save-on-evict /
clean shutdown (`idatui/pool.py`), `BinaryState` snapshot+restore and the switch
itself, the `Ctrl+O` switcher palette, the active binary in the status line, and
`--project` (which creates the project when given binaries). One active binary;
no cross-binary search yet.

Project mode is **additive**: with no `--project` the app is byte-for-byte the
single-binary tool it was, which is what keeps the 167-check pilot honest.
Switching reuses the `_after_reconnect` shape — swap client+program, rebuild the
index, reopen the entry. A binary whose worker is still resident restores
instantly (its `Program` and index are still in memory); an evicted one comes
back with a fresh worker but keeps its nav history, since that is just addresses.

**Phase 2 — index cache + project-wide search. (symbols done)**
`idatui/index.py` keeps one **SQLite FTS5 trigram** index at
`<sidecar>/idx/project.db`, populated per binary after its functions load and
re-done only when the source's size/mtime changes.

Why that and not a library: nothing needed installing (FTS5 + the trigram
tokenizer are stdlib), and trigram indexes arbitrary *substrings*, which is what
symbol names and string bodies need. Measured on 300k entries: **1.9 ms** per
query vs 11.8 ms for a Python scan and 28.9 ms for plain `LIKE`; 0.2 ms per
incremental insert.

Size turned out to be a non-issue. Real binaries: `bash` = 5.9k entries /
0.15 MB of text, `libcrypto.so.3` = 30.7k / 0.52 MB. The index runs ~5.7x the
text, so a 20-binary project lands around 12-23 MB — next to the `.i64` files
already in the sidecar (libcrypto's alone is 72 MB) that is ~1% of what the
project already costs. The reason to keep it on disk is **residency, not size**:
search has to work for binaries whose worker isn't running.

Querying is two-stage: the trigram index narrows across every binary, then the
existing `_fuzzy` ranks what's left, so project scope keeps the same
fuzzy-subsequence feel as local scope. Queries shorter than 3 characters fall
back to `LIKE` — a trigram index silently returns *nothing* below that, which
would make incremental typing look broken until the third keystroke.

Both palettes gain a scope toggle on **F2** (not `ctrl+a`: the focused `Input`
binds that to `home`) — `Ctrl+N` for symbols and `"` for strings. Project-scope
hits are prefixed with their binary, and choosing one in another binary switches
to it and jumps. Project scope caps at 60 rows; it saturates any cap because
every binary contributes, and a long list is what makes arrowing feel sluggish.

Ranking, both palettes: the index already guarantees the match, so ordering only
sorts — earliest match, then shortest, then `(binary, addr)`. That last tiebreak
is load-bearing: without it, a name shared by two binaries (`main`, `textdomain`,
most of libc) ties on every other field and Python falls through to comparing the
hit objects, which raises `TypeError`.

**Phase 3 — cross-binary linking (done).** Each binary's imports and exports go
into the project index alongside its functions and strings (`KIND_IMPORT`,
`KIND_EXPORT`), and following an import resolves it to the binary that provides
it.

Following a call to `strcmp` used to reach the PLT/extern entry and stop:
Hex-Rays has nothing to decompile because the code lives in a library this
binary only references. Now `_follow_import` checks whether the target address is
one of this binary's import stubs, asks the index who exports that name, and
switches there. From an `echo` + `libc` project, Enter on `strrchr(a1, 47)` lands
on `strrchr` at `0xaf960` in libc.

Three things this depends on:

*ELF symbol versioning.* The importer sees `strrchr@@GLIBC_2.2.5`; the provider
may export `strrchr`, `strrchr@GLIBC_2.2.5`, or the versioned spelling. Comparing
raw names resolves almost nothing, so `domain.link_name()` cuts at the first `@`
and both sides meet on the bare symbol. `Linkage.raw` keeps what IDA reported,
which is what the listing shows.

*Exact match, not substring.* The join uses `ProjectIndex.exact()`, not
`search()`: an import must bind to its own name, or `read` picks up `pread`,
`read_line` and `thread_start`. Exact match also works below the 3-character
trigram floor, which matters — plenty of real exports are one or two characters.

*Resolution reads the index, not a worker.* A provider resolves while its worker
is evicted; that is the reason the index is on disk. The switch itself then pages
the provider in through the normal pool path.

The reverse direction is in the xrefs dialog. `xrefs_to` only ever sees the
current database, so an exported function looks unused from the inside even when
half the project calls it. `_foreign_importers` appends the project binaries that
IMPORT the symbol — read from the on-disk index, so a caller shows up whether or
not its worker is resident — and choosing one carries a `(binary, addr)` payload
through the same switch path as a search hit, hop included. From libc's
`strrchr`: `0000C318  import  [echo] strrchr`.

It only fires for a symbol this binary actually exports. A local name that
happens to collide with another binary's import is not a caller of ours.

When nothing in the project provides the symbol, `_follow_import` declines and
the local navigation proceeds — landing on the stub is still the honest answer,
and a single-binary session behaves exactly as before.

This does not retire the **PLT/import-stub** backlog item. An import whose
provider isn't in the project still lands on a stub, and that case wants the
original fix — recognise the thunk and present it as an import ("strcmp —
imported, provider not in project") rather than surfacing a decompiler error.
Phase 3 covers the valuable half; stub *presentation* stays worth doing.

**Phase 4 — polish (mostly done).**

*Nav history is per-binary, with a cross-binary way back.* Each binary keeps its
own stack in `BinaryState`; switching restores it (addresses outlive the worker,
so it survives eviction). But phase 3 made a cross-binary jump an ordinary
action, and arriving in a binary whose history is empty made it a one-way door.
`_switch_then_goto` now records the binary it came FROM, and `action_back` falls
through to that hop once local history is spent. So Esc walks back through the
function you were in, then the binary you were in. Manual switching (Ctrl+O)
records nothing — that's not navigation.

*Pre-warm follows the linkage graph, not list order.* When a binary finishes
indexing, `_prewarm_provider` warms the binary that provides the most of its
imports — where a follow is most likely to take you, so its startup is paid
before you ask. `WorkerPool.prewarm()` refuses rather than evicting: spending a
binary you visited on one you haven't is a straight downgrade, and it would throw
away that binary's caches too. At a tight budget pre-warm simply does nothing. It
estimates the cost of a not-yet-spawned worker from the largest resident one,
which is the only evidence available; if the estimate proves wrong, the
speculative worker is the one that goes.

*Driving a project.* `pane spawn --project FILE [--open BIN]` opens a project
pane. `binaries` lists the inventory (active / resident / indexed count / where
Esc returns to) and `switch {binary,addr?}` makes another one active — with an
address it goes through the same path as a search hit, so it records a hop. The
state snapshot now carries `binary` and `hops`.

Still open: project-level persistence (remember the active binary and each
binary's position across sessions — today a fresh launch auto-lands).


## Headerless blobs

An ELF/PE/Mach-O says what it is, so IDA figures it out. A raw firmware dump
says nothing, and IDA falls back to **x86 at address 0** — which on an ARM image
analyses to zero functions. It doesn't fail; it just quietly gives you nothing.

Open one without saying anything and idatui asks, the way interactive IDA does
when no loader matches:

    ┌ unrecognised file — processor? (20) ────────────────┐
    │ fw.bin  (32,768 bytes) — no loader matched; without │
    │ a processor IDA assumes x86 at 0                    │
    │ arm                                                 │
    │   arm          ARM / AArch64 — little-endian        │
    │   armb         ARM — big-endian                     │
    │ load address, e.g. 0x8000000 (blank = 0)            │
    │ Enter accept · Tab base address · Esc load as IDA   │
    └─────────────────────────────────────────────────────┘

Every name in that list is verified against a real IDA by `tools/verify_procs.py`
(open a scratch blob with `-p<name>`, read back `inf_get_procname()`). A wrong
name is *rejected* — rc=4, nothing useful said — which would be a dead end handed
out from inside the dialog meant to rescue you. Two of the first twenty were
wrong: `h8` and `sparc` are module filenames (`procs/h8.so`), not processor
names; the real ones are `h8300` and `sparcb`/`sparcl`. The aliases people reach
for first — `arm64`, `aarch64`, `mips`, `m68k` — are all invalid too, so they
live in the human labels where the filter still finds them.

`formats.sniff()` decides whether to ask, and only recognises formats IDA
definitely handles (ELF, PE, Mach-O, dex, wasm, ar, COFF, Intel HEX, S-records).
Being cautious the wrong way costs one dismissible dialog; being cautious the
other way is the silent-nothing case we're trying to kill. Esc loads it the way
IDA would have, because the sniff can be wrong.

A text container is only accepted when the whole header is printable — a raw
image whose first byte happens to be `:` is far likelier than Intel HEX.

The dialog is skipped when the answer already exists: options on the command
line, options in the project, or a database that already records them.

Or say it up front:

    ida-tui fw.bin --processor arm --base 0x8000000

or per binary in a project, which is where it belongs for a multi-image firmware:

    {"binaries": [
       {"path": "bootloader.bin", "processor": "arm",  "base": "0x0"},
       {"path": "app.bin",        "processor": "arm",  "base": "0x8000000"},
       {"path": "dsp.bin",        "processor": "mipsb", "base": "0x10000000"}
    ]}

`base` is written the way you'd say it (`0x8000000`, any base, int or string).
IDA's own `-b` switch is in **paragraphs** — `-b1000` loads at 0x10000 — so the
conversion happens in `BinaryRef.load_args`, and a base that isn't 16-byte
aligned is rejected rather than silently landing somewhere else. `ida_args`
passes anything else through untouched.

### When analysis finds nothing

Zero functions is what a raw image described wrongly looks like — right file,
wrong architecture, and IDA has no complaint to make about it. The app used to
fall through to the symbol picker, which had nothing to show, so you got empty
panes and a status reading "functions still loading…" long after loading had
finished.

Now it opens the listing at the start of the image (the bytes are there even when
no code was recognised) and the status bar carries the diagnosis for as long as
it's true:

    seg000 @ 0x0  [listing]  — no functions: wrong processor/base? Ctrl+L to reload

`Ctrl+L` re-asks. The database IDA already built has the old processor and base
baked into it and wins over any switches, so reloading means deleting it — hence
the confirmation, which tells you what you'd lose (and, in this case, that you'd
lose nothing).

Two things worth knowing:

* The options apply to the **first** open only. Afterwards the `.i64` records how
  the image was loaded, and passing the switches again makes IDA refuse to open
  it — so the worker skips them once a database exists.
* Changing the processor or base of a binary you've already analysed does
  nothing, because the database wins. Delete its `.i64` from the sidecar to
  reload with new options.

## Decisions

- Residency is a **memory budget** (default 25% of RAM, `memory_pct`), not a
  worker count; pinning supported.
- **Nav history is per-binary** to start.
- Search scope is a **toggle inside the existing palettes**.
- The project file is **explicit** (`--project`), not auto-discovered.
- IDA artifacts are **centralized** in the sidecar dir via staging.

## Open questions

- Re-staging a changed source drops its DB (renames included) — right call, but
  it should warn loudly.
- Should `Esc`/back ever cross binaries (global history)? Deferred.
- Pre-warm policy: analyze the whole project up front, or purely on demand?