1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
|
# Projects — grouping many binaries under one target
RE'ing a real target means several binaries at once: a daemon, the libraries it
links, maybe a loader. IDA has no notion of this (one IDB, one window); Ghidra
does. A **project** groups binaries so you can switch between them instantly,
search across all of them, and (later) follow calls from one into another.
## The constraint that shapes everything
`idatui/worker.py` is `serve(sock, binpath)` — **one worker process holds exactly
one database** (idalib is main-thread-only and single-DB). So N binaries = N
worker processes, each with the analyzed DB resident.
Measured cost (this box, `targets/`):
| binary | size | `.i64` | worker RSS / PSS |
|--------|------|--------|------------------|
| `bash` | 1.2 MB | 14 MB | **126 MB / 117 MB** |
| `libcrypto.so.3` | 5.9 MB | 72 MB | several hundred MB (DB working set dominates) |
PSS ≈ RSS, so workers share almost nothing — the Nth worker costs roughly its
full footprint. A ~120 MB baseline (Python + the idalib kernel) dominates for
small binaries; the DB working set dominates for big ones.
**Therefore residency is budgeted by memory, not by a fixed worker count** — a
count is the wrong knob when one project holds both a 50 KB helper and a 6 MB
crypto library.
## The key split
Two capabilities that feel like one, but aren't:
1. **Switching** to a binary needs a *live worker*.
2. **Searching across** binaries does *not* — if a per-binary index (functions,
strings, imports/exports) is cached on disk.
That split is the unlock: project-wide search stays instant across every binary,
including ones never opened this session, and only *jumping* to a hit costs a
worker spawn.
## Layout
The project file is explicit (`--project router-fw.json`); everything IDA
generates lives in a sidecar dir beside it, so **source trees stay pristine**:
```
router-fw.json # the project file
router-fw.idatui.d/
bin/httpd # staged copy of the source binary
bin/httpd.i64 # IDA's DB + scratch land here automatically
idx/httpd.json # cached index (phase 2)
```
Binaries are **staged** into `bin/` and IDA opens the staged file, so the `.i64`
and all the `.id0/.id1/.id2/.nam/.til` scratch are created there rather than next
to the original. (Today they pile up beside the source: `targets/` holds ~244 MB
of IDA litter around ~13 MB of binaries, much of it stale wedge files from
hard-killed workers.) A source whose size/mtime no longer matches the staged copy
is re-staged, and its now-stale DB dropped.
Staging **copies** rather than hardlinks. A hardlink is free, but source and
staged would be one inode: an in-place rebuild (`cp newbuild /path/bin`
truncates rather than replacing) would silently swap the bytes under an already
analysed database, with nothing to detect it. A copy costs a fraction of the
`.i64` it will grow, and leaves the sidecar self-contained — the project still
opens after the sources go away (an unmounted firmware image, a cleaned build
tree). *The unit test caught this: with hardlinks an in-place rewrite never went
stale.*
```json
{
"name": "router-fw",
"binaries": [
{"path": "/abs/sbin/httpd", "label": "httpd"},
{"path": "../lib/libauth.so"}
],
"memory_pct": 25
}
```
Relative paths resolve against the project file. `label` defaults to the
basename and must be unique (it names the staged file).
## Runtime
- **`WorkerPool`** — one `WorkerClient` per binary, spawned lazily on first
switch, kept resident until the memory budget is exceeded, then LRU-evicted.
Eviction **saves the DB first**, so returning to a binary is a DB load, not a
re-analysis. Binaries can be pinned to stay resident.
- **`BinaryState`** — per binary: `client, program, nav, cur, func_index,
pref/active/split, filter`. Switching snapshots the current state and restores
the target's. `_after_reconnect` already does exactly this swap (client +
program, reload the index, re-open the entry) — switching reuses that seam.
- **Clean shutdown** — the worker currently does `close_database(save=False)` and
is hard-killed on exit, which is why wedge files accumulate. Projects need
save-on-evict and an orderly close anyway, so that gets fixed here.
## UI
- **Binary switcher** — a palette (mirroring `SymbolPalette`/`StringsPalette`)
listing each binary with resident/cold state and function count; Enter
switches.
- **Status bar** shows the active binary.
- **Scope toggle** — the existing symbol (`Ctrl+N`) and strings (`"`) palettes
gain a this-binary ↔ whole-project toggle rather than new keys, so there's one
mental model. Project-scope hits are prefixed with the binary label; Enter
switches binary *and* jumps.
## Phases
**Phase 1 — project model + switching. DONE.** Project file + staging
(`idatui/project.py`), `WorkerPool` with budget eviction / save-on-evict /
clean shutdown (`idatui/pool.py`), `BinaryState` snapshot+restore and the switch
itself, the `Ctrl+O` switcher palette, the active binary in the status line, and
`--project` (which creates the project when given binaries). One active binary;
no cross-binary search yet.
Project mode is **additive**: with no `--project` the app is byte-for-byte the
single-binary tool it was, which is what keeps the 167-check pilot honest.
Switching reuses the `_after_reconnect` shape — swap client+program, rebuild the
index, reopen the entry. A binary whose worker is still resident restores
instantly (its `Program` and index are still in memory); an evicted one comes
back with a fresh worker but keeps its nav history, since that is just addresses.
**Phase 2 — index cache + project-wide search. (symbols done)**
`idatui/index.py` keeps one **SQLite FTS5 trigram** index at
`<sidecar>/idx/project.db`, populated per binary after its functions load and
re-done only when the source's size/mtime changes.
Why that and not a library: nothing needed installing (FTS5 + the trigram
tokenizer are stdlib), and trigram indexes arbitrary *substrings*, which is what
symbol names and string bodies need. Measured on 300k entries: **1.9 ms** per
query vs 11.8 ms for a Python scan and 28.9 ms for plain `LIKE`; 0.2 ms per
incremental insert.
Size turned out to be a non-issue. Real binaries: `bash` = 5.9k entries /
0.15 MB of text, `libcrypto.so.3` = 30.7k / 0.52 MB. The index runs ~5.7x the
text, so a 20-binary project lands around 12-23 MB — next to the `.i64` files
already in the sidecar (libcrypto's alone is 72 MB) that is ~1% of what the
project already costs. The reason to keep it on disk is **residency, not size**:
search has to work for binaries whose worker isn't running.
Querying is two-stage: the trigram index narrows across every binary, then the
existing `_fuzzy` ranks what's left, so project scope keeps the same
fuzzy-subsequence feel as local scope. Queries shorter than 3 characters fall
back to `LIKE` — a trigram index silently returns *nothing* below that, which
would make incremental typing look broken until the third keystroke.
Both palettes gain a scope toggle on **F2** (not `ctrl+a`: the focused `Input`
binds that to `home`) — `Ctrl+N` for symbols and `"` for strings. Project-scope
hits are prefixed with their binary, and choosing one in another binary switches
to it and jumps. Project scope caps at 60 rows; it saturates any cap because
every binary contributes, and a long list is what makes arrowing feel sluggish.
Ranking, both palettes: the index already guarantees the match, so ordering only
sorts — earliest match, then shortest, then `(binary, addr)`. That last tiebreak
is load-bearing: without it, a name shared by two binaries (`main`, `textdomain`,
most of libc) ties on every other field and Python falls through to comparing the
hit objects, which raises `TypeError`.
**Phase 3 — cross-binary linking.** Index what each binary *imports* and
*exports*, then join them: "who in the project calls this export", and following
an import in A lands on the real implementation in B.
This is where the standing **PLT/import-stub** backlog item mostly goes away.
Today, following a call to `strcmp` reaches the PLT thunk and stops at
`extrn strcmp:near` — Hex-Rays has nothing to decompile because the code lives in
a library the app never opened. With an export index, that stub resolves to
`libc`'s implementation *when that library is in the project*, so the dead end
becomes a normal cross-binary jump.
It does not retire the item completely: a single-binary session, or an import
whose provider isn't in the project, still lands on a stub, and that case still
wants the original fix — recognise the thunk and present it as an import
("strcmp — imported, provider not in project") instead of surfacing a decompiler
error. So phase 3 covers the valuable half; stub *presentation* stays worth doing
on its own.
**Phase 4 — polish.** Background pre-warm, nav-history policy (per-binary to
start; a global stack is a later question), RPC/`drive` verbs (`binaries`,
`switch`), project-level persistence.
## Decisions
- Residency is a **memory budget** (default 25% of RAM, `memory_pct`), not a
worker count; pinning supported.
- **Nav history is per-binary** to start.
- Search scope is a **toggle inside the existing palettes**.
- The project file is **explicit** (`--project`), not auto-discovered.
- IDA artifacts are **centralized** in the sidecar dir via staging.
## Open questions
- Re-staging a changed source drops its DB (renames included) — right call, but
it should warn loudly.
- Should `Esc`/back ever cross binaries (global history)? Deferred.
- Pre-warm policy: analyze the whole project up front, or purely on demand?
|
