diff options
| author | blasty <blasty@local> | 2026-07-25 14:33:05 +0200 |
|---|---|---|
| committer | blasty <blasty@local> | 2026-07-25 14:33:05 +0200 |
| commit | 27fd6fb26c16e1c8fc6e21ebb116aa9127b72520 (patch) | |
| tree | 0cb3689b2c570d7c92edfd6061e7526720ff2a45 /docs | |
| parent | exit: ask before quitting with unsaved database changes (diff) | |
| download | ida-tui-27fd6fb26c16e1c8fc6e21ebb116aa9127b72520.tar.gz ida-tui-27fd6fb26c16e1c8fc6e21ebb116aa9127b72520.tar.xz ida-tui-27fd6fb26c16e1c8fc6e21ebb116aa9127b72520.zip | |
projects: project-wide symbol search over a SQLite FTS5 index (phase 2)
idatui/index.py — one on-disk index (<sidecar>/idx/project.db) over every binary
in a project, so search works for binaries whose worker isn't running.
Indexing choice, measured rather than guessed:
* SQLite FTS5 with the TRIGRAM tokenizer — stdlib, no dependency (nothing else
was installed and nothing is needed), and unlike a prefix index it matches
arbitrary substrings, which is what symbol names and string bodies need.
* 300k-entry corpus: 1.9 ms per query vs 11.8 ms for a Python scan and 28.9 ms
for plain LIKE; 0.2 ms per incremental insert.
* Size was the stated worry and turned out not to bite: bash contributes 5.9k
entries / 0.15MB of text, libcrypto.so.3 30.7k / 0.52MB. At ~5.7x the text a
20-binary project is ~12-23MB — against .i64 files already in the sidecar
(libcrypto's alone is 72MB), roughly 1% of what the project already costs. The
reason to be on disk is residency, not size.
* Trigram can't answer queries under 3 chars and returns nothing rather than
erroring, so search() falls back to LIKE — otherwise incremental typing would
look broken until the third keystroke.
Wiring: after a binary's functions load, its symbols + strings are folded into
the index (skipped when the source's size/mtime is unchanged). Ctrl+N gains a
scope toggle on F2 — not ctrl+a, which the focused Input binds to "home" so it
never reaches the palette. Project scope narrows via the index then ranks with
the existing _fuzzy, keeping the same feel; hits are prefixed with their binary,
and choosing one elsewhere switches binary and jumps to it.
Also fixes another instance of the Textual-markup trap: the palette titles ate
"[project]" as a style tag (same class of bug as the status bar), so the pal
titles are markup=False now.
tests/test_index.py: 24 stdlib checks — substring/case-insensitive matching, kind
filter, the <3 char fallback, multi-binary search, per-binary incremental
reindex, staleness, forget, persistence. Suite 191/0.
Strings (") still needs the same scope toggle; the index already carries them.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/PROJECTS.md | 31 |
1 files changed, 28 insertions, 3 deletions
diff --git a/docs/PROJECTS.md b/docs/PROJECTS.md index 1f39c05..8af1dd9 100644 --- a/docs/PROJECTS.md +++ b/docs/PROJECTS.md @@ -122,9 +122,34 @@ index, reopen the entry. A binary whose worker is still resident restores instantly (its `Program` and index are still in memory); an evicted one comes back with a fresh worker but keeps its nav history, since that is just addresses. -**Phase 2 — index cache + project-wide search.** Per-binary index (functions, -strings) persisted after first open, keyed by source size+mtime. Scope toggle in -the symbol/strings palettes, working for never-opened binaries. +**Phase 2 — index cache + project-wide search. (symbols done)** +`idatui/index.py` keeps one **SQLite FTS5 trigram** index at +`<sidecar>/idx/project.db`, populated per binary after its functions load and +re-done only when the source's size/mtime changes. + +Why that and not a library: nothing needed installing (FTS5 + the trigram +tokenizer are stdlib), and trigram indexes arbitrary *substrings*, which is what +symbol names and string bodies need. Measured on 300k entries: **1.9 ms** per +query vs 11.8 ms for a Python scan and 28.9 ms for plain `LIKE`; 0.2 ms per +incremental insert. + +Size turned out to be a non-issue. Real binaries: `bash` = 5.9k entries / +0.15 MB of text, `libcrypto.so.3` = 30.7k / 0.52 MB. The index runs ~5.7x the +text, so a 20-binary project lands around 12-23 MB — next to the `.i64` files +already in the sidecar (libcrypto's alone is 72 MB) that is ~1% of what the +project already costs. The reason to keep it on disk is **residency, not size**: +search has to work for binaries whose worker isn't running. + +Querying is two-stage: the trigram index narrows across every binary, then the +existing `_fuzzy` ranks what's left, so project scope keeps the same +fuzzy-subsequence feel as local scope. Queries shorter than 3 characters fall +back to `LIKE` — a trigram index silently returns *nothing* below that, which +would make incremental typing look broken until the third keystroke. + +`Ctrl+N` gains a scope toggle on **F2** (not `ctrl+a`: the focused `Input` binds +that to `home`). Project-scope hits are prefixed with their binary, and choosing +one in another binary switches to it and jumps. **Strings (`"`) still needs the +same toggle** — the index already carries them. **Phase 3 — cross-binary linking.** Import/export index; "who in the project calls this export"; follow an import stub in A into its implementation in B. |
