aboutsummaryrefslogtreecommitdiffstats
path: root/docs
diff options
context:
space:
mode:
authorblasty <blasty@local>2026-07-25 14:33:05 +0200
committerblasty <blasty@local>2026-07-25 14:33:05 +0200
commit27fd6fb26c16e1c8fc6e21ebb116aa9127b72520 (patch)
tree0cb3689b2c570d7c92edfd6061e7526720ff2a45 /docs
parentexit: ask before quitting with unsaved database changes (diff)
downloadida-tui-27fd6fb26c16e1c8fc6e21ebb116aa9127b72520.tar.gz
ida-tui-27fd6fb26c16e1c8fc6e21ebb116aa9127b72520.tar.xz
ida-tui-27fd6fb26c16e1c8fc6e21ebb116aa9127b72520.zip
projects: project-wide symbol search over a SQLite FTS5 index (phase 2)
idatui/index.py — one on-disk index (<sidecar>/idx/project.db) over every binary in a project, so search works for binaries whose worker isn't running. Indexing choice, measured rather than guessed: * SQLite FTS5 with the TRIGRAM tokenizer — stdlib, no dependency (nothing else was installed and nothing is needed), and unlike a prefix index it matches arbitrary substrings, which is what symbol names and string bodies need. * 300k-entry corpus: 1.9 ms per query vs 11.8 ms for a Python scan and 28.9 ms for plain LIKE; 0.2 ms per incremental insert. * Size was the stated worry and turned out not to bite: bash contributes 5.9k entries / 0.15MB of text, libcrypto.so.3 30.7k / 0.52MB. At ~5.7x the text a 20-binary project is ~12-23MB — against .i64 files already in the sidecar (libcrypto's alone is 72MB), roughly 1% of what the project already costs. The reason to be on disk is residency, not size. * Trigram can't answer queries under 3 chars and returns nothing rather than erroring, so search() falls back to LIKE — otherwise incremental typing would look broken until the third keystroke. Wiring: after a binary's functions load, its symbols + strings are folded into the index (skipped when the source's size/mtime is unchanged). Ctrl+N gains a scope toggle on F2 — not ctrl+a, which the focused Input binds to "home" so it never reaches the palette. Project scope narrows via the index then ranks with the existing _fuzzy, keeping the same feel; hits are prefixed with their binary, and choosing one elsewhere switches binary and jumps to it. Also fixes another instance of the Textual-markup trap: the palette titles ate "[project]" as a style tag (same class of bug as the status bar), so the pal titles are markup=False now. tests/test_index.py: 24 stdlib checks — substring/case-insensitive matching, kind filter, the <3 char fallback, multi-binary search, per-binary incremental reindex, staleness, forget, persistence. Suite 191/0. Strings (") still needs the same scope toggle; the index already carries them.
Diffstat (limited to 'docs')
-rw-r--r--docs/PROJECTS.md31
1 files changed, 28 insertions, 3 deletions
diff --git a/docs/PROJECTS.md b/docs/PROJECTS.md
index 1f39c05..8af1dd9 100644
--- a/docs/PROJECTS.md
+++ b/docs/PROJECTS.md
@@ -122,9 +122,34 @@ index, reopen the entry. A binary whose worker is still resident restores
instantly (its `Program` and index are still in memory); an evicted one comes
back with a fresh worker but keeps its nav history, since that is just addresses.
-**Phase 2 — index cache + project-wide search.** Per-binary index (functions,
-strings) persisted after first open, keyed by source size+mtime. Scope toggle in
-the symbol/strings palettes, working for never-opened binaries.
+**Phase 2 — index cache + project-wide search. (symbols done)**
+`idatui/index.py` keeps one **SQLite FTS5 trigram** index at
+`<sidecar>/idx/project.db`, populated per binary after its functions load and
+re-done only when the source's size/mtime changes.
+
+Why that and not a library: nothing needed installing (FTS5 + the trigram
+tokenizer are stdlib), and trigram indexes arbitrary *substrings*, which is what
+symbol names and string bodies need. Measured on 300k entries: **1.9 ms** per
+query vs 11.8 ms for a Python scan and 28.9 ms for plain `LIKE`; 0.2 ms per
+incremental insert.
+
+Size turned out to be a non-issue. Real binaries: `bash` = 5.9k entries /
+0.15 MB of text, `libcrypto.so.3` = 30.7k / 0.52 MB. The index runs ~5.7x the
+text, so a 20-binary project lands around 12-23 MB — next to the `.i64` files
+already in the sidecar (libcrypto's alone is 72 MB) that is ~1% of what the
+project already costs. The reason to keep it on disk is **residency, not size**:
+search has to work for binaries whose worker isn't running.
+
+Querying is two-stage: the trigram index narrows across every binary, then the
+existing `_fuzzy` ranks what's left, so project scope keeps the same
+fuzzy-subsequence feel as local scope. Queries shorter than 3 characters fall
+back to `LIKE` — a trigram index silently returns *nothing* below that, which
+would make incremental typing look broken until the third keystroke.
+
+`Ctrl+N` gains a scope toggle on **F2** (not `ctrl+a`: the focused `Input` binds
+that to `home`). Project-scope hits are prefixed with their binary, and choosing
+one in another binary switches to it and jumps. **Strings (`"`) still needs the
+same toggle** — the index already carries them.
**Phase 3 — cross-binary linking.** Import/export index; "who in the project
calls this export"; follow an import stub in A into its implementation in B.