From 27fd6fb26c16e1c8fc6e21ebb116aa9127b72520 Mon Sep 17 00:00:00 2001 From: blasty Date: Sat, 25 Jul 2026 14:33:05 +0200 Subject: projects: project-wide symbol search over a SQLite FTS5 index (phase 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit idatui/index.py — one on-disk index (/idx/project.db) over every binary in a project, so search works for binaries whose worker isn't running. Indexing choice, measured rather than guessed: * SQLite FTS5 with the TRIGRAM tokenizer — stdlib, no dependency (nothing else was installed and nothing is needed), and unlike a prefix index it matches arbitrary substrings, which is what symbol names and string bodies need. * 300k-entry corpus: 1.9 ms per query vs 11.8 ms for a Python scan and 28.9 ms for plain LIKE; 0.2 ms per incremental insert. * Size was the stated worry and turned out not to bite: bash contributes 5.9k entries / 0.15MB of text, libcrypto.so.3 30.7k / 0.52MB. At ~5.7x the text a 20-binary project is ~12-23MB — against .i64 files already in the sidecar (libcrypto's alone is 72MB), roughly 1% of what the project already costs. The reason to be on disk is residency, not size. * Trigram can't answer queries under 3 chars and returns nothing rather than erroring, so search() falls back to LIKE — otherwise incremental typing would look broken until the third keystroke. Wiring: after a binary's functions load, its symbols + strings are folded into the index (skipped when the source's size/mtime is unchanged). Ctrl+N gains a scope toggle on F2 — not ctrl+a, which the focused Input binds to "home" so it never reaches the palette. Project scope narrows via the index then ranks with the existing _fuzzy, keeping the same feel; hits are prefixed with their binary, and choosing one elsewhere switches binary and jumps to it. Also fixes another instance of the Textual-markup trap: the palette titles ate "[project]" as a style tag (same class of bug as the status bar), so the pal titles are markup=False now. tests/test_index.py: 24 stdlib checks — substring/case-insensitive matching, kind filter, the <3 char fallback, multi-binary search, per-binary incremental reindex, staleness, forget, persistence. Suite 191/0. Strings (") still needs the same scope toggle; the index already carries them. --- docs/PROJECTS.md | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) (limited to 'docs') diff --git a/docs/PROJECTS.md b/docs/PROJECTS.md index 1f39c05..8af1dd9 100644 --- a/docs/PROJECTS.md +++ b/docs/PROJECTS.md @@ -122,9 +122,34 @@ index, reopen the entry. A binary whose worker is still resident restores instantly (its `Program` and index are still in memory); an evicted one comes back with a fresh worker but keeps its nav history, since that is just addresses. -**Phase 2 — index cache + project-wide search.** Per-binary index (functions, -strings) persisted after first open, keyed by source size+mtime. Scope toggle in -the symbol/strings palettes, working for never-opened binaries. +**Phase 2 — index cache + project-wide search. (symbols done)** +`idatui/index.py` keeps one **SQLite FTS5 trigram** index at +`/idx/project.db`, populated per binary after its functions load and +re-done only when the source's size/mtime changes. + +Why that and not a library: nothing needed installing (FTS5 + the trigram +tokenizer are stdlib), and trigram indexes arbitrary *substrings*, which is what +symbol names and string bodies need. Measured on 300k entries: **1.9 ms** per +query vs 11.8 ms for a Python scan and 28.9 ms for plain `LIKE`; 0.2 ms per +incremental insert. + +Size turned out to be a non-issue. Real binaries: `bash` = 5.9k entries / +0.15 MB of text, `libcrypto.so.3` = 30.7k / 0.52 MB. The index runs ~5.7x the +text, so a 20-binary project lands around 12-23 MB — next to the `.i64` files +already in the sidecar (libcrypto's alone is 72 MB) that is ~1% of what the +project already costs. The reason to keep it on disk is **residency, not size**: +search has to work for binaries whose worker isn't running. + +Querying is two-stage: the trigram index narrows across every binary, then the +existing `_fuzzy` ranks what's left, so project scope keeps the same +fuzzy-subsequence feel as local scope. Queries shorter than 3 characters fall +back to `LIKE` — a trigram index silently returns *nothing* below that, which +would make incremental typing look broken until the third keystroke. + +`Ctrl+N` gains a scope toggle on **F2** (not `ctrl+a`: the focused `Input` binds +that to `home`). Project-scope hits are prefixed with their binary, and choosing +one in another binary switches to it and jumps. **Strings (`"`) still needs the +same toggle** — the index already carries them. **Phase 3 — cross-binary linking.** Import/export index; "who in the project calls this export"; follow an import stub in A into its implementation in B. -- cgit v1.3.1-sl0p