stress paging on 10k-func binary; document scale constraints

Measured against libcrypto.so.3 (10,092 funcs, biggest 52,120 insns):
- list_* count cap ~700, disasm max_instructions cap ~500, then SILENT
  collapse to 10 (not clamped) -> must clamp client-side (use 500).
- pagination must advance by len(data); next_offset=offset+count skips data.
- disasm offset paging is O(offset): 6ms@0 -> 180ms@50k, no resumable cursor
  -> domain layer must cache windows + prefetch + over-fetch.
- include_total scans whole func (~217ms on monster) -> fetch once, cache.
- decompile hard-fails on huge funcs as a soft error (code=null) -> handle.
- idb_open needs a writable path for the .i64.

tests/stress_paging.py: durable paging benchmark harness
docs/PAGING_FINDINGS.md: constraints that drive the paging layer

3 files changed, 254 insertions, 0 deletions

diff --git a/.gitignore b/.gitignore
index 2b1bd85..7b08e88 100644
--- a/.gitignore
+++ b/.gitignore
@@ -10,3 +10,6 @@ build/
 *.id2
 *.nam
 *.til
+
+# large analysis targets (copied in, not source)
+targets/
diff --git a/docs/PAGING_FINDINGS.md b/docs/PAGING_FINDINGS.md
new file mode 100644
index 0000000..1c7abff
--- /dev/null
+++ b/docs/PAGING_FINDINGS.md
@@ -0,0 +1,101 @@
+# Paging & scale findings (the "millions of lines" problem)
+
+Measured against a real target: `libcrypto.so.3` (5.7 MB, **10,092 functions**,
+biggest function **52,120 instructions**). These constraints drive the domain /
+paging layer. Reproduce with `tests/stress_paging.py --db <session>`.
+
+## Response shape (list_* / *_query tools)
+
+A query tool payload is a *list of per-query results*:
+
+```json
+{ "result": [ { "data": [ ... ], "next_offset": <int|null> } ] }
+```
+
+## Per-call caps are silent and dangerous
+
+Every `count` / `max_instructions` parameter is honored up to a cap, then
+**silently collapses to a tiny default (10)** — it does NOT clamp to the max.
+
+| tool        | param            | honored up to | above cap |
+|-------------|------------------|---------------|-----------|
+| `list_funcs`| `count`          | ~700          | → 10      |
+| `disasm`    | `max_instructions`| ~500         | → 10      |
+
+**Rule:** clamp page sizes client-side. Use `list` page ≤ **500**, disasm window
+≤ **500**. Never pass a large count hoping for clamping.
+
+## Pagination: advance by `len(data)`, NOT `next_offset`
+
+`next_offset` is just `offset + count`. If `count` exceeds the cap you get 10
+rows but `next_offset` still jumps by `count`, **skipping ~99% of the data**.
+Correct enumeration:
+
+```python
+offset = 0
+while True:
+    data = call("list_funcs", queries=[{"offset": offset, "count": 500}])[...]
+    yield from data
+    if len(data) < 500:   # short page => end
+        break
+    offset += len(data)   # advance by what we actually got
+```
+
+Full 10,092-function enumeration = 21 pages, **~3.1 s** (0.31 ms/func). Do this
+once in the background with a progress indicator; page lazily otherwise.
+
+## disasm offset paging is **O(offset)** — the main scroll hazard
+
+`disasm addr=F offset=N` re-walks from the function start, so latency grows with
+depth (window = 60 instrs):
+
+| offset | latency |
+|--------|---------|
+| 0      | ~6 ms   |
+| 1,000  | ~10 ms  |
+| 5,000  | ~23 ms  |
+| 20,000 | ~75 ms  |
+| 50,000 | ~180 ms |
+
+There is **no resumable cursor**: the payload's `cursor: {"next": N}` is just
+informational; `disasm` has no cursor input param. So O(offset) is unavoidable
+via this API.
+
+Implications for the domain/paging layer:
+- **Cache fetched windows** by `(func, offset)`. Re-scrolling a deep window
+  currently re-pays the full ~180 ms (no server-side cache); our cache makes
+  revisits instant.
+- **Prefetch ahead** in background threads (client is concurrency-safe) so the
+  next window is warm before the user reaches it.
+- **Over-fetch** up to the ~500 cap per call and slice locally for the viewport,
+  amortizing both RTT and the O(offset) walk during fast scrolling.
+- **This only bites pathological functions.** The two 52k-insn outliers are
+  almost certainly mis-analyzed data/unrolled tables; the next largest real
+  function is 2,748 instrs (deepest offset ≈ 10 ms). Typical scrolling is fine.
+
+## `include_total` is expensive — fetch once, cache
+
+Computing `total_instructions` scans the whole function: **~217 ms** on the 52k
+monster (vs ~10 ms without). Fetch total once when a function view opens (to size
+the scrollbar) and cache it; never request it per-scroll.
+
+disasm totals are **top-level** fields, not under `asm`:
+`total_instructions`, `instruction_count`, `cursor`.
+
+## Decompile: can hard-fail on huge functions (soft error)
+
+`decompile` on the 52k-insn function returns, in ~3 ms:
+`{"code": null, "error": "Decompilation failed at 0x3349c0"}` with
+`isError == false`. The client surfaces this as **data, not an exception**
+(correct). The pseudocode view must handle "decompilation failed" gracefully —
+fall back to the disassembly view or show an error panel.
+
+Normal decompile bodies are server-truncated with a `[N chars total]` marker
+(still to be solved for full-body display — see Phase 2).
+
+## Writable path requirement (operational)
+
+`idb_open` writes the `.i64` next to the input binary, so the path must be
+**writable**. Opening from read-only dirs (e.g. `/usr/lib`) fails with
+`"Failed to open database"`. Copy targets into a writable dir first
+(`targets/` in this repo).
diff --git a/tests/stress_paging.py b/tests/stress_paging.py
new file mode 100644
index 0000000..b821354
--- /dev/null
+++ b/tests/stress_paging.py
@@ -0,0 +1,150 @@
+#!/usr/bin/env python3
+"""Stress the 'many lines of text' problem: paging huge listings/functions.
+
+Validates the windowing strategy the TUI will use so we never hand a widget more
+than a viewport-sized slice. Prints concise stats only.
+
+    python3 tests/stress_paging.py --db <session_id>
+"""
+import os
+import statistics
+import sys
+import time
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+from idatui.client import IDAClient  # noqa: E402
+
+URL = "http://127.0.0.1:8745/mcp"
+# Verified server cap: list_* honors count up to ~700, then silently collapses
+# to a 10-item default. Clamp with margin.
+SAFE_PAGE = 500
+
+
+def ms(fn):
+    t = time.time()
+    r = fn()
+    return (time.time() - t) * 1e3, r
+
+
+def q0(payload):
+    """Unwrap a *_query/list_* payload: {'result':[{'data':[...],'next_offset':N}]}."""
+    res = payload.get("result", payload)
+    if isinstance(res, list):
+        res = res[0] if res else {}
+    return res.get("data", []), res.get("next_offset")
+
+
+def page_all_funcs(c, page=SAFE_PAGE):
+    """Enumerate the entire function list correctly: advance by len(data), NOT by
+    next_offset (which is just offset+count and skips over the per-call cap)."""
+    funcs, offset, pages, t0 = [], 0, 0, time.time()
+    while True:
+        payload = c.call("list_funcs", queries=[{"offset": offset, "count": page}])
+        data, _ = q0(payload)
+        funcs.extend(data)
+        pages += 1
+        if len(data) < page:  # short page => reached the end
+            break
+        offset += len(data)
+    return funcs, pages, (time.time() - t0) * 1e3
+
+
+def disasm_meta(c, addr):
+    # total_instructions is a TOP-LEVEL field, not under 'asm'.
+    payload = c.call("disasm", addr=addr, max_instructions=1, include_total=True)
+    return payload.get("total_instructions", payload.get("instruction_count"))
+
+
+def main(argv):
+    db = None
+    it = iter(argv)
+    for a in it:
+        if a == "--db":
+            db = next(it)
+    c = IDAClient(URL, db=db)
+    c.connect()
+    if db is None:
+        c.set_db(c.resolve_db())
+    print(f"db={c.db}  ({c.health().get('module')})")
+
+    # ---- 1. Full function list via cursor pagination ------------------- #
+    print("\n[1] page entire function list (cursor)")
+    funcs, pages, total_ms = page_all_funcs(c, page=SAFE_PAGE)
+    n = len(funcs)
+    print(f"  {n} functions in {pages} pages, {total_ms:.0f}ms "
+          f"({total_ms / max(n,1):.3f}ms/func)")
+
+    def sz(f):
+        s = f["size"]
+        return int(s, 16) if isinstance(s, str) else s
+    biggest = sorted(funcs, key=sz, reverse=True)[:5]
+    print("  biggest by bytes:")
+    for f in biggest:
+        print(f"    {f['addr']:>12}  {sz(f):#8x}  {f['name']}")
+
+    # ---- 2. Instruction counts of the biggest -------------------------- #
+    print("\n[2] instruction totals of biggest funcs")
+    fattest = None
+    fattest_n = 0
+    for f in biggest:
+        dt, total = ms(lambda f=f: disasm_meta(c, f["addr"]))
+        print(f"    {f['name']:<28} {str(total):>8} insns  (meta {dt:.1f}ms)")
+        if isinstance(total, int) and total > fattest_n:
+            fattest, fattest_n = f, total
+
+    # ---- 3. Windowed paging INTO the fattest function ------------------ #
+    print(f"\n[3] window-scroll fattest func {fattest['name']} ({fattest_n} insns)")
+    WIN = 60  # a viewport
+    offsets = list(range(0, max(fattest_n - WIN, 1), max((fattest_n // 12), 1)))
+    times = []
+    for off in offsets:
+        dt, payload = ms(lambda off=off: c.call(
+            "disasm", addr=fattest["addr"], offset=off, max_instructions=WIN))
+        lines = payload.get("asm", {}).get("lines", [])
+        times.append(dt)
+        assert len(lines) <= WIN, f"got {len(lines)} > window {WIN}"
+    print(f"  {len(offsets)} windowed reads @win={WIN}: "
+          f"min={min(times):.1f} med={statistics.median(times):.1f} "
+          f"max={max(times):.1f} ms  (payload capped to <= {WIN} lines)")
+
+    # ---- 3b. O(offset) latency curve (the key hazard) ------------------ #
+    print("\n[3b] disasm offset-latency curve (offset paging is O(offset))")
+    for off in [0, 1000, 5000, 20000, min(50000, fattest_n - WIN)]:
+        if off < 0:
+            continue
+        dt, _ = ms(lambda off=off: c.call(
+            "disasm", addr=fattest["addr"], offset=off, max_instructions=WIN))
+        print(f"    offset={off:>6}: {dt:6.1f}ms")
+
+    # ---- 4. Simulated 'hold page-down' throughput ---------------------- #
+    print("\n[4] rapid sequential scroll (hold page-down)")
+    reads, t0, off = 0, time.time(), 0
+    while time.time() - t0 < 2.0:
+        c.call("disasm", addr=fattest["addr"], offset=off, max_instructions=WIN)
+        off = (off + WIN) % max(fattest_n - WIN, 1)
+        reads += 1
+    dur = time.time() - t0
+    print(f"  {reads} windowed reads in {dur:.1f}s = {reads / dur:.0f} reads/s "
+          f"(~{reads / dur * WIN:.0f} lines/s)")
+
+    # ---- 5. Decompile: truncation AND hard-failure on monsters --------- #
+    print("\n[5] decompile on fattest func (may fail on huge funcs)")
+    dt, payload = ms(lambda: c.call("decompile", addr=fattest["addr"]))
+    code = payload.get("code") if isinstance(payload, dict) else None
+    if not code:
+        err = payload.get("error") if isinstance(payload, dict) else payload
+        print(f"  decompile {dt:.0f}ms -> FAILED (soft error, not raised): {err}")
+    else:
+        marker = "chars total]" in code[-40:]
+        print(f"  decompile {dt:.0f}ms, returned {len(code)} chars, "
+              f"server-truncated={marker}")
+        if marker:
+            print(f"    tail: ...{code[-60:]!r}")
+
+    c.close()
+    print("\nOK")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv[1:]))