path: root/TUI_DRIVING_BLUEPRINT.md

# Blueprint: a generic TUI-driving layer

How to bring the "spawn a pane, drive the real TUI over a socket, read structured
state back" experience (built for **idatui**, see `docs/RPC.md`) to *other* TUI
software — most notably **gdb**, but the pattern generalizes to any terminal app
(vim, k9s, lazygit, htop, a REPL, …).

This is a design reference, not code. It captures the invariants worth keeping,
the parts that are reusable as-is, and the one hard problem you re-solve per
target (knowing when the UI has *settled*).

---

## 1. The reference implementation (what we're generalizing)

idatui is a Textual app we **own**, so we embedded an RPC server directly in its
asyncio event loop. The stack that emerged:

```
 tmux pane A (viewers see this)          tmux pane B (the agent)
 ┌────────────────────────────┐  unix    ┌────────────────────────┐
 │ the TUI, rendering normally │◀socket──▶│ ergonomic client (drive)│
 │ + in-process RPC server     │  JSONL   │  + pane manager         │
 └────────────────────────────┘          └────────────────────────┘
```

Pieces (all reusable ideas): a **unix-socket JSONL server**, **three method
tiers** (raw keys / semantic verbs / structured introspection), a **settle**
primitive (block until the UI is quiescent before returning state), a
**single-driver gate**, a **tmux pane manager** (spawn/stop/list + backend
auto-start + readiness polling), and an **ergonomic client** (auto socket
discovery, terse text, composite gestures).

---

## 2. Invariants worth preserving (why it felt good)

1. **Drive the *real* UI, not a hidden API.** Every action goes through the same
   input path a keyboard would, so the tmux pane shows it happening (livestream /
   debuggability). Bypassing the renderer to mutate state directly is faster but
   defeats the purpose.
2. **Three tiers, always.** `raw` (keystrokes) for fidelity/coverage; `semantic`
   verbs (typed, with an optional per-char delay for the hand-typed look) for
   ergonomics; `introspection` (structured reads) so the agent reasons over data,
   not screen-scraping.
3. **Every mutating call returns *after* the UI settled**, and returns fresh
   state. The driver never acts on a half-rendered frame.
4. **Single-driver by default.** One client at a time; a second concurrent
   connection is refused. No multi-driver coordination to reason about.
5. **Self-documenting surface.** A `methods` call returns the verb table; a
   `state`/`screen` call returns what's happening. The agent can discover the API.
6. **Lifecycle is first-class.** Spawn a pane, wait until *ready*, drive, tear it
   down; never leak panes/processes. Auto-start the backing daemon if it's down.
7. **An ergonomic layer on top.** The raw JSON transport is correct but verbose;
   a thin CLI that auto-resolves the socket and prints terse text is what you
   actually use all day.

---

## 3. Generic architecture

Split into a **reusable core** and a per-target **Adapter**. Only the adapter
changes between targets.

```
        ┌──────────────── reusable core ────────────────┐
client ─┤ transport (unix sock, JSONL, single-driver)   │
 (drive)│ dispatch (method table, tiers, error framing) │
        │ pane manager (tmux spawn/stop/list, readiness) │
        └───────────────────────┬───────────────────────┘
                                 │  Adapter interface
                    ┌────────────┴────────────┐
                    │  Target Adapter          │  ← the only per-target code
                    │  inject / settle /       │
                    │  snapshot / screen /     │
                    │  semantic verbs          │
                    └──────────────────────────┘
```

### The Adapter interface (the contract)

Everything a target must provide. Keep it small:

| method | purpose |
|--------|---------|
| `inject_keys(keys[])` | push keystrokes into the app's real input path |
| `inject_text(str, delay_ms)` | type a literal string (visible typing) |
| `settle(pred?, timeout) -> bool` | block until quiescent, then until `pred` (if given) |
| `snapshot() -> dict` | structured "where am I / what's on screen" state |
| `screen(fmt) -> {w,h,text}` | full render (plain / colored) of the pane |
| `verbs` | target-specific semantic ops (each = compose inject + settle + a predicate) |
| `ready() -> bool` | is the target drivable yet |
| `quit()` | graceful shutdown |

The core turns these into the wire method table and handles transport, framing,
the single-driver gate, and the pane lifecycle.

---

## 4. Target taxonomy — how much the app cooperates

The adapter's implementation depends entirely on how much access you get. Three
levels, best to worst:

### Level 1 — **Embedded** (you own or can patch the app)
Run an RPC server *inside* the app's event loop (idatui). Injection uses the
app's own key API; `settle` uses its message pump + task/worker state; `snapshot`
reads app state directly. **Highest fidelity, cheapest settle** (you have real
signals). Use when you can add ~200 lines to the target.

### Level 2 — **Control-channel** (app has a machine interface)
Many serious TUIs expose a second, structured channel alongside the visual one:
gdb (**GDB/MI** + a **Python API**), tmux (**control mode**, `-CC`), vim
(**channels / `--remote-expr`**), lldb, many REPLs. Strategy: the **visual** pane
is driven with keystrokes for fidelity; **semantic** verbs and **introspection**
go through the control channel, which is request/response (so quiescence is
*given* — a command is done when the channel replies). This is the sweet spot for
gdb (see §8).

### Level 3 — **Black-box PTY** (you can't modify or query it)
The universal fallback: the app is just bytes in / a screen out (vim without
channels, htop, an arbitrary curses app). You inject keystrokes into its PTY and
read state by scraping the rendered screen. **Settle is the hard part** (no
completion signal — you debounce on screen changes). Everything is a keystroke
macro + screen assertion.

> **Most targets are Level 2 or 3.** Design the core so an adapter can *mix*
> levels: e.g. gdb = Level 2 for `break/step/bt/read-mem`, Level 3 (screen
> scrape) for whatever the control channel doesn't expose.

---

## 5. The three hard problems, generically

### 5a. Input injection — "how do keystrokes get in"
- **Embedded:** call the app's key-press API (idatui used Textual's `_press_keys`,
  the same path its own tests use). Supports a `wait:<ms>` token → free typed delay.
- **Control-channel:** structured actions bypass keys entirely (gdb `-exec-*`);
  but *also* keep a keystroke path for TUI navigation.
- **Black-box:** **`tmux send-keys -t <pane> …`** is the universal injector — no
  PTY plumbing. It even names special keys (`Enter`, `C-c`, `Escape`). For a
  literal string, `send-keys -l 'text'`. For the typed aesthetic, send char-by-char
  with a sleep between.

### 5b. Settle / quiescence — "when did the UI finish reacting" *(the crux)*
Pick the strongest signal the target offers. A taxonomy, best → worst:

1. **Completion signal** *(embedded / control-channel).* The app tells you it's
   done: a worker/task queue drains (idatui: message pump + `WorkerManager`), or a
   request/response channel returns (`^done` in GDB/MI). Deterministic; prefer this.
2. **Predicate wait.** Poll a cheap boolean that means "the thing I asked for
   happened" (idatui: `dec.loaded_ea == target`). Always pair a semantic verb with
   its own predicate where one exists.
3. **Prompt / marker detection** *(black-box, structured-ish).* The screen returns
   to a known prompt regex, or the app emits a sentinel you injected (e.g. drive it
   to `echo <nonce>` and wait for `<nonce>` on screen). Robust when a prompt exists.
4. **Output debounce** *(black-box fallback).* Snapshot the screen every ~20 ms;
   consider it settled once it's unchanged for N consecutive samples (e.g. 3× =
   ~60 ms) or a hard timeout. Flakiest; tune N per app; combine with (3).

Factor settle into a shared helper that takes the *yield/poll* strategy and an
optional predicate (idatui's `_sync.settle(app, pred)` is exactly this — reused by
both the live driver and the test suite). That sharing is high-value: your tests
and your driver then agree on "settled."

### 5c. Introspection — "what is on screen / what is the state"
Same best→worst gradient:
1. **Native state** (embedded): read the app's model directly → richest snapshot.
2. **Control-channel queries** (gdb: frames, registers, `-data-read-memory`,
   breakpoints as JSON) → structured, no scraping.
3. **Screen scrape** (black-box): render the pane to a text grid and parse it.
   `tmux capture-pane -p -t <pane>` (add `-e` to keep colors) is the universal
   screen read. For a machine-parseable grid without a real terminal, feed the PTY
   stream through a headless emulator (e.g. **pyte**) and read its buffer.

Always expose a raw `screen()` too (plain + colored). It's the agent's "look with
your eyes" fallback and a great debugging aid (idatui added `format=text|html|svg`;
`html` is nice to pipe to an out-of-band web viewer).

---

## 6. tmux as the universal substrate

For **Level 2/3** targets, don't own the PTY yourself — let **tmux** own it and
drive through tmux. This collapses three problems into three commands and reuses
the pane manager we already built:

| need | tmux primitive |
|------|----------------|
| a visible pane running the target | `tmux split-window -P -F '#{pane_id}' '<cmd>'` |
| inject keystrokes | `tmux send-keys -t <pane> …` (named keys, `-l` literal) |
| read the screen | `tmux capture-pane -p [-e] -t <pane>` |
| is the pane alive | `tmux list-panes -a -F '#{pane_id}'` |
| tear down | `tmux kill-pane -t <pane>` |

A black-box adapter can be built *entirely* on these four. The pane manager
(`spawn/stop/list`, a small registry in `$XDG_RUNTIME_DIR`, backend auto-start,
readiness polling) carries over unchanged — only "what counts as ready" and "what
command to spawn" are per-target.

Caveat: `capture-pane` gives you the *rendered* grid but not semantic structure;
and `send-keys` is fire-and-forget (no completion signal) → you lean on §5b (3)/(4)
for settle. That's the price of black-box.

---

## 7. Reusable vs. per-target

**Reuse verbatim** (target-independent):
- Transport: unix socket, newline-delimited JSON, `{id,method,params}` →
  `{id,result|error}`; single-driver gate; per-request dispatch + error framing.
- Pane manager: `spawn/stop/list`, registry, backend `_ensure_server`, readiness
  poll, graceful `quit` (answer then exit).
- Client + ergonomic CLI: socket auto-resolution (single live pane), terse text
  output, composite verbs, `raw <method> k=v` passthrough, self-documenting
  `methods`.
- The settle *shape* (`wait_for(pred, tick)` + `settle(pred?)`), even though the
  concrete signals differ.

**Write per-target** (the adapter):
- Input injection binding (app key API / control channel / `tmux send-keys`).
- Settle signal (which of §5b applies).
- `snapshot()` and `screen()` sources.
- The semantic verb set (the app's real vocabulary).
- `ready()` and the spawn command.

Rule of thumb: **~80% reuse, ~20% adapter.** Keep the adapter interface narrow so
that stays true.

---

## 8. Worked example: **gdb**

gdb is a *Level 2* target with a great control channel, so aim for a hybrid.

**Layout.** A tmux pane runs gdb in TUI mode (`gdb -q -tui` / `layout src`,
`layout asm`, `layout regs`) — that's what viewers see. Alongside it, a **gdb
Python plugin** (loaded with `-x driver.py`, running inside gdb's own process)
opens the unix socket and *is* the adapter. gdb's Python runs on gdb's thread, so
handlers touch gdb state directly — the embedded pattern, for free, inside a
program you didn't write.

**Injection.**
- *Semantic* verbs call the API directly (visible in the TUI because gdb echoes
  and repaints): `gdb.execute("break main", to_string=True)`, `-exec-run`,
  `-exec-next`, `-exec-continue`, `-exec-finish`. Or GDB/MI via a second channel
  if you prefer strict JSON.
- *Raw* verbs (for TUI-only navigation: `C-x o` to switch windows, PgUp/PgDn in
  the source window, `C-x 2` layouts) go through **`tmux send-keys`** to the pane.

**Settle.** Mostly *given*: `gdb.execute(..., to_string=True)` and MI commands are
synchronous — they return when the command completed, so a semantic verb is
settled the moment the call returns. For *async* execution (`-exec-continue` while
the inferior runs), settle = wait for the next **stop event** (`gdb.events.stop`)
or the MI `*stopped` async record. For raw `send-keys` TUI moves, fall back to
`capture-pane` debounce (§5b-4).

**Introspection** (all structured, no scraping):
- `state()` → `{running|stopped, pc, function, file:line, thread, selected_frame}`
  from `gdb.selected_frame()`, `gdb.selected_thread()`.
- `backtrace()` → walk `gdb.newest_frame()` → `[{level,pc,func,file,line,args}]`.
- `regs()` → `frame.read_register(...)` for the ABI set.
- `mem(addr,len)` → `gdb.selected_inferior().read_memory(...)` (hex/ascii).
- `locals()`, `breakpoints()` (`gdb.breakpoints()` → JSON), `disas(addr?)`.
- `screen()` → `tmux capture-pane -ep` of the TUI pane (the "as a viewer sees it"
  read), *plus* the structured reads above for reasoning.

**Semantic verb set** (the gdb vocabulary): `run/start`, `cont`, `next`, `step`,
`finish`, `until`, `break <loc>`, `tbreak`, `delete <n>`, `watch <expr>`,
`bt [n]`, `frame <n>`, `up/down`, `print <expr>`, `x/<fmt> <addr>`, `set var`,
`layout <src|asm|regs|split>`, `focus <win>`, `raw keys …`.

**Spawn / readiness.** `spawn --bin ./prog [--args …]` → tmux pane runs
`gdb -q -tui -x driver.py --args ./prog …`; **ready** when the plugin's socket is
up *and* gdb reached its prompt (the plugin can signal readiness once loaded).
`stop` = graceful `quit` verb (plugin calls `gdb.execute("quit")`) then kill-pane.

This gives the same feel as idatui: `drive where` → `#3 main at foo.c:42`,
`drive bt`, `drive break foo`, `drive cont`, `drive x/16xb $sp`, `drive screen` —
terse, socket auto-resolved, every action visible in the gdb TUI pane.

---

## 9. Method-surface conventions (keep these consistent across targets)

- **Tiers, named the same everywhere:** `keys`/`text` (raw); `state`/`view`/
  `screen`/`<structured reads>` (introspection); `<semantic verbs>` (per target);
  `ping`/`methods`/`quit` (lifecycle).
- **`ping`** returns `{ok, proto, ready, …}`; **`methods`** returns the verb table
  (self-documentation); **`quit`** answers *then* exits (so the reply flushes).
- Mutating verbs **settle then return fresh `state`**. Read verbs never settle.
- Program-dependent verbs return a clean **`error: not ready`** before init.
- Typed verbs accept **`delay_ms`** (visible typing; `0` = fast). Movement verbs
  are fast (no delay, light settle).
- Errors are data (`{id,error:{message}}`), never drop the connection.

## 10. Client ergonomics (the part you use all day)

- **Auto-resolve the socket**: if exactly one live pane, use it; else `--sock` /
  env; with several, list and ask. Kills the "copy the socket path everywhere"
  tax — the single biggest quality-of-life win.
- **Terse text out**, not raw JSON (`where` → `main @ 0x… [src] L42`); keep a
  `raw <method> k=v` passthrough for the long tail.
- **Composite gestures** for the common multi-step flows (idatui's
  `rename old new` = goto+cursor+type; gdb's `break-and-run`, `stepn N`).
- Fire calls **sequentially** (single-driver); each invocation opens/closes its
  own connection.

## 11. Security & operations

- **Unix socket, mode 0600, local only.** No auth by design (whoever can r/w the
  socket drives it). Add a first-line shared token only if you bind TCP.
- **Single-driver gate** prevents two clients interleaving mutations.
- **Never leak panes/daemons.** Registry + `list --prune`; auto-start shared
  backends idempotently (probe before spawn; a port/socket already-in-use guards
  duplicates); don't kill shared backends on a per-pane `stop`.
- Note the **stale-pane** failure mode: a pane can outlive its backend/session;
  `list` should check liveness, not just tmux presence.

## 12. Onboarding a new target — checklist

1. **Classify** it (§4): can you embed? does it have a control channel? else
   black-box.
2. Pick the **injection** binding (§5a) and the **settle** signal (§5b) — decide
   this first; everything else is easy.
3. Implement the **Adapter** (§3): `inject_keys/text`, `settle`, `snapshot`,
   `screen`, `ready`, `quit`, and the **semantic verbs** (the app's real
   vocabulary — don't invent; mirror what a power user types).
4. Wire the **spawn command** + **readiness** into the pane manager.
5. Reuse transport, client, and ergonomic CLI unchanged.
6. Add a **smoke test** that spawns the target on a socket and drives the whole
   surface end-to-end (idatui's `rpc_smoke.py` is the template — it also locks the
   protocol against regressions).

## 13. Pitfalls & lessons (paid for once already)

- **Settle is where the bugs live.** A verb that "passes" by timing out then
  reading stale state is the classic false-green (idatui hit this: a 25 s hang that
  a check passed *trivially*). Prefer a real signal/predicate; print per-op timing
  so a suddenly-slow verb (a hidden timeout) is visible.
- **Newlines / control bytes** matter when injecting via a line-based reader —
  pick an escaping convention (idatui: literal `\n` → real newline at apply time).
- **Black-box screen scraping is lossy** — no semantic structure, colors optional,
  non-deterministic chrome (a live clock makes frames differ). Crop chrome; prefer
  structured reads when a control channel exists.
- **Match on what's shown, resolve by canonical id.** Names can render differently
  than they're stored (idatui: `.init_proc` shows as `init_proc`).
- **Keep the app visibly driven.** If you ever bypass the UI for speed, gate it
  behind an explicit `fast:true` — the default should always render, because
  "shown as if a user did it" is the whole point.

---

*Reference implementation: `idatui/{rpc,rpcclient,drive,pane,_sync}.py`,
`docs/RPC.md`, `tests/rpc_smoke.py` in this repo.*