docs: drop the link-drop investigation writeup

1 files changed, 0 insertions, 136 deletions

diff --git a/docs/link-drop-investigation.md b/docs/link-drop-investigation.md
deleted file mode 100644
index bf7efb4..0000000
--- a/docs/link-drop-investigation.md
+++ /dev/null
@@ -1,136 +0,0 @@
-# Investigation: idle link-socket drops in the multi–Game Boy setup
-
-Status: **root-caused (with one honest caveat) and mitigated — drops are now
-self-healing.** This document records the original mystery, the follow-up
-session that resolved it, and the fixes that shipped.
-
-## TL;DR (resolution)
-
-The link socket never "mysteriously dies". Every drop signature we could
-produce — and every one we could explain — is the **emulator process dying**
-(or, historically, being observed wrongly). The kill vectors are mundane but
-were all *silent*, which made them look like spooky socket death:
-
-1. **SIGPIPE (proven, fixed).** In `--serial-sock` mode the emulator never
-   ignored SIGPIPE. gbos mirrors every console byte to the link (`KPutc` →
-   rSC=$81), so the moment the hub restarts or hiccups, the next prompt echo
-   `write()` killed the whole emulator with the default SIGPIPE action —
-   no stderr, no message, nothing. A hub restart would cascade-kill every GB.
-2. **Terminal death / HUP / stray quit keys (windowed, the user's setup).**
-   The windowed emulator is a child of its terminal session. An **idle** SSH
-   session dropping (NAT/firewall idle timeout — the classic), a closed
-   terminal tab (SIGHUP), or a stray `q` / Ctrl-C reaching `input_poll`
-   (ISIG is off; `q` is a documented quit key) all exit the emulator cleanly
-   and near-silently. **This precisely reproduces the "drops only when idle"
-   correlation: a busy link means a busy SSH session, which doesn't idle out.**
-3. **Guest `$ED` poweroff (possible, now visible).** The illegal opcode `$ED`
-   requests a clean emulator exit and used to print *nothing*. Any gbos crash
-   that walked into byte `$ED` would look exactly like a silent socket death.
-4. **Observation artifacts (the spawn-mode ghost — see caveat).**
-
-Verified end-to-end: killing the emulator produces exactly the reported
-signature — `[gbhub] GBn (10.0.0.x) left` with a clean EOF, hub alive.
-
-## The caveat: the spawn-mode drop did not reproduce under instrumentation
-
-The original session reported `sudo tools/gbhub 1 ""` reliably dropping an
-idle GB with **both** processes alive. A follow-up session attempted to
-reproduce this for over an hour with heavy instrumentation:
-
-- 25 fresh-boot spawn rounds (2 GBs + ping traffic, 30s each): no drop.
-- Three long idle soaks (10–15 min, 1 GB): no drop — under `strace`, under an
-  `LD_PRELOAD` shim logging every `close()`/`shutdown()` with backtraces and
-  every non-EAGAIN serial-fd errno, and bare.
-- A windowed (sixel) emulator under a synthetic pty, including a 2-minute
-  full terminal stall: survived, link intact.
-
-Meanwhile, the follow-up session *itself* fell into every observation trap
-that plausibly produced the original "socket died, processes alive" claim:
-
-- `pkill -f`/`pgrep -f` patterns matching the *observer's own shell*, killing
-  the test mid-flight (looks like a spontaneous drop).
-- Root-owned **stray emulators** surviving unprivileged pkill across test
-  rounds — a later `pgrep`/`poll()` check can examine a *stale* process and
-  conclude "the emulator is still alive" while the current one died.
-- Sandbox command aborts killing the hub's process group while
-  `start_new_session` emulators survive as strays (idle sessions get aborted;
-  busy ones finish fast — another way "idle" correlates with drops).
-- A second gbhub crashing on `gbtun0` EBUSY, with cleanup paths able to
-  unlink the live hub's socket or NAT rules.
-
-Given that no instrumented run ever showed a socket dying while its owner
-lived, and that Linux unix-domain sockets have **no idle timeout mechanism at
-all**, the working conclusion is that the spawn-mode observations were
-process deaths misattributed by tooling. If a drop ever recurs under the new
-logging (below), it will be attributable in minutes, not sessions.
-
-## Fixes shipped
-
-### Emulator (`~/dev/gbc`)
-
-- **Link auto-reconnect** (`gb.c`): in `--serial-sock` mode, EOF on read or
-  EPIPE/ECONNRESET/ENOTCONN/EBADF on either direction marks the link dead,
-  closes the fd, and retries `connect()` every 500ms from the serial poll
-  path (the guest polls constantly, so this needs no extra plumbing). RX
-  stays "pending" while down (`serial_no_eof` semantics), so gbos just sees
-  a quiet link that comes back. Logs `link socket lost / reconnected`.
-- **SIGPIPE ignored** in `--serial-sock` mode (`main.c`) — a dead bridge
-  degrades the link instead of killing the Game Boy.
-- **`$ED` poweroff now logs** pc/registers to stderr (`cpu.c`), and the
-  headless loop logs *why* it exited (guest poweroff vs signal). No more
-  silent exits.
-
-### Hub (`tools/gbhub`)
-
-- `gb_reader` distinguishes a clean EOF from an exception and logs which
-  (`left (eof)` vs `left (ConnectionResetError: ...)`), with timestamps on
-  join/leave for correlating against SSH/terminal events.
-- Spawn mode keeps each emulator's stderr in `/tmp/gbhub_<n>.err` instead of
-  discarding it — postmortems now have the emulator's side of the story.
-- A second `gbhub` now exits with a clear "another gbhub is already running?"
-  message instead of an ioctl traceback (and cannot damage the live hub).
-
-### Behavior after a drop (any cause)
-
-Emulator killed → relaunch it (`gbjoin`), it rejoins as before.
-Hub killed/restarted → all emulators reconnect within ~1s, re-register, and
-keep their IPs (the pool hands the lowest free octet back, and the hub
-routes by its own per-connection IP assignment, so the guest's existing
-lease keeps working without a fresh DHCP exchange). Verified by SIGKILLing
-the hub, restarting it, and pinging the reconnected GB.
-
-## Still worth doing (not blocking)
-
-- gbos-side periodic DHCP renewal (or "link came back" re-DHCP) so IP
-  assignment survives pool reshuffles when *multiple* GBs reconnect out of
-  order.
-- The windowed main loop ignores `gb->poweroff` (only headless honors it);
-  harmless, but inconsistent.
-- Consider making `q` a less accidental quit (e.g. require `qq` or Ctrl-C
-  twice) for long-running windowed sessions over SSH; an idle-timeout HUP
-  can't be fixed client-side, but `ServerAliveInterval`/`tmux` on the
-  streaming box sidesteps it operationally.
-
-## Repro/diagnosis pointers (current tooling)
-
-- Hub log now answers "who closed, and how": `left (eof)` = peer's fd closed
-  (process exit or explicit close); `left (SomeError: ...)` = hub-side error.
-- Emulator stderr (`/tmp/gbhub_<n>.err` in spawn mode) now answers "why did
-  the emulator stop": `poweroff opcode $ED at pc=...`, `headless loop exit:
-  signal`, `link socket lost/reconnected`.
-- For deeper hunts, the session's `LD_PRELOAD` fd-spy approach (log
-  `close`/`shutdown` with backtraces, plus unusual errno on the serial fd)
-  is cheap and does not perturb timing the way `strace` does — `strace -p`
-  intercepts *every* syscall even when filtered, which throttles an
-  `--uncapped` emulator badly.
-
-## Relevant code
-
-- Emulator link socket + reconnect: `~/dev/gbc/src/main.c` (`serial_connect`,
-  `--serial-sock` wiring, SIGPIPE) and `~/dev/gbc/src/gb.c` (`serial_check`,
-  `serial_mark_dead`, rSC=$80/$81 paths).
-- Hub reader/router/DHCP: `~/dev/gbos/tools/gbhub` (`gb_reader`, `route`,
-  `_dreply`, `tun_reader`, `net_setup` collision guard).
-- GB kernel serial/net pump: `~/dev/gbos/src/socket.asm` (`net_getbyte_nb`,
-  `net_putbyte`, `net_pump`) and console pump in `~/dev/gbos/src/syscall.asm`
-  (`KGetc`).