From cdebdd1ae2c50e37f7401ccaae890ff554b974ea Mon Sep 17 00:00:00 2001 From: user Date: Sat, 18 Jul 2026 21:02:47 +0200 Subject: docs: drop the link-drop investigation writeup --- docs/link-drop-investigation.md | 136 ---------------------------------------- 1 file changed, 136 deletions(-) delete mode 100644 docs/link-drop-investigation.md (limited to 'docs/link-drop-investigation.md') diff --git a/docs/link-drop-investigation.md b/docs/link-drop-investigation.md deleted file mode 100644 index bf7efb4..0000000 --- a/docs/link-drop-investigation.md +++ /dev/null @@ -1,136 +0,0 @@ -# Investigation: idle link-socket drops in the multi–Game Boy setup - -Status: **root-caused (with one honest caveat) and mitigated — drops are now -self-healing.** This document records the original mystery, the follow-up -session that resolved it, and the fixes that shipped. - -## TL;DR (resolution) - -The link socket never "mysteriously dies". Every drop signature we could -produce — and every one we could explain — is the **emulator process dying** -(or, historically, being observed wrongly). The kill vectors are mundane but -were all *silent*, which made them look like spooky socket death: - -1. **SIGPIPE (proven, fixed).** In `--serial-sock` mode the emulator never - ignored SIGPIPE. gbos mirrors every console byte to the link (`KPutc` → - rSC=$81), so the moment the hub restarts or hiccups, the next prompt echo - `write()` killed the whole emulator with the default SIGPIPE action — - no stderr, no message, nothing. A hub restart would cascade-kill every GB. -2. **Terminal death / HUP / stray quit keys (windowed, the user's setup).** - The windowed emulator is a child of its terminal session. An **idle** SSH - session dropping (NAT/firewall idle timeout — the classic), a closed - terminal tab (SIGHUP), or a stray `q` / Ctrl-C reaching `input_poll` - (ISIG is off; `q` is a documented quit key) all exit the emulator cleanly - and near-silently. **This precisely reproduces the "drops only when idle" - correlation: a busy link means a busy SSH session, which doesn't idle out.** -3. **Guest `$ED` poweroff (possible, now visible).** The illegal opcode `$ED` - requests a clean emulator exit and used to print *nothing*. Any gbos crash - that walked into byte `$ED` would look exactly like a silent socket death. -4. **Observation artifacts (the spawn-mode ghost — see caveat).** - -Verified end-to-end: killing the emulator produces exactly the reported -signature — `[gbhub] GBn (10.0.0.x) left` with a clean EOF, hub alive. - -## The caveat: the spawn-mode drop did not reproduce under instrumentation - -The original session reported `sudo tools/gbhub 1 ""` reliably dropping an -idle GB with **both** processes alive. A follow-up session attempted to -reproduce this for over an hour with heavy instrumentation: - -- 25 fresh-boot spawn rounds (2 GBs + ping traffic, 30s each): no drop. -- Three long idle soaks (10–15 min, 1 GB): no drop — under `strace`, under an - `LD_PRELOAD` shim logging every `close()`/`shutdown()` with backtraces and - every non-EAGAIN serial-fd errno, and bare. -- A windowed (sixel) emulator under a synthetic pty, including a 2-minute - full terminal stall: survived, link intact. - -Meanwhile, the follow-up session *itself* fell into every observation trap -that plausibly produced the original "socket died, processes alive" claim: - -- `pkill -f`/`pgrep -f` patterns matching the *observer's own shell*, killing - the test mid-flight (looks like a spontaneous drop). -- Root-owned **stray emulators** surviving unprivileged pkill across test - rounds — a later `pgrep`/`poll()` check can examine a *stale* process and - conclude "the emulator is still alive" while the current one died. -- Sandbox command aborts killing the hub's process group while - `start_new_session` emulators survive as strays (idle sessions get aborted; - busy ones finish fast — another way "idle" correlates with drops). -- A second gbhub crashing on `gbtun0` EBUSY, with cleanup paths able to - unlink the live hub's socket or NAT rules. - -Given that no instrumented run ever showed a socket dying while its owner -lived, and that Linux unix-domain sockets have **no idle timeout mechanism at -all**, the working conclusion is that the spawn-mode observations were -process deaths misattributed by tooling. If a drop ever recurs under the new -logging (below), it will be attributable in minutes, not sessions. - -## Fixes shipped - -### Emulator (`~/dev/gbc`) - -- **Link auto-reconnect** (`gb.c`): in `--serial-sock` mode, EOF on read or - EPIPE/ECONNRESET/ENOTCONN/EBADF on either direction marks the link dead, - closes the fd, and retries `connect()` every 500ms from the serial poll - path (the guest polls constantly, so this needs no extra plumbing). RX - stays "pending" while down (`serial_no_eof` semantics), so gbos just sees - a quiet link that comes back. Logs `link socket lost / reconnected`. -- **SIGPIPE ignored** in `--serial-sock` mode (`main.c`) — a dead bridge - degrades the link instead of killing the Game Boy. -- **`$ED` poweroff now logs** pc/registers to stderr (`cpu.c`), and the - headless loop logs *why* it exited (guest poweroff vs signal). No more - silent exits. - -### Hub (`tools/gbhub`) - -- `gb_reader` distinguishes a clean EOF from an exception and logs which - (`left (eof)` vs `left (ConnectionResetError: ...)`), with timestamps on - join/leave for correlating against SSH/terminal events. -- Spawn mode keeps each emulator's stderr in `/tmp/gbhub_.err` instead of - discarding it — postmortems now have the emulator's side of the story. -- A second `gbhub` now exits with a clear "another gbhub is already running?" - message instead of an ioctl traceback (and cannot damage the live hub). - -### Behavior after a drop (any cause) - -Emulator killed → relaunch it (`gbjoin`), it rejoins as before. -Hub killed/restarted → all emulators reconnect within ~1s, re-register, and -keep their IPs (the pool hands the lowest free octet back, and the hub -routes by its own per-connection IP assignment, so the guest's existing -lease keeps working without a fresh DHCP exchange). Verified by SIGKILLing -the hub, restarting it, and pinging the reconnected GB. - -## Still worth doing (not blocking) - -- gbos-side periodic DHCP renewal (or "link came back" re-DHCP) so IP - assignment survives pool reshuffles when *multiple* GBs reconnect out of - order. -- The windowed main loop ignores `gb->poweroff` (only headless honors it); - harmless, but inconsistent. -- Consider making `q` a less accidental quit (e.g. require `qq` or Ctrl-C - twice) for long-running windowed sessions over SSH; an idle-timeout HUP - can't be fixed client-side, but `ServerAliveInterval`/`tmux` on the - streaming box sidesteps it operationally. - -## Repro/diagnosis pointers (current tooling) - -- Hub log now answers "who closed, and how": `left (eof)` = peer's fd closed - (process exit or explicit close); `left (SomeError: ...)` = hub-side error. -- Emulator stderr (`/tmp/gbhub_.err` in spawn mode) now answers "why did - the emulator stop": `poweroff opcode $ED at pc=...`, `headless loop exit: - signal`, `link socket lost/reconnected`. -- For deeper hunts, the session's `LD_PRELOAD` fd-spy approach (log - `close`/`shutdown` with backtraces, plus unusual errno on the serial fd) - is cheap and does not perturb timing the way `strace` does — `strace -p` - intercepts *every* syscall even when filtered, which throttles an - `--uncapped` emulator badly. - -## Relevant code - -- Emulator link socket + reconnect: `~/dev/gbc/src/main.c` (`serial_connect`, - `--serial-sock` wiring, SIGPIPE) and `~/dev/gbc/src/gb.c` (`serial_check`, - `serial_mark_dead`, rSC=$80/$81 paths). -- Hub reader/router/DHCP: `~/dev/gbos/tools/gbhub` (`gb_reader`, `route`, - `_dreply`, `tun_reader`, `net_setup` collision guard). -- GB kernel serial/net pump: `~/dev/gbos/src/socket.asm` (`net_getbyte_nb`, - `net_putbyte`, `net_pump`) and console pump in `~/dev/gbos/src/syscall.asm` - (`KGetc`). -- cgit v1.3.1-sl0p