# Investigation: idle link-socket drops in the multi–Game Boy setup

Status: **root-caused (with one honest caveat) and mitigated — drops are now
self-healing.** This document records the original mystery, the follow-up
session that resolved it, and the fixes that shipped.

## TL;DR (resolution)

The link socket never "mysteriously dies". Every drop signature we could
produce — and every one we could explain — is the **emulator process dying**
(or, historically, being observed wrongly). The kill vectors are mundane but
were all *silent*, which made them look like spooky socket death:

1. **SIGPIPE (proven, fixed).** In `--serial-sock` mode the emulator never
   ignored SIGPIPE. gbos mirrors every console byte to the link (`KPutc` →
   rSC=$81), so the moment the hub restarts or hiccups, the next prompt echo
   `write()` killed the whole emulator with the default SIGPIPE action —
   no stderr, no message, nothing. A hub restart would cascade-kill every GB.
2. **Terminal death / HUP / stray quit keys (windowed, the user's setup).**
   The windowed emulator is a child of its terminal session. An **idle** SSH
   session dropping (NAT/firewall idle timeout — the classic), a closed
   terminal tab (SIGHUP), or a stray `q` / Ctrl-C reaching `input_poll`
   (ISIG is off; `q` is a documented quit key) all exit the emulator cleanly
   and near-silently. **This precisely reproduces the "drops only when idle"
   correlation: a busy link means a busy SSH session, which doesn't idle out.**
3. **Guest `$ED` poweroff (possible, now visible).** The illegal opcode `$ED`
   requests a clean emulator exit and used to print *nothing*. Any gbos crash
   that walked into byte `$ED` would look exactly like a silent socket death.
4. **Observation artifacts (the spawn-mode ghost — see caveat).**

Verified end-to-end: killing the emulator produces exactly the reported
signature — `[gbhub] GBn (10.0.0.x) left` with a clean EOF, hub alive.

## The caveat: the spawn-mode drop did not reproduce under instrumentation

The original session reported `sudo tools/gbhub 1 ""` reliably dropping an
idle GB with **both** processes alive. A follow-up session attempted to
reproduce this for over an hour with heavy instrumentation:

- 25 fresh-boot spawn rounds (2 GBs + ping traffic, 30s each): no drop.
- Three long idle soaks (10–15 min, 1 GB): no drop — under `strace`, under an
  `LD_PRELOAD` shim logging every `close()`/`shutdown()` with backtraces and
  every non-EAGAIN serial-fd errno, and bare.
- A windowed (sixel) emulator under a synthetic pty, including a 2-minute
  full terminal stall: survived, link intact.

Meanwhile, the follow-up session *itself* fell into every observation trap
that plausibly produced the original "socket died, processes alive" claim:

- `pkill -f`/`pgrep -f` patterns matching the *observer's own shell*, killing
  the test mid-flight (looks like a spontaneous drop).
- Root-owned **stray emulators** surviving unprivileged pkill across test
  rounds — a later `pgrep`/`poll()` check can examine a *stale* process and
  conclude "the emulator is still alive" while the current one died.
- Sandbox command aborts killing the hub's process group while
  `start_new_session` emulators survive as strays (idle sessions get aborted;
  busy ones finish fast — another way "idle" correlates with drops).
- A second gbhub crashing on `gbtun0` EBUSY, with cleanup paths able to
  unlink the live hub's socket or NAT rules.

Given that no instrumented run ever showed a socket dying while its owner
lived, and that Linux unix-domain sockets have **no idle timeout mechanism at
all**, the working conclusion is that the spawn-mode observations were
process deaths misattributed by tooling. If a drop ever recurs under the new
logging (below), it will be attributable in minutes, not sessions.

## Fixes shipped

### Emulator (`~/dev/gbc`)

- **Link auto-reconnect** (`gb.c`): in `--serial-sock` mode, EOF on read or
  EPIPE/ECONNRESET/ENOTCONN/EBADF on either direction marks the link dead,
  closes the fd, and retries `connect()` every 500ms from the serial poll
  path (the guest polls constantly, so this needs no extra plumbing). RX
  stays "pending" while down (`serial_no_eof` semantics), so gbos just sees
  a quiet link that comes back. Logs `link socket lost / reconnected`.
- **SIGPIPE ignored** in `--serial-sock` mode (`main.c`) — a dead bridge
  degrades the link instead of killing the Game Boy.
- **`$ED` poweroff now logs** pc/registers to stderr (`cpu.c`), and the
  headless loop logs *why* it exited (guest poweroff vs signal). No more
  silent exits.

### Hub (`tools/gbhub`)

- `gb_reader` distinguishes a clean EOF from an exception and logs which
  (`left (eof)` vs `left (ConnectionResetError: ...)`), with timestamps on
  join/leave for correlating against SSH/terminal events.
- Spawn mode keeps each emulator's stderr in `/tmp/gbhub_<n>.err` instead of
  discarding it — postmortems now have the emulator's side of the story.
- A second `gbhub` now exits with a clear "another gbhub is already running?"
  message instead of an ioctl traceback (and cannot damage the live hub).

### Behavior after a drop (any cause)

Emulator killed → relaunch it (`gbjoin`), it rejoins as before.
Hub killed/restarted → all emulators reconnect within ~1s, re-register, and
keep their IPs (the pool hands the lowest free octet back, and the hub
routes by its own per-connection IP assignment, so the guest's existing
lease keeps working without a fresh DHCP exchange). Verified by SIGKILLing
the hub, restarting it, and pinging the reconnected GB.

## Still worth doing (not blocking)

- gbos-side periodic DHCP renewal (or "link came back" re-DHCP) so IP
  assignment survives pool reshuffles when *multiple* GBs reconnect out of
  order.
- The windowed main loop ignores `gb->poweroff` (only headless honors it);
  harmless, but inconsistent.
- Consider making `q` a less accidental quit (e.g. require `qq` or Ctrl-C
  twice) for long-running windowed sessions over SSH; an idle-timeout HUP
  can't be fixed client-side, but `ServerAliveInterval`/`tmux` on the
  streaming box sidesteps it operationally.

## Repro/diagnosis pointers (current tooling)

- Hub log now answers "who closed, and how": `left (eof)` = peer's fd closed
  (process exit or explicit close); `left (SomeError: ...)` = hub-side error.
- Emulator stderr (`/tmp/gbhub_<n>.err` in spawn mode) now answers "why did
  the emulator stop": `poweroff opcode $ED at pc=...`, `headless loop exit:
  signal`, `link socket lost/reconnected`.
- For deeper hunts, the session's `LD_PRELOAD` fd-spy approach (log
  `close`/`shutdown` with backtraces, plus unusual errno on the serial fd)
  is cheap and does not perturb timing the way `strace` does — `strace -p`
  intercepts *every* syscall even when filtered, which throttles an
  `--uncapped` emulator badly.

## Relevant code

- Emulator link socket + reconnect: `~/dev/gbc/src/main.c` (`serial_connect`,
  `--serial-sock` wiring, SIGPIPE) and `~/dev/gbc/src/gb.c` (`serial_check`,
  `serial_mark_dead`, rSC=$80/$81 paths).
- Hub reader/router/DHCP: `~/dev/gbos/tools/gbhub` (`gb_reader`, `route`,
  `_dreply`, `tun_reader`, `net_setup` collision guard).
- GB kernel serial/net pump: `~/dev/gbos/src/socket.asm` (`net_getbyte_nb`,
  `net_putbyte`, `net_pump`) and console pump in `~/dev/gbos/src/syscall.asm`
  (`KGetc`).