1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
|
# Investigation: idle link-socket drops in the multi–Game Boy setup
Status: **root-caused (with one honest caveat) and mitigated — drops are now
self-healing.** This document records the original mystery, the follow-up
session that resolved it, and the fixes that shipped.
## TL;DR (resolution)
The link socket never "mysteriously dies". Every drop signature we could
produce — and every one we could explain — is the **emulator process dying**
(or, historically, being observed wrongly). The kill vectors are mundane but
were all *silent*, which made them look like spooky socket death:
1. **SIGPIPE (proven, fixed).** In `--serial-sock` mode the emulator never
ignored SIGPIPE. gbos mirrors every console byte to the link (`KPutc` →
rSC=$81), so the moment the hub restarts or hiccups, the next prompt echo
`write()` killed the whole emulator with the default SIGPIPE action —
no stderr, no message, nothing. A hub restart would cascade-kill every GB.
2. **Terminal death / HUP / stray quit keys (windowed, the user's setup).**
The windowed emulator is a child of its terminal session. An **idle** SSH
session dropping (NAT/firewall idle timeout — the classic), a closed
terminal tab (SIGHUP), or a stray `q` / Ctrl-C reaching `input_poll`
(ISIG is off; `q` is a documented quit key) all exit the emulator cleanly
and near-silently. **This precisely reproduces the "drops only when idle"
correlation: a busy link means a busy SSH session, which doesn't idle out.**
3. **Guest `$ED` poweroff (possible, now visible).** The illegal opcode `$ED`
requests a clean emulator exit and used to print *nothing*. Any gbos crash
that walked into byte `$ED` would look exactly like a silent socket death.
4. **Observation artifacts (the spawn-mode ghost — see caveat).**
Verified end-to-end: killing the emulator produces exactly the reported
signature — `[gbhub] GBn (10.0.0.x) left` with a clean EOF, hub alive.
## The caveat: the spawn-mode drop did not reproduce under instrumentation
The original session reported `sudo tools/gbhub 1 ""` reliably dropping an
idle GB with **both** processes alive. A follow-up session attempted to
reproduce this for over an hour with heavy instrumentation:
- 25 fresh-boot spawn rounds (2 GBs + ping traffic, 30s each): no drop.
- Three long idle soaks (10–15 min, 1 GB): no drop — under `strace`, under an
`LD_PRELOAD` shim logging every `close()`/`shutdown()` with backtraces and
every non-EAGAIN serial-fd errno, and bare.
- A windowed (sixel) emulator under a synthetic pty, including a 2-minute
full terminal stall: survived, link intact.
Meanwhile, the follow-up session *itself* fell into every observation trap
that plausibly produced the original "socket died, processes alive" claim:
- `pkill -f`/`pgrep -f` patterns matching the *observer's own shell*, killing
the test mid-flight (looks like a spontaneous drop).
- Root-owned **stray emulators** surviving unprivileged pkill across test
rounds — a later `pgrep`/`poll()` check can examine a *stale* process and
conclude "the emulator is still alive" while the current one died.
- Sandbox command aborts killing the hub's process group while
`start_new_session` emulators survive as strays (idle sessions get aborted;
busy ones finish fast — another way "idle" correlates with drops).
- A second gbhub crashing on `gbtun0` EBUSY, with cleanup paths able to
unlink the live hub's socket or NAT rules.
Given that no instrumented run ever showed a socket dying while its owner
lived, and that Linux unix-domain sockets have **no idle timeout mechanism at
all**, the working conclusion is that the spawn-mode observations were
process deaths misattributed by tooling. If a drop ever recurs under the new
logging (below), it will be attributable in minutes, not sessions.
## Fixes shipped
### Emulator (`~/dev/gbc`)
- **Link auto-reconnect** (`gb.c`): in `--serial-sock` mode, EOF on read or
EPIPE/ECONNRESET/ENOTCONN/EBADF on either direction marks the link dead,
closes the fd, and retries `connect()` every 500ms from the serial poll
path (the guest polls constantly, so this needs no extra plumbing). RX
stays "pending" while down (`serial_no_eof` semantics), so gbos just sees
a quiet link that comes back. Logs `link socket lost / reconnected`.
- **SIGPIPE ignored** in `--serial-sock` mode (`main.c`) — a dead bridge
degrades the link instead of killing the Game Boy.
- **`$ED` poweroff now logs** pc/registers to stderr (`cpu.c`), and the
headless loop logs *why* it exited (guest poweroff vs signal). No more
silent exits.
### Hub (`tools/gbhub`)
- `gb_reader` distinguishes a clean EOF from an exception and logs which
(`left (eof)` vs `left (ConnectionResetError: ...)`), with timestamps on
join/leave for correlating against SSH/terminal events.
- Spawn mode keeps each emulator's stderr in `/tmp/gbhub_<n>.err` instead of
discarding it — postmortems now have the emulator's side of the story.
- A second `gbhub` now exits with a clear "another gbhub is already running?"
message instead of an ioctl traceback (and cannot damage the live hub).
### Behavior after a drop (any cause)
Emulator killed → relaunch it (`gbjoin`), it rejoins as before.
Hub killed/restarted → all emulators reconnect within ~1s, re-register, and
keep their IPs (the pool hands the lowest free octet back, and the hub
routes by its own per-connection IP assignment, so the guest's existing
lease keeps working without a fresh DHCP exchange). Verified by SIGKILLing
the hub, restarting it, and pinging the reconnected GB.
## Still worth doing (not blocking)
- gbos-side periodic DHCP renewal (or "link came back" re-DHCP) so IP
assignment survives pool reshuffles when *multiple* GBs reconnect out of
order.
- The windowed main loop ignores `gb->poweroff` (only headless honors it);
harmless, but inconsistent.
- Consider making `q` a less accidental quit (e.g. require `qq` or Ctrl-C
twice) for long-running windowed sessions over SSH; an idle-timeout HUP
can't be fixed client-side, but `ServerAliveInterval`/`tmux` on the
streaming box sidesteps it operationally.
## Repro/diagnosis pointers (current tooling)
- Hub log now answers "who closed, and how": `left (eof)` = peer's fd closed
(process exit or explicit close); `left (SomeError: ...)` = hub-side error.
- Emulator stderr (`/tmp/gbhub_<n>.err` in spawn mode) now answers "why did
the emulator stop": `poweroff opcode $ED at pc=...`, `headless loop exit:
signal`, `link socket lost/reconnected`.
- For deeper hunts, the session's `LD_PRELOAD` fd-spy approach (log
`close`/`shutdown` with backtraces, plus unusual errno on the serial fd)
is cheap and does not perturb timing the way `strace` does — `strace -p`
intercepts *every* syscall even when filtered, which throttles an
`--uncapped` emulator badly.
## Relevant code
- Emulator link socket + reconnect: `~/dev/gbc/src/main.c` (`serial_connect`,
`--serial-sock` wiring, SIGPIPE) and `~/dev/gbc/src/gb.c` (`serial_check`,
`serial_mark_dead`, rSC=$80/$81 paths).
- Hub reader/router/DHCP: `~/dev/gbos/tools/gbhub` (`gb_reader`, `route`,
`_dreply`, `tun_reader`, `net_setup` collision guard).
- GB kernel serial/net pump: `~/dev/gbos/src/socket.asm` (`net_getbyte_nb`,
`net_putbyte`, `net_pump`) and console pump in `~/dev/gbos/src/syscall.asm`
(`KGetc`).
|
