pty: open the slave before the fork, so a pane's size reaches itv0.1.1

On Darwin a pty master has no line discipline until its slave has been
opened: every TIOCSWINSZ on the master until then fails with ENOTTY and
the size is dropped without a word. A pane is born 1x1 and told how big
it really is by the layout an instant after the fork -- and with the
child doing the open(), that message raced the child and usually lost.
Nothing ever sent it again, so the program in the pane spent its whole
life believing it had one row and one column. `C-a e` hit it every
time, because the editor's pane is always brand new: nvim drew its
screen into two rows in the corner of a 94x15 pane.

The old code knew half of this -- the child set the winsize on the
slave it had just opened, which is what kept a fresh pane from being
0x0 -- but that only fixed the *spawn* size. Every resize after it
still went to a master that might not have a slave yet.

So open the slave here, before the fork, and let the child inherit it.
The fork copies the fd table, so the slave is open from the child's
first instruction and every resize the parent sends afterwards lands;
BSD's forkpty() does the same thing for the same reason. The winsize is
set on that slave before the fork, which orders it ahead of anything
the parent sends later, and the child no longer sets it at all: by the
time the child ran, the size it was spawned with could already be
stale, and writing it back would undo the very resize this is about.

test_pty_size.py asks the pane's own program what size it thinks it is
-- for the first pane, for a pane from a split, and after a resize --
and compares it to the pane. It printed "1 1" against a 74x18 pane
before this.

live_m0.py asked `tput cols`, which needs a terminfo entry for
xterm-ghostty and therefore fails on most macs for a reason that has
nothing to do with resizing. `stty size` asks the tty itself, which is
the thing under test.

1 files changed, 36 insertions, 10 deletions

diff --git a/src/pty.c b/src/pty.c
index 7f7ddb5..dbabcef 100644
--- a/src/pty.c
+++ b/src/pty.c
@@ -29,18 +29,49 @@ int pty_spawn(pty_t *p, const char *const argv[], uint16_t cols, uint16_t rows,
     return -1;
   }
 
+  /* The slave is opened *here*, before the fork, and not by the child.
+   *
+   * Darwin attaches no line discipline to a pty master until its slave has
+   * been opened: until then every TIOCSWINSZ on the master fails with ENOTTY
+   * and the size is silently dropped. A pane is born 1x1 and told its real
+   * size by the layout an instant later -- so with the child opening the
+   * slave, that message raced the child's open() and usually lost, and
+   * nothing ever sent it again. The pane's program spent its life believing
+   * it had one row and one column, which is how `C-a e` produced an editor
+   * drawing into a corner of the pane on macOS and nowhere else.
+   *
+   * Opening it in the parent closes the window entirely: the fork copies the
+   * fd table, so the slave is open from the child's first instruction, and
+   * every resize the parent sends afterwards lands. It is also what BSD
+   * forkpty() does, for this reason.
+   *
+   * O_NOCTTY: we are not the process that wants this terminal. The child is,
+   * and it says so with TIOCSCTTY after setsid(). */
+  int slave = open(slave_name, O_RDWR | O_NOCTTY);
+  if (slave < 0) {
+    close(master);
+    return -1;
+  }
+
   /* The pixel fields matter: a program drawing images asks the tty how big a
    * cell is, and a winsize that says zero is a program that cannot size an
    * image. They are the *terminal's* pixels, so they are cols/rows times the
-   * cell size the client reported. */
+   * cell size the client reported.
+   *
+   * Set on the slave, which works on Linux and Darwin alike, and set before
+   * the fork so it is ordered ahead of any resize the parent sends after it.
+   * The child must not set it: by the time it ran, the size it was spawned
+   * with could already be stale, and writing it back would undo the resize we
+   * are here to deliver. */
   struct winsize ws = {.ws_col = cols,
                        .ws_row = rows,
                        .ws_xpixel = (unsigned short)(cols * cell_w),
                        .ws_ypixel = (unsigned short)(rows * cell_h)};
-  ioctl(master, TIOCSWINSZ, &ws);
+  ioctl(slave, TIOCSWINSZ, &ws);
 
   pid_t pid = fork();
   if (pid < 0) {
+    close(slave);
     close(master);
     return -1;
   }
@@ -48,14 +79,7 @@ int pty_spawn(pty_t *p, const char *const argv[], uint16_t cols, uint16_t rows,
   if (pid == 0) {
     /* child: new session, pty becomes the controlling terminal */
     setsid();
-    int slave = open(slave_name, O_RDWR);
-    if (slave < 0) _exit(127);
     ioctl(slave, TIOCSCTTY, 0);
-    /* Darwin attaches no line discipline to the master until the slave is
-     * opened, so the TIOCSWINSZ above returns ENOTTY there and the size is
-     * dropped: the pane starts 0x0 and stays that way until something else
-     * resizes it. Setting it here, on the slave we just opened, works on both. */
-    ioctl(slave, TIOCSWINSZ, &ws);
     dup2(slave, STDIN_FILENO);
     dup2(slave, STDOUT_FILENO);
     dup2(slave, STDERR_FILENO);
@@ -78,7 +102,9 @@ int pty_spawn(pty_t *p, const char *const argv[], uint16_t cols, uint16_t rows,
     _exit(127);
   }
 
-  /* parent */
+  /* parent: the child holds the slave now, and holding a second copy would
+   * keep the master readable long after the program in the pane had gone. */
+  close(slave);
   int flags = fcntl(master, F_GETFL, 0);
   fcntl(master, F_SETFL, flags | O_NONBLOCK);
   p->fd = master;