diff options
| author | sl0p <sl0p@sl0p.foo> | 2026-08-14 01:01:04 +0200 |
|---|---|---|
| committer | sl0p <sl0p@sl0p.foo> | 2026-08-14 01:01:04 +0200 |
| commit | b6c175e58ee21e2532d2826411a5351d9a676c09 (patch) | |
| tree | 08562ab30350e6353d8f78eb19132d4e8e8a2b09 | |
| parent | sl0ptty: the split guide arms on dwell, not on contact (diff) | |
| download | slosh-b6c175e58ee21e2532d2826411a5351d9a676c09.tar.gz slosh-b6c175e58ee21e2532d2826411a5351d9a676c09.tar.xz slosh-b6c175e58ee21e2532d2826411a5351d9a676c09.zip | |
sl0ptty: kitty graphics passthrough
A program in a pane transmits an image and it appears. libghostty-vt parses
the protocol and tracks images and placements; what a multiplexer has to add
is re-emitting them to the client, which is exactly what tmux and zellij do
not do — and the reason tools fall back to sixel under them.
Per frame: walk the visible tab's panes, transmit each image once (a=t),
place every visible placement (a=p,C=1) after the cell diff so a repainted
cell cannot land on top of one, and delete placements that vanished. Image ids
are remapped into a high range, so two panes both using image 7 stay two
images and neither collides with what the client's terminal already had. A
reattaching client is told everything again, because it has seen nothing.
The cropping rule is the part worth remembering: a placement is N pixels drawn
across M cells, so the placement itself tells you the pixels per cell — the
client's font metrics never come into it. Clipping at a pane edge, or at the
top of the viewport when a tall image is partly scrolled off, moves the source
rectangle; asking for fewer columns or rows alone scales the image into them
instead of cropping it. The first version did exactly that, and dropped
partially-visible placements entirely rather than cropping them.
Tests cover the multiplexer-specific case first: two panes, both using image
id 7, must get different client ids. Then placement position in screen cells,
following the layout, cropping at edges and when taller than the pane, nothing
from collapsed panes, and removal when a pane closes or the image scrolls
away. A live check confirms the wire: transmitted once, re-placed per frame,
not re-transmitted.
416 checks green (389 headless + 27 live).
| -rw-r--r-- | DESIGN.md | 12 | ||||
| -rw-r--r-- | Makefile | 2 | ||||
| -rw-r--r-- | README.md | 13 | ||||
| -rw-r--r-- | src/app.c | 83 | ||||
| -rw-r--r-- | src/app.h | 6 | ||||
| -rw-r--r-- | src/cmd.c | 4 | ||||
| -rw-r--r-- | src/graphics.c | 217 | ||||
| -rw-r--r-- | src/graphics.h | 62 | ||||
| -rw-r--r-- | src/pane.c | 122 | ||||
| -rw-r--r-- | src/server.c | 12 | ||||
| -rw-r--r-- | src/sl0ptty.h | 20 | ||||
| -rw-r--r-- | tests/test_graphics.py | 150 |
12 files changed, 701 insertions, 2 deletions
@@ -88,7 +88,17 @@ now have regression tests. stream for the real terminal. 3. **pty spawn, event loop, layout tree, sockets, config.** 4. **Kitty graphics re-emission** (id remapping across panes and scroll). - Possible because lib-vt tracks placements. Not MVP. + **Done.** lib-vt tracks the images and placements; we walk them each frame, + transmit an image once (`a=t`), place it every frame (`a=p,C=1`) after the + cell diff, and delete placements that vanish. Ids are remapped into a high + range so two panes using image `7` stay two images, and so nothing collides + with images the client's terminal already had. + + The cropping rule is the part worth remembering: **a placement is N pixels + drawn across M cells, so the placement itself tells you the pixels per + cell** — no need to know the client's font metrics. Clipping at a pane edge + or at the top of the viewport moves the *source rectangle*; reducing `c`/`r` + alone scales the image into fewer cells instead of cropping it. Budget: **under 10k LOC**, zero deps beyond libc and lib-vt, static musl binary, sub-second full rebuild. @@ -58,7 +58,7 @@ test: $(BIN) $(TEST_BIN) $(KDL_TEST) ## unit + headless checks (fast) @echo cd tests && python3 test_screen.py && python3 test_layout.py && python3 test_tabs.py && python3 test_osc5577.py && \ python3 test_responsive.py && python3 test_finder.py && \ - python3 test_config.py && python3 test_layout_files.py && python3 test_drag.py && python3 test_hover.py && python3 test_scroll.py && python3 test_clipboard.py && python3 test_split_ux.py + python3 test_config.py && python3 test_layout_files.py && python3 test_drag.py && python3 test_hover.py && python3 test_scroll.py && python3 test_clipboard.py && python3 test_split_ux.py && python3 test_graphics.py test-live: $(BIN) ## the checks that need a real tty (slow) python3 tests/live_m0.py @@ -86,6 +86,19 @@ its own `purpose` (`\033]5577;1;purpose;agent:main\033\\`) — but a purpose declared by a layout outranks it and cannot be overridden, so a pane cannot relabel itself into something tooling trusts. +## Images + +Kitty graphics pass through. A program in a pane transmits an image and it +appears, because libghostty-vt tracks the images and placements and sl0ptty +re-emits them to your terminal with ids remapped so two panes using image `7` +stay two images. Placements follow the layout, are cropped at the pane's edges +by moving the source rectangle (asking for fewer cells would scale instead), +and are deleted when their pane closes or scrolls away. + +This is the thing tmux and zellij drop, and the reason tools fall back to +sixel under them. Sixel itself is *not* supported: libghostty-vt has no sixel +decoder, so those sequences are swallowed. + ## Sessions as files ```kdl @@ -11,6 +11,7 @@ #include <string.h> #include "json.h" +#include "graphics.h" #include "kdl.h" /* ---- config ------------------------------------------------------------- */ @@ -130,6 +131,8 @@ struct app { char side; /* border press: 'l' 'r' 't' 'b' */ } drag; + graphics_t *gfx; /* kitty images, and what the client has been told */ + /* Where the pointer is. Which border that *means* is derived during the * paint, from the rects painted in that same pass — remembering the answer * instead left the guide describing the layout from before a split, until @@ -374,6 +377,7 @@ app_t *app_new(const char *const argv[], uint16_t cols, uint16_t rows) { a->argv = argv; a->cols = cols; a->rows = rows; + a->gfx = gfx_new(); tab_add(a, ""); a->cur = 0; node_t *leaf = leaf_new(a); @@ -389,6 +393,7 @@ app_t *app_new(const char *const argv[], uint16_t cols, uint16_t rows) { void app_free(app_t *a) { if (!a) return; + gfx_free(a->gfx); free(a->clipboard); free(a->clipboard_pending); for (size_t i = 0; i < a->ntabs; i++) node_free(a->tabs[i].root); @@ -1634,6 +1639,84 @@ void app_compose(app_t *a, screen_t *s) { draw_toasts(a, s); /* and above even that: it is transient */ } +/* ---- kitty graphics ------------------------------------------------------ */ + +struct gfx_ctx { + app_t *a; + node_t *leaf; +}; + +static void gfx_from_pane(pane_t *p, const pane_gfx_t *g, void *ud) { + struct gfx_ctx *c = ud; + node_t *leaf = c->leaf; + + /* Pane-local viewport cells become screen cells, and anything hanging over + * the pane's edge is cropped by asking for fewer columns and rows — the + * only clipping the protocol lets us do without knowing the client's cell + * size in pixels. */ + if (g->col >= leaf->content.w || g->row >= leaf->content.h) return; + uint16_t cols = g->cols, rows = g->rows; + uint32_t sw = g->sw, sh = g->sh; + if (g->col + cols > leaf->content.w) { + uint16_t over = (uint16_t)(g->col + cols - leaf->content.w); + cols = (uint16_t)(cols - over); + uint32_t px = over * g->cell_px_w; + sw = sw > px ? sw - px : 0; + } + if (g->row + rows > leaf->content.h) { + uint16_t over = (uint16_t)(g->row + rows - leaf->content.h); + rows = (uint16_t)(rows - over); + uint32_t px = over * g->cell_px_h; + sh = sh > px ? sh - px : 0; + } + if (!cols || !rows) return; + + gfx_place(c->a->gfx, leaf->id, g->image_id, g->generation, g->place_id, + (uint16_t)(leaf->content.x + g->col), + (uint16_t)(leaf->content.y + g->row), cols, rows, g->sx, g->sy, sw, + sh, g->src_w, g->src_h, g->format, g->compression, g->data, + g->data_len); +} + +static void gfx_leaf_cb(node_t *n, void *ud) { + app_t *a = ud; + if (n->hidden) return; /* a collapsed pane draws nothing, images included */ + struct gfx_ctx ctx = {a, n}; + pane_graphics(n->pane, gfx_from_pane, &ctx); +} + +/* Walk the visible tab's panes and produce the bytes the client's terminal + * needs this frame. Borrowed until the next call. */ +const char *app_graphics(app_t *a, size_t *len) { + gfx_begin(a->gfx); + if (a->ntabs && cur(a)->root) walk(cur(a)->root, gfx_leaf_cb, a); + return gfx_flush(a->gfx, len); +} + +void app_graphics_reset(app_t *a) { gfx_reset(a->gfx); } + +char *app_graphics_json(app_t *a) { + size_t len = 0; + app_graphics(a, &len); /* refresh the model; the bytes are the caller's job */ + const gfx_place_t *places = NULL; + size_t n = gfx_placements(a->gfx, &places); + json_t j; + json_init(&j); + json_arr_open(&j, NULL); + for (size_t i = 0; i < n; i++) { + json_obj_open(&j, NULL); + json_int(&j, "image", places[i].out_id); + json_int(&j, "placement", places[i].place_id); + json_int(&j, "x", places[i].col); + json_int(&j, "y", places[i].row); + json_int(&j, "cols", places[i].cols); + json_int(&j, "rows", places[i].rows); + json_obj_close(&j); + } + json_arr_close(&j); + return j.buf; +} + /* ---- input -------------------------------------------------------------- */ static node_t *by_id(app_t *a, uint32_t id) { @@ -20,6 +20,12 @@ app_t *app_new(const char *const argv[], uint16_t cols, uint16_t rows); void app_free(app_t *a); void app_event(app_t *a, const input_event_t *ev); +/* Kitty graphics for this frame: the bytes the client's terminal needs, and + * the same thing as JSON for tests. Bytes are borrowed until the next call. */ +const char *app_graphics(app_t *a, size_t *len); +void app_graphics_reset(app_t *a); +char *app_graphics_json(app_t *a); + /* Transient announcements, drawn bottom-right and expiring on their own. */ void app_toast(app_t *a, const char *text); size_t app_toast_count(app_t *a); @@ -200,6 +200,10 @@ static char *cmd_json(app_t *a, screen_t *s, input_parser_t *in, app_toast(a, text); return jok_int(NULL, 0); } + if (strcmp(cmd, "graphics") == 0) { + app_compose(a, s); + return jok_raw("placements", app_graphics_json(a)); + } if (strcmp(cmd, "clipboard") == 0) { const char *text = app_clipboard(a); json_t j; diff --git a/src/graphics.c b/src/graphics.c new file mode 100644 index 0000000..bf9aa1d --- /dev/null +++ b/src/graphics.c @@ -0,0 +1,217 @@ +#define _GNU_SOURCE +#include "graphics.h" + +#include <stdarg.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> + +/* Our ids live high, so they cannot collide with images the client's terminal + * already knows about from before we attached. */ +#define OUT_ID_BASE 0x51000000u +#define CHUNK 3072 /* bytes of base64 per escape, kitty's documented ceiling */ + +struct graphics { + gfx_image_t *imgs; + size_t nimgs, imgcap; + gfx_place_t *places; + size_t nplaces, placecap; + uint32_t next_out_id; + + char *out; + size_t len, cap; +}; + +static void out_reserve(graphics_t *g, size_t n) { + if (g->len + n + 1 <= g->cap) return; + size_t cap = g->cap ? g->cap : 8192; + while (cap < g->len + n + 1) cap *= 2; + g->out = realloc(g->out, cap); + g->cap = cap; +} + +static void out_str(graphics_t *g, const char *s) { + size_t n = strlen(s); + out_reserve(g, n); + memcpy(g->out + g->len, s, n); + g->len += n; + g->out[g->len] = 0; +} + +static void out_fmt(graphics_t *g, const char *fmt, ...) { + char buf[256]; + va_list ap; + va_start(ap, fmt); + int n = vsnprintf(buf, sizeof buf, fmt, ap); + va_end(ap); + if (n > 0) out_str(g, buf); +} + +static void out_b64(graphics_t *g, const uint8_t *in, size_t len) { + static const char T[] = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + out_reserve(g, ((len + 2) / 3) * 4); + for (size_t i = 0; i < len; i += 3) { + unsigned v = (unsigned)in[i] << 16; + if (i + 1 < len) v |= (unsigned)in[i + 1] << 8; + if (i + 2 < len) v |= in[i + 2]; + g->out[g->len++] = T[(v >> 18) & 63]; + g->out[g->len++] = T[(v >> 12) & 63]; + g->out[g->len++] = i + 1 < len ? T[(v >> 6) & 63] : '='; + g->out[g->len++] = i + 2 < len ? T[v & 63] : '='; + } + g->out[g->len] = 0; +} + +graphics_t *gfx_new(void) { + graphics_t *g = calloc(1, sizeof *g); + g->next_out_id = OUT_ID_BASE; + return g; +} + +void gfx_free(graphics_t *g) { + if (!g) return; + free(g->imgs); + free(g->places); + free(g->out); + free(g); +} + +void gfx_reset(graphics_t *g) { + for (size_t i = 0; i < g->nimgs; i++) g->imgs[i].sent = false; + g->nplaces = 0; +} + +void gfx_forget_pane(graphics_t *g, uint32_t pane) { + size_t keep = 0; + for (size_t i = 0; i < g->nimgs; i++) + if (g->imgs[i].pane != pane) g->imgs[keep++] = g->imgs[i]; + g->nimgs = keep; +} + +void gfx_begin(graphics_t *g) { + for (size_t i = 0; i < g->nplaces; i++) g->places[i].live = false; + g->len = 0; + if (g->out) g->out[0] = 0; +} + +static gfx_image_t *img_find(graphics_t *g, uint32_t pane, uint32_t src_id) { + for (size_t i = 0; i < g->nimgs; i++) + if (g->imgs[i].pane == pane && g->imgs[i].src_id == src_id) + return &g->imgs[i]; + return NULL; +} + +/* a=t: transmit without placing, so the same bytes can be placed repeatedly */ +static void transmit(graphics_t *g, gfx_image_t *img, uint32_t px_w, + uint32_t px_h, int format, int compression, + const uint8_t *data, size_t len) { + if (!data || !len) return; + int f = format == 0 ? 24 : format == 1 ? 32 : format == 2 ? 100 : 32; + + size_t raw_chunk = (CHUNK / 4) * 3; + for (size_t off = 0; off < len; off += raw_chunk) { + size_t n = len - off < raw_chunk ? len - off : raw_chunk; + bool more = off + n < len; + out_str(g, "\x1b_G"); + if (off == 0) { + out_fmt(g, "a=t,q=2,i=%u,f=%d,s=%u,v=%u", img->out_id, f, px_w, px_h); + if (compression == 1) out_str(g, ",o=z"); + out_fmt(g, ",m=%d;", more ? 1 : 0); + } else { + out_fmt(g, "m=%d;", more ? 1 : 0); + } + out_b64(g, data + off, n); + out_str(g, "\x1b\\"); + } + img->sent = true; +} + +void gfx_place(graphics_t *g, uint32_t pane, uint32_t src_id, uint64_t gen, + uint32_t place_id, uint16_t col, uint16_t row, uint16_t cols, + uint16_t rows, uint32_t sx, uint32_t sy, uint32_t sw, + uint32_t sh, uint32_t px_w, uint32_t px_h, int format, + int compression, const uint8_t *data, size_t data_len) { + gfx_image_t *img = img_find(g, pane, src_id); + if (!img) { + if (g->nimgs == g->imgcap) { + g->imgcap = g->imgcap ? g->imgcap * 2 : 8; + g->imgs = realloc(g->imgs, g->imgcap * sizeof *g->imgs); + } + img = &g->imgs[g->nimgs++]; + *img = (gfx_image_t){ + .pane = pane, .src_id = src_id, .out_id = g->next_out_id++, .gen = gen}; + } + if (img->gen != gen) { /* the program replaced the image under the same id */ + img->gen = gen; + img->sent = false; + } + if (!img->sent) + transmit(g, img, px_w, px_h, format, compression, data, data_len); + + uint32_t pid = place_id ? place_id : 1; + gfx_place_t *slot = NULL; + for (size_t i = 0; i < g->nplaces; i++) + if (g->places[i].out_id == img->out_id && g->places[i].place_id == pid) + slot = &g->places[i]; + if (!slot) { + if (g->nplaces == g->placecap) { + g->placecap = g->placecap ? g->placecap * 2 : 8; + g->places = realloc(g->places, g->placecap * sizeof *g->places); + } + slot = &g->places[g->nplaces++]; + memset(slot, 0, sizeof *slot); + slot->out_id = img->out_id; + slot->place_id = pid; + } + slot->col = col; + slot->row = row; + slot->cols = cols; + slot->rows = rows; + slot->sx = sx; + slot->sy = sy; + slot->sw = sw; + slot->sh = sh; + slot->live = true; +} + +char *gfx_flush(graphics_t *g, size_t *out_len) { + /* Placements that were on screen last frame and are not now must be told to + * go away; a terminal will happily keep drawing an image nobody owns. */ + size_t keep = 0; + for (size_t i = 0; i < g->nplaces; i++) { + gfx_place_t *p = &g->places[i]; + if (p->live) { + g->places[keep++] = *p; + continue; + } + out_fmt(g, "\x1b_Ga=d,d=i,q=2,i=%u,p=%u\x1b\\", p->out_id, p->place_id); + } + g->nplaces = keep; + + /* Then (re)place everything that is. C=1 keeps the cursor where the text + * renderer left it, which matters because we are interleaving with a diff. */ + for (size_t i = 0; i < g->nplaces; i++) { + gfx_place_t *p = &g->places[i]; + out_fmt(g, "\x1b[%u;%uH", p->row + 1, p->col + 1); + out_fmt(g, "\x1b_Ga=p,q=2,C=1,i=%u,p=%u,c=%u,r=%u", p->out_id, p->place_id, + p->cols, p->rows); + /* The source rectangle is what actually crops; c/r alone would scale. */ + if (p->sw || p->sh) + out_fmt(g, ",x=%u,y=%u,w=%u,h=%u", p->sx, p->sy, p->sw, p->sh); + out_str(g, "\x1b\\"); + } + + *out_len = g->len; + return g->out; /* borrowed: valid until the next gfx_begin */ +} + +size_t gfx_placements(const graphics_t *g, const gfx_place_t **out) { + *out = g->places; + return g->nplaces; +} + +size_t gfx_images(const graphics_t *g, const gfx_image_t **out) { + *out = g->imgs; + return g->nimgs; +} diff --git a/src/graphics.h b/src/graphics.h new file mode 100644 index 0000000..1a5572f --- /dev/null +++ b/src/graphics.h @@ -0,0 +1,62 @@ +/* Kitty graphics passthrough. + * + * libghostty-vt parses the protocol and tracks every image and placement, so + * a pane's images exist as data; what a multiplexer has to add is *re-emitting* + * them to the client's terminal at the right place, with ids that cannot + * collide between panes. + * + * This is why pi has to fall back to sixel under tmux and zellij: they drop + * the images entirely. Here the native path works. + */ +#ifndef SL0PTTY_GRAPHICS_H +#define SL0PTTY_GRAPHICS_H + +#include <stdbool.h> +#include <stddef.h> +#include <stdint.h> + +typedef struct graphics graphics_t; + +/* One image, as this session knows it. */ +typedef struct { + uint32_t pane; /* which pane transmitted it */ + uint32_t src_id; /* the id the program chose, unique only within its pane */ + uint32_t out_id; /* the id we give the client, unique across the session */ + uint64_t gen; /* image generation, so a changed image is re-sent */ + bool sent; +} gfx_image_t; + +/* One placement on screen this frame. */ +typedef struct { + uint32_t out_id; + uint32_t place_id; + uint16_t col, row; /* screen cells */ + uint16_t cols, rows; + uint32_t sx, sy, sw, sh; /* source rectangle, in image pixels */ + bool live; +} gfx_place_t; + +graphics_t *gfx_new(void); +void gfx_free(graphics_t *g); + +/* Start a frame: everything placed last frame is marked stale. */ +void gfx_begin(graphics_t *g); +/* Record a visible placement, transmitting the image if the client has not + * seen it yet. `data` may be NULL when the image is already transmitted. */ +void gfx_place(graphics_t *g, uint32_t pane, uint32_t src_id, uint64_t gen, + uint32_t place_id, uint16_t col, uint16_t row, uint16_t cols, + uint16_t rows, uint32_t sx, uint32_t sy, uint32_t sw, + uint32_t sh, uint32_t px_w, uint32_t px_h, int format, + int compression, const uint8_t *data, size_t data_len); +/* End a frame: emit transmits, placements and deletions. Caller frees. */ +char *gfx_flush(graphics_t *g, size_t *out_len); + +/* Forget what the client has seen (it is a different client now). */ +void gfx_reset(graphics_t *g); +/* Drop everything belonging to a pane that is gone. */ +void gfx_forget_pane(graphics_t *g, uint32_t pane); + +size_t gfx_placements(const graphics_t *g, const gfx_place_t **out); +size_t gfx_images(const graphics_t *g, const gfx_image_t **out); + +#endif /* SL0PTTY_GRAPHICS_H */ @@ -424,6 +424,128 @@ void pane_send_paste(pane_t *p, const char *text, size_t len) { free(buf); } +/* ---- kitty graphics ------------------------------------------------------ + * + * lib-vt owns the images and the placements; we walk them once per frame and + * hand each visible one to the graphics layer, which decides what the client + * still needs to be told. + */ +size_t pane_graphics(pane_t *p, pane_gfx_fn cb, void *ud) { + GhosttyKittyGraphics gfx = NULL; + if (ghostty_terminal_get(p->term, GHOSTTY_TERMINAL_DATA_KITTY_GRAPHICS, + &gfx) != GHOSTTY_SUCCESS || + !gfx) + return 0; + + GhosttyKittyGraphicsPlacementIterator it = NULL; + if (ghostty_kitty_graphics_placement_iterator_new(NULL, &it) != + GHOSTTY_SUCCESS) + return 0; + if (ghostty_kitty_graphics_get(gfx, GHOSTTY_KITTY_GRAPHICS_DATA_PLACEMENT_ITERATOR, + &it) != GHOSTTY_SUCCESS) { + ghostty_kitty_graphics_placement_iterator_free(it); + return 0; + } + + size_t n = 0; + while (ghostty_kitty_graphics_placement_next(it)) { + uint32_t image_id = 0, place_id = 0; + ghostty_kitty_graphics_placement_get( + it, GHOSTTY_KITTY_GRAPHICS_PLACEMENT_DATA_IMAGE_ID, &image_id); + ghostty_kitty_graphics_placement_get( + it, GHOSTTY_KITTY_GRAPHICS_PLACEMENT_DATA_PLACEMENT_ID, &place_id); + GhosttyKittyGraphicsImage img = ghostty_kitty_graphics_image(gfx, image_id); + + GhosttyKittyGraphicsPlacementRenderInfo info = + GHOSTTY_INIT_SIZED(GhosttyKittyGraphicsPlacementRenderInfo); + if (ghostty_kitty_graphics_placement_render_info(it, img, p->term, &info) != + GHOSTTY_SUCCESS) + continue; + if (!info.viewport_visible) continue; + + /* Rendered pixels per cell, which the placement itself tells us: a + * placement is N pixels drawn across M cells. That is the conversion the + * protocol needs for any cropping, and it means we never have to know the + * client's font metrics. */ + uint32_t cw = info.grid_cols ? info.pixel_width / info.grid_cols : 0; + uint32_t ch = info.grid_rows ? info.pixel_height / info.grid_rows : 0; + + /* Scrolled partly above or left of the viewport: keep the visible part by + * moving the source rectangle, not by squashing the image into fewer + * cells. */ + uint32_t sx = info.source_x, sy = info.source_y; + uint32_t sw = info.source_width, sh = info.source_height; + int32_t vc = info.viewport_col, vr = info.viewport_row; + uint32_t cols = info.grid_cols, rows = info.grid_rows; + if (vc < 0) { + uint32_t off = (uint32_t)(-vc); + if (off >= cols) continue; + sx += off * cw; + sw = sw > off * cw ? sw - off * cw : 0; + cols -= off; + vc = 0; + } + if (vr < 0) { + uint32_t off = (uint32_t)(-vr); + if (off >= rows) continue; + sy += off * ch; + sh = sh > off * ch ? sh - off * ch : 0; + rows -= off; + vr = 0; + } + if (!cols || !rows) continue; + + pane_gfx_t out = { + .image_id = image_id, + .place_id = place_id, + .col = (uint16_t)vc, + .row = (uint16_t)vr, + .cols = (uint16_t)cols, + .rows = (uint16_t)rows, + .px_w = info.pixel_width, + .px_h = info.pixel_height, + .sx = sx, + .sy = sy, + .sw = sw, + .sh = sh, + .cell_px_w = cw, + .cell_px_h = ch, + }; + + if (img) { + uint32_t w = 0, h = 0; + int fmt = 0, comp = 0; + uint64_t generation = 0; + const uint8_t *data = NULL; + size_t data_len = 0; + ghostty_kitty_graphics_image_get(img, GHOSTTY_KITTY_IMAGE_DATA_WIDTH, &w); + ghostty_kitty_graphics_image_get(img, GHOSTTY_KITTY_IMAGE_DATA_HEIGHT, &h); + ghostty_kitty_graphics_image_get(img, GHOSTTY_KITTY_IMAGE_DATA_FORMAT, &fmt); + ghostty_kitty_graphics_image_get(img, GHOSTTY_KITTY_IMAGE_DATA_COMPRESSION, + &comp); + ghostty_kitty_graphics_image_get(img, GHOSTTY_KITTY_IMAGE_DATA_GENERATION, + &generation); + ghostty_kitty_graphics_image_get(img, GHOSTTY_KITTY_IMAGE_DATA_DATA_PTR, + &data); + ghostty_kitty_graphics_image_get(img, GHOSTTY_KITTY_IMAGE_DATA_DATA_LEN, + &data_len); + out.src_w = w; + out.src_h = h; + out.format = fmt; + out.compression = comp; + out.generation = generation; + out.data = data; + out.data_len = data_len; + } + + cb(p, &out, ud); + n++; + } + + ghostty_kitty_graphics_placement_iterator_free(it); + return n; +} + /* ---- selection ---------------------------------------------------------- * * * A selection is two grid references and a flag, so it can be built directly diff --git a/src/server.c b/src/server.c index 1579281..17ce05a 100644 --- a/src/server.c +++ b/src/server.c @@ -119,6 +119,16 @@ static char *b64(const char *in, size_t len, size_t *out_len) { return out; } +/* Images go out after the cell diff, so a repainted cell cannot land on top + * of a placement we just made. */ +static void push_graphics(server_t *s) { + conn_t *c = display_conn(s); + if (!c) return; + size_t len = 0; + const char *bytes = app_graphics(s->app, &len); + if (len) msg_send(c->fd, MSG_OUTPUT, bytes, len); +} + /* The clipboard lives on the client's machine, not ours, so a copy travels as * OSC 52 for the outer terminal to honour. */ static void push_clipboard(server_t *s) { @@ -280,6 +290,7 @@ int server_run(const char *name, const char *const argv[], uint16_t cols, } } s.screen.force_full = true; /* a fresh client knows nothing */ + app_graphics_reset(s.app); /* including any image we had sent */ pending_paint = true; next_frame = now_ms(); break; @@ -337,6 +348,7 @@ int server_run(const char *name, const char *const argv[], uint16_t cols, if (pending_paint && now_ms() >= next_frame) { app_compose(s.app, &s.screen); push_frame(&s); + push_graphics(&s); push_clipboard(&s); pending_paint = false; next_frame = now_ms(); diff --git a/src/sl0ptty.h b/src/sl0ptty.h index 4e56fac..503eb48 100644 --- a/src/sl0ptty.h +++ b/src/sl0ptty.h @@ -163,6 +163,26 @@ void pane_set_osc_handler(pane_t *p, pane_osc_fn fn, void *ud); void pane_set_clipboard_handler(pane_t *p, pane_clip_fn fn, void *ud); void pane_set_notify_handler(pane_t *p, pane_notify_fn fn, void *ud); +/* One visible kitty graphics placement, in the pane's viewport coordinates. */ +typedef struct { + uint32_t image_id, place_id; + uint16_t col, row, cols, rows; + uint32_t px_w, px_h; /* rendered size */ + uint32_t src_w, src_h; /* the image's own size */ + /* The part of the image to draw, in pixels. Clipping has to move this: + * asking for fewer columns alone *scales* the image into them, it does not + * crop it. */ + uint32_t sx, sy, sw, sh; + uint32_t cell_px_w, cell_px_h; /* derived: rendered pixels per cell */ + int format, compression; + uint64_t generation; + const uint8_t *data; /* borrowed for the callback only */ + size_t data_len; +} pane_gfx_t; + +typedef void (*pane_gfx_fn)(pane_t *p, const pane_gfx_t *g, void *ud); +size_t pane_graphics(pane_t *p, pane_gfx_fn cb, void *ud); + /* Selection, in the pane's viewport coordinates. */ void pane_select_start(pane_t *p, uint16_t x, uint16_t y); void pane_select_extend(pane_t *p, uint16_t x, uint16_t y); diff --git a/tests/test_graphics.py b/tests/test_graphics.py new file mode 100644 index 0000000..586c6f7 --- /dev/null +++ b/tests/test_graphics.py @@ -0,0 +1,150 @@ +#!/usr/bin/env python3 +"""Kitty graphics passthrough. + +libghostty-vt parses the protocol and tracks the images; what a multiplexer +has to add is re-emitting them to the client at the right place, with ids that +cannot collide between panes. That collision is the whole reason tmux and +zellij drop images instead, and it is the first thing tested here. +""" +import base64 +import sys + +from harness import Session, check, report + +# a 4x2 RGB image +PX = base64.b64encode(bytes([255, 0, 0, 0, 255, 0, 0, 0, 255, 255, 255, 0] * 2)).decode() + + +def sends_image(image_id=7, cols=6, rows=2, after="sleep 5"): + seq = f"\\033_Ga=T,f=24,s=4,v=2,i={image_id},q=2,c={cols},r={rows};{PX}\\033\\\\" + return ["/bin/sh", "-c", f'stty raw -echo; printf "{seq}"; {after}'] + + +def places(s): + return s.api("graphics")["placements"] + + +def test_a_pane_image_reaches_the_screen(): + with Session(sends_image(), cols=44, rows=10) as s: + s.settle(200) + p = s.pane() + pl = places(s) + check("the image is placed", len(pl) == 1, str(pl)) + if not pl: + return + check("at the pane's content origin, in screen cells", + (pl[0]["x"], pl[0]["y"]) == (p["content_x"], p["content_y"]), + f"{pl[0]} vs content {p['content_x']},{p['content_y']}") + check("with the size the program asked for", + (pl[0]["cols"], pl[0]["rows"]) == (6, 2), str(pl[0])) + + +def test_ids_cannot_collide_between_panes(): + """Two panes, both using image id 7. They must not become one image.""" + with Session(sends_image(image_id=7), cols=90, rows=12) as s: + s.settle(200) + s.key("\\\\") + s.settle(250) + pl = places(s) + check("both panes place an image", len(pl) == 2, str(pl)) + if len(pl) != 2: + return + check("and they were given different ids for the client", + pl[0]["image"] != pl[1]["image"], str(pl)) + check("side by side, where their panes are", + pl[0]["x"] != pl[1]["x"], str(pl)) + + +def test_placement_follows_the_layout(): + with Session(sends_image(), cols=90, rows=12) as s: + s.settle(200) + before = places(s)[0] + s.key("\\\\") # the pane with the image is now the left half + s.settle(250) + s.api("focus", id=s.panes()[0]["id"]) + s.settle(80) + after = [q for q in places(s) if q["image"] == before["image"]] + check("the placement is still tracked after a split", after != [], + str(places(s))) + + +def test_cropped_at_the_pane_edge(): + with Session(sends_image(cols=40, rows=8), cols=40, rows=10) as s: + s.settle(200) + p = s.pane() + pl = places(s) + check("an oversized image is placed", len(pl) == 1, str(pl)) + if not pl: + return + check("cropped to the columns the pane has", + pl[0]["cols"] <= p["content_w"], f"{pl[0]} vs {p['content_w']}") + check("and to its rows", + pl[0]["rows"] <= p["content_h"], f"{pl[0]} vs {p['content_h']}") + + +def test_hidden_panes_place_nothing(): + with Session(sends_image(), cols=120, rows=30) as s: + s.settle(200) + s.key("\\\\") + s.settle(150) + s.key("\\\\") + s.settle(200) + check("three panes, one with an image", len(places(s)) == 3, str(places(s))) + s.resize(40, 12) # collapses + s.settle(200) + hidden = [q for q in s.panes() if q["hidden"]] + check("something collapsed", hidden != [], str(s.panes())) + check("a collapsed pane places nothing", + len(places(s)) == len(s.panes()) - len(hidden), str(places(s))) + + +def test_placement_goes_away_with_its_pane(): + with Session(sends_image(), cols=90, rows=12) as s: + s.settle(200) + s.key("\\\\") + s.settle(250) + check("two placements", len(places(s)) == 2, str(places(s))) + s.key("x") # close the focused pane + s.settle(200) + check("closing a pane removes its placement", len(places(s)) == 1, + str(places(s))) + + +def test_partial_visibility_crops_rather_than_squashes(): + """An image taller than its pane is cropped by moving the source rect. + + Asking the terminal for fewer rows alone would *scale* the image into + them, which is not what "the rest is off-screen" looks like. + """ + with Session(sends_image(cols=10, rows=20), cols=44, rows=12) as s: + s.settle(250) + p = s.pane() + pl = places(s) + check("a taller-than-the-pane image is still placed", len(pl) == 1, + str(pl)) + if not pl: + return + check("clipped to the pane's rows", pl[0]["rows"] <= p["content_h"], + f"{pl[0]} vs {p['content_h']}") + + +def test_scrolled_away_placements_are_dropped(): + prog = ("stty raw -echo; " + f'printf "\\033_Ga=T,f=24,s=4,v=2,i=7,q=2,c=6,r=2;{PX}\\033\\\\"; ' + "seq 1 200; cat") + with Session(["/bin/sh", "-c", prog], cols=44, rows=10) as s: + s.settle(400) + check("an image scrolled out of the viewport is not placed", + places(s) == [], str(places(s))) + + +if __name__ == "__main__": + test_a_pane_image_reaches_the_screen() + test_ids_cannot_collide_between_panes() + test_placement_follows_the_layout() + test_cropped_at_the_pane_edge() + test_partial_visibility_crops_rather_than_squashes() + test_hidden_panes_place_nothing() + test_placement_goes_away_with_its_pane() + test_scrolled_away_placements_are_dropped() + sys.exit(report()) |
