aboutsummaryrefslogtreecommitdiffstats
path: root/tests/test_skill.py
diff options
context:
space:
mode:
authoruser <user@clank>2026-08-17 13:08:06 +0200
committeruser <user@clank>2026-08-17 13:08:06 +0200
commit244068e86c03b6c1bf457fac7e576bf5f68a43b7 (patch)
treec294bf9fdd75d6fb487cc3d820ec955d0f799146 /tests/test_skill.py
parentsl0ppty: workspaces, and layouts that write themselves (diff)
downloadslosh-244068e86c03b6c1bf457fac7e576bf5f68a43b7.tar.gz
slosh-244068e86c03b6c1bf457fac7e576bf5f68a43b7.tar.xz
slosh-244068e86c03b6c1bf457fac7e576bf5f68a43b7.zip
sl0ppty: a skill for driving it, and the two variables that made one possible
`.agents/skills/driving-sl0ppty/SKILL.md`. The convention is `.agents/skills/<name>/SKILL.md` with YAML frontmatter -- already in this tree under `vendor/libghostty-vt`, so it is not an invention -- and it means an agent working in a checkout picks the skill up without being told, while a directory per skill leaves room for adjuncts later. `contrib/README.md` and `docs/scripting.md` say where it is and how to use it elsewhere: copy or symlink the directory into wherever your agent keeps skills. **Writing it turned up the reason there was no skill worth writing.** A pane is told `SL0PPTY=1` and nothing else, so a program inside one knows it is in a session and cannot find out *which* -- `ls` lists them all and none of them is labelled "the one you are in". So the whole most valuable case, an agent driving the session it is running in, was guesswork. The server now sets `SL0PPTY_SESSION` and `SL0PPTY_BIN` in its own environment, which every pane inherits through fork: one place, no signatures threaded, and unset under `--script`, which has no socket to name. `SL0PPTY_BIN` is read from `/proc/self/exe` rather than `argv[0]`, and exists because the failure it prevents is the one I hit writing the examples -- a session started from a build tree, and `sl0ppty` not on the pane's `PATH`, so the skill's own commands did not run. What the skill actually teaches, none of which `docs/scripting.md` says because a reference is not allowed to have opinions: - Work belongs in a pane that was **given** a command, not typed into somebody's shell. Then `alive` and `exit_code` off `panes` are how you wait, and the output survives the program (D14) so it is still there to read. - There is **no wait primitive over the socket**: `settle` is headless-only, and `deadline` is about repaints and says nothing about program output. Poll. - `purpose` is the handle. Titles change underneath you; a declared purpose cannot be relabelled by the program in the pane, which is what makes it safe to address. - `snapshot` is the whole composited session, and only what is on screen. Cut a pane out with its `content_*` rect; do not expect scrollback. - **The echoed command line matches your own marker.** `echo DONE` puts `DONE` on screen twice. Count, or use a marker you did not send. - Write `\e`, never `\033`: the unescaper takes `\0` first, so `\033` arrives as NUL followed by `33` and nothing complains. That bug was live in this repo's own test suite until last week. - A program in a pane can draw its own status and real buttons with OSC 5577 and read the click off its stdin, which is how it asks a question instead of printing a prompt and hoping. - Close what you made; do not `quit` a session you did not start. `tests/test_skill.py` is the part that keeps it true. A skill is documentation an agent acts on without a human reading it first, so a stale one is worse than a missing one: a wrong verb is a failed tool call somebody else has to debug. Every verb the file names is sent to a live session and must not answer `unknown cmd`; every `panes` field its central recipe polls must exist; the environment it promises is read out of a real pane; the recipe itself is run end to end and must report exit status 3 with its output still on screen. It also greps the file for paths out of a home directory, because "plug and play" is a property somebody has to keep. That test found `close-pane` in the draft, which is the *keybinding* action name -- the verb is `close`. Exactly the class of error the test exists for, found before anybody's agent ran it. Verified: 1795 checks, all green, 43 of them in `tests/test_skill.py`. Then the skill's recipe run verbatim against a real session -- tagged pane, polled to `alive:false`, `exit status: 3`, `compiling`/`built`/`exited: status 3` still readable -- and from inside a pane, `session=agentdemo bin=/…/sl0ppty` and a `building 3/7` status drawn in its own frame.
Diffstat (limited to 'tests/test_skill.py')
-rwxr-xr-xtests/test_skill.py162
1 files changed, 162 insertions, 0 deletions
diff --git a/tests/test_skill.py b/tests/test_skill.py
new file mode 100755
index 0000000..5bf348b
--- /dev/null
+++ b/tests/test_skill.py
@@ -0,0 +1,162 @@
+#!/usr/bin/env python3
+"""`.agents/skills/driving-sl0ppty/SKILL.md`: the agent-facing interface, checked.
+
+A skill is documentation an agent acts on without a human reading it first, which
+makes a stale one worse than a missing one: a wrong verb becomes a failed tool call
+somebody has to debug, and a verb we removed becomes an agent that cannot do the
+thing it was told it could. So every verb, every environment variable and every
+`panes` field the skill names is checked against the program here.
+
+It is deliberately not checked for prose. It is checked for claims that can rot.
+"""
+import json
+import os
+import re
+import subprocess
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from harness import BIN, Session, check, report
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+SKILL = os.path.join(ROOT, ".agents", "skills", "driving-sl0ppty", "SKILL.md")
+SH = ["/bin/sh", "-c", "read x"]
+
+
+def text():
+ with open(SKILL) as f:
+ return f.read()
+
+
+def test_the_skill_is_where_agents_look_for_it():
+ check("it exists", os.path.exists(SKILL), SKILL)
+ body = text()
+ check("it opens with YAML frontmatter", body.startswith("---\n"), body[:40])
+ fm = body.split("---")[1]
+ check("naming itself for its directory", "name: driving-sl0ppty" in fm, fm)
+ check("and saying when it applies", "description:" in fm, fm)
+ # The description is what a harness matches on, so it has to mention the
+ # things a user would say.
+ for word in ("sl0ppty", "pane", "workspace", "SL0PPTY"):
+ check("the description mentions " + word, word in fm, fm)
+
+
+def test_every_verb_it_names_is_a_verb_the_program_has():
+ """The check that matters: a skill naming a verb we do not have is a tool call
+ that fails in front of somebody."""
+ body = text()
+ named = set(re.findall(r'\{"cmd":"([a-z-]+)"', body))
+ named |= set(re.findall(r'^\| `([a-z-]+)`(?: `([a-z-]+)`)*', body, re.M))
+ # the verb table lists several per row, backticked
+ for row in re.findall(r"^\| ((?:`[a-z-]+` ?)+)\|", body, re.M):
+ named |= set(re.findall(r"`([a-z-]+)`", row))
+ named = {v for v in named if isinstance(v, str) and v}
+ check("the skill names a useful number of verbs", len(named) > 20, str(named))
+
+ with Session(SH, cols=80, rows=12) as s:
+ unknown = []
+ for verb in sorted(named):
+ if verb == "quit": # would end the session under test
+ continue
+ r = s.api(verb)
+ if r.get("ok") is False and "unknown cmd" in r.get("error", ""):
+ unknown.append(verb)
+ check("every verb it names exists", not unknown,
+ "unknown: " + str(unknown))
+
+
+def test_the_environment_it_promises_is_the_environment_panes_get():
+ body = text()
+ for var in ("SL0PPTY", "SL0PPTY_SESSION", "SL0PPTY_BIN"):
+ check("the skill documents " + var, var in body, "missing " + var)
+
+ # SL0PPTY is set by pty.c for every pane, in every mode.
+ with Session(["/bin/sh", "-c", 'echo "V=[$SL0PPTY]"; read x']) as s:
+ s.settle(120)
+ check("a pane really is told it is in one",
+ "V=[1]" in s.snapshot().screen(), s.snapshot().screen())
+
+ # The other two are set by the server, which --script is not. The skill says
+ # so; this is that sentence being true.
+ with Session(["/bin/sh", "-c", 'echo "S=[$SL0PPTY_SESSION]"; read x']) as s:
+ s.settle(120)
+ check("and that --script has no session to name",
+ "S=[]" in s.snapshot().screen(), s.snapshot().screen())
+ flat = " ".join(body.split()) # the file is wrapped; the claim is not
+ check("which the skill says out loud",
+ "unset under `--script`" in flat, "not documented")
+
+
+def test_the_panes_fields_it_tells_agents_to_poll_are_real():
+ """The skill's central recipe polls `alive` and `exit_code` off `panes` and
+ filters on `purpose`. If any of those three moved, the recipe is a hang."""
+ body = text()
+ with Session(SH, cols=80, rows=12) as s:
+ pane = s.panes()[0]
+ for field in ("id", "purpose", "alive", "exit_code", "tab_id",
+ "content_x", "content_y", "content_w", "content_h"):
+ check("panes reports " + field, field in pane, str(pane))
+ check("and the skill names it", field in body, "missing " + field)
+
+
+def test_the_recipe_in_the_skill_actually_works():
+ """Run the skill's own worked example: a tagged command pane, polled to
+ completion, exit status read off `panes`. Copied from the file rather than
+ paraphrased, so a change to the file is a change to what is tested."""
+ with Session(SH, cols=90, rows=14) as s:
+ s.api("apply-layout", kdl='layout { tab name="build" '
+ '{ pane purpose="task:build" command="sh -c \'echo built; exit 3\'" } }')
+ target = None
+ for _ in range(200):
+ s.settle(30)
+ got = [p for p in s.panes() if p["purpose"] == "task:build"]
+ if got and not got[0]["alive"]:
+ target = got[0]
+ break
+ check("the tagged pane is findable by purpose", target is not None,
+ str(s.panes()))
+ check("it reports the real exit status once it stops",
+ target and target["exit_code"] == 3, str(target))
+ check("and its output is still readable afterwards",
+ "built" in s.snapshot().screen(), s.snapshot().screen())
+
+
+def test_it_is_free_of_this_machine():
+ """"Plug and play" is a property somebody has to keep. A path out of the
+ author's home directory in a skill is a path an end user cannot use."""
+ body = text()
+ bad = []
+ for pat in (r"/home/[a-z]", r"/Users/[a-z]", r"/tmp/[a-z0-9]",
+ r"\bclank\b", re.escape(os.path.expanduser("~"))):
+ bad += [m.group(0) for m in re.finditer(pat, body)]
+ check("no path from the machine it was written on", not bad, str(set(bad)))
+ check("it reaches the binary through the environment",
+ "SL0PPTY_BIN" in body and "not on your `PATH`" in body, "no PATH note")
+
+
+def test_settle_is_not_promised_over_the_socket():
+ """The skill tells agents to poll, because the thing that would save them from
+ polling is headless-only. If that ever changes, the skill is understating what
+ the program can do -- which is the good direction, but it should not be
+ accidental."""
+ with Session(SH, cols=80, rows=10) as s:
+ r = s.api("settle", ms=10)
+ check("settle is not a socket verb", r.get("ok") is False, str(r))
+ check("and the skill says there is no wait primitive",
+ "no wait primitive" in text(), "not documented")
+
+
+def test_the_skill_is_reachable_from_the_docs_people_read():
+ """An agent finds it by convention. A human has to be told it is there."""
+ for path, what in ((os.path.join(ROOT, "contrib", "README.md"), "contrib"),
+ (os.path.join(ROOT, "docs", "scripting.md"), "scripting")):
+ with open(path) as f:
+ body = f.read()
+ check("%s points at the skill" % what, "driving-sl0ppty" in body, path)
+
+
+if __name__ == "__main__":
+ for name, fn in sorted(list(globals().items())):
+ if name.startswith("test_"):
+ fn()
+ sys.exit(report())