listing: syntax-highlight assembly from IDA's own token tags

The listing showed the mnemonic bright and every operand in one body colour.
IDA already classifies each token, for every processor it supports:
generate_disasm_line() emits \x01<tag>text\x02<tag> and the tag says what the
text IS. We were calling tag_remove() and throwing that away.

So: no lexer. A pygments asm lexer would be a worse guess and would need one
dialect per architecture — this is arch-correct for free, including the ARM/MIPS
blobs the loader work just made openable.

  lea rcx, function; "usage"
  insn  reg punct   name   cmt

_idatui_spans() parses the tags into [[kind, text], ...], heads rows carry
"spans", Head.spans holds them, and _span_segments() renders them with a
fallback to the old mnemonic/rest split for older workers.

Palette rule: NEUTRALS for the machine (mnemonic brightest — it's the column you
scan; registers at body weight because they're most of the text), HUES only where
they mean something (numbers, strings, symbols), structure recedes so commas and
brackets stop competing with operands.

Two things that fail SILENTLY and are now encoded:
* The constants are SCOLOR_DATNAME / SCOLOR_CODNAME. There is no SCOLOR_DNAME —
  a wrong guess leaves the tag unmapped, symbols render as plain body text, and
  nothing tells you why. Probed the live IDA to get the real names.
* Spans must be whitespace-collapsed exactly as `text` is, walking characters
  rather than per span, because a run of IDA's column padding straddles span
  boundaries. A row only gets spans when they reconstruct `text` exactly, so a
  mismatch degrades to the old rendering instead of corrupting the line.

The reason this was parked yesterday was NOT a bug in it. listing_view's
"undefining a data head yields an unknown run" waits for
`index_of_ea(dea) >= 0` — but dea is the head it just undefined, so it is in the
OLD model too and the predicate passes instantly, asserting against pre-edit
rows. It only ever passed because the model swap won the race; spans made pages
3x bigger, the swap lost, and the check accused working code. It now waits for
the model to be REPLACED.

Cost measured on libcrypto: 95KB per 500-row page, 50ms; model ensure(2000)
228ms. Acceptable for what it buys.

tests: new asm_highlight scenario (+7) — >90% of code rows carry spans, insn/reg/
punct present, every span kind has a style, spans reconstruct the row text
exactly, mnemonic is the first span. 202/0 scenarios, 26/0 blob, 30/0 project UI.

TODO: DisasmView appears to be dead code (never instantiated; Ctx.dis returns
ListingView), which is why this only needed doing once.

1 files changed, 55 insertions, 1 deletions

diff --git a/tests/test_scenarios.py b/tests/test_scenarios.py
index 6ccbf0d..abffe3c 100644
--- a/tests/test_scenarios.py
+++ b/tests/test_scenarios.py
@@ -395,6 +395,53 @@ async def s_load_options(c: Ctx):
     await c.wait(lambda: not isinstance(app.screen, LoadOptionsScreen), 10)
 
 
+@scenario("asm_highlight")
+async def s_asm_highlight(c: Ctx):
+    """Assembly is highlighted from IDA's OWN token tags.
+
+    generate_disasm_line() emits \x01<tag>text\x02<tag>, and the tag says what
+    the text is — for every processor IDA supports. Nothing is lexed here, so
+    what this pins is that the tags survive the trip: parsed server-side, agreed
+    with the plain text, carried on the Head, and mapped to a style.
+    """
+    from idatui.app import _S_SPAN
+    app = c.app
+    await c.open_biggest("listing")
+    lst = c.lst
+    ok = await c.wait(lambda: lst.model is not None and lst.total > 20, 25)
+    if not ok:
+        c.check("a listing to highlight", False, f"total={lst.total}")
+        return
+    lst.model.ensure(400)
+    rows = [lst.model.get(i) for i in range(min(lst.model.loaded(), 400))]
+    rows = [h for h in rows if h is not None]
+    code = [h for h in rows if h.kind == "code"]
+    c.check("code rows carry IDA's token spans",
+            code and sum(1 for h in code if h.spans) > len(code) * 0.9,
+            f"{sum(1 for h in code if h.spans)}/{len(code)} have spans")
+
+    kinds = {k for h in rows for k, _ in (h.spans or ())}
+    # If a tag isn't mapped it renders as body text and nothing says why, so the
+    # ones that carry real meaning are worth asserting explicitly.
+    for want in ("insn", "reg", "punct"):
+        c.check(f"the palette sees {want} tokens", want in kinds, f"{sorted(kinds)}")
+    c.check("every span kind has a style",
+            all(k in _S_SPAN for k in kinds), f"unstyled: {sorted(kinds - set(_S_SPAN))}")
+
+    # Spans must describe the SAME text the row shows, or the row renders
+    # different characters than search/width calculations think it has.
+    bad = [h for h in rows if h.spans
+           and "".join(t for _k, t in h.spans) != h.text]
+    c.check("spans reconstruct the row text exactly", not bad,
+            f"{[(hex(h.ea), h.text) for h in bad[:2]]}")
+
+    # And the mnemonic must be the loudest thing on the line (the column you
+    # scan), not just any styled token.
+    mn = next((h for h in code if h.spans and h.spans[0][0] == "insn"), None)
+    c.check("the mnemonic is the first span",
+            mn is not None, f"{code[0].spans if code else None}")
+
+
 @scenario("command_palette")
 async def s_command_palette(c: Ctx):
     app = c.app
@@ -1949,9 +1996,16 @@ async def s_listing_view(c: Ctx):
     try:
         c.prog.undefine(dea, size=4)
         c.prog.bump_items()
-        # reopen the listing so the model reflects the new undefined run
+        # Reopen the listing so the model reflects the new undefined run, and
+        # wait for the model to be REPLACED. Waiting on index_of_ea(dea) >= 0 is
+        # no test at all: dea is in the OLD model too (it's the head we just
+        # undefined), so the predicate passes instantly and we then assert
+        # against pre-edit rows. It only ever passed because the swap happened to
+        # win the race, and it started failing the moment page loads got bigger.
+        stale = c.lst.model
         await c.goto_ui(hex(dea))
         await c.wait(lambda: app._active == "listing" and c.lst.total > 0
+                     and c.lst.model is not stale
                      and c.lst.model.index_of_ea(dea) >= 0, 25)
         ui = c.lst.model.index_of_ea(dea)
         c.check("undefining a data head yields an unknown run in the listing",