Export findings as markdown (Ctrl+E), and the journal that makes it true

The output of an RE session is what you worked out, and it was locked in a
.i64 that only IDA can read. Ctrl+E (or `drive export`, or the `export` RPC
verb) writes it out: your comments grouped by function with the line each
annotates, the names and prototypes you set, the types you declared.

**The hard part was provenance, and it needed a mechanism, not a filter.**
A database does not record WHO wrote a comment or a name. IDA's analyzer
sets `; switch 73 cases` and `; s1` with the same `set_cmt` a person uses,
and the ELF loader sets `elf_gnu_hash_nbuckets` and `File class: 64-bit`
the same way. Four probes, all negative: the FF_COMM flag is identical,
`get_cmt` returns them all, `generate_disasm_line` tags every one of them
COLOR_REGCMT (not COLOR_AUTOCMT), and they survive with auto-comments
switched off. A first cut filtered by shape and produced a report whose
first screen was ELF header trivia and `; jumptable ... case 99`.

So idatui journals its own edits (idatui/journal.py) into a netnode in the
database: it rides along in the .i64, it is still there next session, and
the report is then exactly what was done here -- 2 findings out of a
database carrying 693 other annotations. Recorded at the choke points in
edit_ctl (rename, name-address, comment, retype) and in the struct editor;
flushed on save, on export and on quit, so no edit pays a round trip.

Without a journal (a database worked on in the IDA GUI, or predating this)
the report falls back to filtering by shape -- dummy names, imports, loader
segments, the analyzer's stereotyped switch/jumptable strings -- and says
so in the document rather than claiming authorship it cannot prove.

idatui/findings.py splits gather (needs IDA) from render (does not), so the
formatting, grouping, sorting, escaping and the empty cases are tested
offline: tests/test_findings.py, 32 checks, no worker, 0.1s. The pilot
scenario covers the round trip that matters -- edit through the UI, export,
find it in the file, and reload the journal from the .i64.

Full suite: 842 passed, 0 failed, 51.2s.

2 files changed, 276 insertions, 0 deletions

diff --git a/tests/test_findings.py b/tests/test_findings.py
new file mode 100644
index 0000000..cf813c2
--- /dev/null
+++ b/tests/test_findings.py
@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+"""The findings export: gathering and, mostly, RENDERING.
+
+`idatui.findings.render` takes plain data and returns markdown, so all of the
+interesting behaviour -- grouping, sorting, what an empty section says, and
+whether hostile text can break out of a table cell or a code fence -- is
+testable with no IDA, no worker and no binary. `gather` is covered against a
+fake Program that answers like the real one, including by raising.
+"""
+
+#: pure: stdlib only.
+#: Read by tests/run.py (--fast skips every NEEDS_IDA file).
+NEEDS_IDA = False
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from idatui.domain import Comment, NamedItem, Struct  # noqa: E402
+from idatui.findings import (  # noqa: E402
+    Findings, default_path, from_loader, gather, is_dummy, render,
+)
+
+PASS = FAIL = 0
+
+
+def check(name, cond, detail=""):
+    global PASS, FAIL
+    if cond:
+        PASS += 1
+        print(f"  ok   {name}")
+    else:
+        FAIL += 1
+        print(f"  FAIL {name}   {detail}")
+
+
+def sample() -> Findings:
+    return Findings(
+        binary="echo", path="/tmp/echo",
+        sections=[(0x1000, 0x2000, ".text"), (0x2000, 0x2100, ".data")],
+        n_functions=128,
+        comments=[
+            Comment(addr=0x1100, text="length is attacker controlled",
+                    line="mov edi, [rbp+len]", func="parse", func_addr=0x1000),
+            Comment(addr=0x1010, text="entry", line="push rbp",
+                    func="parse", func_addr=0x1000),
+            Comment(addr=0x1000, text="parses the header", whole_func=True,
+                    func="parse", func_addr=0x1000),
+            Comment(addr=0x2004, text="magic", line="dd 0DEADBEEFh"),
+            # What the ELF loader writes into every database, through the very
+            # same set_cmt a person uses.
+            Comment(addr=0x4, text="File class: 64-bit", line="db 2",
+                    seg="LOAD"),
+        ],
+        names=[
+            NamedItem(addr=0x1000, name="parse", is_func=True, size=0x120,
+                      proto="int __fastcall parse(char *)"),
+            NamedItem(addr=0x1200, name="sub_1200", is_func=True, size=0x30),
+            NamedItem(addr=0x2004, name="hdr_magic", seg=".data"),
+            NamedItem(addr=0x1400, name="memcpy", is_func=True, size=0x40),
+            NamedItem(addr=0x390, name="elf_gnu_hash_nbuckets", seg="LOAD"),
+        ],
+        types=[(Struct(name="hdr", size=0x10, is_union=False, members=3,
+                       ordinal=42), "struct hdr\n{\n  int magic;\n};\n"),
+               (Struct(name="Elf64_Dyn", size=0x10, is_union=False, members=2,
+                       ordinal=3), "struct Elf64_Dyn\n{\n  int d_tag;\n};\n")],
+        linked={"memcpy"},
+        stripped=True,
+    )
+
+
+def main() -> int:
+    doc = render(sample())
+
+    check("the report names the binary", doc.startswith("# Findings — echo"), doc[:40])
+    # 1 function, not 3: sub_1200 is IDA's invention and memcpy is the linker's.
+    check("the summary counts only what a person contributed",
+          "1 named functions · 1 named data · 4 comments · 2 local types" in doc,
+          doc.splitlines()[2] if len(doc.splitlines()) > 2 else "")
+
+    # A name IDA invented is not a finding, and neither is one the linker gave.
+    check("dummy names are excluded", "sub_1200" not in doc)
+    check("imported names are excluded", "`memcpy`" not in doc)
+    check("real names survive", "`parse`" in doc and "`hdr_magic`" in doc)
+
+    # The loader annotates every database it makes; none of it is a finding.
+    check("the loader's own comments are left out", "File class" not in doc)
+    check("the loader's own names are left out", "elf_gnu_hash" not in doc)
+    check("but the report says how many it dropped",
+          "4 annotations left out as the loader's own" in doc,   # 1 comment + 3 names
+          [l for l in doc.splitlines() if "left out" in l])
+    check("from_loader knows both shapes",
+          from_loader("LOAD") and from_loader("", "elf_gnu_hash_x")
+          and not from_loader(".text", "parse"))
+    check("is_dummy knows the shapes IDA invents",
+          all(is_dummy(n) for n in ("sub_1234", "loc_A0", "unk_4000", "j_free"))
+          and not any(is_dummy(n) for n in ("parse", "sub_parse", "main", "")),
+          "")
+
+    # Comments lead, grouped by function, address-ordered within a group.
+    check("comments come before the name tables",
+          doc.index("## Comments") < doc.index("## Named functions"))
+    body = doc[doc.index("## Comments"):doc.index("## Named functions")]
+    check("comments are grouped under their function", "### `parse`" in body)
+    check("a commentless region is grouped separately",
+          "### outside any function" in body)
+    check("comments are ordered by address inside a group",
+          body.index("0x1010") < body.index("0x1100"))
+    check("a function comment says that is what it is",
+          "*whole function*: parses the header" in body)
+    check("an instruction comment carries the line it annotates",
+          "`mov edi, [rbp+len]`" in body)
+
+    # Types: newest ordinal first, because that is the one you just wrote.
+    types = doc[doc.index("## Local types"):]
+    check("your newest type is first",
+          types.index("hdr") < types.index("Elf64_Dyn"))
+    check("type source is fenced as C", "```c\nstruct hdr" in types)
+
+    # Escaping.
+    hostile = Findings(binary="x", comments=[
+        Comment(addr=1, text="a | b", line="mov | rax"),
+    ], names=[NamedItem(addr=2, name="a|b")])
+    hdoc = render(hostile)
+    check("a pipe cannot break a table row", "a\\|b" in hdoc, hdoc)
+    check("a pipe in a comment is escaped too", "a \\| b" in hdoc)
+
+    # The empty database must still produce a document that says something.
+    empty = render(Findings(binary="nothing"))
+    check("an empty report is still a document",
+          empty.startswith("# Findings — nothing") and "## Comments" in empty)
+    check("and it says why it is empty", "Comments are the part" in empty)
+    check("an empty report has no dangling type section",
+          "## Local types" not in empty)
+
+    # Provenance must be stated, not implied. Without a journal the report is a
+    # scan and says so; with one it is exactly what idatui recorded doing.
+    scanned = render(Findings(binary="x", stripped=False,
+                              names=[NamedItem(addr=1, name="main", is_func=True)]))
+    check("a scanned report admits it cannot know who wrote what",
+          "**source**: a scan of the database" in scanned
+          and "include its work as well as yours" in scanned)
+    check("and warns when the binary brought its own symbols",
+          "include ones it shipped with" in scanned)
+
+    j = sample()
+    j.recorded = {0x1100, 0x1000}
+    j.n_recorded = 7
+    jdoc = render(j)
+    check("a journalled report says so", "idatui's edit journal" in jdoc
+          and "7 recorded edits" in jdoc, "")
+    jbody = jdoc[jdoc.index("## Comments"):jdoc.index("## Named functions")]
+    check("and lists only the comments it recorded",
+          "length is attacker controlled" in jbody and "0x2004" not in jbody,
+          jbody)
+    check("a journalled report drops names it did not record",
+          "`parse`" in jdoc and "hdr_magic" not in jdoc)
+
+    # -- gather ------------------------------------------------------------- #
+    class FakeProgram:
+        def sections(self):
+            return [(0x1000, 0x2000, ".text")]
+
+        def annotations(self, limit=4000):
+            return ([Comment(addr=1, text="hi")],
+                    [NamedItem(addr=1, name="parse", is_func=True)])
+
+        def linkage(self):
+            return ([], [])
+
+        def functions(self):
+            raise RuntimeError("index unavailable")
+
+        def list_structs(self):
+            raise RuntimeError("no types")
+
+        def struct_source(self, name):
+            return ""
+
+    f = gather(FakeProgram(), "/tmp/echo")
+    check("gather reads the annotations", len(f.comments) == 1 and len(f.names) == 1)
+    check("gather takes the binary name from the path", f.binary == "echo", f.binary)
+    check("a failing backend degrades the report instead of raising",
+          f.n_functions == 0 and f.types == [] and "# Findings" in render(f))
+
+    check("the default path sits beside the binary",
+          default_path("/tmp/echo") == "/tmp/echo.findings.md")
+
+    print(f"\n{PASS} passed, {FAIL} failed")
+    return 1 if FAIL else 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
diff --git a/tests/test_scenarios.py b/tests/test_scenarios.py
index 6c3177a..0b004b3 100644
--- a/tests/test_scenarios.py
+++ b/tests/test_scenarios.py
@@ -1093,6 +1093,88 @@ async def s_structs(c: Ctx):
     c.check("Esc closes the struct editor", not isinstance(app.screen, StructEditor))
 
 
+@scenario("export_findings")
+async def s_export_findings(c: Ctx):
+    """Ctrl+E writes a markdown report of what this session worked out.
+
+    Deliberately end-to-end: the interesting failure is not the formatting (that
+    is covered offline in test_findings.py) but whether a comment and a rename
+    made through the UI come back out of the database and into the file.
+    """
+    import tempfile
+
+    from idatui.findings import default_path
+
+    app = c.app
+    fn = c.biggest()
+    tag = os.getpid()
+    newname, note = f"exp_{tag}", f"found_it_{tag}"
+    old = fn.name
+    await c.open(fn.addr, "listing")
+
+    # Make something to find: a rename and a comment, through the real paths.
+    app.program.client.invoke(
+        "rename", batch={"func": {"addr": hex(fn.addr), "name": newname}})
+    app.program.bump_names()
+    app.program.set_comment(fn.addr, note)
+    app.program.invalidate(fn.addr)
+    # ...and tell the journal, exactly as the edit controller would. The
+    # database cannot say who wrote a comment (IDA's own analyzer uses the same
+    # call), so the journal is what makes this MY finding rather than noise.
+    app.journal.record("rename", fn.addr, f"{old} → {newname}")
+    app.journal.record("comment", fn.addr, note)
+
+    out = os.path.join(tempfile.gettempdir(), f"idatui-findings-{tag}.md")
+    try:
+        await c.press("ctrl+e")
+        inp = app.query_one("#export", Input)
+        opened = await c.wait(lambda: inp.display, 5)
+        c.check("Ctrl+E opens the export prompt", opened, f"display={inp.display}")
+        c.check("the prompt is prefilled with a path beside the binary",
+                inp.value == default_path(app._open_path), f"value={inp.value!r}")
+        inp.value = out
+        await c.press("enter")
+        written = await c.wait(lambda: os.path.exists(out), 30)
+        c.check("Enter writes the report", written, f"no {out}")
+        if not written:
+            return
+        doc = open(out, encoding="utf-8").read()
+        c.check("the report is markdown with the expected sections",
+                doc.startswith("# Findings") and "## Comments" in doc
+                and "## Named functions" in doc, doc[:60])
+        c.check("a comment written this session is in it", note in doc,
+                doc[:200])
+        c.check("and the function it belongs to is named", newname in doc,
+                doc[:200])
+        c.check("the report is sourced from the journal, not a scan",
+                "idatui's edit journal" in doc,
+                [l for l in doc.splitlines() if "**source**" in l])
+        c.check("the analyzer's own comments stay out of it",
+                "switch jump" not in doc and "jumptable" not in doc,
+                [l for l in doc.splitlines() if "switch" in l][:2])
+        c.check("the status line says where it went",
+                out in c.status(), c.status())
+        # The journal has to survive the database, or a report is only ever
+        # about the session that happened to be open.
+        from idatui.journal import Journal
+
+        app.journal.flush(app.program)
+        reloaded = Journal()
+        reloaded.load(app.program)
+        c.check("the journal round-trips through the .i64",
+                fn.addr in reloaded.addresses(),
+                f"{len(reloaded)} entries, {sorted(reloaded.addresses())[:3]}")
+    finally:
+        # Idempotent: hand the database back exactly as we found it.
+        app.program.set_comment(fn.addr, "")
+        app.program.client.invoke(
+            "rename", batch={"func": {"addr": hex(fn.addr), "name": old}})
+        app.program.bump_names()
+        app.program.invalidate(fn.addr)
+        if os.path.exists(out):
+            os.remove(out)
+
+
 @scenario("struct_filter")
 async def s_struct_filter(c: Ctx):
     app = c.app