Export findings as markdown (Ctrl+E), and the journal that makes it true

The output of an RE session is what you worked out, and it was locked in a
.i64 that only IDA can read. Ctrl+E (or `drive export`, or the `export` RPC
verb) writes it out: your comments grouped by function with the line each
annotates, the names and prototypes you set, the types you declared.

**The hard part was provenance, and it needed a mechanism, not a filter.**
A database does not record WHO wrote a comment or a name. IDA's analyzer
sets `; switch 73 cases` and `; s1` with the same `set_cmt` a person uses,
and the ELF loader sets `elf_gnu_hash_nbuckets` and `File class: 64-bit`
the same way. Four probes, all negative: the FF_COMM flag is identical,
`get_cmt` returns them all, `generate_disasm_line` tags every one of them
COLOR_REGCMT (not COLOR_AUTOCMT), and they survive with auto-comments
switched off. A first cut filtered by shape and produced a report whose
first screen was ELF header trivia and `; jumptable ... case 99`.

So idatui journals its own edits (idatui/journal.py) into a netnode in the
database: it rides along in the .i64, it is still there next session, and
the report is then exactly what was done here -- 2 findings out of a
database carrying 693 other annotations. Recorded at the choke points in
edit_ctl (rename, name-address, comment, retype) and in the struct editor;
flushed on save, on export and on quit, so no edit pays a round trip.

Without a journal (a database worked on in the IDA GUI, or predating this)
the report falls back to filtering by shape -- dummy names, imports, loader
segments, the analyzer's stereotyped switch/jumptable strings -- and says
so in the document rather than claiming authorship it cannot prove.

idatui/findings.py splits gather (needs IDA) from render (does not), so the
formatting, grouping, sorting, escaping and the empty cases are tested
offline: tests/test_findings.py, 32 checks, no worker, 0.1s. The pilot
scenario covers the round trip that matters -- edit through the UI, export,
find it in the file, and reload the journal from the .i64.

Full suite: 842 passed, 0 failed, 51.2s.

1 files changed, 101 insertions, 0 deletions

diff --git a/idatui/codemode_client.py b/idatui/codemode_client.py
index 88d7b31..4c3ec99 100644
--- a/idatui/codemode_client.py
+++ b/idatui/codemode_client.py
@@ -501,6 +501,107 @@ for item in page:
 result = {"strings": rows, "total": len(items), "next_offset": offset + len(rows)}
 result
 ''',
+    # Everything a person ADDED to the database: comments, non-dummy names, and
+    # the prototypes they set.
+    #
+    # Names come from IDA's name list, which is already an index -- no scan at
+    # all. Comments have no index, so they need a walk, and the walk is over
+    # HEADS: `next_that`'s predicate is a *Python* callback (SWIG calls it with
+    # one argument, so `f_has_cmt` does not even fit), which would be one call
+    # per BYTE -- 400 million of them on a big image. `max_scan` bounds it and
+    # reports `truncated` rather than sitting there.
+    "list_annotations": r'''
+import ida_bytes, ida_funcs, ida_lines, ida_nalt, ida_name
+import ida_segment, ida_typeinf, idautils
+limit = max(1, int(a.get("limit", 4000)))
+max_scan = max(1000, int(a.get("max_scan", 2000000)))
+comments, names = [], []
+scanned = 0
+
+def _line(ea):
+    try:
+        txt = ida_lines.generate_disasm_line(ea, ida_lines.GENDSM_REMOVE_TAGS)
+    except Exception:
+        txt = ""
+    return " ".join((txt or "").split())
+
+for ea, nm in idautils.Names():
+    if len(names) >= limit:
+        break
+    if not nm or not ida_bytes.has_user_name(ida_bytes.get_flags(ea)):
+        continue
+    fn = ida_funcs.get_func(ea)
+    is_fn = fn is not None and int(fn.start_ea) == int(ea)
+    proto = None
+    if is_fn:
+        try:
+            ti = ida_typeinf.tinfo_t()
+            if ida_nalt.get_tinfo(ti, ea):
+                proto = str(ti)
+        except Exception:
+            proto = None
+    seg = ida_segment.getseg(ea)
+    names.append({"addr": hex(int(ea)), "name": nm, "func": is_fn,
+                  "size": (int(fn.end_ea - fn.start_ea) if is_fn else 0),
+                  "proto": proto,
+                  "seg": (ida_segment.get_segm_name(seg) if seg else "")})
+
+for i in range(ida_segment.get_segm_qty()):
+    seg = ida_segment.getnseg(i)
+    if seg is None or len(comments) >= limit or scanned >= max_scan:
+        continue
+    for ea in idautils.Heads(seg.start_ea, seg.end_ea):
+        scanned += 1
+        if len(comments) >= limit or scanned >= max_scan:
+            break
+        if not ida_bytes.has_cmt(ida_bytes.get_flags(ea)):
+            continue
+        for rep in (False, True):
+            text = ida_bytes.get_cmt(ea, rep)
+            if text:
+                fn = ida_funcs.get_func(ea)
+                comments.append({
+                    "addr": hex(int(ea)), "text": text, "repeatable": rep,
+                    "line": _line(ea), "seg": ida_segment.get_segm_name(seg),
+                    "func": (ida_funcs.get_func_name(fn.start_ea) if fn else None),
+                    "func_addr": (hex(int(fn.start_ea)) if fn else None)})
+
+# Whole-function comments are not on the byte flags, so the scan cannot see them.
+for fn_ea in idautils.Functions():
+    fn = ida_funcs.get_func(fn_ea)
+    if fn is None or len(comments) >= limit:
+        continue
+    for rep in (False, True):
+        text = ida_funcs.get_func_cmt(fn, rep)
+        if text:
+            seg = ida_segment.getseg(fn_ea)
+            comments.append({"addr": hex(int(fn_ea)), "text": text,
+                             "repeatable": rep, "line": "", "whole_func": True,
+                             "seg": (ida_segment.get_segm_name(seg) if seg else ""),
+                             "func": ida_funcs.get_func_name(fn_ea),
+                             "func_addr": hex(int(fn_ea))})
+result = {"comments": comments, "names": names, "scanned": scanned,
+          "truncated": (len(comments) >= limit or len(names) >= limit
+                        or scanned >= max_scan)}
+result
+''',
+    # The findings journal (idatui/journal.py). A netnode blob rides along in
+    # the .i64, so "what did I work out here" survives closing the database.
+    "journal_get": r'''
+import ida_netnode
+n = ida_netnode.netnode(a.get("node", "$ idatui.journal"))
+blob = n.getblob(0, "I") if ida_netnode.exist(n) else None
+result = {"data": blob.decode("utf-8", "replace") if blob else ""}
+result
+''',
+    "journal_put": r'''
+import ida_netnode
+n = ida_netnode.netnode(a.get("node", "$ idatui.journal"), 0, True)
+payload = (a.get("data") or "").encode("utf-8")
+n.setblob(payload, 0, "I")
+result = {"ok": True, "bytes": len(payload)}
+result
+''',
     "list_linkage": r'''
 imports = [{"addr": hex(int(item.address)), "name": item.name, "module": item.module_name}
            for item in db.imports.get_all_imports() if item.name]