1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
|
# Bridge GCP cmd-buffer TOCTOU hammer (ASM2464PD, opcode 0xEB)
Custom bridge firmware that rapidly flips a single 32-bit PCIe target between two
values, to race the PSP GCP TA's triple-read of `cmdBuf[0x818]` (the double-fetch
in `fw/FINDING-GCP-FE09-CONFIGINIT-HOST-OFFSET-WRITE.md`, Add.70). The bridge is
the concurrent writer the host USB path can't be (submit-then-poll is too slow).
## What the bridge actually runs (IMPORTANT)
The box's bridge runs the **handmade** firmware from `tinygrad/asm2464pd-firmware`
built at HEAD `ed4e39b7` (clean) → USB id **`3801:0001 "custom ed4e39b7-CLEAN"`**,
~9 KB. tinygrad's `usb.py` asserts the product string starts with `"custom"`, and
`ops_amd.py` accepts VID `0x3801`/`0xADD1`.
The repo's prebuilt `fw.bin` / `fw_tinygrad.bin` (98 KB, product "USB 3.2 …") are a
DIFFERENT, protocol-incompatible variant — flashing them bricks the GPU link
(first `E4` control-read fails with a USB pipe error). **Do not flash those.**
- `fw_handmade_CLEAN.bin` — exact restore baseline (`ed4e39b7-CLEAN`, sha `21d1fce4…`)
- `fw_handmade_HAMMER.bin` — the hammer build currently flashed (`ed4e39b7-DIRTY`, sha `f2cbf85b…`)
- `handmade_gcp_hammer_0xEB.patch` — the source change (apply to `handmade/src/main.c`)
- `eb_hammer_test.py` — on-box functional test (verified PASS)
- `_obsolete/` — the old binary-patch approach (built against the wrong 98 KB image)
## The handler (integrated in `handmade/src/main.c`, not binary-patched)
New vendor control command `0xEB`, reusing the firmware's own posted-MWr TLP
primitive (`REG_PCIE_*` @ 0xB210–0xB296; `STATUS=0x07; TRIGGER=0x0F`, fire-and-forget).
It **rings the PSP ring doorbell itself** (one MemWr), then immediately flips the
target between A and B — the single 8051 can't both service a host submit and hammer,
so the final doorbell must come from the bridge.
Config lives in a pinned firmware global `hammer_cfg` (XDATA, 32 B), poked by the
host via the `0xE5` XDATA-write path (no timing constraint). Read its address from
the build map (`_hammer_cfg`, currently **0x0009** — in the firmware's global RAM;
note `0x6000` is NOT backed by RAM). The `0xEB` trigger carries only the 4-byte
arming magic.
```
hammer_cfg (all little-endian):
[0:8] doorbell PCIe addr (0 => skip) [8:12] doorbell value
[12:20] hammer target PCIe addr [20:24] value A
[24:28] value B [28:30] iterations (<=0x8000)
[30] fmt_type (0 => 0x60 MWr64) [31] byte_en (0 => 0x0F)
trigger: control_write(0xEB, data=b'HMR\x01')
```
Byte maps (match the proven host path): value DATA_0 = LE LSB (natural store);
addr ADDR_0<-byte3 .. ADDR_3<-byte0, HIGH_0<-byte7 .. HIGH_3<-byte4. The doorbell
reg `{reg_pref}_67` (regMP0_SMN_C2PMSG) resolves to PCIe addr `0x1005820c`.
Safety gates: distinct opcode; arming magic; doorbell skipped if addr==0; iter
clamp; always ACKs with a ZLP. Host side: `host_hammer.py` (arm/fire/capture_doorbell).
## Build / flash / restore
```sh
# build (from a clean clone of tinygrad/asm2464pd-firmware @ ed4e39b7)
cd handmade && patch -p1 < .../handmade_gcp_hammer_0xEB.patch # or edit main.c
make wrapped # -> build/firmware_wrapped.bin ("custom ed4e39b7-DIRTY")
# flash (on the box; FTDI + bridge both on 192.168.1.200)
python ftdi_debug.py --bootloader --no-read # -> ROM bootloader 174c:2463
# wait until `lsusb | grep 174c:246` shows the bootloader, THEN:
PYTHONPATH=/root/psp/tinygrad python flash.py firmware_wrapped.bin
python ftdi_debug.py --reset --no-read
# restore the clean firmware the same way with fw_handmade_CLEAN.bin
```
## Recovery (PROVEN on this board)
The FTDI (`ftdi://ftdi:230x/1`, already EEPROM-provisioned CBUS=GPIO) can force the
ASM2464's mask-ROM bootloader via `--bootloader`, independent of flash contents, so
a bad image is always reflashable. Demonstrated repeatedly: running fw → bootloader
(`174c:2463`) → reflash → boot. The FT230X has no MPSSE, so the ONLY out-of-band
path is this strap; if it ever fails, recovery needs a SPI programmer on the chip.
## Status
- [x] handler integrated in source, asm-verified, flashed (`ed4e39b7-DIRTY`)
- [x] box fully functional on it: LINK UP, GPU init, vram `0x5ff000000`
- [x] **doorbell + hammer** handler built, asm-verified (FMT once, doorbell gated,
target-addr set, A/B loop), flashed; config RAM at `0x0009` round-trips
- [x] exercised on HW (`mech2.py`): doorbell writes the configured value to its
address (Test A: `0xD00DB011`); hammer runs the A/B loop (Test B: `0xBBBBBBBB`,
last-write wins); arming-magic gate no-ops; identical behaviour to the proven
`pcie_mem_write`
- [~] **VRAM read-back cannot verify distinct-address writes**: `pcie_mem_read` over
the USB tunnel is cache-confounded (stale write-latch without flush, 0 with
`flush_hdp`) — a read-tool limit, not a handler bug (positive control with
`pcie_mem_write` fails identically). Distinct-address correctness must be
settled by the PSP-side oracle in the chain.
- [x] **race RUN** (`race_experiment.py`, on-box `hw/race.py`): fault-free observability
test against the live state-3 chain. Positive controls green (signal `0`/`256`;
overlap plumbing completes). **24/24 the TA read the STAGED `0x818`, never the
hammered value** (`stage 0x1000 flip->2 => {0:12}`; `stage 2 flip->0x1000 =>
{256:12}`). => the cmd buffer is cached/snapshotted at dispatch; the double-fetch
TOCTOU is **NOT exploitable by a concurrent PCIe writer** in this ordering. See
FINDING Add.72.
- [~] bridge-issued doorbell is DEAD: a bridge single-TLP MWr to the wptr reg
`0x1005820c` does not trigger PSP processing (host streaming write does). This
blocks the one untested ordering (hammer live *before* dispatch snapshot).
- [ ] win condition (NOT achieved): a chosen 32-bit value at a far-OOB target with
normal status. The write ceiling stands; the double-fetch does not lift it here.
|
