diff options
Diffstat (limited to 'TODO')
| -rw-r--r-- | TODO | 28 |
1 files changed, 13 insertions, 15 deletions
@@ -72,21 +72,19 @@ done-ish: -## The scenario suite mutates a PERSISTENT database +## Test hygiene (fixed, worth remembering) -tests/test_scenarios.py runs against targets/echo.i64 and every edit it makes is -saved there. A scenario that undefines an instruction leaves that instruction -undefined for every later run — decomp_follow_self started failing "for no -reason" and stayed failing until the .i64 was deleted and re-analysed. +tests/test_scenarios.py used to run against targets/echo.i64 and IDA saved every +edit it made, so each run inherited the previous run's damage. It cost real time +twice: decomp_follow_self "started failing" with no code change (an earlier +scenario had undefined an instruction), and an edit-position check looked flaky +one run in three, which nearly got written up as an async race. -That also poisoned an investigation: an `edit_keeps_view` check looked flaky -(cursor jumping 0x20c6 -> 0x2094 about one run in three) and was almost -certainly the database drifting between runs, not a race. Any conclusion drawn -from repeated runs of a mutating scenario is suspect. +Now: the suite copies the binary into a temp dir and seeds it from a golden +database (<target>.pristine.i64, built once, never written back). Every run +starts from identical bytes and the tracked target is never touched. -Worth fixing properly: either give the suite a scratch copy of the binary per -run, or have mutating scenarios undo themselves. Until then, `rm targets/*.i64` -before trusting a failure that appeared without a code change. - -Coverage for "an edit must not move the view" lives in tests/test_blob_ui.py, -which builds its own throwaway binary and can mutate freely. +The general rule this came from: a suite whose result depends on its own history +can't be trusted to accuse the code. tests/test_blob_ui.py builds a throwaway +binary; test_project_ui.py stages copies; test_thumb_ui.py deletes the .i64 +before each phase because the T flag and segment bitness are SAVED in it. |