Repository navigation
CBEF-123: Group spec-test failures by root cause in a whole-run report - #190
davidkelly wants to merge 2 commits into
Conversation
Most spec-test failures come from a few causes that stop a test before it checks anything (in the baseline, a handful of causes account for over 80% of them), so the total pass rate says little about what a change fixed. - scripts/root-causes.tsv lists the known causes as message patterns. A fix PR deletes its cause's row; a new common cause gets one. - scripts/root-causes.py reports the pass rate, the failures for each cause, the pass rate of the tests no known cause stopped, and the most common messages that match no cause. - A report job in spec-tests.yml downloads both shards' results and writes one summary for the whole run: the root causes and the baseline comparison. On the baseline run (43463541551): 27.6% pass; known causes account for 15,567 of 18,191 failures; the other tests pass at 72.7%.
emilienbev
left a comment
There was a problem hiding this comment.
Nice, grouping failures by cause makes it much easier to see what a fix actually moved.
| return causes | ||
|
|
||
|
|
||
| def read_results(dirs): |
There was a problem hiding this comment.
Nit. This parser is a copy of the one in compare-baseline.py, and the two already differ. This one finds TRX files recursively and that one doesn't, so the two reports could count different tests. Could we share one reader, or give compare-baseline.py a --causes flag?
There was a problem hiding this comment.
Good catch. They differed in two ways: one searched subdirectories and the other didn't, and one counted result rows while the other counted test names. Both scripts now use one reader, scripts/trx.py, which searches only the given directories and counts each test name once. I kept it as a separate script instead of adding --causes, so compare-baseline.py stays a pass/fail gate. Output on the baseline run is unchanged. cd51442
root-causes.py had its own copy of compare-baseline.py's TRX reader, and the copies already differed: one searched subdirectories and one did not, and one counted result rows while the other counted test names. The two reports could count different tests. - scripts/trx.py reads the TRX files; both scripts use it. - Only the given directory is searched, which is where run-suites.sh writes the TRX files. - root-causes.py counts each test name once, as the baseline does. On the baseline run (43463541551) both reports are unchanged.
Most spec-test failures come from a few causes that stop a test before it checks anything: in
the baseline, a handful of causes account for over 80% of the 18,191 failures. The total pass
rate says little about what a change fixed, and each shard's summary covers only half the run.
This adds a whole-run report that groups the failures by cause.
Change
scripts/root-causes.tsv: the known causes, one row each (id, regex on the error message,description; the first match wins). A fix PR deletes its cause's row; a new common cause gets
a row.
scripts/root-causes.py: reads the TRX files and prints the pass rate, the failures for eachcause, the pass rate of the tests no known cause stopped, and the most common messages that
match no cause.
spec-tests.yml: areportjob, after both shards, writes one summary for the whole run:the root causes, then the baseline comparison.
Test infrastructure only; no provider changes.
Tests
Run against the baseline run's artifacts (43463541551): 27.6% pass; known causes account for
15,567 of 18,191 failures; the other tests pass at 72.7%. Both shards together match the
baseline exactly. The report job's script was run locally against the same artifacts.
First in a stack: #188 and #189 build on it and each deletes the rows it fixes.
Jira: CBEF-123