Skip to content

CBEF-123: Group spec-test failures by root cause in a whole-run report - #190

Draft
davidkelly wants to merge 2 commits into
mainfrom
spec-root-causes
Draft

davidkelly wants to merge 2 commits into
mainfrom
spec-root-causes

Conversation

@davidkelly

Copy link
Copy Markdown
Contributor

Most spec-test failures come from a few causes that stop a test before it checks anything: in
the baseline, a handful of causes account for over 80% of the 18,191 failures. The total pass
rate says little about what a change fixed, and each shard's summary covers only half the run.
This adds a whole-run report that groups the failures by cause.

Change

  • scripts/root-causes.tsv: the known causes, one row each (id, regex on the error message,
    description; the first match wins). A fix PR deletes its cause's row; a new common cause gets
    a row.
  • scripts/root-causes.py: reads the TRX files and prints the pass rate, the failures for each
    cause, the pass rate of the tests no known cause stopped, and the most common messages that
    match no cause.
  • spec-tests.yml: a report job, after both shards, writes one summary for the whole run:
    the root causes, then the baseline comparison.
  • CONTRIBUTING.md: how to use and update it.

Test infrastructure only; no provider changes.

Tests

Run against the baseline run's artifacts (43463541551): 27.6% pass; known causes account for
15,567 of 18,191 failures; the other tests pass at 72.7%. Both shards together match the
baseline exactly. The report job's script was run locally against the same artifacts.

First in a stack: #188 and #189 build on it and each deletes the rows it fixes.

Jira: CBEF-123

Most spec-test failures come from a few causes that stop a test before
it checks anything (in the baseline, a handful of causes account for
over 80% of them), so the total pass rate says little about what a
change fixed.

- scripts/root-causes.tsv lists the known causes as message patterns.
  A fix PR deletes its cause's row; a new common cause gets one.
- scripts/root-causes.py reports the pass rate, the failures for each
  cause, the pass rate of the tests no known cause stopped, and the
  most common messages that match no cause.
- A report job in spec-tests.yml downloads both shards' results and
  writes one summary for the whole run: the root causes and the
  baseline comparison.

On the baseline run (43463541551): 27.6% pass; known causes account for
15,567 of 18,191 failures; the other tests pass at 72.7%.

@emilienbev emilienbev left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, grouping failures by cause makes it much easier to see what a fix actually moved.

return causes


def read_results(dirs):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit. This parser is a copy of the one in compare-baseline.py, and the two already differ. This one finds TRX files recursively and that one doesn't, so the two reports could count different tests. Could we share one reader, or give compare-baseline.py a --causes flag?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. They differed in two ways: one searched subdirectories and the other didn't, and one counted result rows while the other counted test names. Both scripts now use one reader, scripts/trx.py, which searches only the given directories and counts each test name once. I kept it as a separate script instead of adding --causes, so compare-baseline.py stays a pass/fail gate. Output on the baseline run is unchanged. cd51442

root-causes.py had its own copy of compare-baseline.py's TRX reader,
and the copies already differed: one searched subdirectories and one
did not, and one counted result rows while the other counted test
names. The two reports could count different tests.

- scripts/trx.py reads the TRX files; both scripts use it.
- Only the given directory is searched, which is where run-suites.sh
  writes the TRX files.
- root-causes.py counts each test name once, as the baseline does.

On the baseline run (43463541551) both reports are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants