Repository navigation
CBEF-123: Group spec-test failures by root cause in a whole-run report #190
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
Draft
Changes from 1 commit
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
98 changes: 98 additions & 0 deletions
98
tests/Couchbase.EntityFrameworkCore.Specification.Tests/scripts/root-causes.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| #!/usr/bin/env python3 | ||
| """Group a spec-suite run's failures by root cause. | ||
|
|
||
| Reads the TRX files in one or more result directories (for example one per CI shard) and matches | ||
| each failed test's error message against scripts/root-causes.tsv. Prints the pass rate, the count | ||
| for each known cause, the pass rate of the tests no known cause stopped, and the most common | ||
| messages that match no cause (candidates for new rows). | ||
|
|
||
| Usage: root-causes.py --results DIR [DIR...] [--causes FILE] [--top N] | ||
| """ | ||
| import argparse | ||
| import collections | ||
| import glob | ||
| import os | ||
| import re | ||
| import xml.etree.ElementTree as ET | ||
|
|
||
| NS = {"t": "http://microsoft.com/schemas/VisualStudio/TeamTest/2010"} | ||
| DEFAULT_CAUSES = os.path.join(os.path.dirname(os.path.abspath(__file__)), "root-causes.tsv") | ||
|
|
||
|
|
||
| def read_causes(path): | ||
| causes = [] | ||
| with open(path) as f: | ||
| for line in f: | ||
| line = line.rstrip("\n") | ||
| if not line or line.startswith("#"): | ||
| continue | ||
| cause_id, pattern, description = line.split("\t", 2) | ||
| causes.append((cause_id, re.compile(pattern), description)) | ||
| return causes | ||
|
|
||
|
|
||
| def read_results(dirs): | ||
| for results_dir in dirs: | ||
| for path in glob.glob(os.path.join(results_dir, "**", "*.trx"), recursive=True): | ||
| try: | ||
| root = ET.parse(path).getroot() | ||
| except ET.ParseError: | ||
| continue | ||
| for result in root.iterfind(".//t:UnitTestResult", NS): | ||
| message = result.find("t:Output/t:ErrorInfo/t:Message", NS) | ||
| text = message.text or "" if message is not None else "" | ||
| yield result.get("testName"), result.get("outcome"), text | ||
|
|
||
|
|
||
| def normalize(message): | ||
| """First line of a message, with names and numbers masked so similar messages group.""" | ||
| line = message.strip().split("\n")[0] | ||
| line = re.sub(r"'[^']*'", "'…'", line) | ||
| line = re.sub(r'"[^"]*"', '"…"', line) | ||
| line = re.sub(r"\d+", "N", line) | ||
| return line[:160] | ||
|
|
||
|
|
||
| def main(): | ||
| parser = argparse.ArgumentParser() | ||
| parser.add_argument("--results", required=True, nargs="+") | ||
| parser.add_argument("--causes", default=DEFAULT_CAUSES) | ||
| parser.add_argument("--top", type=int, default=25) | ||
| args = parser.parse_args() | ||
|
|
||
| causes = read_causes(args.causes) | ||
| outcomes = collections.Counter() | ||
| by_cause = collections.Counter() | ||
| unmatched = collections.Counter() | ||
| for _, outcome, message in read_results(args.results): | ||
| outcomes[outcome] += 1 | ||
| if outcome != "Failed": | ||
| continue | ||
| cause = next((c for c, pattern, _ in causes if pattern.search(message)), None) | ||
| if cause: | ||
| by_cause[cause] += 1 | ||
| else: | ||
| unmatched[normalize(message)] += 1 | ||
|
|
||
| passed, failed = outcomes["Passed"], outcomes["Failed"] | ||
| total = sum(outcomes.values()) | ||
| blocked = sum(by_cause.values()) | ||
| other = failed - blocked | ||
| print(f"Tests: {total} passed {passed} ({passed / max(1, total):.1%}) failed {failed} " | ||
| f"not run {total - passed - failed}") | ||
| print(f"Failures with a known cause: {blocked}; other failures: {other}") | ||
| print(f"Pass rate without the known causes: {passed / max(1, passed + other):.1%}") | ||
|
|
||
| print("\nKnown causes:") | ||
| for cause_id, _, description in causes: | ||
| if by_cause[cause_id]: | ||
| print(f" {by_cause[cause_id]:6d} {cause_id:22s} {description}") | ||
|
|
||
| print(f"\nMost common other failures (top {args.top} of {len(unmatched)} distinct messages):") | ||
| for message, count in unmatched.most_common(args.top): | ||
| print(f" {count:6d} {message}") | ||
| return 0 | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| raise SystemExit(main()) | ||
15 changes: 15 additions & 0 deletions
15
tests/Couchbase.EntityFrameworkCore.Specification.Tests/scripts/root-causes.tsv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| # Known root causes of spec-test failures, matched against each failed test's error message. | ||
| # Columns (tab-separated): id, Python regex, description. The first matching row wins, so put | ||
| # narrow patterns before broad ones. When a fix removes a cause, delete its row. | ||
| sync-io does not support synchronous I/O Sync API throws; option B (opt-in via ConfigureWarnings) not landed | ||
| int-key-not-generated Insert failed for key 0 in keyspace .*\n.*DocumentExistsException A generated integer key gets no value, so every new entity is written as document 0 | ||
| seed-doc-exists Insert failed for key .*\n.*DocumentExistsException Seeding inserts a key that already exists (owned types with explicit keys, among others) | ||
| seed-empty-key Insert failed for key in keyspace Document key is empty: the save path reads the key by CLR reflection, not from the update entry | ||
| key-getter Property Get method was not found Document key read by CLR reflection fails for a property with no getter | ||
| tpc-key-generation TpcStoreGeneratedIdentityWarning A generated TPC key is rejected by EF's model validation (the harness forces TPC keys to be generated) | ||
| split-query SplitQueryResultCoordinator Split queries are compiled with the single-query coordinator | ||
| proxies-harness AddEntityFrameworkProxies Test harness: the internal service provider lacks EF proxies | ||
| default-mappings annotation 'Relational:DefaultMappings' cannot be added Relational model built twice for JSON entity types | ||
| owned-no-owner owned entity without a corresponding owner Tracking query projects an owned entity without its owner | ||
| cross-join syntax error .*at: CROSS CROSS JOIN is emitted, which SQL++ does not support | ||
| outer-keyword syntax error .*at: OUTER SQL++ rejects an OUTER keyword the generator emits (OUTER APPLY or similar) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit. This parser is a copy of the one in
compare-baseline.py, and the two already differ. This one finds TRX files recursively and that one doesn't, so the two reports could count different tests. Could we share one reader, or givecompare-baseline.pya--causesflag?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch. They differed in two ways: one searched subdirectories and the other didn't, and one counted result rows while the other counted test names. Both scripts now use one reader,
scripts/trx.py, which searches only the given directories and counts each test name once. I kept it as a separate script instead of adding--causes, socompare-baseline.pystays a pass/fail gate. Output on the baseline run is unchanged. cd51442