Skip to content
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .github/workflows/spec-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -156,3 +156,35 @@ jobs:
- name: Remove cluster
if: always() && env.CBDINO_CLUSTER_ID != ''
run: cbdinocluster rm "$CBDINO_CLUSTER_ID" || true

# One view of the whole run: both shards compared with the baseline together, and the
# failures grouped by root cause (scripts/root-causes.tsv).
report:
name: report
needs: spec-tests
if: always()
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Download results
uses: actions/download-artifact@v4
with:
pattern: spec-test-results-*
path: ${{ runner.temp }}/shards

- name: Summary
run: |
shopt -s nullglob
dirs=("$RUNNER_TEMP"/shards/*/spec-results)
[ ${#dirs[@]} -gt 0 ] || { echo "No shard results."; exit 0; }
{
echo '## EF Core specification tests, all shards'
echo
echo '```'
python3 "$SPEC_PROJECT/scripts/root-causes.py" --results "${dirs[@]}" || true
echo
python3 "$SPEC_PROJECT/scripts/compare-baseline.py" --results "${dirs[@]}" | head -n 300 || true
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
5 changes: 5 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,11 @@ pass. Each shard's TRX files are uploaded as a `spec-test-results-<shard>` artif
the baseline from a CI run, download both and run
`scripts/compare-baseline.py --results <shard 1>/spec-results <shard 2>/spec-results --update`.

A `report` job then summarizes both shards together and groups the failures by root cause, using
the patterns in `scripts/root-causes.tsv`; run `scripts/root-causes.py --results <dirs>` to do the
same locally. When a change fixes a cause, delete its row; when the "other failures" list shows a
new common cause, add one.

## Cutting a release

`.github/workflows/release.yml` runs on manual dispatch (`workflow_dispatch`) against an existing
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
#!/usr/bin/env python3
"""Group a spec-suite run's failures by root cause.

Reads the TRX files in one or more result directories (for example one per CI shard) and matches
each failed test's error message against scripts/root-causes.tsv. Prints the pass rate, the count
for each known cause, the pass rate of the tests no known cause stopped, and the most common
messages that match no cause (candidates for new rows).

Usage: root-causes.py --results DIR [DIR...] [--causes FILE] [--top N]
"""
import argparse
import collections
import glob
import os
import re
import xml.etree.ElementTree as ET

NS = {"t": "http://microsoft.com/schemas/VisualStudio/TeamTest/2010"}
DEFAULT_CAUSES = os.path.join(os.path.dirname(os.path.abspath(__file__)), "root-causes.tsv")


def read_causes(path):
causes = []
with open(path) as f:
for line in f:
line = line.rstrip("\n")
if not line or line.startswith("#"):
continue
cause_id, pattern, description = line.split("\t", 2)
causes.append((cause_id, re.compile(pattern), description))
return causes


def read_results(dirs):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit. This parser is a copy of the one in compare-baseline.py, and the two already differ. This one finds TRX files recursively and that one doesn't, so the two reports could count different tests. Could we share one reader, or give compare-baseline.py a --causes flag?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. They differed in two ways: one searched subdirectories and the other didn't, and one counted result rows while the other counted test names. Both scripts now use one reader, scripts/trx.py, which searches only the given directories and counts each test name once. I kept it as a separate script instead of adding --causes, so compare-baseline.py stays a pass/fail gate. Output on the baseline run is unchanged. cd51442

for results_dir in dirs:
for path in glob.glob(os.path.join(results_dir, "**", "*.trx"), recursive=True):
try:
root = ET.parse(path).getroot()
except ET.ParseError:
continue
for result in root.iterfind(".//t:UnitTestResult", NS):
message = result.find("t:Output/t:ErrorInfo/t:Message", NS)
text = message.text or "" if message is not None else ""
yield result.get("testName"), result.get("outcome"), text


def normalize(message):
"""First line of a message, with names and numbers masked so similar messages group."""
line = message.strip().split("\n")[0]
line = re.sub(r"'[^']*'", "'…'", line)
line = re.sub(r'"[^"]*"', '"…"', line)
line = re.sub(r"\d+", "N", line)
return line[:160]


def main():
parser = argparse.ArgumentParser()
parser.add_argument("--results", required=True, nargs="+")
parser.add_argument("--causes", default=DEFAULT_CAUSES)
parser.add_argument("--top", type=int, default=25)
args = parser.parse_args()

causes = read_causes(args.causes)
outcomes = collections.Counter()
by_cause = collections.Counter()
unmatched = collections.Counter()
for _, outcome, message in read_results(args.results):
outcomes[outcome] += 1
if outcome != "Failed":
continue
cause = next((c for c, pattern, _ in causes if pattern.search(message)), None)
if cause:
by_cause[cause] += 1
else:
unmatched[normalize(message)] += 1

passed, failed = outcomes["Passed"], outcomes["Failed"]
total = sum(outcomes.values())
blocked = sum(by_cause.values())
other = failed - blocked
print(f"Tests: {total} passed {passed} ({passed / max(1, total):.1%}) failed {failed} "
f"not run {total - passed - failed}")
print(f"Failures with a known cause: {blocked}; other failures: {other}")
print(f"Pass rate without the known causes: {passed / max(1, passed + other):.1%}")

print("\nKnown causes:")
for cause_id, _, description in causes:
if by_cause[cause_id]:
print(f" {by_cause[cause_id]:6d} {cause_id:22s} {description}")

print(f"\nMost common other failures (top {args.top} of {len(unmatched)} distinct messages):")
for message, count in unmatched.most_common(args.top):
print(f" {count:6d} {message}")
return 0


if __name__ == "__main__":
raise SystemExit(main())
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Known root causes of spec-test failures, matched against each failed test's error message.
# Columns (tab-separated): id, Python regex, description. The first matching row wins, so put
# narrow patterns before broad ones. When a fix removes a cause, delete its row.
sync-io does not support synchronous I/O Sync API throws; option B (opt-in via ConfigureWarnings) not landed
int-key-not-generated Insert failed for key 0 in keyspace .*\n.*DocumentExistsException A generated integer key gets no value, so every new entity is written as document 0
seed-doc-exists Insert failed for key .*\n.*DocumentExistsException Seeding inserts a key that already exists (owned types with explicit keys, among others)
seed-empty-key Insert failed for key in keyspace Document key is empty: the save path reads the key by CLR reflection, not from the update entry
key-getter Property Get method was not found Document key read by CLR reflection fails for a property with no getter
tpc-key-generation TpcStoreGeneratedIdentityWarning A generated TPC key is rejected by EF's model validation (the harness forces TPC keys to be generated)
split-query SplitQueryResultCoordinator Split queries are compiled with the single-query coordinator
proxies-harness AddEntityFrameworkProxies Test harness: the internal service provider lacks EF proxies
default-mappings annotation 'Relational:DefaultMappings' cannot be added Relational model built twice for JSON entity types
owned-no-owner owned entity without a corresponding owner Tracking query projects an owned entity without its owner
cross-join syntax error .*at: CROSS CROSS JOIN is emitted, which SQL++ does not support
outer-keyword syntax error .*at: OUTER SQL++ rejects an OUTER keyword the generator emits (OUTER APPLY or similar)
Loading