Repository navigation
test: match expected failures that Behat reports with a line suffix - #41916
Open
oc-tmueller wants to merge 1 commit into
Open
oc-tmueller wants to merge 1 commit into
oc-tmueller wants to merge 1 commit into
Conversation
Behat can print a failed scenario as 'features/suite/file.feature:9 (on line 15)'. Behat 3.34 does so for every failed scenario, where 3.13 printed the bare path. run.sh checks that every listed expected failure really failed by grepping for the path at the end of the line, so with Behat 3.34 each expected failure was reported as 'expected to fail but did not fail' and the run failed. Allow the optional suffix, keeping file.feature:9 distinct from file.feature:902. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Thomas Müller <323649642+oc-tmueller@users.noreply.github.com>
phil-davis
approved these changes
Oct 23, 2026
phil-davis
left a comment
phil-davis
left a comment
Contributor
There was a problem hiding this comment.
Looks good. This has always been a bit of a pain, because we have to parse the human-readable text output of the test run, rather than being able to get any well-specified data format holding the test results.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
run.shchecks that every scenario inEXPECTED_FAILURES_FILEreally failed withso the failed path has to end the line. Behat 3.34 prints every failed scenario with the line of the failed step appended:
The grep then never matches. Every expected failure is reported as "expected to fail but did not fail", and the run fails even though exactly the listed scenarios failed. The forward check is unaffected, because it strips everything after the first space. Behat 3.13 (used for 10.16) printed these paths without the suffix, which is why it worked there.
This showed up in owncloud-docker/server#738, which runs core's acceptance tests from the 11.0.1
-qatarball against the Docker image. Its 11.0.1 CLI jobs failed this way; the 10.16.6 jobs, with Behat 3.13, passed with lists of the same kind. That PR patches the extractedrun.shthe same way as this one.The fix allows an optional
(on line N)suffix. It still keepsfile.feature:9distinct fromfile.feature:902, as the comment above it requires. Checked by hand:comments.feature:9matches both…comments.feature:9and…comments.feature:9 (on line 15), and does not match…comments.feature:902.🤖 Generated with Claude Code