TestNG retry tests that are marked as skipped with gradle-retry-plugin

[itkhanz]

Hi,
I am using TestNG with gradle, and for retrying flaky tests, I am using gradle-retry-plugin instead of the IRetryAnalayzer from TestNG.
Could you please have a look at this issue, and share your input on how to best deal with this situation:-

gradle/test-retry-gradle-plugin#290

I would like to be able to retry tests that are marked as skipped in TestNG but this is not possible at the moment with gradle-retry-plugin, so therefore I am making the configuration methods annotated with @BeforeMethod and @BeforeClass in BaseTest as failed which causes Gradle/TestNG to report the result of configuration method separately causing duplicatate total results count where the configuration method is shown as failed, and the actual test is shown as skipped.

I would really appreciate your input on dealing with this situation.

[Sun-Wukhan]

The root cause is how the result is reported. The Gradle retry plugin retries tests with a failed result. When a @BeforeMethod or @BeforeClass fails, TestNG reports the test itself as skipped, not failed, so the plugin sees nothing to retry. Failing the configuration method on purpose does not change that: Gradle then counts the configuration failure as one result and the skipped test as another, which is the duplication you are seeing.

For this case, keep the retries in TestNG and drop the Gradle plugin for these suites:

• Implement IRetryAnalyzer and apply it to every test with an IAnnotationTransformer, so you do not have to annotate each method.
• TestNG re-runs a test whose configuration method failed when the analyzer returns true. The earlier attempts are reported as skipped retries and the last attempt carries the real pass or fail, so testng-results.xml and the Gradle totals stay consistent.

One distinction worth making: if the configuration method throws SkipException because a precondition genuinely does not apply (wrong browser, environment not deployed), retrying will only skip again. Reserve SkipException for that. For transient setup failures you actually want retried, throw a normal exception from the setup code and let the retry analyzer handle it.