Rely on your tests. An AI-native Playwright pipeline with pre-flight locator validation and failure-aware retries.
Requirements -> scenarios -> locators -> validation -> execution.
pip install -e .
rely plan examples/requirements/login.md --url https://app.example.com
rely run out/plan.json --url https://app.example.comPlaywright's built-in retries are dumb: retries: 3 on everything. rely validates every locator before the browser navigates:
| Check | Rule |
|---|---|
| Match count | exactly 1 element |
| Visibility | visible |
| Enabled | enabled |
| Stability score | >= RELY_MIN_LOCATOR_SCORE (default 0.4) |
Anything that fails becomes UNRUNNABLE, and the scenario is skipped instead of executed with a broken selector. No wasted run time. No misleading failures.
Every failure is classified, and the class determines the retry policy:
| Class | Retries | Action |
|---|---|---|
REAL_BUG |
0 | Report immediately |
FLAKY |
3 | Exponential backoff |
LOCATOR_DRIFT |
1 | Re-resolve the locator |
ENV_ISSUE |
1 | Health check, then retry |
UNCLASSIFIED |
1 | Report as unknown |
The classifier is heuristic-first (fast, free, deterministic) and falls back to an LLM only for residue.
flowchart LR
REQ[Requirements<br/>Markdown] --> PLAN[Scenario Planner<br/>LLM, once]
PLAN --> SCN[Scenarios]
SCN --> DISC[Locator Discovery<br/>10-tier hierarchy]
DISC --> VAL{Pre-flight<br/>Validation}
VAL -->|valid| RUN[Playwright<br/>Execution]
VAL -->|invalid| SKIP[Unrunnable]
RUN --> CLS{Failure<br/>Classifier}
CLS -->|REAL_BUG| FAIL[Fail, 0 retries]
CLS -->|FLAKY| F3[3 retries, backoff]
CLS -->|LOCATOR_DRIFT| RE[Re-resolve, 1 retry]
CLS -->|ENV_ISSUE| HC[Health check, 1 retry]
| Tier | Strategy | Score |
|---|---|---|
| 1 | get_by_role (W3C) |
0.95 |
| 2 | data-testid |
0.90 |
| 3 | aria-label |
0.85 |
| 4 | name attribute |
0.75 |
| 5 | id attribute |
0.70 |
| 6 | CSS class fragment | 0.55 |
| 7 | Visible text | 0.50 |
| 8 | Scoped CSS | 0.35 |
| 9 | Scoped XPath | 0.25 |
| 10 | Full XPath | 0.10 |
Tiers 1–3 score above 0.85 and pass validation. Tiers 9–10 fall below the 0.4 threshold and are rejected by the gate.
rely plan <requirements.md> --url <url> # plan scenarios from requirements
rely run <plan.json> --url <url> # execute with pre-flight + classified retries
rely run <plan.json> --url <url> --headed # show the browser
rely report <traces.json> # summarize a traces file.env:
OLLAMA_URL=http://localhost:11434
OLLAMA_MODEL=llama3.2:3b
RELY_MIN_LOCATOR_SCORE=0.4
RELY_HEADLESS=true
Early. Pipeline works end-to-end. Locator discovery supports role, testid, aria, name, id, text, and scoped XPath. Classifier covers the four deterministic failure signatures plus an LLM fallback.
Roadmap: visual diff on assertion failures, HTML trace report, parallel execution, self-healing locator re-resolution on drift.
MIT