Skip to content
kallurayaankitPublic

About

AI-native Playwright pipeline: requirements → scenarios → validated locators → classified retries

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

rely

Tests License: MIT Python 3.10+

Rely on your tests. An AI-native Playwright pipeline with pre-flight locator validation and failure-aware retries.

Requirements -> scenarios -> locators -> validation -> execution.

pip install -e .
rely plan examples/requirements/login.md --url https://app.example.com
rely run out/plan.json --url https://app.example.com

Two ideas, nothing else

1. Pre-flight locator validation

Playwright's built-in retries are dumb: retries: 3 on everything. rely validates every locator before the browser navigates:

Check Rule
Match count exactly 1 element
Visibility visible
Enabled enabled
Stability score >= RELY_MIN_LOCATOR_SCORE (default 0.4)

Anything that fails becomes UNRUNNABLE, and the scenario is skipped instead of executed with a broken selector. No wasted run time. No misleading failures.

2. Classification-driven retries

Every failure is classified, and the class determines the retry policy:

Class Retries Action
REAL_BUG 0 Report immediately
FLAKY 3 Exponential backoff
LOCATOR_DRIFT 1 Re-resolve the locator
ENV_ISSUE 1 Health check, then retry
UNCLASSIFIED 1 Report as unknown

The classifier is heuristic-first (fast, free, deterministic) and falls back to an LLM only for residue.

Architecture

flowchart LR
    REQ[Requirements<br/>Markdown] --> PLAN[Scenario Planner<br/>LLM, once]
    PLAN --> SCN[Scenarios]
    SCN --> DISC[Locator Discovery<br/>10-tier hierarchy]
    DISC --> VAL{Pre-flight<br/>Validation}
    VAL -->|valid| RUN[Playwright<br/>Execution]
    VAL -->|invalid| SKIP[Unrunnable]
    RUN --> CLS{Failure<br/>Classifier}
    CLS -->|REAL_BUG| FAIL[Fail, 0 retries]
    CLS -->|FLAKY| F3[3 retries, backoff]
    CLS -->|LOCATOR_DRIFT| RE[Re-resolve, 1 retry]
    CLS -->|ENV_ISSUE| HC[Health check, 1 retry]
Loading

The 10-tier locator hierarchy

Tier Strategy Score
1 get_by_role (W3C) 0.95
2 data-testid 0.90
3 aria-label 0.85
4 name attribute 0.75
5 id attribute 0.70
6 CSS class fragment 0.55
7 Visible text 0.50
8 Scoped CSS 0.35
9 Scoped XPath 0.25
10 Full XPath 0.10

Tiers 1–3 score above 0.85 and pass validation. Tiers 9–10 fall below the 0.4 threshold and are rejected by the gate.

CLI

rely plan <requirements.md> --url <url>       # plan scenarios from requirements
rely run <plan.json> --url <url>              # execute with pre-flight + classified retries
rely run <plan.json> --url <url> --headed     # show the browser
rely report <traces.json>                     # summarize a traces file

Configuration

.env:

OLLAMA_URL=http://localhost:11434
OLLAMA_MODEL=llama3.2:3b
RELY_MIN_LOCATOR_SCORE=0.4
RELY_HEADLESS=true

Status

Early. Pipeline works end-to-end. Locator discovery supports role, testid, aria, name, id, text, and scoped XPath. Classifier covers the four deterministic failure signatures plus an LLM fallback.

Roadmap: visual diff on assertion failures, HTML trace report, parallel execution, self-healing locator re-resolution on drift.

License

MIT

About

AI-native Playwright pipeline: requirements → scenarios → validated locators → classified retries

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages