Skip to content

feat: verified-facts audit, code compliance gate, and benchmark - #15

Merged
AnxForever merged 2 commits into
mainfrom
feat/verified-facts-benchmark
Aug 16, 2026
Merged

AnxForever merged 2 commits into
mainfrom
feat/verified-facts-benchmark

Conversation

@AnxForever

Copy link
Copy Markdown
Owner

Adds the remaining two quality layers from the gap analysis:

  1. verify-spec.py — audits every style's API spec for internal contradictions (forbidden classes in component templates, required entries with no template backing, token colors outside the palette). Full-catalog run exposed real data-source defects (e.g. neo-brutalist required table vs template mismatch: 124/146 styles have at least one issue).

  2. eval-check.py — mechanical compliance gate for generated code: forbidden classes, off-palette colors, and (with --component) missing required classes. Template-backed only, so spec-data inconsistencies never produce false failures on generated code.

  3. benchmark.py — with/without-skill pass-rate suite. Current: 0/4 without skill, 4/4 with skill (identical tasks, same eval-check scorer).

SKILL.md gains eval-check in the quality gate, spec-data-health guidance, and new scripts in Resources. CI gates all three and pins the benchmark.

Adds the remaining two quality layers from the gap analysis:

1. scripts/verify-spec.py — audits every style's API spec for internal
   contradictions (forbidden classes in component templates, required
   entries with no template backing, token colors outside the palette).
   Running it over the full catalog exposed real data-source defects
   (e.g. neo-brutalist required table vs template mismatch).

2. scripts/eval-check.py — mechanical compliance gate for generated code:
   forbidden classes, off-palette colors, and (with --component) missing
   required classes. Template-backed only, so spec-data inconsistencies
   never produce false failures on generated code.

3. scripts/benchmark.py — with/without-skill pass-rate suite. Current
   result: 0/4 without skill, 4/4 with skill (identical tasks, same
   eval-check scorer).

SKILL.md gains: eval-check in the quality gate, spec-data-health guidance
(trust template over required table; report inconsistencies), and the new
scripts in Resources. CI gates all three new scripts and pins the benchmark.
@coderabbitai

coderabbitai Bot commented Aug 16, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

@AnxForever, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 36 minutes

Limit details: You’ve used all 1 included review currently available under your plan.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: aff6a98b-3e1d-44a4-9da6-ed7dc1429d1e

📥 Commits

Reviewing files that changed from the base of the PR and between 134cfc7 and 472dcb5.

⛔ Files ignored due to path filters (1)
  • scripts/__pycache__/verify-spec.cpython-312.pyc is excluded by !**/*.pyc
📒 Files selected for processing (5)
  • .github/workflows/regression-gate.yml
  • SKILL.md
  • scripts/benchmark.py
  • scripts/eval-check.py
  • scripts/verify-spec.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

cmd && { exit 1; } failed whenever eval-check correctly rejected violating
code (its nonzero exit propagated). Use if cmd; then fail; fi so the step
only fails when eval-check wrongly ACCEPTS violating code.
@AnxForever
AnxForever merged commit 394132b into main Aug 16, 2026
2 checks passed
@AnxForever
AnxForever deleted the feat/verified-facts-benchmark branch August 16, 2026 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant