Skip to content

fix: validate faithfulness statement-score alignment - #13424

Merged
sjrl merged 2 commits into
deepset-ai:mainfrom
Jacqueline122:fix/faithfulness-statement-score-alignment
Nov 16, 2026
Merged

sjrl merged 2 commits into
deepset-ai:mainfrom
Jacqueline122:fix/faithfulness-statement-score-alignment

Conversation

@Jacqueline122

@Jacqueline122 Jacqueline122 commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Related Issues

Fixes #13423

Proposed Changes:

FaithfulnessEvaluator previously averaged the returned scores even when their count differed from the number of extracted statements. For example, two statements with only [1] as scores produced a perfect score and status="evaluated".

Validate that each statement has one score before aggregation. With raise_on_failure=True, mismatched counts raise a descriptive ValueError. With False, the row receives status="error" and a NaN score and is excluded from the mean. Valid aligned replies, including empty lists, retain their existing behavior.

Update the docstring, component documentation, and release note.

How did you test it?

  • Extended the existing raise_on_failure=False test with a mismatched statement/score case and added one targeted ValueError test for the default raise_on_failure=True behavior.
  • Existing tests cover valid aligned/empty results and async execution through the shared postprocessing path.
  • I personally ran the three related evaluator test modules: 77 passed, 4 integration tests deselected.
  • Ruff lint/format, targeted mypy checks, pre-commit hooks, and git diff --check pass.
  • The full repository suite and live API integration tests were not run locally.

Notes for the reviewer

Codex generated the implementation, regression tests, documentation, and release note. The reproduction uses synthetic replies and does not measure the frequency of malformed judgments in production.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I have used a conventional commit type for the PR title.
  • I have documented my code.
  • I have added a release note file.
  • I have run pre-commit hooks and fixed any issue.

@Jacqueline122
Jacqueline122 requested a review from a team as a code owner October 24, 2026 20:40
@Jacqueline122
Jacqueline122 requested review from julian-risch and removed request for a team October 24, 2026 20:40
@vercel

vercel Bot commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

@Jacqueline122 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Oct 24, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @Jacqueline122, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from julian-risch October 24, 2026 21:51
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Oct 24, 2026
@HaystackBot
HaystackBot marked this pull request as draft October 24, 2026 21:51
@HaystackBot
HaystackBot marked this pull request as ready for review October 24, 2026 21:53
@HaystackBot HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Oct 24, 2026
@HaystackBot

Copy link
Copy Markdown
Contributor

Thanks for signing the CLA, @Jacqueline122! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

@HaystackBot
HaystackBot removed the request for review from julian-risch October 24, 2026 22:25
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Oct 24, 2026
@HaystackBot
HaystackBot marked this pull request as draft October 24, 2026 22:25
@HaystackBot
HaystackBot marked this pull request as ready for review October 25, 2026 11:32
@HaystackBot HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Oct 25, 2026
@HaystackBot

Copy link
Copy Markdown
Contributor

Thanks for signing the CLA, @Jacqueline122! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

@Jacqueline122

Copy link
Copy Markdown
Contributor Author

Hi @julian-risch, the CLA check is now passing and the PR is ready for review. Could you please take a look when you have a chance?

Thanks!

@HaystackBot
HaystackBot removed the request for review from julian-risch November 5, 2026 08:27
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Nov 5, 2026
@HaystackBot
HaystackBot marked this pull request as draft November 5, 2026 08:27
@HaystackBot
HaystackBot marked this pull request as ready for review November 5, 2026 08:52
@HaystackBot HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Nov 5, 2026
@HaystackBot

Copy link
Copy Markdown
Contributor

Thanks for signing the CLA, @Jacqueline122! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

@github-actions github-actions Bot added the type:documentation Improvements on the docs label Nov 12, 2026
@sjrl
sjrl requested review from sjrl and removed request for julian-risch November 12, 2026 08:15
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/components/evaluators
  faithfulness.py
Project Total  

This report was generated by python-coverage-comment-action

@Jacqueline122

Copy link
Copy Markdown
Contributor Author

Thanks @sjrl! Commit 6136e07 simplifies the tests as requested and shortens the release note to your suggested version.

I reran the related evaluator tests locally: 77 passed, 4 integration tests deselected. Could you please re-review when you have a chance?

@sjrl sjrl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@sjrl
sjrl merged commit 0380875 into deepset-ai:main Nov 16, 2026
27 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic:tests type:documentation Improvements on the docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Title: FaithfulnessEvaluator accepts mismatched statement and score counts as evaluated results

4 participants