Repository navigation
Conversation
|
@Jacqueline122 is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
|
Hi @Jacqueline122, thanks a lot for your contribution! 🙏 We noticed that the Contributor License Agreement (CLA) check ( To get your PR reviewed, please sign the CLA via the link in the |
|
Thanks for signing the CLA, @Jacqueline122! 🎉 This PR is now ready for review again and the reviewer has been re-assigned. |
|
Thanks for signing the CLA, @Jacqueline122! 🎉 This PR is now ready for review again and the reviewer has been re-assigned. |
|
Hi @julian-risch, the CLA check is now passing and the PR is ready for review. Could you please take a look when you have a chance? Thanks! |
|
Thanks for signing the CLA, @Jacqueline122! 🎉 This PR is now ready for review again and the reviewer has been re-assigned. |
Coverage reportClick to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||
Related Issues
Fixes #13423
Proposed Changes:
FaithfulnessEvaluatorpreviously averaged the returned scores even when their count differed from the number of extracted statements. For example, two statements with only[1]as scores produced a perfect score andstatus="evaluated".Validate that each statement has one score before aggregation. With
raise_on_failure=True, mismatched counts raise a descriptiveValueError. WithFalse, the row receivesstatus="error"and a NaN score and is excluded from the mean. Valid aligned replies, including empty lists, retain their existing behavior.Update the docstring, component documentation, and release note.
How did you test it?
raise_on_failure=Falsetest with a mismatched statement/score case and added one targetedValueErrortest for the defaultraise_on_failure=Truebehavior.git diff --checkpass.Notes for the reviewer
Codex generated the implementation, regression tests, documentation, and release note. The reproduction uses synthetic replies and does not measure the frequency of malformed judgments in production.
Checklist