Repository navigation
fix: raise ValueError on empty inputs in LLM-based evaluators - #13746
buzzedaldrinx86-creator wants to merge 2 commits into
Conversation
LLMEvaluator.validate_input_parameters accepted empty input lists. FaithfulnessEvaluator and ContextRelevanceEvaluator then returned a NaN score with no error, and LLMEvaluator returned empty results. Raise a ValueError instead, as DocumentNDCGEvaluator and the evaluators changed in deepset-ai#11958 already do. run and run_async both go through the same check. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
@buzzedaldrinx86-creator is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
| predicted_answers=[["Berlin"]], | ||
| ) | ||
|
|
||
| def test_run_with_empty_inputs(self, monkeypatch): |
There was a problem hiding this comment.
All four new tests go through the same check in validate_input_parameters. Could we add one case to the existing test_invalid_input_parameters instead and drop the four new tests?
# Received inputs are empty lists
with pytest.raises(ValueError, match="non-empty input lists"):
component.validate_input_parameters(expected={"questions": list[str]}, received={"questions": []})| def test_run_with_empty_inputs(self, monkeypatch): | ||
| monkeypatch.setenv("OPENAI_API_KEY", "test-api-key") | ||
| component = FaithfulnessEvaluator() | ||
| # Without the check this returns a NaN score and empty results, with no error and no warning. | ||
| with pytest.raises(ValueError, match="non-empty input lists"): | ||
| component.run(questions=[], contexts=[], predicted_answers=[]) | ||
|
|
There was a problem hiding this comment.
remove
| def test_run_with_empty_inputs(self, monkeypatch): | |
| monkeypatch.setenv("OPENAI_API_KEY", "test-api-key") | |
| component = FaithfulnessEvaluator() | |
| # Without the check this returns a NaN score and empty results, with no error and no warning. | |
| with pytest.raises(ValueError, match="non-empty input lists"): | |
| component.run(questions=[], contexts=[], predicted_answers=[]) |
| def test_run_with_empty_inputs(self, monkeypatch): | ||
| monkeypatch.setenv("OPENAI_API_KEY", "test-api-key") | ||
| component = ContextRelevanceEvaluator() | ||
| # Without the check this returns a NaN score and empty results, with no error and no warning. | ||
| with pytest.raises(ValueError, match="non-empty input lists"): | ||
| component.run(questions=[], contexts=[]) | ||
|
|
There was a problem hiding this comment.
remove
| def test_run_with_empty_inputs(self, monkeypatch): | |
| monkeypatch.setenv("OPENAI_API_KEY", "test-api-key") | |
| component = ContextRelevanceEvaluator() | |
| # Without the check this returns a NaN score and empty results, with no error and no warning. | |
| with pytest.raises(ValueError, match="non-empty input lists"): | |
| component.run(questions=[], contexts=[]) |
| @pytest.mark.asyncio | ||
| async def test_run_async_with_empty_inputs(self, monkeypatch): | ||
| monkeypatch.setenv("OPENAI_API_KEY", "test-api-key") | ||
| component = LLMEvaluator( | ||
| instructions="test-instruction", | ||
| inputs=[("questions", list[str]), ("predicted_answers", list[str])], | ||
| outputs=["score"], | ||
| examples=[{"inputs": {"questions": "What is 2 + 2?", "predicted_answers": "4"}, "outputs": {"score": 1}}], | ||
| ) | ||
|
|
||
| with pytest.raises(ValueError, match="non-empty input lists"): | ||
| await component.run_async(questions=[], predicted_answers=[]) | ||
|
|
There was a problem hiding this comment.
remove
| @pytest.mark.asyncio | |
| async def test_run_async_with_empty_inputs(self, monkeypatch): | |
| monkeypatch.setenv("OPENAI_API_KEY", "test-api-key") | |
| component = LLMEvaluator( | |
| instructions="test-instruction", | |
| inputs=[("questions", list[str]), ("predicted_answers", list[str])], | |
| outputs=["score"], | |
| examples=[{"inputs": {"questions": "What is 2 + 2?", "predicted_answers": "4"}, "outputs": {"score": 1}}], | |
| ) | |
| with pytest.raises(ValueError, match="non-empty input lists"): | |
| await component.run_async(questions=[], predicted_answers=[]) |
| If the received inputs are missing, are not lists, have different lengths or are empty lists, or, only if | ||
| `raise_on_failure` is set to True, if the output is not a valid JSON or doesn't contain the expected keys. |
There was a problem hiding this comment.
Could we split this into two sentences?
| If the received inputs are missing, are not lists, have different lengths or are empty lists, or, only if | |
| `raise_on_failure` is set to True, if the output is not a valid JSON or doesn't contain the expected keys. | |
| If the received inputs are missing, are not lists, have different lengths, or are empty lists. | |
| If `raise_on_failure` is set to True, also if the output is not a valid JSON or doesn't contain the | |
| expected keys. |
| If the received inputs are missing, are not lists, have different lengths or are empty lists, or, only if | ||
| `raise_on_failure` is set to True, if the output is not a valid JSON or doesn't contain the expected keys. |
There was a problem hiding this comment.
Could we split this into two sentences?
| If the received inputs are missing, are not lists, have different lengths or are empty lists, or, only if | |
| `raise_on_failure` is set to True, if the output is not a valid JSON or doesn't contain the expected keys. | |
| If the received inputs are missing, are not lists, have different lengths, or are empty lists. | |
| If `raise_on_failure` is set to True, also if the output is not a valid JSON or doesn't contain the | |
| expected keys. |
Coverage reportClick to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||
|
Hi @buzzedaldrinx86-creator, thanks a lot for your contribution! 🙏 We noticed that the Contributor License Agreement (CLA) check ( To get your PR reviewed, please sign the CLA via the link in the |
|
Hi @buzzedaldrinx86-creator, just a friendly reminder: this PR is still in draft because the Contributor License Agreement (CLA) hasn't been signed yet. We'd love to review your contribution! Please sign the CLA via the link in the |
|
Hey @buzzedaldrinx86-creator, commit 25605af is authored by "Jared Condon", which isn't linked to a GitHub account, so the CLA check can't pass. Could you either have them sign the CLA (if this is your own commit, adding that commit email to your GitHub account should be enough) or rewrite the history so that commit is authored by your account? Thanks! |
|
Hi @buzzedaldrinx86-creator, this PR is still waiting for the Contributor License Agreement (CLA) to be signed. Please note that if the CLA isn't signed within the next 24 hours, this PR will be automatically closed. Signing only takes a minute via the link in the |
Address review: replace the four per-evaluator empty-input tests with one case in the existing test_invalid_input_parameters, since all of them go through validate_input_parameters. Split the Raises docstring into two sentences. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
25605af to
6281d3c
Compare
|
Thanks for signing the CLA, @buzzedaldrinx86-creator! 🎉 This PR is now ready for review again and the reviewer has been re-assigned. |
Related Issues
AnswerExactMatchEvaluator,DocumentMAPEvaluator,DocumentMRREvaluatorandDocumentRecallEvaluatorraise aValueErroron empty input.Proposed Changes:
LLMEvaluator.validate_input_parameterschecks that every expected input is present, that each one is a list, and that the lists have the same length. Empty lists pass all three checks.FaithfulnessEvaluatorandContextRelevanceEvaluatorthen return a NaN score with no error and no warning, andLLMEvaluatorreturns empty results:This PR adds one check after the length check. Empty input lists now raise
ValueError("LLM evaluator expects non-empty input lists ...").runandrun_asyncboth callvalidate_input_parameters, so both paths are covered. This matchesDocumentNDCGEvaluatorand the four evaluators changed in #11958.SASEvaluatoris not changed: it returns0.0for empty input, andtest_run_with_empty_inputspins that behavior.The
:raises ValueError:docstrings ofrunandrun_asyncnow list empty input lists. They also said that invalid inputs raise only whenraise_on_failure=True, butvalidate_input_parametersraises regardless, so the docstrings now say that.How did you test it?
Four unit tests:
test_run_with_empty_inputsinTestLLMEvaluator,TestFaithfulnessEvaluatorandTestContextRelevanceEvaluator, andTestLLMEvaluatorAsync.test_run_async_with_empty_inputs. All four fail without the change and pass with it.With the change, the evaluator unit tests pass (196 passed;
test_sas_evaluator.pynot run, since it needssentence-transformers), andruff checkandruff format --checkare clean on the changed files.Notes for the reviewer
This changes behavior for callers who pass empty lists: they now get a
ValueErrorinstead of a NaN score. The release note is underfixes, as in #11958.This PR was fully generated with an AI assistant. I have reviewed the changes and run the relevant tests.
Checklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:and added!in case the PR includes breaking changes.codespellwas not run.