Skip to content

fix: raise ValueError on empty inputs in LLM-based evaluators - #13746

Open
buzzedaldrinx86-creator wants to merge 2 commits into
deepset-ai:mainfrom
buzzedaldrinx86-creator:fix-llm-evaluators-empty-inputs
Open

buzzedaldrinx86-creator wants to merge 2 commits into
deepset-ai:mainfrom
buzzedaldrinx86-creator:fix-llm-evaluators-empty-inputs

Conversation

@buzzedaldrinx86-creator

Copy link
Copy Markdown

Related Issues

Proposed Changes:

LLMEvaluator.validate_input_parameters checks that every expected input is present, that each one is a list, and that the lists have the same length. Empty lists pass all three checks. FaithfulnessEvaluator and ContextRelevanceEvaluator then return a NaN score with no error and no warning, and LLMEvaluator returns empty results:

FaithfulnessEvaluator().run(questions=[], contexts=[], predicted_answers=[])
# {'results': [], 'meta': None, 'score': nan, 'individual_scores': []}

This PR adds one check after the length check. Empty input lists now raise ValueError("LLM evaluator expects non-empty input lists ..."). run and run_async both call validate_input_parameters, so both paths are covered. This matches DocumentNDCGEvaluator and the four evaluators changed in #11958.

SASEvaluator is not changed: it returns 0.0 for empty input, and test_run_with_empty_inputs pins that behavior.

The :raises ValueError: docstrings of run and run_async now list empty input lists. They also said that invalid inputs raise only when raise_on_failure=True, but validate_input_parameters raises regardless, so the docstrings now say that.

How did you test it?

Four unit tests: test_run_with_empty_inputs in TestLLMEvaluator, TestFaithfulnessEvaluator and TestContextRelevanceEvaluator, and TestLLMEvaluatorAsync.test_run_async_with_empty_inputs. All four fail without the change and pass with it.
With the change, the evaluator unit tests pass (196 passed; test_sas_evaluator.py not run, since it needs sentence-transformers), and ruff check and ruff format --check are clean on the changed files.

Notes for the reviewer

This changes behavior for callers who pass empty lists: they now get a ValueError instead of a NaN score. The release note is under fixes, as in #11958.

This PR was fully generated with an AI assistant. I have reviewed the changes and run the relevant tests.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run the ruff and release-note pre-commit hooks on the changed files and fixed any issue. codespell was not run.

LLMEvaluator.validate_input_parameters accepted empty input lists.
FaithfulnessEvaluator and ContextRelevanceEvaluator then returned a NaN
score with no error, and LLMEvaluator returned empty results. Raise a
ValueError instead, as DocumentNDCGEvaluator and the evaluators changed
in deepset-ai#11958 already do. run and run_async both go through the same check.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@buzzedaldrinx86-creator
buzzedaldrinx86-creator requested a review from a team as a code owner November 14, 2026 05:49
@buzzedaldrinx86-creator
buzzedaldrinx86-creator requested review from sjrl and removed request for a team November 14, 2026 05:49
@vercel

vercel Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

@buzzedaldrinx86-creator is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Nov 14, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions Bot added topic:tests P3 Low priority, leave it in the backlog labels Nov 14, 2026
@sjrl sjrl self-assigned this Nov 16, 2026
@github-actions github-actions Bot added the type:documentation Improvements on the docs label Nov 16, 2026
predicted_answers=[["Berlin"]],
)

def test_run_with_empty_inputs(self, monkeypatch):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All four new tests go through the same check in validate_input_parameters. Could we add one case to the existing test_invalid_input_parameters instead and drop the four new tests?

        # Received inputs are empty lists
        with pytest.raises(ValueError, match="non-empty input lists"):
            component.validate_input_parameters(expected={"questions": list[str]}, received={"questions": []})

Comment on lines +259 to +265
def test_run_with_empty_inputs(self, monkeypatch):
monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
component = FaithfulnessEvaluator()
# Without the check this returns a NaN score and empty results, with no error and no warning.
with pytest.raises(ValueError, match="non-empty input lists"):
component.run(questions=[], contexts=[], predicted_answers=[])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

Suggested change
def test_run_with_empty_inputs(self, monkeypatch):
monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
component = FaithfulnessEvaluator()
# Without the check this returns a NaN score and empty results, with no error and no warning.
with pytest.raises(ValueError, match="non-empty input lists"):
component.run(questions=[], contexts=[], predicted_answers=[])

Comment on lines +218 to +224
def test_run_with_empty_inputs(self, monkeypatch):
monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
component = ContextRelevanceEvaluator()
# Without the check this returns a NaN score and empty results, with no error and no warning.
with pytest.raises(ValueError, match="non-empty input lists"):
component.run(questions=[], contexts=[])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

Suggested change
def test_run_with_empty_inputs(self, monkeypatch):
monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
component = ContextRelevanceEvaluator()
# Without the check this returns a NaN score and empty results, with no error and no warning.
with pytest.raises(ValueError, match="non-empty input lists"):
component.run(questions=[], contexts=[])

Comment on lines +468 to +480
@pytest.mark.asyncio
async def test_run_async_with_empty_inputs(self, monkeypatch):
monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
component = LLMEvaluator(
instructions="test-instruction",
inputs=[("questions", list[str]), ("predicted_answers", list[str])],
outputs=["score"],
examples=[{"inputs": {"questions": "What is 2 + 2?", "predicted_answers": "4"}, "outputs": {"score": 1}}],
)

with pytest.raises(ValueError, match="non-empty input lists"):
await component.run_async(questions=[], predicted_answers=[])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

Suggested change
@pytest.mark.asyncio
async def test_run_async_with_empty_inputs(self, monkeypatch):
monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
component = LLMEvaluator(
instructions="test-instruction",
inputs=[("questions", list[str]), ("predicted_answers", list[str])],
outputs=["score"],
examples=[{"inputs": {"questions": "What is 2 + 2?", "predicted_answers": "4"}, "outputs": {"score": 1}}],
)
with pytest.raises(ValueError, match="non-empty input lists"):
await component.run_async(questions=[], predicted_answers=[])

Comment on lines +242 to +243
If the received inputs are missing, are not lists, have different lengths or are empty lists, or, only if
`raise_on_failure` is set to True, if the output is not a valid JSON or doesn't contain the expected keys.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we split this into two sentences?

Suggested change
If the received inputs are missing, are not lists, have different lengths or are empty lists, or, only if
`raise_on_failure` is set to True, if the output is not a valid JSON or doesn't contain the expected keys.
If the received inputs are missing, are not lists, have different lengths, or are empty lists.
If `raise_on_failure` is set to True, also if the output is not a valid JSON or doesn't contain the
expected keys.

Comment on lines +315 to +316
If the received inputs are missing, are not lists, have different lengths or are empty lists, or, only if
`raise_on_failure` is set to True, if the output is not a valid JSON or doesn't contain the expected keys.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we split this into two sentences?

Suggested change
If the received inputs are missing, are not lists, have different lengths or are empty lists, or, only if
`raise_on_failure` is set to True, if the output is not a valid JSON or doesn't contain the expected keys.
If the received inputs are missing, are not lists, have different lengths, or are empty lists.
If `raise_on_failure` is set to True, also if the output is not a valid JSON or doesn't contain the
expected keys.

@github-actions

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/components/evaluators
  llm_evaluator.py
  haystack/core/pipeline
  pipeline.py
Project Total  

This report was generated by python-coverage-comment-action

@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @buzzedaldrinx86-creator, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Nov 18, 2026
@HaystackBot
HaystackBot marked this pull request as draft November 18, 2026 00:42
@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @buzzedaldrinx86-creator, just a friendly reminder: this PR is still in draft because the Contributor License Agreement (CLA) hasn't been signed yet. We'd love to review your contribution! Please sign the CLA via the link in the license/cla check, and this PR will automatically be marked ready for review.

@sjrl

sjrl commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Hey @buzzedaldrinx86-creator, commit 25605af is authored by "Jared Condon", which isn't linked to a GitHub account, so the CLA check can't pass. Could you either have them sign the CLA (if this is your own commit, adding that commit email to your GitHub account should be enough) or rewrite the history so that commit is authored by your account? Thanks!

@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @buzzedaldrinx86-creator, this PR is still waiting for the Contributor License Agreement (CLA) to be signed. Please note that if the CLA isn't signed within the next 24 hours, this PR will be automatically closed. Signing only takes a minute via the link in the license/cla check, and the PR will then automatically be marked ready for review.

Address review: replace the four per-evaluator empty-input tests with one
case in the existing test_invalid_input_parameters, since all of them go
through validate_input_parameters. Split the Raises docstring into two
sentences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@buzzedaldrinx86-creator
buzzedaldrinx86-creator force-pushed the fix-llm-evaluators-empty-inputs branch from 25605af to 6281d3c Compare November 20, 2026 01:25
@HaystackBot
HaystackBot marked this pull request as ready for review November 20, 2026 01:25
@HaystackBot HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Nov 20, 2026
@HaystackBot

Copy link
Copy Markdown
Contributor

Thanks for signing the CLA, @buzzedaldrinx86-creator! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P3 Low priority, leave it in the backlog topic:tests type:documentation Improvements on the docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 of 5 Document*Evaluators crash with ZeroDivisionError on empty input

4 participants