Repository navigation
fix: tolerate Wikipedia failures, non-root image, measured speedups - #1
Merged
Merged
Conversation
- Pin every third-party action to a full commit SHA (version kept in a comment), so a moved or compromised tag cannot change what runs. - Check out with persist-credentials: false; no step needs the token left in .git/config afterwards. - Default the workflow token to contents: read and grant write scopes only to the jobs that publish.
A timeout or 429 on any retriever branch raised out of the LangGraph superstep and killed the run, discarding the other branches' results. The retriever now catches httpx errors per branch, returns no docs for that sub-question (the analyzer answers "insufficient evidence") and records the error in the telemetry note. The sequential baseline uses the same helper so both benchmarks still degrade identically.
The image ran the CLI and the smoke benchmark as root. It now switches to an unprivileged user; the benchmark results directory is the only path the container writes to, so that is the only path chowned.
Since 59e7fa6 a failed search degrades its branch to "insufficient evidence", but nothing downstream could tell: the sequential baseline dropped the error, the parallel record ignored telemetry, and the CLI exited 0 with no warning. A rerun under rate limiting would have fed faster, lower-F1 runs into the published speedup and F1 numbers. - retrieve_or_empty logs a warning on failure. - SupervisorState.retrieval_errors and SequentialResult.retrieval_errors map sub-question id to the error. - hotpotqa_eval records the errors and leaves failed records (retrieval errors, or a raised exception as before in single mode) out of the summary, counted in n_failed, with a warning. Exception records used to be averaged in as 0 ms and 0 F1; they are now excluded too, so all three modes treat a failed retrieval the same way. - `mas` prints a warning line and adds retrieval_errors to --json.
The package description promised "2.4x speedup vs sequential on HotpotQA multi-hop, +F1 over single-prompt". Nothing in the repo measured either: the live run gave 1.31x on 5 questions and the smoke benchmark 1.99x, and no single-agent F1 comparison was published. RESULTS.md also said real Haiku latency "usually pushes the parallel speedup higher" than 1.99x, which the live 1.31x contradicts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One Wikipedia timeout or 429 on a retriever branch killed the whole graph run, and once caught, nothing downstream could tell a degraded run from a clean one. This branch isolates the failure per branch, records it end to end, runs the container as an unprivileged user, replaces the unmeasured "2.4x" package claim with the measured figures, and pins the CI workflow. Five commits on top of
main.Security
USER app;/app/benchmarks/results, the only path the container writes to, is the only path chowned.Correctness and reliability
httpx.HTTPError, returns no docs for its sub-question (the analyzer answers "insufficient evidence") and notes the error in telemetry; the sequential baseline uses the same helper, so both benchmarks degrade identically.retrieve_or_emptylogs a warning,SupervisorState.retrieval_errorsandSequentialResult.retrieval_errorsmap sub-question id to error,hotpotqa_evalkeeps failed records out of the summary and counts them inn_failed(raised exceptions used to average in as 0 ms and 0 F1), andmasprints a warning and addsretrieval_errorsto--json.Docs and numbers
CI
persist-credentials: false, workflow tokencontents: read.Tests
Before: 58 passed on Python 3.12 (
python -m pytest -q -p no:cacheprovider);ruff check src tests benchmarksclean. After, atd296ff5: 65 passed, 1 warning (LangChain deprecation), 89% coverage (494 statements, 55 missed) frompython -m pytest --cov=multi_agent_supervisor --cov-report=term -q.python -m benchmarks.smoke_parallel --delay 0.3 --repeats 2ran.Not done
dockerjob builds it.