Skip to content

fix: tolerate Wikipedia failures, non-root image, measured speedups - #1

Merged
Tajaddin merged 5 commits into
mainfrom
engineering-2026-11
Nov 7, 2026
Merged

Tajaddin merged 5 commits into
mainfrom
engineering-2026-11

Conversation

@Tajaddin

@Tajaddin Tajaddin commented Nov 7, 2026

Copy link
Copy Markdown
Owner

One Wikipedia timeout or 429 on a retriever branch killed the whole graph run, and once caught, nothing downstream could tell a degraded run from a clean one. This branch isolates the failure per branch, records it end to end, runs the container as an unprivileged user, replaces the unmeasured "2.4x" package claim with the measured figures, and pins the CI workflow. Five commits on top of main.

Security

  • The image creates uid 10001 and runs as USER app; /app/benchmarks/results, the only path the container writes to, is the only path chowned.

Correctness and reliability

  • Each retriever branch catches httpx.HTTPError, returns no docs for its sub-question (the analyzer answers "insufficient evidence") and notes the error in telemetry; the sequential baseline uses the same helper, so both benchmarks degrade identically.
  • Failed searches are recorded: retrieve_or_empty logs a warning, SupervisorState.retrieval_errors and SequentialResult.retrieval_errors map sub-question id to error, hotpotqa_eval keeps failed records out of the summary and counts them in n_failed (raised exceptions used to average in as 0 ms and 0 F1), and mas prints a warning and adds retrieval_errors to --json.

Docs and numbers

  • The pyproject description promised "2.4x speedup vs sequential on HotpotQA multi-hop, +F1 over single-prompt"; nothing in the repo measured either. It now states 1.31x (5 live questions) and 1.99x (smoke benchmark), and RESULTS.md no longer claims real Haiku latency pushes the speedup above 1.99x.
  • README: 65 tests, 89% coverage.

CI

  • Third-party actions pinned to full commit SHAs (version in a comment), persist-credentials: false, workflow token contents: read.

Tests

Before: 58 passed on Python 3.12 (python -m pytest -q -p no:cacheprovider); ruff check src tests benchmarks clean. After, at d296ff5: 65 passed, 1 warning (LangChain deprecation), 89% coverage (494 statements, 55 missed) from python -m pytest --cov=multi_agent_supervisor --cov-report=term -q. python -m benchmarks.smoke_parallel --delay 0.3 --repeats 2 ran.

Not done

  • The Docker image was not built locally (no Docker daemon available); the CI docker job builds it.
  • No explicit timeout or retry count on the Anthropic client.

- Pin every third-party action to a full commit SHA (version kept in a
  comment), so a moved or compromised tag cannot change what runs.
- Check out with persist-credentials: false; no step needs the token
  left in .git/config afterwards.
- Default the workflow token to contents: read and grant write scopes
  only to the jobs that publish.
A timeout or 429 on any retriever branch raised out of the LangGraph
superstep and killed the run, discarding the other branches' results.
The retriever now catches httpx errors per branch, returns no docs for
that sub-question (the analyzer answers "insufficient evidence") and
records the error in the telemetry note. The sequential baseline uses
the same helper so both benchmarks still degrade identically.
The image ran the CLI and the smoke benchmark as root. It now switches
to an unprivileged user; the benchmark results directory is the only
path the container writes to, so that is the only path chowned.
Since 59e7fa6 a failed search degrades its branch to "insufficient
evidence", but nothing downstream could tell: the sequential baseline
dropped the error, the parallel record ignored telemetry, and the CLI
exited 0 with no warning. A rerun under rate limiting would have fed
faster, lower-F1 runs into the published speedup and F1 numbers.

- retrieve_or_empty logs a warning on failure.
- SupervisorState.retrieval_errors and SequentialResult.retrieval_errors
  map sub-question id to the error.
- hotpotqa_eval records the errors and leaves failed records (retrieval
  errors, or a raised exception as before in single mode) out of the
  summary, counted in n_failed, with a warning. Exception records used
  to be averaged in as 0 ms and 0 F1; they are now excluded too, so all
  three modes treat a failed retrieval the same way.
- `mas` prints a warning line and adds retrieval_errors to --json.
The package description promised "2.4x speedup vs sequential on HotpotQA
multi-hop, +F1 over single-prompt". Nothing in the repo measured either:
the live run gave 1.31x on 5 questions and the smoke benchmark 1.99x,
and no single-agent F1 comparison was published. RESULTS.md also said
real Haiku latency "usually pushes the parallel speedup higher" than
1.99x, which the live 1.31x contradicts.
@Tajaddin
Tajaddin merged commit 373c439 into main Nov 7, 2026
5 checks passed
@Tajaddin
Tajaddin deleted the engineering-2026-11 branch November 7, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant