Repository navigation
fix: stop Sydney's progress placeholders leaking into reply content - #22
Merged
Merged
Conversation
Every plain (non-tool-call) reply came back with one of Sydney's own
status placeholders glued to the front of its content -- observed live:
{"content": "Making it happen\u2026pong"}
and, streaming, a first chunk of "Working on it\u2026" ahead of the real
"pong".
Sydney interleaves that UI chrome into the very same `target:"update"`
frames that carry the answer, and `stream_chat_reply` treated every
`author:"bot"` entry's `text` as reply text. The chrome is distinguished
ONLY by carrying a `messageType` (`Progress`, `InternalLoaderMessage`,
`ReferencesListComplete`, ...) where the real answer entry carries none
-- so accept an entry as content only when its `messageType` says it is
content (`ANSWER_MESSAGE_TYPES`), and skip the rest, logging lengths
only. Unrecognized types are skipped too but WARNed, so an answer
wrongly skipped by a future protocol change is diagnosable from the log
rather than surfacing as an unexplained empty reply.
`GeneratedCode` is deliberately KEPT as content, which is the
counterintuitive part and is now covered by a test: a live probe
(`scripts/probe_generated_code_messages.py`, added here) shows that in
code tool-calling mode the fenced ```python block holding the
`invoke_capability(...)` call arrives as a `messageType:"GeneratedCode"`
entry while the prose answer entry of that same turn holds only prose.
Filtering it out silently breaks all code-mode tool calling -- there is
then nothing left for `_extract_code_mode_calls` to parse. The
`"Coding and executing"` prefix long noted in REVERSE_ENGINEERING.md is
the adjacent `Progress` entry, and is what this change actually removes.
Two latent bugs fall out of the same fix:
- Snapshots are now diffed per `messageId`. The chrome entries are
separate messages with their own ids, so one shared "text so far"
cursor made a switch between two messages look like a snapshot that
did not extend the previous text, and the whole message was
re-emitted -- duplicating it in the reply.
- A turn that emitted only a placeholder and then failed is reported as
the failure it is. The type-2 `turnState:"Failed"` throttle check is
guarded on "no answer text yet", and a placeholder used to satisfy
that guard, so the turn "succeeded" with the placeholder as its
entire content.
Verified live against real credentials: plain and streaming replies come
back as exactly "pong" with no placeholder, tool calling still lands
3/3, and the log shows only `Progress`/`ReferencesListComplete` skipped
with zero warnings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Every plain (non-tool-call) reply came back with one of Sydney's own status
placeholders glued to the front of its content. Reproduced live against real
credentials on
master:…and on the streaming path the first chunk delivered to the client was the
placeholder on its own:
Any client that shows the assistant's message verbatim therefore showed
"Making it happen…pong". Not credential- or auth-related: authentication,token rotation and the Chathub transport were all working fine.
Why
Sydney interleaves its own transient UI chrome into the very same
target:"update"frames that carry the answer, andstream_chat_reply()treated every
author:"bot"entry'stextas reply text. The chrome isdistinguished ONLY by carrying a
messageTypewhere the real answer entrycarries none (verbatim from a live
scripts/dump_frames.pycapture):The placeholder strings are model-generated and unbounded (
"Gathering details…","Working on it…","Making it happen…","Looking into it…"),so this has to be filtered by type, never by matching text.
The fix
Accept an entry as content only when its
messageTypesays it is content(
ANSWER_MESSAGE_TYPES) and skip everything else, logging lengths only per thefile's logging convention. Types in neither set are skipped but WARNed, so
an answer wrongly skipped by some future protocol change is diagnosable from
the log file alone instead of surfacing as an unexplained empty reply.
GeneratedCodeis kept as content — the counterintuitive partIt looks exactly like code-interpreter noise, and dropping it silently breaks
all code-mode tool calling. A new live probe
(
scripts/probe_generated_code_messages.py) dumps one code-mode turn and showswhy — the tool call is not in the prose answer:
messageTypecontentTypetextProgressEarlyProgress"Looking into it…"ProgressCode"Coding and executing"GeneratedCode```python\nresult = invoke_capability('get_weather', {'city': 'Prague'})\n```GeneratedCode{"executedCode":…,"status":"Failed","stderr":"NameError: name 'invoke_capability' is not defined…"}ProgressSearchResults"OK, I'll search for 'Prague current weather'…"Entry 3 — the
invoke_capability(...)call that_extract_code_mode_calls()exists to parse — rides entirely on
GeneratedCode, while the prose answer(entry 6) holds only prose. Entry 2 is the
"Coding and executing"prefix longnoted in
REVERSE_ENGINEERING.md, and that is what this change removes.I hit this regression while verifying the first version of the fix; it is now
pinned by
test_generated_code_entry_is_kept_as_content, which asserts thecode survives and that
_extract_code_mode_calls()still finds the call.Two latent bugs fall out of the same fix
messageId. The chrome entries are separatemessages with their own ids, so one shared "text so far" cursor made a switch
between two messages look like a snapshot that didn't extend the previous
text — and the diff's only recovery is to re-yield the whole thing,
duplicating it in the reply.
the failure it is. The type-2
turnState:"Failed"throttle check is guardedon "no answer text yet", and a placeholder used to satisfy that guard — so
such a turn "succeeded" with the placeholder as its entire content instead of
surfacing as a 429.
Verification
Live, against real credentials:
'pong'(was'Making it happen…pong'), 3/3 runs['pong'](was['Working on it…', 'pong'])tool_calls -> [('get_weather', '{"city": "Prague"}')], 3/3Progress/ReferencesListCompleteskipped,GeneratedCodenever skipped, zero warnings
Offline suite: 38 passed (13 new). 10 of the new tests fail on
masterandpass here — including the duplication and masked-failure cases. Full local CI
gate green:
ruff format --check,ruff check,bandit,py_compile,--helpsmoke (pure-Python AES self-check),pytest.Proxy stays pure-stdlib and single-file; the new probe is a developer-only
script under
scripts/, not part of the shipped proxy and not run in CI.Patch version bump
0.10.0→0.10.1in bothPROXY_VERSIONandpyproject.toml.