Skip to content

fix: stop Sydney's progress placeholders leaking into reply content - #22

Merged
sideeffffect merged 1 commit into
masterfrom
fix/progress-message-leak
Aug 1, 2026
Merged

sideeffffect merged 1 commit into
masterfrom
fix/progress-message-leak

Conversation

@sideeffffect

Copy link
Copy Markdown
Owner

The bug

Every plain (non-tool-call) reply came back with one of Sydney's own status
placeholders glued to the front of its content. Reproduced live against real
credentials on master:

$ curl -s localhost:8001/v1/chat/completions -d '{"messages":[{"role":"user","content":"Say the single word: pong"}], ...}'
{"choices":[{"message":{"role":"assistant","content":"Making it happen…pong"}}]}

…and on the streaming path the first chunk delivered to the client was the
placeholder on its own:

data: {"choices":[{"delta":{"role":"assistant","content":"Working on it…"}}]}
data: {"choices":[{"delta":{"content":"pong"}}]}

Any client that shows the assistant's message verbatim therefore showed
"Making it happen…pong". Not credential- or auth-related: authentication,
token rotation and the Chathub transport were all working fine.

Why

Sydney interleaves its own transient UI chrome into the very same
target:"update" frames that carry the answer, and stream_chat_reply()
treated every author:"bot" entry's text as reply text. The chrome is
distinguished ONLY by carrying a messageType where the real answer entry
carries none (verbatim from a live scripts/dump_frames.py capture):

// chrome — arrives BEFORE the first answer token
{"text": "Gathering details…", "author": "bot", "messageType": "Progress",
 "contentType": "EarlyProgress", "isPersisted": false, "messageId": "9d4aa596…"}

// the answer — no messageType key at all
{"text": "hello world", "author": "bot", "contentOrigin": "DeepLeo",
 "responseIdentifier": "Default", "messageId": "8e4511c6…", "adaptiveCards": [...]}

The placeholder strings are model-generated and unbounded ("Gathering details…", "Working on it…", "Making it happen…", "Looking into it…"),
so this has to be filtered by type, never by matching text.

The fix

Accept an entry as content only when its messageType says it is content
(ANSWER_MESSAGE_TYPES) and skip everything else, logging lengths only per the
file's logging convention. Types in neither set are skipped but WARNed, so
an answer wrongly skipped by some future protocol change is diagnosable from
the log file alone instead of surfacing as an unexplained empty reply.

GeneratedCode is kept as content — the counterintuitive part

It looks exactly like code-interpreter noise, and dropping it silently breaks
all code-mode tool calling. A new live probe
(scripts/probe_generated_code_messages.py) dumps one code-mode turn and shows
why — the tool call is not in the prose answer:

# messageType contentType text
1 Progress EarlyProgress "Looking into it…"
2 Progress Code "Coding and executing"
3 GeneratedCode — ```python\nresult = invoke_capability('get_weather', {'city': 'Prague'})\n```
4 GeneratedCode — {"executedCode":…,"status":"Failed","stderr":"NameError: name 'invoke_capability' is not defined…"}
5 Progress SearchResults "OK, I'll search for 'Prague current weather'…"
6… absent — the prose answer, streamed as growing snapshots

Entry 3 — the invoke_capability(...) call that _extract_code_mode_calls()
exists to parse — rides entirely on GeneratedCode, while the prose answer
(entry 6) holds only prose. Entry 2 is the "Coding and executing" prefix long
noted in REVERSE_ENGINEERING.md, and that is what this change removes.
I hit this regression while verifying the first version of the fix; it is now
pinned by test_generated_code_entry_is_kept_as_content, which asserts the
code survives and that _extract_code_mode_calls() still finds the call.

Two latent bugs fall out of the same fix

  • Snapshots are now diffed per messageId. The chrome entries are separate
    messages with their own ids, so one shared "text so far" cursor made a switch
    between two messages look like a snapshot that didn't extend the previous
    text — and the diff's only recovery is to re-yield the whole thing,
    duplicating it in the reply.
  • A turn that emitted only a placeholder and then failed is now reported as
    the failure it is.
    The type-2 turnState:"Failed" throttle check is guarded
    on "no answer text yet", and a placeholder used to satisfy that guard — so
    such a turn "succeeded" with the placeholder as its entire content instead of
    surfacing as a 429.

Verification

Live, against real credentials:

  • plain reply → 'pong' (was 'Making it happen…pong'), 3/3 runs
  • streaming → one single chunk, ['pong'] (was ['Working on it…', 'pong'])
  • tool calling → tool_calls -> [('get_weather', '{"city": "Prague"}')], 3/3
  • longer multi-snapshot prose reply → no duplication, no placeholder
  • log shows only Progress / ReferencesListComplete skipped, GeneratedCode
    never skipped, zero warnings

Offline suite: 38 passed (13 new). 10 of the new tests fail on master and
pass here — including the duplication and masked-failure cases. Full local CI
gate green: ruff format --check, ruff check, bandit, py_compile,
--help smoke (pure-Python AES self-check), pytest.

Proxy stays pure-stdlib and single-file; the new probe is a developer-only
script under scripts/, not part of the shipped proxy and not run in CI.
Patch version bump 0.10.0 → 0.10.1 in both PROXY_VERSION and
pyproject.toml.

Every plain (non-tool-call) reply came back with one of Sydney's own
status placeholders glued to the front of its content -- observed live:

  {"content": "Making it happen\u2026pong"}

and, streaming, a first chunk of "Working on it\u2026" ahead of the real
"pong".

Sydney interleaves that UI chrome into the very same `target:"update"`
frames that carry the answer, and `stream_chat_reply` treated every
`author:"bot"` entry's `text` as reply text. The chrome is distinguished
ONLY by carrying a `messageType` (`Progress`, `InternalLoaderMessage`,
`ReferencesListComplete`, ...) where the real answer entry carries none
-- so accept an entry as content only when its `messageType` says it is
content (`ANSWER_MESSAGE_TYPES`), and skip the rest, logging lengths
only. Unrecognized types are skipped too but WARNed, so an answer
wrongly skipped by a future protocol change is diagnosable from the log
rather than surfacing as an unexplained empty reply.

`GeneratedCode` is deliberately KEPT as content, which is the
counterintuitive part and is now covered by a test: a live probe
(`scripts/probe_generated_code_messages.py`, added here) shows that in
code tool-calling mode the fenced ```python block holding the
`invoke_capability(...)` call arrives as a `messageType:"GeneratedCode"`
entry while the prose answer entry of that same turn holds only prose.
Filtering it out silently breaks all code-mode tool calling -- there is
then nothing left for `_extract_code_mode_calls` to parse. The
`"Coding and executing"` prefix long noted in REVERSE_ENGINEERING.md is
the adjacent `Progress` entry, and is what this change actually removes.

Two latent bugs fall out of the same fix:

- Snapshots are now diffed per `messageId`. The chrome entries are
  separate messages with their own ids, so one shared "text so far"
  cursor made a switch between two messages look like a snapshot that
  did not extend the previous text, and the whole message was
  re-emitted -- duplicating it in the reply.
- A turn that emitted only a placeholder and then failed is reported as
  the failure it is. The type-2 `turnState:"Failed"` throttle check is
  guarded on "no answer text yet", and a placeholder used to satisfy
  that guard, so the turn "succeeded" with the placeholder as its
  entire content.

Verified live against real credentials: plain and streaming replies come
back as exactly "pong" with no placeholder, tool calling still lands
3/3, and the log shows only `Progress`/`ReferencesListComplete` skipped
with zero warnings.
@sideeffffect
sideeffffect merged commit 83d04be into master Aug 1, 2026
14 checks passed
@sideeffffect
sideeffffect deleted the fix/progress-message-leak branch August 1, 2026 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant