Skip to content

feat: image generation + image understanding + an MCP endpoint at /mcp (and the image-reported-as-throttling fix) - #26

Merged
sideeffffect merged 10 commits into
masterfrom
fix/image-not-throttle
Aug 4, 2026
Merged

sideeffffect merged 10 commits into
masterfrom
fix/image-not-throttle

Conversation

@sideeffffect

@sideeffffect sideeffffect commented Aug 2, 2026 •

Copy link
Copy Markdown
Owner

Started as "what tools does Sydney actually offer?" and grew into a capabilities release: three new ways to use Copilot — image understanding, image generation, and an MCP endpoint — on top of the original bug fix and capability probe.

Every claim below was re-verified live end-to-end against this branch on 2026-08-03, driving the real proxy over HTTP with real credentials — see Verification.

1. Vision input — image understanding

Send an image and ask about it with the OpenAI-standard image_url content part (an inline data:image/...;base64,... URI). Sydney's GPT-V reads it and answers as ordinary text.

Reverse-engineered from data/har1.har (the turn "what is on this image?") plus the UploadFile POST in the 26-08-02 capture. Two steps, each with a non-obvious, load-bearing detail found only by testing live:

  • Upload to substrate.office.com/m365Copilot/UploadFile with the proxy's existing Sydney bearer token. Requires X-Variants: feature.EnableImageSupportInUploadFile — without it the endpoint feature-gates image upload off and returns an empty HTTP 403, no body, no hint — plus X-AnchorMailbox / X-Scenario.
  • The multipart FileBase64 field is the entire data: URI string — despite the name it is neither raw bytes nor bare base64; either gets {"fileSanitizer":"None","result":{"value":"InvalidRequest"}}. Derived arithmetically from the capture before sending a request: the browser's field was 716,422 chars while the server echoed fileSize: 716400, a difference of exactly len("data:image/png;base64,"). The OpenAI caller already sends exactly that URI, so it is forwarded verbatim.
  • Reference the returned docId from the chat turn as an ImageFile messageAnnotations entry. The gptv optionsSets that make Sydney read it were already in OPTIONS_SETS, so the invocation is otherwise unchanged.

Only inline data: URIs are accepted — a remote http(s):// URL is skipped rather than fetched, because fetching a caller-supplied URL server-side is a request-forgery footgun. Images attach to the latest user turn; tools + images in one request is unsupported and warns.

2. Image generation — POST /v1/images/generations and MCP generate_image

Image output now works too, on OpenAI's own image endpoint rather than on chat completions:

curl http://127.0.0.1:8001/v1/images/generations \
  -H 'Content-Type: application/json' \
  -d '{"prompt": "a red bicycle leaning against a white brick wall"}'

→ {"created": …, "data": [{"b64_json": "…"}], "output_format": "png"} — live: a valid 1536×1024 PNG, 2.67 MB.

Two steps under the hood, both established live:

  • Ask for the image in an ordinary chat turn (Sydney has no separate image API) and collect the finished file's URL off the Progress frames' contentGenerationProgressList — only entries with status: 2 carry a populated ImageReferenceUrls, so a fetch must use the final one.
  • Download it with a second access token, for the designerappservice resource, minted from the very same FOCI refresh token — so no extra credential is involved. The URL's fileToken query parameter must be moved into a filetoken request header; leaving it in the query string gets HTTP 400.

Deliberately not on /v1/chat/completions: that response can only carry text, so the options there would be a link the caller cannot authenticate to or megabytes of base64 in content. On MCP it is a first-class image content block ({"type": "image", "data": …, "mimeType": …}), which is exactly why it is offered there as well.

Security of the download path. url is server-supplied and this is the one place the proxy attaches a real access token to a URL it did not construct, so the host is asserted against designerapp.officeapps.live.com before anything is sent and redirects are refused. The URL and fileToken are never logged — that token grants access to the file, so it falls under the project's "never log secrets" rule; only sizes and the host are.

response_format is accepted but only b64_json is honored (a url would 401 for the caller) — mirroring OpenAI's own GPT-image models. n runs n independent turns, since Sydney produces one image per turn and has no batch parameter; capped at 4. size/quality/style/background are accepted-and-warned: Copilot exposes no such control. usage is omitted rather than faked as zeros — OpenAI's schema makes it optional here, unlike on chat completions.

Two honest caveats, both measured

  • Generation is prompt-steered, not contractual. It is reached by asking, so Copilot can decline or answer in prose. That surfaces as HTTP 502 upstream_no_image carrying Copilot's own wording, never as an empty success.
  • There is a daily per-account image cap — discovered by hitting it mid-review. Copilot reports it only as prose ("Sorry, I can't generate any more images today"), and its own throttling.metering block does not reflect it: ImageGeneration still read 100 with the cap already exhausted. Independent confirmation that the metering numbers are not interpretable (see §4). The cap was observed to reset at UTC midnight — 23:25 UTC failed, 00:02 UTC succeeded.

3. MCP endpoint at /mcp, always on

The same server also speaks MCP over the Streamable HTTP transport at POST /mcp, on the same host/port as /v1, always — there is no flag to turn it off. No per-request auth, same as /v1. Hand-rolled JSON-RPC on the stdlib http.server — no MCP SDK, per the single-file/stdlib-only rule.

Implements the minimal spec-compliant subset a client needs: initialize (echoes the client's protocolVersion), notifications/* (202, no body), ping, tools/list, tools/call. Stateless; single application/json response. GET /mcp → 405 (this server never initiates messages). Three tools:

  • ask_copilot {prompt} → Copilot's text answer.
  • describe_image {image: data-URI, prompt?} → the vision path from §1.
  • generate_image {prompt} → the image itself, as a real MCP image content block.

Bad tool arguments come back as isError results (visible to the model); unknown method/tool as JSON-RPC errors (-32601 / -32602).

There is deliberately no off switch. /mcp reaches the same backend, as the same single configured identity, with the same absence of per-request auth, as the /v1 API on the same localhost port — so disabling it protects nothing /v1 does not already expose, and it is one of the two ways this proxy is meant to be used rather than an optional extra. (A --disable-mcp flag existed earlier on this branch and was removed in dbd0ccd; it never appeared in a release, so nothing that shipped can name it.)

Two deliberate absences, documented next to the code so they don't get "fixed" later:

  • No tools parameter, ever. Routing the /v1 tool-calling emulation through an MCP tool would nest two loops — the client's model calls ask_copilot, which prompt-steers Sydney into emitting a fake tool call, which returns as plain text to a model that already has a real, schema-enforced mechanism. Copilot is a leaf in the MCP role.
  • No conversation_id yet. Each call is an independent turn, which each tool description states plainly. The eventual fix is the opposite of /v1's: /v1 infers continuity by fingerprinting the resent messages[], which has no equivalent when the caller sends one prompt. Purely additive when wanted.

4. The original bug: image generation reported as rate limiting

Sydney's image generation works — it invokes image_gen, streams Loading image progress with a contentGenerationProgressList, and finishes with status: 2 and a real PNG link. What it never streams is answer text. So the turn completed with total_length=0 chars, indistinguishable to _looks_like_throttled_empty_reply from the silent empty-completion throttle:

HTTP 429  {"type": "upstream_throttled",
           "message": "...Microsoft is temporarily throttling this account -- wait a bit and try again."}

Wrong in every part. Nothing was throttled, and the identical request fails identically forever — so a client that honors 429 by backing off and retrying loops indefinitely.

Now surfaced as HTTP 502 unsupported_upstream_content, stating outright that this is not a rate limit, that retrying will not help, and pointing at the two endpoints that do return the image. A genuinely empty reply still surfaces as 429, because there "back off and retry" is correct advice.

Two earlier revisions of this PR got the explanation wrong, in instructive ways — both now retired. First, the 429 above. Then the docs claimed the generated file "needs the browser's Office (OHP) cookie session, which the proxy lacks", and that image output was therefore unreachable. The observation behind that was real — the link 401s both anonymously and with the proxy's Sydney bearer token — but the inference was not: it wants a different token from the same refresh token (§2). What was documented as a protocol dead end shipped as a feature in this same PR.

5. Sydney's capability surface, measured

scripts/probe_sydney_capabilities.py drives one turn per suspected capability and records what fires on the wire instead of asking Copilot what it can do.

throttling.metering — 15 named capabilities, on the type-2 StreamItem frame (v0.12.0's quota logging read type-1 update frames only, so it never saw this):

LLMOnly=100  ImageGeneration=100  WXPAgentMode=7  FileReference=3
TenantDataAccess=0  PersonalDataAccess=0  CodeInterpreter=0  ImageAnalysis=0
VisualCreator=0  GraphicArt=0  DeepResearch=0  DeepWork=0
CopilotTuning=0  NotebookCowork=0  CostQuota=0

Logged, and nothing else — the numbers were measured not to mean what they look like: CodeInterpreter read 0 on the very turn whose code interpreter demonstrably ran; no value moved across 8 consecutive turns; and ImageGeneration read 100 while the daily cap was exhausted (§2). Names are solid; numbers aren't interpretable, so the proxy never branches on them.

Sydney emits real, native, OpenAI-shaped tool_calls for its own built-ins — {"function":{"name":"image_gen",...},"id":"call_zvq9...","type":"function"} — which disproves the module docstring's standing claim that no such mechanism exists. Corrected, and the invocations are now logged. Deliberately no behavior change on the strength of it: whether a client-declared tool can be registered into that namespace is unanswered, and that question is now the highest-value open thread in the project.

No tenant grounding on this path — mail / files / calendar / directory all refuse, consistent with TenantDataAccess: 0. Caveat: one account, one tenant, one license; this may be a licensing artifact.

Verification

Live sweep against this branch on 2026-08-03, driving the real proxy over HTTP with live credentials.

group result
Image generation — 200 / valid 1536×1024 PNG / output_format: png / usage omitted 4/4
Closed loop — generated image fed back through the proxy's own vision path 2/2
n=2 → two distinct images (differing SHA-256) from two independent turns 1/1
MCP generate_image → real image content block, image/png, valid 1254×1254 PNG 2/2
Daily-cap path → 502 upstream_no_image + Copilot's own wording; reset at UTC midnight 3/3
Chat endpoint asked for an image → 502 unsupported_upstream_content (the original bug) 2/2
Image-endpoint error paths (no prompt, blank prompt, response_format=url, bad JSON, GET) 5/5
Vision (chat + MCP describe_image, SSRF guard skips a remote URL) 3/3
MCP (initialize, notification 202, tools/list = 3 tools, −32601, −32602, GET 405) 6/6
/mcp served unconditionally; --disable-mcp rejected as unrecognized 3/3
Text / streaming SSE / /v1/models / /healthz regression 4/4
Total 35/35

The closed loop is the strongest single result: /v1/images/generations produced an image from "a red bicycle leaning against a white brick wall", and feeding that PNG back through the proxy's own vision path returned "A red bicycle is standing in front of a white brick wall." Both directions, verified against each other, through the shipped HTTP API. The MCP pair round-trips the same way (generate_image duck → describe_image → "Yellow rubber duck.").

The log for the successful generation also confirms the credential story end to end: a designerappservice token minted from the same refresh token (opaque=True — it is a JWE), the fetch logged as host-only, and zero occurrences of the URL or its fileToken.

This sweep found four real defects, all in documentation this PR itself had added, and all found by probing the running proxy rather than reading it — --help and the MCP initialize result are what a user and a client actually see. Fixed in 56af8d8:

  • the module docstring's opening paragraph (printed by --help) advertised only two MCP tools;
  • MCP_INSTRUCTIONS — surfaced to a real client's model — likewise listed two, so a client could reasonably conclude generate_image did not exist;
  • the KNOWN LIMITATIONS headline still read "image OUTPUT (generation) does not [work]", contradicting the next sentence of its own bullet;
  • README's MCP section said "Two tools are exposed:" above a list of three.

A first pass at the tenant probes reported 2/4 — that was my detector failing, not the claim: Sydney writes curly apostrophes (can’t) and inserts adverbs (can't **directly** view), which brittle substring matching misses. Re-run with a normalising regex and full replies: 4/4.

Testing, version, docs

  • 167 offline tests pass, up from 61 on master (+106): test_image_generation.py 44, test_generated_content.py 26, test_vision_input.py 20, test_mcp.py 16. Plus 2 new stdlib-only selfcheck cases (10 → 12) so the vision and MCP surfaces are exercised on every supported interpreter, including the ones pytest cannot be installed on.
  • The image tests cover the security-critical bits specifically: host pinning, redirect refusal, the filetoken header move, scope separation, and an assertion that a refusal never echoes the capability token.
  • Fixtures are verbatim live captures — the doubly-JSON-encoded invocation string, all 15 metering entries.
  • Full local gate green at HEAD: ruff format/check, bandit (0 issues), vermin (floor still 3.7), py_compile, --help, selfcheck 12/12, pytest 167.
  • Version 0.12.0 → 0.14.0 (master is 0.12.0), in sync across PROXY_VERSION, pyproject.toml, uv.lock.
  • Docs: module docstring, README (MCP section, "Generating images" section, revised image/limitations bullets), REVERSE_ENGINEERING.md (Vision input; Fetching a generated image; the vision negative results and method).
  • data/ is now gitignored wholesale — it holds credential-grade captures, and one of them was previously matched only by a machine-local global gitignore, i.e. one git add -A away from being committed on any other clone.

No new dependencies; still a single stdlib-only file.

Probing Sydney's own capabilities live (new
scripts/probe_sydney_capabilities.py, 8 turns on a real account) turned up a
real bug and two documentation errors.

THE BUG: an image request came back as HTTP 429 "Microsoft is temporarily
throttling this account -- wait a bit and try again". Every part of that was
wrong. Sydney's image generation works: it invokes its `image_gen` tool,
streams `Loading image` progress carrying a `contentGenerationProgressList`,
and finishes with `status: 2` and a real PNG link. What it never streams is
answer TEXT -- so the turn completed with `total_length=0 chars`, which
`_looks_like_throttled_empty_reply` could not tell apart from the silent
empty-completion throttle. Nothing was throttled, and waiting never helps:
the identical request fails identically forever, so a client that honors 429
by backing off and retrying loops indefinitely.

Delivering the image instead was tested, not assumed: the
`ImageReferenceUrls` link (designerapp.officeapps.live.com) returns HTTP 401
both anonymously AND with this proxy's own Sydney bearer token. It needs the
browser's Office cookie session, which this proxy deliberately does not have.

So the fix is accurate reporting: `stream_chat_reply` now notices generated
non-text content and raises `UnsupportedContentError`, surfaced as HTTP 502
`unsupported_upstream_content` with a message that says explicitly that this
is not a rate limit and retrying will not help. A genuinely empty reply --
no generated content -- still surfaces as 429, where that advice is right.
The retry-as-a-fresh-conversation path skips it too, since a second attempt
produces the same image and the same absence of text.

ALSO FOUND, and now logged:

- `throttling.metering`, Sydney's own list of 15 named metered capabilities
  (CodeInterpreter, TenantDataAccess, PersonalDataAccess, ImageGeneration,
  ...). It rides on the type-2 StreamItem frame, so v0.12.0's quota logging
  -- which read type-1 `update` frames only -- never saw it. Logged and
  nothing more: the allowance numbers were measured NOT to mean what they
  look like (`CodeInterpreter` read 0 on a turn whose code interpreter
  demonstrably ran; no value moved across 8 consecutive turns), so the proxy
  never branches on them.

- `invocation`, direct wire evidence that Sydney emits REAL native
  OpenAI-shaped tool calls for its own built-ins:
  {"function":{"name":"image_gen","arguments":"{...}"},
   "id":"call_zvq9VI82lh3kvZdNlbDl5c","type":"function"}
  This disproves the module docstring's claim that "Sydney has no native
  OpenAI-style tools/tool_calls mechanism", which is corrected. Whether a
  CLIENT-declared tool can be registered into that namespace is unanswered,
  and nothing here assumes it can -- the emulation is untouched.

Also documented: no tenant grounding on this path (mail, files, calendar and
directory all refused, 4 of 4), consistent with TenantDataAccess=0 /
PersonalDataAccess=0. What does work is the Python code interpreter (plugin
id `Pyexec`) and web search.

New tests/test_generated_content.py (26 cases) with fixtures taken verbatim
from the live capture. Verified live end-to-end after the change: the image
request now returns 502 `unsupported_upstream_content`, an ordinary text
request still returns 200, and both new log lines fire against real frames.

Version 0.12.0 -> 0.13.0 (PROXY_VERSION, pyproject.toml, uv.lock in sync).
COMMITEOF && git log --oneline -1
…nnotations

Sydney's image *understanding* works end-to-end through this proxy's own
bearer token (no browser cookie session needed) -- the mirror image of the
generation case, which can't be delivered. Wired to the OpenAI-standard
`image_url` content part; verified live 2026-08-02 (solid red/green/blue PNGs
each described correctly, streaming and non-streaming).

Reverse-engineered from data/har1.har ("what is on this image?") plus the
UploadFile POST in the 26-08-02 capture. Two steps, both with two
non-obvious, load-bearing details found only by testing live:

- POST the image to substrate's /m365Copilot/UploadFile. Requires header
  X-Variants: feature.EnableImageSupportInUploadFile (without it the endpoint
  feature-gates image upload off and returns an empty HTTP 403), plus
  X-AnchorMailbox / X-Scenario. The multipart `FileBase64` field is the ENTIRE
  `data:<mime>;base64,...` data URI STRING -- not raw bytes, not bare base64
  (either makes the sanitizer reject the POST with
  {"fileSanitizer":"None","result":{"value":"InvalidRequest"}}). The OpenAI
  image_url value already is that data URI, so it's forwarded verbatim.
- Reference the returned docId from the chat turn's user message as an
  `ImageFile` messageAnnotations entry. The gptv optionsSets that make Sydney
  read it are already in OPTIONS_SETS, so the invocation is otherwise
  unchanged.

Surface: images come from the latest user turn; only inline data: URIs are
accepted (a remote http(s) image URL is skipped, to avoid server-side request
forgery); `tools` + images in one request is unsupported and warns. New
offline tests in tests/test_vision_input.py (20 cases, network stubbed).
Docs: REVERSE_ENGINEERING.md "Vision input", README, module docstring.
MSG && git log --oneline -1
…e/ask_copilot

Serve the same Sydney backend a second way: an MCP (Model Context Protocol)
endpoint at POST /mcp over the Streamable HTTP transport, on the same
host/port as the OpenAI /v1 API, enabled by default (--disable-mcp to turn
off). No per-request auth, same as /v1. Hand-rolled JSON-RPC on the stdlib
http.server -- no MCP SDK, per the single-file/stdlib-only rule.

Implements the minimal spec-compliant subset a client needs: the initialize
handshake (echoes the client's protocolVersion), notifications/* (202, no
body), ping, tools/list, and tools/call. Stateless (no session id issued);
returns a single application/json JSON-RPC response. GET /mcp -> 405 (no
server-initiated SSE); with --disable-mcp, /mcp -> 404.

Two tools, both relaying to Copilot:
- ask_copilot {prompt} -> Copilot's text answer (its web search + Python code
  interpreter still apply).
- describe_image {image: data: URI, prompt?} -> reuses the new vision-input
  path (UploadFile + ImageFile annotation) to ask about an image.

Tool-argument problems come back as isError results (visible to the model);
unknown method/tool come back as JSON-RPC errors (-32601/-32602). Upstream
throttling and empty replies are surfaced as isError text too.

Verified live 2026-08-02: initialize / tools/list / ask_copilot ('gamma') /
describe_image (a blue PNG -> 'Blue') / every error path. New offline suite
tests/test_mcp.py (16 cases) drives the real handler with run_chat_turn
stubbed; two stdlib-only selfcheck cases cover the vision + MCP surface on
every supported interpreter. Bumps version 0.13.0 -> 0.14.0 (PROXY_VERSION,
pyproject, uv.lock). Docs: module docstring, README "MCP endpoint (/mcp)".
MSG && git log --oneline -3
@sideeffffect sideeffffect changed the title fix: stop reporting Sydney's image generation as rate limiting (+ Sydney's capability surface, probed) feat: vision input (image understanding) + MCP endpoint at /mcp (+ image-not-throttle fix & capability probe) Aug 2, 2026
No behavior change -- comments and README only. Both of these look like
obvious missing features and are not, so the reasoning belongs next to the
code rather than only in a PR discussion.

1. No `tools` parameter on the MCP tools, ever. Routing this proxy's /v1
   tool-calling emulation through an MCP tool would nest two tool loops: the
   MCP client's own model calls ask_copilot, which prompt-steers Sydney into
   emitting a fake tool call, which returns as plain TEXT to a model that
   already has a real, schema-enforced tool mechanism. Incoherent even before
   accounting for the emulation being probabilistic. Copilot is a leaf in the
   MCP role.

2. No conversation_id -- yet. Each MCP call is already documented in its own
   tool description as an independent, stateless turn, but the README didn't
   say so, which is where a user would look before pointing a client at it.
   Records why the eventual fix is the OPPOSITE of /v1's: /v1 infers
   continuity by fingerprinting the resent messages[] array, which has no
   equivalent when the caller sends one prompt rather than a transcript,
   whereas an explicit id is honest in MCP precisely because arguments are
   supplied by a model reading a schema. Purely additive when wanted.
MSG && git log --oneline -1 && git push -q origin fix/image-not-throttle && echo pushed
…ookie claim

Moves everything the throwaway scripts/probe_image_io.py knew into
REVERSE_ENGINEERING.md so it can be deleted losslessly (it was never tracked
in git -- it only ever existed on one machine, which is exactly the problem).
Documentation only: no behavior change, the proxy diff is docstrings.

The important part is not transcription but a CORRECTION. Three places --
REVERSE_ENGINEERING.md, the module docstring, and README -- asserted that
Sydney's generated images need "the browser's Office (OHP) cookie session"
and are therefore out of reach. That was WRONG, and the probe disproved it
live. The observation behind it was real (the ImageReferenceUrls link 401s
both anonymously and with this proxy's Sydney bearer token); the inference
from it was not. The link wants a DIFFERENT bearer token, minted from the
very same FOCI refresh token for the designerappservice.officeapps.live.com
resource, plus the URL's fileToken query param moved into a `filetoken`
header. That fetch returns HTTP 200 and a valid 2,693,151-byte PNG.

So image output is UNIMPLEMENTED, not impossible, and all three docs now say
so -- with the corrected reason: chat-completions can only carry text, so the
choices would be a link the caller cannot authenticate to or megabytes of
inline base64. MCP's image content blocks are the natural home if it is ever
added. The 502 unsupported_upstream_content behavior is unchanged and still
correct.

New in REVERSE_ENGINEERING.md:

- "Fetching a generated image" -- the full recipe: URL shape, the fileToken
  payload ({TokenPrefix:"AAD-<guid>", UserObjectId, ClientName:"CopilotSydney"}
  -- the "AAD-" prefix is the tell that it wants a token, not a cookie), the
  exact token request (byte-identical to the Sydney one except `scope`), the
  5-part JWE response (so its claims are NOT parseable), the refresh-token
  rotation hazard, and both live controls: `Bearer ` prefix is optional,
  `filetoken` header is required (omitting it -> 400).
- "Vision input -- the negative results and method behind it" -- the
  guess-proof digit-PNG methodology and its CAPTCHA-refusal confound, the
  finding that conversationId binding alone is NOT enough, the nine candidate
  payload shapes that all missed before messageAnnotations was recovered, and
  the Firefox-vs-Chrome HAR difference (only Chrome records WebSocket
  payloads) that unblocked it.
- The arithmetic that derived the FileBase64 data-URI format from the capture
  without sending a request: 716,422 field chars vs the echoed fileSize
  716,400 = exactly len("data:image/png;base64,").
- A reproduction recipe for both halves from shipped code, so no probe script
  is needed to re-derive any of it.

Side effect: `uv run ruff check .` now passes across the whole repo (the
probe was carrying all 26 remaining lint errors).
MSG && git log --oneline -1 && git push -q origin fix/image-not-throttle && echo "pushed"
…tures

The debug artifacts (HARs, the SignalR frame dump, the curl scratch, a saved
proxy log) now all live in data/. Ignore that directory as a whole rather
than relying on the per-extension rules that happened to cover them.

This closes a real gap rather than just tidying. Every file moved was already
ignored, but `data/m365_openai_proxy.petr.log` was matched ONLY by
~/.gitignore_global's `*.log` -- a machine-local file that is not part of the
repo. On any other clone, or for any other contributor, that credential-grade
capture would have shown up as untracked and been one `git add -A` away from
being committed.

Ignoring the directory also makes the rule prospective instead of reactive: a
capture saved with an unanticipated name (a mitmproxy .flow, a .txt frame
dump, a saved JSON response body) is covered without anyone having to
remember to add a pattern first. Verified with `git check-ignore -v` that all
15 files in data/ now resolve to this rule, and that hypothetical
.flow/.txt/.json names there resolve to it too. The per-extension rules stay
as a safety net for a capture saved outside data/ by accident.

REVERSE_ENGINEERING.md's "Not actually in this repo" note now names data/ as
the convention, so the filenames it cites throughout can be resolved to
data/<name> by anyone who has their own captures.
MSG && git log --oneline -1 && git push -q origin fix/image-not-throttle && echo pushed && git status -sb | head -1
Caught by following my own written recipe literally instead of assuming it
worked. When scripts/probe_image_io.py was deleted, the replacement recipe
said to "pull the ImageReferenceUrls value out of m365_openai_proxy.log at
DEBUG". That step is false and always was: measured live, a DEBUG log of the
exact turn that generates an image contains ZERO occurrences of
`designerapp`, `ImageReferenceUrls` or `fileToken`. Anyone following the
recipe would have found nothing and been stuck -- i.e. the probe's deletion
was NOT lossless the way the commit claimed.

The proxy is right not to log it, and the fix is the doc, not the code:
`_note_generated_content` deliberately tallies only generated-content KINDS
("image x3"), and the `fileToken` in that URL is an access-granting
capability for the file -- logging it verbatim would violate this project's
"never log secrets" rule. The recipe now says so explicitly so nobody
"helpfully" adds that logging later.

The corrected recipe reads the URL from the RAW FRAMES
(open_chathub/send_chat_message/SignalRBuffer -- scripts/dump_frames.py with
its prompt changed), and was then run end to end to confirm it actually
works: raw frames -> URL on designerapp.officeapps.live.com, scope swap ->
2513-char 5-part JWE, fetch -> HTTP 200 image/png 3,217,818 bytes valid PNG.
Both controls reproduced independently of the original run: the `Bearer `
prefix is optional, and omitting the `filetoken` header still gives 400.

So the MECHANISM claim was sound and is now doubly confirmed; only the
recipe's step 2 was wrong. Also records the mandatory
CredentialStore.rotate() persistence step, which the old wording left
implicit.
MSG && git log --oneline -1
@sideeffffect sideeffffect changed the title feat: vision input (image understanding) + MCP endpoint at /mcp (+ image-not-throttle fix & capability probe) feat: vision input (image understanding) + MCP endpoint at /mcp, plus the image-generation-misreported-as-throttling fix Aug 2, 2026
The `/v1/images/generations` commit added a third MCP tool and a second
working image direction, but four doc sites still described the world as
it was before it:

- the module docstring's opening paragraph (and so `--help`, which prints
  it) advertised only `ask_copilot` and `describe_image`;
- `MCP_INSTRUCTIONS` -- which real MCP clients surface to their model --
  likewise listed two tools, so a client could reasonably conclude
  `generate_image` was not available;
- the KNOWN LIMITATIONS headline still read "image OUTPUT (generation)
  does not [work]", contradicting the very next sentence in its own
  bullet;
- README's MCP section said "Two tools are exposed:" above a list of
  three.

All four found by live-probing the running proxy rather than by reading:
`--help` and the `initialize` result are what a user and a client
actually see.
EOF && git --no-pager log --oneline -2 && git push -q origin fix/image-not-throttle && echo "PUSHED"
@sideeffffect sideeffffect changed the title feat: vision input (image understanding) + MCP endpoint at /mcp, plus the image-generation-misreported-as-throttling fix feat: image generation + image understanding + an MCP endpoint at /mcp (and the image-reported-as-throttling fix) Aug 3, 2026
/mcp is one of the two ways this proxy is meant to be used, not an
optional extra bolted onto the /v1 API, so an off switch was pure config
surface for no benefit.

It bought no security either, which was the only argument for keeping
it: /mcp reaches exactly the same Copilot backend, as the same single
configured identity, with the same absence of per-request auth, as the
/v1 API sitting next to it on the same localhost port. Disabling it
protected nothing that /v1 did not already expose.

Removes the flag, the `mcp_enabled` parameter on `make_handler` that sat
behind it, and the conditional in both route tables; /mcp is now reached
the same unconditional way as /v1.

Not a breaking change: `--disable-mcp` was introduced earlier on this
same branch and has never appeared in a release (master is 0.12.0, and
there is no v0.13/v0.14 tag), so no released invocation can name it. The
version therefore stays 0.14.0.

The `--disable-mcp -> 404` test is replaced rather than deleted, by one
that guards the decision itself: `make_handler` takes no toggle, the
docstring advertises no flag, and the only handler you can build serves
/mcp. Live-verified: `--disable-mcp` is now rejected as an unrecognized
argument, and a default instance answers `initialize` with 3 tools while
GET /mcp still 405s and /v1 is unaffected.
EOF && git --no-pager log --oneline -1 && git push -q origin fix/image-not-throttle && echo "PUSHED" && git rev-list --left-right --count origin/fix/image-not-throttle...HEAD
@sideeffffect
sideeffffect merged commit 80b6118 into master Aug 4, 2026
22 checks passed
@sideeffffect
sideeffffect deleted the fix/image-not-throttle branch August 4, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant