Let a builder that sees open a picture itself by youkaicountry · Pull Request #1183 · storytold/craft-anything

youkaicountry · 2026-11-08T06:00:37Z

Stacked: on #1182 (shrinking large pictures) → #1179 (pictures to builders that see) → #1177 → #1161 → #1155. It reuses #1182's shrinking. Only the last two commits are this PR: the claim and Let a builder that sees open a picture itself. I'll rebase as those merge.

Why

With #1179 an API builder whose model sees is shown the pictures the run attaches: the original's with the first message, and each check's after it. It still couldn't look at anything else. build_file_read refused any binary file, so a screenshot it saved of its own build, or a seventh picture in reference/, was out of reach.

What changes

Engine (build_file_read).

A PNG, JPEG, GIF or WebP gets a typed answer instead of the binary-file refusal: {path, picture, media_type, bytes, note}. Here picture is the path as shown in the project.
Other binary files are still refused.
The picture sniffer moves from the ai crate to web_common::picture_media_type, and model::image_media_type calls it, so there's one copy.
The tool's description says how pictures are handled.

Agent loop.

LoopConfig::project says which folder the build tools' paths are relative to.
A tool result with a picture field (a typed field, not a tool name) marks the picture as opened.
After the step's results, a model that sees is shown each opened picture after a line naming it. Pictures are read with Shrink a picture too large to show a model #1182's pictures::load, so a big one is shrunk and named with its scale. A step shows at most six, and the rest are named so the model can open them next.
A model that doesn't see gets "[app/shot.png is a picture, but this model isn't shown pictures.]".
Agent apps are unchanged: they open pictures with their own Read tool.

Run. AiHub::start_builder takes the project folder (its only caller is the run). The sighted builder's note now says it can open other pictures with build_file_read.

Tests

Engine: build_file_read on a PNG (by a root-style path, /app/shot.png) answers with picture: "app/shot.png", media_type: "image/png" and no text, and other binary files are still refused.
Agent: a_picture_a_tool_opens_is_shown_after_its_result_to_a_model_that_sees. One step opens a picture, a missing picture and a non-picture result. A sighted model gets the three results, then Picture: app/shot.png and the image, then the missing one named as not shown. A blind model gets no image and the note. With the agent no longer noting opened pictures, it fails.
Totals: 557 engine, 528 ai, and every MCP, CLI and ui-egui test pass. clippy, fmt, layers, locks, secrets and i18n-coverage are clean, and cargo xtask tools leaves docs/tools.md as it was.

Not tested: the run giving the builder its project folder is one line that no test covers (run tests pass in their own conversation). It hasn't been run live yet.

🤖 Generated with Claude Code

coderabbitai · 2026-11-08T06:00:44Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

youkaicountry · 2026-11-09T23:51:24Z

Rebased onto main (2e7f276) with the stack under it. #1258 gave start_builder and start_loop the run's mode (deep); project now comes after it, and main's new tests that start a builder pass None for it.

Add the plan for local models that can do more than the lean defaults assume: the context window read from the server, an Auto/Lean/Full tool list per AI, honest prompts for builders that can't see pictures, and pictures for models that can. The first line is claimed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A model on this computer or its network was budgeted at 24k tokens whatever it could hold, and the real window was only learned from a refusal, which never came once we trimmed first. Before a backend's first request, and once more after it while the server hasn't said, the agent now asks the server for the window it has loaded: the model list (context_length, max_model_len, meta.n_ctx), llama-swap's upstream route for a loaded model, llama.cpp's /props, LM Studio's loaded_context_length and Ollama's /api/ps. Three quarters of it is a local model's budget; it only lowers a paid service's. Each server is a WindowProbe, so a fork can add one. context_tokens gets a field in Settings -> AI (Auto by default) and `ai context <name> <tokens|auto>` in the CLI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ings The gate skipped every service off this network but OpenRouter, while its comment and the docs said they were asked through their model list. OpenAI-compatible services now are (official OpenAI and Gemini, whose lists say no window, still aren't), and Groq's `context_window` field is read. Adds tests for the hub's set_context_tokens, `ai context` and the Settings field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The llama-swap upstream route and Ollama 0.40.15 were asked live; note Ollama's 4096-token default under the line about setting a server's size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The budget was a bare text box that wrapped away from its label and didn't say what Auto meant. It is now a dropdown on its own line (Auto, 8k to 1M, Custom… for any number, so nothing is capped), and under it a note worked out with the agent loop's own rule from the window the server names: "Auto: 393,216 tokens of the server's 524,288", or "196,608 will be used: the server holds 262,144". `ai context` shows the same window and budget. Strings are in all 13 catalogs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…fault The 150k cap for a paid service's Auto budget was applied before the person's own budget was compared with the window, so a 390k budget became 150k. Also escape a pipe that split a STATUS table row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

llama-swap starts a model for a /props?model= request too, so the llama.cpp probe swapped out the loaded model whenever the list gave no window for an unloaded one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>