Skip to content

fix(llm-mapper): include cache_read tokens in non-streaming Anthropic prompt_tokens - #5866

Open
pcbeingused333 wants to merge 1 commit into
Helicone:mainfrom
pcbeingused333:fix/anthropic-nonstream-prompt-tokens-cache
Open

pcbeingused333 wants to merge 1 commit into
Helicone:mainfrom
pcbeingused333:fix/anthropic-nonstream-prompt-tokens-cache

Conversation

@pcbeingused333

@pcbeingused333 pcbeingused333 commented Nov 6, 2026 •

Copy link
Copy Markdown

Ticket

None, found by code review (streaming and non-streaming converters disagree).

Component/Service

  • Web (Frontend)
  • Jawn (Backend)
  • Worker (Proxy)
  • Bifrost (Marketing)
  • AI Gateway
  • Packages
  • Infrastructure/Docker
  • Documentation

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Performance improvement
  • Refactoring

Deployment Notes

  • No special deployment steps required
  • Database migrations need to run
  • Environment variable changes required
  • Coordination with other teams needed

Context

Cause. The Anthropic→OpenAI converters disagree on usage.prompt_tokens. The streaming one (streamedResponse/toOpenai.ts) reports input_tokens + cache_read_input_tokens, which is the OpenAI convention (cached tokens are part of prompt_tokens). The non-streaming one (response/toOpenai.ts) reported only input_tokens, so with prompt caching the same request showed e.g. prompt_tokens: 4 instead of 51717. The existing non-streaming test even says "should be input_tokens + cache_read_input_tokens" but asserted 4.

Fix. Non-streaming prompt_tokens is now input_tokens + cache_read_input_tokens, matching the streaming path. total_tokens and prompt_tokens_details are unchanged.

Test. Updated the non-streaming cache-read case in packages/__tests__/llm-mapper/anthropic-stream-cache-tokens.test.ts to expect 4 + 51713: red before (received 4), green after; the rest of the Anthropic mapper suites still pass.


This change was prepared, and the tests above were run, with an AI coding assistant (Claude Code).

🤖 Generated with Claude Code


Note

Low Risk
Corrects token accounting in the mapper only; aligns behavior with streaming and avoids misleading usage for cached prompts.

Overview
Non-streaming toOpenAI now sets usage.prompt_tokens to input_tokens + cache_read_input_tokens, matching the streaming converter and OpenAI’s convention (cached prompt bytes count toward prompt_tokens). total_tokens and prompt_tokens_details are unchanged.

The non-streaming cache-hit test now expects 4 + 51713 instead of 4, matching the comment that prompt tokens should include read-cache tokens.

Reviewed by Cursor Bugbot for commit 2cbbe10. Bugbot is set up for automated code reviews on this repo. Configure here.

@vercel

vercel Bot commented Nov 6, 2026

Copy link
Copy Markdown

@pcbeingused333 is attempting to deploy a commit to the Helicone Team on Vercel.

A member of the Team first needs to authorize it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant