From c400e2c4af3b89c89288041e029d47dfd0eaa0b7 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 03:18:30 +0000 Subject: [PATCH 01/15] fix(judges): Track judge harness usage Co-Authored-By: David Cramer --- README.md | 10 ++- docs/architecture.md | 13 ++- docs/github-actions.md | 5 +- packages/core/src/report/metadata.ts | 1 + .../src/content/docs/docs/judges/custom.mdx | 38 +++++++-- packages/github-reporter/src/collect.ts | 75 ++++++++++++++--- packages/github-reporter/src/merge.ts | 13 +++ packages/github-reporter/src/report.test.ts | 34 +++++++- packages/github-reporter/src/summary.ts | 34 +++++++- packages/github-reporter/src/types.ts | 4 + packages/harness-ai-sdk/README.md | 3 +- packages/harness-ai-sdk/src/index.ts | 29 ++++++- packages/harness-openai-agents/README.md | 4 +- .../harness-openai-agents/src/index.test.ts | 34 +++++++- packages/harness-openai-agents/src/index.ts | 11 ++- packages/harness-pi-ai/README.md | 4 +- packages/harness-pi-ai/src/index.ts | 17 +++- packages/vitest-evals/README.md | 12 ++- packages/vitest-evals/src/index.ts | 30 +++++-- .../src/judges/factualityJudge.test.ts | 5 +- packages/vitest-evals/src/judges/index.ts | 2 + .../src/judges/judgeHarness.test.ts | 80 ++++++++++++++++++- .../vitest-evals/src/judges/judgeHarness.ts | 35 +++++--- packages/vitest-evals/src/judges/types.ts | 4 +- packages/vitest-evals/src/reporter.ts | 39 ++++++++- skills/vitest-evals/SPEC.md | 7 +- .../references/judges-and-assertions.md | 13 +++ 27 files changed, 486 insertions(+), 70 deletions(-) diff --git a/README.md b/README.md index 8d865c1..28234ae 100644 --- a/README.md +++ b/README.md @@ -232,8 +232,9 @@ Harness-backed suites stay close to plain Vitest: `ctx.runJudge(...)` from a configured `judgeHarness` - pass scenario-specific judge criteria explicitly to matcher options, or bind suite-wide criteria on the judge instance -- reporter output, replay, usage, tool calls, and spans come from the - normalized run. First-party harnesses attach native spans when +- reporter output, replay, usage, tool calls, and spans come from normalized + runs. App usage and judge usage stay distinguishable; stable token usage from + both contributes to report totals. First-party harnesses attach native spans when provider/runtime data is available; `createHarness(...)` attaches a fallback run span when a custom harness does not return traces itself. Span attributes include typed OpenTelemetry GenAI semantic keys while keeping @@ -247,7 +248,10 @@ matcher options only when a judge calls `ctx.runJudge(...)`. A `judgeHarness` is a dedicated judge-model adapter, so the same factuality judge can be reused across AI SDK, Pi, OpenAI Agents, or custom app harnesses. Automatic inference for explicit matcher calls only applies when suite judges share the same judge -harness instance. +harness instance. Each `ctx.runJudge(...)` call returns judge output and records +the complete normalized judge run under the score's `judgeRuns` field. Custom +judge harnesses should return full runs when usage is available; keep +provider-specific cost estimates under `usage.metadata`. Tool replay is available for opt-in tools in the first-party harnesses. Configure the replay mode and directory globally in Vitest, then opt individual diff --git a/docs/architecture.md b/docs/architecture.md index b1fb7ac..e381c45 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -141,7 +141,11 @@ adapters. Custom judges should use `createJudge("Name", assess)` for stable reporter labels, or `createJudge({ name, judgeHarness, assess })` when the judge should carry a reusable judge-side harness default. `createJudgeHarness(...)` is the shared abstraction for judge-side provider -shims. +shims. Each `ctx.runJudge(...)` call returns the normalized judge output while +core records the complete judge harness run on the resulting score. Multiple +calls from one judge remain separate runs. Reporters aggregate their stable +usage fields alongside application usage while preserving provider-specific +cost estimates under each run's `usage.metadata`. ### `packages/vitest-evals/src/legacy/*` @@ -238,8 +242,11 @@ For each eval test in a harness-backed suite: `usage`, `timings`, `artifacts`, and `errors`. 6. Core stores that run on `task.meta.harness` for the reporter. 7. Automatic suite-level judges run against the normalized run/session pair. -8. The eval test asserts on the same returned result and session. -9. The reporter renders the recorded metadata without re-executing the harness. +8. Each judge-harness invocation is stored under its judge score as `judgeRuns`. +9. Reporters include stable judge usage in totals while keeping it distinguishable + from application usage. +10. The eval test asserts on the same returned result and session. +11. The reporter renders the recorded metadata without re-executing the harness. Explicit `expect(result).toSatisfyJudge(...)` calls use the run's typed output and reuse registered input, metadata, and harness context diff --git a/docs/github-actions.md b/docs/github-actions.md index 0902659..24bd401 100644 --- a/docs/github-actions.md +++ b/docs/github-actions.md @@ -8,7 +8,10 @@ Public docs mirror this guide at [vitest-evals.sentry.dev/docs/github](https://vitest-evals.sentry.dev/docs/github). Use JSON as the eval artifact because it preserves `task.meta.eval` and -`task.meta.harness`; JUnit XML does not carry the full eval metadata. +`task.meta.harness`; JUnit XML does not carry the full eval metadata. Usage +totals include normalized application and judge token usage. Judge usage stays +separate in the collected report, and provider-specific costs remain preserved +under each run's `usage.metadata`. ## Minimal Workflow diff --git a/packages/core/src/report/metadata.ts b/packages/core/src/report/metadata.ts index 09fdc03..163459b 100644 --- a/packages/core/src/report/metadata.ts +++ b/packages/core/src/report/metadata.ts @@ -19,6 +19,7 @@ export const EvalScoreSchema = z .object({ name: z.string().optional(), score: NullableFiniteNumberSchema, + judgeRuns: z.array(HarnessRunSchema).optional(), metadata: JsonObjectSchema.optional(), }) .strict(); diff --git a/packages/docs/src/content/docs/docs/judges/custom.mdx b/packages/docs/src/content/docs/docs/judges/custom.mdx index f527f2c..c9c9fd9 100644 --- a/packages/docs/src/content/docs/docs/judges/custom.mdx +++ b/packages/docs/src/content/docs/docs/judges/custom.mdx @@ -68,12 +68,14 @@ await expect(result).toSatisfyJudge(CapitalJudge, { If a custom judge needs an LLM call, configure a `judgeHarness` on the matcher, the judge object, or the suite, then call `ctx.runJudge(...)` from the judge. -Core curries the current abort signal into that function. Matcher options win -over a judge default, and a judge default wins over the suite default. Explicit -matcher calls can also reuse a single unambiguous judge-level harness from the -suite's automatic judges, but automatic judges do not inherit inferred -harnesses from sibling judges. Leave `judgeHarness` unset for suites that only -use deterministic judges. +Core curries the current abort signal into that function. Each call returns the +judge output to your scoring code and records the underlying normalized judge +run on the score as `judgeRuns`. If one judge calls `runJudge()` more than once, +every run is retained. Matcher options win over a judge default, and a judge +default wins over the suite default. Explicit matcher calls can also reuse a +single unambiguous judge-level harness from the suite's automatic judges, but +automatic judges do not inherit inferred harnesses from sibling judges. Leave +`judgeHarness` unset for suites that only use deterministic judges. ```ts title="evals/judges/rubricJudge.ts" import { createJudge } from "vitest-evals"; @@ -95,6 +97,30 @@ export const RubricJudge = createJudge({ }); ``` +Return a full `HarnessRun` from a custom judge harness when the provider exposes +usage. Stable token fields are included in terminal and GitHub report totals. +Keep provider-specific cost estimates under `usage.metadata`; reports preserve +that metadata but do not treat it as a normalized cost field. + +```ts title="evals/judges/judgeHarness.ts" +const judgeHarness = createJudgeHarness({ + name: "rubric-model", + async run(input) { + const response = await callJudgeModel(input); + return { + output: response.text, + session: response.session, + usage: { + inputTokens: response.usage.inputTokens, + outputTokens: response.usage.outputTokens, + metadata: { costUsd: response.costUsd }, + }, + errors: [], + }; + }, +}); +``` + ## Failure Behavior Custom judges fail when the returned score falls below the active threshold. diff --git a/packages/github-reporter/src/collect.ts b/packages/github-reporter/src/collect.ts index 547df13..ce3f228 100644 --- a/packages/github-reporter/src/collect.ts +++ b/packages/github-reporter/src/collect.ts @@ -1,5 +1,6 @@ import { type HarnessRun, + type UsageSummary as HarnessUsageSummary, type ReportCase, type ToolCall, type TranscriptToolCallEvent, @@ -36,7 +37,8 @@ export function collectEvalReport( const evalScores = cases .map((testCase) => testCase.eval?.avgScore) .filter((score): score is number => isFiniteNumber(score)); - const usage = sumUsage(cases); + const judgeUsage = sumJudgeUsage(cases); + const usage = addUsage(sumHarnessUsage(cases), judgeUsage); const durationMs = workspace.runs[0]?.durationMs; return { @@ -63,6 +65,7 @@ export function collectEvalReport( } : undefined, usage, + judgeUsage, cases, failures, }; @@ -89,6 +92,7 @@ function collectEvalCase(reportCase: ReportCase): EvalCase { thresholdFailed: reportCase.eval.thresholdFailed, output: reportCase.eval.output, scores, + ...(hasJudgeRuns(scores) ? { judgeUsage: sumJudgeRuns(scores) } : {}), } : undefined, harness: reportCase.harness @@ -231,31 +235,76 @@ function stringifyReason(value: unknown) { return typeof value === "string" ? value : stringifyValue(value, 4000); } -function sumUsage(cases: EvalCase[]) { - const usage: Required = { +function emptyUsage(): Required { + return { inputTokens: 0, outputTokens: 0, reasoningTokens: 0, totalTokens: 0, toolCalls: 0, }; +} +function addUsage( + left: Required, + right: Required, +): Required { + return { + inputTokens: left.inputTokens + right.inputTokens, + outputTokens: left.outputTokens + right.outputTokens, + reasoningTokens: left.reasoningTokens + right.reasoningTokens, + totalTokens: left.totalTokens + right.totalTokens, + toolCalls: left.toolCalls + right.toolCalls, + }; +} + +function addRunUsage( + total: Required, + usage: HarnessUsageSummary | undefined, +) { + total.inputTokens += usage?.inputTokens ?? 0; + total.outputTokens += usage?.outputTokens ?? 0; + total.reasoningTokens += usage?.reasoningTokens ?? 0; + total.totalTokens += + usage?.totalTokens ?? + (usage?.inputTokens ?? 0) + + (usage?.outputTokens ?? 0) + + (usage?.reasoningTokens ?? 0); + total.toolCalls += usage?.toolCalls ?? 0; +} + +function sumHarnessUsage(cases: EvalCase[]) { + const usage = emptyUsage(); for (const testCase of cases) { - const caseUsage = testCase.harness?.usage; - usage.inputTokens += caseUsage?.inputTokens ?? 0; - usage.outputTokens += caseUsage?.outputTokens ?? 0; - usage.reasoningTokens += caseUsage?.reasoningTokens ?? 0; - usage.totalTokens += - caseUsage?.totalTokens ?? - (caseUsage?.inputTokens ?? 0) + - (caseUsage?.outputTokens ?? 0) + - (caseUsage?.reasoningTokens ?? 0); - usage.toolCalls += toolCallCount(testCase); + addRunUsage(usage, testCase.harness?.usage); + usage.toolCalls += + toolCallCount(testCase) - (testCase.harness?.usage?.toolCalls ?? 0); } + return usage; +} + +function hasJudgeRuns(scores: EvalScore[]) { + return scores.some((score) => (score.judgeRuns?.length ?? 0) > 0); +} +function sumJudgeRuns(scores: EvalScore[]) { + const usage = emptyUsage(); + for (const score of scores) { + for (const run of score.judgeRuns ?? []) { + addRunUsage(usage, run.usage); + } + } return usage; } +function sumJudgeUsage(cases: EvalCase[]) { + return cases.reduce( + (usage, testCase) => + addUsage(usage, testCase.eval?.judgeUsage ?? emptyUsage()), + emptyUsage(), + ); +} + function toolCallCount(testCase: EvalCase) { const usageToolCalls = testCase.harness?.usage?.toolCalls; if (usageToolCalls !== undefined) { diff --git a/packages/github-reporter/src/merge.ts b/packages/github-reporter/src/merge.ts index 387ce34..460b7d3 100644 --- a/packages/github-reporter/src/merge.ts +++ b/packages/github-reporter/src/merge.ts @@ -44,11 +44,24 @@ export function mergeEvalReports(reports: EvalReport[]): EvalReport { } : undefined, usage: mergeUsage(reports.map((report) => report.usage)), + judgeUsage: mergeUsage( + reports.map((report) => report.judgeUsage ?? emptyUsage()), + ), cases, failures, }; } +function emptyUsage(): Required { + return { + inputTokens: 0, + outputTokens: 0, + reasoningTokens: 0, + totalTokens: 0, + toolCalls: 0, + }; +} + function mergeUsage(usages: Array>) { return { inputTokens: sum(usages, (usage) => usage.inputTokens), diff --git a/packages/github-reporter/src/report.test.ts b/packages/github-reporter/src/report.test.ts index 6758b4e..68b448e 100644 --- a/packages/github-reporter/src/report.test.ts +++ b/packages/github-reporter/src/report.test.ts @@ -63,6 +63,19 @@ const sampleJson: VitestJsonReport = { { name: "StructuredOutputJudge", score: 0.2, + judgeRuns: [ + { + output: "denied", + session: { events: [] }, + usage: { + inputTokens: 80, + outputTokens: 20, + totalTokens: 100, + metadata: { costUsd: 0.02 }, + }, + errors: [], + }, + ], metadata: { rationale: "status mismatch\nExpected status=approved, got status=denied", @@ -162,7 +175,17 @@ describe("collectEvalReport", () => { average: 0.2, minimum: 0.2, }); - expect(report.usage.totalTokens).toBe(1220); + expect(report.usage.totalTokens).toBe(1320); + expect(report.judgeUsage).toEqual({ + inputTokens: 80, + outputTokens: 20, + reasoningTokens: 0, + totalTokens: 100, + toolCalls: 0, + }); + expect( + report.cases[0]?.eval?.scores[0]?.judgeRuns?.[0]?.usage.metadata, + ).toEqual({ costUsd: 0.02 }); expect(report.usage.toolCalls).toBe(2); expect(report.failures[0]).toMatchObject({ displayFile: "apps/demo/evals/refund.eval.ts", @@ -391,7 +414,8 @@ describe("mergeEvalReports", () => { average: 0.5, minimum: 0.2, }); - expect(report.usage.totalTokens).toBe(1520); + expect(report.usage.totalTokens).toBe(1620); + expect(report.judgeUsage?.totalTokens).toBe(100); expect(report.usage.toolCalls).toBe(3); expect(report.cases).toHaveLength(2); expect(report.failures).toHaveLength(1); @@ -902,7 +926,7 @@ describe("renderJobSummary", () => { expect(summary).toContain("| Pass Rate | 0.0% |"); expect(summary).not.toContain("| Tests |"); expect(summary).toContain("| Score | avg 0.20, min 0.20 |"); - expect(summary).not.toContain("| Usage |"); + expect(summary).toContain("| Usage | 1,320 tokens (100 judge), 2 tools |"); expect(summary).toContain("## Scores"); expect(summary.indexOf("## Scores")).toBeLessThan( summary.indexOf("## Results"), @@ -916,7 +940,9 @@ describe("renderJobSummary", () => { expect(details).toContain("```text\nResult\n------"); expect(details).toContain("Case 1. refund agent > rejects fraud"); expect(details).toContain("Location apps/demo/evals/refund.eval.ts:42"); - expect(details).toContain("Usage 1,220 tokens, 2 tools, 4.1s"); + expect(details).toContain( + "Usage 1,320 tokens (100 judge), 2 tools, 4.1s", + ); expect(details).toContain("Reason\n------"); expect(details).toContain("Expected status=approved, got status=denied"); expect(details).toContain("Judge Score"); diff --git a/packages/github-reporter/src/summary.ts b/packages/github-reporter/src/summary.ts index ce70f4e..d479ebe 100644 --- a/packages/github-reporter/src/summary.ts +++ b/packages/github-reporter/src/summary.ts @@ -114,6 +114,11 @@ function renderSummaryTable( rows.push(["Score", formatScoreSummary(report.score)]); } + const usage = formatReportUsage(report); + if (usage) { + rows.push(["Usage", usage]); + } + if (gate?.enforced) { rows.push(["Gate", gate.message]); } @@ -145,6 +150,24 @@ function formatScoreSummary(score: NonNullable) { }`; } +function formatReportUsage(report: EvalReport) { + const parts: string[] = []; + if (report.usage.totalTokens > 0) { + const judgeTokens = report.judgeUsage?.totalTokens ?? 0; + parts.push( + judgeTokens > 0 + ? `${formatNumber(report.usage.totalTokens)} tokens (${formatNumber(judgeTokens)} judge)` + : `${formatNumber(report.usage.totalTokens)} tokens`, + ); + } + if (report.usage.toolCalls > 0) { + parts.push( + `${formatNumber(report.usage.toolCalls)} tool${report.usage.toolCalls === 1 ? "" : "s"}`, + ); + } + return parts.join(", "); +} + function escapeTableCell(value: string) { return value .replace(/\r?\n/g, " ") @@ -388,19 +411,26 @@ function renderAsciiTable(headers: string[], rows: string[][]) { function formatCaseUsage(testCase: EvalCase) { const usage = testCase.harness?.usage; + const judgeUsage = testCase.eval?.judgeUsage; const parts: string[] = []; - const totalTokens = + const appTokens = usage?.totalTokens ?? (usage?.inputTokens ?? 0) + (usage?.outputTokens ?? 0) + (usage?.reasoningTokens ?? 0); + const judgeTokens = judgeUsage?.totalTokens ?? 0; + const totalTokens = appTokens + judgeTokens; const toolCalls = usage?.toolCalls !== undefined ? Math.max(usage.toolCalls, testCase.toolCalls.length) : testCase.toolCalls.length; if (totalTokens > 0) { - parts.push(`${formatNumber(totalTokens)} tokens`); + parts.push( + judgeTokens > 0 + ? `${formatNumber(totalTokens)} tokens (${formatNumber(judgeTokens)} judge)` + : `${formatNumber(totalTokens)} tokens`, + ); } if (toolCalls > 0) { parts.push(`${formatNumber(toolCalls)} tool${toolCalls === 1 ? "" : "s"}`); diff --git a/packages/github-reporter/src/types.ts b/packages/github-reporter/src/types.ts index 33ed5c9..061f52a 100644 --- a/packages/github-reporter/src/types.ts +++ b/packages/github-reporter/src/types.ts @@ -30,6 +30,7 @@ export type EvalCase = { thresholdFailed?: boolean; output?: unknown; scores: EvalScore[]; + judgeUsage?: Required; }; harness?: { name?: string; @@ -67,7 +68,10 @@ export type EvalReport = { average: number; minimum?: number; }; + /** Combined application and judge usage. */ usage: Required; + /** Usage from judge harness runs only. */ + judgeUsage?: Required; cases: EvalCase[]; failures: EvalCase[]; }; diff --git a/packages/harness-ai-sdk/README.md b/packages/harness-ai-sdk/README.md index 53a7f98..edc42a2 100644 --- a/packages/harness-ai-sdk/README.md +++ b/packages/harness-ai-sdk/README.md @@ -87,7 +87,8 @@ const harness = aiSdkHarness({ `run` executes the system under test. Judges are created separately; keep judge prompts and model calls on a judge harness instead of putting them on the app -harness. +harness. `aiSdkJudgeHarness()` returns a normalized judge run, including AI SDK +usage, so reports account for judge tokens separately from app tokens. ```ts import { openai } from "@ai-sdk/openai"; diff --git a/packages/harness-ai-sdk/src/index.ts b/packages/harness-ai-sdk/src/index.ts index 7462c8b..90765bd 100644 --- a/packages/harness-ai-sdk/src/index.ts +++ b/packages/harness-ai-sdk/src/index.ts @@ -166,18 +166,39 @@ async function runAiSdkJudgeHarness( input.responseFormat?.type === "json" && input.responseFormat.schema !== undefined ) { - const { object } = await generateObject({ + const result = await generateObject({ ...requestOptions, schema: jsonSchema( input.responseFormat.schema as Parameters[0], ), }); - return object; + return createAiSdkJudgeRun(input, result, normalizeContent(result.object)); } - const { text } = await generateText(requestOptions); - return text; + const result = await generateText(requestOptions); + return createAiSdkJudgeRun(input, result, result.text); +} + +function createAiSdkJudgeRun( + input: JudgeHarnessInput, + result: unknown, + output: JsonValue | undefined, +): HarnessRun { + const usage = resolveUsage(result); + return { + output, + session: resolveSession( + input.prompt, + result, + output, + new Map(), + [], + undefined, + ), + usage, + errors: [], + }; } function formatAiSdkJudgeSystemPrompt( diff --git a/packages/harness-openai-agents/README.md b/packages/harness-openai-agents/README.md index 536f990..73a5072 100644 --- a/packages/harness-openai-agents/README.md +++ b/packages/harness-openai-agents/README.md @@ -88,7 +88,9 @@ const harness = openaiAgentsHarness({ `run` executes the OpenAI agent under test. Judges are created separately; keep judge prompts on the judge and model calls on a judge harness instead of -putting a judge model call on the app harness. +putting a judge model call on the app harness. `openaiAgentsJudgeHarness()` +returns a normalized judge run, including SDK usage, so reports account for +judge tokens separately from app tokens. ```ts import { openaiAgentsJudgeHarness } from "@vitest-evals/harness-openai-agents"; diff --git a/packages/harness-openai-agents/src/index.test.ts b/packages/harness-openai-agents/src/index.test.ts index ff38ae6..83a4710 100644 --- a/packages/harness-openai-agents/src/index.test.ts +++ b/packages/harness-openai-agents/src/index.test.ts @@ -1,11 +1,12 @@ import { mkdtempSync, readFileSync, rmSync } from "node:fs"; import { join } from "node:path"; -import { Agent, tool } from "@openai/agents"; +import { Agent, type Runner, tool } from "@openai/agents"; import { afterEach, expect, test, vi } from "vitest"; import { describeEval, getHarnessRunFromError, messagesToTranscriptEvents, + runJudgeHarnessRun, spansByKind, toolCalls, } from "vitest-evals"; @@ -15,7 +16,11 @@ import type { JsonValue, NormalizedSession, } from "vitest-evals/harness"; -import { openaiAgentsHarness, type OpenAiAgentsTool } from "./index"; +import { + openaiAgentsHarness, + openaiAgentsJudgeHarness, + type OpenAiAgentsTool, +} from "./index"; type Classification = { label: "bourbon" | "scotch"; @@ -34,6 +39,31 @@ function firstAssistantToolCall(session: NormalizedSession) { return session.events.find((event) => event.type === "tool_call"); } +test("openai judge harness preserves usage", async () => { + const judgeHarness = openaiAgentsJudgeHarness({ + model: "gpt-4.1-mini", + runner: { + run: vi.fn(async () => ({ + finalOutput: "approved", + state: { + usage: { inputTokens: 12, outputTokens: 3, totalTokens: 15 }, + }, + })), + } as unknown as Runner, + }); + + const run = await runJudgeHarnessRun(judgeHarness, { + prompt: "Grade this answer.", + }); + + expect(run.output).toBe("approved"); + expect(run.usage).toMatchObject({ + inputTokens: 12, + outputTokens: 3, + totalTokens: 15, + }); +}); + function createOpenAiToolCallDetails(callId: string, name = "lookupBottle") { return { toolCall: { diff --git a/packages/harness-openai-agents/src/index.ts b/packages/harness-openai-agents/src/index.ts index ef196bd..15e2445 100644 --- a/packages/harness-openai-agents/src/index.ts +++ b/packages/harness-openai-agents/src/index.ts @@ -313,8 +313,17 @@ export function openaiAgentsJudgeHarness( ...(outputType ? { outputType } : {}), }); const result = await runner.run(agent, prompt, { signal }); + const output = normalizeContent(resolveOpenAiAgentsJudgeOutput(result)); + const usage = resolveUsage(result); - return resolveOpenAiAgentsJudgeOutput(result); + return { + output, + session: resolveSession(prompt, result, output, usage, { + runtimeEvents: [], + }), + usage, + errors: [], + }; }, }); } diff --git a/packages/harness-pi-ai/README.md b/packages/harness-pi-ai/README.md index bb7eee8..f049c4e 100644 --- a/packages/harness-pi-ai/README.md +++ b/packages/harness-pi-ai/README.md @@ -43,7 +43,9 @@ describeEval("refund agent", { harness }, (it) => { `run` executes the Pi agent under test. Judges are created separately; keep judge prompts on the judge and model calls on a judge harness instead of -putting a judge model call on the app harness. +putting a judge model call on the app harness. `piAiJudgeHarness()` returns a +normalized judge run, including Pi usage, so reports account for judge tokens +separately from app tokens. ```ts import { getModel } from "@mariozechner/pi-ai"; diff --git a/packages/harness-pi-ai/src/index.ts b/packages/harness-pi-ai/src/index.ts index d8676d5..1f9d963 100644 --- a/packages/harness-pi-ai/src/index.ts +++ b/packages/harness-pi-ai/src/index.ts @@ -151,7 +151,22 @@ export function piAiJudgeHarness( }, ); - return resolvePiAiJudgeText(message); + const output = resolvePiAiJudgeText(message); + const usage = resolveUsage(message, 0); + return { + output, + session: resolveSession( + message, + [ + { type: "message", role: "user", content: prompt }, + { type: "message", role: "assistant", content: output }, + ], + output, + usage, + ), + usage, + errors: [], + }; }, }); } diff --git a/packages/vitest-evals/README.md b/packages/vitest-evals/README.md index 21484f1..e7e915d 100644 --- a/packages/vitest-evals/README.md +++ b/packages/vitest-evals/README.md @@ -368,9 +368,11 @@ projections through helpers such as `toolCalls(result)`, `userMessages(result)`, of manually walking provider payloads. Return a full `HarnessRun` only when you need exact canonical `session.events`, trace, or usage control. -Provider setup and rubric parsing stay in your judge. The core -package only requires the judge to return a `JudgeResult` with a score and -optional metadata. +Provider setup and rubric parsing stay in your judge. The core package only +requires the judge to return a `JudgeResult` with a score and optional metadata. +Each `ctx.runJudge(...)` call also records its complete normalized judge run +under that score's `judgeRuns` field. Stable judge usage contributes to report +totals; provider-specific cost estimates remain under `usage.metadata`. Automatic suite-level judges are a good fit when every `run(...)` should get the same scoring. For cases where only some runs need an LLM judge, keep the @@ -484,7 +486,9 @@ judges, but automatic judges do not inherit inferred harnesses from sibling judges. That inference requires those judges to share the same judge harness instance. Leave `judgeHarness` unset for suites that only use deterministic judges. Calling `harness.run(...)` from a judge executes the application again, -so use that only when a second run is intentional. +so use that only when a second run is intentional. Custom judge harnesses +should return a full `HarnessRun` when provider usage is available. Multiple +`runJudge(...)` calls are retained as separate judge runs. For an `EvalHarnessRun` returned by fixture `run(...)`, `toSatisfyJudge(...)` uses the run's typed `output` and reuses the registered diff --git a/packages/vitest-evals/src/index.ts b/packages/vitest-evals/src/index.ts index 21c5cad..97891f0 100644 --- a/packages/vitest-evals/src/index.ts +++ b/packages/vitest-evals/src/index.ts @@ -29,7 +29,7 @@ import type { JudgeOptions, JudgeResult, } from "./judges/types"; -import type { JudgeHarness } from "./judges/judgeHarness"; +import type { JudgeHarness, JudgeHarnessRun } from "./judges/judgeHarness"; import { createRunJudge } from "./judges/judgeHarness"; import { wrapText } from "./wrapText"; @@ -449,14 +449,17 @@ expect.extend({ ) { const { threshold = 1.0, ...context } = (options ?? {}) as JudgeAssertionOptions; + const judgeRuns: JudgeHarnessRun[] = []; const judgeOptions = buildJudgeAssertionOptions( received, judge, context, isEvalTaskLike(this.task) ? this.task : undefined, + judgeRuns, ); - const result = await judge.assess(judgeOptions); + const assessed = await judge.assess(judgeOptions); + const result = attachJudgeRuns(assessed, judgeRuns); const score = result.score ?? 0; const pass = threshold === null ? true : score >= threshold; @@ -597,10 +600,12 @@ async function applyAutomaticJudges< ) { const runToolCalls = toolCalls(run.session); const scores = await Promise.all( - judges.map((judge) => { + judges.map(async (judge) => { + const judgeRuns: JudgeHarnessRun[] = []; const runJudge = createRunJudge( resolveJudgeHarnessForJudge(judge, judgeHarness), signal, + (judgeRun) => judgeRuns.push(judgeRun), ); const judgeOptions = { input, @@ -613,7 +618,8 @@ async function applyAutomaticJudges< runJudge, } as unknown as JudgeContext; - return Promise.resolve(judge.assess(judgeOptions)); + const result = await judge.assess(judgeOptions); + return attachJudgeRuns(result, judgeRuns); }), ); @@ -762,6 +768,13 @@ function formatJudgeTextOutput(run: HarnessRun) { : JSON.stringify(assistantOutput); } +function attachJudgeRuns( + result: JudgeResult, + judgeRuns: JudgeHarnessRun[], +): JudgeResult { + return judgeRuns.length > 0 ? { ...result, judgeRuns } : result; +} + function buildJudgeAssertionOptions< TJudgeOptions extends JudgeContext = JudgeContext, >( @@ -769,6 +782,7 @@ function buildJudgeAssertionOptions< judge: Judge, options: Omit, "threshold">, task?: EvalTaskLike, + judgeRuns: JudgeHarnessRun[] = [], ): TJudgeOptions { const registeredContext = resolveRegisteredJudgeRunContext( received, @@ -779,7 +793,11 @@ function buildJudgeAssertionOptions< const judgeHarness = options.judgeHarness ?? resolveJudgeHarnessForJudge(judge, registeredContext?.judgeHarness); - const runJudge = createRunJudge(judgeHarness, registeredContext?.signal); + const runJudge = createRunJudge( + judgeHarness, + registeredContext?.signal, + (judgeRun) => judgeRuns.push(judgeRun), + ); const signal = registeredContext?.signal; const input = options.input ?? @@ -1265,11 +1283,13 @@ export { type FactualityJudgeVerdict, createJudgeHarness, runJudgeHarness, + runJudgeHarnessRun, type CreateJudgeHarnessOptions, type CreateJudgeHarnessRunOptions, type JudgeHarness, type JudgeHarnessInput, type JudgeHarnessOutput, + type JudgeHarnessRun, type RunJudge, type RunJudgeOptions, StructuredOutputJudge, diff --git a/packages/vitest-evals/src/judges/factualityJudge.test.ts b/packages/vitest-evals/src/judges/factualityJudge.test.ts index bd763c7..ae77e68 100644 --- a/packages/vitest-evals/src/judges/factualityJudge.test.ts +++ b/packages/vitest-evals/src/judges/factualityJudge.test.ts @@ -115,15 +115,16 @@ describeEval( }), ); expect(task.meta.eval?.scores).toEqual([ - { + expect.objectContaining({ name: "FactualityJudge", score: 1, metadata: { rationale: "The submitted answer matches the expert answer.", choice: "C", }, - }, + }), ]); + expect(task.meta.eval?.scores[0]?.judgeRuns).toHaveLength(1); }); }, ); diff --git a/packages/vitest-evals/src/judges/index.ts b/packages/vitest-evals/src/judges/index.ts index 007b92a..337fb02 100644 --- a/packages/vitest-evals/src/judges/index.ts +++ b/packages/vitest-evals/src/judges/index.ts @@ -11,11 +11,13 @@ export { export { createJudgeHarness, runJudgeHarness, + runJudgeHarnessRun, type CreateJudgeHarnessOptions, type CreateJudgeHarnessRunOptions, type JudgeHarness, type JudgeHarnessInput, type JudgeHarnessOutput, + type JudgeHarnessRun, type RunJudge, type RunJudgeOptions, } from "./judgeHarness"; diff --git a/packages/vitest-evals/src/judges/judgeHarness.test.ts b/packages/vitest-evals/src/judges/judgeHarness.test.ts index b5a34e1..58a518c 100644 --- a/packages/vitest-evals/src/judges/judgeHarness.test.ts +++ b/packages/vitest-evals/src/judges/judgeHarness.test.ts @@ -1,5 +1,11 @@ import { expect, test } from "vitest"; -import { createJudgeHarness, runJudgeHarness } from "./judgeHarness"; +import { createHarness } from "../harness"; +import { createJudge, describeEval } from "../index"; +import { + createJudgeHarness, + runJudgeHarness, + runJudgeHarnessRun, +} from "./judgeHarness"; test("runJudgeHarness preserves null output values", async () => { const judgeHarness = createJudgeHarness({ @@ -39,6 +45,78 @@ test("runJudgeHarness falls back to assistant content when output is missing", a expect(result).toBe('{"choice":"C"}'); }); +test("runJudgeHarnessRun preserves normalized usage", async () => { + const judgeHarness = createJudgeHarness({ + run: async () => ({ + output: "approved", + session: { events: [] }, + usage: { + inputTokens: 12, + outputTokens: 3, + totalTokens: 15, + metadata: { costUsd: 0.02 }, + }, + errors: [], + }), + }); + + const run = await runJudgeHarnessRun(judgeHarness, { + prompt: "Return JSON.", + }); + + expect(run.usage).toEqual({ + inputTokens: 12, + outputTokens: 3, + totalTokens: 15, + metadata: { costUsd: 0.02 }, + }); +}); + +const appHarness = createHarness({ + name: "app", + run: async ({ input }: { input: string }) => ({ + output: input, + messages: [ + { role: "user", content: input }, + { role: "assistant", content: input }, + ], + }), +}); +const usageJudgeHarness = createJudgeHarness({ + run: async () => ({ + output: "approved", + session: { events: [] }, + usage: { inputTokens: 12, outputTokens: 3, totalTokens: 15 }, + errors: [], + }), +}); +const usageJudge = createJudge("UsageJudge", async ({ runJudge }) => { + await runJudge?.({ prompt: "Grade this." }); + await runJudge?.({ prompt: "Check it again." }); + return { score: 1 }; +}); + +describeEval( + "judge usage", + { + harness: appHarness, + judgeHarness: usageJudgeHarness, + judges: [usageJudge], + }, + (it) => { + it("records every judge harness run", async ({ run, task }) => { + await run("answer"); + + expect(task.meta.eval?.scores?.[0]?.judgeRuns).toHaveLength(2); + expect(task.meta.eval?.scores?.[0]?.judgeRuns?.[0]?.usage).toEqual({ + inputTokens: 12, + outputTokens: 3, + totalTokens: 15, + }); + }); + }, +); + test("runJudgeHarness preserves structured values with an output field", async () => { const judgeHarness = createJudgeHarness({ run: async () => ({ diff --git a/packages/vitest-evals/src/judges/judgeHarness.ts b/packages/vitest-evals/src/judges/judgeHarness.ts index 6086f1a..d2e74bf 100644 --- a/packages/vitest-evals/src/judges/judgeHarness.ts +++ b/packages/vitest-evals/src/judges/judgeHarness.ts @@ -65,6 +65,9 @@ export type RunJudgeOptions = { signal?: AbortSignal; }; +/** Full normalized run produced by one judge-model invocation. */ +export type JudgeHarnessRun = HarnessRun; + /** * Curried judge-harness runner available inside `JudgeContext`. * @@ -153,33 +156,43 @@ export async function runJudgeHarness( input: JudgeHarnessInput, options: RunJudgeOptions = {}, ): Promise { + const run = await runJudgeHarnessRun(judgeHarness, input, options); + return resolveJudgeHarnessOutput(run); +} + +/** Runs a judge harness and returns its complete normalized run. */ +export async function runJudgeHarnessRun( + judgeHarness: JudgeHarness, + input: JudgeHarnessInput, + options: RunJudgeOptions = {}, +): Promise { const artifacts: HarnessContext["artifacts"] = {}; - const run = await judgeHarness.run(input, { + return judgeHarness.run(input, { signal: options.signal, artifacts, setArtifact: (name, value) => { artifacts[name] = value; }, }); - - return run.output !== undefined - ? run.output - : resolveJudgeHarnessAssistantOutput(run); } /** Binds a judge harness to the current eval run context. */ export function createRunJudge( judgeHarness: JudgeHarness | undefined, signal?: AbortSignal, + onRun?: (run: JudgeHarnessRun) => void, ): RunJudge | undefined { if (!judgeHarness) { return undefined; } - return (input, options) => - runJudgeHarness(judgeHarness, input, { + return async (input, options) => { + const run = await runJudgeHarnessRun(judgeHarness, input, { signal: options?.signal ?? signal, }); + onRun?.(run); + return resolveJudgeHarnessOutput(run); + }; } function normalizeJudgeHarnessResult( @@ -233,8 +246,8 @@ function createJudgeHarnessMessages( ]; } -function resolveJudgeHarnessAssistantOutput( - run: HarnessRun, -): JudgeHarnessOutput { - return latestAssistantMessageContent(run.session) ?? ""; +function resolveJudgeHarnessOutput(run: JudgeHarnessRun): JudgeHarnessOutput { + return run.output !== undefined + ? run.output + : (latestAssistantMessageContent(run.session) ?? ""); } diff --git a/packages/vitest-evals/src/judges/types.ts b/packages/vitest-evals/src/judges/types.ts index e881693..3e4c9d4 100644 --- a/packages/vitest-evals/src/judges/types.ts +++ b/packages/vitest-evals/src/judges/types.ts @@ -1,5 +1,5 @@ import type { Harness, HarnessRun, JsonValue, ToolCall } from "../harness"; -import type { JudgeHarness, RunJudge } from "./judgeHarness"; +import type { JudgeHarness, JudgeHarnessRun, RunJudge } from "./judgeHarness"; /** * Score payload returned by a judge. @@ -17,6 +17,8 @@ import type { JudgeHarness, RunJudge } from "./judgeHarness"; export type JudgeResult = { /** Numeric score. `null` records an intentionally unscored result. */ score: number | null; + /** Normalized judge-model runs recorded while producing this score. */ + judgeRuns?: JudgeHarnessRun[]; /** JSON-like judge metadata shown by assertions and reporters. */ metadata?: { /** Human-readable explanation for the score. */ diff --git a/packages/vitest-evals/src/reporter.ts b/packages/vitest-evals/src/reporter.ts index 9bb6552..13307e5 100644 --- a/packages/vitest-evals/src/reporter.ts +++ b/packages/vitest-evals/src/reporter.ts @@ -300,18 +300,55 @@ export default class DefaultEvalReporter extends VerboseReporter { scores: Array<{ name?: string; score?: number | null; + judgeRuns?: Array<{ + usage?: { + inputTokens?: number; + outputTokens?: number; + reasoningTokens?: number; + totalTokens?: number; + }; + }>; }>, ) { for (const score of scores) { this.log( this.formatDetailLine( "score", - `${score.name || "Unknown"} ${this.formatScore(score.score ?? 0)}`, + [ + `${score.name || "Unknown"} ${this.formatScore(score.score ?? 0)}`, + this.formatJudgeUsage(score.judgeRuns), + ] + .filter(Boolean) + .join(" · "), ), ); } } + private formatJudgeUsage( + runs: + | Array<{ + usage?: { + inputTokens?: number; + outputTokens?: number; + reasoningTokens?: number; + totalTokens?: number; + }; + }> + | undefined, + ) { + const totalTokens = (runs ?? []).reduce( + (total, run) => + total + + (run.usage?.totalTokens ?? + (run.usage?.inputTokens ?? 0) + + (run.usage?.outputTokens ?? 0) + + (run.usage?.reasoningTokens ?? 0)), + 0, + ); + return totalTokens > 0 ? `${totalTokens} judge tok` : undefined; + } + private formatFieldLabel( label: | "tool" diff --git a/skills/vitest-evals/SPEC.md b/skills/vitest-evals/SPEC.md index 67af2fc..36f622c 100644 --- a/skills/vitest-evals/SPEC.md +++ b/skills/vitest-evals/SPEC.md @@ -33,8 +33,9 @@ Out of scope: - Required first actions: inspect the touched code, choose the runtime target, open the matching reference. - Required outputs: implementation or review guidance that uses the harness-backed API only, plus targeted verification. - Non-negotiable constraints: normalized run data remains JSON-serializable, - suite tests call `run(...)` explicitly, and judge model calls stay on judges - or judge-side helpers rather than the app harness. + suite tests call `run(...)` explicitly, judge model calls stay on judges or + judge-side helpers rather than the app harness, and model-backed judge runs + retain normalized usage for reports. - Expected bundled files loaded at runtime: `SKILL.md` first, then one or more focused files under `references/`. ## Source And Evidence Model @@ -78,6 +79,8 @@ Data that must not be stored: - The skill does not automate installation; it is installable as a self-contained directory. - Provider-specific model credentials and environment setup remain application concerns. - API details must be refreshed when package options, peer ranges, or reporter behavior change. +- Provider-specific cost remains metadata; the skill does not define a stable + cost field or pricing model. ## Maintenance Notes diff --git a/skills/vitest-evals/references/judges-and-assertions.md b/skills/vitest-evals/references/judges-and-assertions.md index 8aeb2bd..ccfb886 100644 --- a/skills/vitest-evals/references/judges-and-assertions.md +++ b/skills/vitest-evals/references/judges-and-assertions.md @@ -71,6 +71,17 @@ const RefundRubricJudge = createJudge< - Normalized sessions infer output from the latest assistant message content. - Calling `ctx.harness.run(...)` inside a judge executes the app again; only do this when the judge intentionally needs a second run. +## Judge Usage + +- A judge harness is separate from the app harness. Return a full `HarnessRun` + when the provider exposes usage. +- `ctx.runJudge(...)` returns the judge output, while vitest-evals records the + complete normalized run under the score's `judgeRuns` field. +- Multiple `runJudge(...)` calls from one judge are retained separately. +- Stable token fields contribute to report totals. Provider-specific cost + estimates stay under `usage.metadata` and are preserved without being + normalized. + ## Review Checklist - Custom judges should usually use `createJudge("Name", assess)`. @@ -78,3 +89,5 @@ const RefundRubricJudge = createJudge< - Rationale or parsed judge output is placed under `metadata`. - LLM-backed judges provide the judge prompt/rubric text. Shared provider setup belongs in a judge-side helper, not on the app harness. +- Model-backed judge harnesses retain normalized usage instead of returning only + provider text when usage is available. From 6ad85765bc6274a4b838f388e7c8fa6ca7b8092f Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 03:20:25 +0000 Subject: [PATCH 02/15] test(reporter): Expect aggregate usage summary --- .github/workflows/github-reporter-smoke.yml | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/.github/workflows/github-reporter-smoke.yml b/.github/workflows/github-reporter-smoke.yml index c81c39f..a79cf88 100644 --- a/.github/workflows/github-reporter-smoke.yml +++ b/.github/workflows/github-reporter-smoke.yml @@ -373,10 +373,7 @@ jobs: echo "test totals row should not be shown" exit 1 fi - if grep -F "| Usage |" "$GITHUB_STEP_SUMMARY"; then - echo "top-level usage row should not be shown" - exit 1 - fi + grep -F "| Usage | 1,302 tokens, 9 tools |" "$GITHUB_STEP_SUMMARY" grep -F "::error file=apps/demo/evals/smoke.eval.ts,line=12,col=5,title=vitest-evals::refund eval > rejects fraud - score 0.10 - SmokeJudge - expected approved but got denied" tmp/github-reporter-smoke/stdout.txt grep -F "::error file=apps/demo/evals/smoke.eval.ts,line=28,col=5,title=vitest-evals::refund eval > returns wrong refund amount - score 0.35 - AmountJudge - expected amount=5000, got amount=499" tmp/github-reporter-smoke/stdout.txt grep -F "::warning::GitHub Check Run skipped: missing GITHUB_TOKEN" tmp/github-reporter-smoke/stdout.txt From 7d7e3ad7e25c7a0d70ad5dc820fb4229e308341a Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 03:36:39 +0000 Subject: [PATCH 03/15] refactor(judges): simplify usage tracking Co-Authored-By: David Cramer --- .github/workflows/github-reporter-smoke.yml | 5 +- README.md | 10 ++-- docs/architecture.md | 7 ++- docs/github-actions.md | 5 +- .../src/content/docs/docs/judges/custom.mdx | 9 ++-- packages/github-reporter/src/collect.ts | 45 +++--------------- packages/github-reporter/src/merge.ts | 13 ----- packages/github-reporter/src/report.test.ts | 14 +----- packages/github-reporter/src/summary.ts | 34 +------------- packages/github-reporter/src/types.ts | 4 -- packages/harness-ai-sdk/README.md | 3 +- .../harness-ai-sdk/src/judgeHarness.test.ts | 6 +++ packages/harness-openai-agents/README.md | 4 +- .../harness-openai-agents/src/index.test.ts | 8 ++-- packages/harness-pi-ai/README.md | 4 +- .../harness-pi-ai/src/judgeHarness.test.ts | 4 ++ packages/vitest-evals/src/index.ts | 22 +++++---- packages/vitest-evals/src/judges/index.ts | 2 - .../src/judges/judgeHarness.test.ts | 33 +------------ .../vitest-evals/src/judges/judgeHarness.ts | 47 +++++++++---------- packages/vitest-evals/src/judges/types.ts | 4 +- packages/vitest-evals/src/reporter.ts | 39 +-------------- skills/vitest-evals/SPEC.md | 5 +- 23 files changed, 82 insertions(+), 245 deletions(-) diff --git a/.github/workflows/github-reporter-smoke.yml b/.github/workflows/github-reporter-smoke.yml index a79cf88..c81c39f 100644 --- a/.github/workflows/github-reporter-smoke.yml +++ b/.github/workflows/github-reporter-smoke.yml @@ -373,7 +373,10 @@ jobs: echo "test totals row should not be shown" exit 1 fi - grep -F "| Usage | 1,302 tokens, 9 tools |" "$GITHUB_STEP_SUMMARY" + if grep -F "| Usage |" "$GITHUB_STEP_SUMMARY"; then + echo "top-level usage row should not be shown" + exit 1 + fi grep -F "::error file=apps/demo/evals/smoke.eval.ts,line=12,col=5,title=vitest-evals::refund eval > rejects fraud - score 0.10 - SmokeJudge - expected approved but got denied" tmp/github-reporter-smoke/stdout.txt grep -F "::error file=apps/demo/evals/smoke.eval.ts,line=28,col=5,title=vitest-evals::refund eval > returns wrong refund amount - score 0.35 - AmountJudge - expected amount=5000, got amount=499" tmp/github-reporter-smoke/stdout.txt grep -F "::warning::GitHub Check Run skipped: missing GITHUB_TOKEN" tmp/github-reporter-smoke/stdout.txt diff --git a/README.md b/README.md index 28234ae..30b22a9 100644 --- a/README.md +++ b/README.md @@ -233,9 +233,8 @@ Harness-backed suites stay close to plain Vitest: - pass scenario-specific judge criteria explicitly to matcher options, or bind suite-wide criteria on the judge instance - reporter output, replay, usage, tool calls, and spans come from normalized - runs. App usage and judge usage stay distinguishable; stable token usage from - both contributes to report totals. First-party harnesses attach native spans when - provider/runtime data is available; `createHarness(...)` attaches a fallback + runs, including model-backed judge runs. First-party harnesses attach native + spans when provider/runtime data is available; `createHarness(...)` attaches a fallback run span when a custom harness does not return traces itself. Span attributes include typed OpenTelemetry GenAI semantic keys while keeping provider-specific keys JSON-safe. @@ -248,10 +247,7 @@ matcher options only when a judge calls `ctx.runJudge(...)`. A `judgeHarness` is a dedicated judge-model adapter, so the same factuality judge can be reused across AI SDK, Pi, OpenAI Agents, or custom app harnesses. Automatic inference for explicit matcher calls only applies when suite judges share the same judge -harness instance. Each `ctx.runJudge(...)` call returns judge output and records -the complete normalized judge run under the score's `judgeRuns` field. Custom -judge harnesses should return full runs when usage is available; keep -provider-specific cost estimates under `usage.metadata`. +harness instance. Tool replay is available for opt-in tools in the first-party harnesses. Configure the replay mode and directory globally in Vitest, then opt individual diff --git a/docs/architecture.md b/docs/architecture.md index e381c45..b71ed5f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -243,10 +243,9 @@ For each eval test in a harness-backed suite: 6. Core stores that run on `task.meta.harness` for the reporter. 7. Automatic suite-level judges run against the normalized run/session pair. 8. Each judge-harness invocation is stored under its judge score as `judgeRuns`. -9. Reporters include stable judge usage in totals while keeping it distinguishable - from application usage. -10. The eval test asserts on the same returned result and session. -11. The reporter renders the recorded metadata without re-executing the harness. +9. The eval test asserts on the same returned result and session. +10. The reporter renders the recorded metadata and includes judge usage in totals + without re-executing either harness. Explicit `expect(result).toSatisfyJudge(...)` calls use the run's typed output and reuse registered input, metadata, and harness context diff --git a/docs/github-actions.md b/docs/github-actions.md index 24bd401..f2ec471 100644 --- a/docs/github-actions.md +++ b/docs/github-actions.md @@ -9,9 +9,8 @@ Public docs mirror this guide at Use JSON as the eval artifact because it preserves `task.meta.eval` and `task.meta.harness`; JUnit XML does not carry the full eval metadata. Usage -totals include normalized application and judge token usage. Judge usage stays -separate in the collected report, and provider-specific costs remain preserved -under each run's `usage.metadata`. +totals include normalized application and judge token usage. Judge runs remain +attached to their scores, including provider-specific `usage.metadata`. ## Minimal Workflow diff --git a/packages/docs/src/content/docs/docs/judges/custom.mdx b/packages/docs/src/content/docs/docs/judges/custom.mdx index c9c9fd9..f337601 100644 --- a/packages/docs/src/content/docs/docs/judges/custom.mdx +++ b/packages/docs/src/content/docs/docs/judges/custom.mdx @@ -69,9 +69,8 @@ await expect(result).toSatisfyJudge(CapitalJudge, { If a custom judge needs an LLM call, configure a `judgeHarness` on the matcher, the judge object, or the suite, then call `ctx.runJudge(...)` from the judge. Core curries the current abort signal into that function. Each call returns the -judge output to your scoring code and records the underlying normalized judge -run on the score as `judgeRuns`. If one judge calls `runJudge()` more than once, -every run is retained. Matcher options win over a judge default, and a judge +judge output and records the normalized judge run in report metadata. Multiple +calls remain separate runs. Matcher options win over a judge default, and a judge default wins over the suite default. Explicit matcher calls can also reuse a single unambiguous judge-level harness from the suite's automatic judges, but automatic judges do not inherit inferred harnesses from sibling judges. Leave @@ -98,8 +97,8 @@ export const RubricJudge = createJudge({ ``` Return a full `HarnessRun` from a custom judge harness when the provider exposes -usage. Stable token fields are included in terminal and GitHub report totals. -Keep provider-specific cost estimates under `usage.metadata`; reports preserve +usage. Stable token fields are included in GitHub report totals. Keep +provider-specific cost estimates under `usage.metadata`; reports preserve that metadata but do not treat it as a normalized cost field. ```ts title="evals/judges/judgeHarness.ts" diff --git a/packages/github-reporter/src/collect.ts b/packages/github-reporter/src/collect.ts index ce3f228..89ae93e 100644 --- a/packages/github-reporter/src/collect.ts +++ b/packages/github-reporter/src/collect.ts @@ -37,8 +37,7 @@ export function collectEvalReport( const evalScores = cases .map((testCase) => testCase.eval?.avgScore) .filter((score): score is number => isFiniteNumber(score)); - const judgeUsage = sumJudgeUsage(cases); - const usage = addUsage(sumHarnessUsage(cases), judgeUsage); + const usage = sumUsage(cases); const durationMs = workspace.runs[0]?.durationMs; return { @@ -65,7 +64,6 @@ export function collectEvalReport( } : undefined, usage, - judgeUsage, cases, failures, }; @@ -92,7 +90,6 @@ function collectEvalCase(reportCase: ReportCase): EvalCase { thresholdFailed: reportCase.eval.thresholdFailed, output: reportCase.eval.output, scores, - ...(hasJudgeRuns(scores) ? { judgeUsage: sumJudgeRuns(scores) } : {}), } : undefined, harness: reportCase.harness @@ -245,19 +242,6 @@ function emptyUsage(): Required { }; } -function addUsage( - left: Required, - right: Required, -): Required { - return { - inputTokens: left.inputTokens + right.inputTokens, - outputTokens: left.outputTokens + right.outputTokens, - reasoningTokens: left.reasoningTokens + right.reasoningTokens, - totalTokens: left.totalTokens + right.totalTokens, - toolCalls: left.toolCalls + right.toolCalls, - }; -} - function addRunUsage( total: Required, usage: HarnessUsageSummary | undefined, @@ -273,38 +257,21 @@ function addRunUsage( total.toolCalls += usage?.toolCalls ?? 0; } -function sumHarnessUsage(cases: EvalCase[]) { +function sumUsage(cases: EvalCase[]) { const usage = emptyUsage(); for (const testCase of cases) { addRunUsage(usage, testCase.harness?.usage); usage.toolCalls += toolCallCount(testCase) - (testCase.harness?.usage?.toolCalls ?? 0); - } - return usage; -} - -function hasJudgeRuns(scores: EvalScore[]) { - return scores.some((score) => (score.judgeRuns?.length ?? 0) > 0); -} - -function sumJudgeRuns(scores: EvalScore[]) { - const usage = emptyUsage(); - for (const score of scores) { - for (const run of score.judgeRuns ?? []) { - addRunUsage(usage, run.usage); + for (const score of testCase.eval?.scores ?? []) { + for (const run of score.judgeRuns ?? []) { + addRunUsage(usage, run.usage); + } } } return usage; } -function sumJudgeUsage(cases: EvalCase[]) { - return cases.reduce( - (usage, testCase) => - addUsage(usage, testCase.eval?.judgeUsage ?? emptyUsage()), - emptyUsage(), - ); -} - function toolCallCount(testCase: EvalCase) { const usageToolCalls = testCase.harness?.usage?.toolCalls; if (usageToolCalls !== undefined) { diff --git a/packages/github-reporter/src/merge.ts b/packages/github-reporter/src/merge.ts index 460b7d3..387ce34 100644 --- a/packages/github-reporter/src/merge.ts +++ b/packages/github-reporter/src/merge.ts @@ -44,24 +44,11 @@ export function mergeEvalReports(reports: EvalReport[]): EvalReport { } : undefined, usage: mergeUsage(reports.map((report) => report.usage)), - judgeUsage: mergeUsage( - reports.map((report) => report.judgeUsage ?? emptyUsage()), - ), cases, failures, }; } -function emptyUsage(): Required { - return { - inputTokens: 0, - outputTokens: 0, - reasoningTokens: 0, - totalTokens: 0, - toolCalls: 0, - }; -} - function mergeUsage(usages: Array>) { return { inputTokens: sum(usages, (usage) => usage.inputTokens), diff --git a/packages/github-reporter/src/report.test.ts b/packages/github-reporter/src/report.test.ts index 68b448e..bb19a69 100644 --- a/packages/github-reporter/src/report.test.ts +++ b/packages/github-reporter/src/report.test.ts @@ -176,13 +176,6 @@ describe("collectEvalReport", () => { minimum: 0.2, }); expect(report.usage.totalTokens).toBe(1320); - expect(report.judgeUsage).toEqual({ - inputTokens: 80, - outputTokens: 20, - reasoningTokens: 0, - totalTokens: 100, - toolCalls: 0, - }); expect( report.cases[0]?.eval?.scores[0]?.judgeRuns?.[0]?.usage.metadata, ).toEqual({ costUsd: 0.02 }); @@ -415,7 +408,6 @@ describe("mergeEvalReports", () => { minimum: 0.2, }); expect(report.usage.totalTokens).toBe(1620); - expect(report.judgeUsage?.totalTokens).toBe(100); expect(report.usage.toolCalls).toBe(3); expect(report.cases).toHaveLength(2); expect(report.failures).toHaveLength(1); @@ -926,7 +918,7 @@ describe("renderJobSummary", () => { expect(summary).toContain("| Pass Rate | 0.0% |"); expect(summary).not.toContain("| Tests |"); expect(summary).toContain("| Score | avg 0.20, min 0.20 |"); - expect(summary).toContain("| Usage | 1,320 tokens (100 judge), 2 tools |"); + expect(summary).not.toContain("| Usage |"); expect(summary).toContain("## Scores"); expect(summary.indexOf("## Scores")).toBeLessThan( summary.indexOf("## Results"), @@ -940,9 +932,7 @@ describe("renderJobSummary", () => { expect(details).toContain("```text\nResult\n------"); expect(details).toContain("Case 1. refund agent > rejects fraud"); expect(details).toContain("Location apps/demo/evals/refund.eval.ts:42"); - expect(details).toContain( - "Usage 1,320 tokens (100 judge), 2 tools, 4.1s", - ); + expect(details).toContain("Usage 1,220 tokens, 2 tools, 4.1s"); expect(details).toContain("Reason\n------"); expect(details).toContain("Expected status=approved, got status=denied"); expect(details).toContain("Judge Score"); diff --git a/packages/github-reporter/src/summary.ts b/packages/github-reporter/src/summary.ts index d479ebe..ce70f4e 100644 --- a/packages/github-reporter/src/summary.ts +++ b/packages/github-reporter/src/summary.ts @@ -114,11 +114,6 @@ function renderSummaryTable( rows.push(["Score", formatScoreSummary(report.score)]); } - const usage = formatReportUsage(report); - if (usage) { - rows.push(["Usage", usage]); - } - if (gate?.enforced) { rows.push(["Gate", gate.message]); } @@ -150,24 +145,6 @@ function formatScoreSummary(score: NonNullable) { }`; } -function formatReportUsage(report: EvalReport) { - const parts: string[] = []; - if (report.usage.totalTokens > 0) { - const judgeTokens = report.judgeUsage?.totalTokens ?? 0; - parts.push( - judgeTokens > 0 - ? `${formatNumber(report.usage.totalTokens)} tokens (${formatNumber(judgeTokens)} judge)` - : `${formatNumber(report.usage.totalTokens)} tokens`, - ); - } - if (report.usage.toolCalls > 0) { - parts.push( - `${formatNumber(report.usage.toolCalls)} tool${report.usage.toolCalls === 1 ? "" : "s"}`, - ); - } - return parts.join(", "); -} - function escapeTableCell(value: string) { return value .replace(/\r?\n/g, " ") @@ -411,26 +388,19 @@ function renderAsciiTable(headers: string[], rows: string[][]) { function formatCaseUsage(testCase: EvalCase) { const usage = testCase.harness?.usage; - const judgeUsage = testCase.eval?.judgeUsage; const parts: string[] = []; - const appTokens = + const totalTokens = usage?.totalTokens ?? (usage?.inputTokens ?? 0) + (usage?.outputTokens ?? 0) + (usage?.reasoningTokens ?? 0); - const judgeTokens = judgeUsage?.totalTokens ?? 0; - const totalTokens = appTokens + judgeTokens; const toolCalls = usage?.toolCalls !== undefined ? Math.max(usage.toolCalls, testCase.toolCalls.length) : testCase.toolCalls.length; if (totalTokens > 0) { - parts.push( - judgeTokens > 0 - ? `${formatNumber(totalTokens)} tokens (${formatNumber(judgeTokens)} judge)` - : `${formatNumber(totalTokens)} tokens`, - ); + parts.push(`${formatNumber(totalTokens)} tokens`); } if (toolCalls > 0) { parts.push(`${formatNumber(toolCalls)} tool${toolCalls === 1 ? "" : "s"}`); diff --git a/packages/github-reporter/src/types.ts b/packages/github-reporter/src/types.ts index 061f52a..33ed5c9 100644 --- a/packages/github-reporter/src/types.ts +++ b/packages/github-reporter/src/types.ts @@ -30,7 +30,6 @@ export type EvalCase = { thresholdFailed?: boolean; output?: unknown; scores: EvalScore[]; - judgeUsage?: Required; }; harness?: { name?: string; @@ -68,10 +67,7 @@ export type EvalReport = { average: number; minimum?: number; }; - /** Combined application and judge usage. */ usage: Required; - /** Usage from judge harness runs only. */ - judgeUsage?: Required; cases: EvalCase[]; failures: EvalCase[]; }; diff --git a/packages/harness-ai-sdk/README.md b/packages/harness-ai-sdk/README.md index edc42a2..53a7f98 100644 --- a/packages/harness-ai-sdk/README.md +++ b/packages/harness-ai-sdk/README.md @@ -87,8 +87,7 @@ const harness = aiSdkHarness({ `run` executes the system under test. Judges are created separately; keep judge prompts and model calls on a judge harness instead of putting them on the app -harness. `aiSdkJudgeHarness()` returns a normalized judge run, including AI SDK -usage, so reports account for judge tokens separately from app tokens. +harness. ```ts import { openai } from "@ai-sdk/openai"; diff --git a/packages/harness-ai-sdk/src/judgeHarness.test.ts b/packages/harness-ai-sdk/src/judgeHarness.test.ts index 3128486..65f020e 100644 --- a/packages/harness-ai-sdk/src/judgeHarness.test.ts +++ b/packages/harness-ai-sdk/src/judgeHarness.test.ts @@ -82,6 +82,7 @@ test("aiSdkJudgeHarness uses generateText without a JSON schema", async () => { }); generateTextMock.mockResolvedValueOnce({ text: "plain verdict", + usage: { inputTokens: 12, outputTokens: 3, totalTokens: 15 }, } as any); const result = await judgeHarness.run( @@ -99,6 +100,11 @@ test("aiSdkJudgeHarness uses generateText without a JSON schema", async () => { expect(judgeHarness.name).toBe("test-judge"); expect(result.output).toBe("plain verdict"); + expect(result.usage).toMatchObject({ + inputTokens: 12, + outputTokens: 3, + totalTokens: 15, + }); expect(generateTextMock).toHaveBeenCalledWith( expect.objectContaining({ model: mockModel, diff --git a/packages/harness-openai-agents/README.md b/packages/harness-openai-agents/README.md index 73a5072..536f990 100644 --- a/packages/harness-openai-agents/README.md +++ b/packages/harness-openai-agents/README.md @@ -88,9 +88,7 @@ const harness = openaiAgentsHarness({ `run` executes the OpenAI agent under test. Judges are created separately; keep judge prompts on the judge and model calls on a judge harness instead of -putting a judge model call on the app harness. `openaiAgentsJudgeHarness()` -returns a normalized judge run, including SDK usage, so reports account for -judge tokens separately from app tokens. +putting a judge model call on the app harness. ```ts import { openaiAgentsJudgeHarness } from "@vitest-evals/harness-openai-agents"; diff --git a/packages/harness-openai-agents/src/index.test.ts b/packages/harness-openai-agents/src/index.test.ts index 83a4710..2991cff 100644 --- a/packages/harness-openai-agents/src/index.test.ts +++ b/packages/harness-openai-agents/src/index.test.ts @@ -6,7 +6,6 @@ import { describeEval, getHarnessRunFromError, messagesToTranscriptEvents, - runJudgeHarnessRun, spansByKind, toolCalls, } from "vitest-evals"; @@ -52,9 +51,10 @@ test("openai judge harness preserves usage", async () => { } as unknown as Runner, }); - const run = await runJudgeHarnessRun(judgeHarness, { - prompt: "Grade this answer.", - }); + const run = await judgeHarness.run( + { prompt: "Grade this answer." }, + { artifacts: {}, setArtifact: vi.fn() }, + ); expect(run.output).toBe("approved"); expect(run.usage).toMatchObject({ diff --git a/packages/harness-pi-ai/README.md b/packages/harness-pi-ai/README.md index f049c4e..bb7eee8 100644 --- a/packages/harness-pi-ai/README.md +++ b/packages/harness-pi-ai/README.md @@ -43,9 +43,7 @@ describeEval("refund agent", { harness }, (it) => { `run` executes the Pi agent under test. Judges are created separately; keep judge prompts on the judge and model calls on a judge harness instead of -putting a judge model call on the app harness. `piAiJudgeHarness()` returns a -normalized judge run, including Pi usage, so reports account for judge tokens -separately from app tokens. +putting a judge model call on the app harness. ```ts import { getModel } from "@mariozechner/pi-ai"; diff --git a/packages/harness-pi-ai/src/judgeHarness.test.ts b/packages/harness-pi-ai/src/judgeHarness.test.ts index b93a578..f45478e 100644 --- a/packages/harness-pi-ai/src/judgeHarness.test.ts +++ b/packages/harness-pi-ai/src/judgeHarness.test.ts @@ -71,6 +71,10 @@ test("piAiJudgeHarness runs judge prompts through Pi AI", async () => { ); expect(result.output).toBe('{"choice":"C","rationale":"Matches."}'); + expect(result.usage).toMatchObject({ + totalTokens: 2, + metadata: { input: 1, output: 1 }, + }); expect(completeSimpleMock).toHaveBeenCalledWith( mockModel, { diff --git a/packages/vitest-evals/src/index.ts b/packages/vitest-evals/src/index.ts index 97891f0..fa08410 100644 --- a/packages/vitest-evals/src/index.ts +++ b/packages/vitest-evals/src/index.ts @@ -29,13 +29,17 @@ import type { JudgeOptions, JudgeResult, } from "./judges/types"; -import type { JudgeHarness, JudgeHarnessRun } from "./judges/judgeHarness"; +import type { JudgeHarness } from "./judges/judgeHarness"; import { createRunJudge } from "./judges/judgeHarness"; import { wrapText } from "./wrapText"; +type RecordedJudgeResult = JudgeResult & { + judgeRuns?: HarnessRun[]; +}; + type EvalTaskMeta = { eval?: { - scores: (JudgeResult & { name: string })[]; + scores: (RecordedJudgeResult & { name: string })[]; avgScore: number; output?: unknown; toolCalls?: ToolCall[]; @@ -449,7 +453,7 @@ expect.extend({ ) { const { threshold = 1.0, ...context } = (options ?? {}) as JudgeAssertionOptions; - const judgeRuns: JudgeHarnessRun[] = []; + const judgeRuns: HarnessRun[] = []; const judgeOptions = buildJudgeAssertionOptions( received, judge, @@ -601,7 +605,7 @@ async function applyAutomaticJudges< const runToolCalls = toolCalls(run.session); const scores = await Promise.all( judges.map(async (judge) => { - const judgeRuns: JudgeHarnessRun[] = []; + const judgeRuns: HarnessRun[] = []; const runJudge = createRunJudge( resolveJudgeHarnessForJudge(judge, judgeHarness), signal, @@ -701,7 +705,7 @@ function appendJudgeScore( thresholdFailed, toolCalls: judgeToolCalls, }: { - score: JudgeResult & { name: string }; + score: RecordedJudgeResult & { name: string }; output?: unknown; thresholdFailed: boolean; toolCalls?: ToolCall[]; @@ -770,8 +774,8 @@ function formatJudgeTextOutput(run: HarnessRun) { function attachJudgeRuns( result: JudgeResult, - judgeRuns: JudgeHarnessRun[], -): JudgeResult { + judgeRuns: HarnessRun[], +): RecordedJudgeResult { return judgeRuns.length > 0 ? { ...result, judgeRuns } : result; } @@ -782,7 +786,7 @@ function buildJudgeAssertionOptions< judge: Judge, options: Omit, "threshold">, task?: EvalTaskLike, - judgeRuns: JudgeHarnessRun[] = [], + judgeRuns: HarnessRun[] = [], ): TJudgeOptions { const registeredContext = resolveRegisteredJudgeRunContext( received, @@ -1283,13 +1287,11 @@ export { type FactualityJudgeVerdict, createJudgeHarness, runJudgeHarness, - runJudgeHarnessRun, type CreateJudgeHarnessOptions, type CreateJudgeHarnessRunOptions, type JudgeHarness, type JudgeHarnessInput, type JudgeHarnessOutput, - type JudgeHarnessRun, type RunJudge, type RunJudgeOptions, StructuredOutputJudge, diff --git a/packages/vitest-evals/src/judges/index.ts b/packages/vitest-evals/src/judges/index.ts index 337fb02..007b92a 100644 --- a/packages/vitest-evals/src/judges/index.ts +++ b/packages/vitest-evals/src/judges/index.ts @@ -11,13 +11,11 @@ export { export { createJudgeHarness, runJudgeHarness, - runJudgeHarnessRun, type CreateJudgeHarnessOptions, type CreateJudgeHarnessRunOptions, type JudgeHarness, type JudgeHarnessInput, type JudgeHarnessOutput, - type JudgeHarnessRun, type RunJudge, type RunJudgeOptions, } from "./judgeHarness"; diff --git a/packages/vitest-evals/src/judges/judgeHarness.test.ts b/packages/vitest-evals/src/judges/judgeHarness.test.ts index 58a518c..474f373 100644 --- a/packages/vitest-evals/src/judges/judgeHarness.test.ts +++ b/packages/vitest-evals/src/judges/judgeHarness.test.ts @@ -1,11 +1,7 @@ import { expect, test } from "vitest"; import { createHarness } from "../harness"; import { createJudge, describeEval } from "../index"; -import { - createJudgeHarness, - runJudgeHarness, - runJudgeHarnessRun, -} from "./judgeHarness"; +import { createJudgeHarness, runJudgeHarness } from "./judgeHarness"; test("runJudgeHarness preserves null output values", async () => { const judgeHarness = createJudgeHarness({ @@ -45,33 +41,6 @@ test("runJudgeHarness falls back to assistant content when output is missing", a expect(result).toBe('{"choice":"C"}'); }); -test("runJudgeHarnessRun preserves normalized usage", async () => { - const judgeHarness = createJudgeHarness({ - run: async () => ({ - output: "approved", - session: { events: [] }, - usage: { - inputTokens: 12, - outputTokens: 3, - totalTokens: 15, - metadata: { costUsd: 0.02 }, - }, - errors: [], - }), - }); - - const run = await runJudgeHarnessRun(judgeHarness, { - prompt: "Return JSON.", - }); - - expect(run.usage).toEqual({ - inputTokens: 12, - outputTokens: 3, - totalTokens: 15, - metadata: { costUsd: 0.02 }, - }); -}); - const appHarness = createHarness({ name: "app", run: async ({ input }: { input: string }) => ({ diff --git a/packages/vitest-evals/src/judges/judgeHarness.ts b/packages/vitest-evals/src/judges/judgeHarness.ts index d2e74bf..9c974ff 100644 --- a/packages/vitest-evals/src/judges/judgeHarness.ts +++ b/packages/vitest-evals/src/judges/judgeHarness.ts @@ -65,9 +65,6 @@ export type RunJudgeOptions = { signal?: AbortSignal; }; -/** Full normalized run produced by one judge-model invocation. */ -export type JudgeHarnessRun = HarnessRun; - /** * Curried judge-harness runner available inside `JudgeContext`. * @@ -156,38 +153,23 @@ export async function runJudgeHarness( input: JudgeHarnessInput, options: RunJudgeOptions = {}, ): Promise { - const run = await runJudgeHarnessRun(judgeHarness, input, options); - return resolveJudgeHarnessOutput(run); -} - -/** Runs a judge harness and returns its complete normalized run. */ -export async function runJudgeHarnessRun( - judgeHarness: JudgeHarness, - input: JudgeHarnessInput, - options: RunJudgeOptions = {}, -): Promise { - const artifacts: HarnessContext["artifacts"] = {}; - return judgeHarness.run(input, { - signal: options.signal, - artifacts, - setArtifact: (name, value) => { - artifacts[name] = value; - }, - }); + return resolveJudgeHarnessOutput( + await executeJudgeHarness(judgeHarness, input, options), + ); } /** Binds a judge harness to the current eval run context. */ export function createRunJudge( judgeHarness: JudgeHarness | undefined, signal?: AbortSignal, - onRun?: (run: JudgeHarnessRun) => void, + onRun?: (run: HarnessRun) => void, ): RunJudge | undefined { if (!judgeHarness) { return undefined; } return async (input, options) => { - const run = await runJudgeHarnessRun(judgeHarness, input, { + const run = await executeJudgeHarness(judgeHarness, input, { signal: options?.signal ?? signal, }); onRun?.(run); @@ -195,6 +177,21 @@ export function createRunJudge( }; } +async function executeJudgeHarness( + judgeHarness: JudgeHarness, + input: JudgeHarnessInput, + options: RunJudgeOptions, +) { + const artifacts: HarnessContext["artifacts"] = {}; + return judgeHarness.run(input, { + signal: options.signal, + artifacts, + setArtifact: (name, value) => { + artifacts[name] = value; + }, + }); +} + function normalizeJudgeHarnessResult( input: JudgeHarnessInput, result: Awaited>, @@ -246,7 +243,9 @@ function createJudgeHarnessMessages( ]; } -function resolveJudgeHarnessOutput(run: JudgeHarnessRun): JudgeHarnessOutput { +function resolveJudgeHarnessOutput( + run: HarnessRun, +): JudgeHarnessOutput { return run.output !== undefined ? run.output : (latestAssistantMessageContent(run.session) ?? ""); diff --git a/packages/vitest-evals/src/judges/types.ts b/packages/vitest-evals/src/judges/types.ts index 3e4c9d4..e881693 100644 --- a/packages/vitest-evals/src/judges/types.ts +++ b/packages/vitest-evals/src/judges/types.ts @@ -1,5 +1,5 @@ import type { Harness, HarnessRun, JsonValue, ToolCall } from "../harness"; -import type { JudgeHarness, JudgeHarnessRun, RunJudge } from "./judgeHarness"; +import type { JudgeHarness, RunJudge } from "./judgeHarness"; /** * Score payload returned by a judge. @@ -17,8 +17,6 @@ import type { JudgeHarness, JudgeHarnessRun, RunJudge } from "./judgeHarness"; export type JudgeResult = { /** Numeric score. `null` records an intentionally unscored result. */ score: number | null; - /** Normalized judge-model runs recorded while producing this score. */ - judgeRuns?: JudgeHarnessRun[]; /** JSON-like judge metadata shown by assertions and reporters. */ metadata?: { /** Human-readable explanation for the score. */ diff --git a/packages/vitest-evals/src/reporter.ts b/packages/vitest-evals/src/reporter.ts index 13307e5..9bb6552 100644 --- a/packages/vitest-evals/src/reporter.ts +++ b/packages/vitest-evals/src/reporter.ts @@ -300,55 +300,18 @@ export default class DefaultEvalReporter extends VerboseReporter { scores: Array<{ name?: string; score?: number | null; - judgeRuns?: Array<{ - usage?: { - inputTokens?: number; - outputTokens?: number; - reasoningTokens?: number; - totalTokens?: number; - }; - }>; }>, ) { for (const score of scores) { this.log( this.formatDetailLine( "score", - [ - `${score.name || "Unknown"} ${this.formatScore(score.score ?? 0)}`, - this.formatJudgeUsage(score.judgeRuns), - ] - .filter(Boolean) - .join(" · "), + `${score.name || "Unknown"} ${this.formatScore(score.score ?? 0)}`, ), ); } } - private formatJudgeUsage( - runs: - | Array<{ - usage?: { - inputTokens?: number; - outputTokens?: number; - reasoningTokens?: number; - totalTokens?: number; - }; - }> - | undefined, - ) { - const totalTokens = (runs ?? []).reduce( - (total, run) => - total + - (run.usage?.totalTokens ?? - (run.usage?.inputTokens ?? 0) + - (run.usage?.outputTokens ?? 0) + - (run.usage?.reasoningTokens ?? 0)), - 0, - ); - return totalTokens > 0 ? `${totalTokens} judge tok` : undefined; - } - private formatFieldLabel( label: | "tool" diff --git a/skills/vitest-evals/SPEC.md b/skills/vitest-evals/SPEC.md index 36f622c..dcc3932 100644 --- a/skills/vitest-evals/SPEC.md +++ b/skills/vitest-evals/SPEC.md @@ -34,8 +34,7 @@ Out of scope: - Required outputs: implementation or review guidance that uses the harness-backed API only, plus targeted verification. - Non-negotiable constraints: normalized run data remains JSON-serializable, suite tests call `run(...)` explicitly, judge model calls stay on judges or - judge-side helpers rather than the app harness, and model-backed judge runs - retain normalized usage for reports. + judge-side helpers, and model-backed judge harnesses return normalized usage. - Expected bundled files loaded at runtime: `SKILL.md` first, then one or more focused files under `references/`. ## Source And Evidence Model @@ -79,8 +78,6 @@ Data that must not be stored: - The skill does not automate installation; it is installable as a self-contained directory. - Provider-specific model credentials and environment setup remain application concerns. - API details must be refreshed when package options, peer ranges, or reporter behavior change. -- Provider-specific cost remains metadata; the skill does not define a stable - cost field or pricing model. ## Maintenance Notes From e23345567a309ae38c85706abd6f8e8eb84ca794 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 04:42:08 +0000 Subject: [PATCH 04/15] feat(usage): standardize and report eval costs Co-Authored-By: David Cramer --- .github/workflows/github-reporter-smoke.yml | 7 ++-- README.md | 4 +- docs/architecture.md | 12 +++--- docs/github-actions.md | 4 +- docs/provider-transformations.md | 4 +- packages/core/src/harness/index.ts | 5 ++- packages/core/src/index.test.ts | 13 +++--- .../src/content/docs/docs/judges/custom.mdx | 8 ++-- packages/github-reporter/src/collect.ts | 31 +++++++++++++- packages/github-reporter/src/merge.ts | 3 ++ packages/github-reporter/src/report.test.ts | 25 ++++++++---- packages/github-reporter/src/summary.ts | 40 ++++++++++++++++++- packages/github-reporter/src/types.ts | 6 +++ packages/harness-pi-ai/src/index.test.ts | 8 ++-- packages/harness-pi-ai/src/index.ts | 14 +++++++ .../harness-pi-ai/src/judgeHarness.test.ts | 1 + packages/vitest-evals/README.md | 4 +- skills/vitest-evals/SPEC.md | 2 +- .../references/judges-and-assertions.md | 6 +-- 19 files changed, 149 insertions(+), 48 deletions(-) diff --git a/.github/workflows/github-reporter-smoke.yml b/.github/workflows/github-reporter-smoke.yml index c81c39f..db8b4e5 100644 --- a/.github/workflows/github-reporter-smoke.yml +++ b/.github/workflows/github-reporter-smoke.yml @@ -373,10 +373,9 @@ jobs: echo "test totals row should not be shown" exit 1 fi - if grep -F "| Usage |" "$GITHUB_STEP_SUMMARY"; then - echo "top-level usage row should not be shown" - exit 1 - fi + grep -F "| App Usage | 1,302 tokens, 9 tools |" "$GITHUB_STEP_SUMMARY" + grep -F "| Judge Usage | none |" "$GITHUB_STEP_SUMMARY" + grep -F "| Total Usage | 1,302 tokens, 9 tools |" "$GITHUB_STEP_SUMMARY" grep -F "::error file=apps/demo/evals/smoke.eval.ts,line=12,col=5,title=vitest-evals::refund eval > rejects fraud - score 0.10 - SmokeJudge - expected approved but got denied" tmp/github-reporter-smoke/stdout.txt grep -F "::error file=apps/demo/evals/smoke.eval.ts,line=28,col=5,title=vitest-evals::refund eval > returns wrong refund amount - score 0.35 - AmountJudge - expected amount=5000, got amount=499" tmp/github-reporter-smoke/stdout.txt grep -F "::warning::GitHub Check Run skipped: missing GITHUB_TOKEN" tmp/github-reporter-smoke/stdout.txt diff --git a/README.md b/README.md index 30b22a9..961f33c 100644 --- a/README.md +++ b/README.md @@ -232,8 +232,8 @@ Harness-backed suites stay close to plain Vitest: `ctx.runJudge(...)` from a configured `judgeHarness` - pass scenario-specific judge criteria explicitly to matcher options, or bind suite-wide criteria on the judge instance -- reporter output, replay, usage, tool calls, and spans come from normalized - runs, including model-backed judge runs. First-party harnesses attach native +- reporter output separates application, judge, and combined token and + `costUsd` usage from normalized runs. First-party harnesses attach native spans when provider/runtime data is available; `createHarness(...)` attaches a fallback run span when a custom harness does not return traces itself. Span attributes include typed OpenTelemetry GenAI semantic keys while keeping diff --git a/docs/architecture.md b/docs/architecture.md index b71ed5f..6a4c563 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -94,10 +94,9 @@ not return traces themselves. Span attributes include typed OpenTelemetry GenAI semantic keys for common model, agent, tool, and token fields while still allowing provider-specific attributes. -`UsageSummary` is intentionally limited to stable usage units such as tokens, -tool counts, retries, provider, and model. Provider-specific cost estimates are -not normalized because pricing semantics vary by runtime and can be stale; if a -harness needs to retain them, store them under `usage.metadata`. +`UsageSummary` standardizes tokens, USD cost, tool counts, retries, provider, +and model. Harnesses report provider-supplied or suite-estimated cost through +`costUsd`; provider-specific pricing details remain under `usage.metadata`. ### `packages/vitest-evals/src/index.ts` @@ -143,9 +142,8 @@ judge should carry a reusable judge-side harness default. `createJudgeHarness(...)` is the shared abstraction for judge-side provider shims. Each `ctx.runJudge(...)` call returns the normalized judge output while core records the complete judge harness run on the resulting score. Multiple -calls from one judge remain separate runs. Reporters aggregate their stable -usage fields alongside application usage while preserving provider-specific -cost estimates under each run's `usage.metadata`. +calls from one judge remain separate runs. Reporters aggregate tokens and +`costUsd` separately for application and judge usage, plus a combined total. ### `packages/vitest-evals/src/legacy/*` diff --git a/docs/github-actions.md b/docs/github-actions.md index f2ec471..6c5afea 100644 --- a/docs/github-actions.md +++ b/docs/github-actions.md @@ -9,8 +9,8 @@ Public docs mirror this guide at Use JSON as the eval artifact because it preserves `task.meta.eval` and `task.meta.harness`; JUnit XML does not carry the full eval metadata. Usage -totals include normalized application and judge token usage. Judge runs remain -attached to their scores, including provider-specific `usage.metadata`. +reports show separate application, judge, and combined token and `costUsd` +totals. Judge runs remain attached to their scores for detailed usage data. ## Minimal Workflow diff --git a/docs/provider-transformations.md b/docs/provider-transformations.md index 2858a81..dcc1763 100644 --- a/docs/provider-transformations.md +++ b/docs/provider-transformations.md @@ -110,8 +110,8 @@ Harness adapters should: transcript events remain unambiguous - preserve the application-facing result separately in `run.output` - attach provider/model and stable usage data when available -- keep provider-specific cost estimates in `usage.metadata`, not as normalized - usage fields +- map provider-reported or suite-estimated USD cost to `usage.costUsd`; keep + provider-specific pricing details in `usage.metadata` - attach replay/cache metadata in the tool record metadata rather than in provider-specific side channels - expose common assertion data through helpers such as `toolCalls(result)`, diff --git a/packages/core/src/harness/index.ts b/packages/core/src/harness/index.ts index 60f3449..d37b563 100644 --- a/packages/core/src/harness/index.ts +++ b/packages/core/src/harness/index.ts @@ -39,6 +39,7 @@ export const UsageSummarySchema = z outputTokens: FiniteNumberSchema.optional(), reasoningTokens: FiniteNumberSchema.optional(), totalTokens: FiniteNumberSchema.optional(), + costUsd: FiniteNumberSchema.optional(), toolCalls: FiniteNumberSchema.optional(), retries: FiniteNumberSchema.optional(), metadata: JsonObjectSchema.optional(), @@ -59,11 +60,13 @@ export type UsageSummary = { reasoningTokens?: number; /** Total token count reported by the provider or adapter. */ totalTokens?: number; + /** Cost of the run in US dollars, reported or estimated by the provider or adapter. */ + costUsd?: number; /** Count of tool calls observed during the run. */ toolCalls?: number; /** Retry count observed during the run. */ retries?: number; - /** Provider-specific JSON-safe usage details. Cost estimates belong here. */ + /** Provider-specific JSON-safe usage details. */ metadata?: Record; }; diff --git a/packages/core/src/index.test.ts b/packages/core/src/index.test.ts index 44beedf..2f49acb 100644 --- a/packages/core/src/index.test.ts +++ b/packages/core/src/index.test.ts @@ -884,22 +884,21 @@ describe("messagesToTranscriptEvents", () => { }); describe("UsageSummarySchema", () => { - test("keeps provider-specific usage data under metadata", () => { + test("accepts standardized USD cost and keeps provider details in metadata", () => { expect( UsageSummarySchema.safeParse({ totalTokens: 120, - estimatedCostUsd: 0.02, + costUsd: 0.02, + metadata: { providerRequestId: "req_123" }, }).success, - ).toBe(false); + ).toBe(true); expect( UsageSummarySchema.safeParse({ totalTokens: 120, - metadata: { - estimatedCostUsd: 0.02, - }, + estimatedCostUsd: 0.02, }).success, - ).toBe(true); + ).toBe(false); }); }); diff --git a/packages/docs/src/content/docs/docs/judges/custom.mdx b/packages/docs/src/content/docs/docs/judges/custom.mdx index f337601..603012f 100644 --- a/packages/docs/src/content/docs/docs/judges/custom.mdx +++ b/packages/docs/src/content/docs/docs/judges/custom.mdx @@ -97,9 +97,9 @@ export const RubricJudge = createJudge({ ``` Return a full `HarnessRun` from a custom judge harness when the provider exposes -usage. Stable token fields are included in GitHub report totals. Keep -provider-specific cost estimates under `usage.metadata`; reports preserve -that metadata but do not treat it as a normalized cost field. +usage. Set `usage.costUsd` from provider-reported cost or a suite estimate. +GitHub reports show application, judge, and combined token and cost totals; +provider-specific pricing details can remain under `usage.metadata`. ```ts title="evals/judges/judgeHarness.ts" const judgeHarness = createJudgeHarness({ @@ -112,7 +112,7 @@ const judgeHarness = createJudgeHarness({ usage: { inputTokens: response.usage.inputTokens, outputTokens: response.usage.outputTokens, - metadata: { costUsd: response.costUsd }, + costUsd: response.costUsd, }, errors: [], }; diff --git a/packages/github-reporter/src/collect.ts b/packages/github-reporter/src/collect.ts index 89ae93e..608dab9 100644 --- a/packages/github-reporter/src/collect.ts +++ b/packages/github-reporter/src/collect.ts @@ -37,7 +37,9 @@ export function collectEvalReport( const evalScores = cases .map((testCase) => testCase.eval?.avgScore) .filter((score): score is number => isFiniteNumber(score)); - const usage = sumUsage(cases); + const appUsage = sumAppUsage(cases); + const judgeUsage = sumJudgeUsage(cases); + const usage = addUsage(appUsage, judgeUsage); const durationMs = workspace.runs[0]?.durationMs; return { @@ -64,6 +66,8 @@ export function collectEvalReport( } : undefined, usage, + appUsage, + judgeUsage, cases, failures, }; @@ -238,6 +242,7 @@ function emptyUsage(): Required { outputTokens: 0, reasoningTokens: 0, totalTokens: 0, + costUsd: 0, toolCalls: 0, }; } @@ -254,15 +259,37 @@ function addRunUsage( (usage?.inputTokens ?? 0) + (usage?.outputTokens ?? 0) + (usage?.reasoningTokens ?? 0); + total.costUsd += usage?.costUsd ?? 0; total.toolCalls += usage?.toolCalls ?? 0; } -function sumUsage(cases: EvalCase[]) { +function addUsage( + left: Required, + right: Required, +): Required { + return { + inputTokens: left.inputTokens + right.inputTokens, + outputTokens: left.outputTokens + right.outputTokens, + reasoningTokens: left.reasoningTokens + right.reasoningTokens, + totalTokens: left.totalTokens + right.totalTokens, + costUsd: left.costUsd + right.costUsd, + toolCalls: left.toolCalls + right.toolCalls, + }; +} + +function sumAppUsage(cases: EvalCase[]) { const usage = emptyUsage(); for (const testCase of cases) { addRunUsage(usage, testCase.harness?.usage); usage.toolCalls += toolCallCount(testCase) - (testCase.harness?.usage?.toolCalls ?? 0); + } + return usage; +} + +function sumJudgeUsage(cases: EvalCase[]) { + const usage = emptyUsage(); + for (const testCase of cases) { for (const score of testCase.eval?.scores ?? []) { for (const run of score.judgeRuns ?? []) { addRunUsage(usage, run.usage); diff --git a/packages/github-reporter/src/merge.ts b/packages/github-reporter/src/merge.ts index 387ce34..d9083f7 100644 --- a/packages/github-reporter/src/merge.ts +++ b/packages/github-reporter/src/merge.ts @@ -44,6 +44,8 @@ export function mergeEvalReports(reports: EvalReport[]): EvalReport { } : undefined, usage: mergeUsage(reports.map((report) => report.usage)), + appUsage: mergeUsage(reports.map((report) => report.appUsage)), + judgeUsage: mergeUsage(reports.map((report) => report.judgeUsage)), cases, failures, }; @@ -55,6 +57,7 @@ function mergeUsage(usages: Array>) { outputTokens: sum(usages, (usage) => usage.outputTokens), reasoningTokens: sum(usages, (usage) => usage.reasoningTokens), totalTokens: sum(usages, (usage) => usage.totalTokens), + costUsd: sum(usages, (usage) => usage.costUsd), toolCalls: sum(usages, (usage) => usage.toolCalls), }; } diff --git a/packages/github-reporter/src/report.test.ts b/packages/github-reporter/src/report.test.ts index bb19a69..bd26bac 100644 --- a/packages/github-reporter/src/report.test.ts +++ b/packages/github-reporter/src/report.test.ts @@ -71,7 +71,7 @@ const sampleJson: VitestJsonReport = { inputTokens: 80, outputTokens: 20, totalTokens: 100, - metadata: { costUsd: 0.02 }, + costUsd: 0.02, }, errors: [], }, @@ -95,6 +95,7 @@ const sampleJson: VitestJsonReport = { }, usage: { totalTokens: 1220, + costUsd: 0.08, toolCalls: 2, }, timings: { @@ -175,11 +176,19 @@ describe("collectEvalReport", () => { average: 0.2, minimum: 0.2, }); - expect(report.usage.totalTokens).toBe(1320); - expect( - report.cases[0]?.eval?.scores[0]?.judgeRuns?.[0]?.usage.metadata, - ).toEqual({ costUsd: 0.02 }); - expect(report.usage.toolCalls).toBe(2); + expect(report.appUsage).toMatchObject({ + totalTokens: 1220, + costUsd: 0.08, + }); + expect(report.judgeUsage).toMatchObject({ + totalTokens: 100, + costUsd: 0.02, + }); + expect(report.usage).toMatchObject({ + totalTokens: 1320, + costUsd: 0.1, + toolCalls: 2, + }); expect(report.failures[0]).toMatchObject({ displayFile: "apps/demo/evals/refund.eval.ts", displayName: "refund agent > rejects fraud", @@ -918,7 +927,9 @@ describe("renderJobSummary", () => { expect(summary).toContain("| Pass Rate | 0.0% |"); expect(summary).not.toContain("| Tests |"); expect(summary).toContain("| Score | avg 0.20, min 0.20 |"); - expect(summary).not.toContain("| Usage |"); + expect(summary).toContain("| App Usage | 1,220 tokens, $0.08, 2 tools |"); + expect(summary).toContain("| Judge Usage | 100 tokens, $0.02 |"); + expect(summary).toContain("| Total Usage | 1,320 tokens, $0.10, 2 tools |"); expect(summary).toContain("## Scores"); expect(summary.indexOf("## Scores")).toBeLessThan( summary.indexOf("## Results"), diff --git a/packages/github-reporter/src/summary.ts b/packages/github-reporter/src/summary.ts index ce70f4e..72b68e2 100644 --- a/packages/github-reporter/src/summary.ts +++ b/packages/github-reporter/src/summary.ts @@ -1,6 +1,11 @@ import type { EvalGateResult } from "./gate"; import { formatPercent } from "./gate"; -import type { EvalCase, EvalReport, ToolCallSummary } from "./types"; +import type { + EvalCase, + EvalReport, + ToolCallSummary, + UsageSummary, +} from "./types"; import { compactLine, escapeFence, @@ -114,6 +119,12 @@ function renderSummaryTable( rows.push(["Score", formatScoreSummary(report.score)]); } + if (hasUsage(report.usage)) { + rows.push(["App Usage", formatUsage(report.appUsage)]); + rows.push(["Judge Usage", formatUsage(report.judgeUsage)]); + rows.push(["Total Usage", formatUsage(report.usage)]); + } + if (gate?.enforced) { rows.push(["Gate", gate.message]); } @@ -145,6 +156,33 @@ function formatScoreSummary(score: NonNullable) { }`; } +function hasUsage(usage: Required) { + return usage.totalTokens > 0 || usage.costUsd > 0 || usage.toolCalls > 0; +} + +function formatUsage(usage: Required) { + const parts: string[] = []; + if (usage.totalTokens > 0) { + parts.push(`${formatNumber(usage.totalTokens)} tokens`); + } + if (usage.costUsd > 0) { + parts.push( + usage.costUsd.toLocaleString("en-US", { + style: "currency", + currency: "USD", + minimumFractionDigits: 2, + maximumFractionDigits: 6, + }), + ); + } + if (usage.toolCalls > 0) { + parts.push( + `${formatNumber(usage.toolCalls)} tool${usage.toolCalls === 1 ? "" : "s"}`, + ); + } + return parts.join(", ") || "none"; +} + function escapeTableCell(value: string) { return value .replace(/\r?\n/g, " ") diff --git a/packages/github-reporter/src/types.ts b/packages/github-reporter/src/types.ts index 33ed5c9..243ce72 100644 --- a/packages/github-reporter/src/types.ts +++ b/packages/github-reporter/src/types.ts @@ -67,7 +67,12 @@ export type EvalReport = { average: number; minimum?: number; }; + /** Combined application and judge usage. */ usage: Required; + /** Application and other upstream usage. */ + appUsage: Required; + /** Judge-model usage. */ + judgeUsage: Required; cases: EvalCase[]; failures: EvalCase[]; }; @@ -78,6 +83,7 @@ export type UsageSummary = { outputTokens?: number; reasoningTokens?: number; totalTokens?: number; + costUsd?: number; toolCalls?: number; }; diff --git a/packages/harness-pi-ai/src/index.test.ts b/packages/harness-pi-ai/src/index.test.ts index 4b57cd1..bc346bd 100644 --- a/packages/harness-pi-ai/src/index.test.ts +++ b/packages/harness-pi-ai/src/index.test.ts @@ -1060,7 +1060,7 @@ test("applies output selectors to HarnessRun-shaped results", async () => { expect(result.usage.totalTokens).toBe(7); }); -test("moves provider-specific usage fields into metadata", async () => { +test("normalizes USD cost and moves other provider usage fields into metadata", async () => { const normalizedHarness = piAiHarness({ agent: () => ({ id: "refund-agent" }), run: async () => ({ @@ -1073,7 +1073,8 @@ test("moves provider-specific usage fields into metadata", async () => { totalTokens: 7, toolCalls: 0, retries: 1, - costUSD: 20, + costUsd: 20, + costUSD: 21, estimatedCost: 20, metadata: { cacheReadTokens: 3, @@ -1093,9 +1094,10 @@ test("moves provider-specific usage fields into metadata", async () => { totalTokens: 7, toolCalls: 0, retries: 1, + costUsd: 20, metadata: { cacheReadTokens: 3, - costUSD: 20, + costUSD: 21, estimatedCost: 20, }, }); diff --git a/packages/harness-pi-ai/src/index.ts b/packages/harness-pi-ai/src/index.ts index 1f9d963..ab0b1ac 100644 --- a/packages/harness-pi-ai/src/index.ts +++ b/packages/harness-pi-ai/src/index.ts @@ -1819,6 +1819,7 @@ function resolveUsage(result: unknown, toolCallCount: number): UsageSummary { outputTokens: numberField(usageRecord.outputTokens), reasoningTokens: numberField(usageRecord.reasoningTokens), totalTokens: numberField(usageRecord.totalTokens), + costUsd: resolveCostUsd(usageRecord), toolCalls: numberField(usageRecord.toolCalls), retries: numberField(usageRecord.retries), metadata: collectUsageMetadata(usageRecord), @@ -1859,10 +1860,23 @@ const USAGE_SUMMARY_KEYS = new Set([ "outputTokens", "reasoningTokens", "totalTokens", + "costUsd", "toolCalls", "retries", ]); +function resolveCostUsd(usage: Record) { + const direct = numberField(usage.costUsd); + if (direct !== undefined) { + return direct; + } + + const cost = usage.cost; + return cost && typeof cost === "object" && !Array.isArray(cost) + ? numberField((cost as Record).total) + : undefined; +} + function stringField(value: unknown) { return typeof value === "string" ? value : undefined; } diff --git a/packages/harness-pi-ai/src/judgeHarness.test.ts b/packages/harness-pi-ai/src/judgeHarness.test.ts index f45478e..6f84021 100644 --- a/packages/harness-pi-ai/src/judgeHarness.test.ts +++ b/packages/harness-pi-ai/src/judgeHarness.test.ts @@ -73,6 +73,7 @@ test("piAiJudgeHarness runs judge prompts through Pi AI", async () => { expect(result.output).toBe('{"choice":"C","rationale":"Matches."}'); expect(result.usage).toMatchObject({ totalTokens: 2, + costUsd: 0, metadata: { input: 1, output: 1 }, }); expect(completeSimpleMock).toHaveBeenCalledWith( diff --git a/packages/vitest-evals/README.md b/packages/vitest-evals/README.md index e7e915d..83f932e 100644 --- a/packages/vitest-evals/README.md +++ b/packages/vitest-evals/README.md @@ -371,8 +371,8 @@ need exact canonical `session.events`, trace, or usage control. Provider setup and rubric parsing stay in your judge. The core package only requires the judge to return a `JudgeResult` with a score and optional metadata. Each `ctx.runJudge(...)` call also records its complete normalized judge run -under that score's `judgeRuns` field. Stable judge usage contributes to report -totals; provider-specific cost estimates remain under `usage.metadata`. +under that score's `judgeRuns` field. Tokens and `costUsd` contribute to +separate application, judge, and combined report totals. Automatic suite-level judges are a good fit when every `run(...)` should get the same scoring. For cases where only some runs need an LLM judge, keep the diff --git a/skills/vitest-evals/SPEC.md b/skills/vitest-evals/SPEC.md index dcc3932..10f4264 100644 --- a/skills/vitest-evals/SPEC.md +++ b/skills/vitest-evals/SPEC.md @@ -34,7 +34,7 @@ Out of scope: - Required outputs: implementation or review guidance that uses the harness-backed API only, plus targeted verification. - Non-negotiable constraints: normalized run data remains JSON-serializable, suite tests call `run(...)` explicitly, judge model calls stay on judges or - judge-side helpers, and model-backed judge harnesses return normalized usage. + judge-side helpers, and model-backed harnesses normalize tokens and `costUsd`. - Expected bundled files loaded at runtime: `SKILL.md` first, then one or more focused files under `references/`. ## Source And Evidence Model diff --git a/skills/vitest-evals/references/judges-and-assertions.md b/skills/vitest-evals/references/judges-and-assertions.md index ccfb886..9105151 100644 --- a/skills/vitest-evals/references/judges-and-assertions.md +++ b/skills/vitest-evals/references/judges-and-assertions.md @@ -78,9 +78,9 @@ const RefundRubricJudge = createJudge< - `ctx.runJudge(...)` returns the judge output, while vitest-evals records the complete normalized run under the score's `judgeRuns` field. - Multiple `runJudge(...)` calls from one judge are retained separately. -- Stable token fields contribute to report totals. Provider-specific cost - estimates stay under `usage.metadata` and are preserved without being - normalized. +- Stable token fields and `costUsd` contribute to separate application, judge, + and combined report totals. Provider-specific pricing details stay under + `usage.metadata`. ## Review Checklist From 267b16c6c824b176387ac5cde96c4bcc45a1d462 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 04:52:26 +0000 Subject: [PATCH 05/15] refactor(usage): tighten schema semantics Co-Authored-By: David Cramer --- .github/workflows/github-reporter-smoke.yml | 2 +- docs/architecture.md | 12 +++-- docs/github-actions.md | 4 +- docs/provider-transformations.md | 5 +- packages/core/src/harness/index.ts | 4 +- packages/core/src/index.test.ts | 5 +- packages/core/src/report/workspace.ts | 2 +- packages/docs/src/content/docs/api.mdx | 2 +- .../src/content/docs/docs/judges/custom.mdx | 5 +- packages/github-reporter/src/collect.ts | 29 +++--------- packages/github-reporter/src/merge.ts | 12 +++-- packages/github-reporter/src/report.test.ts | 30 ++++++++---- packages/github-reporter/src/summary.ts | 46 +++++++++++++++---- packages/github-reporter/src/types.ts | 10 ++-- packages/report-ui/src/app/App.test.ts | 4 +- packages/report-ui/src/app/model.test.ts | 2 +- packages/report-ui/src/server.test.ts | 4 +- 17 files changed, 112 insertions(+), 66 deletions(-) diff --git a/.github/workflows/github-reporter-smoke.yml b/.github/workflows/github-reporter-smoke.yml index db8b4e5..7bd98fb 100644 --- a/.github/workflows/github-reporter-smoke.yml +++ b/.github/workflows/github-reporter-smoke.yml @@ -373,7 +373,7 @@ jobs: echo "test totals row should not be shown" exit 1 fi - grep -F "| App Usage | 1,302 tokens, 9 tools |" "$GITHUB_STEP_SUMMARY" + grep -F "| Application Usage | 1,302 tokens, 9 tools |" "$GITHUB_STEP_SUMMARY" grep -F "| Judge Usage | none |" "$GITHUB_STEP_SUMMARY" grep -F "| Total Usage | 1,302 tokens, 9 tools |" "$GITHUB_STEP_SUMMARY" grep -F "::error file=apps/demo/evals/smoke.eval.ts,line=12,col=5,title=vitest-evals::refund eval > rejects fraud - score 0.10 - SmokeJudge - expected approved but got denied" tmp/github-reporter-smoke/stdout.txt diff --git a/docs/architecture.md b/docs/architecture.md index 6a4c563..dd7df7e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -96,7 +96,8 @@ allowing provider-specific attributes. `UsageSummary` standardizes tokens, USD cost, tool counts, retries, provider, and model. Harnesses report provider-supplied or suite-estimated cost through -`costUsd`; provider-specific pricing details remain under `usage.metadata`. +`costUsd`; omit it when cost is unknown, and use zero only for known-free runs. +Provider-specific pricing details remain under `usage.metadata`. ### `packages/vitest-evals/src/index.ts` @@ -142,8 +143,8 @@ judge should carry a reusable judge-side harness default. `createJudgeHarness(...)` is the shared abstraction for judge-side provider shims. Each `ctx.runJudge(...)` call returns the normalized judge output while core records the complete judge harness run on the resulting score. Multiple -calls from one judge remain separate runs. Reporters aggregate tokens and -`costUsd` separately for application and judge usage, plus a combined total. +calls from one judge remain separate runs. The GitHub reporter aggregates tokens +and `costUsd` separately for application and judge usage, plus a derived total. ### `packages/vitest-evals/src/legacy/*` @@ -173,7 +174,10 @@ Provides the custom Vitest reporter that reads normalized run metadata from `packages/core` owns dependency-light primitives shared by the Vitest integration, GitHub reporter, and report UI. Its main entry stays browser-safe, while `@vitest-evals/core/node` exposes filesystem helpers for local and CI -report consumers. It exports stable schemas, TypeScript types, and helpers for: +report consumers. The collected workspace is versioned. Judge runs and standardized cost change +its persisted shape in schema version 2; readers reject unsupported versions +instead of silently misreading them. It exports schemas, TypeScript types, and +helpers for: - JSON-safe values - normalized harness runs, sessions, transcript-derived tool calls, usage, diff --git a/docs/github-actions.md b/docs/github-actions.md index 6c5afea..d4bb25c 100644 --- a/docs/github-actions.md +++ b/docs/github-actions.md @@ -10,7 +10,9 @@ Public docs mirror this guide at Use JSON as the eval artifact because it preserves `task.meta.eval` and `task.meta.harness`; JUnit XML does not carry the full eval metadata. Usage reports show separate application, judge, and combined token and `costUsd` -totals. Judge runs remain attached to their scores for detailed usage data. +totals. A combined cost is shown only when every usage category reports cost; +omitted cost means unknown, while zero means known free. Judge runs remain +attached to their scores for detailed usage data. ## Minimal Workflow diff --git a/docs/provider-transformations.md b/docs/provider-transformations.md index dcc1763..67ea95f 100644 --- a/docs/provider-transformations.md +++ b/docs/provider-transformations.md @@ -110,8 +110,9 @@ Harness adapters should: transcript events remain unambiguous - preserve the application-facing result separately in `run.output` - attach provider/model and stable usage data when available -- map provider-reported or suite-estimated USD cost to `usage.costUsd`; keep - provider-specific pricing details in `usage.metadata` +- map provider-reported or suite-estimated USD cost to `usage.costUsd`; omit it + when unknown, use zero only for known-free runs, and keep provider-specific + pricing details in `usage.metadata` - attach replay/cache metadata in the tool record metadata rather than in provider-specific side channels - expose common assertion data through helpers such as `toolCalls(result)`, diff --git a/packages/core/src/harness/index.ts b/packages/core/src/harness/index.ts index d37b563..66e0c7f 100644 --- a/packages/core/src/harness/index.ts +++ b/packages/core/src/harness/index.ts @@ -39,7 +39,7 @@ export const UsageSummarySchema = z outputTokens: FiniteNumberSchema.optional(), reasoningTokens: FiniteNumberSchema.optional(), totalTokens: FiniteNumberSchema.optional(), - costUsd: FiniteNumberSchema.optional(), + costUsd: FiniteNumberSchema.nonnegative().optional(), toolCalls: FiniteNumberSchema.optional(), retries: FiniteNumberSchema.optional(), metadata: JsonObjectSchema.optional(), @@ -60,7 +60,7 @@ export type UsageSummary = { reasoningTokens?: number; /** Total token count reported by the provider or adapter. */ totalTokens?: number; - /** Cost of the run in US dollars, reported or estimated by the provider or adapter. */ + /** USD-denominated cost attributed to this run. Omit when unknown; zero means known free. */ costUsd?: number; /** Count of tool calls observed during the run. */ toolCalls?: number; diff --git a/packages/core/src/index.test.ts b/packages/core/src/index.test.ts index 2f49acb..e26bcb5 100644 --- a/packages/core/src/index.test.ts +++ b/packages/core/src/index.test.ts @@ -899,6 +899,9 @@ describe("UsageSummarySchema", () => { estimatedCostUsd: 0.02, }).success, ).toBe(false); + expect(UsageSummarySchema.safeParse({ costUsd: -0.01 }).success).toBe( + false, + ); }); }); @@ -932,7 +935,7 @@ describe("collectReportWorkspace", () => { }, }, ]); - expect(workspace.schemaVersion).toBe(1); + expect(workspace.schemaVersion).toBe(2); expect(workspace.cases).toHaveLength(1); expect(workspace.cases[0]).toMatchObject({ displayFile: "apps/demo/evals/refund.eval.ts", diff --git a/packages/core/src/report/workspace.ts b/packages/core/src/report/workspace.ts index 5c0baed..4d5b687 100644 --- a/packages/core/src/report/workspace.ts +++ b/packages/core/src/report/workspace.ts @@ -14,7 +14,7 @@ import { import { VitestJsonStatusSchema } from "./vitest-json"; /** Current schema version for collected report workspaces. */ -export const REPORT_WORKSPACE_SCHEMA_VERSION = 1; +export const REPORT_WORKSPACE_SCHEMA_VERSION = 2; /** One collected Vitest JSON report source in a multi-run workspace. */ export const ReportRunSchema = z diff --git a/packages/docs/src/content/docs/api.mdx b/packages/docs/src/content/docs/api.mdx index b87fbac..12f89c2 100644 --- a/packages/docs/src/content/docs/api.mdx +++ b/packages/docs/src/content/docs/api.mdx @@ -38,7 +38,7 @@ type parameters, options, and examples. UsageSummary - Stable provider, model, token, tool, retry, and metadata usage fields. + Stable provider, model, token, USD cost, tool, retry, and metadata usage fields. diff --git a/packages/docs/src/content/docs/docs/judges/custom.mdx b/packages/docs/src/content/docs/docs/judges/custom.mdx index 603012f..49704d0 100644 --- a/packages/docs/src/content/docs/docs/judges/custom.mdx +++ b/packages/docs/src/content/docs/docs/judges/custom.mdx @@ -98,8 +98,9 @@ export const RubricJudge = createJudge({ Return a full `HarnessRun` from a custom judge harness when the provider exposes usage. Set `usage.costUsd` from provider-reported cost or a suite estimate. -GitHub reports show application, judge, and combined token and cost totals; -provider-specific pricing details can remain under `usage.metadata`. +Omit it when cost is unknown; zero means the run is known to be free. GitHub +reports show application, judge, and combined token and cost totals. +Provider-specific pricing details can remain under `usage.metadata`. ```ts title="evals/judges/judgeHarness.ts" const judgeHarness = createJudgeHarness({ diff --git a/packages/github-reporter/src/collect.ts b/packages/github-reporter/src/collect.ts index 608dab9..e136811 100644 --- a/packages/github-reporter/src/collect.ts +++ b/packages/github-reporter/src/collect.ts @@ -8,13 +8,13 @@ import { collectReportWorkspace, } from "@vitest-evals/core"; import type { + AggregatedUsageSummary, CollectOptions, EvalCase, EvalFailure, EvalReport, EvalScore, ToolCallSummary, - UsageSummary, VitestJsonReport, } from "./types"; import { compactLine, stringifyValue } from "./utils"; @@ -37,9 +37,8 @@ export function collectEvalReport( const evalScores = cases .map((testCase) => testCase.eval?.avgScore) .filter((score): score is number => isFiniteNumber(score)); - const appUsage = sumAppUsage(cases); + const usage = sumAppUsage(cases); const judgeUsage = sumJudgeUsage(cases); - const usage = addUsage(appUsage, judgeUsage); const durationMs = workspace.runs[0]?.durationMs; return { @@ -66,7 +65,6 @@ export function collectEvalReport( } : undefined, usage, - appUsage, judgeUsage, cases, failures, @@ -236,19 +234,18 @@ function stringifyReason(value: unknown) { return typeof value === "string" ? value : stringifyValue(value, 4000); } -function emptyUsage(): Required { +function emptyUsage(): AggregatedUsageSummary { return { inputTokens: 0, outputTokens: 0, reasoningTokens: 0, totalTokens: 0, - costUsd: 0, toolCalls: 0, }; } function addRunUsage( - total: Required, + total: AggregatedUsageSummary, usage: HarnessUsageSummary | undefined, ) { total.inputTokens += usage?.inputTokens ?? 0; @@ -259,24 +256,12 @@ function addRunUsage( (usage?.inputTokens ?? 0) + (usage?.outputTokens ?? 0) + (usage?.reasoningTokens ?? 0); - total.costUsd += usage?.costUsd ?? 0; + if (usage?.costUsd !== undefined) { + total.costUsd = (total.costUsd ?? 0) + usage.costUsd; + } total.toolCalls += usage?.toolCalls ?? 0; } -function addUsage( - left: Required, - right: Required, -): Required { - return { - inputTokens: left.inputTokens + right.inputTokens, - outputTokens: left.outputTokens + right.outputTokens, - reasoningTokens: left.reasoningTokens + right.reasoningTokens, - totalTokens: left.totalTokens + right.totalTokens, - costUsd: left.costUsd + right.costUsd, - toolCalls: left.toolCalls + right.toolCalls, - }; -} - function sumAppUsage(cases: EvalCase[]) { const usage = emptyUsage(); for (const testCase of cases) { diff --git a/packages/github-reporter/src/merge.ts b/packages/github-reporter/src/merge.ts index d9083f7..7de017c 100644 --- a/packages/github-reporter/src/merge.ts +++ b/packages/github-reporter/src/merge.ts @@ -1,4 +1,4 @@ -import type { EvalReport, UsageSummary } from "./types"; +import type { AggregatedUsageSummary, EvalReport } from "./types"; /** Merges multiple collected eval reports into one combined report. */ export function mergeEvalReports(reports: EvalReport[]): EvalReport { @@ -44,20 +44,24 @@ export function mergeEvalReports(reports: EvalReport[]): EvalReport { } : undefined, usage: mergeUsage(reports.map((report) => report.usage)), - appUsage: mergeUsage(reports.map((report) => report.appUsage)), judgeUsage: mergeUsage(reports.map((report) => report.judgeUsage)), cases, failures, }; } -function mergeUsage(usages: Array>) { +function mergeUsage(usages: AggregatedUsageSummary[]): AggregatedUsageSummary { + const costs = usages + .map((usage) => usage.costUsd) + .filter((cost): cost is number => cost !== undefined); return { inputTokens: sum(usages, (usage) => usage.inputTokens), outputTokens: sum(usages, (usage) => usage.outputTokens), reasoningTokens: sum(usages, (usage) => usage.reasoningTokens), totalTokens: sum(usages, (usage) => usage.totalTokens), - costUsd: sum(usages, (usage) => usage.costUsd), + ...(costs.length > 0 + ? { costUsd: costs.reduce((total, cost) => total + cost, 0) } + : {}), toolCalls: sum(usages, (usage) => usage.toolCalls), }; } diff --git a/packages/github-reporter/src/report.test.ts b/packages/github-reporter/src/report.test.ts index bd26bac..a4e5330 100644 --- a/packages/github-reporter/src/report.test.ts +++ b/packages/github-reporter/src/report.test.ts @@ -176,19 +176,15 @@ describe("collectEvalReport", () => { average: 0.2, minimum: 0.2, }); - expect(report.appUsage).toMatchObject({ + expect(report.usage).toMatchObject({ totalTokens: 1220, costUsd: 0.08, + toolCalls: 2, }); expect(report.judgeUsage).toMatchObject({ totalTokens: 100, costUsd: 0.02, }); - expect(report.usage).toMatchObject({ - totalTokens: 1320, - costUsd: 0.1, - toolCalls: 2, - }); expect(report.failures[0]).toMatchObject({ displayFile: "apps/demo/evals/refund.eval.ts", displayName: "refund agent > rejects fraud", @@ -416,7 +412,8 @@ describe("mergeEvalReports", () => { average: 0.5, minimum: 0.2, }); - expect(report.usage.totalTokens).toBe(1620); + expect(report.usage.totalTokens).toBe(1520); + expect(report.judgeUsage.totalTokens).toBe(100); expect(report.usage.toolCalls).toBe(3); expect(report.cases).toHaveLength(2); expect(report.failures).toHaveLength(1); @@ -927,7 +924,9 @@ describe("renderJobSummary", () => { expect(summary).toContain("| Pass Rate | 0.0% |"); expect(summary).not.toContain("| Tests |"); expect(summary).toContain("| Score | avg 0.20, min 0.20 |"); - expect(summary).toContain("| App Usage | 1,220 tokens, $0.08, 2 tools |"); + expect(summary).toContain( + "| Application Usage | 1,220 tokens, $0.08, 2 tools |", + ); expect(summary).toContain("| Judge Usage | 100 tokens, $0.02 |"); expect(summary).toContain("| Total Usage | 1,320 tokens, $0.10, 2 tools |"); expect(summary).toContain("## Scores"); @@ -960,6 +959,21 @@ describe("renderJobSummary", () => { expect(details).not.toContain("Final:\n\n```text"); }); + test("does not present a partial cost as the combined total", () => { + const json = structuredClone(sampleJson); + const usage = (json.testResults[0]?.assertionResults[0]?.meta as any) + .harness.run.usage; + usage.costUsd = undefined; + + const summary = renderJobSummary( + collectEvalReport(json, { workspace: "/repo" }), + ); + + expect(summary).toContain("| Application Usage | 1,220 tokens, 2 tools |"); + expect(summary).toContain("| Judge Usage | 100 tokens, $0.02 |"); + expect(summary).toContain("| Total Usage | 1,320 tokens, 2 tools |"); + }); + test("surfaces non-eval failures without pretending the run passed", () => { const json: VitestJsonReport = { ...sampleJson, diff --git a/packages/github-reporter/src/summary.ts b/packages/github-reporter/src/summary.ts index 72b68e2..860d154 100644 --- a/packages/github-reporter/src/summary.ts +++ b/packages/github-reporter/src/summary.ts @@ -1,10 +1,10 @@ import type { EvalGateResult } from "./gate"; import { formatPercent } from "./gate"; import type { + AggregatedUsageSummary, EvalCase, EvalReport, ToolCallSummary, - UsageSummary, } from "./types"; import { compactLine, @@ -119,10 +119,13 @@ function renderSummaryTable( rows.push(["Score", formatScoreSummary(report.score)]); } - if (hasUsage(report.usage)) { - rows.push(["App Usage", formatUsage(report.appUsage)]); + if (hasUsage(report.usage) || hasUsage(report.judgeUsage)) { + rows.push(["Application Usage", formatUsage(report.usage)]); rows.push(["Judge Usage", formatUsage(report.judgeUsage)]); - rows.push(["Total Usage", formatUsage(report.usage)]); + rows.push([ + "Total Usage", + formatUsage(sumUsage(report.usage, report.judgeUsage)), + ]); } if (gate?.enforced) { @@ -156,16 +159,43 @@ function formatScoreSummary(score: NonNullable) { }`; } -function hasUsage(usage: Required) { - return usage.totalTokens > 0 || usage.costUsd > 0 || usage.toolCalls > 0; +function hasUsage(usage: AggregatedUsageSummary) { + return ( + usage.totalTokens > 0 || usage.costUsd !== undefined || usage.toolCalls > 0 + ); +} + +function sumUsage( + app: AggregatedUsageSummary, + judge: AggregatedUsageSummary, +): AggregatedUsageSummary { + const appCostKnown = !hasUsageWithoutCost(app); + const judgeCostKnown = !hasUsageWithoutCost(judge); + return { + inputTokens: app.inputTokens + judge.inputTokens, + outputTokens: app.outputTokens + judge.outputTokens, + reasoningTokens: app.reasoningTokens + judge.reasoningTokens, + totalTokens: app.totalTokens + judge.totalTokens, + ...(appCostKnown && judgeCostKnown + ? { costUsd: (app.costUsd ?? 0) + (judge.costUsd ?? 0) } + : {}), + toolCalls: app.toolCalls + judge.toolCalls, + }; +} + +function hasUsageWithoutCost(usage: AggregatedUsageSummary) { + return ( + usage.costUsd === undefined && + (usage.totalTokens > 0 || usage.toolCalls > 0) + ); } -function formatUsage(usage: Required) { +function formatUsage(usage: AggregatedUsageSummary) { const parts: string[] = []; if (usage.totalTokens > 0) { parts.push(`${formatNumber(usage.totalTokens)} tokens`); } - if (usage.costUsd > 0) { + if (usage.costUsd !== undefined) { parts.push( usage.costUsd.toLocaleString("en-US", { style: "currency", diff --git a/packages/github-reporter/src/types.ts b/packages/github-reporter/src/types.ts index 243ce72..54c2945 100644 --- a/packages/github-reporter/src/types.ts +++ b/packages/github-reporter/src/types.ts @@ -67,12 +67,10 @@ export type EvalReport = { average: number; minimum?: number; }; - /** Combined application and judge usage. */ - usage: Required; /** Application and other upstream usage. */ - appUsage: Required; + usage: AggregatedUsageSummary; /** Judge-model usage. */ - judgeUsage: Required; + judgeUsage: AggregatedUsageSummary; cases: EvalCase[]; failures: EvalCase[]; }; @@ -87,6 +85,10 @@ export type UsageSummary = { toolCalls?: number; }; +/** Aggregated usage; cost stays absent when no run reported one. */ +export type AggregatedUsageSummary = Required> & + Pick; + /** Tool-call summary shown in reporter output. */ export type ToolCallSummary = { name: string; diff --git a/packages/report-ui/src/app/App.test.ts b/packages/report-ui/src/app/App.test.ts index df8a5e6..2e30541 100644 --- a/packages/report-ui/src/app/App.test.ts +++ b/packages/report-ui/src/app/App.test.ts @@ -97,7 +97,7 @@ describe("visible summary", () => { expect( summarizeVisibleWorkspace( { - schemaVersion: 1, + schemaVersion: 2, runs, cases: [ cases[0]!, @@ -163,7 +163,7 @@ describe("visible summary", () => { expect( summarizeVisibleWorkspace( { - schemaVersion: 1, + schemaVersion: 2, runs: [ { id: "run-1", diff --git a/packages/report-ui/src/app/model.test.ts b/packages/report-ui/src/app/model.test.ts index d9adf73..4d8e688 100644 --- a/packages/report-ui/src/app/model.test.ts +++ b/packages/report-ui/src/app/model.test.ts @@ -14,7 +14,7 @@ import { } from "./model"; const workspace: ReportWorkspace = { - schemaVersion: 1, + schemaVersion: 2, runs: [ { id: "shard-a.json", diff --git a/packages/report-ui/src/server.test.ts b/packages/report-ui/src/server.test.ts index aadcc7e..52361a2 100644 --- a/packages/report-ui/src/server.test.ts +++ b/packages/report-ui/src/server.test.ts @@ -7,7 +7,7 @@ import { parseCliArgs } from "./cli-options"; import { serveReportWorkspace } from "./server"; const workspace: ReportWorkspace = { - schemaVersion: 1, + schemaVersion: 2, runs: [ { id: "vitest-results.json", @@ -93,7 +93,7 @@ describe("serveReportWorkspace", () => { try { const dataResponse = await fetch(`${server.url}/data/workspace.json`); await expect(dataResponse.json()).resolves.toMatchObject({ - schemaVersion: 1, + schemaVersion: 2, cases: [{ id: "case-1" }], }); From bd4630fdd7fa0dd29abd6c3ab409187496666ff2 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 04:53:00 +0000 Subject: [PATCH 06/15] docs: note usage schema compatibility Co-Authored-By: David Cramer --- docs/architecture.md | 5 +++-- docs/github-actions.md | 4 +++- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index dd7df7e..03cc965 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -176,8 +176,9 @@ integration, GitHub reporter, and report UI. Its main entry stays browser-safe, while `@vitest-evals/core/node` exposes filesystem helpers for local and CI report consumers. The collected workspace is versioned. Judge runs and standardized cost change its persisted shape in schema version 2; readers reject unsupported versions -instead of silently misreading them. It exports schemas, TypeScript types, and -helpers for: +instead of silently misreading them. Raw Vitest `task.meta` is strict but not +independently versioned, so producers and artifact readers must use compatible +package releases. Core exports schemas, TypeScript types, and helpers for: - JSON-safe values - normalized harness runs, sessions, transcript-derived tool calls, usage, diff --git a/docs/github-actions.md b/docs/github-actions.md index d4bb25c..799a62e 100644 --- a/docs/github-actions.md +++ b/docs/github-actions.md @@ -12,7 +12,9 @@ Use JSON as the eval artifact because it preserves `task.meta.eval` and reports show separate application, judge, and combined token and `costUsd` totals. A combined cost is shown only when every usage category reports cost; omitted cost means unknown, while zero means known free. Judge runs remain -attached to their scores for detailed usage data. +attached to their scores for detailed usage data. Keep the package producing +the JSON artifact and the `getsentry/vitest-evals` action on compatible +releases; raw Vitest task metadata is strict and not independently versioned. ## Minimal Workflow From 98d079faff1850c37d4c2c1985612bb7011d15cf Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 05:31:26 +0000 Subject: [PATCH 07/15] docs: align usage and judge guidance Co-Authored-By: David Cramer --- AGENTS.md | 2 +- packages/docs/public/llms.txt | 48 ++++++++++++++++--- .../docs/src/content/docs/docs/github.mdx | 17 +++++++ .../docs/src/content/docs/docs/harnesses.mdx | 5 +- skills/vitest-evals/SKILL.md | 7 +-- skills/vitest-evals/SPEC.md | 5 +- .../vitest-evals/references/custom-harness.md | 14 ++++-- .../vitest-evals/references/harness-ai-sdk.md | 26 +++++++--- .../vitest-evals/references/harness-pi-ai.md | 5 +- .../references/judges-and-assertions.md | 38 ++++++++------- .../references/suite-authoring.md | 4 +- .../references/troubleshooting.md | 3 +- 12 files changed, 127 insertions(+), 47 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 155b3e6..23bd868 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,7 +29,7 @@ apps/ # demo-ai-sdk, demo-openai-agents, demo-pi (realistic - Root API work is harness-first and judge-first. - Legacy scorer-first changes stay under `packages/vitest-evals/src/legacy/...`. - Keep normalized session and run data JSON-serializable. -- Keep `UsageSummary` to stable usage units such as tokens, tools, retries, provider, and model. Do not add first-class cost fields; provider-specific cost estimates belong in `usage.metadata`. +- Keep `UsageSummary` to stable usage units such as tokens, USD cost, tools, retries, provider, and model. Set `costUsd` only when cost is known; provider-specific pricing details belong in `usage.metadata`. - GitHub reporting is action-first: document workflows with `uses: getsentry/vitest-evals@v0`, not CLI commands. - Sharded eval reporting uses distinct JSON artifacts per matrix job and one reducer job that publishes the combined action report. - Root GitHub Action releases keep source and bundled tags separate: Craft publishes the source release, `vX.Y.Z-src` preserves the source baseline, and `vX`/`vX.Y.Z` point at the bundled action commit. Keep the Craft `github` target filtered away from package artifacts. diff --git a/packages/docs/public/llms.txt b/packages/docs/public/llms.txt index d780fc6..653ac4e 100644 --- a/packages/docs/public/llms.txt +++ b/packages/docs/public/llms.txt @@ -353,12 +353,14 @@ duration, and metadata. Tool-result events have `type: "tool_result"`, required - `outputTokens` - `reasoningTokens` - `totalTokens` +- `costUsd` - `toolCalls` - `retries` - `metadata` -Do not add first-class cost fields to usage. Provider-specific cost estimates -belong in `usage.metadata`. +Set `costUsd` to the provider-reported cost or the suite's estimate in US +dollars. Omit it when cost is unknown. Use `0` only when the run is known to be +free. Keep provider-specific pricing details in `usage.metadata`. ## First-party harness adapters @@ -461,6 +463,7 @@ The common judge context includes: - `session` - `toolCalls` - `harness` +- `runJudge` when a judge harness is configured - `signal` Create a deterministic judge: @@ -515,9 +518,38 @@ order, avoiding extra calls, or matching arguments. `StructuredOutputJudge()` checks JSON-safe structured output fields. Use it when correctness depends on app-facing output shape or values. -Built-ins are deterministic contract checks. For model-graded evals, put model -selection, prompt text, rubrics, parsing, and thresholding inside a custom -judge. +Built-ins other than `FactualityJudge()` are deterministic checks. For a +model-backed judge, keep the prompt, rubric, parsing, and threshold in the +judge. Put the provider call in a separate `judgeHarness`, then call it through +`ctx.runJudge(...)`. This records tokens and cost as judge usage instead of +application usage. + +```ts +import { aiSdkJudgeHarness } from "@vitest-evals/harness-ai-sdk"; +import { createJudge } from "vitest-evals"; + +const judgeHarness = aiSdkJudgeHarness({ model: rubricModel }); + +const RubricJudge = createJudge({ + name: "RubricJudge", + judgeHarness, + async assess(ctx) { + if (!ctx.runJudge) { + throw new Error("RubricJudge requires a judge harness."); + } + + const verdict = await ctx.runJudge({ + prompt: formatRubricPrompt(ctx), + responseFormat: { type: "json" }, + }); + return parseVerdict(verdict); + }, +}); +``` + +Use a first-party judge harness from the matching adapter package, or +`createJudgeHarness(...)` for another provider. Return a full `HarnessRun` when +usage is available. Each `runJudge(...)` call is recorded separately. ## Case criteria guidance @@ -555,7 +587,7 @@ The reporter surfaces: - judge metadata/rationales - app output summaries - tool activity -- usage summaries +- separate application, judge, and total token and USD cost summaries - replay metadata - non-eval failures without pretending the run passed @@ -630,6 +662,10 @@ Action inputs: gate. Defaults to true only after a successful Check Run publish. - `min-pass-rate` / `min-score-average`: optional aggregate floors in `0`-`1`. +The job summary and Check Run show application usage, judge usage, and their +combined total separately. If either side has usage but no cost, the combined +cost is omitted. + Check Run SHA resolution order on `pull_request`: 1. explicit `sha` input diff --git a/packages/docs/src/content/docs/docs/github.mdx b/packages/docs/src/content/docs/docs/github.mdx index 4374f75..1e42c5b 100644 --- a/packages/docs/src/content/docs/docs/github.mdx +++ b/packages/docs/src/content/docs/docs/github.mdx @@ -71,6 +71,23 @@ the gate decision (for example `Eval pass rate 63.7% — required 80.0%`) instea of failing on any single case. Quality misses stay visible as warnings when the suite is still above the floor. +## Usage And Cost + +The job summary and Check Run show three rows: + +- **Application Usage**: the app under test, including upstream model calls + collected by its harness +- **Judge Usage**: model calls made through `ctx.runJudge(...)` +- **Total Usage**: application and judge usage combined + +Each row can include tokens, USD cost, and tool calls. Harnesses should set +`usage.costUsd` to the provider-reported cost or the suite's estimate. Omit it +when cost is unknown; use `0` only for a run known to be free. + +If either application or judge usage has no cost, the combined cost is omitted +instead of showing a partial total. Sharded reports sum each usage group across +all result files before calculating the combined row. + ## Check Run Ownership A Check Run is a **separate** GitHub check from the workflow job row. Use it diff --git a/packages/docs/src/content/docs/docs/harnesses.mdx b/packages/docs/src/content/docs/docs/harnesses.mdx index 2c5c800..764a2cb 100644 --- a/packages/docs/src/content/docs/docs/harnesses.mdx +++ b/packages/docs/src/content/docs/docs/harnesses.mdx @@ -46,7 +46,10 @@ same shape regardless of runtime: | `session.events` | Ordered transcript events, including messages, tool-call requests, and tool results. | | `toolCalls(result)` | Deterministic tool-call projection for tool judges and replay checks. | | `traces` | Operation spans for runs, model calls, tools, guardrails, handoffs, or custom work. | -| `usage` | Stable usage units such as provider, model, tokens, tools, and retries. | +| `usage` | Provider, model, tokens, USD cost, tools, and retries. Omit unknown values. | + +Set `usage.costUsd` to the provider-reported cost or the suite's estimate. Omit +it when cost is unknown; use `0` only when the run is known to be free. First-party harnesses attach trace spans automatically from native runtime data they observe. `createHarness()` attaches a fallback run span when a custom diff --git a/skills/vitest-evals/SKILL.md b/skills/vitest-evals/SKILL.md index f83bdc1..3855783 100644 --- a/skills/vitest-evals/SKILL.md +++ b/skills/vitest-evals/SKILL.md @@ -35,9 +35,10 @@ Use the harness-backed API as the only authoring model. - Use `toolCalls(result)` and message helpers for normalized session assertions. - Use `spans(result)`, `spansByKind(result, kind)`, and `failedSpans(result)` for span assertions. - Keep `HarnessRun`, `NormalizedSession`, usage, artifacts, and tool records JSON-serializable. -- Keep judge model calls on judges. Use `createJudge("Name", assess)` for - custom judges; use the provider-helper overload only when multiple judges - reuse setup and need curried run options. +- Report known tokens and USD cost in `run.usage`. Omit `costUsd` when unknown; + use `0` only when the run is known to be free. +- Keep rubric text and parsing in the judge. Put provider calls in a + `judgeHarness`, then call them through `ctx.runJudge(...)` so usage is recorded. - Put scenario-owned criteria on the input value. Put direct-check expected values in Vitest case rows. Pass per-case judge criteria through explicit matcher options, and suite-wide criteria through judge config. diff --git a/skills/vitest-evals/SPEC.md b/skills/vitest-evals/SPEC.md index 10f4264..d189ab8 100644 --- a/skills/vitest-evals/SPEC.md +++ b/skills/vitest-evals/SPEC.md @@ -33,8 +33,9 @@ Out of scope: - Required first actions: inspect the touched code, choose the runtime target, open the matching reference. - Required outputs: implementation or review guidance that uses the harness-backed API only, plus targeted verification. - Non-negotiable constraints: normalized run data remains JSON-serializable, - suite tests call `run(...)` explicitly, judge model calls stay on judges or - judge-side helpers, and model-backed harnesses normalize tokens and `costUsd`. + suite tests call `run(...)` explicitly, judges call models through a + `judgeHarness` and `ctx.runJudge(...)`, and harnesses normalize tokens and + `costUsd`. - Expected bundled files loaded at runtime: `SKILL.md` first, then one or more focused files under `references/`. ## Source And Evidence Model diff --git a/skills/vitest-evals/references/custom-harness.md b/skills/vitest-evals/references/custom-harness.md index ece50a9..aedbd44 100644 --- a/skills/vitest-evals/references/custom-harness.md +++ b/skills/vitest-evals/references/custom-harness.md @@ -25,9 +25,10 @@ const appHarness: Harness = { return { output, session: { - messages: [ - { role: "user", content: normalizeContent(input) }, + events: [ + { type: "message", role: "user", content: normalizeContent(input) }, { + type: "message", role: "assistant", content: normalizeContent(appResult.reply), metadata: normalizeMetadata({ channel: appResult.channel }), @@ -35,7 +36,10 @@ const appHarness: Harness = { ], metadata: normalizeMetadata({ caseId: appResult.caseId }), }, - usage: appResult.usage ?? {}, + usage: { + ...appResult.usage, + costUsd: appResult.costUsd, + }, errors: [], }; }, @@ -48,8 +52,8 @@ const appHarness: Harness = { |-------|-------------| | `name` | Stable short label shown in reporter output. | | `run` | Executes the application once and returns a normalized `HarnessRun`. | -| `session.messages` | JSON-safe user, assistant, and tool trace. | -| `usage` | Empty object when unknown; include provider/model/tokens when available. | +| `session.events` | Ordered, JSON-safe messages and tool activity. | +| `usage` | Include provider, model, tokens, tools, and `costUsd` when known. Omit unknown fields. | | `errors` | Empty array on success; serialized error records on partial results. | ## Implementation Rules diff --git a/skills/vitest-evals/references/harness-ai-sdk.md b/skills/vitest-evals/references/harness-ai-sdk.md index 25a4929..035ca8d 100644 --- a/skills/vitest-evals/references/harness-ai-sdk.md +++ b/skills/vitest-evals/references/harness-ai-sdk.md @@ -5,7 +5,7 @@ Open this for `ai` package integrations such as `generateText`, `generateObject` ## Install And Import ```sh -npm install -D ai vitest-evals @vitest-evals/harness-ai-sdk +pnpm add -D ai vitest-evals @vitest-evals/harness-ai-sdk ``` ```ts @@ -44,14 +44,26 @@ const harness = aiSdkHarness({ Agent factories receive `{ input, context }` before execution so apps can derive instructions or seeded state without side-channel setup. -When LLM-backed judges need shared provider setup, keep that helper in the -judge module. Do not put judge-model calls on the app harness: +Use a separate judge harness for rubric calls. This keeps rubric usage out of +application usage and lets reporters show both totals: ```ts -const verdict = await generateText({ - model: rubricModel, - prompt: formatJudgePrompt(ctx), -}).then((result) => result.text); +import { aiSdkJudgeHarness } from "@vitest-evals/harness-ai-sdk"; +import { createJudge } from "vitest-evals"; + +const judgeHarness = aiSdkJudgeHarness({ model: rubricModel }); + +const RubricJudge = createJudge({ + name: "RubricJudge", + judgeHarness, + async assess(ctx) { + if (!ctx.runJudge) { + throw new Error("RubricJudge requires a judge harness."); + } + const verdict = await ctx.runJudge({ prompt: formatJudgePrompt(ctx) }); + return parseVerdict(verdict); + }, +}); ``` ## Normalization Behavior diff --git a/skills/vitest-evals/references/harness-pi-ai.md b/skills/vitest-evals/references/harness-pi-ai.md index d2514e4..aa07d6e 100644 --- a/skills/vitest-evals/references/harness-pi-ai.md +++ b/skills/vitest-evals/references/harness-pi-ai.md @@ -5,7 +5,7 @@ Open this for Pi AI or Pi Mono-style agents. ## Install And Import ```sh -npm install -D vitest-evals @vitest-evals/harness-pi-ai +pnpm add -D vitest-evals @vitest-evals/harness-pi-ai ``` ```ts @@ -68,7 +68,8 @@ const harness = piAiHarness({ - Starts sessions with a user message for the input. - Uses recorded runtime events as normalized messages. - Adds tool calls from wrapped runtime tools and native tool instrumentation. -- Reads usage from `usage` or `metrics`, and adds tool call counts when absent. +- Reads usage from `usage` or `metrics`, adds tool counts when absent, and maps + Pi's total USD cost to `costUsd` when available. - Uses `result.session` or `result.trace` directly when they already match `NormalizedSession`. - Attaches partial runs to thrown errors. diff --git a/skills/vitest-evals/references/judges-and-assertions.md b/skills/vitest-evals/references/judges-and-assertions.md index 9105151..2ac8263 100644 --- a/skills/vitest-evals/references/judges-and-assertions.md +++ b/skills/vitest-evals/references/judges-and-assertions.md @@ -13,33 +13,37 @@ Every judge receives: | `toolCalls` | Flattened calls from the run. | | `run` | Full normalized `HarnessRun`. | | `session` | Normalized session from the run. | -| `harness` | Suite harness for intentional second runs. | +| `harness` | App harness for intentional second runs. | +| `runJudge` | Runs the configured judge harness and records its usage. | ## Custom Judge Pattern ```ts -import { - createJudge, -} from "vitest-evals"; - -const RefundRubricJudge = createJudge< - string, - RefundOutput, - { expectedStatus: string } ->( - "RefundRubricJudge", - async (ctx) => { - const verdict = await callJudgeModel({ +import { aiSdkJudgeHarness } from "@vitest-evals/harness-ai-sdk"; +import { createJudge } from "vitest-evals"; + +const judgeHarness = aiSdkJudgeHarness({ model: rubricModel }); + +const RefundRubricJudge = createJudge({ + name: "RefundRubricJudge", + judgeHarness, + async assess(ctx) { + if (!ctx.runJudge) { + throw new Error("RefundRubricJudge requires a judge harness."); + } + + const verdict = await ctx.runJudge({ prompt: formatRubric({ input: ctx.input, output: ctx.output, expectedStatus: ctx.expectedStatus, }), + responseFormat: { type: "json" }, }); return parseVerdict(verdict); }, -); +}); ``` ## Automatic Vs Explicit @@ -52,7 +56,7 @@ const RefundRubricJudge = createJudge< | Only one assertion needs the judge | `await expect(result).toSatisfyJudge(Judge)` | | Judge needs structured app output | Type `JudgeContext<..., TOutput>` and read `ctx.output` | | Judge needs text | Use a text `TOutput` or explicitly project structured output to text | -| Judge needs shared model setup | Keep a local helper in the judge module, or use the provider-helper overload of `createJudge(...)` when curried run options are needed | +| Judge needs a model call | Configure a `judgeHarness`, then call `ctx.runJudge(...)` | ## Built-In Judges @@ -87,7 +91,7 @@ const RefundRubricJudge = createJudge< - Custom judges should usually use `createJudge("Name", assess)`. - Scores are numbers or `null`; failure behavior is controlled by thresholds. - Rationale or parsed judge output is placed under `metadata`. -- LLM-backed judges provide the judge prompt/rubric text. Shared provider setup - belongs in a judge-side helper, not on the app harness. +- Model-backed judges own the prompt, rubric, and parser. Provider setup belongs + in a `judgeHarness`, not the app harness. - Model-backed judge harnesses retain normalized usage instead of returning only provider text when usage is available. diff --git a/skills/vitest-evals/references/suite-authoring.md b/skills/vitest-evals/references/suite-authoring.md index cab22c4..93f875d 100644 --- a/skills/vitest-evals/references/suite-authoring.md +++ b/skills/vitest-evals/references/suite-authoring.md @@ -44,8 +44,8 @@ describeEval("refund agent", { harness }, (it) => { ## Normalized Run Checklist - `run.output` is JSON-serializable or omitted. -- `run.session.messages` contains user, assistant, and tool records worth reporting. -- `run.usage` includes provider/model/token/tool data when available. +- `run.session.events` contains the ordered messages and tool activity worth reporting. +- `run.usage` includes provider, model, tokens, tools, and `costUsd` when known. - `run.artifacts` contains only JSON-safe diagnostics set through `context.setArtifact(...)`. - `run.errors` is an array, even on success. diff --git a/skills/vitest-evals/references/troubleshooting.md b/skills/vitest-evals/references/troubleshooting.md index 449568d..fcf1e41 100644 --- a/skills/vitest-evals/references/troubleshooting.md +++ b/skills/vitest-evals/references/troubleshooting.md @@ -8,7 +8,8 @@ Open this for failing evals, missing reporter data, or uncertain verification. |---------|--------------|-----| | Test has no `run` fixture | Suite is not inside `describeEval(...)`. | Wrap the suite with `describeEval(name, { harness }, (it) => { ... })`. | | App executes more than once per test | Judge or assertion calls the app directly. | Use the returned `result`; reserve `harness.run(...)` inside a judge for intentional second runs. | -| Reporter shows little harness detail | `HarnessRun.session`, `usage`, or `errors` is sparse. | Return a richer normalized run with assistant messages. | +| Reporter shows little harness detail | `HarnessRun.session`, `usage`, or `errors` is sparse. | Return a richer run with ordered session events and usage. | +| Usage cost is missing | The harness did not provide `usage.costUsd`. | Map provider cost or a suite estimate to `costUsd`; omit it when unknown. | | Tool calls are missing | Tools were not passed through the harness runtime or native tool inference missed them. | Use wrapped `runtime.tools`, explicit `tools`, or emit events from the app seam. | | Tool arguments/results disappear | Values are not JSON-serializable or normalize to `undefined`. | Convert to records, arrays, strings, numbers, booleans, or `null`. | | Judge receives blank text | The judge expects text but the run output or assistant content is not text. | Use a text `run.output` or project structured output to text inside the judge. | From a3ac1d92b766704ea11754e755d9a6e9abfa33de Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 06:07:48 +0000 Subject: [PATCH 08/15] fix(reporting): complete usage coverage Co-Authored-By: David Cramer --- README.md | 4 +- packages/docs/public/llms.txt | 12 +- .../docs/src/content/docs/docs/report-ui.mdx | 15 +- packages/github-reporter/src/collect.ts | 16 ++ packages/github-reporter/src/merge.ts | 7 +- packages/github-reporter/src/report.test.ts | 20 ++ packages/report-ui/README.md | 3 +- .../fixtures/vitest-results.visual.json | 182 ++++++++++++------ packages/report-ui/src/app/App.test.ts | 4 + .../src/app/components/OverviewTab.tsx | 45 ++++- .../src/app/components/ReportChrome.tsx | 20 +- packages/report-ui/src/app/model.test.ts | 23 ++- packages/report-ui/src/app/model.ts | 58 +++++- packages/report-ui/src/server.test.ts | 25 ++- packages/vitest-evals/README.md | 5 +- packages/vitest-evals/src/reporter.test.ts | 20 +- packages/vitest-evals/src/reporter.ts | 118 +++++++++--- 17 files changed, 446 insertions(+), 131 deletions(-) diff --git a/README.md b/README.md index 961f33c..fa67eaf 100644 --- a/README.md +++ b/README.md @@ -128,8 +128,8 @@ soft-fail defaults, and score-gate options. Use `vitest-evals serve` to inspect one or more JSON reports locally. It accepts files, simple globs, and directories of JSON artifacts, then serves a -React SPA with run summaries, eval cases, harness output, sessions, tools, and -trace details. +React SPA with run summaries, eval cases, separate application and judge usage, +harness output, sessions, tools, and trace details. ```sh pnpm exec vitest-evals serve vitest-results.json diff --git a/packages/docs/public/llms.txt b/packages/docs/public/llms.txt index 653ac4e..19fb397 100644 --- a/packages/docs/public/llms.txt +++ b/packages/docs/public/llms.txt @@ -602,8 +602,9 @@ pnpm exec vitest-evals serve eval-results/ ``` The command accepts JSON files, simple globs, and directories. It opens a local -React report viewer for run summaries, eval cases, scores, harness sessions, -tool calls, session transcripts, trace span details, and raw case metadata. +report viewer for run summaries, eval cases, scores, application and judge +usage, harness sessions, tool calls, transcripts, trace details, and raw case +metadata. Unknown costs stay `n/a`; partial costs are not shown as totals. Use the same JSON artifacts for local investigation and GitHub reporting. @@ -771,13 +772,14 @@ transcript events, usage, and partial output when execution fails. - Keep expected values in case rows, explicit matcher options, or judge config. - Prefer deterministic judges for contracts such as tool calls and structured output. -- Keep model-graded judge prompts and rubrics inside custom judges. +- Keep model-graded prompts and rubrics in the judge. Run provider calls through + `ctx.runJudge(...)` and a separate `judgeHarness`. - Run the app once per eval case, then judge the captured result. - Publish CI results with `getsentry/vitest-evals@v0`. - For sharded CI, emit distinct JSON artifacts per shard and combine them in one reducer reporting job. -- Put provider-specific cost estimates in `usage.metadata`, not as first-class - `UsageSummary` fields. +- Put known USD cost in `usage.costUsd`. Keep provider-specific pricing details + in `usage.metadata`. ## Quick command reference diff --git a/packages/docs/src/content/docs/docs/report-ui.mdx b/packages/docs/src/content/docs/docs/report-ui.mdx index c2fbb42..42a0517 100644 --- a/packages/docs/src/content/docs/docs/report-ui.mdx +++ b/packages/docs/src/content/docs/docs/report-ui.mdx @@ -5,7 +5,8 @@ editUrl: false --- The local report UI reads Vitest JSON artifacts and serves a local viewer for -drilling into runs, cases, scores, harness sessions, tool calls, and trace spans. +drilling into runs, cases, scores, application and judge usage, harness +sessions, tool calls, and trace spans. ## Emit JSON @@ -36,10 +37,14 @@ environment where the artifacts were produced. ## What to Inspect -Use the report UI when a summary is not enough. It is built for comparing runs, -finding failed eval cases, reading judge scores, inspecting harness output, and -following session transcripts with tool call arguments, results, errors, and -related trace span details. +Use the report UI when a summary is not enough. The top summary separates +application, judge, and total tokens and cost. A case's Usage section shows the +same breakdown for that case. If any contributing run has unknown cost, its +combined cost is shown as `n/a` rather than as a partial total. + +You can also compare runs, find failed cases, read judge scores, inspect harness +output, and follow session transcripts with tool arguments, results, errors, +and related trace details. The same JSON artifacts can also be uploaded to the GitHub reporter, so local investigation and CI reporting stay on one durable report format. diff --git a/packages/github-reporter/src/collect.ts b/packages/github-reporter/src/collect.ts index e136811..db5c2d6 100644 --- a/packages/github-reporter/src/collect.ts +++ b/packages/github-reporter/src/collect.ts @@ -264,26 +264,42 @@ function addRunUsage( function sumAppUsage(cases: EvalCase[]) { const usage = emptyUsage(); + const runUsages = cases.flatMap((testCase) => + testCase.harness ? [testCase.harness.usage] : [], + ); for (const testCase of cases) { addRunUsage(usage, testCase.harness?.usage); usage.toolCalls += toolCallCount(testCase) - (testCase.harness?.usage?.toolCalls ?? 0); } + omitPartialCost(usage, runUsages); return usage; } function sumJudgeUsage(cases: EvalCase[]) { const usage = emptyUsage(); + const runUsages: Array = []; for (const testCase of cases) { for (const score of testCase.eval?.scores ?? []) { for (const run of score.judgeRuns ?? []) { + runUsages.push(run.usage); addRunUsage(usage, run.usage); } } } + omitPartialCost(usage, runUsages); return usage; } +function omitPartialCost( + total: AggregatedUsageSummary, + usages: Array, +) { + if (usages.some((usage) => usage?.costUsd === undefined)) { + total.costUsd = undefined; + } +} + function toolCallCount(testCase: EvalCase) { const usageToolCalls = testCase.harness?.usage?.toolCalls; if (usageToolCalls !== undefined) { diff --git a/packages/github-reporter/src/merge.ts b/packages/github-reporter/src/merge.ts index 7de017c..af0fd5e 100644 --- a/packages/github-reporter/src/merge.ts +++ b/packages/github-reporter/src/merge.ts @@ -54,12 +54,17 @@ function mergeUsage(usages: AggregatedUsageSummary[]): AggregatedUsageSummary { const costs = usages .map((usage) => usage.costUsd) .filter((cost): cost is number => cost !== undefined); + const costComplete = !usages.some( + (usage) => + usage.costUsd === undefined && + (usage.totalTokens > 0 || usage.toolCalls > 0), + ); return { inputTokens: sum(usages, (usage) => usage.inputTokens), outputTokens: sum(usages, (usage) => usage.outputTokens), reasoningTokens: sum(usages, (usage) => usage.reasoningTokens), totalTokens: sum(usages, (usage) => usage.totalTokens), - ...(costs.length > 0 + ...(costComplete && costs.length > 0 ? { costUsd: costs.reduce((total, cost) => total + cost, 0) } : {}), toolCalls: sum(usages, (usage) => usage.toolCalls), diff --git a/packages/github-reporter/src/report.test.ts b/packages/github-reporter/src/report.test.ts index a4e5330..f616008 100644 --- a/packages/github-reporter/src/report.test.ts +++ b/packages/github-reporter/src/report.test.ts @@ -413,7 +413,9 @@ describe("mergeEvalReports", () => { minimum: 0.2, }); expect(report.usage.totalTokens).toBe(1520); + expect(report.usage.costUsd).toBeUndefined(); expect(report.judgeUsage.totalTokens).toBe(100); + expect(report.judgeUsage.costUsd).toBe(0.02); expect(report.usage.toolCalls).toBe(3); expect(report.cases).toHaveLength(2); expect(report.failures).toHaveLength(1); @@ -974,6 +976,24 @@ describe("renderJobSummary", () => { expect(summary).toContain("| Total Usage | 1,320 tokens, 2 tools |"); }); + test("does not present a partial cost within one usage category", () => { + const json = structuredClone(sampleJson); + const secondAssertion = structuredClone( + json.testResults[0]!.assertionResults[0]!, + ); + secondAssertion.title = "second case"; + secondAssertion.fullName = "refund agent second case"; + (secondAssertion.meta as any).harness.run.usage = { + totalTokens: 50, + }; + json.testResults[0]!.assertionResults.push(secondAssertion); + + const report = collectEvalReport(json, { workspace: "/repo" }); + + expect(report.usage.totalTokens).toBe(1270); + expect(report.usage.costUsd).toBeUndefined(); + }); + test("surfaces non-eval failures without pretending the run passed", () => { const json: VitestJsonReport = { ...sampleJson, diff --git a/packages/report-ui/README.md b/packages/report-ui/README.md index eac08a9..45a8f21 100644 --- a/packages/report-ui/README.md +++ b/packages/report-ui/README.md @@ -11,7 +11,8 @@ pnpm exec vitest-evals serve eval-results/ The product CLI accepts Vitest JSON result files, simple `*` and `**` globs, and directories containing JSON files. It collects them into the shared `ReportWorkspace` model from `@vitest-evals/core`, serves the SPA locally, and -exposes the collected data at `/data/workspace.json`. +exposes the collected data at `/data/workspace.json`. The viewer separates +application, judge, and total token and cost usage at workspace and case level. For visual QA, build and serve the intentionally awkward fixture: diff --git a/packages/report-ui/fixtures/vitest-results.visual.json b/packages/report-ui/fixtures/vitest-results.visual.json index f3d5b2e..0ef2b67 100644 --- a/packages/report-ui/fixtures/vitest-results.visual.json +++ b/packages/report-ui/fixtures/vitest-results.visual.json @@ -48,7 +48,22 @@ "score": 0.2, "metadata": { "rationale": "status mismatch; expected denied, got manual_review" - } + }, + "judgeRuns": [ + { + "output": { + "choice": "C" + }, + "usage": { + "totalTokens": 96, + "costUsd": 0.012 + }, + "session": { + "events": [] + }, + "errors": [] + } + ] }, { "name": "ToolCallJudge", @@ -72,6 +87,7 @@ "inputTokens": 980, "outputTokens": 240, "totalTokens": 1220, + "costUsd": 0.084, "toolCalls": 3, "metadata": { "cacheReadTokens": 320 @@ -81,55 +97,81 @@ "totalMs": 4100 }, "session": { - "messages": [ + "events": [ { + "type": "message", "role": "system", "content": "You are a refund policy agent. Follow fraud controls exactly." }, { + "type": "message", "role": "user", "content": "Refund invoice inv_404 for account acct_suspicious" }, { + "type": "message", "role": "assistant", - "content": "I need to inspect the invoice and account risk profile.", - "toolCalls": [ - { - "name": "lookupInvoice", - "arguments": { - "invoiceId": "inv_404" - }, - "result": { - "refundable": true, - "amount": 19900, - "currency": "usd" - }, - "durationMs": 6 - }, - { - "name": "inspectAccountRisk", - "arguments": { - "accountId": "acct_suspicious" - }, - "result": { - "risk": "high", - "signals": ["velocity", "chargeback"] - }, - "durationMs": 18 - }, - { - "name": "createRefund", - "arguments": { - "invoiceId": "inv_404" - }, - "error": { - "message": "blocked by fraud policy" - }, - "durationMs": 11 - } - ] + "content": "I need to inspect the invoice and account risk profile." }, { + "type": "tool_call", + "id": "visual-call-1", + "name": "lookupInvoice", + "arguments": { + "invoiceId": "inv_404" + }, + "durationMs": 6 + }, + { + "type": "tool_result", + "toolCallId": "visual-call-1", + "name": "lookupInvoice", + "durationMs": 6, + "content": { + "refundable": true, + "amount": 19900, + "currency": "usd" + } + }, + { + "type": "tool_call", + "id": "visual-call-2", + "name": "inspectAccountRisk", + "arguments": { + "accountId": "acct_suspicious" + }, + "durationMs": 18 + }, + { + "type": "tool_result", + "toolCallId": "visual-call-2", + "name": "inspectAccountRisk", + "durationMs": 18, + "content": { + "risk": "high", + "signals": ["velocity", "chargeback"] + } + }, + { + "type": "tool_call", + "id": "visual-call-3", + "name": "createRefund", + "arguments": { + "invoiceId": "inv_404" + }, + "durationMs": 11 + }, + { + "type": "tool_result", + "toolCallId": "visual-call-3", + "name": "createRefund", + "durationMs": 11, + "error": { + "message": "blocked by fraud policy" + } + }, + { + "type": "message", "role": "assistant", "content": "This should be sent to manual review because the account risk is high." } @@ -320,24 +362,28 @@ "totalMs": 2100 }, "session": { - "messages": [ + "events": [ { + "type": "message", "role": "user", "content": "Refund invoice inv_123" }, { + "type": "message", "role": "assistant", - "content": "The invoice is refundable and the refund has been created.", - "toolCalls": [ - { - "name": "lookupInvoice", - "durationMs": 4 - }, - { - "name": "createRefund", - "durationMs": 21 - } - ] + "content": "The invoice is refundable and the refund has been created." + }, + { + "type": "tool_call", + "id": "visual-call-1", + "name": "lookupInvoice", + "durationMs": 4 + }, + { + "type": "tool_call", + "id": "visual-call-2", + "name": "createRefund", + "durationMs": 21 } ] }, @@ -475,26 +521,34 @@ "totalMs": 1800 }, "session": { - "messages": [ + "events": [ { + "type": "message", "role": "user", "content": "how long do i have to get money back???" }, { + "type": "message", "role": "assistant", - "content": "The refund window is 30 days for standard purchases.", - "toolCalls": [ - { - "name": "searchKnowledgeBase", - "arguments": { - "query": "standard purchase refund window" - }, - "result": { - "documents": 3 - }, - "durationMs": 33 - } - ] + "content": "The refund window is 30 days for standard purchases." + }, + { + "type": "tool_call", + "id": "visual-call-1", + "name": "searchKnowledgeBase", + "arguments": { + "query": "standard purchase refund window" + }, + "durationMs": 33 + }, + { + "type": "tool_result", + "toolCallId": "visual-call-1", + "name": "searchKnowledgeBase", + "durationMs": 33, + "content": { + "documents": 3 + } } ] }, diff --git a/packages/report-ui/src/app/App.test.ts b/packages/report-ui/src/app/App.test.ts index 2e30541..d923c2f 100644 --- a/packages/report-ui/src/app/App.test.ts +++ b/packages/report-ui/src/app/App.test.ts @@ -201,6 +201,8 @@ describe("visible summary", () => { passed: 1, runCount: 1, skipped: 1, + appTokens: 0, + judgeTokens: 0, totalTokens: 0, toolCallCount: 0, }), @@ -215,6 +217,8 @@ describe("visible summary", () => { passed: 0, runCount: 1, skipped: 1, + appTokens: 0, + judgeTokens: 0, totalTokens: 0, toolCallCount: 0, }), diff --git a/packages/report-ui/src/app/components/OverviewTab.tsx b/packages/report-ui/src/app/components/OverviewTab.tsx index c8585f6..cfa6dc1 100644 --- a/packages/report-ui/src/app/components/OverviewTab.tsx +++ b/packages/report-ui/src/app/components/OverviewTab.tsx @@ -1,5 +1,10 @@ import type { HarnessRun, ReportCase } from "@vitest-evals/core"; -import { formatDuration, formatNumber } from "../model"; +import { + formatDuration, + formatNumber, + formatUsd, + summarizeRuns, +} from "../model"; import { EmptyState } from "../ui"; import { DetailContent, DetailSection } from "./DetailLayout"; import { Fact, FactsGrid, JsonBlock, ScoreValue } from "./ReportPrimitives"; @@ -20,7 +25,7 @@ export function OverviewTab({ - + {testCase.failureMessages.length > 0 ? ( @@ -139,16 +144,38 @@ function EvidenceLine({ label, value }: { label: string; value: string }) { ); } -function UsageGrid({ run }: { run: HarnessRun | undefined }) { +function UsageGrid({ + testCase, + run, +}: { + testCase: ReportCase; + run: HarnessRun | undefined; +}) { const usage = run?.usage; + const app = summarizeRuns(run ? [run] : []); + const judge = summarizeRuns( + (testCase.eval?.scores ?? []).flatMap((score) => score.judgeRuns ?? []), + ); + const totalCost = + (app.runCount === 0 || app.costUsd !== undefined) && + (judge.runCount === 0 || judge.costUsd !== undefined) && + app.runCount + judge.runCount > 0 + ? (app.costUsd ?? 0) + (judge.costUsd ?? 0) + : undefined; + return ( - - - - - - + + + + + + + + diff --git a/packages/report-ui/src/app/components/ReportChrome.tsx b/packages/report-ui/src/app/components/ReportChrome.tsx index 8749caa..798555e 100644 --- a/packages/report-ui/src/app/components/ReportChrome.tsx +++ b/packages/report-ui/src/app/components/ReportChrome.tsx @@ -3,6 +3,7 @@ import { formatDuration, formatNumber, formatScore, + formatUsd, type summarizeWorkspace, } from "../model"; import { cx, toneTextClass, type Tone } from "../ui"; @@ -126,14 +127,16 @@ export function SummaryBar({ @@ -200,7 +203,15 @@ export function RunStrip({ ); } -function SummaryCounter({ label, value }: { label: string; value: string }) { +function SummaryCounter({ + label, + value, + detail, +}: { + label: string; + value: string; + detail?: string; +}) { return (
@@ -209,6 +220,9 @@ function SummaryCounter({ label, value }: { label: string; value: string }) {
{value}
+ {detail ? ( +
{detail}
+ ) : null}
); } diff --git a/packages/report-ui/src/app/model.test.ts b/packages/report-ui/src/app/model.test.ts index 4d8e688..c704da9 100644 --- a/packages/report-ui/src/app/model.test.ts +++ b/packages/report-ui/src/app/model.test.ts @@ -49,7 +49,20 @@ const workspace: ReportWorkspace = { failureMessages: ["Score: 0.20 below threshold: 1.00"], eval: { avgScore: 0.2, - scores: [{ name: "StructuredOutputJudge", score: 0.2 }], + scores: [ + { + name: "StructuredOutputJudge", + score: 0.2, + judgeRuns: [ + { + output: { choice: "C" }, + usage: { totalTokens: 100, costUsd: 0.02 }, + session: { events: [] }, + errors: [], + }, + ], + }, + ], }, harness: { name: "pi-ai", @@ -57,6 +70,7 @@ const workspace: ReportWorkspace = { output: { status: "denied" }, usage: { totalTokens: 1220, + costUsd: 0.08, toolCalls: 1, }, session: { @@ -139,7 +153,12 @@ describe("summarizeWorkspace", () => { passed: 1, failed: 1, averageScore: 0.6, - totalTokens: 1220, + appTokens: 1220, + judgeTokens: 100, + totalTokens: 1320, + appCostUsd: 0.08, + judgeCostUsd: 0.02, + totalCostUsd: 0.1, toolCallCount: 2, durationMs: 4500, }); diff --git a/packages/report-ui/src/app/model.ts b/packages/report-ui/src/app/model.ts index eadf324..27169f5 100644 --- a/packages/report-ui/src/app/model.ts +++ b/packages/report-ui/src/app/model.ts @@ -26,7 +26,12 @@ export type WorkspaceSummary = { failed: number; skipped: number; averageScore?: number; + appTokens: number; + judgeTokens: number; totalTokens: number; + appCostUsd?: number; + judgeCostUsd?: number; + totalCostUsd?: number; toolCallCount: number; durationMs?: number; }; @@ -69,6 +74,15 @@ export function summarizeWorkspace( .map((testCase) => testCase.eval?.avgScore) .filter((score): score is number => typeof score === "number"); + const appRuns = workspace.cases.flatMap((testCase) => + testCase.harness?.run ? [testCase.harness.run] : [], + ); + const judgeRuns = workspace.cases.flatMap((testCase) => + (testCase.eval?.scores ?? []).flatMap((score) => score.judgeRuns ?? []), + ); + const appUsage = summarizeRuns(appRuns); + const judgeUsage = summarizeRuns(judgeRuns); + return { runCount: workspace.runs.length, caseCount: workspace.cases.length, @@ -83,10 +97,12 @@ export function summarizeWorkspace( scores.length > 0 ? scores.reduce((total, score) => total + score, 0) / scores.length : undefined, - totalTokens: workspace.cases.reduce( - (total, testCase) => total + totalTokensFor(testCase.harness?.run), - 0, - ), + appTokens: appUsage.tokens, + judgeTokens: judgeUsage.tokens, + totalTokens: appUsage.tokens + judgeUsage.tokens, + appCostUsd: appUsage.costUsd, + judgeCostUsd: judgeUsage.costUsd, + totalCostUsd: combinedCost(appUsage, judgeUsage), toolCallCount: workspace.cases.reduce( (total, testCase) => total + (toolCallCountForCase(testCase) ?? 0), 0, @@ -210,6 +226,17 @@ export function formatNumber(value: number | undefined) { return value === undefined ? "n/a" : new Intl.NumberFormat().format(value); } +export function formatUsd(value: number | undefined) { + return value === undefined + ? "n/a" + : value.toLocaleString("en-US", { + style: "currency", + currency: "USD", + minimumFractionDigits: 2, + maximumFractionDigits: 6, + }); +} + export function formatJson(value: unknown) { if (value === undefined) { return ""; @@ -232,6 +259,29 @@ function totalTokensFor(run: HarnessRun | undefined) { ); } +/** Sums tokens and complete USD cost across normalized harness runs. */ +export function summarizeRuns(runs: HarnessRun[]) { + return { + tokens: runs.reduce((total, run) => total + totalTokensFor(run), 0), + costUsd: + runs.length > 0 && runs.every((run) => run.usage.costUsd !== undefined) + ? runs.reduce((total, run) => total + (run.usage.costUsd ?? 0), 0) + : undefined, + runCount: runs.length, + }; +} + +function combinedCost( + app: ReturnType, + judge: ReturnType, +) { + const appKnown = app.runCount === 0 || app.costUsd !== undefined; + const judgeKnown = judge.runCount === 0 || judge.costUsd !== undefined; + return appKnown && judgeKnown && app.runCount + judge.runCount > 0 + ? (app.costUsd ?? 0) + (judge.costUsd ?? 0) + : undefined; +} + function toolCallsForCase(testCase: ReportCase) { const run = testCase.harness?.run; const runToolCalls = run ? toolCalls(run.session) : []; diff --git a/packages/report-ui/src/server.test.ts b/packages/report-ui/src/server.test.ts index 52361a2..7957960 100644 --- a/packages/report-ui/src/server.test.ts +++ b/packages/report-ui/src/server.test.ts @@ -1,8 +1,9 @@ import { chmod, mkdir, mkdtemp, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; -import { join } from "node:path"; +import { join, resolve } from "node:path"; import { describe, expect, test } from "vitest"; import type { ReportWorkspace } from "@vitest-evals/core"; +import { readReportWorkspace } from "@vitest-evals/core/node"; import { parseCliArgs } from "./cli-options"; import { serveReportWorkspace } from "./server"; @@ -76,6 +77,28 @@ describe("parseCliArgs", () => { inputs: ["vitest-results.json"], }); }); + + test("keeps the visual fixture compatible with the current report schema", async () => { + const fixture = resolve( + process.cwd(), + "packages/report-ui/fixtures/vitest-results.visual.json", + ); + const { workspace } = await readReportWorkspace([fixture], { + workspace: "/repo", + }); + + expect(workspace.cases).toHaveLength(5); + expect(workspace.cases[0]?.harness?.run?.usage).toMatchObject({ + totalTokens: 1220, + costUsd: 0.084, + }); + expect( + workspace.cases[0]?.eval?.scores?.[0]?.judgeRuns?.[0]?.usage, + ).toMatchObject({ + totalTokens: 96, + costUsd: 0.012, + }); + }); }); describe("serveReportWorkspace", () => { diff --git a/packages/vitest-evals/README.md b/packages/vitest-evals/README.md index 83f932e..f2dfd2b 100644 --- a/packages/vitest-evals/README.md +++ b/packages/vitest-evals/README.md @@ -133,7 +133,8 @@ describeEval("refund agent", { harness }, (it) => { ## Terminal Reporting The terminal reporter has two eval report levels. Normal mode prints compact -test, score, usage, and tool-count summaries. Info mode adds per-tool summaries, +test and score summaries, with application and judge tokens and cost shown +separately. Info mode adds per-tool summaries, arguments, timing/size metadata, replay status, and final output summaries. Set `VITEST_EVALS_REPORT_LEVEL=info`, or pass `--info` through the workspace eval scripts, to enable it. `--verbose` and `-v` remain aliases for @@ -145,7 +146,7 @@ Full transcripts and spans are preserved in the Vitest JSON report metadata. The local report UI reads the same Vitest JSON artifacts and serves a React SPA for drilling into runs, eval cases, harness output, sessions, tool calls, -scores, and trace spans. +scores, trace spans, and separate application, judge, and total usage. ```sh pnpm exec vitest-evals serve vitest-results.json diff --git a/packages/vitest-evals/src/reporter.test.ts b/packages/vitest-evals/src/reporter.test.ts index b55dc8c..bc19158 100644 --- a/packages/vitest-evals/src/reporter.test.ts +++ b/packages/vitest-evals/src/reporter.test.ts @@ -59,6 +59,14 @@ function createTestCase({ rationale?: string; output?: unknown; }; + judgeRuns?: Array<{ + session: { events: TranscriptEvent[] }; + usage: { + totalTokens?: number; + costUsd?: number; + }; + errors: unknown[]; + }>; }>; }; harness?: { @@ -70,6 +78,7 @@ function createTestCase({ }; usage?: { totalTokens?: number; + costUsd?: number; toolCalls?: number; }; errors?: unknown[]; @@ -246,7 +255,7 @@ describe("DefaultEvalReporter", () => { expect(logger.log).toHaveBeenCalledTimes(1); expect(stripVTControlCharacters(logger.log.mock.calls[0][0])).toContain( - "fixtures/reporter.eval.ts:12:3 > demo pi refund agent > streams eval progress [12 tok | 1 tool] 42ms", + "fixtures/reporter.eval.ts:12:3 > demo pi refund agent > streams eval progress [app 12 tok | 1 tool] 42ms", ); }); @@ -759,6 +768,13 @@ describe("DefaultEvalReporter", () => { { name: "StructuredOutputJudge", score: 1, + judgeRuns: [ + { + session: { events: [] }, + usage: { totalTokens: 3, costUsd: 0.002 }, + errors: [], + }, + ], }, { name: "ToolCallJudge", @@ -771,7 +787,7 @@ describe("DefaultEvalReporter", () => { expect(logger.log).toHaveBeenCalledTimes(3); expect(stripVTControlCharacters(logger.log.mock.calls[0][0])).toContain( - "fixtures/reporter.eval.ts:12:3 > demo pi refund agent > streams eval progress [12 tok] 42ms", + "fixtures/reporter.eval.ts:12:3 > demo pi refund agent > streams eval progress [app 12 tok | judge 3 tok / $0.0020 | total 15 tok] 42ms", ); expect(stripVTControlCharacters(logger.log.mock.calls[1][0])).toContain( "score StructuredOutputJudge 1.00", diff --git a/packages/vitest-evals/src/reporter.ts b/packages/vitest-evals/src/reporter.ts index 9bb6552..10abc13 100644 --- a/packages/vitest-evals/src/reporter.ts +++ b/packages/vitest-evals/src/reporter.ts @@ -1,6 +1,6 @@ import { DefaultReporter, VerboseReporter } from "vitest/node"; import c from "tinyrainbow"; -import type { ToolCall } from "./harness"; +import type { HarnessRun, ToolCall, UsageSummary } from "./harness"; import { toolCalls } from "./harness"; const TEST_NAME_SEPARATOR = c.dim(" > "); @@ -50,7 +50,7 @@ export default class DefaultEvalReporter extends VerboseReporter { } if (meta.harness) { - this.logHarnessTestCase(test, meta.harness); + this.logHarnessTestCase(test, meta.harness, meta.eval); if (testResult.state !== "failed" && meta.eval?.scores?.length) { this.logJudgeScoreDetails(meta.eval.scores); } @@ -119,19 +119,16 @@ export default class DefaultEvalReporter extends VerboseReporter { run: { session: Parameters[0]; output?: unknown; - usage?: { - totalTokens?: number; - inputTokens?: number; - outputTokens?: number; - reasoningTokens?: number; - toolCalls?: number; - }; + usage?: UsageSummary; errors?: unknown[]; }; }, + evalMeta?: { + scores?: Array<{ judgeRuns?: HarnessRun[] }>; + }, ): void { let title = this.getFormattedTestTitle(test); - const summary = this.formatHarnessSummary(harnessMeta); + const summary = this.formatHarnessSummary(harnessMeta, evalMeta); if (summary) { title += c.dim(` [${summary}]`); } @@ -144,32 +141,35 @@ export default class DefaultEvalReporter extends VerboseReporter { } } - private formatHarnessSummary(harnessMeta: { - name: string; - run: { - session: Parameters[0]; - usage?: { - totalTokens?: number; - inputTokens?: number; - outputTokens?: number; - reasoningTokens?: number; - toolCalls?: number; + private formatHarnessSummary( + harnessMeta: { + name: string; + run: { + session: Parameters[0]; + usage?: UsageSummary; + errors?: unknown[]; }; - errors?: unknown[]; - }; - }) { + }, + evalMeta?: { + scores?: Array<{ judgeRuns?: HarnessRun[] }>; + }, + ) { const parts: string[] = []; - const totalTokens = - harnessMeta.run.usage?.totalTokens ?? - (harnessMeta.run.usage?.inputTokens ?? 0) + - (harnessMeta.run.usage?.outputTokens ?? 0) + - (harnessMeta.run.usage?.reasoningTokens ?? 0); + const appUsage = summarizeUsage(harnessMeta.run.usage); + const judgeRuns = (evalMeta?.scores ?? []).flatMap( + (score) => score.judgeRuns ?? [], + ); + const judgeUsage = summarizeRunUsage(judgeRuns); const totalTools = harnessMeta.run.usage?.toolCalls ?? toolCalls(harnessMeta.run.session).length; - if (totalTokens > 0) { - parts.push(`${totalTokens} tok`); + if (appUsage) { + parts.push(`app ${formatUsage(appUsage)}`); + } + if (judgeUsage) { + parts.push(`judge ${formatUsage(judgeUsage)}`); + parts.push(`total ${formatUsage(sumUsage(appUsage, judgeUsage))}`); } if (totalTools > 0) { parts.push(`${totalTools} tool${totalTools === 1 ? "" : "s"}`); @@ -795,3 +795,61 @@ export default class DefaultEvalReporter extends VerboseReporter { return title; } } + +function summarizeRunUsage(runs: HarnessRun[]) { + if (runs.length === 0) { + return undefined; + } + const totalTokens = runs.reduce( + (total, run) => total + usageTokens(run.usage), + 0, + ); + const costUsd = runs.every((run) => run.usage.costUsd !== undefined) + ? runs.reduce((total, run) => total + (run.usage.costUsd ?? 0), 0) + : undefined; + return totalTokens > 0 || costUsd !== undefined + ? { totalTokens, costUsd } + : undefined; +} + +function summarizeUsage(usage: UsageSummary | undefined) { + if (!usage) { + return undefined; + } + const totalTokens = usageTokens(usage); + return totalTokens > 0 || usage.costUsd !== undefined + ? { totalTokens, costUsd: usage.costUsd } + : undefined; +} + +function sumUsage(app: UsageSummary | undefined, judge: UsageSummary) { + const appCostKnown = !app || app.costUsd !== undefined; + const judgeCostKnown = judge.costUsd !== undefined; + return { + totalTokens: usageTokens(app) + usageTokens(judge), + ...(appCostKnown && judgeCostKnown + ? { costUsd: (app?.costUsd ?? 0) + (judge.costUsd ?? 0) } + : {}), + }; +} + +function formatUsage(usage: UsageSummary) { + const parts: string[] = []; + const totalTokens = usageTokens(usage); + if (totalTokens > 0) { + parts.push(`${totalTokens} tok`); + } + if (usage.costUsd !== undefined) { + parts.push(`$${usage.costUsd.toFixed(usage.costUsd < 0.01 ? 4 : 2)}`); + } + return parts.join(" / "); +} + +function usageTokens(usage: UsageSummary | undefined) { + return ( + usage?.totalTokens ?? + (usage?.inputTokens ?? 0) + + (usage?.outputTokens ?? 0) + + (usage?.reasoningTokens ?? 0) + ); +} From 3e5e87ed33d22e9bd479e5235e5fd60583436fa8 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 06:14:06 +0000 Subject: [PATCH 09/15] fix(reporter): treat eval-only tools as uncosted usage --- packages/github-reporter/src/collect.ts | 5 ++++ packages/github-reporter/src/report.test.ts | 26 +++++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/packages/github-reporter/src/collect.ts b/packages/github-reporter/src/collect.ts index db5c2d6..f34ae37 100644 --- a/packages/github-reporter/src/collect.ts +++ b/packages/github-reporter/src/collect.ts @@ -273,6 +273,11 @@ function sumAppUsage(cases: EvalCase[]) { toolCallCount(testCase) - (testCase.harness?.usage?.toolCalls ?? 0); } omitPartialCost(usage, runUsages); + if ( + cases.some((testCase) => !testCase.harness && toolCallCount(testCase) > 0) + ) { + usage.costUsd = undefined; + } return usage; } diff --git a/packages/github-reporter/src/report.test.ts b/packages/github-reporter/src/report.test.ts index f616008..832d3fc 100644 --- a/packages/github-reporter/src/report.test.ts +++ b/packages/github-reporter/src/report.test.ts @@ -994,6 +994,32 @@ describe("renderJobSummary", () => { expect(report.usage.costUsd).toBeUndefined(); }); + test("treats eval-only tool calls as application usage with unknown cost", () => { + const json = structuredClone(sampleJson); + const secondAssertion = structuredClone( + json.testResults[0]!.assertionResults[0]!, + ); + secondAssertion.title = "eval-only tools"; + secondAssertion.fullName = "refund agent eval-only tools"; + secondAssertion.meta = { + eval: { + avgScore: 1, + scores: [], + toolCalls: [ + { name: "lookupInvoice", status: "ok" }, + { name: "notifyCustomer", status: "ok" }, + ], + }, + }; + json.testResults[0]!.assertionResults.push(secondAssertion); + + const report = collectEvalReport(json, { workspace: "/repo" }); + + expect(report.usage.totalTokens).toBe(1220); + expect(report.usage.toolCalls).toBe(4); + expect(report.usage.costUsd).toBeUndefined(); + }); + test("surfaces non-eval failures without pretending the run passed", () => { const json: VitestJsonReport = { ...sampleJson, From d2aad046a404c8e6e66d7488baee5ed9323e1b44 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 06:19:07 +0000 Subject: [PATCH 10/15] fix(reporting): preserve costs with empty usage --- packages/github-reporter/src/collect.ts | 42 ++++++++++++++------- packages/github-reporter/src/report.test.ts | 17 +++++++++ packages/report-ui/src/app/model.test.ts | 21 ++++++++++- packages/report-ui/src/app/model.ts | 20 +++++++--- 4 files changed, 81 insertions(+), 19 deletions(-) diff --git a/packages/github-reporter/src/collect.ts b/packages/github-reporter/src/collect.ts index f34ae37..418dbcb 100644 --- a/packages/github-reporter/src/collect.ts +++ b/packages/github-reporter/src/collect.ts @@ -264,23 +264,28 @@ function addRunUsage( function sumAppUsage(cases: EvalCase[]) { const usage = emptyUsage(); - const runUsages = cases.flatMap((testCase) => - testCase.harness ? [testCase.harness.usage] : [], - ); - for (const testCase of cases) { - addRunUsage(usage, testCase.harness?.usage); - usage.toolCalls += - toolCallCount(testCase) - (testCase.harness?.usage?.toolCalls ?? 0); + const runUsages = cases + .map(appUsageForCase) + .filter((item): item is HarnessUsageSummary => item !== undefined); + for (const runUsage of runUsages) { + addRunUsage(usage, runUsage); } omitPartialCost(usage, runUsages); - if ( - cases.some((testCase) => !testCase.harness && toolCallCount(testCase) > 0) - ) { - usage.costUsd = undefined; - } return usage; } +function appUsageForCase(testCase: EvalCase): HarnessUsageSummary | undefined { + const usage = testCase.harness?.usage; + const effectiveToolCalls = toolCallCount(testCase); + if (!usage && effectiveToolCalls === 0) { + return undefined; + } + return { + ...usage, + ...(effectiveToolCalls > 0 ? { toolCalls: effectiveToolCalls } : {}), + }; +} + function sumJudgeUsage(cases: EvalCase[]) { const usage = emptyUsage(); const runUsages: Array = []; @@ -300,11 +305,22 @@ function omitPartialCost( total: AggregatedUsageSummary, usages: Array, ) { - if (usages.some((usage) => usage?.costUsd === undefined)) { + if (usages.some(hasUsageWithoutCost)) { total.costUsd = undefined; } } +function hasUsageWithoutCost(usage: HarnessUsageSummary | undefined) { + return ( + usage?.costUsd === undefined && + ((usage?.totalTokens ?? + (usage?.inputTokens ?? 0) + + (usage?.outputTokens ?? 0) + + (usage?.reasoningTokens ?? 0)) > 0 || + (usage?.toolCalls ?? 0) > 0) + ); +} + function toolCallCount(testCase: EvalCase) { const usageToolCalls = testCase.harness?.usage?.toolCalls; if (usageToolCalls !== undefined) { diff --git a/packages/github-reporter/src/report.test.ts b/packages/github-reporter/src/report.test.ts index 832d3fc..00d37b5 100644 --- a/packages/github-reporter/src/report.test.ts +++ b/packages/github-reporter/src/report.test.ts @@ -994,6 +994,23 @@ describe("renderJobSummary", () => { expect(report.usage.costUsd).toBeUndefined(); }); + test("does not let empty usage hide a known sibling cost", () => { + const json = structuredClone(sampleJson); + const secondAssertion = structuredClone( + json.testResults[0]!.assertionResults[0]!, + ); + secondAssertion.title = "deterministic case"; + secondAssertion.fullName = "refund agent deterministic case"; + (secondAssertion.meta as any).harness.run.usage = {}; + (secondAssertion.meta as any).harness.run.session.events = []; + json.testResults[0]!.assertionResults.push(secondAssertion); + + const report = collectEvalReport(json, { workspace: "/repo" }); + + expect(report.usage.totalTokens).toBe(1220); + expect(report.usage.costUsd).toBe(0.08); + }); + test("treats eval-only tool calls as application usage with unknown cost", () => { const json = structuredClone(sampleJson); const secondAssertion = structuredClone( diff --git a/packages/report-ui/src/app/model.test.ts b/packages/report-ui/src/app/model.test.ts index c704da9..e74461d 100644 --- a/packages/report-ui/src/app/model.test.ts +++ b/packages/report-ui/src/app/model.test.ts @@ -1,6 +1,6 @@ -import { describe, expect, test } from "vitest"; import { messagesToTranscriptEvents } from "@vitest-evals/core"; import type { ReportWorkspace } from "@vitest-evals/core"; +import { describe, expect, test } from "vitest"; import { buildSpanTree, buildTranscript, @@ -164,6 +164,25 @@ describe("summarizeWorkspace", () => { }); }); + test("does not let an empty usage run hide a known sibling cost", () => { + const testCase = structuredClone(workspace.cases[0]!); + const emptyUsageCase = structuredClone(workspace.cases[0]!); + emptyUsageCase.id = "case-empty-usage"; + emptyUsageCase.harness!.run!.usage = {}; + emptyUsageCase.eval = undefined; + + expect( + summarizeWorkspace({ + ...workspace, + cases: [testCase, emptyUsageCase], + }), + ).toMatchObject({ + appTokens: 1220, + appCostUsd: 0.08, + totalCostUsd: 0.1, + }); + }); + test("uses the wall-clock envelope for parallel runs", () => { expect( summarizeWorkspace({ diff --git a/packages/report-ui/src/app/model.ts b/packages/report-ui/src/app/model.ts index 27169f5..97f6d14 100644 --- a/packages/report-ui/src/app/model.ts +++ b/packages/report-ui/src/app/model.ts @@ -1,5 +1,4 @@ import { - toolCalls, type HarnessRun, type JsonValue, type NormalizedError, @@ -9,6 +8,7 @@ import { type TranscriptMessageEvent, type TranscriptToolCallEvent, type TranscriptToolResultEvent, + toolCalls, } from "@vitest-evals/core"; export type CaseStatusFilter = "all" | ReportCase["status"]; @@ -261,16 +261,26 @@ function totalTokensFor(run: HarnessRun | undefined) { /** Sums tokens and complete USD cost across normalized harness runs. */ export function summarizeRuns(runs: HarnessRun[]) { + const usageRuns = runs.filter(hasUsage); return { - tokens: runs.reduce((total, run) => total + totalTokensFor(run), 0), + tokens: usageRuns.reduce((total, run) => total + totalTokensFor(run), 0), costUsd: - runs.length > 0 && runs.every((run) => run.usage.costUsd !== undefined) - ? runs.reduce((total, run) => total + (run.usage.costUsd ?? 0), 0) + usageRuns.length > 0 && + usageRuns.every((run) => run.usage.costUsd !== undefined) + ? usageRuns.reduce((total, run) => total + (run.usage.costUsd ?? 0), 0) : undefined, - runCount: runs.length, + runCount: usageRuns.length, }; } +function hasUsage(run: HarnessRun) { + return ( + totalTokensFor(run) > 0 || + (run.usage.toolCalls ?? 0) > 0 || + run.usage.costUsd !== undefined + ); +} + function combinedCost( app: ReturnType, judge: ReturnType, From c2e2505e2ecdf6deefc90a2a66bb337dbf6dada5 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 06:25:15 +0000 Subject: [PATCH 11/15] fix(report-ui): include session-only tool usage --- packages/report-ui/src/app/model.test.ts | 20 ++++++++++++++++++++ packages/report-ui/src/app/model.ts | 1 + 2 files changed, 21 insertions(+) diff --git a/packages/report-ui/src/app/model.test.ts b/packages/report-ui/src/app/model.test.ts index e74461d..0b89072 100644 --- a/packages/report-ui/src/app/model.test.ts +++ b/packages/report-ui/src/app/model.test.ts @@ -169,6 +169,7 @@ describe("summarizeWorkspace", () => { const emptyUsageCase = structuredClone(workspace.cases[0]!); emptyUsageCase.id = "case-empty-usage"; emptyUsageCase.harness!.run!.usage = {}; + emptyUsageCase.harness!.run!.session.events = []; emptyUsageCase.eval = undefined; expect( @@ -183,6 +184,25 @@ describe("summarizeWorkspace", () => { }); }); + test("treats session-only tool calls as usage with unknown cost", () => { + const testCase = structuredClone(workspace.cases[0]!); + const sessionOnlyCase = structuredClone(workspace.cases[0]!); + sessionOnlyCase.id = "case-session-only"; + sessionOnlyCase.harness!.run!.usage = {}; + sessionOnlyCase.eval = undefined; + + expect( + summarizeWorkspace({ + ...workspace, + cases: [testCase, sessionOnlyCase], + }), + ).toMatchObject({ + appTokens: 1220, + appCostUsd: undefined, + totalCostUsd: undefined, + }); + }); + test("uses the wall-clock envelope for parallel runs", () => { expect( summarizeWorkspace({ diff --git a/packages/report-ui/src/app/model.ts b/packages/report-ui/src/app/model.ts index 97f6d14..8aa9683 100644 --- a/packages/report-ui/src/app/model.ts +++ b/packages/report-ui/src/app/model.ts @@ -277,6 +277,7 @@ function hasUsage(run: HarnessRun) { return ( totalTokensFor(run) > 0 || (run.usage.toolCalls ?? 0) > 0 || + toolCalls(run.session).length > 0 || run.usage.costUsd !== undefined ); } From 8d6344f93fdf33f2506b3dcb4854bf9e4811ae92 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 06:29:59 +0000 Subject: [PATCH 12/15] fix(report-ui): migrate version 1 workspaces --- packages/core/src/index.test.ts | 22 ++++++++++++++---- packages/core/src/report/workspace.ts | 33 +++++++++++++++++++++++---- 2 files changed, 45 insertions(+), 10 deletions(-) diff --git a/packages/core/src/index.test.ts b/packages/core/src/index.test.ts index e26bcb5..0b6b326 100644 --- a/packages/core/src/index.test.ts +++ b/packages/core/src/index.test.ts @@ -1,12 +1,16 @@ import { describe, expect, test } from "vitest"; import { + HarnessRunSchema, + TranscriptToolResultEventSchema, + UsageSummarySchema, + type VitestJsonReport, assistantMessages, collectReportWorkspace, failedSpans, - HarnessRunSchema, latestAssistantMessageContent, - messagesToTranscriptEvents, messagesByRole, + messagesToTranscriptEvents, + parseReportWorkspace, parseVitestJsonReport, readEvalTaskMeta, spans, @@ -15,10 +19,7 @@ import { toolCalls, toolMessages, traceSpans, - TranscriptToolResultEventSchema, - UsageSummarySchema, userMessages, - type VitestJsonReport, } from "./index"; const sampleJson: VitestJsonReport = { @@ -967,6 +968,17 @@ describe("collectReportWorkspace", () => { }); }); + test("migrates schema version 1 workspaces in memory", () => { + const workspace = collectReportWorkspace(sampleJson); + + expect( + parseReportWorkspace({ + ...workspace, + schemaVersion: 1, + }), + ).toEqual(workspace); + }); + test("defaults harness-only case scores from Vitest status", () => { const json = structuredClone(sampleJson); const passed = json.testResults[0]!.assertionResults[0]!; diff --git a/packages/core/src/report/workspace.ts b/packages/core/src/report/workspace.ts index 4d5b687..4da5302 100644 --- a/packages/core/src/report/workspace.ts +++ b/packages/core/src/report/workspace.ts @@ -1,15 +1,15 @@ import { z } from "zod"; +import { FiniteNumberSchema, isRecord, parseWithSchema } from "../schema-utils"; import { EvalMetaSchema, HarnessMetaSchema, readEvalTaskMeta, } from "./metadata"; -import { FiniteNumberSchema, parseWithSchema, isRecord } from "../schema-utils"; import { - parseVitestJsonReport, - VitestJsonLocationSchema, type VitestJsonAssertion, + VitestJsonLocationSchema, type VitestJsonReport, + parseVitestJsonReport, } from "./vitest-json"; import { VitestJsonStatusSchema } from "./vitest-json"; @@ -64,12 +64,23 @@ export const ReportCaseSchema = z /** One eval or harness-backed test case collected from Vitest JSON. */ export type ReportCase = z.infer; +const reportWorkspaceShape = { + runs: z.array(ReportRunSchema), + cases: z.array(ReportCaseSchema), +}; + +const LegacyReportWorkspaceSchema = z + .object({ + schemaVersion: z.literal(1), + ...reportWorkspaceShape, + }) + .strict(); + /** Full multi-run report workspace consumed by rich report UIs. */ export const ReportWorkspaceSchema = z .object({ schemaVersion: z.literal(REPORT_WORKSPACE_SCHEMA_VERSION), - runs: z.array(ReportRunSchema), - cases: z.array(ReportCaseSchema), + ...reportWorkspaceShape, }) .strict(); @@ -92,6 +103,18 @@ export type CollectReportWorkspaceOptions = { /** Parses and validates an unknown value as a collected report workspace. */ export function parseReportWorkspace(input: unknown): ReportWorkspace { + if (isRecord(input) && input.schemaVersion === 1) { + const legacy = parseWithSchema( + LegacyReportWorkspaceSchema, + input, + "report workspace", + ); + return { + ...legacy, + schemaVersion: REPORT_WORKSPACE_SCHEMA_VERSION, + }; + } + return parseWithSchema(ReportWorkspaceSchema, input, "report workspace"); } From 496a008108330987610427f36248a07a30ef3d22 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 13:45:05 +0000 Subject: [PATCH 13/15] refactor(reporter): clarify GitHub summaries Co-Authored-By: David Cramer --- packages/github-reporter/src/gate.ts | 38 ++++++------- packages/github-reporter/src/report.test.ts | 62 ++++++++++++--------- packages/github-reporter/src/summary.ts | 53 ++++++++++-------- 3 files changed, 85 insertions(+), 68 deletions(-) diff --git a/packages/github-reporter/src/gate.ts b/packages/github-reporter/src/gate.ts index 58f5467..20941b6 100644 --- a/packages/github-reporter/src/gate.ts +++ b/packages/github-reporter/src/gate.ts @@ -87,10 +87,10 @@ export function evaluateEvalGate( status: "failed", enforced: true, passRate, - title: "Eval report hard failure", - message: `${formatNumber(nonEvalFailures)} non-eval test failure${ + title: "Eval run failed", + message: `${formatNumber(nonEvalFailures)} test failure${ nonEvalFailures === 1 ? "" : "s" - }; ${counts}`, + } outside eval cases; ${counts}`, }; } @@ -100,7 +100,7 @@ export function evaluateEvalGate( status: "failed", enforced: true, passRate: null, - title: "Eval report hard failure", + title: "Eval run failed", message: "no eval cases were reported", }; } @@ -117,8 +117,8 @@ export function evaluateEvalGate( status: "failed", enforced: true, passRate, - title: "Eval report hard failure", - message: `vitest run failed without counted test failures; ${counts}`, + title: "Eval run failed", + message: `Vitest failed without reporting a failed test; ${counts}`, }; } @@ -132,7 +132,7 @@ export function evaluateEvalGate( enforced: true, passRate, title: `Eval pass rate ${formatPercent(passRate)} — required ${formatPercent(minPassRate)}`, - message: `eval pass rate below floor: ${counts}; required >= ${formatPercent(minPassRate)}`, + message: `pass rate is below the minimum: ${counts}; minimum ${formatPercent(minPassRate)}`, }; } @@ -144,8 +144,8 @@ export function evaluateEvalGate( status: "failed", enforced: true, passRate, - title: "Eval score gate failed", - message: `no score average available; required avg score >= ${formatScore(minScoreAverage)}`, + title: "Average score unavailable", + message: `no average score was reported; minimum ${formatScore(minScoreAverage)}`, }; } if (average + Number.EPSILON < minScoreAverage) { @@ -154,8 +154,8 @@ export function evaluateEvalGate( status: "failed", enforced: true, passRate, - title: `Avg score ${formatScore(average)} — required ${formatScore(minScoreAverage)}`, - message: `avg score below floor: ${counts}; required avg score >= ${formatScore(minScoreAverage)}`, + title: `Average score ${formatScore(average)} — required ${formatScore(minScoreAverage)}`, + message: `average score is below the minimum: ${counts}; minimum ${formatScore(minScoreAverage)}`, }; } } @@ -166,7 +166,7 @@ export function evaluateEvalGate( enforced: true, passRate, title: enforcedPassTitle(report, passRate, minPassRate, minScoreAverage), - message: `eval gate passed: ${counts}${formatFloorSuffix(minPassRate, minScoreAverage)}`, + message: `requirements met: ${counts}${formatMinimumSuffix(minPassRate, minScoreAverage)}`, }; } @@ -207,7 +207,7 @@ function resolveMinPassRate(policy: EvalGatePolicy) { function defaultCheckTitle(report: EvalReport) { if (report.failures.length === 0 && report.status === "passed") { - return "No eval failures"; + return "No eval cases failed"; } if (report.failures.length === 0) { return "Vitest run failed"; @@ -224,10 +224,10 @@ function enforcedPassTitle( minScoreAverage: number | undefined, ) { if (minPassRate !== undefined) { - return `Eval pass rate ${formatPercent(passRate)} — floor ${formatPercent(minPassRate)}`; + return `Eval pass rate ${formatPercent(passRate)} — minimum ${formatPercent(minPassRate)}`; } if (minScoreAverage !== undefined) { - return `Avg score ${formatScore(report.score?.average)} — floor ${formatScore(minScoreAverage)}`; + return `Average score ${formatScore(report.score?.average)} — minimum ${formatScore(minScoreAverage)}`; } return defaultCheckTitle(report); } @@ -240,19 +240,19 @@ function formatEvalCounts(report: EvalReport, passRate: number | null) { const passRateText = passRate === null ? "n/a" : formatPercent(passRate); return `${formatNumber(report.totals.evalPassed)}/${formatNumber( report.totals.evalTotal, - )} passed (${passRateText}), avg score ${scoreText}`; + )} passed (${passRateText}), average score ${scoreText}`; } -function formatFloorSuffix( +function formatMinimumSuffix( minPassRate: number | undefined, minScoreAverage: number | undefined, ) { const parts: string[] = []; if (minPassRate !== undefined) { - parts.push(`pass rate floor ${formatPercent(minPassRate)}`); + parts.push(`minimum pass rate ${formatPercent(minPassRate)}`); } if (minScoreAverage !== undefined) { - parts.push(`avg score floor ${formatScore(minScoreAverage)}`); + parts.push(`minimum average score ${formatScore(minScoreAverage)}`); } return parts.length === 0 ? "" : `; ${parts.join(", ")}`; } diff --git a/packages/github-reporter/src/report.test.ts b/packages/github-reporter/src/report.test.ts index 00d37b5..56e59e1 100644 --- a/packages/github-reporter/src/report.test.ts +++ b/packages/github-reporter/src/report.test.ts @@ -751,7 +751,7 @@ describe("evaluateEvalGate", () => { expect(gate.enforced).toBe(true); expect(gate.passRate).toBe(0.8); expect(gate.title).toContain("80.0%"); - expect(gate.message).toContain("pass rate floor 80.0%"); + expect(gate.message).toContain("minimum pass rate 80.0%"); }); test("fails a suite below the pass-rate floor with a titled annotation", () => { @@ -766,7 +766,7 @@ describe("evaluateEvalGate", () => { expect(gate.ok).toBe(false); expect(gate.title).toBe("Eval pass rate 63.7% — required 80.0%"); expect(renderGateWorkflowCommand(gate)).toBe( - "::error title=Eval pass rate 63.7%25 — required 80.0%25::eval pass rate below floor: 65/102 passed (63.7%25), avg score 0.72; required >= 80.0%25", + "::error title=Eval pass rate 63.7%25 — required 80.0%25::pass rate is below the minimum: 65/102 passed (63.7%25), average score 0.72; minimum 80.0%25", ); }); @@ -782,8 +782,8 @@ describe("evaluateEvalGate", () => { const gate = evaluateEvalGate(report, { minPassRate: 0.8 }); expect(gate.ok).toBe(false); - expect(gate.title).toBe("Eval report hard failure"); - expect(gate.message).toContain("non-eval test failure"); + expect(gate.title).toBe("Eval run failed"); + expect(gate.message).toContain("test failure outside eval cases"); }); test("treats fail-on-failures as a 100% pass-rate floor", () => { @@ -921,16 +921,18 @@ describe("renderJobSummary", () => { "1. refund agent > rejects fraud - StructuredOutputJudge - 0.20", ); expect(summary).toContain("| Metric | Value |"); - expect(summary).toContain("| Status | failed |"); - expect(summary).toContain("| Evals | 0 passed, 1 failed, 1 total |"); - expect(summary).toContain("| Pass Rate | 0.0% |"); + expect(summary).toContain("| Status | Failed |"); + expect(summary).toContain("| Eval cases | 0 passed, 1 failed, 1 total |"); + expect(summary).toContain("| Pass rate | 0.0% |"); expect(summary).not.toContain("| Tests |"); - expect(summary).toContain("| Score | avg 0.20, min 0.20 |"); + expect(summary).toContain("| Score | average 0.20, lowest 0.20 |"); expect(summary).toContain( - "| Application Usage | 1,220 tokens, $0.08, 2 tools |", + "| App usage | 1,220 tokens, $0.08, 2 tool calls |", + ); + expect(summary).toContain("| Judge usage | 100 tokens, $0.02 |"); + expect(summary).toContain( + "| Total usage | 1,320 tokens, $0.10, 2 tool calls |", ); - expect(summary).toContain("| Judge Usage | 100 tokens, $0.02 |"); - expect(summary).toContain("| Total Usage | 1,320 tokens, $0.10, 2 tools |"); expect(summary).toContain("## Scores"); expect(summary.indexOf("## Scores")).toBeLessThan( summary.indexOf("## Results"), @@ -942,14 +944,14 @@ describe("renderJobSummary", () => { const details = summary.match(/
[\s\S]*?<\/details>/)?.[0] ?? ""; expect(details.match(/```/g)).toHaveLength(2); expect(details).toContain("```text\nResult\n------"); - expect(details).toContain("Case 1. refund agent > rejects fraud"); - expect(details).toContain("Location apps/demo/evals/refund.eval.ts:42"); - expect(details).toContain("Usage 1,220 tokens, 2 tools, 4.1s"); + expect(details).toContain("Case 1. refund agent > rejects fraud"); + expect(details).toContain("Location apps/demo/evals/refund.eval.ts:42"); + expect(details).toContain("Usage 1,220 tokens, 2 tool calls, 4.1s"); expect(details).toContain("Reason\n------"); expect(details).toContain("Expected status=approved, got status=denied"); expect(details).toContain("Judge Score"); expect(details).toContain("StructuredOutputJudge 0.20"); - expect(details).toContain("Final Output\n------------"); + expect(details).toContain("Output\n------"); expect(details).toContain('"reason": "invoice is not refundable"'); expect(details).toContain("Tool Status"); expect(details).toContain( @@ -971,9 +973,13 @@ describe("renderJobSummary", () => { collectEvalReport(json, { workspace: "/repo" }), ); - expect(summary).toContain("| Application Usage | 1,220 tokens, 2 tools |"); - expect(summary).toContain("| Judge Usage | 100 tokens, $0.02 |"); - expect(summary).toContain("| Total Usage | 1,320 tokens, 2 tools |"); + expect(summary).toContain( + "| App usage | 1,220 tokens, cost unavailable, 2 tool calls |", + ); + expect(summary).toContain("| Judge usage | 100 tokens, $0.02 |"); + expect(summary).toContain( + "| Total usage | 1,320 tokens, cost unavailable, 2 tool calls |", + ); }); test("does not present a partial cost within one usage category", () => { @@ -1077,12 +1083,14 @@ describe("renderJobSummary", () => { }), ); - expect(summary).toContain("| Status | failed |"); - expect(summary).toContain("| Evals | 0 passed, 0 failed, 0 total |"); + expect(summary).toContain("| Status | Failed |"); + expect(summary).toContain("| Eval cases | 0 passed, 0 failed, 0 total |"); expect(summary).not.toContain("| Tests |"); - expect(summary).toContain("| Other Failures | 1 non-eval test failure |"); + expect(summary).toContain( + "| Other test failures | 1 test failure outside eval cases |", + ); expect(summary).toContain("## Results"); - expect(summary).toContain("No eval metadata was found"); + expect(summary).toContain("No eval results were found"); }); test("escapes table cell control characters", () => { @@ -1092,7 +1100,7 @@ describe("renderJobSummary", () => { report.status = "failed \\ | escaped" as typeof report.status; expect(renderJobSummary(report)).toContain( - String.raw`| Status | failed \\ \| escaped |`, + String.raw`| Status | Failed \\ \| escaped |`, ); }); }); @@ -1397,10 +1405,10 @@ describe("publishCheckRun", () => { const body = JSON.parse(request.body); expect(body.conclusion).toBe("success"); expect(body.output.title).toContain("90.0%"); - expect(body.output.summary).toContain("| Status | passed |"); - expect(body.output.summary).toContain("| Pass Rate | 90.0% |"); - expect(body.output.summary).toContain("| Gate |"); - expect(body.output.summary).toContain("### Quality Misses"); + expect(body.output.summary).toContain("| Status | Passed |"); + expect(body.output.summary).toContain("| Pass rate | 90.0% |"); + expect(body.output.summary).toContain("| Requirements |"); + expect(body.output.summary).toContain("### Cases Below Target"); expect(body.output.annotations[0]?.annotation_level).toBe("warning"); }); diff --git a/packages/github-reporter/src/summary.ts b/packages/github-reporter/src/summary.ts index 860d154..ecde012 100644 --- a/packages/github-reporter/src/summary.ts +++ b/packages/github-reporter/src/summary.ts @@ -60,7 +60,7 @@ export function renderJobSummary( if (report.failures.length > 0) { const failureHeading = - options.gate?.ok === true ? "### Quality Misses" : "### Failures"; + options.gate?.ok === true ? "### Cases Below Target" : "### Failures"; lines.push(failureHeading, ""); failures.forEach((testCase, index) => { lines.push(...renderFailureDetails(testCase, index + 1, options), ""); @@ -68,18 +68,18 @@ export function renderJobSummary( if (report.failures.length > failures.length) { const omittedLabel = - options.gate?.ok === true ? "quality misses" : "failures"; + options.gate?.ok === true ? "cases below target" : "failures"; lines.push( `${report.failures.length - failures.length} more ${omittedLabel} omitted from this summary.`, "", ); } } else if (report.totals.evalTotal > 0) { - lines.push("### Failures", "", "No eval failures.", ""); + lines.push("### Failures", "", "No eval cases failed.", ""); } if (report.totals.evalTotal === 0) { - lines.push("No eval metadata was found in the Vitest JSON report.", ""); + lines.push("No eval results were found in the Vitest JSON report.", ""); } return `${lines.join("\n")}\n`; @@ -95,9 +95,9 @@ function renderSummaryTable( gate?: EvalGateResult, ) { const rows: Array<[string, string]> = [ - ["Status", gate?.status ?? report.status], + ["Status", capitalize(gate?.status ?? report.status)], [ - "Evals", + "Eval cases", formatCountLine( report.totals.evalPassed, report.totals.evalFailed, @@ -107,10 +107,10 @@ function renderSummaryTable( ]; if (gate?.passRate !== undefined && gate.passRate !== null) { - rows.push(["Pass Rate", formatPercent(gate.passRate)]); + rows.push(["Pass rate", formatPercent(gate.passRate)]); } else if (report.totals.evalTotal > 0) { rows.push([ - "Pass Rate", + "Pass rate", formatPercent(report.totals.evalPassed / report.totals.evalTotal), ]); } @@ -120,24 +120,24 @@ function renderSummaryTable( } if (hasUsage(report.usage) || hasUsage(report.judgeUsage)) { - rows.push(["Application Usage", formatUsage(report.usage)]); - rows.push(["Judge Usage", formatUsage(report.judgeUsage)]); + rows.push(["App usage", formatUsage(report.usage)]); + rows.push(["Judge usage", formatUsage(report.judgeUsage)]); rows.push([ - "Total Usage", + "Total usage", formatUsage(sumUsage(report.usage, report.judgeUsage)), ]); } if (gate?.enforced) { - rows.push(["Gate", gate.message]); + rows.push(["Requirements", gate.message]); } if (nonEvalFailures > 0) { rows.push([ - "Other Failures", - `${formatNumber(nonEvalFailures)} non-eval test failure${ + "Other test failures", + `${formatNumber(nonEvalFailures)} test failure${ nonEvalFailures === 1 ? "" : "s" - }`, + } outside eval cases`, ]); } @@ -154,11 +154,15 @@ function renderSummaryTable( } function formatScoreSummary(score: NonNullable) { - return `avg ${formatScore(score.average)}${ - score.minimum === undefined ? "" : `, min ${formatScore(score.minimum)}` + return `average ${formatScore(score.average)}${ + score.minimum === undefined ? "" : `, lowest ${formatScore(score.minimum)}` }`; } +function capitalize(value: string) { + return value.charAt(0).toUpperCase() + value.slice(1); +} + function hasUsage(usage: AggregatedUsageSummary) { return ( usage.totalTokens > 0 || usage.costUsd !== undefined || usage.toolCalls > 0 @@ -205,9 +209,12 @@ function formatUsage(usage: AggregatedUsageSummary) { }), ); } + if (hasUsageWithoutCost(usage)) { + parts.push("cost unavailable"); + } if (usage.toolCalls > 0) { parts.push( - `${formatNumber(usage.toolCalls)} tool${usage.toolCalls === 1 ? "" : "s"}`, + `${formatNumber(usage.toolCalls)} tool call${usage.toolCalls === 1 ? "" : "s"}`, ); } return parts.join(", ") || "none"; @@ -326,7 +333,7 @@ function renderFailureBlock( ["Case", `${number}. ${testCase.displayName}`], ["Status", testCase.status], ["Location", formatLocation(testCase.displayFile, testCase.location)], - ["Harness", testCase.harness?.name ?? "n/a"], + ["App runner", testCase.harness?.name ?? "n/a"], ["Score", formatScore(failure?.score ?? testCase.eval?.avgScore)], ["Judge", failure?.judgeName ?? "n/a"], ]; @@ -368,7 +375,7 @@ function renderFailureBlock( if (finalOutput !== undefined) { lines.push( ...renderAsciiSection( - "Final Output", + "Output", stringifyValue(finalOutput, maxOutputChars).split(/\r?\n/), ), "", @@ -400,7 +407,7 @@ function renderFailureBlock( if (testCase.harness?.errors.length) { lines.push( ...renderAsciiSection( - "Harness Errors", + "App errors", stringifyValue(testCase.harness.errors, maxReasonChars).split(/\r?\n/), ), "", @@ -471,7 +478,9 @@ function formatCaseUsage(testCase: EvalCase) { parts.push(`${formatNumber(totalTokens)} tokens`); } if (toolCalls > 0) { - parts.push(`${formatNumber(toolCalls)} tool${toolCalls === 1 ? "" : "s"}`); + parts.push( + `${formatNumber(toolCalls)} tool call${toolCalls === 1 ? "" : "s"}`, + ); } if (testCase.harness?.timingMs !== undefined) { parts.push(formatDuration(testCase.harness.timingMs)); From fa95ce22767a9958f17f0b7e8f1854d362ba2946 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 13:47:53 +0000 Subject: [PATCH 14/15] test(reporter): align smoke summary copy --- .github/workflows/github-reporter-smoke.yml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/workflows/github-reporter-smoke.yml b/.github/workflows/github-reporter-smoke.yml index 7bd98fb..e5cb7a7 100644 --- a/.github/workflows/github-reporter-smoke.yml +++ b/.github/workflows/github-reporter-smoke.yml @@ -361,21 +361,21 @@ jobs: grep -F "Judge Score" "$GITHUB_STEP_SUMMARY" grep -F "SmokeJudge 0.10" "$GITHUB_STEP_SUMMARY" grep -F "AmountJudge 0.35" "$GITHUB_STEP_SUMMARY" - grep -F "Harness Errors" "$GITHUB_STEP_SUMMARY" + grep -F "App errors" "$GITHUB_STEP_SUMMARY" grep -F "refund ledger rejected amount" "$GITHUB_STEP_SUMMARY" - grep -F "Final Output" "$GITHUB_STEP_SUMMARY" + grep -F "Output" "$GITHUB_STEP_SUMMARY" grep -F "Tool Status" "$GITHUB_STEP_SUMMARY" grep -F "lookupInvoice ok" "$GITHUB_STEP_SUMMARY" - grep -F "| Status | failed |" "$GITHUB_STEP_SUMMARY" - grep -F "| Evals | 8 passed, 2 failed, 10 total |" "$GITHUB_STEP_SUMMARY" - grep -F "| Score | avg 0.59, min 0.10 |" "$GITHUB_STEP_SUMMARY" + grep -F "| Status | Failed |" "$GITHUB_STEP_SUMMARY" + grep -F "| Eval cases | 8 passed, 2 failed, 10 total |" "$GITHUB_STEP_SUMMARY" + grep -F "| Score | average 0.59, lowest 0.10 |" "$GITHUB_STEP_SUMMARY" if grep -F "| Tests |" "$GITHUB_STEP_SUMMARY"; then echo "test totals row should not be shown" exit 1 fi - grep -F "| Application Usage | 1,302 tokens, 9 tools |" "$GITHUB_STEP_SUMMARY" - grep -F "| Judge Usage | none |" "$GITHUB_STEP_SUMMARY" - grep -F "| Total Usage | 1,302 tokens, 9 tools |" "$GITHUB_STEP_SUMMARY" + grep -F "| App usage | 1,302 tokens, cost unavailable, 9 tool calls |" "$GITHUB_STEP_SUMMARY" + grep -F "| Judge usage | none |" "$GITHUB_STEP_SUMMARY" + grep -F "| Total usage | 1,302 tokens, cost unavailable, 9 tool calls |" "$GITHUB_STEP_SUMMARY" grep -F "::error file=apps/demo/evals/smoke.eval.ts,line=12,col=5,title=vitest-evals::refund eval > rejects fraud - score 0.10 - SmokeJudge - expected approved but got denied" tmp/github-reporter-smoke/stdout.txt grep -F "::error file=apps/demo/evals/smoke.eval.ts,line=28,col=5,title=vitest-evals::refund eval > returns wrong refund amount - score 0.35 - AmountJudge - expected amount=5000, got amount=499" tmp/github-reporter-smoke/stdout.txt grep -F "::warning::GitHub Check Run skipped: missing GITHUB_TOKEN" tmp/github-reporter-smoke/stdout.txt From 936e14f60af962386e5ee30c3c4ba13bb4bcb148 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 13:49:21 +0000 Subject: [PATCH 15/15] test(reporter): update gated smoke title --- .github/workflows/github-reporter-smoke.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/github-reporter-smoke.yml b/.github/workflows/github-reporter-smoke.yml index e5cb7a7..21cb809 100644 --- a/.github/workflows/github-reporter-smoke.yml +++ b/.github/workflows/github-reporter-smoke.yml @@ -442,7 +442,7 @@ jobs: test "$ACTION_GATE_STATUS" = "passed" test "$ACTION_PASS_RATE" = "0.80" test "$ACTION_EVALS_FAILED" = "2" - test "$ACTION_GATE_TITLE" = "Eval pass rate 80.0% — floor 70.0%" + test "$ACTION_GATE_TITLE" = "Eval pass rate 80.0% — minimum 70.0%" - name: Run gated reporter action below floor id: reporter_action_gated_fail