Repository navigation
Reuse benchmark builds within each CI pipeline run - #10
Closed
LoopedBard3 wants to merge 6 commits into
Closed
LoopedBard3 wants to merge 6 commits into
LoopedBard3 wants to merge 6 commits into
Conversation
* Update Azure Cobalt profiles to four cores Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Remove unnecessary README update Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Let workload defaults select the listening port instead of overriding container scenarios to port 5000. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Add generated Minimal Todo container benchmarks Adds a frozen, reviewed Minimal Todo benchmark cohort (C#, Go, Rust) with a versioned HTTP contract and one native Crank scenario file. Source generation, admission review, pre-promotion comparison, and quota calibration tooling for this cohort -- including the HTTP contract fixture and the one-time state/body verification that uses it -- are owned by the separate dotnet-performance-tools repository; this change is the shipped, day-to-day Crank runtime surface only. - One scenario/minimal-todo/minimal-todo.benchmarks.yml defines all three language scenarios with two roles each (application, load; no separate contract-check/before/after roles and no checks/ fixture fetch). Regular runs validate: application readiness (the startup gate), then the measured Bombardier result strictly (exact URL/method/client/rate/duration/ connections/timeout/request headers, response counters all 2xx with zero other status classes, zero transport errors, delivered rate/duration within 5% of offered), plus CPU/memory tail coverage and sample-gap checks. No requiredArchitecture/architecture gating: Docker images are multi-arch, so the same scenario runs unmodified on x64 or ARM64 agents. Manual connected runs use the shared build/ci.profile.yml / build/azure.profile.yml profiles like every other scenario; there is no scenario-specific profile file. - Each app's own Dockerfile builds directly from floating, multi-arch base images: .NET 11 nightly SDK/ASP.NET (no stable .NET 12 tag is published yet), golang:1.27-bookworm, and rust:1.96-bookworm builders, all finishing on a shared ubuntu:26.04 runtime image (verified against the live .NET 11 nightly manifest). No pinned digests, no Dockerfile.daily/global.json SDK pin, no build-context receipt tooling. - Scheduler template (build/minimal-todo-scenarios.yml) is enabled with the previously accepted matched-contract rate (25200 req/s, 1 CPU, 512 MiB, 128 connections, 30s warmup, 120s duration, 2s request timeout, port 8080), dispatched to the gold-lin, azure-arm64, and cobalt-cloud-lin pods. Validation: offline Crank config resolution (--profile offline) succeeds for all three scenarios (now two roles each); the full scripts/pod-scheduler test suite (43 tests, including the committed-YAML snapshot check) passes; no dead references to any removed file remain. Docker and the floating .NET 11/ Go 1.27/Rust 1.96 toolchains were not available in this environment, so the Dockerfiles could not be build-verified here. This round changed the Docker base images and the C# project's target framework; the previously accepted 25200 req/s rate was calibrated against the prior pinned images and has not been re-run against these new images. The first post-merge scheduled run is the first operational confirmation under the new images, not a re-validation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d9d74a4-18fa-4e63-a55d-792c3aabe5f8 * Remove redundant job options and duplicate metadata from Minimal Todo scenario Follow-up nits on the generated Minimal Todo scenario, no history rewrite: - Remove the custom offline variable/profile entirely (no other scenario in this repository uses one). The application/contract-check jobs no longer clear endpoints conditionally; the bombardier job's onConfigure simply validates that application/load each resolve to exactly one agent endpoint, matching the connected-run assumption every other scenario makes. - Remove job-level options that only restated Crank's own defaults (reuseBuild/reuseSource/collectCounters/downloadOutput/ downloadOutputOutput/displayBuild/displayOutput/noClean: false) and that have no prior art anywhere else in scenarios/*.benchmarks.yml; options blocks now contain only requiredOperatingSystem: linux, matching the repository-wide convention. - Remove the per-app job variables language/framework/sdkImage/runtimeImage and the onResultsCreated block that copied them into todo/* properties: they duplicated the todo/language and todo/framework --property values the scheduler template already passes, and were not read by the build itself. The calibration summary (todo/calibration/*, todo/cpu/*, todo/memory/*) is unchanged. - README: removed the offline-resolution section describing the now-removed profile; no other content changed. Validation: static config assertions confirm the offline variable/profile, the removed options, and the removed job variables are all absent. The full scripts/pod-scheduler test suite (43 tests) passes; regenerating both CI pipelines produces no diff. Structural resolution was re-verified with crank --debug against a test-only sanitized temp copy of the scenario file (outside this repository, never committed) with a non-listening loopback placeholder endpoint per role -- --debug only resolves and serializes the job graph, so no real agent was contacted. Docker and the floating .NET 11/ Go 1.27/Rust 1.96 toolchains remain unavailable in this environment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d9d74a4-18fa-4e63-a55d-792c3aabe5f8 * Use the official dotnet/crank Bombardier job instead of redefining it Follow-up nit, no history rewrite: - Import the official, unmodified Bombardier job via https://raw.githubusercontent.com/dotnet/crank/main/src/Microsoft.Crank.Jobs.Bombardier/bombardier.yml instead of redefining its source/framework/SDK/runtime/arguments/ onConfigure/options locally. Deleted the entire local jobs.bombardier definition, including its workload-variable range checks and the application/load endpoint-count/URL-shape validation (no custom endpoint validation needed: the shared ci/azure profile always supplies a real endpoint, matching every other scenario in this repository). - Renamed the top-level requestTimeout variable to timeout (default unchanged at 2) to match the imported job's own variable name, and updated build/minimal-todo-scenarios.yml's --variable requestTimeout=2 to --variable timeout=2. - Each scenario's load role now references the imported bombardier job with explicit variables path: /todos, transport: http1, presetHeaders: none, and customHeaders for Accept: application/json, Connection: keep-alive, and Accept-Encoding: identity. rate/connections/warmup/duration/timeout/ serverAddress/serverPort are intentionally not set per-role: they are supplied the same way as before, via the scheduler template's explicit --variable flags (or an operator's own CLI variables for a manual run), not a scenario-file override. - README: describes the imported Bombardier job and its explicit variables instead of the old pinned-commit wrapper description. Validation: resolved a test-only sanitized temp copy of the scenario file (outside this repository, never committed) with crank --debug, using a non-listening loopback placeholder endpoint -- confirmed the import resolves and merges correctly for all three languages, both with no CLI variables (imported job's own defaults apply, as expected) and with the same --variable overrides the scheduler template passes (connections, warmup, duration, rate, timeout, serverPort all bind correctly into the imported job, and the resulting Bombardier arguments match exactly what was previously hardcoded). The only network access during this validation was crank's own normal fetch of the imported YAML's raw GitHub URL (the same mechanism every scenario that imports wrk.yml/bombardier.yml already relies on) plus a refused local loopback connection; no real agent was contacted. The full scripts/pod-scheduler test suite (43 tests) passes; regenerating both CI pipelines produces no diff beyond the already-staged timeout rename in build/minimal-todo-scenarios.yml. Docker and the floating .NET 11/Go 1.27/Rust 1.96 toolchains remain unavailable in this environment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d9d74a4-18fa-4e63-a55d-792c3aabe5f8 * Add optional direct-Crank-build C# comparison scenario (not scheduled) Follow-up, no history rewrite. Adds a side-by-side comparison point alongside the existing Docker csharp scenario, which remains unchanged (verified byte-identical in the diff). - New job todo-csharp-project builds src/TodoApi/TodoApi.csproj directly with Crank (project: + framework: net11.0 + channel: latest, confirmed real Crank Job model fields via the actual dotnet/crank source), no Docker fields at all. Reuses the exact same sources/sourceRevision main-or-40hex validation and CPU/memory/port onConfigure logic as todo-csharp. - New scenario csharp-project pairs todo-csharp-project with the same imported Bombardier load job and variables (path: /todos, transport: http1, presetHeaders: none, customHeaders for Accept/Connection/Accept-Encoding) as the csharp scenario. - Not added to build/minimal-todo-scenarios.yml or any pod/CI configuration: the scheduled csharp scenario stays the Docker baseline; csharp-project is only ever run manually via an explicit --scenario csharp-project. - README: documents the optional scenario, why channel: latest is used (coherent latest SDK/runtime/ASP.NET Core, not a floating Docker digest), and that Crank's own CPU/memory limits (cgroups) apply identically to both jobs so a comparison isolates packaging/runtime/host-OS effects rather than a resource-quota difference. Validation: resolved a test-only sanitized temp copy of the scenario file (outside this repository, never committed) with crank --debug for all four scenarios (csharp, go, rust, csharp-project) using a non-listening loopback placeholder endpoint -- all resolve cleanly; confirmed the csharp-project application job resolves Project/Framework/Channel/Port correctly and has no DockerFile. The full scripts/pod-scheduler test suite (43 tests) passes; regenerating both CI pipelines produces no diff (this scenario is not referenced by any pod/CI configuration). No real agent was contacted. Docker and the floating .NET 11/Go 1.27/Rust 1.96 toolchains remain unavailable in this environment. Live/lab execution (the requested one-time paired Docker-vs-direct comparison on real agents) is out of scope for this commit and this session: this checkout has no credentials or network path to real lab agents, consistent with this effort's established division of responsibility where live dispatch is performed by the parent/native executor, never by this candidate-integration checkout. Reported this limitation back to the requester; the comparison itself has not been run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d9d74a4-18fa-4e63-a55d-792c3aabe5f8 * Fix C# Dockerfile locked-mode restore order; correct stale README claims Addresses five PR review comments on aspnet#2215 (normal follow-up commit, no history rewrite, no scenario/scheduler/workload change): 1. (csharp/Dockerfile, review 4213810222) The pre-restore COPY only included TodoApi.csproj, so `dotnet restore --locked-mode` ran with no packages.lock.json on disk; the real lock file only arrived in the later `COPY src/TodoApi/ src/TodoApi/` step, after restore had already run. Added packages.lock.json to the pre-restore COPY alongside the csproj (not excluded by .dockerignore). Locked-mode and the lock file itself are unchanged -- this only fixes when the lock file becomes visible to restore. 2. (csharp/README.md, review 4213810258) The "Verification in the supplied environment" section (Windows SDK 10.0.401, runtimes 10.0.12) is now explicitly labeled as historical/original .NET 10 evidence, predating this round's floating .NET 11 images; states plainly that the current .NET 11 configuration was not build-verified in this checkout. Also corrected two stale present-tense claims: "strict runtime selection" (the RuntimeFrameworkVersion/roll-forward pin it described was removed from Directory.Build.props this round) and "the pinned runtimes" for framework-dependent RID publishes (now "an installed, compatible runtime"), plus the package-lock description (the app does have its own packages.lock.json, just with an empty recorded dependency set, not "no package lock"). 3/4. (go/README.md, reviews 4213810293 and 4213810316) Corrected the top-of-file claim that "the Docker builder pins that version and its image digest" -- the builder floats on `golang:1.27-bookworm` with no digest pin, and `go.mod`'s `go 1.27` is a minimum-version directive, not an exact pin. Corrected the Container section and Files list, which claimed "native builder to cross-compile for the selected target" and "pinned multi-stage Linux container build": the Dockerfile has no BUILDPLATFORM/TARGETARCH, so `docker build --platform ...` selects a single target platform for the whole build (native for that platform, via emulation when needed) rather than cross-compiling from one builder architecture to another; that's a different mechanism from the standalone GOOS/GOARCH cross-compilation commands already documented separately (left unchanged). 5. (rust/README.md, review 4213810360) Corrected "a minimal Debian runtime" in the current "Linux containers" section to `ubuntu:26.04` (the actual current runtime base). The Validation section's historical "Runtime image" row (Debian glibc 2.36-9+deb12u14) is preserved verbatim but its heading/intro now explicitly label it as the original Debian-based build from before this round's switch to Ubuntu, so it can't be misread as evidence the current image was tested. Validation: for item 1, Docker is unavailable in this environment, so I built the smallest possible local reproduction instead -- a byte-for-byte copy of the real build context, staged exactly like each Dockerfile COPY step, restored/published with the local SDK (TargetFramework temporarily overridden net11.0->net10.0 in that throwaway copy only, since net11.0 isn't installed locally; no change to the real project). Both the old and the fixed COPY order restore and publish successfully (exit 0) for this specific project today: TodoApi.csproj has zero package references, and neither the csproj nor Directory.Build.props sets RestorePackagesWithLockFile, so no lock file is auto-generated and --locked-mode has nothing to validate against either way right now -- reproducing NU1004 was not possible today, and I am not claiming it was observed. The fix is still correct and makes --locked-mode meaningful going forward (e.g. once a real dependency is added, restore will then actually see and validate against the committed lock file before building, instead of silently not reading it). All four touched files: git diff --check is clean (one pre-existing CRLF/cr-at-eol cosmetic flag per file, consistent with each file's existing line-ending convention, not a real defect, same pattern as every prior round in this PR). No scenario, scheduler, pod, or workload file was touched. Per-comment coverage: all five identified review comments (4213810222, 4213810258, 4213810293, 4213810316, 4213810360) are addressed by this commit. Threads are intentionally left unresolved pending confirmation this fix is accepted. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d9d74a4-18fa-4e63-a55d-792c3aabe5f8 * Schedule Minimal Todo C# daily via direct build; Go/Rust weekly via Docker Normal follow-up commit, no history rewrite/scheduler-architecture change/ main-branch change. Implements a new scheduling policy while keeping the Docker generation/comparison path fully intact: - build/minimal-todo-scenarios.yml: the C# scenario entry now submits `--scenario csharp-project` (the existing direct, non-Docker build) instead of `--scenario csharp`, unconditionally on every regular scheduled run (condition: 'true', same cadence as before). todo/language=csharp and todo/framework=net11.0 properties are unchanged. The Go and Rust entries keep their existing Docker `go`/`rust` scenarios but add `condition: Math.round(Date.now() / 43200000) % 14 == 0` (true once every 14 half-day scheduled slots, i.e. once per ~7 days for each pipeline's own 12-hour cron), sharing the same slot value so both languages fall on the same calendar day within a given pipeline. The published Service Bus message's condition now combines both: `(parameters.condition) && (scenario.condition)`, matching the existing idiom in build/containers-scenarios.yml, rather than bypassing the pod-level condition. - The Docker todo-csharp/csharp, todo-go/go, and todo-rust/rust jobs and scenarios in scenarios/minimal-todo/minimal-todo.benchmarks.yml are completely unchanged (not renamed, not removed) and remain the canonical generation/update/admission/pre-promotion comparison path. Rate (25200), 1 CPU, 512 MiB, 128 connections, 30s/120s warmup/duration, 2s timeout, port 8080, and the shared steadystate config are all unchanged. - scenarios/minimal-todo/README.md: replaced the old "optional, not scheduled" description of csharp-project with a new "Execution paths" section mapping which scenario is scheduled where and why, including an explicit, hedged disclosure of the preliminary Docker-vs-direct comparison (~3% lower CPU quota for direct, similar latency; process-exit verification had an unresolved local-reader-bug caveat) -- stated plainly as not a fully accepted paired test or recalibration; the retained 25200 rate is the prior operational policy value, not re-derived from this comparison. - build/README.md: updated the Minimal Todo section to describe the new per-language cadence and point at the scenario README for the full mapping. Validation: regenerating both CI pipelines produces no diff (the cadence condition lives entirely inside the scenario-dispatch template, which is expanded at actual pipeline-run time, not at pod-scheduler generation time). The full scripts/pod-scheduler test suite (43 tests) passes. Confirmed no other file overrides minimal-todo-scenarios.yml's `scenarios` parameter default (the only two pipeline call sites pass only connection/serviceBusQueueName/serviceBusNamespace/arguments), so adding a `condition` key to every default scenario item cannot cause an Azure template "object does not contain property" expansion error elsewhere. Verified the weekly-condition formula numerically against both pipelines' actual cron schedules (ci01/ci02 "0 3/12 * * *", Azure "0 2/12 * * *") across 28 consecutive triggers (14 days) from several arbitrary start dates: exactly 2 of 28 (i.e. exactly 1 per 7 days) evaluate true for each cadence, every time -- a direct consequence of the triggers being an exact-12-hour arithmetic sequence, mod 14. Re-verified all four scenarios (csharp, go, rust, csharp-project) still resolve cleanly via `crank --debug` against a test-only sanitized temp copy (outside this repository, never committed, non-listening loopback placeholder endpoint; no real agent contacted). git diff --check is clean (pre-existing per-file CRLF/cr-at-eol cosmetic flag, same pattern as every prior round, not a real defect). No app code, build mode, or toolchain was changed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d9d74a4-18fa-4e63-a55d-792c3aabe5f8 * Simplify app job runtime config to normal Crank job/application.* options Normal follow-up commit, no history rewrite/scheduler-architecture change/ live jobs. Replaces the bespoke per-job onConfigure hooks (resource-variable -> typed-property mapping, main-or-SHA format validation, forced LocalFolder=null) with plain Crank job properties, matching the established pattern in build/containers-scenarios.yml (--application.cpuLimitRatio / --application.memoryLimitInBytes). - Removed all four app onConfigure blocks (todo-csharp, todo-csharp-project, todo-go, todo-rust): no replacement runtime JS, no forced LocalFolder clear, no main-or-40hex script guard. Normal Crank job/profile/CLI semantics now apply, including local-source overrides (previously impossible: the removed hook unconditionally reset LocalFolder to null on every run regardless of any --application.sources.app.localFolder override). - Each app job now sets typed defaults directly: cpuLimitRatio: 1, memoryLimitInBytes: 536870912, port: 8080 -- preserving the existing 1 CPU / 512 MiB manual default (deleting the hook with no replacement would have silently made these 0/unlimited). All three are overridable via --application.cpuLimitRatio / --application.memoryLimitInBytes / --application.port, same as any other scenario. - The "app" source keeps its name/destination/remote-repository/cache policy; branchOrCommit now defaults to the literal main (a normal branch clone) instead of a templated "#{{sourceRevision}}" marker. The scenario's sourceRevision variable is removed; pinning an exact commit is now a normal CLI source override, --application.sources.app.branchOrCommit "#<sha>" (the nested sources dictionary form, not the obsolete --application.source.* alias -- "app" is a named source, not the default unnamed one). The obsolete cpu/memoryBytes/sourceRevision scenario variables are removed now that every consumer has been rewired; the load/workload variables (serverAddress/serverPort/rate/warmup/duration/ connections/timeout) and the imported Bombardier headers are unchanged. - Each app's PORT environment variable now derives from the job's own resolved port ("{{job.Port}}") instead of the removed serverPort-reading hook, so --application.port overrides correctly flow into the container's listening port. The imported Bombardier load job's own default port (5000) no longer matches the app's default (8080); each scenario's load role now sets variables.serverPort: 8080 (a plain per-role variable default, no JS, the same mechanism already used for path/transport/ presetHeaders/customHeaders) so the two match by default, while a CLI --variable serverPort=<port> still overrides both app and load together when passed explicitly. - build/minimal-todo-scenarios.yml: replaced --variable cpu=1 --variable memoryBytes=536870912 --variable sourceRevision=$(Build.SourceVersion) with --application.cpuLimitRatio 1 --application.memoryLimitInBytes 536870912 --application.port 8080 --application.sources.app.branchOrCommit \"#$(Build.SourceVersion)\" (JSON- escaped double quotes around the commit value, matching this same messageBody's existing quoting convention for other values, so the leading "#" cannot be read as a shell comment start by whatever splits this string downstream). --variable serverPort=8080 and the other workload variables are unchanged; result properties (cpu/memoryBytes/ todo/*) are unchanged and remain truthful. - Docs: updated scenarios/minimal-todo/README.md's "Scenario" section to describe the new typed-property/CLI-override shape instead of sourceRevision/onConfigure, and fixed the two stale runtime YAML comment blocks (one job comment, one scenario comment) that still described csharp-project as "optional"/"not part of any scheduled lane" -- it has been the scheduled day-to-day C# lane since commit 3a802b0. Also corrected a "ci01/ci02" imprecision to "ci01" (ci02's cron and contents are unrelated to Minimal Todo). Validation: all four scenarios (csharp, go, rust, csharp-project) resolve cleanly via crank --debug against a test-only sanitized temp copy (outside this repository, never committed, non-listening loopback placeholder endpoint; no real agent contacted), confirmed for every combination the exact requirement calls for: - No CLI overrides: CpuLimitRatio=1.0, MemoryLimitInBytes=536870912, Port=8080, BranchOrCommit="main", EnvironmentVariables.PORT="8080", and the load job's resolved URL is http://.../8080/todos (serverPort default now matches) -- for all four scenarios. - With --application.cpuLimitRatio 0.5 --application.memoryLimitInBytes 268435456 --application.port 8091 --application.sources.app.branchOrCommit "#<40-hex-sha>" --variable serverPort=8091: CpuLimitRatio=0.5, MemoryLimitInBytes=268435456, Port=8091, BranchOrCommit="#<sha>" (not truncated at the "#"), EnvironmentVariables.PORT="8091" (matches Job.Port, not left stale), and the load job's resolved URL is http://.../8091/todos. - --application.sources.app.localFolder override is honored (previously impossible due to the removed forced-clear hook). - The imported official Bombardier job is still used and its arguments (headers, --http1, path) are unchanged in both cases. Separately simulated the exact build/minimal-todo-scenarios.yml messageBody JSON with placeholder values substituted for every ${{ }}/$() token: the result parses as valid JSON and the resolved args string contains --application.sources.app.branchOrCommit "#<sha>" with the quotes intact around the "#", matching this exact quoting convention already used successfully elsewhere in the same messageBody for other special-character values. The full scripts/pod-scheduler test suite (43 tests) passes; regenerating both CI pipelines produces no diff beyond the minimal-todo-scenarios.yml edit itself. git diff --check is clean (the one remaining flag is the same pre-existing per-file CRLF/cr-at-eol cosmetic pattern as every prior round, confirmed not a real defect). No app code, Dockerfile, job/scenario rename, SelfContained/framework/channel, or generation/comparison Docker path was changed; the Go/Rust weekly and C# regular-cadence scheduling from the prior commit is unchanged. No live agents or Docker commands were used. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d9d74a4-18fa-4e63-a55d-792c3aabe5f8 * Add memory/rate annotations; remove test/fixture gunk from runtime repo Normal follow-up commit, no history rewrite/scheduler-architecture change/ merge/live jobs. The PR was already marked ready for review (draft: false); that state is preserved. RUNTIME CONFIG (no workload/value change): - Added a `# 512 MiB` human comment to each of the four app jobs' memoryLimitInBytes: 536870912 (todo-csharp, todo-csharp-project, todo-go, todo-rust). - Added an explicit `rate: 100 # Override with --variable rate=...` to each of the four scenarios' load role (csharp, go, rust, csharp-project). The imported Bombardier job's own default rate is 0 (uncapped); the scenario's former root-level `rate: 100` variable never actually reached the load job (the same non-propagation behavior already documented for serverPort), so a manual run with no --variable override was silently uncapped rather than the intended low smoke-test default. Removed the now-redundant root `rate` variable (nothing in this file templates `{{rate}}` directly; only the per-role default has any effect). A CLI `--variable rate=...` override (e.g. the scheduled 25200) still wins over this default, confirmed empirically. TEST/FIXTURE/EVIDENCE REMOVAL (runtime repo only -- tools-side canonical archive is untouched, already independently preserved per 248's prior confirmation): - Deleted csharp/tests/TodoApi.Tests/ entirely (ApiFixture.cs, AppProcess.cs, ContractTests.cs, LifecycleTests.cs, TodoApi.Tests.csproj, minimal-todo-v1.expected.json, packages.lock.json). - Deleted go/lifecycle_test.go, go/server_test.go, go/testdata/ (the golden fixture). - Deleted rust/tests/ entirely (contract.rs, lifecycle.rs, fixtures/minimal-todo-v1.expected.json). - Zero production .cs/.go/.rs files were touched (confirmed via `git diff --cached --stat` filtered to those paths: no output). The fixed five-Todo dataset, Program.cs/Todo.cs, main.go/server.go, main.rs/lib.rs (including its embedded `#[cfg(test)]` unit tests, which are part of the production file and were left untouched), Cargo.toml/Cargo.lock/go.mod, and all Dockerfiles needed for regular runs are all unchanged. WIRED BUILD SURFACES: - rust/Dockerfile: removed `COPY tests ./tests` and the `FROM build AS test` / `RUN cargo test --release --locked` stage; kept the production `cargo build --release --locked`, the ubuntu:26.04 runtime stage, nonroot user, and entrypoint unchanged. - csharp/TodoApi.slnx: removed the `/tests/` folder and TodoApi.Tests.csproj reference, now a single-project solution (src/TodoApi/TodoApi.csproj). - csharp/src/TodoApi/TodoApi.csproj: removed the dead `<InternalsVisibleTo Include="TodoApi.Tests" />` (csproj metadata only, no .cs edit). - Dead dockerignore/gitignore patterns removed: csharp/.dockerignore's `tests` line, csharp/.gitignore's now-unreachable `TestResults/` line, go/.dockerignore's `*_test.go`/`testdata/` lines, rust/.dockerignore's `!tests/`/`!tests/**` allow-list lines (its "ignore everything, allow-list specific paths" pattern no longer needs to admit a path that doesn't exist). README CLEANUP (operator-facing only; full generated test suite, fixture, and validation/campaign history remain owned by the separate dotnet-performance-tools canonical archive): - csharp/README.md, go/README.md, rust/README.md: rewritten concisely to app/framework description, build, run, contract, PORT, readiness, shutdown, and container build/run instructions only. Removed the historical Windows-SDK-10/original-Debian-image validation sections, test run commands/counts, fixture SHA-256 hashes, local artifact/bin/log/ evidence directory listings, and the Files sections' now-dead test-file entries. - scenarios/minimal-todo/README.md: documents the new rate/memory/serverPort per-role defaults; states plainly that production code only ships here (no test projects/fixtures); adds an explicit operational note that the 25200 req/s rate is a uniform policy value applied to all three languages and both the Docker and direct C# builds, not a per-language or per-resource-point capacity guarantee; reinforces that the full generated test suite/fixture/checker/review/campaign history is owned by dotnet-performance-tools. The C#-direct-regular/Go-Rust-weekly cadence and Docker-comparison-availability design from the prior two commits is unchanged and explicitly preserved. Validation: `git diff --cached --stat` filtered to *.cs/*.go/*.rs shows zero production file changes (only test-file deletions). Grepped the full repository for references to every deleted file/directory name: zero remaining matches (no dead links in any README or build/*.yml). All four scenarios (csharp, go, rust, csharp-project) resolve cleanly via `crank --debug` against a test-only sanitized temp copy (outside this repository, never committed, non-listening loopback placeholder endpoint; no real agent or Docker command used), confirming: default load rate is now 100 (previously an uncapped 0), memoryLimitInBytes is unchanged at 536870912, and a `--variable rate=25200` CLI override still wins. The full scripts/pod-scheduler test suite (43 tests) passes; regenerating both CI pipelines produces zero diff. git diff --check is clean (the only flagged lines are the same pre-existing per-file CRLF/cr-at-eol cosmetic pattern seen in every prior round for a fully-rewritten file -- confirmed zero genuine trailing whitespace). Docker and the floating toolchains remain unavailable in this environment, so the Dockerfiles could not be build-verified here; this is an honest, disclosed limitation, not a passed build. No shared agent, Docker daemon, install, or remote probe was started. No live/native/lab/SQL/workflow dispatch was performed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d9d74a4-18fa-4e63-a55d-792c3aabe5f8 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d9d74a4-18fa-4e63-a55d-792c3aabe5f8
* Move PR benchmarks to Gold x64 relay machines Replace disconnected Perf/Citrine selections with existing Gold Linux and Windows profiles. Retire unavailable runtime ARM64 dispatch and configs, document support, and validate scenario imports and pipeline architecture selections. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: efd56148-f715-4c87-b1ec-7f70ad6e66c2 * Remove migration-specific PR benchmark tests Keep the Gold machine migration while removing its bespoke validation harness and README command. Completed offline validation remains documented in the PR. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: efd56148-f715-4c87-b1ec-7f70ad6e66c2 --------- Copilot-Session: efd56148-f715-4c87-b1ec-7f70ad6e66c2
Remove retired Citrine and dependent ARM/Siryn profile definitions, update examples to active Gold profiles, and reuse shared profiles in legacy scenarios. Correct the Gold Windows relay database address while preserving controller queues and active machine configurations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 947a7758-5660-4f23-b447-e7006ace68fd
Cache application and supporting builds for container, gRPC, HttpClient, SignalR, and WebSocket matrices using run-scoped source and build keys. Isolate Docker and compression variants, document the cache lifecycle, and add configuration regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d203afa3-f8a9-47b3-9353-e38e91a647e9
Owner
Author
|
Opened in the fork by mistake. Replaced by the draft PR targeting upstream: aspnet#2222 |
There was a problem hiding this comment.
🟡 Changes recommended
Shared Docker image tags can substitute artifacts between pipeline runs despite run-scoped cache keys.
3 open issues
What changed in this PR
This PR adds pipeline-run-scoped build reuse to reduce repeated benchmark setup. It also introduces Minimal Todo workloads and updates benchmark machine selection and scheduling.
Changes:
- Enable build reuse for five benchmark matrices, with variant-specific cache keys and regression checks.
- Add C#, Go, and Rust Minimal Todo applications, container builds, and scheduled scenarios.
- Migrate PR benchmarks to Gold machines, retire unavailable profiles and ARM64 PR jobs, and update documentation.
| File | Description |
|---|---|
| src/BenchmarksApps/Samples/Template/sample.yml | Update the example comment to use Gold. |
| src/BenchmarksApps/MinimalTodo/rust/src/main.rs | Add Rust startup, readiness, and shutdown handling. |
| src/BenchmarksApps/MinimalTodo/rust/src/lib.rs | Add Todo routes, serialization, and port validation. |
| src/BenchmarksApps/MinimalTodo/rust/rust-toolchain.toml | Select the Rust toolchain. |
| src/BenchmarksApps/MinimalTodo/rust/README.md | Document Rust operation and HTTP behavior. |
| src/BenchmarksApps/MinimalTodo/rust/Dockerfile | Add a non-root Rust container build. |
| src/BenchmarksApps/MinimalTodo/rust/Cargo.toml | Define Rust dependencies and package settings. |
| src/BenchmarksApps/MinimalTodo/rust/Cargo.lock | Lock the Rust dependency graph. |
| src/BenchmarksApps/MinimalTodo/rust/.gitignore | Exclude Rust build output. |
| src/BenchmarksApps/MinimalTodo/rust/.dockerignore | Restrict the Rust build context. |
| src/BenchmarksApps/MinimalTodo/go/server.go | Add Go Todo and health handlers. |
| src/BenchmarksApps/MinimalTodo/go/README.md | Document Go operation and HTTP behavior. |
| src/BenchmarksApps/MinimalTodo/go/main.go | Add Go startup, validation, and shutdown handling. |
| src/BenchmarksApps/MinimalTodo/go/go.mod | Define the Go module and minimum toolchain. |
| src/BenchmarksApps/MinimalTodo/go/Dockerfile | Build a static, non-root Go container. |
| src/BenchmarksApps/MinimalTodo/go/.dockerignore | Exclude unnecessary Docker context files. |
| src/BenchmarksApps/MinimalTodo/Directory.Build.targets | Isolate applications from ancestor build targets. |
| src/BenchmarksApps/MinimalTodo/csharp/TodoApi.slnx | Add the C# solution. |
| src/BenchmarksApps/MinimalTodo/csharp/src/TodoApi/TodoApi.csproj | Define the ASP.NET Core project. |
| src/BenchmarksApps/MinimalTodo/csharp/src/TodoApi/Todo.cs | Define immutable records and JSON property order. |
| src/BenchmarksApps/MinimalTodo/csharp/src/TodoApi/Program.cs | Add C# routes, configuration, and lifecycle handling. |
| src/BenchmarksApps/MinimalTodo/csharp/src/TodoApi/packages.lock.json | Record the empty package dependency set. |
| src/BenchmarksApps/MinimalTodo/csharp/README.md | Document C# operation and HTTP behavior. |
| src/BenchmarksApps/MinimalTodo/csharp/NuGet.Config | Define package sources. |
| src/BenchmarksApps/MinimalTodo/csharp/Dockerfile | Add a multi-stage, non-root .NET container. |
| src/BenchmarksApps/MinimalTodo/csharp/Directory.Build.props | Set framework and compiler options. |
| src/BenchmarksApps/MinimalTodo/csharp/.gitignore | Exclude C# build artifacts. |
| src/BenchmarksApps/MinimalTodo/csharp/.dockerignore | Exclude generated files from Docker context. |
| src/BenchmarksApps/MinimalTodo/.gitattributes | Preserve application source bytes across checkouts. |
| src/BenchmarksApps/BuildPerformance/buildperformance.yml | Replace the embedded profile with shared profiles. |
| src/Benchmarks/json.benchmarks.yml | Refresh imports and remove the embedded retired profile. |
| scenarios/README.md | Update machine guidance and introduce Minimal Todo. |
| scenarios/platform.benchmarks.yml | Update the example comment to use Gold. |
| scenarios/mono.benchmarks.yml | Update the example comment to use Gold. |
| scenarios/minimal-todo/README.md | Document workload execution, scheduling, and calibration limits. |
| scenarios/minimal-todo/minimal-todo.benchmarks.yml | Define Docker and direct C# benchmark scenarios. |
| scenarios/minimal-todo/.gitattributes | Set scenario YAML line endings. |
| scenarios/containers.benchmarks.yml | Assign distinct image names to Dockerfile variants. |
| scenarios/aspnet.profiles.yml | Remove retired profiles and correct Gold database addressing. |
| scenarios/aspnet.profiles.standard.yml | Remove retired standard machine profiles. |
| build/websocket-scenarios.yml | Enable run-scoped WebSocket build reuse. |
| build/tests/test_build_reuse.py | Add cache isolation and template wiring checks. |
| build/signalr-scenarios.yml | Enable server and shared client build reuse. |
| build/README.md | Document caching, machine migration, and Minimal Todo scheduling. |
| build/prbenchmarks.yml | Update job labels and remove ARM64 PR-benchmark dispatch. |
| build/prbenchmarks.runtime.windows_x64.config.yml | Select Gold Windows for runtime PR benchmarks. |
| build/prbenchmarks.runtime.windows_arm64.config.yml | Delete the retired Windows ARM64 configuration. |
| build/prbenchmarks.runtime.linux_x64.config.yml | Select Gold Linux for runtime PR benchmarks. |
| build/prbenchmarks.runtime.linux_arm64.config.yml | Delete the retired Linux ARM64 configuration. |
| build/prbenchmarks.efcore.config.yml | Select Gold Windows for EF Core PR benchmarks. |
| build/prbenchmarks.aspnetcore.config.yml | Select Gold Linux and Windows for ASP.NET Core. |
| build/minimal-todo-scenarios.yml | Dispatch regular C# and weekly Go/Rust workloads. |
| build/job-variables.yml | Add shared cache-profile and Minimal Todo configuration references. |
| build/httpclient-scenarios.yml | Enable run-scoped client and server build reuse. |
| build/grpc-scenarios.yml | Identify implementation variants and enable build reuse. |
| build/containers-scenarios.yml | Enable run-scoped application and database build reuse. |
| build/ci.profile.yml | Remove retired CI machine profiles. |
| build/build-reuse.profile.yml | Define opt-in cache keys and input validation. |
| build/benchmarks-ci-azure.yml | Schedule Minimal Todo and update job dependencies. |
| build/benchmarks-ci-02.yml | Rebalance generated Gold benchmark scheduling. |
| build/benchmarks-ci-01.yml | Add Minimal Todo and rebalance generated scheduling. |
| build/benchmarks_ci_pods.json | Register Minimal Todo on Gold Linux. |
| build/benchmarks_ci_azure_pods.json | Register Minimal Todo on Azure ARM64 pods. |
| build/azure.profile.yml | Remove port overrides and update Cobalt core counts. |
Review effort: Balanced
Comment on lines
+12
to
+19
| onConfigure: | ||
| - 'if (!/^[0-9]+$/.test(String(job.Variables.buildCacheId))) { throw "buildCacheId must be set to the pipeline build ID."; }' | ||
| db: | ||
| buildKey: 'containers-build-{{ job.Source.DockerImageName }}-{{ buildCacheId }}' | ||
| source: | ||
| sourceKey: 'containers-source-{{ job.Source.DockerImageName }}-{{ buildCacheId }}' | ||
| options: | ||
| reuseBuild: true |
Comment on lines
+30
to
+38
| onConfigure: | ||
| - 'if (!/^[0-9]+$/.test(String(job.Variables.buildCacheId))) { throw "buildCacheId must be set to the pipeline build ID."; }' | ||
| - 'if (!job.Variables.serverBuildKey || !job.Variables.clientBuildKey || !/^[a-z0-9-]+$/.test(String(job.Variables.serverBuildKey)) || !/^[a-z0-9-]+$/.test(String(job.Variables.clientBuildKey))) { throw "serverBuildKey and clientBuildKey must identify the gRPC variants using lowercase letters, digits, or hyphens."; }' | ||
| load: | ||
| buildKey: 'grpc-client-build-{{ clientBuildKey }}-{{ job.Framework }}-{{ buildCacheId }}' | ||
| source: | ||
| sourceKey: 'grpc-client-source-{{ clientBuildKey }}-{{ buildCacheId }}' | ||
| options: | ||
| reuseBuild: true |
| --publish-results true ` | ||
| --external-log-uri $buildUrl | ||
| displayName: Runtime Windows X64 | ||
| displayName: Runtime Gold Windows X64 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Avoid rebuilding the same benchmark applications for every matrix entry. Fortunes FastHttp currently spends roughly 42 seconds rebuilding the application for each size/rate combination, despite the load phase being only 15 seconds of warmup plus 15 seconds of measurement.
Build.BuildId, so each pipeline run refreshes floating sources and builds once per compatible variant on each agent, then reuses those builds.The branch is based on current
aspnet/Benchmarksmain.Validation
Deployment considerations
No live benchmark timing run has been performed; actual wall-time savings still need to be measured after deployment. These profiles do not add a cache TTL, so retained builds rely on the agents' existing cleanup/restart policy.