Skip to content

Reuse benchmark builds within each CI pipeline run #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LoopedBard3 wants to merge 6 commits into from
Closed

Reuse benchmark builds within each CI pipeline run #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fceba7d
Update Azure Cobalt profiles to four cores (#2217)
LoopedBard3 Oct 14, 2026
d8e9611
Remove workload port overrides from Azure profiles (#2216)
LoopedBard3 Oct 14, 2026
16ab502
Add generated Minimal Todo container benchmarks (#2215)
LoopedBard3 Oct 14, 2026
4db6f23
Move PR benchmarks to Gold Linux and Windows machines (#2219)
LoopedBard3 Oct 19, 2026
9dd895b
Remove disconnected Citrine benchmark profiles (#2218)
LoopedBard3 Oct 19, 2026
cc79cfa
Reuse benchmark builds within each CI pipeline run
LoopedBard3 Oct 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 100 additions & 2 deletions build/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ JSON config that lists the available pods (machine groups) and scenarios.

| Generated YAML | Source config | Notes |
|---|---|---|
| `benchmarks-ci-01.yml`, `benchmarks-ci-02.yml` | `benchmarks_ci_pods.json` | On-prem fleet (citrine + mono); split across two YAMLs to spread load |
| `benchmarks-ci-01.yml`, `benchmarks-ci-02.yml` | `benchmarks_ci_pods.json` | On-prem fleet (Gold); split across two YAMLs to spread load |
| `benchmarks-ci-azure.yml` | `benchmarks_ci_azure_pods.json` | Azure cloud pods (includes the merged eastus2 cobalt cloud pods) |
| `benchmarks-ci-cobalt.yml` | `benchmarks_ci_cobalt_pods.json` | Cobalt-hosted pods |

Expand All @@ -36,6 +36,9 @@ python ./scripts/pod-scheduler/main.py \
The header of each generated YAML embeds the exact regen command for that
file.

The `citrine1`, `citrine2`, and `citrine3` names in the on-prem config are
controller queues, not retired benchmark machines or relay targets.

## Hand-maintained scenario templates

Everything else in this directory is hand-maintained. The most important
Expand All @@ -45,13 +48,40 @@ groups:
(e.g. `proxies-scenarios.yml`, `grpc-scenarios.yml`). One per benchmark
family. The pod-scheduler picks which template to run via the
`template` field on each scenario in the JSON config.
- `prbenchmarks.*.config.yml` — PR-trigger pipelines for individual PR runs.
- `prbenchmarks.yml` — scheduled PR-benchmark bot pipeline.
- `prbenchmarks.*.config.yml` — components, scenarios, and machine choices for
individual PR-benchmark requests.
- `job-variables.yml` — shared variable definitions consumed by the scenario
templates.

When adding a new benchmark family, drop a `*-scenarios.yml` template in this
directory and reference it from the relevant `benchmarks_ci_*_pods.json`.

## PR benchmarks

`prbenchmarks.yml` polls benchmark requests on `dotnet/aspnetcore`,
`dotnet/efcore`, and `dotnet/runtime` every 30 minutes. The controller image
determines where PR components are built, not which machine runs the benchmark.
The first profile in each bot config is the default machine selection.

| Config | Controller | Supported benchmark machines (default first) |
|---|---|---|
| `prbenchmarks.aspnetcore.config.yml` | Windows | `aspnet-gold-lin`, `aspnet-gold-win` (x64, 56 cores) |
| `prbenchmarks.efcore.config.yml` | Windows | `aspnet-gold-win` (x64, 56 cores) |
| `prbenchmarks.runtime.windows_x64.config.yml` | Windows | `aspnet-gold-win` (x64, 56 cores) |
| `prbenchmarks.runtime.linux_x64.config.yml` | Linux | `aspnet-gold-lin` (x64, 56 cores) |

These selections use the existing `aspnet-gold-*-relay` profiles imported by
the benchmark scenarios. Gold Linux uses `goldlin`, `goldload`, and `golddb`;
Gold Windows uses `goldwin`, `goldload2`, and `golddb`. The disconnected Perf
and Citrine choices are no longer offered. Runtime ARM64 jobs/configs have
been retired because their relay machines are unavailable; Gold is x64 and
does not replace ARM64 coverage.

The ASP.NET Core, EF Core, and runtime scenario/component definitions are
unchanged. This migration does not change the polling schedule, controller
images, or Crank SDK/build configuration.

## Editing the schedule

Common edits, all done in JSON:
Expand All @@ -69,3 +99,71 @@ After any edit, regenerate the YAML and commit both the JSON and the YAML
together. The snapshot tests in
`scripts/pod-scheduler/tests/test_snapshots.py` will fail in CI if the
generated YAML drifts from the configs.

## Build reuse within a pipeline run

`build-reuse.profile.yml` supplies opt-in Crank profiles for the high-repeat
container, gRPC, HttpClient, SignalR, and WebSocket matrices. Their scenario
templates load `$(buildReuseProfile)` and pass
`--variable buildCacheId=$(Build.BuildId)`. The first use of each build on an
agent checks out sources and builds; later compatible entries reuse it. A new
pipeline build ID changes both the source key and the build key, so floating
branches, .NET versions, and Docker base images are refreshed on the next run.
`--session` alone does not invalidate Crank caches. Source and build keys have
different prefixes because their metadata lives in the same cache directory.

| Suite | Builds reused across entries |
|---|---|
| Containers | Application and database images across 9 sizes and 3 request rates |
| gRPC | Separate ASP.NET Core/Go servers and .NET/Go/h2load clients across calls, protocols, and connection/stream counts |
| HttpClient | Client and Kestrel server across GET/POST, HTTP versions, concurrency, and payload sizes |
| SignalR | Server and client across scenarios, transports, and protocols; warmup and load share the client build |
| WebSockets | Client across payload sizes; separate compressed/uncompressed server builds |

These profiles reuse builds, not running processes or database contents.
Containers and applications are still started and stopped for each benchmark;
warmup and measurement durations are unchanged. Caches are local to each agent,
not shared between machines or architectures, and remain subject to the agent's
normal cache cleanup/lifetime. These profiles do not add a cache TTL; retained
builds need the same agent cleanup/restart policy as other Crank caches.

Container cache keys include the image name. Different Dockerfiles must have
different image names; the ASP.NET default, Alpine, Composite, and current
variants therefore have separate tags. gRPC scenario entries specify
`serverBuildKey` and `clientBuildKey` to separate implementations, and WebSocket
server keys include the compile-time compression mode. When adding a new
compile-time variant to these matrices, give it a distinct build identity
rather than reusing an existing key.

Other potential follow-ups are the proxy payload/protocol matrix and the
baseline/trend suites. They are not opted in here: proxies vary configuration
files copied into Docker images, while baseline/trend suites vary projects,
frameworks, and publishing settings that need their own cache identities.
The SslStream/QUIC matrix is currently disabled. Build-performance benchmarks
must continue building because build time is what they measure.

To check cache isolation without contacting agents or running benchmarks:

```bash
python -m unittest discover -s build/tests
```

The profile tests use the existing `crank --debug` configuration-only mode and
skip when the Crank CLI is not installed. Template wiring checks run without it.

## Minimal Todo

`minimal-todo-scenarios.yml` dispatches `scenarios/minimal-todo/minimal-todo.benchmarks.yml`
(1 CPU, 536,870,912 bytes / 512 MiB, 25200 req/s, enabled) for the `gold-lin`
and Azure `azure-arm64`/`cobalt-cloud-lin` pods. C# runs on every regular
scheduled slot using the direct (non-Docker) `csharp-project` build; Go and
Rust run the Docker `go`/`rust` builds but only on a shared weekly slot
(`condition: Math.round(Date.now() / 43200000) % 14 == 0`, combined with the
pod's own condition, matching the `containers-scenarios.yml` idiom). The
Docker `csharp`/`go`/`rust` builds (floating, multi-arch base images, no
pinned digests, no `requiredArchitecture` gating) remain the canonical
generation/update/admission/pre-promotion comparison path and are unaffected
-- see `scenarios/minimal-todo/README.md` for the full execution-path
mapping. Generation, admission, and calibration tooling for this cohort live
in the separate `dotnet-performance-tools`
repository; this directory only carries the scheduled dispatch template.
46 changes: 8 additions & 38 deletions build/azure.profile.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
profiles:
aspnet-azure-lin:
variables:
serverPort: 5000
serverAddress: 10.0.4.4
cores: 4
jobs:
Expand All @@ -28,7 +27,6 @@ profiles:

aspnet-azure-lin-relay:
variables:
serverPort: 5000
serverAddress: 10.0.4.4
cores: 4
jobs:
Expand All @@ -53,7 +51,6 @@ profiles:

idna-intel-lin:
variables:
serverPort: 5000
serverAddress: 10.0.4.12
cores: 4
jobs:
Expand All @@ -78,7 +75,6 @@ profiles:

idna-intel-lin-relay:
variables:
serverPort: 5000
serverAddress: 10.0.4.12
cores: 4
jobs:
Expand All @@ -103,7 +99,6 @@ profiles:

idna-amd-lin:
variables:
serverPort: 5000
serverAddress: 10.0.4.13
cores: 4
jobs:
Expand All @@ -128,7 +123,6 @@ profiles:

idna-amd-lin-relay:
variables:
serverPort: 5000
serverAddress: 10.0.4.13
cores: 4
jobs:
Expand All @@ -153,7 +147,6 @@ profiles:

idna-intel-win:
variables:
serverPort: 5000
serverAddress: 10.0.4.14
cores: 4
jobs:
Expand All @@ -178,7 +171,6 @@ profiles:

idna-intel-win-relay:
variables:
serverPort: 5000
serverAddress: 10.0.4.14
cores: 4
jobs:
Expand All @@ -203,7 +195,6 @@ profiles:

idna-amd-win:
variables:
serverPort: 5000
serverAddress: 10.0.4.15
cores: 4
jobs:
Expand All @@ -228,7 +219,6 @@ profiles:

idna-amd-win-relay:
variables:
serverPort: 5000
serverAddress: 10.0.4.15
cores: 4
jobs:
Expand All @@ -253,7 +243,6 @@ profiles:

aspnet-azurearm64-lin:
variables:
serverPort: 5000
serverAddress: 10.0.4.7
cores: 4
jobs:
Expand All @@ -278,7 +267,6 @@ profiles:

aspnet-azurearm64-lin-relay:
variables:
serverPort: 5000
serverAddress: 10.0.4.7
cores: 4
jobs:
Expand All @@ -303,9 +291,8 @@ profiles:

cobalt-cloud-lin:
variables:
serverPort: 5000
serverAddress: 10.0.4.17
cores: 16
cores: 4
jobs:
db:
endpoints:
Expand All @@ -328,9 +315,8 @@ profiles:

cobalt-cloud-lin-relay:
variables:
serverPort: 5000
serverAddress: 10.0.4.17
cores: 16
cores: 4
jobs:
db:
endpoints:
Expand All @@ -353,9 +339,8 @@ profiles:

cobalt-cloud-lin-al3:
variables:
serverPort: 5000
serverAddress: 10.0.4.20
cores: 16
cores: 4
jobs:
db:
endpoints:
Expand All @@ -378,9 +363,8 @@ profiles:

cobalt-cloud-lin-al3-relay:
variables:
serverPort: 5000
serverAddress: 10.0.4.20
cores: 16
cores: 4
jobs:
db:
endpoints:
Expand All @@ -404,7 +388,6 @@ profiles:
# Add individual machine profiles for azure-db similar to ci.profile.yml
azure-db-app: # azurearm64db
variables:
serverPort: 5000
serverAddress: 10.0.4.9
cores: 4
agents:
Expand Down Expand Up @@ -440,7 +423,6 @@ profiles:
# Individual machine profiles for azure-client
azure-client-app: # azurearm64client
variables:
serverPort: 5000
serverAddress: 10.0.4.8
cores: 4
agents:
Expand Down Expand Up @@ -476,7 +458,6 @@ profiles:
# Individual machine profiles for azure-server-arm64
azure-server-arm64-app: # azurearm64server
variables:
serverPort: 5000
serverAddress: 10.0.4.7
cores: 4
agents:
Expand Down Expand Up @@ -512,7 +493,6 @@ profiles:
# Individual machine profiles for azure2-client
azure2-client-app:
variables:
serverPort: 5000
serverAddress: 10.0.4.5
cores: 2
agents:
Expand Down Expand Up @@ -548,7 +528,6 @@ profiles:
# Individual machine profiles for azure2-db
azure2-db-app:
variables:
serverPort: 5000
serverAddress: 10.0.4.6
cores: 2
agents:
Expand Down Expand Up @@ -584,7 +563,6 @@ profiles:
# Individual machine profiles for azure2-server-amd64
azure2-server-amd64-app:
variables:
serverPort: 5000
serverAddress: 10.0.4.4
cores: 4
agents:
Expand Down Expand Up @@ -620,7 +598,6 @@ profiles:
# Individual machine profiles for idna-amd-lin
idna-amd-lin-app:
variables:
serverPort: 5000
serverAddress: 10.0.4.13
cores: 4
agents:
Expand Down Expand Up @@ -656,7 +633,6 @@ profiles:
# Individual machine profiles for idna-amd-win
idna-amd-win-app:
variables:
serverPort: 5000
serverAddress: 10.0.4.15
cores: 4
agents:
Expand Down Expand Up @@ -692,7 +668,6 @@ profiles:
# Individual machine profiles for idna-intel-lin
idna-intel-lin-app:
variables:
serverPort: 5000
serverAddress: 10.0.4.12
cores: 4
agents:
Expand Down Expand Up @@ -728,7 +703,6 @@ profiles:
# Individual machine profiles for idna-intel-win
idna-intel-win-app:
variables:
serverPort: 5000
serverAddress: 10.0.4.14
cores: 4
agents:
Expand Down Expand Up @@ -764,9 +738,8 @@ profiles:
# Individual machine profiles for cobalt-cloud-lin-client
cobalt-cloud-lin-client-app:
variables:
serverPort: 5000
serverAddress: 10.0.4.18
cores: 16
cores: 4
agents:
application:
endpoints:
Expand Down Expand Up @@ -800,9 +773,8 @@ profiles:
# Individual machine profiles for cobalt-cloud-lin-db
cobalt-cloud-lin-db-app:
variables:
serverPort: 5000
serverAddress: 10.0.4.19
cores: 16
cores: 4
agents:
application:
endpoints:
Expand Down Expand Up @@ -836,9 +808,8 @@ profiles:
# Individual machine profiles for cobalt-cloud-lin-server
cobalt-cloud-lin-server-app:
variables:
serverPort: 5000
serverAddress: 10.0.4.17
cores: 16
cores: 4
agents:
application:
endpoints:
Expand Down Expand Up @@ -872,9 +843,8 @@ profiles:
# Individual machine profiles for cobalt-cloud-lin-server-azure-linux3
cobalt-cloud-lin-server-azure-linux3-app:
variables:
serverPort: 5000
serverAddress: 10.0.4.20
cores: 16
cores: 4
agents:
application:
endpoints:
Expand Down
Loading
Loading