feat(ksm): grant DRA RBAC for Cluster Agent 7.85.0+; mount CDI dir for GPU monitoring by zhuminyi · Pull Request #3703 · DataDog/datadog-operator

zhuminyi · 2026-10-30T19:33:06Z

What does this PR do?

When the component running kubernetes_state_core (Cluster Agent, or Cluster Checks Runner when KSM runs there) is 7.85.0+, adds a resource.k8s.io list/watch rule for resourceclaims, resourceslices and devicetaintrules to the KSM ClusterRole. The operator's own ClusterRole gets the same permissions so it can grant them.
With GPU monitoring enabled, mounts host /var/run/cdi read-only at /host/var/run/cdi in the core agent container. The mount is removed on GKE Autopilot.
Documents in docs/kubernetes_state_metrics.md how to enable DRA collection through conf.configData.

Motivation

Cluster Agent 7.85.0 collects Kubernetes DRA metrics when users add the resourceclaims, resourceslices and devicetaintrules collectors to a custom KSM config. Users can add the collectors themselves but can't grant the RBAC. The CDI specs let the node agent attribute GPUs to pods that got them through DRA.

Example

examples/datadogagent/datadog-agent-with-dra.yaml shows a working setup: Cluster Agent and Agent image overrides to 7.85.0, features.gpu.enabled: true, and kubeStateMetricsCore.conf.configData with the full default collector list plus resourceclaims, resourceslices, devicetaintrules (and an optional dra_device_classes filter).

Additional Notes

The operator does not add the collectors and there is no new CRD field.
The version check fails closed (unparseable tags such as :dev get no rule): a Cluster Agent older than 7.85 fails the whole kubernetes_state_core check on DRA clusters when given these collector names.
With no image override, the check uses the bundled Cluster Agent version (currently 7.84.1), so the rule is only added once the bundle moves to 7.85+, or when users set a 7.85+ image.

Minimum Agent Versions

Agent: none (older agents ignore the CDI mount)
Cluster Agent: v7.85.0 (Cluster Checks Runner v7.85.0 when KSM runs there)

Describe your test plan

Unit tests cover the RBAC version check (Cluster Agent and Cluster Checks Runner, rc and :dev tags, no override, custom conf) and the CDI mount including the Autopilot removal. make gotest, make golden-test and make lint pass; make generate && make manifests leaves no diff.

To check on a cluster: set the Cluster Agent image to 7.85.0+, add the three collectors through features.kubeStateMetricsCore.conf.configData, and confirm kubernetes_state.resourceslice.devices reports with no Forbidden errors in the Cluster Agent logs.

Checklist

PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
PR has a milestone or the qa/skip-qa label
All commits are signed (see: signing commits)

datadog-prod-us1-6 · 2026-10-30T19:44:17Z

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 57.26% (+0.03%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: ae6206c | Docs | View more details | Give us feedback!}

Copilot

🟡 Changes recommended

The version check uses incorrect image override precedence, which can omit required DRA permissions for the image actually running and cause collectors to fail with Forbidden errors.

2 open issues

What changed in this PR

Adds support for Kubernetes Dynamic Resource Allocation (DRA) monitoring through version-gated KSM permissions and access to host CDI specifications for GPU attribution.

Changes:

Grants DRA read permissions when the component running KSM is version 7.85.0 or later.
Mounts the CDI directory read-only for GPU monitoring, excluding GKE Autopilot.
Adds tests, documentation, and an example for enabling DRA collectors.

File	Description
`pkg/kubernetes/rbac/const.go`	Defines DRA API group and resource constants.
`internal/controller/testutils/renderer/testdata/golden/suppression-baseline.golden.yaml`	Updates expected output with the CDI mount.
`internal/controller/datadogagent/feature/kubernetesstatecore/rbac.go`	Adds conditional DRA permissions.
`internal/controller/datadogagent/feature/kubernetesstatecore/rbac_test.go`	Tests DRA permission generation.
`internal/controller/datadogagent/feature/kubernetesstatecore/feature.go`	Gates DRA RBAC on the executing component's version.
`internal/controller/datadogagent/feature/kubernetesstatecore/feature_test.go`	Tests version gating for Cluster Agent and runners.
`internal/controller/datadogagent/feature/kubernetesstatecore/const.go`	Defines the minimum DRA collector version.
`internal/controller/datadogagent/feature/gpu/feature.go`	Adds the CDI mount and Autopilot exclusion.
`internal/controller/datadogagent/feature/gpu/feature_test.go`	Tests CDI volumes and provider-specific removal.
`internal/controller/datadogagent/feature/gpu/const.go`	Defines CDI volume paths.
`internal/controller/datadogagent_controller.go`	Declares operator DRA permissions for generation.
`examples/datadogagent/datadog-agent-with-dra.yaml`	Provides a DRA and GPU monitoring example.
`docs/kubernetes_state_metrics.md`	Documents DRA collectors and version requirements.
`config/rbac/role.yaml`	Grants the operator DRA list/watch permissions.

_{Review effort: Balanced}

chatgpt-codex-connector · 2026-10-30T20:17:26Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-30T20:20:00.113809Z	`6299e58`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6299e5873a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…r GPU monitoring

…aintRule requirement

Main now defaults to Agent 7.85.0 (#3708), so the DRA RBAC gated on >= 7.85.0 renders in the comprehensive fixtures. Regenerated with go test ./internal/controller/testutils/renderer/... -update.

Copilot

🟡 Changes recommended

Image overrides that only set pull policy or secrets incorrectly produce an empty component version, withholding DRA permissions from supported Cluster Agents or runners and causing configured collectors to receive Forbidden errors.

1 open issue

Resolve inherited image versions for pull-only overrides · New

2 resolved since last review

_{Review effort: Balanced}

…the DRA gate An image override that sets no tag (only pullPolicy or pullSecrets, or a name without a tag) keeps the bundled image tag when rendered, but GetComponentVersion returned "" or "latest" for it, so the fail-closed DRA gate withheld the RBAC from a supported Cluster Agent or Cluster Checks Runner. Resolve the component's bundled version for those overrides; explicit unparseable tags still fail closed.

Copilot

🟢 Approval recommended

No active review issues are reported for the DRA metrics and GPU-to-pod attribution changes.

No open issues 🎉

1 resolved since last review

Resolve inherited image versions for pull-only overrides

_{Review effort: Balanced}

zhuminyi added the enhancement New feature or request label Oct 30, 2026

github-actions Bot added team/container-platform team/container-integrations team/ebpf-platform labels Oct 30, 2026

zhuminyi added this to the v1.33.0 milestone Oct 30, 2026

zhuminyi requested a balanced review from Copilot October 30, 2026 19:47

Copilot started reviewing on behalf of zhuminyi October 30, 2026 19:48

View session

Copilot AI reviewed Oct 30, 2026

View reviewed changes

Comment thread internal/controller/datadogagent/feature/kubernetesstatecore/feature.go Resolved

Comment thread docs/kubernetes_state_metrics.md Resolved

zhuminyi marked this pull request as ready for review October 30, 2026 20:17

zhuminyi requested a review from a team October 30, 2026 20:17

zhuminyi requested review from a team as code owners October 30, 2026 20:17

chatgpt-codex-connector Bot reviewed Oct 30, 2026

View reviewed changes

Comment thread internal/controller/datadogagent/feature/kubernetesstatecore/feature.go Resolved

tbavelier approved these changes Nov 2, 2026

View reviewed changes

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: queued mergequeue-status: in_progress mergequeue-status: rejected and removed mergequeue-status: queued mergequeue-status: in_progress labels Nov 2, 2026

zhuminyi added 5 commits November 2, 2026 13:28

feat(ksm): grant DRA RBAC for Cluster Agent 7.85.0+; mount CDI dir fo…

9eef841

…r GPU monitoring

docs(ksm): add DRA example DatadogAgent

3c22625

docs(ksm): run DRA docs example on the Cluster Agent; clarify DeviceT…

51ef13b

…aintRule requirement

docs(ksm): show only the DRA collectors in the DRA example

41b8f3f

test(renderer): regenerate golden manifests for the DRA RBAC

1b99adc

Main now defaults to Agent 7.85.0 (#3708), so the DRA RBAC gated on >= 7.85.0 renders in the comprehensive fixtures. Regenerated with go test ./internal/controller/testutils/renderer/... -update.

zhuminyi force-pushed the minyi/dra-collect-metrics branch from 6299e58 to 1b99adc Compare November 2, 2026 18:31

zhuminyi requested a balanced review from Copilot November 2, 2026 18:58

Copilot started reviewing on behalf of zhuminyi November 2, 2026 18:58

View session

Copilot AI reviewed Nov 2, 2026

View reviewed changes

Comment thread internal/controller/datadogagent/feature/kubernetesstatecore/feature.go Resolved

zhuminyi requested a balanced review from Copilot November 2, 2026 19:49

Copilot started reviewing on behalf of zhuminyi November 2, 2026 19:49

View session

Copilot AI reviewed Nov 2, 2026

View reviewed changes

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: waiting mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: rejected mergequeue-status: waiting mergequeue-status: queued labels Nov 2, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit bcb5057 into main Nov 2, 2026
62 of 64 checks passed

gh-worker-dd-devflow-36fce6 Bot removed the mergequeue-status: in_progress label Nov 2, 2026

gh-worker-dd-mergequeue-cf854d Bot deleted the minyi/dra-collect-metrics branch November 2, 2026 21:19

gh-worker-dd-devflow-36fce6 Bot added the mergequeue-status: done label Nov 2, 2026

Navigation Menu

Repository navigation

feat(ksm): grant DRA RBAC for Cluster Agent 7.85.0+; mount CDI dir for GPU monitoring - #3703

feat(ksm): grant DRA RBAC for Cluster Agent 7.85.0+; mount CDI dir for GPU monitoring#3703
gh-worker-dd-mergequeue-cf854d[bot] merged 6 commits into
mainfrom
minyi/dra-collect-metrics

zhuminyi commented Oct 30, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-6 Bot commented Oct 30, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 30, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

zhuminyi commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Example

Additional Notes

Minimum Agent Versions

Describe your test plan

Checklist

Uh oh!

datadog-prod-us1-6 Bot commented Oct 30, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zhuminyi commented Oct 30, 2026 •

edited

Loading

datadog-prod-us1-6 Bot commented Oct 30, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

chatgpt-codex-connector Bot commented Oct 30, 2026 •

edited

Loading