Skip to content

Expose DRA device health in PodStatus#130606

Merged
k8s-ci-robot merged 1 commit into
kubernetes:masterfrom
Jpsassine:dra_device_health_status
Jul 25, 2025
Merged

Expose DRA device health in PodStatus#130606
k8s-ci-robot merged 1 commit into
kubernetes:masterfrom
Jpsassine:dra_device_health_status

Conversation

@Jpsassine

@Jpsassine Jpsassine commented Mar 6, 2025

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

This PR implements the DRA portion of KEP-4680, introducing a mechanism for the Kubelet to monitor and report the health of devices allocated via Dynamic Resource Allocation (DRA). This provides cluster operators and controllers with crucial visibility into device health directly in the Pod's status, enabling better troubleshooting and remediation of pod failures caused by faulty hardware.

The core changes are:

  1. New gRPC Health Service (dra-health/v1alpha1):

    • Defines a new NodeHealth gRPC service with a WatchResources server-streaming RPC.
    • DRA plugins can optionally implement this service to stream device health status updates (Healthy, Unhealthy, Unknown) to the Kubelet.
  2. Kubelet Integration:

    • The DRAPluginManager now detects plugins that implement the NodeHealth service and starts a long-lived WatchResources stream to monitor them. It gracefully handles plugins that do not implement the service.
    • The DRA Manager (pkg/kubelet/cm/dra) consumes these health updates and maintains a new, persistent healthInfoCache to track the last known state of each device. This cache handles device timeouts and survives Kubelet restarts.
    • When a device's health changes, the DRA Manager identifies all affected pods and triggers an asynchronous pod sync via the Container Manager.
  3. API Change (v1.PodStatus):

    • A new field, allocatedResourcesStatus, is added to v1.ContainerStatus.
    • During a pod sync, the DRA Manager's UpdateAllocatedResourcesStatus function populates this field with the current health of the container's allocated devices, read from the health cache.
    • The status_manager's equality check (isPodStatusByKubeletEqual) is updated to correctly detect changes within this new field, ensuring the pod status is patched to the API server.
  4. Testing:

    • Adds comprehensive unit tests for the new health caching and processing logic.
    • Adds a new node e2e test ([Feature:DynamicResourceAllocation] Resource Health) that verifies the end-to-end flow: a change in device health reported by a test driver is correctly reflected in the PodStatus on the API server.
    • Fixes discovered in existing e2e tests during development are included to ensure the suite remains stable.

Which issue(s) this PR fixes:

Fixes #126243

Special notes for your reviewer:

This PR focuses on the DRA implementation path for KEP-4680. All changes are behind the ResourceHealthStatus feature gate.

Does this PR introduce a user-facing change?

Yes. When the ResourceHealthStatus feature gate is enabled, a new allocatedResourcesStatus field will appear in the containerStatuses of a Pod's status if it uses Dynamic Resource Allocation. This allows users to inspect the health of the allocated devices.

The Kubelet can now monitor the health of devices allocated via Dynamic Resource Allocation (DRA) and report it in the `pod.status.containerStatuses.allocatedResourcesStatus` field. This requires the DRA plugin to implement the new v1alpha1 `NodeHealth` gRPC service. This feature is controlled by the `ResourceHealthStatus` feature gate.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/4680-add-resource-health-to-pod-status/README.md


@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Mar 6, 2025
@linux-foundation-easycla

linux-foundation-easycla Bot commented Mar 6, 2025

Copy link
Copy Markdown

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: Jpsassine / name: JP (b7de71f)

@k8s-ci-robot k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Mar 6, 2025
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

Hi @Jpsassine. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot

Copy link
Copy Markdown
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Mar 6, 2025
@k8s-ci-robot k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. wg/device-management Categorizes an issue or PR as relevant to WG Device Management. labels Mar 6, 2025
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Mar 6, 2025
@googs1025

Copy link
Copy Markdown
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 6, 2025
@bart0sh

bart0sh commented Mar 10, 2025

Copy link
Copy Markdown
Contributor

@Jpsassine Thank you for your PR. Please sign the CLA to proceed further, thanks.

BTW, is there any KEP or another design document describing/discussing these changes? If so, please provide links in the PR description.

@bart0sh bart0sh moved this from Triage to Waiting on Author in SIG Node: code and documentation PRs Mar 10, 2025
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Mar 11, 2025
@Jpsassine

Copy link
Copy Markdown
Contributor Author

/easycla

Comment thread pkg/kubelet/cm/dra/healthinfo.go Outdated
Comment thread pkg/kubelet/cm/dra/healthinfo.go Outdated
This change introduces the ability for the Kubelet to monitor and report
the health of devices allocated via Dynamic Resource Allocation (DRA).
This addresses a key part of KEP-4680 by providing visibility into
device failures, which helps users and controllers diagnose pod failures.

The implementation includes:
- A new `v1alpha1.NodeHealth` gRPC service with a `WatchResources`
  stream that DRA plugins can optionally implement.
- A health information cache within the Kubelet's DRA manager to track
  the last known health of each device and handle plugin disconnections.
- An asynchronous update mechanism that triggers a pod sync when a
  device's health changes.
- A new `allocatedResourcesStatus` field in `v1.ContainerStatus` to
  expose the device health information to users via the Pod API.

Update vendor

KEP-4680: Fix lint, boilerplate, and codegen issues

Add another e2e test, add TODO for KEP4680 & update test infra helpers

Add Feature Gate e2e test

Fixing presubmits

Fix var names, feature gating, and nits

Fix DRA Health gRPC API according to review feedback

@SergeyKanzhelev SergeyKanzhelev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

chatted with @mrunalp we are ok on sig node.

@k8s-ci-robot

Copy link
Copy Markdown
Contributor

LGTM label has been added.

DetailsGit tree hash: 2028ed4293b0fbcd309f821750ae67c9a74ad62e

@SergeyKanzhelev

Copy link
Copy Markdown
Member

@dims are you around to approve for hack/?

@SergeyKanzhelev

Copy link
Copy Markdown
Member

@dchen1107 are you OK to approve for DRA? @pohly and @klueska are likely will not be around till code freeze

@dims

dims commented Jul 25, 2025

Copy link
Copy Markdown
Member

/approve

(for /hack)

@dims

dims commented Jul 25, 2025

Copy link
Copy Markdown
Member

/hold for @dchen1107 or folks that @SergeyKanzhelev tagged

@k8s-ci-robot

k8s-ci-robot commented Jul 25, 2025

Copy link
Copy Markdown
Contributor

@Jpsassine: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kubernetes-unit-windows-master b7de71f link false /test pull-kubernetes-unit-windows-master
pull-kubernetes-node-e2e-crio-cgrpv1-dra b7de71f link false /test pull-kubernetes-node-e2e-crio-cgrpv1-dra

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@dchen1107

dchen1107 commented Jul 25, 2025

Copy link
Copy Markdown
Member

/approve to unblock the progress

But you need to fix the failing tests.

@k8s-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dchen1107, dims, Jpsassine, SergeyKanzhelev

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@dims

dims commented Jul 25, 2025

Copy link
Copy Markdown
Member

thanks @dchen1107

/hold cancel

@SergeyKanzhelev

Copy link
Copy Markdown
Member

/retest-required

@pacoxu

pacoxu commented Jul 25, 2025

Copy link
Copy Markdown
Member

/test pull-kubernetes-e2e-kind-ipv6

@dims

dims commented Jul 25, 2025

Copy link
Copy Markdown
Member

/skip

@pacoxu

pacoxu commented Jul 25, 2025

Copy link
Copy Markdown
Member

/milestone v1.34
since this was approved before code freeze.

Comment thread test/e2e_node/dra_test.go

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Jpsassine Jpsassine Jul 25, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, these seem to be failing due to the ResourceHealthStatus feature not being enabled in the DRA e2e tests. Instead of enabling my plan is to disable the flag for DRA e2e tests to conserve resources and then add them to this pre-existing pull-kubernetes-e2e-node-resource-health-status presubmit job kubernetes/test-infra#35197

@pohly

pohly commented Jul 25, 2025

Copy link
Copy Markdown
Contributor

https://prow.k8s.io/pr-history/?org=kubernetes&repo=kubernetes&pr=130606 shows that the new e2e_node test didn't work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api-review Categorizes an issue or PR as actively needing an API review. approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver area/cloudprovider area/code-generation area/dependency Issues or PRs related to dependency changes area/kube-proxy area/kubectl area/kubelet area/release-eng Issues or PRs related to the Release Engineering subproject area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/cli Categorizes an issue or PR as relevant to SIG CLI. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/etcd Categorizes an issue or PR as relevant to SIG Etcd. sig/network Categorizes an issue or PR as relevant to SIG Network. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/release Categorizes an issue or PR as relevant to SIG Release. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. wg/device-management Categorizes an issue or PR as relevant to WG Device Management.

Projects

Status: API review completed, 1.34
Archived in project
Archived in project
Archived in project
Archived in project
Archived in project

Development

Successfully merging this pull request may close these issues.