Expose DRA device health in PodStatus#130606
Conversation
|
|
|
Hi @Jpsassine. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the The DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/ok-to-test |
|
@Jpsassine Thank you for your PR. Please sign the CLA to proceed further, thanks. BTW, is there any KEP or another design document describing/discussing these changes? If so, please provide links in the PR description. |
|
/easycla |
This change introduces the ability for the Kubelet to monitor and report the health of devices allocated via Dynamic Resource Allocation (DRA). This addresses a key part of KEP-4680 by providing visibility into device failures, which helps users and controllers diagnose pod failures. The implementation includes: - A new `v1alpha1.NodeHealth` gRPC service with a `WatchResources` stream that DRA plugins can optionally implement. - A health information cache within the Kubelet's DRA manager to track the last known health of each device and handle plugin disconnections. - An asynchronous update mechanism that triggers a pod sync when a device's health changes. - A new `allocatedResourcesStatus` field in `v1.ContainerStatus` to expose the device health information to users via the Pod API. Update vendor KEP-4680: Fix lint, boilerplate, and codegen issues Add another e2e test, add TODO for KEP4680 & update test infra helpers Add Feature Gate e2e test Fixing presubmits Fix var names, feature gating, and nits Fix DRA Health gRPC API according to review feedback
SergeyKanzhelev
left a comment
There was a problem hiding this comment.
/lgtm
/approve
chatted with @mrunalp we are ok on sig node.
|
LGTM label has been added. DetailsGit tree hash: 2028ed4293b0fbcd309f821750ae67c9a74ad62e |
|
@dims are you around to approve for |
|
@dchen1107 are you OK to approve for DRA? @pohly and @klueska are likely will not be around till code freeze |
|
/approve (for |
|
/hold for @dchen1107 or folks that @SergeyKanzhelev tagged |
|
@Jpsassine: The following tests failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/approve to unblock the progress But you need to fix the failing tests. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dchen1107, dims, Jpsassine, SergeyKanzhelev The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
thanks @dchen1107 /hold cancel |
|
/retest-required |
|
/test pull-kubernetes-e2e-kind-ipv6 |
|
/skip |
|
/milestone v1.34 |
There was a problem hiding this comment.
These tests are failing or flaking:
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/133214/pull-kubernetes-node-e2e-crio-cgrpv1-dra/1948697267229691904
There was a problem hiding this comment.
Yeah, these seem to be failing due to the ResourceHealthStatus feature not being enabled in the DRA e2e tests. Instead of enabling my plan is to disable the flag for DRA e2e tests to conserve resources and then add them to this pre-existing pull-kubernetes-e2e-node-resource-health-status presubmit job kubernetes/test-infra#35197
|
https://prow.k8s.io/pr-history/?org=kubernetes&repo=kubernetes&pr=130606 shows that the new e2e_node test didn't work. |
What type of PR is this?
/kind feature
/kind api-change
What this PR does / why we need it:
This PR implements the DRA portion of KEP-4680, introducing a mechanism for the Kubelet to monitor and report the health of devices allocated via Dynamic Resource Allocation (DRA). This provides cluster operators and controllers with crucial visibility into device health directly in the Pod's status, enabling better troubleshooting and remediation of pod failures caused by faulty hardware.
The core changes are:
New gRPC Health Service (
dra-health/v1alpha1):NodeHealthgRPC service with aWatchResourcesserver-streaming RPC.Healthy,Unhealthy,Unknown) to the Kubelet.Kubelet Integration:
DRAPluginManagernow detects plugins that implement theNodeHealthservice and starts a long-livedWatchResourcesstream to monitor them. It gracefully handles plugins that do not implement the service.pkg/kubelet/cm/dra) consumes these health updates and maintains a new, persistenthealthInfoCacheto track the last known state of each device. This cache handles device timeouts and survives Kubelet restarts.API Change (
v1.PodStatus):allocatedResourcesStatus, is added tov1.ContainerStatus.UpdateAllocatedResourcesStatusfunction populates this field with the current health of the container's allocated devices, read from the health cache.status_manager's equality check (isPodStatusByKubeletEqual) is updated to correctly detect changes within this new field, ensuring the pod status is patched to the API server.Testing:
[Feature:DynamicResourceAllocation] Resource Health) that verifies the end-to-end flow: a change in device health reported by a test driver is correctly reflected in thePodStatuson the API server.Which issue(s) this PR fixes:
Fixes #126243
Special notes for your reviewer:
This PR focuses on the DRA implementation path for KEP-4680. All changes are behind the
ResourceHealthStatusfeature gate.Does this PR introduce a user-facing change?
Yes. When the
ResourceHealthStatusfeature gate is enabled, a newallocatedResourcesStatusfield will appear in thecontainerStatusesof a Pod's status if it uses Dynamic Resource Allocation. This allows users to inspect the health of the allocated devices.Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: