chore: upgrade Kubernetes deps and local images to 1.35 (incl. controller-runtime v0.23 and KAI-scheduler v0.15) by yankay · Pull Request #603 · ai-dynamo/grove

yankay · 2026-05-12T02:59:40Z

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Upgrades Grove's Kubernetes dependency baseline from 1.34 to 1.35.

k8s.io/*: v0.34.3 -> v0.35.5
sigs.k8s.io/controller-runtime: v0.22.4 -> v0.23.3
github.com/kai-scheduler/KAI-scheduler: v0.14.0 -> v0.15.2 (required for controller-runtime v0.23)
kindest/node: v1.34.3 -> v1.35.1
rancher/k3s: v1.34.2-k3s1 -> v1.35.5-k3s1

The installed KAI-scheduler version in docs and e2e tooling (docs/installation.md, operator/e2e/README.md, operator/hack/{e2e-cluster,infra_manager}/dependencies.yaml, operator/hack/e2e-cluster/create-e2e-cluster.py) is also bumped to v0.15.2.

KAI v0.15 changes PodGroupSpec.MinMember / SubGroup.MinMember from int32 to *int32; only the e2e SubGroup verifier needed a deref. Generated code refreshed via make generate and make generate-api-docs.

Two incidental changes that came along with the upgrade:

operator/go.mod: Go toolchain bumped 1.26.1 -> 1.26.3, and the now-unused operator/client require plus the k8s.io/kubelet replace directive were dropped by go mod tidy.
operator/e2e/setup/k8s_clusters.go: raised client-side rate limits (QPS 50 / Burst 100) above the client-go defaults (5/10); e2e polling loops were otherwise hitting client rate limiter Wait ... context deadline exceeded under the rolling/ondelete update tests on 1.35.

Which issue(s) this PR fixes:

Fixes #602

Special notes for your reviewer:

Local: go build ./..., go build -tags e2e ./e2e/..., go test ./..., go vet ./... all pass.

Related design PR: #605

Does this PR introduce a API change?

The following dependencies are updated:
- `k8s.io/*`: `v0.34.3` -> `v0.35.5`
- `sigs.k8s.io/controller-runtime`: `v0.22.4` -> `v0.23.3`
- `github.com/kai-scheduler/KAI-scheduler`: `v0.14.0` -> `v0.15.2`
- `kindest/node`: `v1.34.3` -> `v1.35.1`
- `rancher/k3s`: `v1.34.2-k3s1` -> `v1.35.5-k3s1`

Additional documentation e.g., enhancement proposals, usage docs, etc.:

NONE

copy-pr-bot · 2026-05-12T02:59:44Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

yankay · 2026-05-12T03:22:44Z

CI status: blocked on KAI-Scheduler upstream

All 7 failing E2E jobs share the same root cause, and it is not the kind/k3d 1.35 image bump itself. The k3d cluster (rancher/k3s:v1.35.4-k3s1) comes up fine; failure happens later when make run-e2e tries to compile the Go e2e test binary:

# github.com/kai-scheduler/KAI-scheduler/pkg/apis/scheduling/v2alpha2
.../kai-scheduler@v0.14.0/pkg/apis/scheduling/v2alpha2/podgroup_webhook.go:18:34:
  not enough arguments in call to ctrl.NewWebhookManagedBy
    have (controllerruntime.Manager)
    want (manager.Manager, T)
FAIL  github.com/ai-dynamo/grove/operator/e2e/tests [build failed]
make[1]: *** [Makefile:126: run-e2e] Error 1

controller-runtime v0.23 made ctrl.NewWebhookManagedBy generic (mgr, obj). operator/e2e/tests transitively imports github.com/kai-scheduler/KAI-scheduler v0.14.0, which still uses the v0.22 signature, so the e2e binary no longer compiles once we bump controller-runtime.

KAI-Scheduler side

The fix is already merged on main of kai-scheduler/KAI-Scheduler:

Tracking issue: #1466 Bump the k8s.io/* module group from v0.34.x to v0.35.x (closed)
PR: #1472 chore(deps): bump k8s.io/* module group from v0.34.x to v0.35.4 (merged 2026-04-26)

But no released tag contains it yet — the latest tags v0.14.0 / v0.14.1 / v0.14.2 and the v0.14 release branch are all still on controller-runtime v0.22.3. The change is sitting in CHANGELOG.md under [Unreleased].

Proposed path

Two options, in order of preference:

Wait for the next KAI-Scheduler minor release (presumably v0.15.0) and bump to that here. There is no public ETA yet — happy to ping upstream to ask.
Temporarily pin to kai-scheduler/KAI-Scheduler main via go get github.com/kai-scheduler/KAI-scheduler@1b591f419a01 so this PR is not held up, with a follow-up to swap back to a proper semver once v0.15.0 is cut.

Marking the PR as pending upstream while we decide. Suggestions welcome.

danbar2 · 2026-05-12T07:29:59Z

CI status: blocked on KAI-Scheduler upstream

All 7 failing E2E jobs share the same root cause, and it is not the kind/k3d 1.35 image bump itself. The k3d cluster (rancher/k3s:v1.35.4-k3s1) comes up fine; failure happens later when make run-e2e tries to compile the Go e2e test binary:
# github.com/kai-scheduler/KAI-scheduler/pkg/apis/scheduling/v2alpha2
.../kai-scheduler@v0.14.0/pkg/apis/scheduling/v2alpha2/podgroup_webhook.go:18:34:
  not enough arguments in call to ctrl.NewWebhookManagedBy
    have (controllerruntime.Manager)
    want (manager.Manager, T)
FAIL  github.com/ai-dynamo/grove/operator/e2e/tests [build failed]
make[1]: *** [Makefile:126: run-e2e] Error 1
controller-runtime v0.23 made ctrl.NewWebhookManagedBy generic (mgr, obj). operator/e2e/tests transitively imports github.com/kai-scheduler/KAI-scheduler v0.14.0, which still uses the v0.22 signature, so the e2e binary no longer compiles once we bump controller-runtime.

KAI-Scheduler side

The fix is already merged on main of kai-scheduler/KAI-Scheduler:

Tracking issue: #1466 Bump the k8s.io/* module group from v0.34.x to v0.35.x (closed)

PR: #1472 chore(deps): bump k8s.io/* module group from v0.34.x to v0.35.4 (merged 2026-04-26)

But no released tag contains it yet — the latest tags v0.14.0 / v0.14.1 / v0.14.2 and the v0.14 release branch are all still on controller-runtime v0.22.3. The change is sitting in CHANGELOG.md under [Unreleased].

Proposed path

Two options, in order of preference:

Wait for the next KAI-Scheduler minor release (presumably v0.15.0) and bump to that here. There is no public ETA yet — happy to ping upstream to ask.

Temporarily pin to kai-scheduler/KAI-Scheduler main via go get github.com/kai-scheduler/KAI-scheduler@1b591f419a01 so this PR is not held up, with a follow-up to swap back to a proper semver once v0.15.0 is cut.

Marking the PR as pending upstream while we decide. Suggestions welcome.

Thanks for raising this PR !
Option 1 is the correct one, we should wait for KAI to upgrade as well.
cc: @enoodle

renormalize

https://github.com/kai-scheduler/KAI-Scheduler/releases/tag/v0.15.0 is out. Can the PR be rebased with the kai-scheduler bump? Thanks.

yankay · 2026-05-25T12:47:49Z

https://github.com/kai-scheduler/KAI-Scheduler/releases/tag/v0.15.0 is out. Can the PR be rebased with the kai-scheduler bump? Thanks.

HI @renormalize

Done. Rebased on main and bumped KAI-scheduler v0.14.0 → v0.15.0

renormalize · 2026-05-26T09:59:41Z

@yankay I took the liberty to sync the dependencies across the multiple Go modules we have in this repository, as they were not synced up after your rebased commit.

We can merge once the E2E tests pass.

yankay · 2026-05-26T10:06:44Z

@yankay I took the liberty to sync the dependencies across the multiple Go modules we have in this repository, as they were not synced up after your rebased commit.

We can merge once the E2E tests pass.

Thanks @renormalize for the cross-module dep sync :-)

enoodle · 2026-05-27T10:19:52Z

@yankay Do you want to also update the installed KAI version in the docs / e2e scripts to v0.15.0 ?

yankay · 2026-05-27T10:50:17Z

Thanks @enoodle! Good catch — bumped the installed KAI version to v0.15.0 in docs and e2e tooling PTAL 🙏

Waiting for the fix to be released in KAI v0.5.1 or v0.6.0, then we can re-run and confirm this PR.

renormalize · 2026-06-08T05:17:12Z

@danbar2 any idea on when KAI plans to release v0.15.1 or v0.16.0?

enoodle · 2026-06-08T08:13:27Z

@renormalize I will work on including this fix in v0.15.1 fix of KAI.
On the grove side we can advance #524 to disengage from KAI's logic to create the PodGroups.

enoodle · 2026-06-11T06:59:31Z

@renormalize @yankay KAI v0.15.2 contains the fix you referenced. Can you update to use that?

Update Grove's Kubernetes dependency baseline from 1.34 to 1.35. - Bump k8s.io/* to v0.35.5 and controller-runtime to v0.23.3. - Bump KAI Scheduler to v0.15.2 for controller-runtime v0.23 compatibility. - Bump local kind and k3s images to the 1.35 line. - Raise the operator module Go directive to 1.26.3 for the updated KAI Scheduler dependency. - Refresh generated clients, CRDs, docs, and e2e install pins. Co-authored-by: Saketh Kalaga <saketh.kalaga@sap.com> Signed-off-by: Kay Yan <kay.yan@daocloud.io>

yankay requested review from Ronkahn21, danbar2, gflarity, sanjaychatterjee, shayasoolin and unmarshall as code owners May 12, 2026 02:59

renormalize requested changes May 25, 2026

View reviewed changes

yankay force-pushed the chore/602-upgrade-k8s-1.35 branch 2 times, most recently from b9bdefd to b72fd55 Compare May 25, 2026 12:45

yankay changed the title ~~chore: upgrade Kubernetes dependencies and local images to 1.35~~ chore: upgrade Kubernetes deps and local images to 1.35 (incl. controller-runtime v0.23 and KAI-scheduler v0.15) May 25, 2026

yankay force-pushed the chore/602-upgrade-k8s-1.35 branch from b72fd55 to 8b64e6a Compare May 26, 2026 06:57

renormalize force-pushed the chore/602-upgrade-k8s-1.35 branch from 9d9e1af to 59c8a49 Compare May 26, 2026 09:54

renormalize previously approved these changes May 26, 2026

View reviewed changes

yankay dismissed renormalize’s stale review via 3e72a55 May 27, 2026 10:48

yankay force-pushed the chore/602-upgrade-k8s-1.35 branch from 59c8a49 to 3e72a55 Compare May 27, 2026 10:48

yankay force-pushed the chore/602-upgrade-k8s-1.35 branch 2 times, most recently from 06da07e to 6ca589f Compare May 27, 2026 11:13

yankay mentioned this pull request May 28, 2026

podgrouper: webhook rejects PodGroups with parent SubGroups carrying minMember=0 (breaks Grove topologyConstraintGroupConfigs) kai-scheduler/KAI-Scheduler#1639

Closed

yankay force-pushed the chore/602-upgrade-k8s-1.35 branch from 6ca589f to cd75d03 Compare June 11, 2026 08:19

yankay force-pushed the chore/602-upgrade-k8s-1.35 branch from cd75d03 to 24ffe87 Compare June 11, 2026 08:27

yankay force-pushed the chore/602-upgrade-k8s-1.35 branch 2 times, most recently from f8b9ea1 to c754169 Compare June 11, 2026 10:00

Conversation

yankay commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a API change?

Additional documentation e.g., enhancement proposals, usage docs, etc.:

Uh oh!

copy-pr-bot Bot commented May 12, 2026

Uh oh!

yankay commented May 12, 2026

CI status: blocked on KAI-Scheduler upstream

KAI-Scheduler side

Proposed path

Uh oh!

danbar2 commented May 12, 2026

CI status: blocked on KAI-Scheduler upstream

KAI-Scheduler side

Proposed path

Uh oh!

renormalize left a comment

Choose a reason for hiding this comment

Uh oh!

yankay commented May 25, 2026

Uh oh!

renormalize commented May 26, 2026

Uh oh!

yankay commented May 26, 2026

Uh oh!

enoodle commented May 27, 2026

Uh oh!

yankay commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

renormalize commented Jun 8, 2026

Uh oh!

enoodle commented Jun 8, 2026

Uh oh!

enoodle commented Jun 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yankay commented May 12, 2026 •

edited

Loading

yankay commented May 27, 2026 •

edited

Loading