Skip to content

MachineDeployment observedGeneration updated before other status fields #13738

Open
Open
MachineDeployment observedGeneration updated before other status fields#13738
Labels
kind/bugCategorizes issue or PR as related to a bug.needs-priorityIndicates an issue lacks a `priority/foo` label and requires one.needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
@efarjima

Description

@efarjima
efarjima
opened on May 22, 2026

What steps did you take and what happened?

Similar to #10059 but for MachineDeployments.
I encountered an inconsistency (even if it does not last for long): I use a k8s operator to manage CAPI resources, and at some moment it changed only bootstrap.configRef (not k8sVersion) from a MachineDeployment. The operator checks MD status to determine if it's ready and it can proceed with further steps, but there is a time window in which the ObservedGeneration (from status and all conditions) is updated to the last one without even having started to roll out. If we look at the MachineDeployment at that specific point in time, we would think the latest generation is fully ready when it has not even re-rolled one machine.
A few iterations later we can see the RollingOut condition changed to True and upToDateReplicas == 0.
Here is a condensed view of the MD transitions captured with kubectl get machinedeployments --watch -oyaml

1. Object is updated and CAPI did not react 
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
  annotations:
    machinedeployment.clusters.x-k8s.io/revision: "2"
  creationTimestamp: "2026-05-22T13:39:10Z"
  finalizers:
  - cluster.x-k8s.io/machinedeployment
  generation: 3
  labels:
    cluster.x-k8s.io/cluster-name: workload-docker-01
  name: workload-docker-01-wave-1
  namespace: c-workload-docker-01
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta2
    kind: Cluster
    name: workload-docker-01
    uid: d07ab90f-018e-4ded-adc7-696079357b56
  resourceVersion: "7888632"
  uid: 025e14c3-ad3c-40aa-8ad3-00c411587b23
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2026-05-22T13:39:57Z"
    message: ""
    observedGeneration: 2
    reason: Available
    status: "True"
    type: Available
  - lastTransitionTime: "2026-05-22T14:39:14Z"
    message: ""
    observedGeneration: 2
    reason: NotRollingOut
    status: "False"
    type: RollingOut
  - lastTransitionTime: "2026-05-22T13:39:12Z"
    message: ""
    observedGeneration: 2
    reason: NotRemediating
    status: "False"
    type: Remediating
  - lastTransitionTime: "2026-05-22T14:39:14Z"
    message: ""
    observedGeneration: 2
    reason: NotScalingDown
    status: "False"
    type: ScalingDown
  - lastTransitionTime: "2026-05-22T13:39:16Z"
    message: ""
    observedGeneration: 2
    reason: NotScalingUp
    status: "False"
    type: ScalingUp
  - lastTransitionTime: "2026-05-22T14:39:14Z"
    message: ""
    observedGeneration: 2
    reason: Ready
    status: "True"
    type: MachinesReady
  - lastTransitionTime: "2026-05-22T14:39:14Z"
    message: ""
    observedGeneration: 2
    reason: UpToDate
    status: "True"
    type: MachinesUpToDate
  - lastTransitionTime: "2026-05-22T13:39:11Z"
    message: ""
    observedGeneration: 2
    reason: NotPaused
    status: "False"
    type: Paused
  - lastTransitionTime: "2026-05-22T13:39:12Z"
    message: ""
    observedGeneration: 2
    reason: NotDeleting
    status: "False"
    type: Deleting
  observedGeneration: 2
  phase: Running
  readyReplicas: 1
  replicas: 1
  selector: cluster.x-k8s.io/cluster-name=workload-docker-01,cluster.x-k8s.io/deployment-name=workload-docker-01-wave-1
  upToDateReplicas: 1
2. First CAPI reconciliation setting observedGeneration 
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
  annotations:
    machinedeployment.clusters.x-k8s.io/revision: "3"
  creationTimestamp: "2026-05-22T13:39:10Z"
  finalizers:
  - cluster.x-k8s.io/machinedeployment
  generation: 3
  labels:
    cluster.x-k8s.io/cluster-name: workload-docker-01
  name: workload-docker-01-wave-1
  namespace: c-workload-docker-01
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta2
    kind: Cluster
    name: workload-docker-01
    uid: d07ab90f-018e-4ded-adc7-696079357b56
  resourceVersion: "7888650"
  uid: 025e14c3-ad3c-40aa-8ad3-00c411587b23
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2026-05-22T13:39:57Z"
    message: ""
    observedGeneration: 3
    reason: Available
    status: "True"
    type: Available
  - lastTransitionTime: "2026-05-22T14:39:14Z"
    message: ""
    observedGeneration: 3
    reason: NotRollingOut
    status: "False"
    type: RollingOut
  - lastTransitionTime: "2026-05-22T13:39:12Z"
    message: ""
    observedGeneration: 3
    reason: NotRemediating
    status: "False"
    type: Remediating
  - lastTransitionTime: "2026-05-22T14:39:14Z"
    message: ""
    observedGeneration: 3
    reason: NotScalingDown
    status: "False"
    type: ScalingDown
  - lastTransitionTime: "2026-05-22T13:39:16Z"
    message: ""
    observedGeneration: 3
    reason: NotScalingUp
    status: "False"
    type: ScalingUp
  - lastTransitionTime: "2026-05-22T14:39:14Z"
    message: ""
    observedGeneration: 3
    reason: Ready
    status: "True"
    type: MachinesReady
  - lastTransitionTime: "2026-05-22T14:39:14Z"
    message: ""
    observedGeneration: 3
    reason: UpToDate
    status: "True"
    type: MachinesUpToDate
  - lastTransitionTime: "2026-05-22T13:39:11Z"
    message: ""
    observedGeneration: 3
    reason: NotPaused
    status: "False"
    type: Paused
  - lastTransitionTime: "2026-05-22T13:39:12Z"
    message: ""
    observedGeneration: 3
    reason: NotDeleting
    status: "False"
    type: Deleting
  observedGeneration: 3
  phase: Running
  readyReplicas: 1
  replicas: 1
  selector: cluster.x-k8s.io/cluster-name=workload-docker-01,cluster.x-k8s.io/deployment-name=workload-docker-01-wave-1
  upToDateReplicas: 1
3. CAPI reporting RollingOut condition for the first time 
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
  annotations:
    machinedeployment.clusters.x-k8s.io/revision: "3"
  creationTimestamp: "2026-05-22T13:39:10Z"
  finalizers:
  - cluster.x-k8s.io/machinedeployment
  generation: 3
  labels:
    cluster.x-k8s.io/cluster-name: workload-docker-01
  name: workload-docker-01-wave-1
  namespace: c-workload-docker-01
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta2
    kind: Cluster
    name: workload-docker-01
    uid: d07ab90f-018e-4ded-adc7-696079357b56
  resourceVersion: "7888674"
  uid: 025e14c3-ad3c-40aa-8ad3-00c411587b23
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2026-05-22T13:39:57Z"
    message: ""
    observedGeneration: 3
    reason: Available
    status: "True"
    type: Available
  - lastTransitionTime: "2026-05-22T15:08:01Z"
    message: |-
      Rolling out 1 not up-to-date replicas
      * KubeadmConfig is not up-to-date
    observedGeneration: 3
    reason: RollingOut
    status: "True"
    type: RollingOut
  - lastTransitionTime: "2026-05-22T13:39:12Z"
    message: ""
    observedGeneration: 3
    reason: NotRemediating
    status: "False"
    type: Remediating
  - lastTransitionTime: "2026-05-22T14:39:14Z"
    message: ""
    observedGeneration: 3
    reason: NotScalingDown
    status: "False"
    type: ScalingDown
  - lastTransitionTime: "2026-05-22T13:39:16Z"
    message: ""
    observedGeneration: 3
    reason: NotScalingUp
    status: "False"
    type: ScalingUp
  - lastTransitionTime: "2026-05-22T14:39:14Z"
    message: ""
    observedGeneration: 3
    reason: Ready
    status: "True"
    type: MachinesReady
  - lastTransitionTime: "2026-05-22T15:08:01Z"
    message: |-
      * Machine workload-docker-01-wave-1-lldqr-hhcrl:
        * KubeadmConfig is not up-to-date
    observedGeneration: 3
    reason: NotUpToDate
    status: "False"
    type: MachinesUpToDate
  - lastTransitionTime: "2026-05-22T13:39:11Z"
    message: ""
    observedGeneration: 3
    reason: NotPaused
    status: "False"
    type: Paused
  - lastTransitionTime: "2026-05-22T13:39:12Z"
    message: ""
    observedGeneration: 3
    reason: NotDeleting
    status: "False"
    type: Deleting
  deprecated:
    v1beta1:
      availableReplicas: 1
      conditions:
      - lastTransitionTime: "2026-05-22T13:39:56Z"
        status: "True"
        type: Ready
      - lastTransitionTime: "2026-05-22T13:39:56Z"
        status: "True"
        type: Available
      - lastTransitionTime: "2026-05-22T15:08:00Z"
        reason: WaitingForMachineSet
        severity: Info
        status: "False"
        type: MachineSetReady
      readyReplicas: 1
      unavailableReplicas: 1
      updatedReplicas: 0
  observedGeneration: 3
  phase: Running
  readyReplicas: 1
  replicas: 1
  selector: cluster.x-k8s.io/cluster-name=workload-docker-01,cluster.x-k8s.io/deployment-name=workload-docker-01-wave-1
  upToDateReplicas: 1
4. Finally CAPI reports upToDateReplicas: 0 
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
  annotations:
    machinedeployment.clusters.x-k8s.io/revision: "3"
  creationTimestamp: "2026-05-22T13:39:10Z"
  finalizers:
  - cluster.x-k8s.io/machinedeployment
  generation: 3
  labels:
    cluster.x-k8s.io/cluster-name: workload-docker-01
  name: workload-docker-01-wave-1
  namespace: c-workload-docker-01
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta2
    kind: Cluster
    name: workload-docker-01
    uid: d07ab90f-018e-4ded-adc7-696079357b56
  resourceVersion: "7888696"
  uid: 025e14c3-ad3c-40aa-8ad3-00c411587b23
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2026-05-22T13:39:57Z"
    message: ""
    observedGeneration: 3
    reason: Available
    status: "True"
    type: Available
  - lastTransitionTime: "2026-05-22T15:08:01Z"
    message: |-
      Rolling out 1 not up-to-date replicas
      * KubeadmConfig is not up-to-date
    observedGeneration: 3
    reason: RollingOut
    status: "True"
    type: RollingOut
  - lastTransitionTime: "2026-05-22T13:39:12Z"
    message: ""
    observedGeneration: 3
    reason: NotRemediating
    status: "False"
    type: Remediating
  - lastTransitionTime: "2026-05-22T14:39:14Z"
    message: ""
    observedGeneration: 3
    reason: NotScalingDown
    status: "False"
    type: ScalingDown
  - lastTransitionTime: "2026-05-22T13:39:16Z"
    message: ""
    observedGeneration: 3
    reason: NotScalingUp
    status: "False"
    type: ScalingUp
  - lastTransitionTime: "2026-05-22T14:39:14Z"
    message: ""
    observedGeneration: 3
    reason: Ready
    status: "True"
    type: MachinesReady
  - lastTransitionTime: "2026-05-22T15:08:01Z"
    message: |-
      * Machine workload-docker-01-wave-1-lldqr-hhcrl:
        * KubeadmConfig is not up-to-date
    observedGeneration: 3
    reason: NotUpToDate
    status: "False"
    type: MachinesUpToDate
  - lastTransitionTime: "2026-05-22T13:39:11Z"
    message: ""
    observedGeneration: 3
    reason: NotPaused
    status: "False"
    type: Paused
  - lastTransitionTime: "2026-05-22T13:39:12Z"
    message: ""
    observedGeneration: 3
    reason: NotDeleting
    status: "False"
    type: Deleting
  deprecated:
    v1beta1:
      availableReplicas: 1
      conditions:
      - lastTransitionTime: "2026-05-22T13:39:56Z"
        status: "True"
        type: Ready
      - lastTransitionTime: "2026-05-22T13:39:56Z"
        status: "True"
        type: Available
      - lastTransitionTime: "2026-05-22T15:08:00Z"
        reason: WaitingForMachineSet
        severity: Info
        status: "False"
        type: MachineSetReady
      readyReplicas: 1
      unavailableReplicas: 1
      updatedReplicas: 0
  observedGeneration: 3
  phase: Running
  readyReplicas: 1
  replicas: 1
  selector: cluster.x-k8s.io/cluster-name=workload-docker-01,cluster.x-k8s.io/deployment-name=workload-docker-01-wave-1
  upToDateReplicas: 0

What did you expect to happen?

I would expect MD status to be always consistent regardless of which change is triggering a re-roll.

Cluster API version

1.13.2

Kubernetes version

1.34.0

Anything else you would like to add?

No response

Label(s) to be applied

/kind bug
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.needs-priorityIndicates an issue lacks a `priority/foo` label and requires one.needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions