VKS Cluster is stuck in False state due to error: Machine won't be remediated because it is pending removal due to rollout

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

For a new Cluster creation, or updates to an existing VKS cluster (for example updating the KR version in use), in some cases a new Machine fails to bootstrap and fails to join the VKS Cluster.

In case of such a failure, the Machine does not have a nodeRef  set in its status sub resource. Once the Machine health check conditions are satisfied, this Machine will be marked for deletion and would be cleaned up.

When an in-place update is triggered on the Cluster or the specific node pool of the Cluster before this remediation occurs, (say updating the SSH banner setting via osConfiguration.sshd variable), the Machine ends up being stuck requiring a manual intervention before the Cluster creation/update can progress.

This deadlock would manifest on the Cluster object's conditions as below:

Status:
  Conditions:
    ...
    Last Transition Time:  YYYY-MM-DDTHH:MM:SS.MS+HH:MM
    Message:
    Observed Generation:   6
    Reason:                ReconcileSucceeded
    Status:                True
    Type:                  TopologyReconciled
    Last Transition Time:  YYYY-MM-DDTHH:MM:SS.MS+HH:MM
    Message:               * MachineDeployment vks###-####-##-####-####-##-###-worker-#####:
  * Rolling out 3 not up-to-date replicas
    * KubeadmConfig is not up-to-date
    Observed Generation:   6
    Reason:                RollingOut
    Status:                True
    Type:                  RollingOut
    Last Transition Time:  YYYY-MM-DDTHH:MM:SS.MS+HH:MM
    Message:               * Machine vks###-####-##-####-####-##-###-worker-#####: Machine won't be remediated because it is pending removal due to rollout  <<========== This is the relevant message 
    Observed Generation:   6
    Reason:                Remediating
    Status:                True
    Type:                  Remediating

Until the Machine is remediated via a manual intervention, the node pool of the Cluster object would never have the desired number of replicas.

Note: This deadlock is limited to the worker machines of the Cluster only, control plane nodes do not enter into this state.

Environment

vSphere Kubernetes Service 3.7.x

Cause

When a Machine without a nodeRef (due to bootstrap failures), is updated in-place, VKS attempts to move this stuck Machine to a new MachineSet revision instead of simply deleting it. However, because the move mechanism requires a valid nodeRef to proceed, and standard deletion is disabled for machines pending an in-place update, the controller enters a deadlock. It cannot update, move, or delete the machine, causing the rollout to stall indefinitely.

Control plane machines avoid this issue because the KubeadmControlPlane object maintains exclusive ownership, eliminating the movement of machines between MachineSets.

Resolution

This issue will be fixed in a future release.

Please contact Broadcom Support to workaround this issue.