Newly provisioned VM stuck in Powered Off state fails to join Kubernetes cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Describing the VM object stuck in powered off state shows the following:

Events:
  Type     Reason                 Age                   From                                                                                                 Message
  ----     ------                 ----                  ----                                                                                                 -------
  Warning  CreateOrUpdateFailure  12m (x2 over 12m)     vmware-system-vmop/vmware-system-vmop-controller-manager-####/virtualmachine-controller  status update pending for persistent volume: ####-containerd on VM
  Warning  CreateOrUpdateFailure  7m9s (x16 over 12m)   vmware-system-vmop/vmware-system-vmop-controller-manager-####/virtualmachine-controller  persistent volume: ####-containerd not attached to VM

• Following log entires are seen in the /var/log/vmware/vsan-health/vsanvcmgmtd.log :

YYYY-MM-DDTHH:MM:SSZ error vsanvcmgmtd[1052547] [vSAN@6876 sub=FcdService opID=####] Fail to add a SCSI controller, the number of SCSI controller reached limit
YYYY-MM-DDTHH:MM:SSZ info vsanvcmgmtd[1052547] [vSAN@6876 sub=FcdService opID=####] GetPVSCSIControlerWithFreeSlot return controllerKey=-99
YYYY-MM-DDTHH:MM:SSZ error vsanvcmgmtd[1052547] [vSAN@6876 sub=FcdService opID=####] Failed to attach disk to vm: vim.VirtualMachine:vm-## because missing SCSI controller
YYYY-MM-DDTHH:MM:SSZ error vsanvcmgmtd[1052547] [vSAN@6876 sub=Workflow opID=####] Workflow current action has fault (vim.fault.CnsMissingControllerFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    reason = "Failed to attach disk to vm: vim.VirtualMachine:vm-## because missing SCSI controller"
-->    msg = ""
--> }

Environment

vSphere Kubernetes Service

Cause

During node provisioning, the VM Operator creates the VM but waits for the CSI controller to attach the required persistent storage (used for /var/lib/containerd) before powering it on. Since the CSI controller is unresponsive, the volume attachment remains pending.

Resolution

Restart the vsphere-csi-controller deployment in the Supervisor cluster to clear the hung state.

k rollout restart deployment vsphere-csi-controller -n vmware-system-csi