Workers stuck in SchedulingDisabled and unable to remove nodes in a Tanzu Kubernetes Cluster

Products

• VMware Cloud Director

Issue/Introduction

• Worker nodes are stuck in Ready,SchedulingDisabled state and we are unable to uncordon or remove them from the Tanzu Kubernetes Cluster

• When this issue occurs, the following error message appears in the VMware Cloud Director Container Service Extension log file: 

  Error processing "csi-#########################################################": failed to detach: rpc error: code = Unknown desc = error while waiting for detach task for disk [pvc-########-####-###-####-############] from VM

Environment

VMware Cloud Director 10.6.x

VMware Cloud Director Container Service Extension 4.2.3

Cause

The failure is caused by a mismatch between the 'disk' table in the Cloud Director Database containing a 'vmdk_path' and 'ds_moref' (datastore) value which does not match where the vmdk location and datastore name located in vSphere.

Resolution

To resolve this issue, contact Broadcom Support and note this Article ID (385523) in the problem description. For more information, see Creating and managing Broadcom support cases