Attaching of PVCs to TKC VMs or Service VMs does not work and also error "ReconcileCnsVolumeMetadata: Failed to delete entry in CNS for instance with name" is seen in vSphere Namespace's Events

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Attaching of PVC to TKCs or Service VMs does no longer work and hence TKC nodes can't be provisioned
• When checking events specific to an affected vSphere Namespace, following errors can be observed:

  root@SV [ ~ ]# kubectl get events -n <vsphere-namespace>
  LAST SEEN   TYPE      REASON         OBJECT                    MESSAGE
  1m          Warning   UpdateFailed   cnsvolumemetadata/[...]   ReconcileCnsVolumeMetadata: Failed to delete entry in CNS for instance with name "<TKC>-etcd-0" and entity type "POD" in the guest cluster "#######-####-####-####-############". Requeuing request.
  2m          Warning   UpdateFailed   cnsvolumemetadata/[...]   ReconcileCnsVolumeMetadata: Failed to delete entry in CNS for instance with name "<TKC>-etcd-1" and entity type "POD" in the guest cluster "#######-####-####-####-############". Requeuing request.
  3m          Warning   UpdateFailed   cnsvolumemetadata/[...]   ReconcileCnsVolumeMetadata: Failed to delete entry in CNS for instance with name "<TKC>-etcd-2" and entity type "POD" in the guest cluster "#######-####-####-####-############". Requeuing request.

• Checking vsphere-csi-controller logs of vsphere-sync container, following logs are visible:
  

  root@SV [ ~ ]# kubectl logs -n vmware-system-csi vsphere-csi-controller-#########-##### vsphere-syncer | grep "op timeout"
  {"level":"info","time":"####-##-##T##:##:##.000Z","caller":"volume/listview.go:253","msg":"op timeout. context deadline exceeded. using listview context without a timeout","TraceId":"########-####-####-####-############"}

Environment

VMware vSphere 8.0 / 9.0

VMware vSphere Kubernetes Service

Cause

The vsphere-csi-controller fails to detect task completion in rare circumstances.

Resolution

This issue will be addressed in a future vCenter release.

Workaround

To workaround this issue, please restart the vsphere-csi-controller on the Supervisor:

# kubectl rollout restart deployment -n vmware-system-csi vsphere-csi-controller

 
Depending on the environment size, please allow the environment some minutes (around 5-15 minutes) to catch up.

Additional Information

This issue was also addressed in a future upstream vsphere-csi-driver version in Kubernetes - see: https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/3584 

• • Japanese version: TKC 仮想マシンまたは Service VM への PVC のアタッチが正常に動作せず、あわせて vSphere Namespace のイベントに 「ReconcileCnsVolumeMetadata: Failed to delete entry in CNS for instance with name」