Detaching of PVCs to TKC VMs or Service VMs does not work and also errors, "The object or item referred to could not be found."

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Tasks to "detach a virtual disk" error with "The object or item referred to could not be found."

• When checking events specific to an affected vSphere Namespace, following errors can be observed:

root@SV [ ~ ]# kubectl get events -n <vsphere-namespace>
LAST SEEN   TYPE      REASON         OBJECT                    MESSAGE
1m          Warning   UpdateFailed   cnsvolumemetadata/[...]   ReconcileCnsVolumeMetadata: Failed to delete entry in CNS for instance with name "#####" and entity type "POD" in the guest cluster "#######-####-####-####-############". Requeuing request.
2m          Warning   UpdateFailed   cnsvolumemetadata/[...]   ReconcileCnsVolumeMetadata: Failed to delete entry in CNS for instance with name "#####" and entity type "POD" in the guest cluster "#######-####-####-####-############". Requeuing request.

• Checking vsphere-csi-controller logs of vsphere-sync container, following logs are visible:

root@SV [ ~ ]# kubectl logs -n vmware-system-csi vsphere-csi-controller-#########-##### vsphere-syncer | grep "op timeout"
{"level":"info","time":"####-##-##T##:##:##.000Z","caller":"volume/listview.go:253","msg":"op timeout. context deadline exceeded. using listview context without a timeout","TraceId":"########-####-####-####-############"}

Environment

VMware vSphere 8.0 / 9.0

VMware vSphere Kubernetes Service

Cause

The vsphere-csi-controller fails to detect task completion in rare circumstances.

Resolution

Workaround

To workaround this issue, please restart the vsphere-csi-controller on the Supervisor:

# kubectl rollout restart deployment -n vmware-system-csi vsphere-csi-controller
 
Depending on the environment size, please allow the environment some minutes (around 5-15 minutes) to catch up.