PVC creation is stuck in Pending state with error: "Waiting for a volume to be created either by the external provisioner 'csi.vsphere.vmware.com' or manually by the system administrator."

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When creating new PVCs, they remain in a Pending state and describing the PVCs shows repeated provisioning errors:
  
  kubectl describe pvc <pvc-name> -n <namespace>
  
  Type     Reason                  Age                  From                             Message
----     ------                  ---                  ----                             -------
Normal   ExternalProvisioning    2m47s (x4643 over 19h) persistentvolume-controller    Waiting for a volume to be created either by the external provisioner 'csi.vsphere.vmware.com' or manually by the system administrator.
Normal   Provisioning            2m3s (x317 over 19h)  csi.vsphere.vmware.com_422acce  External provisioner is provisioning volume for claim


• Review of the vCenter logs, /var/log/vmware/vpxd/vpxd.log shows the DatabaseError during disk creation:
  
  YYYY-MM-DDTHH:MM:SSZ info  vpxd[3563332] [Originator@6876 sub=MoVStorageObjectMgr opID=####] Creating disk pvc-#### on datastore = vim.Datastore:datastore-####
YYYY-MM-DDTHH:MM:SSZ error vpxd[3563332] [Originator@6876 sub=Default opID=####] [VpxLRO] -- ERROR task-1337985 -- VStorageObjectManager -- vim.vslm.vcenter.VStorageObjectManager.createDisk: :vim.fault.DatabaseError
--> Result:
--> (vim.fault.DatabaseError) {
-->    msg = "Database temporarily unavailable or has network problems."
--> }
  
  
• On the ESXi host, /var/run/log/hostd.log reports repeated failures to obtain a vClock tick:
  
  YYYY-MM-DDTHH:MM:SSZ In(166) Hostd[...] Transfer to exception error code: 601, message: Unable to obtain a tick for: kAttempts = 32
YYYY-MM-DDTHH:MM:SSZ Wa(164) Hostd[...] Vslm Failure: VslmCreateDisk failed ... with type vim.fault.DatabaseError
  
  
• On the ESXi host, /var/run/log/vmkernel.log shows lock failures on the datastore:
  
  YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu0:#### opID=####)DLX: 2670: vol '<redacted>', lock ... Read Lock(s) held on a file ... 
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu0:#### opID=####)Fil3: 5174: Lock failed on file: <redacted>.vmdk
  
  
• Attempts to rename or move the vClock file fails:
  
  [root@achesxixx:/vmfs/volumes/<datastore_id>/catalog/vclock]
mv vclock-12310 vclock-12310a
mv: can't rename 'vclock-12310': No such file or directory

Environment

• VMware Tanzu Kubernetes Grid
• vSphere CSI Plugin
• vSphere Kubernetes Service

Cause

The vClock file could not be renamed or updated due to an underlying filesystem issue, which prevented VSLM from completing the disk creation process.

Resolution

Use the KB article 320790 to rebuild the FCD Catalog on the datastore. The vclock file will be recreated as part of this process.