PVC creation is stuck in Pending state with error: "Waiting for a volume to be created either by the external provisioner 'csi.vsphere.vmware.com' or manually by the system administrator."

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When creating new PVCs, they remain in a Pending state and describing the PVCs shows repeated provisioning errors:
  
  kubectl describe pvc <pvc-name> -n <namespace>
  
  Type     Reason                  Age                  From                             Message
----     ------                  ---                  ----                             -------
Normal   ExternalProvisioning    2m47s (x4643 over 19h) persistentvolume-controller    Waiting for a volume to be created either by the external provisioner 'csi.vsphere.vmware.com' or manually by the system administrator.
Normal   Provisioning            2m3s (x317 over 19h)  csi.vsphere.vmware.com_422acce  External provisioner is provisioning volume for claim


• Review of the vCenter logs, /var/log/vmware/vpxd/vpxd.log shows the DatabaseError during disk creation:
  
  YYYY-MM-DDTHH:MM:SSZ info  vpxd[3563332] [Originator@6876 sub=MoVStorageObjectMgr opID=####] Creating disk pvc-#### on datastore = vim.Datastore:datastore-####
YYYY-MM-DDTHH:MM:SSZ error vpxd[3563332] [Originator@6876 sub=Default opID=####] [VpxLRO] -- ERROR task-1337985 -- VStorageObjectManager -- vim.vslm.vcenter.VStorageObjectManager.createDisk: :vim.fault.DatabaseError
--> Result:
--> (vim.fault.DatabaseError) {
-->    msg = "Database temporarily unavailable or has network problems."
--> }
  
  
• On the ESXi host, /var/run/log/hostd.log reports repeated failures to obtain a vClock tick:
  
  YYYY-MM-DDTHH:MM:SSZ In(166) Hostd[...] Transfer to exception error code: 601, message: Unable to obtain a tick for: kAttempts = 32
YYYY-MM-DDTHH:MM:SSZ Wa(164) Hostd[...] Vslm Failure: VslmCreateDisk failed ... with type vim.fault.DatabaseError
  
  
• On the ESXi host, /var/run/log/vmkernel.log shows lock failures on the datastore:
  
  YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu0:#### opID=####)DLX: 2670: vol '<redacted>', lock ... Read Lock(s) held on a file ... 
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu0:#### opID=####)Fil3: 5174: Lock failed on file: <redacted>.vmdk
  
  
• Attempts to rename or move the vClock file fails:
  
  [root@achesxixx:/vmfs/volumes/<datastore_id>/catalog/vclock]
mv vclock-12310 vclock-12310a
mv: can't rename 'vclock-12310': No such file or directory

Environment

• VMware Tanzu Kubernetes Grid
• vSphere CSI Plugin
• vSphere Kubernetes Service

Cause

The vClock file could not be renamed or updated due to an underlying filesystem issue, which prevented VSLM from completing the disk creation process.

Resolution

Recreate the vClock file and perform a datastore reconciliation using below steps:

1. Stop CSI controller pods.
   kubectl -n vmware-system-csi scale deployment vsphere-csi-controller --replicas=0
   
   
2. Stop the hostd agent on all ESXi hosts
   /etc/init.d/hostd stop
   
   
3. Backup the catalog folder and remove the vClock folder
   cd /vmfs/volumes/<datastore>/
mv catalog catalog_backup
   
   
4. Recreate catalog and vClock folder, and add a new vClock file with same name.
   cd /vmfs/volumes/<datastore>/catalog/vclock
touch vclock-12310
   
   
5. Start hostd on one ESXi host and perform datastore reconciliation via the ESXi MOB page
   https://<ESXi_host_FQDN_or_IP>/mob/?moid=ha-vstorage-object-manager&method=reconcileDatastoreInventory
Path: MOB > content > vStorageObjectManager > HostReconcileDatastoreInventory_Task
   
   
6. After reconciliation, restart hostd on all ESXi hosts and scale up the CSI controller pods.
   kubectl -n vmware-system-csi scale deployment vsphere-csi-controller --replicas=3
   
   
7. Confirm that all PVCs transition from Pending to Bound state.