PVC creation is stuck in Pending state with error: "Waiting for a volume to be created either by the external provisioner 'csi.vsphere.vmware.com' or manually by the system administrator."

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

1.When creating new PVCs, they remain in a Pending state and describing the PVCs shows repeated provisioning errors:

kubectl describe pvc <pvc-name> -n <namespace>

Type     Reason                  Age                  From                             Message
----     ------                  ---                  ----                             -------
Normal   ExternalProvisioning    2m47s (x4643 over 19h) persistentvolume-controller    Waiting for a volume to be created either by the external provisioner 'csi.vsphere.vmware.com' or manually by the system administrator.
Normal   Provisioning            2m3s (x317 over 19h)  csi.vsphere.vmware.com_422acce  External provisioner is provisioning volume for claim

 

2.Review of vpxd.log shows the DatabaseError during disk creation:

2025-09-03T11:50:29.927Z info  vpxd[3563332] [Originator@6876 sub=MoVStorageObjectMgr opID=ddbc193a-87] Creating disk pvc-9eefe24e-989d-4853-xxxx-01b6dbb87aea on datastore = vim.Datastore:datastore-xxxx
2025-09-03T11:50:33.879Z error vpxd[3563332] [Originator@6876 sub=Default opID=ddbc193a-87] [VpxLRO] -- ERROR task-1337985 -- VStorageObjectManager -- vim.vslm.vcenter.VStorageObjectManager.createDisk: :vim.fault.DatabaseError
--> Result:
--> (vim.fault.DatabaseError) {
-->    msg = "Database temporarily unavailable or has network problems."
--> }

 

3.hostd.log reports repeated failures to obtain a vClock tick:

2025-09-03T11:50:33.866Z In(166) Hostd[...] Transfer to exception error code: 601, message: Unable to obtain a tick for: kAttempts = 32
2025-09-03T11:50:33.866Z Wa(164) Hostd[...] Vslm Failure: VslmCreateDisk failed ... with type vim.fault.DatabaseError

 

4.vmkernel.log shows lock failures on the datastore:

2025-09-03T11:50:30.405Z In(182) vmkernel: cpu0:XXXXXXX opID=XXXXXXXX)DLX: 2670: vol '<redacted>', lock ... Read Lock(s) held on a file ... 
2025-09-03T11:50:30.405Z In(182) vmkernel: cpu0:XXXXXXX opID=XXXXXXXX)Fil3: 5174: Lock failed on file: <redacted>.vmdk

 

5.Attempts to rename or move the vClock file fail:

[root@achesxixx:/vmfs/volumes/680b4d65-0ec61b01-xxxx-fac2fb100002/catalog/vclock]
mv vclock-12310 vclock-12310a
mv: can't rename 'vclock-12310': No such file or directory

Environment

VMware Tanzu Kubernetes Grid

vSphere CSI Plugin

vSphere with Tanzu 7.0

vSphere with Tanzu 8.0

Cause

The vClock file could not be renamed or updated due to an underlying filesystem issue, which prevented VSLM from completing the disk creation process.

Resolution

Recreate the vClock file and perform a datastore reconciliation using below steps:

 

1.Stop CSI controller pods.

kubectl -n vmware-system-csi scale deployment vsphere-csi-controller --replicas=0

 

2.Stop the hostd agent on all ESXi hosts

/etc/init.d/hostd stop

 

3.Backup the catalog folder and remove the vClock folder

cd /vmfs/volumes/<datastore>/

mv catalog catalog_backup

 

4.Recreate catalog and vClock folder, and add a new vClock file with same name.

cd /vmfs/volumes/<datastore>/catalog/vclock

touch vclock-12310

 

5.Start hostd on one ESXi host and perform datastore reconciliation via the ESXi MOB page

https://<ESXi_host_FQDN_or_IP>/mob/?moid=ha-vstorage-object-manager&method=reconcileDatastoreInventory

Path: MOB > content > vStorageObjectManager > HostReconcileDatastoreInventory_Task

 

6.After reconciliation, restart hostd on all ESXi hosts and scale up the CSI controller pods.

kubectl -n vmware-system-csi scale deployment vsphere-csi-controller --replicas=3

 

7.Confirm that all PVCs transition from Pending to Bound state.