After upgrade of vCenter server appliance from v7.x to v8.0U3 or later, Persistent Volumes provisioned by the vSphere CSI driver fail to attach

Products

• VMware vCenter Server 8.0
• VMware vCenter Server 7.0

Issue/Introduction

Customers who use the vSphere CSI driver to provision persistent volumes  may encounter issues after upgrading from  vCenter from 7.x to 8.0U3. Pods do not start and Persistent Volumes fail to attach to the nodes

Cause

The issue occurs due to an upgrade error when modifying database tables to align with the vCenter 8.x database schema.

The following error will be noticed in the vsanvcmgmtd.log (/var/log/vmware/vsan-health/vsanvcmgmtd.log) on the vCenter Server:

"DB Error while inserting volume ***: Column ALLOCATED_SPACE does not exist in Table Definition of table volume_info" or "Cannot process vStorage objectxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx on ds:///vmfs/volumes/****".

The following logging may also be observed in (/var/log/vmware/vsan-health/vsanvcmgmtd.log) on the vCenter Server:

2025-02-27T14:00:00.635+01:00 error vsanvcmgmtd[xxxx] [vSAN@xxx sub=Workflow opId=xxxxxx] Workflow current action has fault (vslm.fault.VslmFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    msg = "DB Update Failed"

Resolution

Ensure to take a snapshot of the vCenter before applying the workaround.

This issue will be fixed in a future release of vCenter 8.x.

Workaround:

1. Login to vPostgres Database on vCenter via SSH client 

# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

2. Now, truncate cns.VPX_TABLE.

  TRUNCATE TABLE cns.VPX_TABLE;

3. Restart vsan-health service.

   # vmon-cli -r vsan-health

4. Confirm that cns.VPX_TABLE has the allocated_space column now.

   SELECT * from cns.VPX_TABLE;

   The column we are interested in is "volume_info | allocated_space | -5 | 8 | 0 "

5. Run fullSync on the datastore containing the volumes that can not be found in the CNS Schema. Go to the MOB page of vCenter using below link

   https://<VC_IP>/vslm/mob/?moid=StorageLifecycleManager&method=VslmSyncDatastore

6. Modify the fields as below

   datastoreUrl(something like ds:///vmfs/volumes/***/) can be queried from the vsanvcmgmt.log as following. fullSync should be true. fcdId should be blank.

7. Wait for 10 mins and check if volumes in datastore ds:///vmfs/volumes/***/ start showing in cns.volume_info table.

SELECT * from cns.volume_info;