Unable to mount vVol datastores on new ESXi hosts.

Products

VMware vCenter Server

Issue/Introduction

A newly added ESXi host in a vSphere cluster is unable to mount or access existing Virtual Volumes (vVol) datastores. This issue typically occurs during cluster expansion or host replacement when the new host fails to synchronize with the VASA storage providers. The failure is often characterized by certificate formatting errors or thumbprint mismatches between the host and the storage provider, resulting in the datastore appearing as 'inaccessible' even if the protocol endpoints are reachable.

Virtual Volumes (vVol) datastores show as 'inaccessible' in the vSphere Client for a newly added host.
Command esxcli storage vvol vasaprovider list reports status as offline or syncError.
The vvold.log on the affected host contains errors such as:
- HandleRequest: unexpected exception handling function UpdateVasaProvider: Wrong X.509 Certificate format
- Storage Fault TRANSPORT_FAULT (1): Wrong X.509 Certificate format
- VVolLib_UpdateVendorProvider:8041: Failed with error (10) The VASA transport encountered an error

Environment

VMware vSphere ESXi 8.0.x
VMware vCenter Server 8.0.x
vVol enabled storage (e.g., HPE Alletra, 3PAR)
HPE Synergy Gen11 servers

Cause

During registration, the certificate of VVOL VP is stored in the PEM format (header and footer added). SPS runs a periodic threads called VasaInfoPoller/VasaProviderCertificatePoller -both fetch the VP certificate conditionally when retainVasaProviderCertificate=true and updates local copy if certificate has changed. But certificate retrieved is compared against PEM formatted certificate and certificate also persisted in non-PEM format. VVOL service expects SSC(self signed certificate) to be PEM format, without which VVOLD->VP communication would be broken.

Resolution

A permanent fix for this defect is targeted for inclusion in upcoming ESXi versions 8.0.x and 9.1.x. Users are encouraged to "Subscribe to this article" to be notified when the fix is available."

Method 1: Standard Certificate Reset

Right-click the affected ESXi host > Certificates > Refresh CA Certificates.
Right-click the affected ESXi host > Certificates > Renew Certificate.

SSH to the host as root and restart the vVol service:

Right-click the host > Storage > Rescan Storage.

Method 2: Manual vvol-vpconfig Workaround

If Method 1 fails and logs confirm a certificate format exception, manually update the VASA configuration from a working host in the same cluster:

On a working host, export the VASA configuration:

Transfer the vvol-vpconfig.json file to the /tmp/ directory of the affected host.

On the affected host, import the configuration:

Restart the vVol service:
```
/etc/init.d/vvold restart
```

Method 3: Changing the certificate polling interval config.

1. Login to the vcenter using SSH and take backup of /usr/lib/vmware-vpx/sps/conf/sms-config.xml file
2. vi /usr/lib/vmware-vpx/sps/conf/sms-config.xml,
Update value of certificatePollingIntervalInDays to -1 or to huge number like (1000).

3. Restart sps service - vmon-cli -r sps

4. Un register and register VP(VASA provider) in the vCenter GUI (vCenter > Configure > Storage provider)

NOTE: Since we are disabling VasaInfoPoller thread, the VP certificate expiry alarm related won't be posted on the vCenter.

Note: If certificatePollingIntervalInDays is updated to -1, this thread is not at all scheduled and you will see following error in sps.log. If updated to huge positive value N, then thread would be scheduled to run after N days.

Additional Information

Contact Broadcom Support

Virtual Volumes (vVOL) datastore inaccessible after moving to another vCenter Server or renewing / replacing custom CA certificate on the ESXi host or vCenter Server