Virtual Volumes datastore is inaccessible after moving to another vCenter Server or refreshing CA certificate or Updating new custom certificate on the ESXi host or the vCenter Server.

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

This article provides information to resolve the certificate issue for vVols after vCenter changes or CA certificate changes. After moving a Host to another vCenter Server or after refreshing CA Certificate, you experience these symptoms:

Virtual Volumes (vVOL) datastores are not accessible.
The command esxcli storage vvol vasaprovider list displays VP status as syncError or Offline
In the ESXi host /var/log/vvold.log, you see similar to:

[YYYY-MM-DDTHH:MM] warning vvold[4AC####] [Originator@6876 sub=Default] VasaSession::GetEndPoint: failed to get endpoint, err=SSL Exception: Verification parameters:

--> PeerThumbprint: ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:E4:85:48:F8

--> ExpectedThumbprint:

--> ExpectedPeerName: <VASA Provider IP address>

--> The remote host certificate has these problems:

-->

--> * unable to get local issuer certificate, using default

[YYYY-MM-DDTHH:MM] info vvold[47B1B70] [Originator@6876 sub=Default] VasaSession::Initialize url is empty

[YYYY-MM-DDTHH:MM] warning vvold[47B1B70] [Originator@6876 sub=Default] VasaSession::DoSetContext: Empty VP URL for VP (xVP)!

[YYYY-MM-DDTHH:MM] info vvold[47B1B70] [Originator@6876 sub=Default] Initialize: Failed to establish connection https://<VASA Provider IP address>:8443/vasa/version.xml

[YYYY-MM-DDTHH:MM] error vvold[47B1B70] [Originator@6876 sub=Default] Initialize: Unable to init session to VP xVP state: 0

[YYYY-MM-DDTHH:MM] info vvold[4770B70] [Originator@6876 sub=Default] VVolUnbindManager::UnbindIdleVVols called

[YYYY-MM-DDTHH:MM] info vvold[4770B70] [Originator@6876 sub=Default] VVolUnbindManager::UnbindIdleVVols done for 0 VVols

[YYYY-MM-DDTHH:MM] info vvold[5ACBB70] [Originator@6876 sub=Default] Came to SI::GetVvolContainer: container <container-GUID>

[YYYY-MM-DDTHH:MM] info vvold[5ACBB70] [Originator@6876 sub=Default] SI:GetVvolContainer successful for Datastore, id=, maxVVol=0 MB

Running the command esxcli storage vvol storagecontainer list returns result similar to:

Datastore

StorageContainer Name: Datastore

UUID: vvol:xxxxxxxxxxxxxxxx-xxxxxxxxxxxx73602

Array: com.vmware.vim:xxxxxxxx3e06-1000000

Size(MB): 0

Free(MB): 0

Accessible: true

Default Policy:

Running the command esxcli storage vvol vasaprovider list returns any of these similar outputs:

Output#1

xVP
VP Name: xVP
URL:https://<VASA Provider IP address>:8443/vasa/version.xml
Status: syncError
Arrays:
Array Id: com.vmware.vim:xxxxxxxx3e06-1000000
Is Active: true
Priority: 0

Output#2

PowerStore VASA provider - PERS
VP Name: PowerStore VASA provider - PERS
URL: https://xx.xx.xx.xx:8443/version.xml

Status: Offline: AuthenticationError [SSL_ERROR_SSL

error:0A000086:SSL routines::certificate verify failed unable to get local issuer certificate / SSL/TLS handshake failed]

This issue can also occur when vCenter and ESXi Hosts are using a custom certificate and when either the Host or vCenter certificate has recently expired or been updated with new custom certificate within the VMware Cluster. The symptoms outlined previously will still be observed in these cases.

Environment

VMware vSphere ESXi
VMware vCenter Server

Cause

This issue occurs because the vVol ssl_reset does not occur automatically when a new VMCA signed certificate is pushed to the Host or if the vCenter custom certificate has been updated recently and the ESXi Host is experiencing a thumbprint mismatch issue.
Also, if the ESXi hosts do not recognize the updated root certificate, they may reject communication with vCenter, leading to connectivity issues with vVols.

Resolution

If issue is seen after VMCA signed CA certificate update in vCenter server, try the below steps:

vSphere UI: Right-click Host > Certificates > Refresh CA Certificates.
vSphere UI: Right-click Host > Certificates > Renew Certificate.
Host CLI (Mandatory): SSH into the host and run the command :
/etc/init.d/vvold ssl_reset && /etc/init.d/vvold restart
UI: Right-click Host > Storage > Rescan Storage.

If issue is seen after custom certificate update in vCenter server or ESXi host, try the below steps:

Download the root certificate from the vCenter server and update the root certificate to the ESXi nodes.
Please follow the steps mentioned in the following KBs to download the root vCenter certificate and update the same in ESXi host:

Manual steps to reset the vVold SSL certificate (If above steps do not resolve the issue):

Migrate the virtual machines from the host and place it in maintenance mode.
Log in to the ESXi Shell with root user.
Run the command - /etc/init.d/vvold ssl_reset && /etc/init.d/vvold restart
Run the command - tail -f /var/log/vvold.log
Look for Empty VP URL for VP messages. If you still see Empty VP URL for VP messages the SSL certificate will need to re-generated on the ESXi host.
Edit the re-generated self-signed certificate. Log in to the ESXi Shell, navigate to /etc/vmware/ssl.
Rename the existing certificates:
mv rui.crt orig.rui.crt
mv rui.key orig.rui.key
Run the command /sbin/generate-certificates to generate new certificates.
Confirm that the host successfully generated new certificates by running command ls -l and comparing the time stamps of the new certificate files with orig.rui.crt and orig.rui.key.
Go to vSphere Client, right click the ESXi host, click Certificates, Click Renew Certificate.
Run ls -l to ensure the date changed on the castore.pem file.
Reboot the ESXi host.
Once the host is up, run the command - tail -f /var/log/vvold.log

If you see errors, update the vCenter Server TRUSTED_ROOTS store.

14. Disconnect and reconnect the ESXi host to the vCenter Server to resolve a mismatched SSL thumbprint in vCenter Server compared to the ESXi host.
15. Run tail -f /var/log/vvold.log. to verify the error is no longer seen.

The expected output should be as below:
[YYYY-MM-DDTHH:MM] info vvold[8355B70] [Originator@6876 sub=default] SI:GetVvolVontainer successful for DataStoreName, id= maxVVol=0 MB ...

Additional Information

Related vVol certificate issues: