Virtual Volumes (vVOL) datastore inaccessible after moving to another vCenter Server or renewing / replacing custom CA certificate on the ESXi host or vCenter Server

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

This article provides information to on how resolve an issue for Virtual Volumes (vVols) that might arise after renewing or replacing certificates in either ESXi or vCenter Server. After moving an ESXi host to another vCenter Server, refreshing or replacing CA Certificates, the following symptoms might be observed:

Virtual Volumes (vVols) datastores are not accessible.
When querying the vVol vasa provider on an ESXi host using the command below, it is reported as offline or with a syncError:
```
# esxcli storage vvol vasaprovider list
```

The vVol daemon log, /var/run/log/vvold.log, contains an error stack similar to the following example:

[YYYY-MM-DDTHH:MM] warning vvold[4AC####] [Originator@6876 sub=Default] VasaSession::GetEndPoint: failed to get endpoint, err=SSL Exception: Verification parameters:
--> PeerThumbprint: ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:E4:85:48:F8
--> ExpectedThumbprint:
--> ExpectedPeerName: <VASA Provider IP address>
--> The remote host certificate has these problems:
-->
--> * unable to get local issuer certificate, using default
[YYYY-MM-DDTHH:MM] info vvold[47B1B70] [Originator@6876 sub=Default] VasaSession::Initialize url is empty
[YYYY-MM-DDTHH:MM] warning vvold[47B1B70] [Originator@6876 sub=Default] VasaSession::DoSetContext: Empty VP URL for VP (xVP)!
[YYYY-MM-DDTHH:MM] info vvold[47B1B70] [Originator@6876 sub=Default] Initialize: Failed to establish connection https://<VASA Provider IP address>:8443/vasa/version.xml
[YYYY-MM-DDTHH:MM] error vvold[47B1B70] [Originator@6876 sub=Default] Initialize: Unable to init session to VP xVP state: 0
[YYYY-MM-DDTHH:MM] info vvold[4770B70] [Originator@6876 sub=Default] VVolUnbindManager::UnbindIdleVVols called
[YYYY-MM-DDTHH:MM] info vvold[4770B70] [Originator@6876 sub=Default] VVolUnbindManager::UnbindIdleVVols done for 0 VVols
[YYYY-MM-DDTHH:MM] info vvold[5ACBB70] [Originator@6876 sub=Default] Came to SI::GetVvolContainer: container <container-GUID>
[YYYY-MM-DDTHH:MM] info vvold[5ACBB70] [Originator@6876 sub=Default] SI:GetVvolContainer successful for Datastore, id=, maxVVol=0 MB

Running the command esxcli storage vvol storagecontainer list returns results similar to:

Datastore
   StorageContainer Name: Datastore
   UUID: vvol:################-############73602
   Array: com.vmware.vim:xxxxxxxx3e06-1000000
   Size(MB): 0
   Free(MB): 0
   Accessible: true
   Default Policy:

Running the command esxcli storage vvol vasaprovider list returns either of the following outputs:

xVP
  VP Name: xVP
  URL:https://<VASA Provider IP address>:8443/vasa/version.xml
 Status: syncError
 Arrays:
    Array Id: com.vmware.vim:########3e06-1000000
    Is Active: true
    Priority: 0

or

 PowerStore VASA provider - PERS
 VP Name: PowerStore VASA provider - PERS
 URL: https://<VASA Provider IP address>:8443/version.xml
 Status: Offline: AuthenticationError [SSL_ERROR_SSL error:0A000086:SSL routines::certificate verify failed unable to get local issuer certificate / SSL/TLS handshake failed]

This issue can also occur when vCenter and ESXi Hosts are using a custom certificate and when either the Host or vCenter certificate has recently expired or been updated with new custom certificate within the VMware Cluster. The symptoms outlined previously will still be observed in these cases.

Environment

VMware vSphere ESXi
VMware vCenter Server

Cause

This issue occurs because the vVol ssl_reset does not occur automatically when a new VMCA signed certificate is pushed to the Host, or if the vCenter custom certificate has been updated recently and the ESXi Host is experiencing a thumbprint mismatch issue.
Also, if the ESXi hosts do not recognize the updated root certificate, they may reject communication with vCenter, leading to connectivity issues with vVols.

Resolution

If the issue is seen after implementing or upgrading the VMCA root certificate in vCenter Server with a CA signed certificate

Apply the following steps:

In the vSphere Client inventory, right click the affected ESXi host, then in the context menu select Certificates > Refresh CA Certificates.
In the vSphere Client inventory, right click the affected ESXi host, then in the context menu select Certificates > Renew Certificate.
Open an SSH connection to the ESXi host, login with the root user and run the following command:
```
# /etc/init.d/vvold ssl_reset && /etc/init.d/vvold restart
```
In the vSphere Client inventory, right click the affected ESXi host and select Storage > Rescan Storage from the context menu.

If issue is seen after custom certificate update in vCenter server or ESXi host

Download the root certificate from the vCenter server and update the root certificate to the ESXi host by implementing the steps outlined in the following Knowledge Base articles:

Manual method to reset the vVold SSL certificate if the above steps do not resolve the issue:

Migrate the virtual machines from the host and place it in maintenance mode.
Open an SSH connection to the ESXi host and login with the root user.

Run the following command:

# /etc/init.d/vvold ssl_reset && /etc/init.d/vvold restart

Next run this command to monitor the vVol daemon log:
```
# tail -f /var/log/vvold.log
```
Look for messages mentioning Empty VP URL for VP. If vvold still reports Empty VP URL for VP despite the steps in the earlier sections having been applied, the hosts SSL ertificate will need to be re-generated.
Reusing the existing SSH connection to the host, navigate to /etc/vmware/ssl.

Rename the existing certificates:

# mv rui.crt orig.rui.crt

# mv rui.key orig.rui.key

Generate a new SSL host certificate by running the command:
```
# /sbin/generate-certificates
```

Confirm that the host successfully generated new certificates by running the command:

# ls -l and comparing the time stamps of the new certificate files with orig.rui.crt and orig.rui.key.

Optionally verify the start date in the rui.crt by running:

# openssl x509 -in /etc/vmware/ssl/rui.crt -noout -startdate -enddate

Connect to vSphere Client, find the ESXi host in the inventory and right click it
In the context menu, select Certificates > Renew Certificate.
Return to the ESXi host and run # ls -lt /etc/vmware/ssl to confirm that the date changed on the castore.pem file.
Reboot the ESXi host.

Once the host is up, run the command:

# tail -f /var/log/vvold.log

to verify the error is no longer seen.
The expected output should be as below:

[YYYY-MM-DDTHH:MM] info vvold[8355B70] [Originator@6876 sub=default] SI:GetVvolVontainer successful for DataStoreName, id= maxVVol=0 MB ...

Additional Information

Related vVol certificate issues: