Virtual Volumes datastore is inaccessible after moving to another vCenter Server or refreshing CA certificate or Updating new custom certificate on the ESXi host or the vCenter Server.

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

This article provides information to resolve the certificate issue for vVols after vCenter changes or CA certificate changes.

Symptoms:

1. After moving a host to another vCenter Server or after refreshing CA Certificate, you experience these symptoms:

Virtual Volumes (vVOL) datastores are not accessible.
The command esxcli storage vvol vasaprovider list displays VP status as syncError.
In the ESXi host /var/log/vvold.log, you see similar to:

[YYYY-MM-DDTHH:MM] warning vvold[4AC6B70] [Originator@6876 sub=Default] VasaSession::GetEndPoint: failed to get endpoint, err=SSL Exception: Verification parameters:

--> PeerThumbprint: XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:E4:85:48:F8

--> ExpectedThumbprint:

--> ExpectedPeerName: <VASA Provider IP address>

--> The remote host certificate has these problems:

-->

--> * unable to get local issuer certificate, using default

[YYYY-MM-DDTHH:MM] info vvold[47B1B70] [Originator@6876 sub=Default] VasaSession::Initialize url is empty

[YYYY-MM-DDTHH:MM] warning vvold[47B1B70] [Originator@6876 sub=Default] VasaSession::DoSetContext: Empty VP URL for VP (xVP)!

[YYYY-MM-DDTHH:MM] info vvold[47B1B70] [Originator@6876 sub=Default] Initialize: Failed to establish connection https://<VASA Provider IP address>:8443/vasa/version.xml

[YYYY-MM-DDTHH:MM] error vvold[47B1B70] [Originator@6876 sub=Default] Initialize: Unable to init session to VP xVP state: 0

[YYYY-MM-DDTHH:MM] info vvold[4770B70] [Originator@6876 sub=Default] VVolUnbindManager::UnbindIdleVVols called

[YYYY-MM-DDTHH:MM] info vvold[4770B70] [Originator@6876 sub=Default] VVolUnbindManager::UnbindIdleVVols done for 0 VVols

[YYYY-MM-DDTHH:MM] info vvold[5ACBB70] [Originator@6876 sub=Default] Came to SI::GetVvolContainer: container <container-GUID>

[YYYY-MM-DDTHH:MM] info vvold[5ACBB70] [Originator@6876 sub=Default] SI:GetVvolContainer successful for Datastore, id=, maxVVol=0 MB

Running the command esxcli storage vvol storagecontainer list returns similar to:

Datastore

StorageContainer Name: Datastore

UUID: vvol:xxxxxxxxxxxxxxxx-xxxxxxxxxxxx73602

Array: com.vmware.vim:xxxxxxxx3e06-1000000

Size(MB): 0

Free(MB): 0

Accessible: true

Default Policy:

Running the command esxcli storage vvol vasaprovider list returns similar to:

xVP

VP Name: xVP

URL:https://<VASA Provider IP address>:8443/vasa/version.xml

Status: syncError

Arrays:

Array Id: com.vmware.vim:xxxxxxxx3e06-1000000

Is Active: true

Priority: 0

2. This issue also occurs in vCenter and ESXi servers using a custom certificate when either the Host or vCenter certificate has recently expired or been updated with new custom certificate within the VMware Cluster. The symptoms outlined previously will still be observed in these cases.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 8.0
VMware vSphere ESXi 7.0
vCenter Server 8.0
vCenter Server 7.0

Cause

This issue occurs because the vVol ssl_reset is not occurring automatically when VMCA signed certificate is pushed to the host.

Suppose the vCenter custom certificate has been updated recently and the ESXi host is experiencing a thumbprint mismatch issue. In that case, this indicates that there is a thumbprint mismatch between the ESXi host and vCenter.

Note: If the ESXi hosts do not recognize the updated root certificate, they may reject communication with vCenter, leading to connectivity issues with vVols.

Resolution

To work around this issue reset the vVold SSL certificate:

Migrate the virtual machines from the host and place it in maintenance mode.
Log in to the ESXi Shell with root user.
Run the command - /etc/init.d/vvold ssl_reset && /etc/init.d/vvold restart
Run the command - tail -f /var/log/vvold.log
Look for Empty VP URL for VP messages. If you still see Empty VP URL for VP messages the SSL certificate will need to re-generated on the ESXi host.
Edit the re-generated self-signed certificate. Log in to the ESXi Shell, navigate to /etc/vmware/ssl.
Rename the existing certificates:
mv rui.crt orig.rui.crt
mv rui.key orig.rui.key
Run the command /sbin/generate-certificates to generate new certificates.
Confirm that the host successfully generated new certificates by running command ls -l and comparing the time stamps of the new certificate files with orig.rui.crt and orig.rui.key.
Go to vSphere Client, right click the ESXi host, click Certificates, Click Renew Certificate.
Run ls -l to ensure the date changed on the castore.pem file.
Reboot the ESXi host.
Once the host is up, run the command - tail -f /var/log/vvold.log

If you see errors, update the vCenter Server TRUSTED_ROOTS store.

14. Disconnect and reconnect the ESXi host to the vCenter Server to resolve a mismatched SSL thumbprint in vCenter Server compared to the ESXi host.
15. Run tail -f /var/log/vvold.log. to verify the error is no longer seen.

The expected output should be as below:
[YYYY-MM-DDTHH:MM] info vvold[8355B70] [Originator@6876 sub=default] SI:GetVvolVontainer successful for DataStoreName, id= maxVVol=0 MB ...

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

For the custom, certificate-updated vCenter or ESXi host try the below steps:

Download the root certificate from the vCenter server and update the root certificate to the ESXi nodes.

Please follow the steps mentioned in the following KB to download the Root vCenter certificate and update the the same in ESXi host.

Refer: