vCenter upgrade fails pre-checks with error: "Unable to enumerate and validate the root certificates from the TRUSTED_ROOTS VECS store."

Products

• VMware vCenter Server 6.0
• VMware vCenter Server 7.0

Issue/Introduction

• When upgrading from vCenter server 6.5/6.7 to vCenter 7.x, the upgrade process fails during the stage 2 pre-checks with the following error
  
  

 

 

Cause

The issue is caused when the VASA (Virtual Appliance Storage API) or vVol is configured with self-signed certificates, and the self-signed certificate is incorrectly pushed into the TRUSTED_ROOTS VECS store. Instead of using the certificate's thumbprint, the registration URL is used as the alias, which leads to the failure during the pre-checks.

Resolution

Follow the steps below to resolve the issue:

1. Ensure that the vmafd service is reachable and started.

Before continuing, make sure the vmafd (VMware vCenter Server Directory Service) is up and running.

2. Take a backup of the source vCenter Server

Make sure you take a valid snapshot or backup of the vCenter Server before making any changes.

3. Check for invalid entries in the TRUSTED_ROOTS store

Run the following command to list all entries in the TRUSTED_ROOTS store on the source appliance:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS | grep Alias

The output should resemble the following:

Alias : 44cc699f8acb000e22cd9d54xxxxxxxxxxxxxxxx 
 
Alias : 4ed81f47dda09fe0d2e28156xxxxxxxxxxxxxxxx 

Alias : https://XX.XX.XX.XX:8443/vasa/version.xml  

Note: The third alias listed (i.e., https://XX.XX.XX.XX:8443/vasa/version.xml) is invalid because it uses a URL instead of a thumbprint.

4. Backup the certificate

Before making any changes, it’s good practice to back up the certificate. Use the following command to export the certificate to a file:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store trusted_roots --alias <URL> --output <location>

For example:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store trusted_roots --alias https://XX.XX.XX.XX:8443/vasa/version.xml --output /storage/core/delete_me.crt

5. Delete the invalid entry

After backing up the certificate, you can safely delete the invalid entry from the TRUSTED_ROOTS store.

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store trusted_roots --alias <URL>

For example:

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store trusted_roots --alias https://XX.XX.XX.XX:8443/vasa/version.xml

6. Publish the certificate to vmdir

If you want to publish a new or existing certificate, use the following command:

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert <location> --login administrator --password <password>

For example:

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /storage/core/delete_me.crt --login administrator --password '<your_password>' 

7. Verify the updated TRUSTED_ROOTS store
To verify that the invalid entry has been removed, run the following command:

 

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS | grep Alias

The output should resemble the following:

Alias : 44cc699f8acb000e22cd9d54xxxxxxxxxxxxxxxx 

Alias : 4ed81f47dda09fe0d2e28156xxxxxxxxxxxxxxxx 
 
Alias : 6a44eff3db63a9699ba3e65axxxxxxxxxxxxxxxx

8. Retry the vCenter upgrade
Once the invalid entries have been addressed, proceed with the vCenter upgrade. The pre-checks should now complete successfully.

9. Unpublish the cert (if no longer needed)
If the certificate is no longer in use, you can unpublish it from the vmdir using the following command:

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert <location> --login administrator --password <password>

For example:

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /storage/core/delete_me.crt --login administrator --password '<your_password>' 

 

Additional Information

Additional Notes:

• Always ensure that any certificates used for VASA or vVol configurations are properly aligned with thumbprints as the alias in the TRUSTED_ROOTS store.
• If you encounter any issues after performing these steps, consider reviewing the vCenter logs or reaching out to VMware support for further assistance.
• If VASA provider is in use, un-publishing certificate you will lose access to vvol datastore.