"Unable to enumerate and validate the root certificates from the TRUSTED_ROOTS VECS store" pre-upgrade check error while upgrading to VC 6.7

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Upgrading from  vSphere 6.5 to 6.7 fails at stage 2 pre-checks with the error:
  • Unable to enumerate and validate the root certificates from the TRUSTED_ROOTS VECS store.

 

 

Environment

VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.7.x

Cause

This issue occurs due to invalid entries in the trusted root store.

Resolution

To resolve this issue, ensure that the vmafd service is reachable and started before continuing.

Note: Take a valid snapshot/backup of the source vCenter Server. 

1. Check for invalid entries (alias should contain thumbprint not a url) in TRUSTED_ROOTS store by running these commands on the source appliance:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS | grep Alias

You see the output similar to: 
 

Alias : 44cc699f8acb000e22cd9d54xxxxxxxxxxxxxxxx
Alias : 4ed81f47dda09fe0d2e28156xxxxxxxxxxxxxxxx
Alias : https://192.168.148.60:8443/vasa/version.xml
 

Note: The third URL in the above list is invalid as it does not match the alias and should be deleted.

2. Take a backup of certs by running this command:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store trusted_roots --alias <URL> --output <location>

For example:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store trusted_roots --alias https://192.168.148.60:8443/vasa/version.xml --output /storage/core/nimble192.168.148.60.crt

3. Delete the entry from the store by running this command:(the connection to VASA is not disrupted by performing this deletion)

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store trusted_roots --alias <URL>

For example:
 

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store trusted_roots --alias https://192.168.148.75:8443/vasa/version.xml

4. To publish the certificate to vmdir run this command:

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert <location> --login administrator --password <password>

For example:
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /storage/core/nimble192.168.148.60.crt  --login administrator --password VMware123

5. To list of TRUSTED_ROOTS certs, run the command:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS | grep Alias

You see the output similar to: 
 

Alias : 44cc699f8acb000e22cd9d54xxxxxxxxxxxxxxxx
Alias : 4ed81f47dda09fe0d2e28156xxxxxxxxxxxxxxxx
Alias : 6a44eff3db63a9699ba3e65axxxxxxxxxxxxxxxx

6. Retry the vCenter upgrade process and pre-checks will complete successfully.
7. If the certificate is no longer in use, you can proceed to unpublish the cert using below command.

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert <location> --login administrator --password <password>

For example:
 

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /storage/core/nimble192.168.148.60.crt  --login administrator --password VMware123

Additional Information

Impact/Risks:
If VASA provider is in use, un-publishing certificate you will lose access to vvol datastore.