Upgrade to vCenter 7.0 fails with "CollectRequirements for com.vmware.sso times out"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
vCenter Upgrade to 7.0 fails with an error:

Upgrade phase timed out. The time planned for the upgrade phase was 15 minutes. The upgrade phase has already been running for 15 minutes.
CollectRequirements for com.vmware.sso times out

In the Tomcat webapp log [catalina.xxxx-xx-xx.log] you see the following error:

2021-04-28T16:01:10.077Z SEVE org.apache.catalina.core.StandardService Failed to initialize connector [Connector[com.vmware.identity.tomcat.VECSAwareHttp11NioProtocol-7444]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1078)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        ...
Caused by: java.io.IOException: Could not get key with alias __MACHINE_CERT from VECS key store
        at com.vmware.identity.tomcat.VECSAwareSSLImplementation.getTransientKeyStore(VECSAwareSSLImplementation.java:162)
        at com.vmware.identity.tomcat.VECSAwareSSLImplementation$1.getKeyManagers(VECSAwareSSLImplementation.java:65)
        at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:245)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:98)
        ... 19 more

When reviewing the content in the STS_INTERNAL_SSL_CERT store in the VECS you find that it contains no entries:

# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store STS_INTERNAL_SSL_CERT
Number of entries in store :    0

Environment

VMware vCenter Server 7.0.x

Cause

When vCenter Server was upgraded from an older version in the past, the installer expects to find an  STS_INTERNAL_SSL_CERTIFICATE store in the VECS, containing the machine SSL certificate.

If this store does not exist, or holds no entries, the vCenter installer will fail to verify the current STS certificate during the pre-check phase and will get stuck.

As a result, when its default timeout is reached, the upgrade fails with the error shown above.

Resolution

To solve this, publish the current machine SSL certificate to the STS_INTERNAL_SSL_CERT store using the following steps:

1. ensure that you have a current backup of the vCenter
2. open an SSH connection to the old vCenter Server Appliance
3. create an export folder:

   # mkdir /certs

4. export the current machine SSL certificate and public key from the MACHINE_SSL_CERT store:

   # /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --alias __MACHINE_CERT --store MACHINE_SSL_CERT >/certs/__MACHINE_CERT.crt
   # /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --alias __MACHINE_CERT --store MACHINE_SSL_CERT >/certs/__MACHINE_CERT.key

5. publish the certificate as a new entry called __MACHINE_CERT in the STS_INTERNAL_CERT store:

   # /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --cert /certs/__MACHINE_CERT.crt --key /certs/__MACHINE_CERT.key

6. Initiate the upgrade process again.