vCenter server Update manager service fails to start

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• Update manager service is stopped and fails to start when done manually. 
• Lifecycle Manager fails to load on the vCenter UI. 
• vCenter patching update may fail to complete. 
• /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log in vCenter logs contains entries similar to:
  
  [<YYYY-MM-DD>T<time> info vmware-vum-server[] [Originator@6876 sub=VcIntegrity] Connecting to host <VCSA FQDN> on port 80 using protocol http
-->
[<YYYY-MM-DD>T<time>  warning vmware-vum-server[08722] [Originator@6876 sub=IO.Connection] Failed to SSL handshake; SSL(<io_obj p:0x00007f3c20a19e18, h:33, <TCP 'x.x.x.x: 55160'>, <TCP 'y.y.y.y: 80'>>), e: nnnnnn(certificate verify failed), duration: 24msec
[<YYYY-MM-DD>T<time>  warning vmware-vum-server[08722] [Originator@6876 sub=HttpConnectionPool-000000] Failed to get pooled connection; <cs p:00007f3c28272c90, TCP:<VCSA FQDN>:80>, SSL(<io_obj p:0x00007f3c20a19e18, h:33, <TCP 'x.x.x.x: 55160'>, <TCP 'y.y.y.y: 80'>>), duration: 68msec, N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
--> PeerThumbprint: x:x:x:x:x:x:x:x:x:x:x:x:x:x:x:x:x:x:x:x
--> ExpectedThumbprint: x:x:x:x:x:x:x:x:x:x:x:x:x:x:x:x:x:x:x:x
--> ExpectedPeerName: <VCSA FQDN>
--> The remote host certificate has these problems:
-->
--> * self signed certificate in certificate chain)
  
  
  
• The actual hostname/FQDN of the vCenter might differ from the FQDN reported in the above log entries. 

Environment

VMware vCenter Server 7.0.x

VMware vCenter Server 8.0.x

Cause

This issue is commonly caused by a custom SSL certificate that has been renewed on the vCenter Server Appliance (VCSA) without properly adding the root and/or intermediate certificates to the trusted store. When the vCenter Server attempts to validate the SSL/TLS handshake with the Update Manager, it fails due to missing or untrusted certificates.

 

Resolution

Note: Ensure there is valid backup/offline snapshot of the VCSA prior to implementing the workaround. Refer to VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice -  KB 313886

 

Step 1: Add the missing Root/Intermediate root Certificate to the Trusted Store

1. Copy the missing root/intermediate root certificate (Intermediatecert.crt) to the /tmp folder.

2. Publish the new intermediate certificate to the vCenter Server:

   /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --chain --cert /tmp/Intermediatecert.crt
   
   

3. Force a refresh of the certificate store again:

   /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh


Step 2: Re-register the Update Manager Service

To ensure the Update Manager is using the new certificates, you need to re-register the service. This can be done by running the following command:

/usr/lib/vmware-updatemgr/bin/refreshCerts-utility.py

Step 3: Restart vCenter Server Services

service-control --stop --all && service-control --start --all

Step 4 (Optional): Remove the Unused Certificate from the Trusted Store

1. Log into the vCenter Server Appliance using an SSH client.

2. Run the following command to get the old certificate:

   /usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id <CERT_ID> --login [email protected] --outcert /tmp/oldcert.cer


    

3. Unpublish the old, unused certificate from the trusted store:

   /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /tmp/oldcert.cer

    

4. Force a refresh of the certificate store:

   /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh


    

5. Verify that the certificate has been removed:

   /usr/lib/vmware-vmafd/bin/dir-cli trustedcert list
    

For more information on guide to unpublished certificates from Trusted store, refer this KB article 

Additional Information

The same steps apply in case of missing Root certificate as well.