Update Manager service failing to start after replacement of the machine SSL certificate with [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)

Products

• VMware vCenter Server

Issue/Introduction

Unable to start the update manager service post the certificate replacement. The issue is seen in the following scenarios

• The vcIntegrity endpoint is updated with the certificate
• Start operation for the VUM service is not returning any error or not updating any log files with error
• When trying yo manually update the certificate, the following error is seen:
  
  

  Command : /usr/lib/vmware-updatemgr/bin/up# su -s /bin/bash updatemgr -c '/usr/lib/vmware-updatemgr/bin/vmware-updatemgr /usr/lib/vmware-updatemgr/bin/vci-integrity.xml'
  
  YYYY-MM-DDT23:33:23.536Z   Getting value for install-parameter: cm.url
  Traceback (most recent call last):
    File "/usr/lib/vmware-updatemgr/bin/updatemgr-utility.py", line 740, in <module>
      main()
    File "/usr/lib/vmware-updatemgr/bin/updatemgr-utility.py", line 735, in main
      ret = args.func(args)
    File "/usr/lib/vmware-updatemgr/bin/updatemgr-utility.py", line 405, in refreshCertificates
      ret = registerWithVc(_args)
    File "/usr/lib/vmware-updatemgr/bin/updatemgr-utility.py", line 464, in registerWithVc
      samlToken = getSamlToken(cmUrl)
    File "/usr/lib/vmware-updatemgr/bin/updatemgr-utility.py", line 142, in getSamlToken
      token_duration=token_duration)
    File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 550, in get_hok_saml_assertion
      private_key)
    File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 234, in perform_request
      webservice.endheaders()
    File "/usr/lib/python2.7/httplib.py", line 1038, in endheaders
      self._send_output(message_body)
    File "/usr/lib/python2.7/httplib.py", line 882, in _send_output
      self.send(msg)
    File "/usr/lib/python2.7/httplib.py", line 844, in send
      self.connect()
    File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 149, in connect
      six.moves.http_client.HTTPSConnection.connect(self)
    File "/usr/lib/python2.7/httplib.py", line 1263, in connect
      server_hostname=server_hostname)
    File "/usr/lib/python2.7/ssl.py", line 387, in wrap_socket
      _context=self)
    File "/usr/lib/python2.7/ssl.py", line 635, in __init__
      self.do_handshake()
    File "/usr/lib/python2.7/ssl.py", line 864, in do_handshake
      self._sslobj.do_handshake()
  
  ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)datemgr-util refresh-certs

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server 6.7.x

Cause

Invalid or stale certificates in the TRUSTED_ROOTS store causes this issue.

Resolution

To resolve the issue,

1. Log in to PSC machine using ssh
2. List all the root certificates from the VECS store
   
   Command: /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store trusted_roots --text | less
   
   
3. Validate the thumbprint from the value under "X509v3 Subject Key Identifier:"
4. Match the following with the valid root certificates and make a note of the Aliases for the stale entries
5. Compare the same using the published certificates using dir-cli
   
   Command: /usr/lib/vmware-vmafd/bin/dir-cli trustedcert list
   
   
6. Take a backup of the certificates from the 'trusted_roots' store to a local location
   
   Command: /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store trusted_roots --alias <stale alias> --output <destination>
   
   
7. Remove the certs from the VECS store
   
   Command: /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store trusted_roots --alias <alias for stale certificate>


8. Unpublish the cert
   
   /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert <cert file> --login administrator@SSO domain --password ''
   
   
9. This process does not require a service restart