'Analytics Service registration with Component Manager failed' 'SSL CERTIFICATE _VERIFY_FAILED' while upgrading vCenter server from 6.x to 6.7 firstboot fails

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter Upgrade from 6.x to 6.7 failed while registering Analytics Service with Component Manager due to cert validation failure.
• Issue can also be caused while migrating a vCenter server to 6.7 with below error

analytics_firstboot.py_xxxx_stderr.log: 

2019-02-12T13:36:10.042Z  Failed to register Analytics Service with Component Manager: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:719)
2019-02-12T13:36:10.045Z  Traceback (most recent call last):
  File "/usr/lib/vmware-analytics/firstboot/analytics_firstboot.py", line 181, in register_with_cm
    cloudvm_sso_cm_register(keystore, cisreg_spec, key_alias, dyn_vars, isPatch=is_patch)
  File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 706, in cloudvm_sso_cm_register
    serviceId = do_lsauthz_operation(cisreg_opts_dict)
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 997, in do_lsauthz_operation
    ls_obj = LookupServiceClient(ls_url, retry_count=60)
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 307, in __init__
    self._init_service_content()
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 287, in do_retry
    return req_method(self, *args, **kargs)
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 297, in _init_service_content
    self.service_content = si.RetrieveServiceContent()
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 557, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 363, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1385, in InvokeMethod conn.request('POST', self.path, req, headers)
  File "/usr/lib/python3.5/http/client.py", line 1107, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1032, in connect
    six.moves.http_client.HTTPSConnection.connect(self)
  File "/usr/lib/python3.5/http/client.py", line 1261, in connect
    server_hostname=server_hostname)
  File "/usr/lib/python3.5/ssl.py", line 385, in wrap_socket
    _context=self)
  File "/usr/lib/python3.5/ssl.py", line 760, in __init__
    self.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:719)

• During handling of the above exception, another exception occurred

Traceback (most recent call last):
  File "/usr/lib/vmware-analytics/firstboot/analytics_firstboot.py", line 288, in main
    fb.register_with_cm(analytics_int_http, is_patch)
  File "/usr/lib/vmware-analytics/firstboot/analytics_firstboot.py", line 192, in register_with_cm
    problem_id='install.analytics.cmregistration.failed')
cis.baseCISException.BaseInstallException: {
    "componentKey": "analytics",
    "problemId": "install.analytics.cmregistration.failed",
    "detail": [
        {
            "translatable": "Analytics Service registration with Component Manager failed.",
            "localized": "Analytics Service registration with Component Manager failed.",
            "id": "install.analytics.cmregistration.failed"
        }
    ],
    "resolution": {
        "translatable": "Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.",
        "localized": "Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.",
        "id": "install.analytics.cmregistration.failed.res"
    }
}

2019-02-12T13:36:10.045Z  VMware Analytics Service firstboot failed

Environment

VMware vCenter Server 6.0.x
VMware vCenter Server 6.5.x
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.7.x
VMware vCenter Server Appliance 6.7.x

Cause

The issue is caused:

• When the machine SSL cert chain is not validated.
• If the root certificate of the Issuing authority of the machine ssl certificate is not available in the TRUSTED_ROOTS store.
• In case of custom certificate the entire chain of certificate (intermediate CA as well as the root CA) should be available in the TRUSTED_ROOTS store.

Resolution

To resolve the issue:

1. Publish the missing certificate to the TRUSTED_ROOTS store.
   • VCSA : /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert <path_of_the_cert>
   • Windows : "%VMWARE_CIS_HOME%"\vmafdd\dir-cli trustedcert publish --cert <path_of_the_cert>
2. If the VC is upgraded from 5.x environment then it may have machine ssl cert issued VMware Installer, in such situation regenerate the machine ssl certificate as it would not be possible to get hold of the Vmware Installer certificate. For more details, refer to Replacing the vSphere 6.0 Machine SSL certificate with a VMware Certificate Authority issued certificate (2112279)

Note : Re-generating the machine SSL certificate would be helpful in all scenarios.