"Cannot retrieve service content from vCenter Server localhost:80", EAM firstboot fails when migrating from Windows vCenter Server 6.0 to vCenter Server Appliance 6.7

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• EAM firstboot fails when migrating from Windows vCenter Server 6.0 to VMware vCenter Server Appliance 6.7

In eam_firstboot.py_10578_stderr.log, you see entries similar to:

INFO:root:Found 1 matching service. ID is e5e50715-####-####-####-########aef
2018-10-31T14:25:46.795Z Error during EAM extension registration in VC.
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:719)
2018-10-31T14:25:46.806Z Traceback (most recent call last):
  File "/usr/lib/vmware-eam/firstboot/eaminstall/extension/vCenter.py", line 217, in _retrieveServiceContent
    self.siC = si.RetrieveContent()
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 557, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
.
.
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:719)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
    .
  File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake
    self._sslobj.do_handshake()
cis.baseCISException.BaseInstallException: {
    "problemId": null,
    "componentKey": "eam",
    "resolution": {
        "translatable": "Search for these symptoms in the VMware knowledge base for any known issues and possible workarounds. If none can be found, collect a support bundle and open a support request.",
        "id": "install.eam.fileSR.resolution",
        "localized": "Search for these symptoms in the VMware knowledge base for any known issues and possible workarounds. If none can be found, collect a support bundle and open a support request."
    },
    "detail": [
        {
            "translatable": "Cannot retrieve service content from vCenter Server %(0)s:%(1)s.",
            "args": [
                "localhost",
                80
            ],
            "id": "install.eam.error.extension.service_content",
            "localized": "Cannot retrieve service content from vCenter Server localhost:80."
        },
        {
            "translatable": "Unable to register ESX Agent Manager as an extension in vCenter Server.",
            "args": [],
            "id": "install.eam.error.extension",
            "localized": "Unable to register ESX Agent Manager as an extension in vCenter Server."
        }
    ]
}

2018-10-31T14:25:46.806Z  VMware ESX Agent Manager firstboot failed.

Or 

2019-03-18T12:08:43.352Z Error during EAM extension registration in VC.[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)
2019-03-18T12:08:43.361Z Traceback (most recent call last):File "/usr/lib/vmware-eam/firstboot/eam_firstboot.py", line 355, in main fb.registerExtension()

• In eam_firstboot.py_10578_stdout.log, you see entries similar to: 
  
  

While Retrieving machine ssl cert:

2018-10-31T14:25:46.613Z ESX Agent Manager service installed
2018-10-31 14:25:46.613495 Retrieving Vpxd certificate.
2018-10-31T14:25:46.613Z Running command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getcert', '--store', 'MACHINE_SSL_CERT', '--alias', '__MACHINE_CERT', '--output', '/var/tmp/vmware/tmp03e4hcol']
2018-10-31T14:25:46.634Z Done running command
2018-10-31 14:25:46.635274 Retrieving Vpxd private key.
2018-10-31T14:25:46.635Z Running command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getkey', '--store', 'MACHINE_SSL_CERT', '--alias', '__MACHINE_CERT', '--output', '/var/tmp/vmware/tmpcci2pk6e']
2018-10-31T14:25:46.660Z Done running command

While retrieving signing certificate of the Machine SSL:

2018-10-31 14:25:46.680541 Retrieving CA certificate.
2018-10-31T14:25:46.680Z Running command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'list', '--store', 'TRUSTED_ROOTS']
2018-10-31T14:25:46.725Z Done running command
2018-10-31T14:25:46.726Z Running command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getcert', '--store', 'TRUSTED_ROOTS', '--alias', '31393071fac886b194715db3f716f16ad298967f', '--output', '/var/tmp/vmware/tmple3kevq7']
2018-10-31T14:25:46.747Z Done running command
2018-10-31 14:25:46.747608 Registering EAM service as a VC extension.

• As per the below code firstboot scripts is only looking (or expecting) the first entry in trusted root store to be signing certificate of the Machine SSL and if it is not migrate fails with eam firstboot.

 

/usr/lib/vmware-eam/firstboot/eaminstall/vecs.py:

self._vecsCAEntry.get_cert(caAliases[0], output_file)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server 6.0.x
VMware vCenter Server 6.7.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server Appliance 6.5.x

Resolution

This issue is resolved in vCenter Server Appliance 6.7 U3g, available at VMware Downloads .

Workaround:
Workaround 1:
To work around the issue, un-publish and re-publish certs from trusted roots store so that the machine SSL signing cert is first entry in trusted store.

1. Create a snapshot of vCenter Server.
2. Identify Alias of Machine SSL signing cert:
   • List the certificates stored in trusted_roots and look for keyword "Alias"
     
     “C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli entry list --store TRUSTED_ROOTS --text | findstr Alias
     /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep Alias
      
   • Identify the machine SSL signing cert alias in the trusted roots store [Alias : __MACHINE_CERT]
3. Take backup of all certs that are available in trusted roots store.
   • Use the below commands to backup certs in trusted root store:
     
     /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <certificate alias> --output /storage/core/<alias.crt>
     "C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli entry getcert --store TRUSTED_ROOTS --alias <certificate alias>--output c:\certificates\<alias>.crt
   • Repeat the above step for all the certs available in trusted store.
4. Unpublish the certificate.
   • Use dir-cli to unpublish for each cert file:
     
     C:\Program Files\VMware\vCenter Server\vmafdd\dir-cli.exe trustedcert unpublish --cert <cert#.crt> --login administrator@<SSO Domain> --password <passwd>
     /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert <cert#.crt> --login administrator@<SSO Domain> --password <passwd>
      
5. Re-publish certs and make sure signing certificate of the Machine SSL is the first certificate published to trusted roots store.
   • Use dir-cli to re-publish for each cert file:
     
     C:\Program Files\VMware\vCenter Server\vmafdd> dir-cli.exe trustedcert publish --cert <C:\Trusted_certs\To be published\cert#.crt> --login administrator@<SSO Domain> --password <passwd>
     /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert </path_to_cer_file/cert#.crt> --login administrator@<SSO Domain> --password <passwd>
6. Force VECS to refresh after republishing certificate to see changes using below command,

 /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

Additional reference KB Article: Firstboot Failed during Install/Deployment, Upgrade or Migration in vCenter Server and vCenter Server Appliance 6.7 (55746)

For example:

Before changes:
vecs-cli.exe entry list --store TRUSTED_ROOTS --text | findstr Alias

Alias : 31393071fac886b194715db3f716f16ad298967f
Alias : ecfba4d095d7aa0678acaae2e205ae84baa7620f
Alias : eeccd9edd4f069c8e8baedb37ba0457062fe4b05
Alias : c8a0707d9240aefc88702949172292578aaf9a56
Alias : 67ed5600354da88909ccda23ad133648313702f2 --> Machine ssl signing cert
Alias : b0e38fe338fd287e8f978cf2d060460a20572688
Alias : 1b01800b62f827005480f06baa34497c60193e65
Alias : 356de6d9badb26bcd18cfb6a091713b8e3d0f5ce
Alias : 24c4e596e5db7dd7db01799a77b649cd405b0363
Alias : 3058753e77ed23b3070e9fafb882cc74ef392bd2
Alias : c7fc47982ce7f2e3b46b6aeb188dd4a9aff65693

After Changes:

Published signing certificate of the Machine SSL to be first entry in trusted root store, also published intermediate and root cert.

vecs-cli.exe entry list --store TRUSTED_ROOTS --text | findstr Alias
Alias : 67ed5600354da88909ccda23ad133648313702f2 --> Machine ssl signing cert
Alias : 75a28c785a3b6cb4a74827a67d6a698609ec9926 --> Intermediate cert
Alias : 47e332b21bafbefa11e8ee2c334b9ac440922ff1 --> Root cert
Alias : 31393071fac886b194715db3f716f16ad298967f
Alias : ecfba4d095d7aa0678acaae2e205ae84baa7620f
Alias : eeccd9edd4f069c8e8baedb37ba0457062fe4b05
Alias : c8a0707d9240aefc88702949172292578aaf9a56
Alias : b0e38fe338fd287e8f978cf2d060460a20572688
Alias : 1b01800b62f827005480f06baa34497c60193e65
Alias : 356de6d9badb26bcd18cfb6a091713b8e3d0f5ce
Alias : 24c4e596e5db7dd7db01799a77b649cd405b0363
Alias : 3058753e77ed23b3070e9fafb882cc74ef392bd2
Alias : c7fc47982ce7f2e3b46b6aeb188dd4a9aff65693

7. Retry the upgrade/migrate process.

Workaroud 2:

To reset the certificates to VMware default certs, you can follow the below steps to reset the certificates to VMware Default certs and make sure if the VMCA root certificate is in first position in the TRUSTED_ROOTS store.

1. Launch the vSphere 6.x Certificate Manager.
   • For vCenter Server 6.x Appliance:
         /usr/lib/vmware-vmca/bin/certificate-manager
   • For Windows vCenter Server 6.x:
         C:\Program Files\VMware\vCenter Server\vmcad\certificate-manager
2. Select Option 4 (Regenerate a new VMCA Root Certificate and replace all certificates) 
3. Once the certificates are replaced verify if the VMCA Root certificate is in first position in TRUSTED_ROOTS store.
4. Use the below command to list the certificates in TRUSTED_ROOTS store
   • /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less
5. Retry the upgrade/migrate process.