This issue is resolved in vCenter Server Appliance 6.7 U3g, available at
VMware Downloads .
Workaround:
Workaround 1:To work around the issue, un-publish and re-publish certs from trusted roots store so that the machine SSL signing cert is first entry in trusted store.
- Create a snapshot of vCenter Server.
- Identify Alias of Machine SSL signing cert:
- List the certificates stored in trusted_roots and look for keyword "Alias"
“C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli entry list --store TRUSTED_ROOTS --text | findstr Alias
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep Alias
- Identify the machine SSL signing cert alias in the trusted roots store [Alias : __MACHINE_CERT]
- Take backup of all certs that are available in trusted roots store.
- Use the below commands to backup certs in trusted root store:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <certificate alias> --output /storage/core/<alias.crt>
"C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli entry getcert --store TRUSTED_ROOTS --alias <certificate alias>--output c:\certificates\<alias>.crt - Repeat the above step for all the certs available in trusted store.
- Unpublish the certificate.
- Use dir-cli to unpublish for each cert file:
C:\Program Files\VMware\vCenter Server\vmafdd\dir-cli.exe trustedcert unpublish --cert <cert#.crt> --login administrator@<SSO Domain> --password <passwd>
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert <cert#.crt> --login administrator@<SSO Domain> --password <passwd>
- Re-publish certs and make sure signing certificate of the Machine SSL is the first certificate published to trusted roots store.
- Use dir-cli to re-publish for each cert file:
C:\Program Files\VMware\vCenter Server\vmafdd> dir-cli.exe trustedcert publish --cert <C:\Trusted_certs\To be published\cert#.crt> --login administrator@<SSO Domain> --password <passwd>
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert </path_to_cer_file/cert#.crt> --login administrator@<SSO Domain> --password <passwd>
- Force VECS to refresh after republishing certificate to see changes using below command,
/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
Additional reference KB Article:
Firstboot Failed during Install/Deployment, Upgrade or Migration in vCenter Server and vCenter Server Appliance 6.7 (55746)For example:Before changes:
vecs-cli.exe entry list --store TRUSTED_ROOTS --text | findstr Alias
Alias : 31393071fac886b194715db3f716f16ad298967f
Alias : ecfba4d095d7aa0678acaae2e205ae84baa7620f
Alias : eeccd9edd4f069c8e8baedb37ba0457062fe4b05
Alias : c8a0707d9240aefc88702949172292578aaf9a56
Alias : 67ed5600354da88909ccda23ad133648313702f2 --> Machine ssl signing cert
Alias : b0e38fe338fd287e8f978cf2d060460a20572688
Alias : 1b01800b62f827005480f06baa34497c60193e65
Alias : 356de6d9badb26bcd18cfb6a091713b8e3d0f5ce
Alias : 24c4e596e5db7dd7db01799a77b649cd405b0363
Alias : 3058753e77ed23b3070e9fafb882cc74ef392bd2
Alias : c7fc47982ce7f2e3b46b6aeb188dd4a9aff65693
After Changes:
Published signing certificate of the Machine SSL to be first entry in trusted root store, also published intermediate and root cert.
vecs-cli.exe entry list --store TRUSTED_ROOTS --text | findstr Alias
Alias : 67ed5600354da88909ccda23ad133648313702f2 --> Machine ssl signing cert
Alias : 75a28c785a3b6cb4a74827a67d6a698609ec9926 --> Intermediate cert
Alias : 47e332b21bafbefa11e8ee2c334b9ac440922ff1 --> Root cert
Alias : 31393071fac886b194715db3f716f16ad298967f
Alias : ecfba4d095d7aa0678acaae2e205ae84baa7620f
Alias : eeccd9edd4f069c8e8baedb37ba0457062fe4b05
Alias : c8a0707d9240aefc88702949172292578aaf9a56
Alias : b0e38fe338fd287e8f978cf2d060460a20572688
Alias : 1b01800b62f827005480f06baa34497c60193e65
Alias : 356de6d9badb26bcd18cfb6a091713b8e3d0f5ce
Alias : 24c4e596e5db7dd7db01799a77b649cd405b0363
Alias : 3058753e77ed23b3070e9fafb882cc74ef392bd2
Alias : c7fc47982ce7f2e3b46b6aeb188dd4a9aff65693
- Retry the upgrade/migrate process.
Workaroud 2:
To reset the certificates to VMware default certs, you can follow the below steps to reset the certificates to VMware Default certs and make sure if the VMCA root certificate is in first position in the TRUSTED_ROOTS store.
- Launch the vSphere 6.x Certificate Manager.
- For vCenter Server 6.x Appliance:
/usr/lib/vmware-vmca/bin/certificate-manager - For Windows vCenter Server 6.x:
C:\Program Files\VMware\vCenter Server\vmcad\certificate-manager
- Select Option 4 (Regenerate a new VMCA Root Certificate and replace all certificates)
- Once the certificates are replaced verify if the VMCA Root certificate is in first position in TRUSTED_ROOTS store.
- Use the below command to list the certificates in TRUSTED_ROOTS store
- /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less
- Retry the upgrade/migrate process.