vCenter upgrade to 7.0 Fails with error "Encountered an internal error. File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py....."

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter upgrade to 7.0 fails during vmidentity firstboot with the error: 

Encountered an internal error.Traceback (most recent call last):  File \"/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py\", line 1449, in main    vmidentityFB.boot()  File \"/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py\", line 343, in boot    self.registerTokenServiceWithLookupService()  File \"/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py\", line 567, in registerTokenServiceWithLookupService    raise e  File \"/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py\", line 563, in registerTokenServiceWithLookupService    dynVars=dynVars)  File \"/usr/lib/vmware-cm/bin/cloudvmcisreg.py\", line 714, in cloudvm_sso_cm_register    serviceId = do_lsauthz_operation(cisreg_opts_dict)  File \"/usr/lib/vmware/site-packages/cis/cisreglib.py\", line 1118, in do_lsauthz_operation    ls_obj.register_service(svc_id, svc_create_spec)  File \"/usr/lib/vmware/site-packages/cis/cisreglib.py\", line 348, in add_securityctx_to_requests    return req_method(self, *args, **kargs)  File \"/usr/lib/vmware/site-packages/cis/cisreglib.py\", line 360, in register_service    svc_create_spec)  File \"/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py\", line 595, in <lambda>    self.f(*(self.args + (obj,) + args), **kwargs)  File \"/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py\", line 385, in _InvokeMethod    return self._stub.InvokeMethod(self, info, args)  File \"/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py\", line 1570, in InvokeMethod    raise obj  # pylint: disable-msg=E0702pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {   dynamicType = <unset>,   dynamicProperty = (vmodl.DynamicProperty) [],   msg = 'LookupFaultServiceFault',   faultCause = <unset>,   faultMessage = (vmodl.LocalizableMessage) [],   reason = 'Invalid fault'}
 

•  In the /var/log/firstboot/vmidentity-firstboot.py_11661_stdout.log you may find entries similar to:

2022-11-05T02:42:45.849Z  VMware Identity Service bootstrap failed.
2022-11-05T02:42:45.850Z  Exception: Traceback (most recent call last):
  File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 1449, in main
     raise obj  # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'LookupFaultServiceFault',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   reason = 'Invalid fault'
 

• In the var/log/vmware/lookupsvc/lookupserver-default.log you may find entries similar to:

[2022-11-05T02:42:45.839Z pool-2-thread-5 ERROR com.vmware.vim.lookup.vlsi.util.VmodlEnhancer] com.vmware.vim.vmomi.core.exception.CertificateValidationException: SSL handshake from 0.0.0.0/0.0.0.0:39030 to vCenterfqdn/127.0.0.1:443 failed in 467 ms
com.vmware.vim.sso.admin.exception.CertificateValidationException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: SSL handshake from 0.0.0.0/0.0.0.0:39030 to vCenterfqdn/127.0.0.1:443 failed in 467 ms
Caused by: java.security.cert.CertificateException: malformed PEM data encountered
        at org.bouncycastle.jcajce.provider.CertificateFactory.readCertificate(Unknown Source) ~[bc-fips-1.0.2.1.jar:1.0.2.1]
Caused by: java.io.IOException: malformed PEM data encountered
        at org.bouncycastle.jcajce.provider.PEMUtil.readPEMObject(Unknown Source) ~[bc-fips-1.0.2.1.jar:1.0.2.1]

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 7.0.x

Cause

• This issue occurs if there are invalid certificates in the VMware Endpoint Certificate Store(VECS) store.
• Non CA certificates in the TRUSTED_ROOTS store are considered invalid.

Resolution

To resolve this issue, follow the below steps:

• List the certificates from TRUSTED_ROOTS store using vecs-cli.  

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less

• Identify the certificates which have the CA bit value set to FALSE under X509v3 Basic Constraints

For example:
X509v3 Basic Constraints: critical
                CA:FALSE

Note: Valid CA Certificates will have "CA:TRUE" under basic Constraints.

• Remove the Non CA certificates from TRUSTED_ROOTS store by referring to the procedure in Removing Expired CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS)

Note: Before removing the certificates, ensure the certificate is not being used by any solution.

Additional Information

Impact/Risks:

• Proceed with caution. Removing a wrong certificate from the VECS store will damage the environment. This can be irreparable.
• Take offline snapshots of all the nodes(PSC and vCenter) in the SSO domain before attempting to remove the certificate.
• In case of reverting snapshots, all the nodes need to be reverted to ensure data consistency.