Unable to deploy OVA in vCenter with error: "A fatal error has occurred to continue"

Products

• VMware vCenter Server
• VMware Telco Cloud Platform

Issue/Introduction

• Failed to deploy OVA in vCenter. The operation terminates with the error: 'A fatal error has occurred. Unable to continue.'
  

• service-control --status --all on vCenter shows, few services are in stopped state 

  Running:

 lookupsvc lwsmd observability-vapi pschealth vc-ws1a-broker vlcm vmafdd vmcad vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-cis-license vmware-content-library vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-infraprofile vmware-perfcharts vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-stsd vmware-trustmanagement vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vsan-health vmware-vsm vtsdb

Stopped:

 applmgmt observability vmcam vmware-certificatemanagement vmware-hvc vmware-imagebuilder vmware-netdumper vmware-pod vmware-rbd-watchdog vmware-sps vmware-topologysvc vmware-updatemgr vmware-vcha vmware-vdtc vmware-vpxd-svcs vsphere-ui vstats wcp

• Services are not starting on the vCenter appliance

• applmgmt.log shows below entries
  
  DEBUG:vmware.appliance.extensions.authorization.authorization_sso:_system_domain: vsphere.local
DEBUG:vmware.appliance.extensions.authorization.authorization_sso:Looking up SSO endpoint for authorization
ERROR:root:Authorization module (authorization_sso) failed to initialize {[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1007)}

Environment

vCenter 8.x
VMware Telco Cloud Platform 5.0.1

Cause

• vpxd-svcs and other core services could not initialize due to a hostname mismatch within the SSL certificates. Below is the log excerpt from vmon.log file:

Wa(03) host-xxxxxx <vpxd-svcs> Service pre-start command's stderr:     self.do_handshake()
Wa(03)+ host-xxxxxx  File "/usr/lib/python3.10/ssl.py", line 1341, in do_handshake
Wa(03)+ host-xxxxxx
Wa(03) host-xxxxxx <vpxd-svcs> Service pre-start command's stderr:     self._sslobj.do_handshake()
Wa(03)+ host-xxxxxx
Wa(03) host-xxxxxx <vpxd-svcs> Service pre-start command's stderr: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'VCENTER PNID/HOSTNAME'. (_ssl.c:997)
Wa(03)+ host-xxxxxx
Er(02) host-xxxxxx <vpxd-svcs> Service pre-start command failed with exit code 1.

• Expired Machine_SSL & Trusted Root Certificates causing vCenter services to fail

Resolution

1. Update/Replace the Machine SSL certificate: by using the vCert script: KB 385107
2. Validate and remove the expired certificates on vCenter appliance
3. Re-initialize the services: service-control --start --all
4. Re-deploy the OVA.