vCenter server is inaccessible using the UI client and multiple services failing to start with error: Failed to start hvc, vpxd, vapi-endpoint, vpxd-svcs services. Error: Operation timed out

Products

• VMware vCenter Server
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• Accessing vCenter Server Appliance (VCSA) using UI may return "Error 500 An error occurred while fetching identity providers. Try again"

• Attempting to start the service manually may result with the error as "Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start hvc, vpxd, vapi-endpoint, vpxd-svcs services. Error: Operation timed out"

#root@vcsa: service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on profile: ALL...
Successfully started service vmware-vmon
Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start hvc, vpxd, vapi-endpoint, vpxd-svcs services. Error: Operation timed out

• /var/log/vmware/vmon/vmon.log:

YYYY-MM-DDTHH:MM:SS Wa(03) host-xxxx <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 116, in update_endpoints
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx     ls_obj.reregister_service(service_info.serviceId, mutable_spec)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 349, in add_securityctx_to_requests

YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx pyVim.sso.SoapException: SoapException:
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx faultcode: ns0:FailedAuthentication
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx faultstring: Invalid credentials
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05
/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>

Cause

This issue is caused due to the solution user certificate being expired on the vCenter server

Resolution

Note: Ensure to take a powered off snapshot on VCSA prior to proceeding with the steps below. In case of Enhanced Linked Mode (ELM), ensure to take a powered off snapshot on all the nodes in ELM.

In order to resolve the issue, proceed with the steps below:

• Download the fixcerts.py script, refer to Replace certificates on vCenter server using the Fixcerts script
• Transfer the downloaded script to the VCSA node using SCP. In case of upload failure, Refer to Connecting to vCenter Server Virtual Appliance using WinSCP fails
• Log in to the VCSA using ssh
• Navigate to the script location
• Execute the below command to renew the solution user certificates

python fixcerts_3_2.py replace --certType solutionusers

Note: The script will provide the option to restart the services on the VCSA

• Post the service restart is complete, proceed to log in to VCSA using UI