vCenter server is inaccessible from the vSphere client and multiple services fail to start with error: Failed to start hvc, vpxd, vapi-endpoint, vpxd-svcs services. Error: Operation timed out
Article ID: 378473
Updated On:
Products
Issue/Introduction
- Accessing vCenter Server Appliance (VCSA) using UI may return the following error: "
Error 500 An error occurred while fetching identity providers. Try again"
OR Accessing vCenter Server Appliance (VCSA) using UI may return "Error An error occurred while fetching identity providers. Try again" - Attempting to start the services manually fails with the following error:
"Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start hvc, vpxd, vapi-endpoint, vpxd-svcs services. Error: Operation timed out"
#root@vcsa: service-control --start --allOperation not cancellable. Please wait for it to finish...Performing start operation on service lwsmd...Successfully started service lwsmdPerforming start operation on service vmafdd...Successfully started service vmafddPerforming start operation on service vmdird...Successfully started service vmdirdPerforming start operation on service vmcad...Successfully started service vmcadPerforming start operation on profile: ALL...Successfully started service vmware-vmonService-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start hvc, vpxd, vapi-endpoint, vpxd-svcs services. Error: Operation timed out
- The vmon log (/var/log/vmware/vmon/vmon.log) includes warnings regarding authentication failures:
YYYY-MM-DDTHH:MM:SS Wa(03) host-xxxx <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 116, in update_endpointsYYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx ls_obj.reregister_service(service_info.serviceId, mutable_spec)YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 349, in add_securityctx_to_requests
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx pyVim.sso.SoapException: SoapException:YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx faultcode: ns0:FailedAuthenticationYYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx faultstring: Invalid credentialsYYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>
Environment
VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x
Cause
This issue is caused by expired certificates on the vCenter server.
Resolution
Note: Make sure to take a snapshot of the vCenter Server Appliance prior to proceeding with the steps below. If the vCenter Server is in Enhanced Linked Mode (ELM), make sure to take a powered off snapshot of all the nodes in ELM.
Use the new and improved certificate management tool vCert - Scripted vCenter Expired Certificate Replacement for all certificate management/replacement workflows. The linked KB has detailed steps on how to replace the expired certificates.
- Once the vCert tool is installed on the VCSA, execute the following command: python vCert.py
- Use Option 1 - Check current certificate status to list all certificates and their expiration status.
- Enter Option 3 - Manage Certificates which lists all of the certificates that can be managed with the tool.
- Select the expired certificate from the list.
- Replace the certificate by providing the SSO admin username and password when prompted.
- Restart the services when prompted.
Additional Information
Determining expired SSL certificates in vCenter Server: https://knowledge.broadcom.com/external/article/343041/determining-expired-ssl-certificates-in.html
Feedback
