vCenter server is inaccessible using the UI client and multiple services failing to start with error: Failed to start hvc, vpxd, vapi-endpoint, vpxd-svcs services. Error: Operation timed out
search cancel

vCenter server is inaccessible using the UI client and multiple services failing to start with error: Failed to start hvc, vpxd, vapi-endpoint, vpxd-svcs services. Error: Operation timed out

book

Article ID: 378473

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

  • Accessing vCenter Server Appliance (VCSA) using UI may return "Error 500 An error occurred while fetching identity providers. Try again"


  • Attempting to start the service manually may result with the error as "Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start hvc, vpxd, vapi-endpoint, vpxd-svcs services. Error: Operation timed out"

#root@vcsa: service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on profile: ALL...
Successfully started service vmware-vmon
Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start hvc, vpxd, vapi-endpoint, vpxd-svcs services. Error: Operation timed out

  • /var/log/vmware/vmon/vmon.log:

YYYY-MM-DDTHH:MM:SS Wa(03) host-xxxx <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 116, in update_endpoints
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx     ls_obj.reregister_service(service_info.serviceId, mutable_spec)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 349, in add_securityctx_to_requests

YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx pyVim.sso.SoapException: SoapException:
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx faultcode: ns0:FailedAuthentication
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx faultstring: Invalid credentials
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-xxxx faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05
/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x

Cause

This issue is caused due to the certificates being expired on the vCenter server

Resolution

Note: Ensure to take a powered off snapshot on VCSA prior to proceeding with the steps below. In case of Enhanced Linked Mode (ELM), ensure to take a powered off snapshot on all the nodes in ELM.

Use the new improved certificate management tool vCert - Scripted vCenter Expired Certificate Replacement for all certificate management/replacement workflow. The KB has detailed steps to replace the expired certificates. 

  • Once the vCert.py is installed on the VCSA, execute the command ./vCert.py
  • Enter Option 3 - Manage Certificates which lists all the certificates. 
  • Select the certificate that is found expired. 
  • Replace the certificate by providing the SSO admin username and password when prompted. 
  • Restart the services when prompted. 

Additional Information

Powered by Wolken Software