vCenter services fail to start after a failed domain repoint operation with the error : Failed to start hvc, vpxd, vapi-endpoint, vpxd-svcs services. Error: Operation timed out

Products

• VMware vCenter Server

Issue/Introduction

After performing a vCenter Server domain repoint, several services including hvc, vpxd, vapi-endpoint, and vpxd-svcs fails to start.

• The vCenter Server UI displays following error: "Error 500 An error occurred while fetching identity providers. Try again"

• Attempting to start all services via service-control --start --all resulted in the below error:

Performing start operation on profile: ALL...
Successfully started service vmware-vmon
Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start hvc, vpxd, vapi-endpoint, vpxd-svcs services. Error: Operation timed out

• Below error message was observed in /var/log/vmware/vmon.log pointing to a pre-start failure of vpxd-svcs service.

YYYY-MM-DDThh:mm:ssZ In(05) host-2459 <vpxd-svcs-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py /storage /var/log
YYYY-MM-DDThh:mm:ssZ Wa(03) host-2459 <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 129, in update_endpoints
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459     is_service_info.serviceId)
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459 AttributeError: 'NoneType' object has no attribute 'serviceId'
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459 During handling of the above exception, another exception occurred:
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459 Traceback (most recent call last):
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 100, in <module>
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459     endpoint_registration_runner(logging_file)
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 65, in endpoint_registration_runner
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459     UpdateTaggingServiceGrpcEndpoint(logging_file).run()
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 54, in run
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459     self.update_endpoints()
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 146, in update_endpoints
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459     raise Exception("Tagging grpc reregistration failed while"
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459 Exception: Tagging grpc reregistration failed while executing vpxd-svcs prestart commands
YYYY-MM-DDThh:mm:ssZ Wa(03)+ host-2459
YYYY-MM-DDThh:mm:ssZ Er(02) host-2459 <vpxd-svcs> Service pre-start command failed with exit code 1.

Environment

vCenter Server 7.X

vCenter Server 8.X

Cause

This issue is due to missing service registrations caused by a failed domain repoint operation.

Resolution

1. Log in to the vCenter Server Appliance (VCSA) via SSH.
   
   
2. Execute the following command to verify if all the solution users are present:
   /usr/lib/vmware-vmafd/bin/dir-cli service list
   
   
3. If solution users are missing, recreate them using the lsdoctor utility:
   python lsdoctor.py -u
   
   
4. Execute the following command to verify if all the service registerations are present:
   /usr/lib/vmware-lookupsvc/tools/lstool.py list --url  https://localhost/lookupservice/sdk --no-check-cert | grep -i "Service Type:" | sort | uniq -c
   
   
5. Rebuild all service registrations if they are missing:
   python lsdoctor.py -r
   
   
6. Restart all vCenter services to apply changes:
   service-control --stop --all && service-control --start --all