Service vpxd-svcs failed to pre-start with AttributeError: 'NoneType' object has no attribute 'serviceId'

Products

• VMware vCenter Server

Issue/Introduction

The vpxd-svcs service failed to pre-start with the log in vmon.log

YYYY-MM-DDTHH:MM:SS Wa(03) host-61220 <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 129, in update_endpoints
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220     is_service_info.serviceId)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220 AttributeError: 'NoneType' object has no attribute 'serviceId'
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220 During handling of the above exception, another exception occurred:
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220 Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 100, in <module>
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220     endpoint_registration_runner(logging_file)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 65, in endpoint_registration_runner
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220     UpdateTaggingServiceGrpcEndpoint(logging_file).run()
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 54, in run
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220     self.update_endpoints()
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 146, in update_endpoints
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220     raise Exception("Tagging grpc reregistration failed while"
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220 Exception: Tagging grpc reregistration failed while executing vpxd-svcs prestart commands
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-61220
YYYY-MM-DDTHH:MM:SS Er(02) host-61220 <vpxd-svcs> Service pre-start command failed with exit code 1

Environment

VMware vCenter Server 7.0.x

VMware vCenter Server 8.0.x

Resolution

Note: before any modification to vCenter server, please do backup or snapshot. 

To resolve the issue

1) Check the PNID of vCenter server with the command:
# /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

Example:
# /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
localhost

In a healthy vCenter server the output could be either FQDN or IP address.
* If vCenter was deployed with FQDN, the PNID is same with FQDN.
* If vCenter was deployed with IP address without DNS resolution, the PNID is the IP address.

In this sample, the PNID output is localhost which neither FQDN nor IP address.
This means that the PNID had been changed. 

2) Check the vCenter server deployment with FQDN or IP, run the command:
# grep -i sso /etc/vmware-vpx/vpxd.cfg

Output:
    <sso>
        <uri>https://xxx.xxx.xxx.xxx/sso-adminserver/sdk/vsphere.local</uri>
        <uri>https://xxx.xxx.xxx.xxx/sso-adminserver/sdk/vsphere.local</uri>
    </sso>

If the uri is FQDN, it was deployed with FQDN. 
If the uri is IP address, it was deployed with IP address. 
In this sample the vCenter was deployed with IP address but somehow the PNID was changed to localhost

2) Manually change the PNID to original with the command:
# /usr/lib/vmware-vmafd/bin/vmafd-cli set-pnid --server-name localhost --pnid <FQDN-or-IP>

Example:
# /usr/lib/vmware-vmafd/bin/vmafd-cli set-pnid --server-name localhost --pnid xxx.xxx.xxx.xxx

3) Stop and start all the services
# service-control --stop --all
# service-control --start --all