"An error occurred while starting service 'vpxd-svcs'" | 7.X | Failed to reregister Tagging service grpc endpoints with Lookup Service

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

/var/log/vmware/vpxd-svcs/pre-start-vpxd-svcs.log :
 
ERROR:tagging_grpc_registration:Failed to reregister Tagging service grpc endpoints with Lookup Service

ERROR:tagging_grpc_registration:generator didn't stop after throw()
Traceback (most recent call last):
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 260, in securityctx_modifier
    yield
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests
    return req_method(self, *args, **kargs)
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 364, in reregister_service
    self.service_content.serviceRegistration.Set(svc_id, svc_set_spec)
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 595, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 385, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod
    raise obj  # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.vmodl.fault.SecurityError: (vmodl.fault.SecurityError) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = '',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) []
}
During handling of the above exception, another exception occurred:

vmon.log : 

XXXX-XX-XXT08:49:46.978Z In(05) host-2111 Received start request for vpxd-svcs
XXXX-XX-XXT08:49:46.978Z In(05) host-2111 <vpxd-svcs-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py /storage /var/log
XXXX-XX-XXT08:49:47.936Z Wa(03) host-2111 <vpxd-svcs> Service pre-start command's stderr: XXXX-XX-XXT08:49:47.936Z  Security error: (vmodl.fault.SecurityError) {
XXXX-XX-XXT08:49:47.936Z Wa(03)+ host-2111    dynamicType = <unset>,
XXXX-XX-XXT08:49:47.936Z Wa(03)+ host-2111    dynamicProperty = (vmodl.DynamicProperty) [],
XXXX-XX-XXT08:49:47.936Z Wa(03)+ host-2111    msg = '',
XXXX-XX-XXT08:49:47.936Z Wa(03)+ host-2111    faultCause = <unset>,
XXXX-XX-XXT08:49:47.936Z Wa(03)+ host-2111    faultMessage = (vmodl.LocalizableMessage) []
XXXX-XX-XXT08:49:47.936Z Wa(03)+ host-2111 }
XXXX-XX-XXT08:49:47.936Z Wa(03)+ host-2111
XXXX-XX-XXT08:49:47.985Z Wa(03) host-2111 <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 260, in securityctx_modifier
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111     yield
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111     return req_method(self, *args, **kargs)
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 364, in reregister_service
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111     self.service_content.serviceRegistration.Set(svc_id, svc_set_spec)
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 595, in <lambda>
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111     self.f(*(self.args + (obj,) + args), **kwargs)
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 385, in _InvokeMethod
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111     return self._stub.InvokeMethod(self, info, args)
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111   File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111     raise obj  # pylint: disable-msg=E0702
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 pyVmomi.VmomiSupport.SecurityError: (vmodl.fault.SecurityError) {
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111    dynamicType = <unset>,
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111    dynamicProperty = (vmodl.DynamicProperty) [],
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111    msg = '',
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111    faultCause = <unset>,
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111    faultMessage = (vmodl.LocalizableMessage) []
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 }
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 During handling of the above exception, another exception occurred:
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 Traceback (most recent call last):
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 119, in update_endpoints
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111     ls_obj.reregister_service(service_info.serviceId, mutable_spec)
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111     return req_method(self, *args, **kargs)
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111   File "/usr/lib/python3.7/contextlib.py", line 161, in __exit__
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111     raise RuntimeError("generator didn't stop after throw()")
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 RuntimeError: generator didn't stop after throw()
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 During handling of the above exception, another exception occurred:
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 Traceback (most recent call last):
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 100, in <module>
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111     endpoint_registration_runner(logging_file)
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 65, in endpoint_registration_runner
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111     UpdateTaggingServiceGrpcEndpoint(logging_file).run()
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 54, in run
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111     self.update_endpoints()
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 146, in update_endpoints
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111     raise Exception("Tagging grpc reregistration failed while"
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 Exception: Tagging grpc reregistration failed while executing vpxd-svcs prestart commands
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111
XXXX-XX-XXT08:49:48.042Z Er(02) host-2111 <vpxd-svcs> Service pre-start command failed with exit code 1.
 

 

Environment

VMware vCenter Server 7.0.x

Cause

This issue occurs when machine information is missing from Builtin - Administrators. 

Resolution

To resolve this issue : 
 

Identify the machine ID :
 

/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

Or 

cat /etc/vmware/install-defaults/sca.hostid
 

In this example : SSO Domain is vsphere.local (change as per customer environment)
 

root@XXXXX [ / ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

56095117-1936-49c7-b7cb-341512a6b9bd

cat /etc/vmware/install-defaults/sca.hostid

56095117-1936-49c7-b7cb-341512a6b9bd
 

Next we need to check if the machine account is missing from Builtin - Administrator. To check run,
 

ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=Administrators,cn=Builtin,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSOPWD' member | less | grep machine
 

Output : 

root@XXXXX[ / ]# ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=Administrators,cn=Builtin,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'VMware@123' member | less | grep machine

member: CN=machine-f6c5c33e-3280-433c-a3c7-a9839f2b1f1e,CN=ServicePrincipals,DC=vsphere,DC=local

As we can see machine-56095117-1936-49c7-b7cb-341512a6b9bd is missing and we should add the entry. 
 

To do that we will create LDIF file with the contents and add the same using ldapadd command. 

cd /var/tmp

vi add.ldif and paste the below contents. 
 

dn: cn=Administrators,cn=Builtin,dc=vsphere,dc=local
changetype: modify
add: member
member: CN=machine-MISSINGMACHINEID,CN=ServicePrincipals,DC=vsphere,DC=local
 

Save the file and quit. 
 

Next run :
 

/opt/likewise/bin/ldapadd -h localhost -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w 'SSOPWD' -f add.ldif 
 

Verify using ldapsearch command and make sure the entry is added properly. 
 

root@XXXXX[ /var/tmp ]# ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=Administrators,cn=Builtin,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSOPWD' member | less | grep machine

member: CN=machine-f6c5c33e-3280-433c-a3c7-a9839f2b1f1e,CN=ServicePrincipals,DC=vsphere,DC=local
member: CN=machine-56095117-1936-49c7-b7cb-341512a6b9bd,CN=ServicePrincipals,DC=vsphere,DC=local
 

Once done, stop and start all services.