"An error occurred while starting service 'vpxd-svcs'" | 7.X | Failed to reregister Tagging service grpc endpoints with Lookup Service

search cancel

"An error occurred while starting service 'vpxd-svcs'" | 7.X | Failed to reregister Tagging service grpc endpoints with Lookup Service

book

Article ID: 315454

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

/var/log/vmware/vpxd-svcs/pre-start-vpxd-svcs.log :

ERROR:tagging_grpc_registration:Failed to reregister Tagging service grpc endpoints with Lookup Service

ERROR:tagging_grpc_registration:generator didn't stop after throw()
Traceback (most recent call last):
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 260, in securityctx_modifier
yield
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests
return req_method(self, *args, **kargs)
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 364, in reregister_service
self.service_content.serviceRegistration.Set(svc_id, svc_set_spec)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 595, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 385, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod
raise obj # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.vmodl.fault.SecurityError: (vmodl.fault.SecurityError) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = '',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) []
}
During handling of the above exception, another exception occurred:

/var/log/vmware/vmware-vmon/vmon.log :

YYYY-MM-DDThh-mm-ss.978Z In(05) host-2111 Received start request for vpxd-svcs

YYYY-MM-DDThh-mm-ss.978Z In(05) host-2111 <vpxd-svcs-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py /storage /var/log

YYYY-MM-DDThh-mm-ss.936Z Wa(03) host-2111 <vpxd-svcs> Service pre-start command's stderr: YYYY-MM-DDThh-mm-ss.936Z  Security error: (vmodl.fault.SecurityError) {

YYYY-MM-DDThh-mm-ss.936Z Wa(03)+ host-2111 dynamicType = <unset>,
YYYY-MM-DDThh-mm-ss.936Z Wa(03)+ host-2111 dynamicProperty = (vmodl.DynamicProperty) [],
YYYY-MM-DDThh-mm-ss.936Z Wa(03)+ host-2111 msg = '',
YYYY-MM-DDThh-mm-ss.936Z Wa(03)+ host-2111 faultCause = <unset>,
YYYY-MM-DDThh-mm-ss.936Z Wa(03)+ host-2111 faultMessage = (vmodl.LocalizableMessage) []
YYYY-MM-DDThh-mm-ss.936Z Wa(03)+ host-2111 }
YYYY-MM-DDThh-mm-ss.936Z Wa(03)+ host-2111

YYYY-MM-DDThh-mm-ss.985Z Wa(03) host-2111 <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 260, in securityctx_modifier

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 yield

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 return req_method(self, *args, **kargs)

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 364, in reregister_service

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 self.service_content.serviceRegistration.Set(svc_id, svc_set_spec)

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 595, in <lambda>

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 self.f(*(self.args + (obj,) + args), **kwargs)

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 385, in _InvokeMethod

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 return self._stub.InvokeMethod(self, info, args)

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111   File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 raise obj # pylint: disable-msg=E0702
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 pyVmomi.VmomiSupport.SecurityError: (vmodl.fault.SecurityError) {
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 dynamicType = <unset>,
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 dynamicProperty = (vmodl.DynamicProperty) [],
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 msg = '',
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 faultCause = <unset>,
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 faultMessage = (vmodl.LocalizableMessage) []
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 }
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 During handling of the above exception, another exception occurred:
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 Traceback (most recent call last):

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 119, in update_endpoints

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 ls_obj.reregister_service(service_info.serviceId, mutable_spec)

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 return req_method(self, *args, **kargs)
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 File "/usr/lib/python3.7/contextlib.py", line 161, in __exit__
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 raise RuntimeError("generator didn't stop after throw()")
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 RuntimeError: generator didn't stop after throw()
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 During handling of the above exception, another exception occurred:
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111
YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 Traceback (most recent call last):

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 100, in <module>

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 endpoint_registration_runner(logging_file)

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 65, in endpoint_registration_runner

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 UpdateTaggingServiceGrpcEndpoint(logging_file).run()

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 54, in run

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 self.update_endpoints()

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 146, in update_endpoints

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 raise Exception("Tagging grpc reregistration failed while"

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111 Exception: Tagging grpc reregistration failed while executing vpxd-svcs prestart commands

YYYY-MM-DDThh-mm-ss.985Z Wa(03)+ host-2111
YYYY-MM-DDThh-mm-ss.042Z Er(02) host-2111 <vpxd-svcs> Service pre-start command failed with exit code 1.

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

This issue occurs when machine information is missing from Builtin - Administrators

Resolution

To resolve this issue :

Identify the machine ID :

Run this command :

/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

Or, cat /etc/vmware/install-defaults/sca.hostid

In this example : SSO Domain is vsphere.local (change as per customer environment)

/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

OUTPUT :
- ########-####-####-####-############

```
cat /etc/vmware/install-defaults/sca.hostid
```
- OUTPUT :
  - ########-####-####-####-############

Next we need to check if the machine account is missing from Builtin - Administrator.
To check run this command :
- ```
ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=Administrators,cn=Builtin,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSOPWD' member | less | grep machine
```
- Output :
  - ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=Administrators,cn=Builtin,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'VMware@123' member | less | grep machine

member: CN=machine-$$$$$$$$-$$$$-$$$$-$$$$-$$$$$$$$$$$$,CN=ServicePrincipals,DC=vsphere,DC=local

As we can see machine-########-####-####-####-############ is missing and we should add the entry.
To do that we will create LDIF file with the contents and add the same using ldapadd command.
Go to this path in vCenter : cd /var/tmp
Run this and paste the below contents :
- ```
vi add.ldif
```
- Paste the below contents :
  - dn: cn=Administrators,cn=Builtin,dc=vsphere,dc=local
    changetype: modify
    add: member
    member: CN=machine-MISSINGMACHINEID,CN=ServicePrincipals,DC=vsphere,DC=local
  - Note : In the above " MISSINGMACHINEID " replace it with above missing machine id.
- Save the file and quit.

Next run :

/opt/likewise/bin/ldapadd -h localhost -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w 'SSOPWD' -f add.ldif

Verify using ldapsearch command and make sure the entry is added properly.
Run the following command :
- ```
ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=Administrators,cn=Builtin,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSOPWD' member | less | grep machine
```
- OUTPUT :
  - ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=Administrators,cn=Builtin,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSOPWD' member | less | grep machine
    
    member: CN=machine-$$$$$$$$-$$$$-$$$$-$$$$-$$$$$$$$$$$$,CN=ServicePrincipals,DC=vsphere,DC=local
    member: CN=machine-########-####-####-####-############,CN=ServicePrincipals,DC=vsphere,DC=local
- Once done, stop and start all services:
  - ```
  service-control --stop --all && service-control --start --all
```

Feedback

thumb_up Yes

thumb_down No