Symptoms:
/var/log/vmware/vpxd-svcs/pre-start-vpxd-svcs.log :
ERROR:tagging_grpc_registration:Failed to reregister Tagging service grpc endpoints with Lookup Service
ERROR:tagging_grpc_registration:generator didn't stop after throw()
Traceback (most recent call last):
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 260, in securityctx_modifier
yield
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests
return req_method(self, *args, **kargs)
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 364, in reregister_service
self.service_content.serviceRegistration.Set(svc_id, svc_set_spec)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 595, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 385, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod
raise obj # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.vmodl.fault.SecurityError: (vmodl.fault.SecurityError) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = '',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) []
}
During handling of the above exception, another exception occurred:
vmon.log :
XXXX-XX-XXT08:49:46.978Z In(05) host-2111 Received start request for vpxd-svcs
XXXX-XX-XXT08:49:46.978Z In(05) host-2111 <vpxd-svcs-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py /storage /var/log
XXXX-XX-XXT08:49:47.936Z Wa(03) host-2111 <vpxd-svcs> Service pre-start command's stderr: XXXX-XX-XXT08:49:47.936Z Security error: (vmodl.fault.SecurityError) {
XXXX-XX-XXT08:49:47.936Z Wa(03)+ host-2111 dynamicType = <unset>,
XXXX-XX-XXT08:49:47.936Z Wa(03)+ host-2111 dynamicProperty = (vmodl.DynamicProperty) [],
XXXX-XX-XXT08:49:47.936Z Wa(03)+ host-2111 msg = '',
XXXX-XX-XXT08:49:47.936Z Wa(03)+ host-2111 faultCause = <unset>,
XXXX-XX-XXT08:49:47.936Z Wa(03)+ host-2111 faultMessage = (vmodl.LocalizableMessage) []
XXXX-XX-XXT08:49:47.936Z Wa(03)+ host-2111 }
XXXX-XX-XXT08:49:47.936Z Wa(03)+ host-2111
XXXX-XX-XXT08:49:47.985Z Wa(03) host-2111 <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 260, in securityctx_modifier
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 yield
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 return req_method(self, *args, **kargs)
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 364, in reregister_service
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 self.service_content.serviceRegistration.Set(svc_id, svc_set_spec)
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 595, in <lambda>
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 self.f(*(self.args + (obj,) + args), **kwargs)
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 385, in _InvokeMethod
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 return self._stub.InvokeMethod(self, info, args)
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 raise obj # pylint: disable-msg=E0702
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 pyVmomi.VmomiSupport.SecurityError: (vmodl.fault.SecurityError) {
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 dynamicType = <unset>,
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 dynamicProperty = (vmodl.DynamicProperty) [],
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 msg = '',
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 faultCause = <unset>,
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 faultMessage = (vmodl.LocalizableMessage) []
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 }
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 During handling of the above exception, another exception occurred:
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 Traceback (most recent call last):
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 119, in update_endpoints
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 ls_obj.reregister_service(service_info.serviceId, mutable_spec)
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 return req_method(self, *args, **kargs)
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 File "/usr/lib/python3.7/contextlib.py", line 161, in __exit__
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 raise RuntimeError("generator didn't stop after throw()")
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 RuntimeError: generator didn't stop after throw()
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 During handling of the above exception, another exception occurred:
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 Traceback (most recent call last):
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 100, in <module>
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 endpoint_registration_runner(logging_file)
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 65, in endpoint_registration_runner
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 UpdateTaggingServiceGrpcEndpoint(logging_file).run()
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 54, in run
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 self.update_endpoints()
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 146, in update_endpoints
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 raise Exception("Tagging grpc reregistration failed while"
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111 Exception: Tagging grpc reregistration failed while executing vpxd-svcs prestart commands
XXXX-XX-XXT08:49:47.985Z Wa(03)+ host-2111
XXXX-XX-XXT08:49:48.042Z Er(02) host-2111 <vpxd-svcs> Service pre-start command failed with exit code 1.
VMware vCenter Server 7.0.x
This issue occurs when machine information is missing from Builtin - Administrators.
To resolve this issue :
Identify the machine ID :
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
Or
cat /etc/vmware/install-defaults/sca.hostid
In this example : SSO Domain is vsphere.local (change as per customer environment)
root@XXXXX [ / ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
56095117-1936-49c7-b7cb-341512a6b9bd
cat /etc/vmware/install-defaults/sca.hostid
56095117-1936-49c7-b7cb-341512a6b9bd
Next we need to check if the machine account is missing from Builtin - Administrator. To check run,
ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=Administrators,cn=Builtin,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSOPWD' member | less | grep machine
Output :
root@XXXXX[ / ]# ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=Administrators,cn=Builtin,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'VMware@123' member | less | grep machine
member: CN=machine-f6c5c33e-3280-433c-a3c7-a9839f2b1f1e,CN=ServicePrincipals,DC=vsphere,DC=local
As we can see machine-56095117-1936-49c7-b7cb-341512a6b9bd is missing and we should add the entry.
To do that we will create LDIF file with the contents and add the same using ldapadd command.
cd /var/tmp
vi add.ldif and paste the below contents.
dn: cn=Administrators,cn=Builtin,dc=vsphere,dc=local
changetype: modify
add: member
member: CN=machine-MISSINGMACHINEID,CN=ServicePrincipals,DC=vsphere,DC=local
Save the file and quit.
Next run :
/opt/likewise/bin/ldapadd -h localhost -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w 'SSOPWD' -f add.ldif
Verify using ldapsearch command and make sure the entry is added properly.
root@XXXXX[ /var/tmp ]# ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=Administrators,cn=Builtin,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSOPWD' member | less | grep machine
member: CN=machine-f6c5c33e-3280-433c-a3c7-a9839f2b1f1e,CN=ServicePrincipals,DC=vsphere,DC=local
member: CN=machine-56095117-1936-49c7-b7cb-341512a6b9bd,CN=ServicePrincipals,DC=vsphere,DC=local
Once done, stop and start all services.