applmgmt/topologysvc/vsm service does not start due to missing Machine Account in Administrators group
search cancel

applmgmt/topologysvc/vsm service does not start due to missing Machine Account in Administrators group

book

Article ID: 382534

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 8.0 VMware vCenter Server 7.0

Issue/Introduction

  • applmgmt/vmware-topologysvc/vmware-vsm service does not start manually or automatically after a reboot of vCenter.
  • The respective logs does not update with latest entries.
  • In vCenter, you will see similar error in /var/log/vmware/vmon/vmon.log.  

YYY-MM-DDTHH:MM:SS.MSz In(05) host-35286 Received start request for applmgmt
YYY-MM-DDTHH:MM:SS.MSz In(05) host-35286 <applmgmt-prestart> Constructed command: /usr/lib/applmgmt/support/scripts/prestart-applmgmt.sh
YYY-MM-DDTHH:MM:SS.MSz Wa(03) host-35286 <applmgmt> Service pre-start command's stderr: Removed /etc/systemd/system/applmgmt.service.
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286
YYY-MM-DDTHH:MM:SS.MSz Wa(03) host-35286 <applmgmt> Service pre-start command's stderr: Created symlink /etc/systemd/system/applmgmt.service → /dev/null.
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286
YYY-MM-DDTHH:MM:SS.MSz Wa(03) host-35286 <applmgmt> Service pre-start command's stderr: Traceback (most recent call last):
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286   File "/usr/lib/applmgmt/support/scripts/create_svc_account.py", line 80, in <module>
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286
YYY-MM-DDTHH:MM:SS.MSz Wa(03) host-35286 <applmgmt> Service pre-start command's stderr:     main()
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286   File "/usr/lib/applmgmt/support/scripts/create_svc_account.py", line 75, in main
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286
YYY-MM-DDTHH:MM:SS.MSz Wa(03) host-35286 <applmgmt> Service pre-start command's stderr:     setup_service_account(args.account, user_perm = args.userPerm,
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286   File "/usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py", line 297, in setup_service_account
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286
YYY-MM-DDTHH:MM:SS.MSz Wa(03) host-35286 <applmgmt> Service pre-start command's stderr:     svcaccount.create_roles(roles)
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286   File "/usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py", line 145, in create_roles
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286
YYY-MM-DDTHH:MM:SS.MSz Wa(03) host-35286 <applmgmt> Service pre-start command's stderr:     count = authz_client.load_roles(roles)
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 619, in load_roles
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286
YYY-MM-DDTHH:MM:SS.MSz Wa(03) host-35286 <applmgmt> Service pre-start command's stderr:     if (self.add_role(
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 636, in add_role
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286
YYY-MM-DDTHH:MM:SS.MSz Wa(03) host-35286 <applmgmt> Service pre-start command's stderr:     self._authz_intservice.AddSolutionRole(
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 598, in <lambda>
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286
YYY-MM-DDTHH:MM:SS.MSz Wa(03) host-35286 <applmgmt> Service pre-start command's stderr:     self.f(*(self.args + (obj,) + args), **kwargs)
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 388, in _InvokeMethod
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286
YYY-MM-DDTHH:MM:SS.MSz Wa(03) host-35286 <applmgmt> Service pre-start command's stderr:     return self._stub.InvokeMethod(self, info, args)
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286   File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1757, in InvokeMethod
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286
YYY-MM-DDTHH:MM:SS.MSz Wa(03) host-35286 <applmgmt> Service pre-start command's stderr:     raise obj
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286
YYY-MM-DDTHH:MM:SS.MSz Wa(03) host-35286 <applmgmt> Service pre-start command's stderr: pyVmomi.VmomiSupport.SecurityError: (vmodl.fault.SecurityError) {
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286    dynamicType = <unset>,
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286    dynamicProperty = (vmodl.DynamicProperty) [],
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286    msg = '',
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286    faultCause = <unset>,
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286    faultMessage = (vmodl.LocalizableMessage) []
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286 }
YYY-MM-DDTHH:MM:SS.MSz Wa(03)+ host-35286
YYY-MM-DDTHH:MM:SS.MSz Er(02) host-35286 <applmgmt> Service pre-start command failed with exit code 1.
YYY-MM-DDTHH:MM:SS.MSz Wa(03) host-35286 [ReadSvcSubStartupData] No startup information from applmgmt.

  • In vCenter - /var/log/vmware/vpxd-svcs/vpxd-svcs.log:

YYY-MM-DDTHH:MM:SS.MSz [dataservice-3 [] WARN  com.vmware.cis.authorization.impl.AclPrivilegeValidator  opId=<OP_ID>] User
<sso_domain>\<machine_account_name> does not have privileges [Authorization.ModifyRoles] on object urn%3Aacl%3Aglobal%3Apermissions
YYY-MM-DDTHH:MM:SS.MSz [dataservice-1 [] WARN  com.vmware.cis.authorization.impl.AclPrivilegeValidator  opId=<OP_ID>] User
<sso_domain>\<machine_account_name> does not have privileges [Authorization.ModifyRoles] on object urn%3Aacl%3Aglobal%3Apermissions
YYY-MM-DDTHH:MM:SS.MSz [dataservice-4 [] WARN  com.vmware.cis.authorization.impl.AclPrivilegeValidator  opId=<OP_ID>] User
<sso_domain>\<machine_account_name> does not have privileges [Authorization.ModifyRoles] on object urn%3Aacl%3Aglobal%3Apermissions

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

The machine account is supposed to be part of administrators group to to perform the all the necessary operations but here for some reason looks like the machine account was removed from the administrators group.

Resolution

  • Before proceeding with the below steps, please ensure to have a valid snapshot/backup of the vCenter. If we have ELM, please take powered off snapshots of all the nodes.
  • Validate with the below command that the Machine Account recorded from vpxd-svcs.log is missing from Administrators group:

/usr/lib/vmware-vmafd/bin/dir-cli group list --name 'Administrators' --password '<admin-password>'

  • If it is missing, please add it with the below command, this is one whole command to be ran once. Update the cn as recorded from the vpxd-svcs.log. The below command is shown with SSO domain as vsphere.local. Please validate if its different in customer's environment, then update the dc values as well :

ldapmodify -x -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w '<admin-password>' << EOF
dn: cn=Administrators,cn=Builtin,dc=vsphere,dc=local
changetype: modify
add: member
member: cn=<machine_account_name>,ou=Domain Controllers,dc=vsphere,dc=local
EOF

  • Run the same command to verify if the machine account is added now:

/usr/lib/vmware-vmafd/bin/dir-cli group list --name 'Administrators' --password '<admin-password>'

  • Restart vCenter services: 

service-control --stop --all && service-control --start --all

  • If vCenter is in ELM, check if the affected vCenter services are up and running and restart vCenter services on all the other nodes.
Powered by Wolken Software