topologysvc and certificatemanagement services fail to start on vCenter Server
search cancel

topologysvc and certificatemanagement services fail to start on vCenter Server

book

Article ID: 427597

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

When starting vCenter Server services, the topologysvc and certificatemanagement services fail to start.

The issue is observed during service startup via `service-control --start --all` or during normal vCenter service initialization.

The following errors are observed in /var/log/vmware/vmon/vmon.log

<timestamp>T07:09:13.914Z Wa(03) host-984218 <topologysvc> Service pre-start command's stderr: Traceback (most recent call last):
<timestamp>T07:09:13.914Z Wa(03)+ host-984218   File "/usr/lib/vmware-topologysvc/scripts/topologysvc_prestart.py", line 194, in <module>
<timestamp>T07:09:13.914Z Wa(03)+ host-984218
<timestamp>T07:09:13.915Z Wa(03) host-984218 <topologysvc> Service pre-start command's stderr:     update_privs()
<timestamp>T07:09:13.915Z Wa(03)+ host-984218   File "/usr/lib/vmware-topologysvc/scripts/topologysvc_prestart.py", line 125, in update_privs
<timestamp>T07:09:13.915Z Wa(03)+ host-984218
<timestamp>T07:09:13.915Z Wa(03) host-984218 <topologysvc> Service pre-start command's stderr:     existing_privileges = authz_client._authz_service.GetPrivileges()
<timestamp>T07:09:13.915Z Wa(03)+ host-984218   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 618, in <lambda>
<timestamp>T07:09:13.915Z Wa(03)+ host-984218
<timestamp>T07:09:13.915Z Wa(03) host-984218 <topologysvc> Service pre-start command's stderr:     self.f(*(self.args + (obj,) + args), **kwargs)
<timestamp>T07:09:13.915Z Wa(03)+ host-984218   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 391, in _InvokeMethod
<timestamp>T07:09:13.915Z Wa(03)+ host-984218
<timestamp>T07:09:13.915Z Wa(03) host-984218 <topologysvc> Service pre-start command's stderr:     return self._stub.InvokeMethod(self, info, args)
<timestamp>T07:09:13.915Z Wa(03)+ host-984218   File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1791, in InvokeMethod
<timestamp>T07:09:13.915Z Wa(03)+ host-984218
<timestamp>T07:09:13.916Z Wa(03) host-984218 <topologysvc> Service pre-start command's stderr:     raise obj
<timestamp>T07:09:13.916Z Wa(03)+ host-984218
<timestamp>T07:09:13.916Z Wa(03) host-984218 <topologysvc> Service pre-start command's stderr: pyVmomi.VmomiSupport.vmodl.fault.SecurityError: (vmodl.fault.SecurityError) {
<timestamp>T07:09:13.916Z Wa(03)+ host-984218    dynamicType = <unset>,
<timestamp>T07:09:13.916Z Wa(03)+ host-984218    dynamicProperty = (vmodl.DynamicProperty) [],
<timestamp>T07:09:13.916Z Wa(03)+ host-984218    msg = '',
<timestamp>T07:09:13.916Z Wa(03)+ host-984218    faultCause = <unset>,
<timestamp>T07:09:13.916Z Wa(03)+ host-984218    faultMessage = (vmodl.LocalizableMessage) []
<timestamp>T07:09:13.916Z Wa(03)+ host-984218 }
<timestamp>T07:09:13.916Z Wa(03)+ host-984218
<timestamp>T07:09:13.996Z Er(02) host-984218 <topologysvc> Service pre-start command failed with exit code 1.
<timestamp>T07:09:16.341Z Wa(03) host-984218 <certificatemanagement> Service pre-start command's stderr: Traceback (most recent call last):
<timestamp>T07:09:16.341Z Wa(03)+ host-984218   File "/usr/lib/vmware-certificatemanagement/scripts/certificatemanagement_prestart.py", line 252, in <module>
<timestamp>T07:09:16.341Z Wa(03)+ host-984218
<timestamp>T07:09:16.341Z Wa(03) host-984218 <certificatemanagement> Service pre-start command's stderr:     update_privs()
<timestamp>T07:09:16.341Z Wa(03)+ host-984218   File "/usr/lib/vmware-certificatemanagement/scripts/certificatemanagement_prestart.py", line 96, in update_privs
<timestamp>T07:09:16.341Z Wa(03)+ host-984218
<timestamp>T07:09:16.341Z Wa(03) host-984218 <certificatemanagement> Service pre-start command's stderr:     existingPrivileges = authz_client._authz_service.GetPrivileges()
<timestamp>T07:09:16.341Z Wa(03)+ host-984218   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 618, in <lambda>
<timestamp>T07:09:16.341Z Wa(03)+ host-984218
<timestamp>T07:09:16.341Z Wa(03) host-984218 <certificatemanagement> Service pre-start command's stderr:     self.f(*(self.args + (obj,) + args), **kwargs)
<timestamp>T07:09:16.341Z Wa(03)+ host-984218   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 391, in _InvokeMethod
<timestamp>T07:09:16.341Z Wa(03)+ host-984218
<timestamp>T07:09:16.341Z Wa(03) host-984218 <certificatemanagement> Service pre-start command's stderr:     return self._stub.InvokeMethod(self, info, args)
<timestamp>T07:09:16.341Z Wa(03)+ host-984218   File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1791, in InvokeMethod
<timestamp>T07:09:16.341Z Wa(03)+ host-984218
<timestamp>T07:09:16.342Z Wa(03) host-984218 <certificatemanagement> Service pre-start command's stderr:     raise obj
<timestamp>T07:09:16.342Z Wa(03)+ host-984218 pyVmomi.VmomiSupport.vmodl.fault.SecurityError: (vmodl.fault.SecurityError) {
<timestamp>T07:09:16.342Z Wa(03)+ host-984218    dynamicType = <unset>,
<timestamp>T07:09:16.342Z Wa(03)+ host-984218    dynamicProperty = (vmodl.DynamicProperty) [],
<timestamp>T07:09:16.342Z Wa(03)+ host-984218    msg = '',
<timestamp>T07:09:16.342Z Wa(03)+ host-984218    faultCause = <unset>,
<timestamp>T07:09:16.342Z Wa(03)+ host-984218    faultMessage = (vmodl.LocalizableMessage) []
<timestamp>T07:09:16.342Z Wa(03)+ host-984218 }
<timestamp>T07:09:16.342Z Wa(03)+ host-984218
<timestamp>T07:09:16.408Z Er(02) host-984218 <certificatemanagement> Service pre-start command failed with exit code 1.


/var/log/vmware/vpxd-svcs/vpxd-svcs.log : 

<timestamp>T06:29:15.973Z [authz-service-4 [] WARN  com.vmware.cis.authorization.impl.AclPrivilegeValidator  opId=b0b1b5c1-0013-4b19-b233-45d6e74abeb7] User VSPHERE.LOCAL\VCMachineAccount does not have privileges [System.View] on object urn%3Aacl%3Aglobal%3Apermissions
<timestamp>T06:29:15.988Z [authz-service-4 [] WARN  com.vmware.cis.core.authz.accesscontrol.impl.CheckPrivilegesRouterRiseImpl  opId=b0b1b5c1-0013-4b19-b233-45d6e74abeb7] User VSPHERE.LOCAL\VCMachineAccount does not have privileges [System.View] on object urn%3Aacl%3Aglobal%3Apermissions

Environment

vCenter Server

Cause

The vCenter Server machine account is missing from the Administrators group in the SSO domain.
This condition can occur due to vCenter replication or synchronization issues, resulting in missing or incomplete privilege assignments for the machine account.
As a result, the authorization service denies required privileges during service initialization, causing topologysvc and certificatemanagement to fail during their pre-start checks.

Resolution

1. Take powered off snapshots of all vCenter Server nodes.
2. SSH to the affected VCSA node as `root`.
3. Copy the provided script `vcsa_machine_account_check.sh` to `/var/tmp`.

   * Alternatively, open a text editor and manually create the script file, then paste the contents.

4. Make the script executable:

   chmod +x /var/tmp/vcsa_machine_account_check.sh
   

5. Execute the script:

   cd /var/tmp
   ./vcsa_machine_account_check.sh

After the script execution completes successfully, start the vCenter services: service-control --start --all

Attachments

vcsa_machine_account_check.sh get_app
Powered by Wolken Software