vpxd-svcs service is not starting in VMware vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

After reboot or restart of all services vpxd-svcs service is not starting.

• Checking log vpxd-svcs.log located in /var/log/vmware/vpxd-svcs you don't see any attempt for service to start.
• Additional symptoms - vCenter Services start but vSphere Client UI spins indefinitely after credential input.
• VDT and lsdoctor logs indicate vmodl.fault.SystemError and thread pool exhaustion in the Lookup Service.
• The /var/log/vmware/vmon/vmon.log shows entries similar to the following

Wa(03)+ host-112767
Wa(03) host-112767 <vpxd-svcs> Service pre-start command's stderr:     return req_method(self, *args, **kargs)
Wa(03)+ host-112767
Wa(03) host-112767 <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 306, in _init_service_content
Wa(03)+ host-112767
Wa(03) host-112767 <vpxd-svcs> Service pre-start command's stderr:     self.service_content = si.RetrieveServiceContent()
Wa(03)+ host-112767
Wa(03) host-112767 <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 618, in <lambda>
Wa(03)+ host-112767
Wa(03) host-112767 <vpxd-svcs> Service pre-start command's stderr:     self.f(*(self.args + (obj,) + args), **kwargs)
Wa(03)+ host-112767
Wa(03) host-112767 <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 391, in _InvokeMethod
Wa(03)+ host-112767
Wa(03) host-112767 <vpxd-svcs> Service pre-start command's stderr:     return self._stub.InvokeMethod(self, info, args)
Wa(03)+ host-112767
Wa(03) host-112767 <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1607, in InvokeMethod
Wa(03)+ host-112767
Wa(03) host-112767 <vpxd-svcs> Service pre-start command's stderr:     raise obj  # pylint: disable-msg=E0702
Wa(03)+ host-112767
Wa(03) host-112767 <vpxd-svcs> Service pre-start command's stderr: pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
Wa(03)+ host-112767    dynamicType = <unset>,
Wa(03)+ host-112767    dynamicProperty = (vmodl.DynamicProperty) [],
Wa(03)+ host-112767    msg = 'Executor rejected to handle task at step com.vmware.vim.vmomi.server.impl.InvocationStep@18674b3',
Wa(03)+ host-112767    faultCause = <unset>,
Wa(03)+ host-112767    faultMessage = (vmodl.LocalizableMessage) [],
Wa(03)+ host-112767    reason = 'Task TracingRunnableWrapper{wrapped=com.vmware.vim.vmomi.server.impl.InvocationTask@7581fb59, span=com.vmware.vim.vmomi.core.tracing.NoopTracer$NoopSpan@79e322cf} rejected from java.util.concurrent.ThreadPoolExecutor@1804e5c[Running, pool size = 100, active threads = 100, queued tasks = 0, completed tasks = 15542]'

• In hosts file there are no localhost entry, and that vCenter don't have FQDN set up and it's using IP as hostname.

In example,

cat /etc/hosts
# Begin /etc/hosts (network card version)

# VAMI_EDIT_BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1  ####.#####.#### ####
::1  ####.#####.#### #### ipv6-localhost ipv6-loopback
# VAMI_EDIT_END

 

Environment

• VMware vCenter Server 8.x

Cause

vCenter PNID is IP and FQDN is localhost. Some services need to resolve IP to FQDN.
The /etc/hosts file is malformed, lacking explicit mapping for the Primary Network Identifier (PNID) to the IP address. This prevents internal services from correctly binding, causing the Lookup Service to hang and exhaust the thread pool.

Resolution

1. Create a backup of the current hosts file: cp /etc/hosts /etc/hosts.bak
2. Edit /etc/hosts using vi /etc/hosts.
3. Append a new line explicitly mapping the vCenter network IP address to localhost
   

   In example (####.#####.#### stands for IP of vCenter)

   # Begin /etc/hosts (network card version)

   # VAMI_EDIT_BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1  ####.#####.#### #### localhost
::1  ####.#####.#### #### localhost ipv6-localhost ipv6-loopback
   ####.#####.#### localhost
   # VAMI_EDIT_END

4. Restart all vCenter services to apply the corrected network bindings: service-control --stop --all && service-control --start --all