Lookup Service thread pool is exhausted and vpxd does not start due to a missing localhost entry in /etc/hosts on vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

After a restart of vCenter Server or all of its services, the vpxd service does not start and remains in PENDING state.

• /var/log/vmware/vpxd/vpxd.log shows timeouts of Lookup Service requests.

YYYY-MM-DDTHH:MM:SS warning vpxd[#######] [Originator@6876 sub=LSClient] Caught exception during lookup service RPC: N7Vmacore16TimeoutExceptionE(Operation timed out)

• /var/log/vmware/lookupsvc/lookupserver-default.log shows that all 100 threads of the Lookup Service thread pool are in use and new requests are rejected.

[YYYY-MM-DDTHH:MM:SS pool-2-thread-64 ERROR com.vmware.vim.vmomi.server.common.impl.BaseProcessingStep] Task TracingRunnableWrapper{wrapped=com.vmware.vim.vmomi.server.impl.InvocationTask@XXXXXXXX, span=com.vmware.vim.vmomi.core.tracing.NoopTracer$NoopSpan@XXXXXXXX} rejected from java.util.concurrent.ThreadPoolExecutor@XXXXXXXX[Running, pool size = 100, active threads = 100, queued tasks = 0, completed tasks = 15000]
java.util.concurrent.RejectedExecutionException: Task TracingRunnableWrapper{...} rejected from java.util.concurrent.ThreadPoolExecutor@XXXXXXXX[Running, pool size = 100, active threads = 100, queued tasks = 0, completed tasks = 15000]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
        at com.vmware.vim.vmomi.server.common.impl.BaseProcessingStep.process(BaseProcessingStep.java:110)

• /etc/hosts has no localhost entry, as in the following example.

# Begin /etc/hosts (network card version)

127.0.0.1   <FQDN>.localdomain
127.0.0.1   <FQDN>

Environment

VMware vCenter Server 8.x

Cause

/etc/hosts is malformed and lacks the localhost entry and the IPv6 loopback entry. Internal services cannot resolve localhost correctly, which causes Lookup Service requests to hang and exhaust the thread pool. vpxd then times out while waiting for Lookup Service and remains in PENDING state.

Resolution

1. Create a backup of the current hosts file.

cp /etc/hosts /etc/hosts.bak

2. Edit /etc/hosts so that the loopback addresses are mapped to the FQDN, the hostname, and localhost, as in the following example.

# Begin /etc/hosts (network card version)
# VAMI_EDIT_BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1 <FQDN> <hostname> localhost
::1 <FQDN> <hostname> localhost ipv6-localhost ipv6-loopback
# VAMI_EDIT_END

3. Restart all vCenter Server services.

service-control --stop --all && service-control --start --all

Additional Information

Japanese version: vCenter Server で /etc/hosts に localhost の記載が無い場合に Lookup Service のスレッドプールが枯渇し vpxd が起動しない(458491)

If localhost cannot be resolved from /etc/hosts, the query is forwarded to dnsmasq. The following command can be used to check this.

grep -c "query\[A\] localhost" /var/log/vmware/dnsmasq.log

For a similar issue where vpxd-svcs does not start due to malformed /etc/hosts, see vpxd-svcs service is not starting in VMware vCenter Server(417188).

If the issue persists after correcting /etc/hosts, check for other name resolution problems such as connectivity to the DNS server.