Unable to power on the VM's across ALL clusters in the environment.

Products

• VMware vCenter Server

Issue/Introduction

-VM power-on is failing with the below error in the vCenter UI

-vpxd.log shows

Environment

VMware vCenter 7.0, 8.0

Cause

The host which is not working/responding properly, can cause this type of issue, as if we have a process that requires a lock to be held until a host responds, that lock may be held far longer than we would consider normal.

Resolution

1)Go to the vpxd-profiler.log and search for the keyword "WAITNG_MOLOCK" which means

-Another task is waiting indefinitely, causing stuck workflows (e.g., VM power-on failures or UI timeouts).

-Deadlocks or lock contention inside vCenter’s inventory service or vpxd process.

Ex:--> ThreadState/ThreadId/10731/State/WAITING_MOLOCK -- host-<host ID>::vim.HostSystem

Thread which is shown in profiler logs as waiting on a lock, we can see that this thread is tied in with that host as well as with the no stats from host messages

2)Find out the host name from the host ID above:

-Get into VCDB /opt/vmware/postgres/current/bin/psql  -d VCDB -U postgres

-Run select dns_name,ip_address from vpx_host where ID =<host ID from the vpxd-profiler.log>;

-Output will be similar to dns_name will be FQDN of the host and ip_address will be IP of the host. 

3)Disconnect the identified host from the  vCenter.

4)Post which able to power on the VM's successfully in the Environment.