vCenter server UI is inaccessible and the vCenter service is stopped in the VAMI

Products

• VMware vCenter Server

Issue/Introduction

• You are not able to log in to the vCenter UI, and logging into the vCenter Server Management Interface (VAMI) at https://<appliance-IP-address-or-FQDN>:5480,under Services, you see that VMware vCenter Server service is in a stopped state.
  
  
• Starting the VMware vCenter Server service allows you to log in to the vCenter UI. 
  
  
• In /var/log/vmware/vpxd/vpxd.log on the vCenter server, you see an error similar to:
  

  YYYY-MM-DDTHH:MM:SS.###Z error vpxd[#######] [Originator@#### sub=Memory checker] Current value ####### exceeds hard limit #######. Shutting down process.
  
  YYYY-MM-DDTHH:MM:SS.###Z panic vpxd[#######] [Originator@#### sub=Default]
  
  -->
  
  --> Panic: Memory exceeds hard limit. Panic

• Running the following command against /var/log/vmware/vpxd/vpxd-profiler.log on the vCenter server:
  

  grep UnfinishedRetrieversCnt vpxd-profiler.log | grep -v 'total 0' | sort -r -k1,1n

  
  You see a high total value such as:
  

  /SessionStats/SessionPool/Session/Id='#######-####-####-####-############'/Username='<exampleUsername>'/ClientIP='###.###.###.###'/PropertyCollector/UnfinishedRetrieversCnt/total 90

• Note: There could be many vpxd and vpxd-profiler logs in the /var/log/vmware/vpxd/ directory that may need to be searched.  

Environment

vCenter 8.x

Cause

A high UnfinishedRetrieversCnt total value causes an out of memory condition that causes the VMware vCenter Server service (vpxd) to panic and crash which renders the vCenter UI inaccessible. 

Resolution

• From the line with the high total value from the vpxd-profiler.log found above, note the Username= and ClientIP= values.
• These indicate the IP and username of the client causing the high UnfinishedRetrieversCnt that is causing the VMware vCenter Server service (vpxd) to crash.
• To resolve this issue, remove the connection to vCenter from the client application.