vCenter inaccessible with no healthy upstream message while VAMI indicates Appliance is running dangerously low on memory. Add more memory to the machine.

Products

• VMware vCenter Server

Issue/Introduction

vCenter is inaccessible with error message: no healthy upstream

VAMI Summary page is reporting Critical error of memory "Appliance is running dangerously low on memory. Add more memory to the machine."

 

"vmware-stsd" service in "Stopped" state

sts-runtime.log.stderr log indicates the following:

Starting service process with pid: 36423.
Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M     -Dcom.sun.org.apache.xml.internal.security.ignoreLineBreaks=true     -Dorg.apache.xml.security.ignoreLineBreaks=true
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000f8400000, 524288, 0) failed; error='Cannot allocate memory' (errno=12)
Starting service process with pid: 456846.
Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M     -Dcom.sun.org.apache.xml.internal.security.ignoreLineBreaks=true     -Dorg.apache.xml.security.ignoreLineBreaks=true
ERROR:  'java.io.IOException: Broken pipe'
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f93a8070000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Starting service process with pid: 704479.
Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M     -Dcom.sun.org.apache.xml.internal.security.ignoreLineBreaks=true     -Dorg.apache.xml.security.ignoreLineBreaks=true
ERROR:  'java.io.IOException: Broken pipe'
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f1b947f0000, 65536, 1) failed; error='Cannot allocate memory' (errno=12)

 

 

Environment

vCenter Appliance 8.0.x

Cause

Wrong sizing defined on vCenter when deployed combined with excessive login from users due to characteristics of the environment.

For instance:

Considering the Hardware Requirements for the vCenter Server Appliance, a vCenter deployed as Small because it has less than 100 hosts and less than 1,000 virtual machines, but it has a very high load of users constantly triggering login requests to vCenter services will make the resources of vCenter VM to do not be sufficient to accommodate the load.

 

Resolution

The resolution is to deploy vCenter considering the size to accommodate the load that may increase over the time.

As workaround for this specific issue, you can Manually increase the heap memory as per KB 320871 of the vmware-stsd service with the command:

• cloudvm-ram-size -C NEW_VALUE vmware-stsd