VMware Cloud Foundation tasks fail with "java.lang.OutOfMemoryError" in the hms-agent-debug.log file

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• When performing a task in SDDC Manager, the task fails. Examples of tasks that have failed:
  • Workload Domain Creation/Deletion/Expansion
  • Host Decommissioning
  • Lifecycle Updates
• Hardware details are not present on the Dashboard page.
• In the /home/vrack/vrm/logs/hms-agent-debug.log file, on the SDDC Manager VM, you see entries similar to:

2017-11-08 03:39:45.180 WARN  [qtp494894055-94] [org.eclipse.jetty.servlet.ServletHandler] Error for /api/1.0/hms/event/switches/R1S2/SWITCH_PORT
java.lang.OutOfMemoryError: Java heap space
2017-11-08 03:41:49.931 WARN  [qtp494894055-75] [org.eclipse.jetty.util.thread.QueuedThreadPool]
java.lang.OutOfMemoryError: Java heap space
2017-11-08 03:43:20.190 DEBUG [qtp494894055-22] [com.vmware.vrack.hms.ipmiservice.IpmiServiceExecutor] Is the node/host available or reachable for Node: R1N2
2017-11-08 03:39:58.404 WARN  [qtp494894055-84] [org.eclipse.jetty.util.thread.QueuedThreadPool]
java.lang.OutOfMemoryError: Java heap space
...
2017-11-08 07:32:29.096 ERROR [qtp494894055-32] [com.vmware.vrack.hms.rest.exception.ExceptionHandler] Returning Bad Request status due to unhandled exception.
java.lang.OutOfMemoryError: Java heap space
...
2017-11-08 08:52:06.533 ERROR [qtp494894055-104] [com.vmware.vrack.hms.rest.exception.ExceptionHandler] Returning Bad Request status due to unhandled exception.
java.lang.OutOfMemoryError: Java heap space
2017-11-08 08:50:16.301 DEBUG [qtp494894055-35] [com.vmware.vrack.hms.rest.services.HmsNodeRateLimiterFilter] Returning response code 200 for request: GET http://localhost:8448/api/1.0/hms/event/host/R1N0/SERVER
2017-11-08 08:51:15.443 WARN  [qtp494894055-78] [org.eclipse.jetty.servlet.ServletHandler] Error for /api/1.0/hms/event/host/R1N12/SERVER
java.lang.OutOfMemoryError: Java heap space
2017-11-08 08:52:02.213 WARN  [qtp494894055-24] [org.eclipse.jetty.servlet.ServletHandler] Error for /api/1.0/hms/event/host/R1N14/CPU
java.lang.OutOfMemoryError: Java heap space

or:  
java.lang.OutOfMemoryError: GC overhead limit exceeded
 

Resolution

This is a known issue affecting VMware Cloud Foundation 2.2. Currently, there is no resolution.

Workaround:

To work around this issue, on the SDDC Manager VM, restart HMS.

If the problem persists, increase the memory limit:

1. SSH to the SDDC Manager VM and change to the /opt/vrack/hms directory.
2. Check the current memory allocation for HMS.​

grep -i "xmx" hms.sh

You should see output similar to:

/bin/su $HMS_USER -c ""$JAVA" -Xmx2048m -XX:MaxPermSize=512m -classpath "$CLASSPATH" Dhms.config.file=$HMS_CONFIG $JETTY_LOGGING_OPT $HMS_LOGGING_CONFIG com.vmware.vrack.hms.HmsApp >> $HMSA_CONSOLE_LOG_FILE 2>&1 &"

In this example, the HMS memory allocation is set to the default, 2048 MB.

3. Increase the memory to 4096MB (or higher as needed).

sed -i -e 's/Xmx2048/Xmx4096/g' hms.sh​

4. Optionally, confirm the new memory configuration.

grep -i "xmx" hms.sh
 

/bin/su $HMS_USER -c ""$JAVA" -Xmx4096m -XX:MaxPermSize=512m -classpath "$CLASSPATH" -Dhms.config.file=$HMS_CONFIG $JETTY_LOGGING_OPT $HMS_LOGGING_CONFIG com.vmware.vrack.hms.HmsApp >> $HMSA_CONSOLE_LOG_FILE 2>&1 &"

5. Restart the HMS by running this command:

systemctl restart hms

6. Restart Workflow containing the affected task.