CM-Inventory runs out of memory when hosts prepared for NSX through VCF

Products

• VMware NSX

Issue/Introduction

Symptoms:

• You're running NSX 3.2.x or NSX 4.0.0.1 or earlier
• You're attempting prepare an ESXi host for NSX in a VCF environment.
• On the NSX manager node that owns the virtual IP, you see the below entries under /var/log/cm-inventory/cm-inventory-tomcat-wrapper.log in the timeframe of the VCF workflow:

INFO | jvm 1 | YYYY/MM/DD hh:mm:ss | java.lang.OutOfMemoryError: Java heap space
STATUS | wrapper | YYYY/MM/DD hh:mm:ss | The JVM has run out of memory. Requesting thread dump.
STATUS | wrapper | YYYY/MM/DD hh:mm:ss | The JVM has run out of memory. Restarting JVM.
...
INFO | jvm 1 | YYYY/MM/DD hh:mm:ss | # java.lang.OutOfMemoryError: Java heap space
STATUS | wrapper | YYYY/MM/DD hh:mm:ss | The JVM has run out of memory. Requesting thread dump.
STATUS | wrapper | YYYY/MM/DD hh:mm:ss | The JVM has run out of memory. Restart JVM (Ignoring, already restarting).
INFO | jvm 1 | YYYY/MM/DD hh:mm:ss | # -XX:OnOutOfMemoryError="gzip -f /image/core/cm_inventory_oom.hpro

• You have a higher number of Distributed Portgroups configured on the vCenter side.

Environment

VMware NSX-T Data Center

Cause

During the VCF workflow, information about the vCenter cluster may be fetched multiple times and as a result cm-inventory saves the retrieved information multiple times which may lead to out of memory errors in larger setups.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.2.2 and VMware NSX 4.0.1, available at VMware downloads.

Workaround:
1. Increase the memory of the cm-inventory service from 512M to 1G by edit the file /usr/tanuki/conf/cm-inventory-tomcat-wrapper.conf on the NSX managers as root user and change wrapper.java.maxmemory line to have a value of 1024 as shown below:

...
wrapper.java.maxmemory=1024
...

2. Restart cm-inventory service with the below command:

/etc/init.d/cm-inventory restart