CM-Inventory runs out of memory when hosts prepared for NSX through VCF

Products

• VMware NSX

Issue/Introduction

• You're running NSX-T 3.2.x or NSX 4.0.0.1 or earlier
• You're attempting prepare an ESXi host for NSX in a VCF environment.
• On the NSX Manager node that owns the virtual IP, you see the below entries under /var/log/cm-inventory/cm-inventory-tomcat-wrapper.log in the timeframe of the VCF workflow:

INFO | jvm 1 | YYYY/MM/DD hh:mm:ss | java.lang.OutOfMemoryError: Java heap space
STATUS | wrapper | YYYY/MM/DD hh:mm:ss | The JVM has run out of memory. Requesting thread dump.
STATUS | wrapper | YYYY/MM/DD hh:mm:ss | The JVM has run out of memory. Restarting JVM.
...
INFO | jvm 1 | YYYY/MM/DD hh:mm:ss | # java.lang.OutOfMemoryError: Java heap space
STATUS | wrapper | YYYY/MM/DD hh:mm:ss | The JVM has run out of memory. Requesting thread dump.
STATUS | wrapper | YYYY/MM/DD hh:mm:ss | The JVM has run out of memory. Restart JVM (Ignoring, already restarting).
INFO | jvm 1 | YYYY/MM/DD hh:mm:ss | # -XX:OnOutOfMemoryError="gzip -f /image/core/cm_inventory_oom.hpro

• You have a higher number of Distributed Portgroups configured on the vCenter side.

Environment

VMware NSX-T Data Center
VMware NSX

Cause

During the VCF workflow, information about the vCenter cluster may be fetched multiple times and as a result cm-inventory saves the retrieved information multiple times which may lead to out of memory errors in larger setups.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.2.2 and NSX 4.0.1, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround

1. 1. Increase the memory of the cm-inventory service from 512M to 1G by edit the file /usr/tanuki/conf/cm-inventory-tomcat-wrapper.conf on the NSX Managers as root user and change wrapper.java.maxmemory line to have a value of 1024 as shown below:
   

   wrapper.java.maxmemory=1024
   

2. Restart cm-inventory service with the below command:
   

   /etc/init.d/cm-inventory restart