Workaround for OOM errors/slowness in application for customers using VCF 4.4.1.x

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

Users on older version of VCF on 4.4.1.x, experiencing Out of Memory errors , can add the logger configuration to remove debug loglevel for this specific class, and instead use just INFO log level if they see OOM errors (over utilized heap).

Environment

VMware Cloud Foundation 4.4.1

Cause

As part of periodic NSX Audit , and subsequent NSX Inventory Sync, the NsxtInventoryLoader logs the NSXInventory JSON to lcm-debug.log, and as part of logging, it happens to consume/hog enormous amount of heap memory.

Resolution

Currently there is no resolution to the issue. This will be fixed in the future releases.

Workaround:

To workaround the issue, please follow the below mentioned steps:

1. Login as root.

su - root
// enter the password

2. Go to Directory.

cd /opt/vmware/vcf/lcm/lcm-app/conf

3. Edit the logback-prod.xml, increase the log level and save the file.

vi logback-prod.xml
// Add the below line to the end of file:
<logger name="com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtInventoryLoader"  level="INFO" additivity="false">
        <appender-ref ref="LCM" />
</logger>

4. Restart LCM service.

systemctl restart LCM

Note: If the logging configuration is removed for support log collection, it must be added back immediately after that. Otherwise, there is a risk that LCM memory usage will be too high, and in scale environments could lead to OOM.