LCM service becomes unresponsive/resource starving/slow in response in idle scaled virtual machines

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• LCM service demands more system resources.
• LCM service becomes unresponsive.  
• LCM service becomes slow in response to API calls; API timeout happens.

Environment

VMware Cloud Foundation 4.2.x
VMware Cloud Foundation 4.3.x
VMware Cloud Foundation 4.0.x
VMware Cloud Foundation 3.x
VMware Cloud Foundation 4.1
VMware Cloud Foundation 2.x
VMware Cloud Foundation 4.4.x

Cause

This is due to the Inventory Sync Service of LCM periodically caching the inventory of the SDDC manager Virtual Machine and each time of caching will generate some garbage due to JVM does not completely release the memory for the reused objects even if LCM always clears these objects before each sync. This issue is more likely to happen in a scaled setup.

Resolution

Currently there is no resolution. The issue will be resolved in future VCF release.

Workaround:

Although users can simply restart the LCM service by the following command to enforce JVM to do garbage collection, it would be better if we can prevent this issue from happening.

Restart LCM service
systemctl restart lcm

Therefore, Inventory Sync Service has exposed the following property to let the user set the inventory sync interval. By default, the interval is 300000 milliseconds (5 minutes). Please be aware of the consequence of this increment - the inventory cache may be stale. Thus, it is recommended to just slightly increase it.  
 
Here is an example to set the interval to 360000 milliseconds (6 minutes), it will reduce the consumption of system resources of LCM to a considerable extent in a scaled setup.

1. Log in SDDC Virtual Machine as root user

su

2. Add "lcm.inventory.sync.interval= 360000 " in /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties

echo "lcm.inventory.sync.interval= 360000 " >> /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties

 

3. Restart LCM service

systemctl restart lcm

 

4. To verify the new setting is being taken up by LCM service, observing lcm-debug.log

grep "Starting logical inventory sync" /var/log/vmware/vcf/lcm/lcm-debug.log