High CPU utilization and sluggishness in SDDC Manager when FIPs is enabled

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

In VMware Cloud Foundation environments, the SDDC Manager VM may exhibit a gradual or sudden ramp-up in CPU utilization, reaching sustained levels of 100%. 

• FIPs is enabled
• SDDC Manager UI is slow or unresponsive.
• CPU utilization on the SDDC Manager VM ramps up to 100% and remains high.
• No active tasks are visible in the UI during the high CPU period.
• Sluggishness when executing commands via SSH.

Environment

SDDC Manager 5.x

Cause

The high CPU utilization is primarily driven by the interaction between FIPs and SDDC Manager API calls.  This causes the following symptoms:

1. Operations Manager JVM Heap Exhaustion: The Operations Manager service reaches its default maximum heap limit (2GB), causing excessive Garbage Collection (GC) activity that consumes CPU cycles.
2. LCM Logger Churn: Frequent log-rotation churn (up to 60-70 times per day) in the Lifecycle Management (LCM) compatibility logger leads to thread pileups and increased overhead.

Resolution

This issue is fixed in SDDC Manager 9.1

Workaround options:

Option 1:

1. Increase Operations Manager JVM Heap

   1. Log in to the SDDC Manager VM as vcf and switch to root.

   2. Open the configuration file for editing:

      1. vi /etc/vmware/vcf/operationsmanager/vcf-operations-manager.conf

          

         1. Locate the -Xmx2G parameter and change it to -Xmx4G.  Ensure there is enough free memory to make this change

         2. Save and close the file.

         3. systemctl restart operationsmanager

2. Silence LCM Compatibility Logger

   1. Edit the LCM application properties file:

      1. vi /etc/vmware/vcf/lcm/lcm-app/conf/application-prod.properties

          

      2. Add the following line to the end of the file: 

         logging.level.com.vmware.vcf.compatibility=OFF

          

      3. Save and close the file.

      4. systemctl restart lcm

Option 2:

Disable FIPs in SDDC Manager

Option 3:

Schedule reboots or restarts of the SDDC Manager services to clear issue: 

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Additional Information

For additional information Contact Support