Troubleshooting suggestions for High CPU Utilization

Products

• VMware Smart Assurance
• Network Observability

Issue/Introduction

This article talks about troubleshooting steps for High CPU Utilization by Smarts processes. 

Environment

All Supported Smarts releases

Resolution

Troubleshooting "High CPU  Utilization" is always tricky and complex. Here are some suggestions on what to look for and how they can impact CPU:

Factors that influence CPU Utilization:

• Processor' Speed
  • www.cpuboss.com
    • This site can provide comparison and details on physical CPU that is being used
  • Single vs Multi core impact
    • processes such as discovery or polling rely on single core not multiple core
  • If on a Linux, what does  top  is telling you?
    • You may consider running top  p <smarts-ip-server-pid> for about an hour and take a look at the cpu utilization
      
      
• VM or Physical
  • If VM, are there other apps/VMs running on the ESXi?
    • IS CPU being shared with other VMs
  • Is the CPU limited for Smarts VM?
    • Smarts doesn't require high cpu all the time, however it does when it comes to discovery and polling
      
      
• 3rd Party API or script components
  • Run the following command to get the list of processes running on the server:
    

    ps aux | head -1; ps aux | sort -rnk 4 >> CaseNumber.log

  • If there are multiple invocations of external scripts towards a singular domain manager via sm_adapter, sm_perl, shell script or any 3rd party script and are not terminated gracefully, this may lead to high CPU growth. 
    
    
• Topology:
  • Number of instrumented devices?
    • To get a list of devices run:
      

      ./sm_tpmgr -s <INCHARGE-AMPM> --sizes

    • Shouldn't be more than 50K.
  • Number of managed devices
    • To get a list of managed devices, you can run:
      

      ./dmctl-s <INCHARGE-AMPM> geti UnitaryComputerSystem | tee | wc -l

  • Number of Discovery threads, shouldn t be more than 40 threads but the more threads the more CPU
    • Edit the discovery.conf file and change the value of the numberProbeThreads to a higher or lower value then restart the IP server . The default value is 10 
  • Number of Polling thread impact CPU utilization. It should be increased by 10 increments but not beyond 50*
    • Edit file with 
      

      BASEDIR/IP/smarts/bin/sm_edit conf/icf/FULL_DEVSTAT.import.

      
      
    • In this file, find the section, "ICIP_SNMPAccessorInterface::DEVSTAT-SNMP-Poller", which controls the number of threads used for SNMP polling.
    • Update the key, "num_threads" to the value you require, not to exceed 50.
    • Restart the server to reflect these changes
    • *Please Note:  If you also have the Smarts Solution Pack being used in your environment do not increase the number of threads beyond 25.  This may cause connections to timeout in a CLOSED_WAIT status and cause your environment to reach the File Descriptor limits and cause the IP domain to crash.
      
      
• To calculate how full the SNMP polling threads are, run the following command:

  • dmctl -s <DOMAIN> invoke SNMP_AccessorInterface::DEVSTAT-SNMP-Poller reset_statistics

  • After at least two of the network polling intervals have passed, run the following DMCTL command to retrieve the SNMP-Poller statistics:
    

    dmctl -s <DOMAIN> get SNMP_AccessorInterface::DEVSTAT-SNMP-Poller

  • Run the following command to generate a complete list of performance metrics:
    

    sm_tpmgr -s <DOMAIN> --show-dm-processes

  • Late polling should be around 1-2 seconds
  • "Get per sec" typically less than 20-30 seconds

Additional Information

The above settings are just some recommended areas to review and monitor if there are any High CPU Utilization concerns pertaining to Smarts IP Manager. For further detail and in-dept review, please refer to Smarts Admin Guide, Smarts scalability Guide,Smarts Deployment Guide.

If you still need further assistance to address this concern, please make sure you collect the above data when contacting Support Team.