How to collect logs for high CPU utilization issues on the Linux platform

Products

• Carbon Black Cloud Enterprise EDR (formerly Cb Threathunter)

Issue/Introduction

Collect pertinent information to help troubleshoot high CPU utilization issues

Environment

• Carbon Black Cloud Linux Sensor: All Supported Versions
• Linux: All Supported Versions

Resolution

1. Determine the baseline for what is considered "typical utilization", for instance: "My application normally handles X transactions/second, with the sensor installed, it handles less transactions/second”
2. Ensure the utility "perf" is installed 
3. Save the following code as a bash file, then run it from the affected machine as superuser, while the issue is present: 

   #!/bin/bash
   
   DIRNAME=$(hostname)_cbc-perf-$(date +%Y-%m-%d_%H-%M-%S)
   mkdir $DIRNAME
   cd $DIRNAME
   
   echo "VMWare Carbon Black Cloud - Performance Metrics Gathering Script"
   echo "Working..."
   
   top -b -n 10 > cbc-kmod.top
   ps -efT > cbc-kmod.ps
   ps -efo uid,pid,pcpu,rss,spid,ppid,c,stime,tty,time,comm > cbc-cpu.ps
   
   #vmstat output 
   vmstat 3 10 > cbc.vmstat 
   
   #iostat output 
   iostat 3 10 > cbc.iostat
   
   #Backup kptr_restrict
   cp /proc/sys/kernel/kptr_restrict ./kptr_restrict_$(date +%Y-%m-%d_%H-%M-%S)
   
   # Get kptr_restrict value
   old_kptr_restrict=$(cat /proc/sys/kernel/kptr_restrict)
   
   # Disable kptr_restrict for a moment
   echo 0 > /proc/sys/kernel/kptr_restrict
   
   # Record everything for 15 seconds
   echo "Sleeping for 15ish seconds ..."
   perf record -o cbc-perf_sleep_15.data -F 99 -a -g sleep 15
   
   # Dump human readable data to file for easier reading
   perf report -i cbc-perf_sleep_15.data --hierarchy > cbc-perf_sleep_15.stdio
   
   # Copy kallsyms for kmod addresses
   cp /proc/kallsyms .
   
   cd ..
   
   # Restore kptr_restrict
   echo "${old_kptr_restrict}" > /proc/sys/kernel/kptr_restrict
   
   tar -zcf $DIRNAME.tgz $DIRNAME
   rm -rf $DIRNAME
   
   echo "Wrote $DIRNAME.tgz"
   echo "Please share $DIRNAME.tgz with support"
   

4. The script will output a compressed file with a .tgz extension, please collect the sensor logs as well as the tgz file and provide to support.
   
   • How to Collect Sensor logs locally (Linux)