How to collect logs for high CPU utilization issues on the Linux platform
book
Article ID: 292325
calendar_today
Updated On:
Products
Carbon Black Cloud Enterprise EDR (formerly Cb Threathunter)
Issue/Introduction
Collect pertinent information to help troubleshoot high CPU utilization issues
Environment
Carbon Black Cloud Linux Sensor: All Supported Versions
Linux: All Supported Versions
Resolution
Determine the baseline for what is considered "typical utilization", for instance: "My application normally handles X transactions/second, with the sensor installed, it handles less transactions/second”
Ensure the utility "perf" is installed
Save the following code as a bash file, then run it from the affected machine as superuser, while the issue is present:
#!/bin/bash
DIRNAME=$(hostname)_cbc-perf-$(date +%Y-%m-%d_%H-%M-%S)
mkdir $DIRNAME
cd $DIRNAME
echo "VMWare Carbon Black Cloud - Performance Metrics Gathering Script"
echo "Working..."
top -b -n 10 > cbc-kmod.top
ps -efT > cbc-kmod.ps
ps -efo uid,pid,pcpu,rss,spid,ppid,c,stime,tty,time,comm > cbc-cpu.ps
#vmstat output
vmstat 3 10 > cbc.vmstat
#iostat output
iostat 3 10 > cbc.iostat
#Backup kptr_restrict
cp /proc/sys/kernel/kptr_restrict ./kptr_restrict_$(date +%Y-%m-%d_%H-%M-%S)
# Get kptr_restrict value
old_kptr_restrict=$(cat /proc/sys/kernel/kptr_restrict)
# Disable kptr_restrict for a moment
echo 0 > /proc/sys/kernel/kptr_restrict
# Record everything for 15 seconds
echo "Sleeping for 15ish seconds ..."
perf record -o cbc-perf_sleep_15.data -F 99 -a -g sleep 15
# Dump human readable data to file for easier reading
perf report -i cbc-perf_sleep_15.data --hierarchy > cbc-perf_sleep_15.stdio
# Copy kallsyms for kmod addresses
cp /proc/kallsyms .
cd ..
# Restore kptr_restrict
echo "${old_kptr_restrict}" > /proc/sys/kernel/kptr_restrict
tar -zcf $DIRNAME.tgz $DIRNAME
rm -rf $DIRNAME
echo "Wrote $DIRNAME.tgz"
echo "Please share $DIRNAME.tgz with support"
The script will output a compressed file with a .tgz extension, please collect the sensor logs as well as the tgz file and provide to support.