Memory information collection script extended with "top" ouput and "ps" output sorted by RSS memory utilization

book

Article ID: 167816

calendar_today

Updated On:

Products

CPM Chassis XOS NPM APM

Issue/Introduction

This script collects the data collected by original cbs_meminfo and adds ps output sorted by RSS, and top output.

Sometimes it might also be useful to collect user space processes memory allocation or CPU utilization data which the original cbs_meminfo does not provide.

 

Resolution

If a system (CPMs or APMs) experiences repeated out of memory conditions or high CPU, you can use the script below, also  attached to this article, to help investigate the factors that may contribute to the issue. The script will collect memory information from /proc/meminfo and /proc/slabinfo, ps and top output on a continuous 5-minute interval. The 5-minute interval is defined at the end of the script by the sleep time in seconds sleep (300);.

Note:
-If the memory or high CPU issue is sporadic and last less than 5 minutes, you will have to change the 'sleep time' accordingly. For example, if the issue disappear after 30 seconds you might want to change the sleep time to 20 or 15 seconds. The idea is to take the data while the issue is happening.
-The script will GZ the file automatically when it reaches 50M to avoid disk utilization issues.

Follow the installation and execution procedure for the original script https://support.symantec.com/en_US/article.TECH243683.html, but using the script code below:

#!/usr/bin/perl
use strict;
 
my $vm_file = "vmstat.txt";
my $cbs_meminfo = "cbs_meminfo.txt";
my $run;
my $date;
my $uptime;
my $file_size;
 
if (-e $cbs_meminfo){
        $date = `date +%d%b%y_%k.%M.%S`;
        chomp($date);
        $run = `tar -czvf cbs_meminfo_bak_"$date".tar.gz $cbs_meminfo`;
 
        if (-e "cbs_meminfo_archive.tar"){
                $run = `tar -f cbs_meminfo_archive.tar -r cbs_meminfo_bak_*`;
        } else {
                $run = `tar -cf cbs_meminfo_archive.tar cbs_meminfo_bak_*`;
        }
 
        $run = `rm -rf $cbs_meminfo cbs_meminfo_bak_*`;
}
 
if (-e $vm_file){
        $run = `rm -rf $vm_file`;
}
 
while (1)
{
        $run = `echo START_ITERATION >> $cbs_meminfo`;
        $date = `date +%d%b%y_%k.%M.%S`;
        $uptime = `uptime`;
        $run = `echo DATE: "$date" >> $cbs_meminfo`;
        $run = `echo UPTIME: "$uptime" >> $cbs_meminfo`;
 
        $run = `vmstat >> $vm_file`;
 
        # summary all to one file
 
        $run = `echo START_VMSTAT_OUTPUT >> $cbs_meminfo`;
        $run = `cat $vm_file >> $cbs_meminfo`;
        $run = `echo END_VMSTAT_OUTPUT >> $cbs_meminfo`;
 
        $run = `echo START_MEMINFO >> $cbs_meminfo`;
        $run = `cat /proc/meminfo >> $cbs_meminfo`;
        $run = `echo END_MEMINFO >> $cbs_meminfo`;
 
        $run = `echo START_SLABINFO >> $cbs_meminfo`;
        $run = `cat /proc/slabinfo >> $cbs_meminfo`;
        $run = `echo END_SLABINFO >> $cbs_meminfo`;
 
       $run = `echo START_PS_RSS >> $cbs_meminfo`;
       $run = `ps aux --sort -rss  >> $cbs_meminfo`;
       $run = `echo END_PS_RSS >> $cbs_meminfo`;
  
       $run = `echo START_TOP >> $cbs_meminfo`;
       $run = `top -b | head -25  >> $cbs_meminfo`;
       $run = `echo END_TOP >> $cbs_meminfo`;
 
        $run = `echo END_ITERATION >> $cbs_meminfo`;
        $run = `rm -rf $vm_file`;
        $file_size = -s $cbs_meminfo;
 
        if ($file_size > 52428800){
                chomp($date);
                $run = `tar -czvf cbs_meminfo_bak_"$date".tar.gz $cbs_meminfo`;
                $run = `rm -rf $cbs_meminfo`;
        }
 
        sleep (300);
}

Workaround

N/A

Attachments

meminfo_with_top.rar get_app