Memory information collection script (cbs_meminfo.pl)

book

Article ID: 167763

calendar_today

Updated On:

Products

XOS

Issue/Introduction

This article describes a memory information collection script (cbs_meminfo.pl) that can be used to investigate out of memory (OOM) conditions.If an out of memory condition occurs, the oom-killer will begin to terminate or kill processes and daemons, and eventually the system will reboot due to the out of memory condition.

Cause

Goal: To provide a tool to use in investigating the contributing factors to an out of memory condition.

Resolution

Using cbs_meminfo.pl to diagnose OOM Conditions on APM

If a system experiences repeated out of memory conditions, you can use the script attached to this article to help investigate the factors that may contribute to the memory leak. The script will collect memory information from /proc/meminfo and /proc/slabinfo on a continuous 5-minute interval.

If you have experienced an out of memory (OOM) condition on the APM, use the attached .tar file and follow the instructions below to collect APM memory utilization information. You can then use the information to help identify memory slab allocations that are used consistently and are possibly "leaking", potentially causing the OOM condition.

Note: For an educational summary on Linux memory in general, see KB Article #5323.

Memory info collection script

    cbs_meminfo.pl.tar
    md5sum: 2d693d9f668ebf14376547222d86d85a

Installation and execution

1.   Copy the cbs_meminfo.pl.tar file to the CPM. Untar the file, make the file executable, and place it in a folder that is accessible from all vap-members.

    tar -xvf cbs_meminfo.pl.tar
    chmod +x cbs_meminfo.pl
    cp cbs_meminfo.pl /tftpboot/fw_common/


2.   Log into each VAP member and copy the script file from the common root directory to the APM's /root directory.

    rsh fw_<vapindex>
    cp /fw_common/cbs_meminfo.pl /root/


3.   From each APM, issue the following command to start the script in the background.
 
/root/cbs_meminfo.pl & <enter>
 
You should be returned the APM prompt.

4.   Confirm the script is running in the background using the following command.
 
ps -ef | grep cbs_meminfo
 
You should see the process running in the process list. For example:
 
root 23567 22812 0 12:32 pts/0 00:00:00 /usr/bin/perl /root/cbs_meminfo.pl
 
5.   Exit the VAP member.

6.   Repeat steps 2-5 for any remaining vap-members.


All the statistics are recorded and appended to:
 
/tftpboot/fw_<vapindex>/root/cbs_meminfo.txt
 
When the process is stopped or if the txt file reaches 50 MB, a time-stamped gzip of the cbs_meminfo.txt file will be placed into the tar file named:
 
/tftpboot/fw_<vapindex>/root/cbs_meminfo_archive.tar
 

Using cbs_meminfo.pl to diagnose OOM Conditions on CPM


1.   Copy the cbs_meminfo.pl.tar file to the CPM under your home directory. Untar the file, make the file executable.

    tar -xvf cbs_meminfo.pl.tar
    chmod +x cbs_meminfo.pl



2.   From the CPM, issue the following command to start the script in the background.

/[HOME]/cbs_meminfo.pl & <enter>
 
You should be returned the prompt.

3.   Confirm the script is running in the background using the following command.

ps -ef | grep cbs_meminfo
 
You should see the process running in the process list. For example:

root 23567 22812 0 12:32 pts/0 00:00:00 /usr/bin/perl /root/cbs_meminfo.pl 
--- 
All the statistics are recorded and appended to:

/[HOME]/cbs_meminfo.txt 
 
When the process is stopped or if the txt file reaches 50 MB, a time-stamped gzip of the cbs_meminfo.txt file will be placed into the tar file named:

/[HOME]/cbs_meminfo_archive.tar
 

Example of file contents

The following information will be recorded and appended to the cbs_meminfo.txt file every 5-minutes:

START_ITERATION
DATE: 16Apr09_12.37.04

UPTIME: 12:37:04 up 3:10, 1 user, load average: 0.06, 0.02, 0.00

START_VMSTAT_OUTPUT
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 3285944 0 66248 0 0 0 0 133 38 0 0 99 0
END_VMSTAT_OUTPUT
START_MEMINFO
total: used: free: shared: buffers: cached:
Mem: 4147269632 781582336 3365687296 0 0 67837952
Swap: 0 0 0
MemTotal: 4050068 kB
MemFree: 3286804 kB
MemShared: 0 kB
Buffers: 0 kB
Cached: 66248 kB
SwapCached: 0 kB
Active: 90048 kB
ActiveAnon: 48000 kB
ActiveCache: 42048 kB
Inact_dirty: 18428 kB
Inact_laundry: 5748 kB
Inact_clean: 0 kB
Inact_target: 22844 kB
HighTotal: 2293760 kB
HighFree: 2014036 kB
LowTotal: 1756308 kB
LowFree: 1272768 kB
SwapTotal: 0 kB
SwapFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB
END_MEMINFO
START_SLABINFO
slabinfo - version: 1.1 (SMP)
kmem_cache 80 80 244 5 5 1 : 2016 504
clip_arp_cache 0 0 256 0 0 1 : 2016 504
ip_fib_hash 448 448 32 4 4 1 : 2016 504
ip_mrt_cache 0 0 128 0 0 1 : 2016 504
tcp_tw_bucket 120 120 128 4 4 1 : 2016 504
tcp_bind_bucket 448 448 32 4 4 1 : 2016 504
tcp_open_request 90 90 128 3 3 1 : 2016 504
inet_peer_cache 116 116 64 2 2 1 : 2016 504
secpath_cache 0 0 128 0 0 1 : 2016 504
xfrm_dst_cache 0 0 256 0 0 1 : 2016 504
ip_dst_cache 165 165 256 11 11 1 : 2016 504
arp_cache 60 60 256 4 4 1 : 2016 504
flow_cache 0 0 128 0 0 1 : 2016 504
blkdev_requests 0 0 128 0 0 1 : 2016 504
nfs_write_data 1274 1770 384 144 177 1 : 992 248
nfs_read_data 1610 1610 384 161 161 1 : 992 248
nfs_page 3576 4080 128 134 136 1 : 2016 504
journal_head 0 0 48 0 0 1 : 2016 504
revoke_table 0 0 12 0 0 1 : 2016 504
revoke_record 0 0 32 0 0 1 : 2016 504
ext3_xattr 0 0 44 0 0 1 : 2016 504
kioctx 0 0 128 0 0 1 : 2016 504
kiocb 0 0 128 0 0 1 : 2016 504
dnotify_cache 0 0 20 0 0 1 : 2016 504
file_lock_cache 160 160 96 4 4 1 : 2016 504
async_poll_table 0 0 140 0 0 1 : 2016 504
fasync_cache 200 200 16 1 1 1 : 2016 504
uid_cache 336 336 32 3 3 1 : 2016 504
skbuff_head_cache 2764 3772 168 159 164 1 : 2016 504
sock 240 240 1408 48 48 2 : 480 120
sigqueue 5470 5974 132 206 206 1 : 2016 504
kiobuf 0 0 128 0 0 1 : 2016 504
cdev_cache 290 290 64 5 5 1 : 2016 504
bdev_cache 116 116 64 2 2 1 : 2016 504
mnt_cache 174 174 64 3 3 1 : 2016 504
inode_cache 2065 2065 512 295 295 1 : 992 248
dentry_cache 3150 3150 128 105 105 1 : 2016 504
dquot 0 0 128 0 0 1 : 2016 504
filp 1740 1740 128 58 58 1 : 2016 504
names_cache 964 964 4096 964 964 1 : 480 120
buffer_head 35 35 108 1 1 1 : 2016 504
mm_struct 1410 1410 384 141 141 1 : 992 248
vm_area_struct 6328 7840 68 124 140 1 : 2016 504
fs_cache 2088 2088 64 36 36 1 : 2016 504
files_cache 1260 1260 512 180 180 1 : 992 248
signal_cache 1740 1740 64 30 30 1 : 2016 504
sighand_cache 835 835 1408 167 167 2 : 480 120
pte_chain 10302 11310 128 344 377 1 : 2016 504
pae_pgd 2088 2088 64 36 36 1 : 2016 504
size-131072(DMA) 0 0 131072 0 0 32 : 0 0
size-131072 7 8 131072 7 8 32 : 0 0
size-65536(DMA) 0 0 65536 0 0 16 : 0 0
size-65536 9 10 65536 9 10 16 : 0 0
size-32768(DMA) 0 0 32768 0 0 8 : 0 0
size-32768 46 47 32768 46 47 8 : 0 0
size-16384(DMA) 0 0 16384 0 0 4 : 0 0
size-16384 6 9 16384 6 9 4 : 0 0
size-8192(DMA) 0 0 8192 0 0 2 : 0 0
size-8192 250 264 8192 250 264 2 : 0 0
size-4096(DMA) 0 0 4096 0 0 1 : 480 120
size-4096 4281 4281 4096 4281 4281 1 : 480 120
size-2048(DMA) 0 0 2048 0 0 1 : 480 120
size-2048 1004 1124 2048 502 562 1 : 480 120
size-1024(DMA) 0 0 1024 0 0 1 : 992 248
size-1024 2524 2524 1024 631 631 1 : 992 248
size-512(DMA) 0 0 512 0 0 1 : 992 248
size-512 1344 1592 512 168 199 1 : 992 248
size-256(DMA) 0 0 256 0 0 1 : 2016 504
size-256 555 555 256 37 37 1 : 2016 504
size-128(DMA) 0 0 128 0 0 1 : 2016 504
size-128 8220 8220 128 274 274 1 : 2016 504
size-64(DMA) 0 0 128 0 0 1 : 2016 504
size-64 420 420 128 14 14 1 : 2016 504
size-32(DMA) 0 0 64 0 0 1 : 2016 504
size-32 870 870 64 15 15 1 : 2016 504
END_SLABINFO
END_ITERATION

 

Workaround

N/A