High CPU utilization on Solaris servers from CDM probe due to multiple "mdb -k" processes

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)
• CA Unified Infrastructure Management On-Premise (Nimsoft / UIM)

Issue/Introduction

CPU Usage on Solaris 10 servers running CDM probe is very high.   Checking the running processes we can see "mdb -k" running many times:

root@hostname[~]# ps -ef|grep mdb

    root  5103     1   1 12:24:31 ?         187:38 mdb -k

    root  4147     1   1 02:08:57 ?          11:39 mdb -k

    root 25004     1   1 01:28:58 ?          18:51 mdb -k

    root  9504     1   1 22:09:38 ?          57:56 mdb -k

    root 13357 13356   1 15:05:32 ?         143:30 mdb -k

    root 29126     1   1 18:43:19 ?          97:14 mdb -k

    root 28606     1   1 01:38:57 ?          17:22 mdb -k   

Environment

Component : UIM - CDM WITH IOSTAT

Cause

This is related to a known issue with Solaris 10 where sometimes the mdb process can take a long time to exit.

CDM will kill the process if it takes too long and this is controlled by a setting in cdm.cfg:

mem_buffer_timeout = 100

This defines how long the probe will wait for the mdk -k command to finish.

Resolution

Lower the timeout to a much lower number, e.g.:

   mem_buffer_timeout = 5

Setting this lower will allow the processes to time out instead of hanging.

Additional Information

https://serverfault.com/questions/383413/echo-memstat-mdb-k-very-slow-on-certain-machines