CDM not capturing correct Disk IO

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

For a server alert we are getting is “Disk IOstat utilization (pu) on sda is now 100.02% which is >= warning threshold (60.00%).

On the Server the output of command iostat -p sda is as below:

avg-cpu: %user %nice %system %iowait %steal %idle
4.35 0.00 2.97 6.15 0.00 86.52

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 908.63 7573.93 7452.69 67063903930 65990375559
sda1 0.00 0.05 0.00 431107 23118
sda2 908.62 7573.88 7452.69 67063432699 65990352441

Environment

Release:
Component: CAUIM

Resolution

I think there is some misunderstanding how cdm collects relevant io data of system.

Below is summary about iostat and how cdm is collecting iostat for linux boxes.
IOSTAT : iostat commands generate reports having In/out stats regarding disks. Each report generated by iostat shows io matrices based on the time covers since the previous report. so in a particular case first iostat report will have time slice from reboot to the command execution. and next report will have time slice from first report to second report and so on.
 iostat -x -d "timeinterval" "noOfReports" will return noOfReports with each one at defined "timeinterval" and will show reports based on the time you provided as timeinterval.

cdm usages command "iostat -x -d 10 2" to collect iostat which will generate two reports at time interval of 10 seconds. where first report should not be considered as it is not relevant and not certain when the last time this command ran on system. cdm picks second run report which is relavant to first report with 10 seconds.

As an example cdm will pics Bold letter iostat metrics to collect metrics.
[root@user ~]# iostat -x -d 10 2
Linux 3.10.0-862.14.4.el7.x86_64 (CHAMO08-CenVM01)      01/04/2019      _x86_64_        (4 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.01    0.63    0.16    15.21     3.92    48.65     0.00    0.86    0.48    2.36   0.37   0.03
dm-0              0.00     0.00    0.49    0.17    13.29     3.66    51.32     0.00    1.05    0.56    2.43   0.40   0.03
dm-1              0.00     0.00    0.01    0.00     0.14     0.00    47.40     0.00    0.19    0.19    0.00   0.16   0.00
dm-2              0.00     0.00    0.01    0.00     0.07     0.13    62.17     0.00    0.58    0.37    5.75   0.33   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00