Upgrading to ESXi 6.5 U1 displays abnormal DAVG and KAVG data in ESXTOP

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Running esxtop displays abnormal DAVG/cmd and KAVG/cmd values.
• Running the below command on the affected ESXi hosts display results as below:

 

vm-support -p -d 1800 -i 5 -w /vmfs/volumes/<VolumeNAme>/

Note: The below results are just examples. Values may vary in different scenarios.

​​
Sample of LUNs that showed abnormal values:

LUN ATS ATSF
naa.############################6733 436046 41
naa.############################6754 438974 385
naa.############################6755 436387 24
naa.############################6757 438984 169
naa.############################6759 435740 23
naa.############################6762 435732 25
naa.############################6764 437415 30
naa.############################6766 435739 29
 

Sample of LUNs that did not show abnormal values:

LUN ATS ATSF
naa.############################6835 785 0
naa.############################6837 629 0
naa.############################6839 629 0
naa.############################6841 630 0
naa.############################6843 629 0
naa.############################6845 629 0
naa.############################6847 644 0
naa.############################6849 633 0

• Repeat offenders are typically busier LUNs in general.

Example of a few LUNs:

BAD

NETAPP Fibre Channel Disk (naa.############################6831)
/> cat /storage/scsifw/devices/naa.############################6831/stats
Statistics {
   Successful Commands:25656883
   Failed Commands:597
   Blocks Read:1591656653
   Failed Blocks Read:0
   Blocks Written:1596779952
   Failed Blocks Written:0
   Read Operations:12494250
   Failed Read Operations:0
   Write Operations:12695671

   total ATS cmds:436086
   total ATS failures:30
   total zero cmds:139
   total zero failures:0
   total blocks zeroed:284672

GOOD

NETAPP Fibre Channel Disk (naa.############################6833)
cat /storage/scsifw/devices/naa.############################6833/stats
Statistics {
   Successful Commands:46839
   Failed Commands:569
   Blocks Read:828213
   Failed Blocks Read:0
   Blocks Written:1802
   Failed Blocks Written:0
   Read Operations:12857
   Failed Read Operations:0
   Write Operations:1802
...
total ATS cmds:639
   total ATS failures:0
   total zero cmds:0
   total zero failures:0
   total blocks zeroed:0

Pattern seen for effected LUNs:

CMDS/s: 0.19
DAVG/cmd: 18446744073709552.00
KAVG/cmd: -18446744073709552.00

CMDS/s: 0.38
DAVG/cmd: 9223372036854776.00
KAVG/cmd: -9223372036854776.00

Environment

VMware vSphere ESXi 6.5 Update 1

Resolution

This issue is resolved in ESXi 6.5 U2, available at Broadcom Downloads.