Abnormal DAVG and KAVG values observed during VAAI operations
Article ID: 344478
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Symptoms:
When performing VAAI operations (such as clone, template deploy, or eagerzeroedthick disk creation), abnormal DAVG and/or KAVG values may be observed on the disk (u) or device (d) views within esxtop such as:
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
mpx.vmhba0:C0:T0:L0 - 1 - 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
naa.6006016061711d0080eebbe70cb1dd11 - 64 - 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
naa.60060160e4a02c00a06219199aa3e011 - 64 - 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
naa.60060160e4a02c00e4cba4ebaba9e011 - 64 - 8 0 12 0.12 62.15 0.39 20.85 0.00 0.06 0.15 94.91 95.06 0.00
naa.600605b002f26c6014f779d81b90cc94 - 128 - 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
9:02:45pm up 2:36, 350 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.00, 0.00, 0.00
ADAPTR PATH NPTH CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
vmhba0 - 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vmhba1 - 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vmhba2 - 10 64.21 0.00 21.47 0.00 0.07 96.74 0.00 96.75 0.00
vmhba3 - 10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vmhba33 - 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
When performing VAAI operations (such as clone, template deploy, or eagerzeroedthick disk creation), abnormal DAVG and/or KAVG values may be observed on the disk (u) or device (d) views within esxtop such as:
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
mpx.vmhba0:C0:T0:L0 - 1 - 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
naa.6006016061711d0080eebbe70cb1dd11 - 64 - 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
naa.60060160e4a02c00a06219199aa3e011 - 64 - 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
naa.60060160e4a02c00e4cba4ebaba9e011 - 64 - 8 0 12 0.12 62.15 0.39 20.85 0.00 0.06 0.15 94.91 95.06 0.00
naa.600605b002f26c6014f779d81b90cc94 - 128 - 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
9:02:45pm up 2:36, 350 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.00, 0.00, 0.00
ADAPTR PATH NPTH CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
vmhba0 - 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vmhba1 - 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vmhba2 - 10 64.21 0.00 21.47 0.00 0.07 96.74 0.00 96.75 0.00
vmhba3 - 10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vmhba33 - 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Environment
VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.5
VMware ESX 4.1.x
VMware vSphere ESXi 5.1
VMware ESXi 4.1.x Installable
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.0
VMware vSphere ESXi 6.5
VMware ESX 4.1.x
VMware vSphere ESXi 5.1
VMware ESXi 4.1.x Installable
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.0
Resolution
When VAAI commands are issued via VAAI Filter, there are actually 2 commands sent. These are top-layer commands which are issued and are never sent to the actual device (they stay within the ESX kernel). These commands are intercepted by the VAAI filter and the VAAI plugin, and are replaced by the vendor-specific commands, which are issued to the device.
This is why esxtop shows device statistics for the top-level commands only, and as a result the values for DAVG and KAVG seem unusual when compared to results obtained when VAAI is not enabled .
In this instance (and only for this instance), the DAVG and KAVG observed in esxtop should not be interpreted as a performance issue, absent of any other symptoms. The DAVG and KAVG values observed for non-VAAI workloads are still valid.
Additional Information
Feedback
Yes
No
Powered by
