Performance issues with VM operations

Products

• VMware vCenter Server
• VMware vCenter Server 8.0
• VMware vSphere ESXi 8.0
• VMware vSphere ESXi

Issue/Introduction

• Operations such as Power on/Off VMs do not complete.
• Creation of a new DVS (Distributed Virtual Switch) fails citing a datastore issue.
• Performance issue on the VM.
• Application issue on the VM
• High load on the storage array.
• VM Cloning takes a longer time

Performance deteriorated on storage devices seen in /var/log/vmkwarning and /var/log/vmkernel

vmkwarning.log
####-##-##T##:##Z cpu1:#######)WARNING: ScsiDeviceIO: 1513: Device naa.################################ performance has deteriorated. I/O latency increased from average value of 3492 microseconds to 320004 microseconds.
..
..
####-##-##T##:##Z cpu30:#######)WARNING: ScsiDeviceIO: 1513: Device naa..################################ performance has deteriorated. I/O latency increased from average value of 3492 microseconds to 645885 microseconds.
####-##-##T##:##Z cpu33:2103416)ALERT: hostd performance has degraded due to high system latency

vmkernel.log
####-##-##T##:##Z cpu29:####### opID=#######)FSAts: 365: [Req mode: 1] Lock at 7471104 on vol 'datastore_name' is not free after 6 miscompare retries
####-##-##T##:##Z cpu29:####### opID=#######)DLX: 2650: vol <datastore_name>, lock at 7471104: Lock type: 10C00001. Exclusive Lock(s) held on a file on volume ########-#######-####-############. numHolders:0 gblNumHolders:0, volume state 10, APD s$
####-##-##T##:##Z cpu29:####### opID=fcc1c967)[type 10c00001 offset 7471104 v 241857, hb offset 4161536
gen 27341, mode 0, owner 00000000-00000000-0000-000000000000 mtime 8525737
num 0 gblnum 0 gblgen 0 gblbrk 0] alloc owner 0
..
..
####-##-##T##:##Z cpu2:#######)ScsiDeviceIO: 4167: Cmd(0x45d90f0d4e48) 0x89, CmdSN 0x2163b3 from world 2101333 to dev "naa..################################" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0
####-##-##T##:##Z cpu31:####### opID=#######)DLX: 5465: vol <datastore_name>, lock at 7471104: Lock type: 10C00001. [Req mode 1] try lock error: Atomic test and set of disk block returned false for equality:

Cause

High IO latency and ATS failures can cause the hostd service of ESXi to enter a degraded state causing the hosts to be unable to perform VM related operations and infrastructure update related operations such as creating a DVS.

 

Resolution

Engage Fabric/Hardware/storage vendor to investigate the cause of the failure.