Performance issues with VM operations

Products

• VMware vCenter Server
• VMware vCenter Server 8.0
• VMware vSphere ESXi 8.0
• VMware vSphere ESXi

Issue/Introduction

• Operations such as Power On/Power Off do not complete or remain in a 'stuck' state

• Creation of a new Distributed Virtual Switch (DVS) fails due to datastore issues

• Performance deterioration or application performance is impacted within the Guest OS

• Tasks remain visible in the vCenter 'Recent Tasks' pane even after a vCenter reboot

• VM Cloning or Snapshot operations take an excessively long time

• There are 'performance deteriorated' errors on storage devices seen in /var/log/vmkwarning.log and /var/log/vmkernel.log

In /var/run/log/vmkwarning.log
[YYYY-MM-DDTHH:MM:SS] cpuXY:WARNING: ScsiDeviceIO: 1513: Device naa.######################## performance has deteriorated. I/O latency increased from average value of XYZ microseconds to ABC microseconds.
[YYYY-MM-DDTHH:MM:SS] cpuXY:WARNING: ScsiDeviceIO: 1513: Device naa.######################## performance has deteriorated. I/O latency increased from average value of XYZ microseconds to ABC microseconds.
[YYYY-MM-DDTHH:MM:SS] cpuXY:ALERT: hostd performance has degraded due to high system latency

In /var/run/log/vmkernel.log
[YYYY-MM-DDTHH:MM:SS] cpuXY: [Req mode: 1] Lock at ＃####### on vol 'datastore_name' is not free after # miscompare retries
[YYYY-MM-DDTHH:MM:SS] cpuXY: 2650: vol <datastore_name>, lock at abcdef: Lock type: 10C00001. Exclusive Lock(s) held on a file on volume ########-#######-####-############. numHolders:0 gblNumHolders:0, volume state 10, APD s$
[YYYY-MM-DDTHH:MM:SS] cpu2:ScsiDeviceIO: 4167: Cmd(0xabcd 0x89, CmdSN 0xabcde from world 012345 to dev "naa.######################## failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0
[YYYY-MM-DDTHH:MM:SS] DLX: 5465: vol <datastore_name>, lock at abcabc: Lock type: 10C00001. [Req mode 1] try lock error: Atomic test and set of disk block returned false for equality:

• ESXTOP data shows DAVG , KAVG , GAVG and QAVG values reports high.

• IO and driver aborts with state in doubt events are observed in the /var/run/log/vmkernel.log

2026-08-31T02:00:52.421Z In(182) vmkernel: cpu97:2097394)ScsiDeviceIO: 4681: Cmd(0x45db329b7280) 0x89, cmdId.initiator=0x430bb534ca00 CmdSN 0x5e42727 from world 2097288 to dev "eui.#########################" failed H:0x5 D:0x0 P:0x0 Cancelled from device layer
2026-08-31T02:00:52.421Z In(182) vmkernel: cpu97:2097394) Cmd count Active:32 Queued:266
2026-08-31T02:00:52.422Z In(182) vmkernel: cpu35:2097343)ScsiDeviceIO: 4616: Cmd(0x45db3234dd80) 0x88, cmdId.initiator=0x43108e5e9080 CmdSN 0xffff9a07d3d083b0 from world 2123878 to dev "eui.#########################" failed H:0x8 D:0x0 P:0x0 Cancelled from
2026-08-31T02:00:52.422Z In(182) vmkernel: cpu35:2097343)device layer
2026-08-31T02:00:52.429Z Wa(180) vmkwarning: cpu45:2098862)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "eui.#######################" state in doubt; requested fast path state update...

 

Environment

VMware vSphere ESXi 8.x
VMware vSphere ESXi 9.x

Cause

High I/O latency and ATS failures can cause the hostd service of ESXi to enter a degraded state, causing the hosts to be unable to perform VM related operations and/or infrastructure update related operations such as creating distributed switches.

 

Resolution

Engage Hardware/Storage vendor to investigate the cause of the failure.

Additional Information

Japanese: VMの操作におけるパフォーマンスの問題