VM was restarted by vSphere HA due to high IO load caused by backup task
search cancel

VM was restarted by vSphere HA due to high IO load caused by backup task

book

Article ID: 426030

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • One of the running virtual machines crashed and was automatically restarted by HA (High Availability), in fdm.log:
[YYYY-MM-DDTHH:MM:SS] Db(167) Fdm[2102736]: [Originator@xxxx sub=FDM opID=WorkQueue-xxxxxxxx] New event: EventEx=com.vmware.vc.ha.VmRestartedByHAEvent vm=/vmfs/volumes/xxxxxxxx/xxxxxxxx/xxxxxxxx.vmx host=host-xxxxxx tag=host-xxxxxx:-xxxxxxxxx:x
  • Performing LWD-based snapshot sync fails, read IOs are aborted by VMkernel. In VMware.log:
[YYYY-MM-DDTHH:MM:SS] Er(02) worker-2328848 293b2adb LWD: Failed to write extent range [62458943,62458954] for the SES /vmfs/volumes/xxxxxxxx/xxxxxxxx/xxxxxxxx.ses.  Error 15: IO was aborted
[YYYY-MM-DDTHH:MM:SS] Wa(03) worker-2328848 293b2adb LWD: Failure writing to SES, snapshot must be canceled, disk 6187408BF0: IO was aborted (15)
[YYYY-MM-DDTHH:MM:SS] Wa(03) worker-2328848 293b2adb LWD: The disk 6187408BF0 has failed SES writes, additional chunk processing is not possible

[YYYY-MM-DDTHH:MM:SS] Er(02) worker-2133979 5ea2d801 LWD: Failed to read extent range [31294929,31294960], offset range [256368058368, 262144], from disk 6187408BF0 (capacity 547608330240), readTxn 10947. Error: 15: IO was aborted
[YYYY-MM-DDTHH:MM:SS] Wa(03) worker-2133979 5ea2d801 LWD: LwdIoTracker_CancelAllIOs: '11' IOs did not complete in the stipulated time for IoTracker 61ED0654A0, disk 6187408BF0
……
[YYYY-MM-DDTHH:MM:SS] Er(02) Upcall-c2e97 eeceb37e LWD: Failed to read extent range [31294684,31294707], offset range [256366051328, 196608], from disk 6187408BF0 (capacity 547608330240), readTxn 10946. Error: 18: Busy
[YYYY-MM-DDTHH:MM:SS] Wa(03) worker-2134172 eeceb37e LWD: Resetting SES IO status from 18 (Busy) to BUSY as IOs are not permitted, disk 6187408BF0
  • The datastore where the virtual machine located shows errors In the /var/log/vmkernel.log file of the ESXi host:
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu72:5586470)ScsiDeviceIO: 5789: Restricting cmd 0x28 (4096 bytes) from WID 5586470 to quiesced dev naa.xxxxxxxxxx (vmkCmd=0x45xxx cmdId: 0x430xxx, 0x6xxxx)
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu84:5585612)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.xxxxxxxxxx" state in doubt; requested fast path state update...
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu2:5585613)ScsiDeviceIO: 4686: Cmd(0x45da691b61c0) 0xfe, CmdSN 0x6xxx from world 2106559 to dev "naa.xxxxxxxxxx" failed H:0x8 D:0x0 P:0x0   
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu84:5585612)ScsiDeviceIO: 4686: Cmd(0x45da60622e80) 0xfe, CmdSN 0x6xxx from world 2106559 to dev "naa.xxxxxxxxxx" failed H:0x8 D:0x0 P:0x0   
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu72:2099107)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.xxxxxxxxxx" state in doubt; requested fast path state update...

 

 

Environment

VMware vCenter Server 8.0.x

Cause

The backup task caused high I/O load on the virtual machine, and I/O was interrupted multiple times, resulting in a virtual machine crash that triggered an HA restart.

Resolution

Engage your storage vendor to check if the storage is getting overloaded and to request further investigation and analysis.

Contact the backup vendor to inquire about the reasons for LWD-based backup failures.

 

Additional Information

Read more at:

"state in doubt; requested fast path state update" error in vmkernel.log

Performing LWD-based snapshot sync fails

vSphere HA restarted the VM

Powered by Wolken Software