Virtual Machine rebooted abruptly by the Guest Operating System

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual Machines running I/O sensitive applications  may reboot when there is high I/O wait time
• Oracle RAC servers running on RHEL may exhibit these symptoms
• VMX logs of the Virtual Machine and the corresponding "hostd" logs of the ESXi will have the below error message.
• This event will also be noticed in the vCenter Server Web Client:

[YYYY-MM-DDTHH:MM:SS Z] Db(167) Hostd[2100167]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5eee5ae3-6868b844-####-###/###.vmx] Handling vmx message 4728: The CPU has been disabled by the guest operating system. Power off or reset the virtual machine.

 

Environment

VMware vSphere ESXi

Cause

• This issue is caused by failed or delayed I/Os:

[YYYY-MM-DDTHH:MM:SS Z] In(182) vmkernel: cpu41:2097863)qlnativefc: vmhba1(3a:0.1): qlnativefcEhAbort:2748:qlnativefcEhAbort: aborting sp 0x45daa3456400 handle d8 from RISC. serialNumber=102c6d, Command timeout=8 sec.
[YYYY-MM-DDTHH:MM:SS Z] In(182) vmkernel: cpu74:2097862)qlnativefc: vmhba1(3a:0.1): qlnativefcEhAbort:2748:qlnativefcEhAbort: aborting sp 0x45daa3456400 handle d8 from RISC. serialNumber=102c6d, Command timeout=8 sec.

[YYYY-MM-DDTHH:MM:SS Z] In(182) vmkernel: cpu30:2098530)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x8a (0x45ca9d22b400, 2117838) to dev "naa.600a09##########" on path "vmhba1:C0:T53:L7" Failed:
[YYYY-MM-DDTHH:MM:SS Z] In(182) vmkernel: cpu27:2118541)qlnativefc: vmhba1(3a:0.1): qlnativefcStatusEntry:2067:C0:T53:L7 - FCP command status: 0x5-0x0 (0x8) portid=3e1061 oxid=0xa cdb=8a0000 len=4096 rspInfo=0x0 resid=0x0 fwResid=0x0 host status = 0x8 device status = 0x$
[YYYY-MM-DDTHH:MM:SS Z] In(182) vmkernel: cpu27:2118809)qlnativefc: vmhba1(3a:0.1): qlnativefcStatusEntry:2067:C0:T53:

• The Guest OS is tuned to reboot if there are failed or the delayed I/Os 
• The issue is outside the VMware vSphere ESXi storage stack
• The issue can be with faults due to the hardware, driver, firmware of the HBA
• This issue can also be caused due to frame drops or issues in the storage array

Resolution

Please contact the hardware vendor, the fabric switch vendor and the storage array vendor to isolate and investigate the cause of the issue

Additional Information

The CPU has been disabled by the guest operating system