Virtual machines experiences abrupt restart or reboot.
search cancel

Virtual machines experiences abrupt restart or reboot.

book

Article ID: 438050

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Virtual machines experiences abrupt restart or reboot.
  • Guest operating system may request a hard reset which can be seen in /vmfs/volumes/devices/VM1/VM/vmware.log.

2026-04-05T16:04:24.917Z In(05) vcpu-9 - Chipset: The guest has requested that the virtual machine be hard reset.

  • Additionally in vCenter may display events "Cannot connect to storage" in "vCenter UI > Host Cluster > Host > Monitor > Events" 
 

 

Environment

VMware vSphere ESXi 8.x

Cause

This issue is caused due to a transient disruption in the underlying storage issue which is triggered by a storage controller failure. This failure caused continuous target port flapping and eventual offline states.

Affected virtual machines may continue attempting to process I/O across paths that are actively experiencing failures and SCSI command aborts. The resulting I/O starvation within the guest operating systems ultimately forces the VMs to initiate a hard reset.

  • Target port flapping causes path failures and aborts which leads to VM reboot and this events are captured in /var/run/log/vmkernel.log with sense code H:0x2 which indicates dropped frames.

2026-04-05T16:02:09.929Z Wa(180) vmkwarning: cpu16:2098444)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.###########################" state in doubt; requested fast path state update...
2026-04-05T16:02:09.929Z In(182) vmkernel: cpu52:9579470)qlnativefc: vmhba65(5f:0.1): qlnativefcStatusEntry:2069:C0:T3:L253 - FCP command status: 0x29-0x0 (0x2) portid=150060 oxid=0x0 cdb=2a00d6 len=4096 rspInfo=0x0 resid=0x0 fwResid=0x0 host status = 0x2 device status =$
2026-04-05T16:02:09.929Z In(182) vmkernel: cpu24:2098437)NMP: nmp_ThrottleLogForDevice:3898: H:0x2 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x4###### CmdSN 0x303491
2026-04-05T16:02:09.929Z In(182) vmkernel: cpu31:2098435)NMP: nmp_ThrottleLogForDevice:3898: H:0x2 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x4###### CmdSN 0x3cd
2026-04-05T16:02:09.929Z In(182) vmkernel: cpu52:9579470)qlnativefc: vmhba65(5f:0.1): qlnativefcStatusEntry:2038:(3:253) Port down status: port-state=0x3

  • Due to the underlying storage connectivity issues, events related to ATS mis compares and SCSI aborts with timeouts are expected in /var/run/log/vmkernel.log , 

2026-04-05T16:04:15.281Z In(182) vmkernel: cpu43:2098437)NMP: nmp_ThrottleLogForDevice:3898: H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0. Act:NONE. cmdId.initiator=0x43###### CmdSN 0x2883e40
2026-04-05T16:04:15.281Z In(182) vmkernel: cpu43:2098437)ScsiDeviceIO: 4686: Cmd(0x4######) 0x89, CmdSN 0x2883e40 from world 19284925 to dev "naa.###########################" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0
2026-04-05T16:04:15.307Z In(182) vmkernel: cpu50:2728956)PVSCSI: 2769: scsi0:8: SCSI ABORT ctx=0x2f7
2026-04-05T16:04:15.307Z In(182) vmkernel: cpu50:2728956)qlnativefc: vmhba3(3b:0.1): qlnativefcTaskMgmt:2308:Task Mgmt abort on serial num 2f7
2026-04-05T16:04:15.308Z In(182) vmkernel: cpu50:2728956)qlnativefc: vmhba3(3b:0.1): qlnativefcEhAbort:2797:C0:T0:L250: Abort command succeeded -- 0
2026-04-05T16:04:15.308Z In(182) vmkernel: cpu50:2728956)qlnativefc: vmhba65(5f:0.1): qlnativefcTaskMgmt:2308:Task Mgmt abort on serial num 2f7
2026-04-05T16:04:15.308Z In(182) vmkernel: cpu50:2728956)qlnativefc: vmhba65(5f:0.1): qlnativefcEhAbort:2751:SCSI command timeout counter incremented to 36
2026-04-05T16:04:28Z In(182) vmkernel:
2026-04-05T16:04:15.308Z In(182) vmkernel: cpu50:2728956)qlnativefc: vmhba65(5f:0.1): qlnativefcEhAbort:2754:qlnativefcEhAbort: aborting sp 0x45bb18e13e00 handle 2f7 from RISC. serialNumber=2f7, Command timeout=55716 sec.

  • Because the storage is unable to handle the load, the driver aborts with sense code H:0x8, followed by SCSI aborts. This leads directly to I/O starvation within the guest operating system:

208446)ScsiDeviceIO: 4633: Cmd(0x45baa7b13440) 0x28, CmdSN 0x2f7 from world 2728942 to dev "naa.###########################" failed H:0x8 D:0x0 P:0x0
2026-04-05T16:04:15.508Z In(182) vmkernel: cpu25:2728956)qlnativefc: vmhba65(5f:0.1): qlnativefcEhAbort:2797:C0:T3:L250: Abort command succeeded -- 1
2026-04-05T16:04:15.508Z In(182) vmkernel: cpu25:2728956)PVSCSI: 2769: scsi0:9: SCSI ABORT ctx=0x3ab

  • The guest operating system will also report SCSI command aborts in /vmfs/volumes/devices/VM/VM1/vmware.log before hard reset.

2026-04-05T16:04:15.307Z In(05) vcpu-13 - PVSCSI: scsi0:8: aborting cmd 0x2f7
2026-04-05T16:04:15.508Z In(05) vcpu-13 - PVSCSI: scsi0:9: aborting cmd 0x3ab

 

Additionally, this behavior in virtual machines can be seen when there is an unsupported firmware and driver mismatch on the HBA adapters. This uncertified configuration degrades multipathing resilience, worsening the overall path instability.

Resolution

  • Engage the storage and switch vendor to further investigate this issue.

  • Furthermore, contact the hardware vendor to ensure HBA adapters are running on supported hardware with compatible driver and firmware versions listed on the Broadcom matrix Broadcom Compatibility Guide.
Powered by Wolken Software