Error: Virtual machine unresponsive during snapshot consolidation
search cancel

Error: Virtual machine unresponsive during snapshot consolidation

book

Article ID: 371539

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere ESXi 8.0

Issue/Introduction

  • Virtual Machine (VM) becomes unresponsive or hangs, often during I/O-intensive tasks such as snapshot consolidation, vMotion, power off or other management tasks on the VM.

  • ESXi host management agents (hostd) become unresponsive causing the host to enter "Not responding" state in vCenter Server.

  • Datastore experiences high latency and failed SCSI commands.

  • Log entries in /var/run/log/vmware/vmkernel.log on the ESXi host shows latency / state in doubt error messages:

[YYYY-MM-DDTHH:MM:SS] cpu6:####)ScsiDeviceIO: 4124: Cmd(0x####) 0x28, CmdSN 0x22ac from world 2100337 to dev "naa.####" failed H:0x7 D:0x28 P:0x0

[YYYY-MM-DDTHH:MM:SS] cpu0:####)WARNING: ScsiDeviceIO: 1513: Device naa.#### performance has deteriorated. I/O latency increased from average value of #### microseconds to #### microseconds.

[YYYY-MM-DDTHH:MM:SS] cpu52:####)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.####" state in doubt; requested fast path state update...

[YYYY-MM-DDTHH:MM:SS] cpu52:####)ScsiDeviceIO: 4115: Cmd(0x####) 0x28, CmdSN 0xbd from world #### to dev "naa.####" failed H:0x2 D:0x0 P:0x0

  • Warnings within the /var/run/log/vmkernel.log file on the ESXi host indicate that a storage path has been classified as flaky.
2025-11-04T06:49:33.664Z Wa(180) vmkwarning: cpu35:2098423)WARNING: NMP: nmpHandleLinkEvent:3998: Marking path vmhba#:C0:T1:L13 flaky on link event 2 with timeoutMS = 20000 flakyMarkTC = 47954528501286772, reEvalFlakyPathTime = 20000

  • Depending on the Host Bus Adapter (HBA) model and driver deployed on the host, additional messaging regarding dropped frames may be observed in the /var/run/log/vmware/vmkernel.log :

    • For Qlogic-based (qlnativefc driver) or Emulex-based (lpfc driver) adapters:

[YYYY-MM-DDTHH:MM:SS] In (182) vmkernel: cpu54:2098257) qlnativefc: vmhba# (12:0.0): qlnativefcStatusEntry:1919: (7:41) Dropped frame (s) detected (106496 of 131072 bytes). 

[YYYY-MM-DDTHH:MM:SS] In (182) vmkernel: cpu54:2098257) qlnativefc: vmhba# (12:0.0): qlnativefcStatusEntry: 2067:C0:T7:L41 FCP command status: 0x15-0x0 (0x2) portid=bc0142 oxid=0x4ba c 80000 len-131072 rspInfo=0x0   resid=0x0 fwResid=0x1a000 host status = 0x2 device  

[YYYY-MM-DDTHH:MM:SS] In (182) vmkernel: cpu54:2098257) qlnativefc: vmhba# (12:0.0): qlnativefcStatusEntry:1919: (7:41) Dropped frame (s) detected (116736 of 131072 bytes).

    • For Cisco VIC CNA adapters (nfnic driver):

[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkernel: cpu35:2098423)nfnic: <1>: fnic_fcpio_icmnd_cmpl_handler: : sc: 0x##############3 tag: 0x## hdr status: FCPIO_DATA_CNT_MISMATCH IO failure! Refer KB340039

[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkernel: cpu35:2098423)nfnic <1>: INFO: fnic_fdis_recv_frame: 4506: Received unknown FCoE frame of len: 52. Dropping Frame
 
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkernel: cpu35:2098423)nfnic <1>: INFO: fnic_fdis_validate_and_get_frame_type: 4166: Received FPIN with some invalid frame bits S_ID: 0xfffffd FCTL: 0x38 R_CTRL: 0x22 type: 0x1. Dropping frame.
  • Windows VM reports following event on windows event viewer:

Windows Event ID 129 ("Reset to device, \Device\RaidPortX, was issued")

  • The following command confirms Link Failure Count, Error Frames, Invalid Tx Word Count and Invalid CRC Count is high:  esxcli storage san fc stats get
[root@esxhostname:~] esxcli storage san fc stats get
   Adapter: vmhba#
   Tx Frames: 1234567890
   Rx Frames: 1234567890
   Lip Count: 0
   Error Frames: 321
   Dumped Frames: 1
   Link Failure Count: 321
   Loss of Signal Count: 0
   PrimSeq Protocol Err Count: 1
   Invalid Tx Word Count: 1234567890
   Invalid CRC Count: 12
   Input Requests: 12345678
   Output Requests: 12345678
   Control Requests: 12345

  • The esxcli storage san fc events get command reports dropped frames event:

YYYY-MM-DDTHH:MM:SS [vmhba#] Dropped frames (10240 of 10 bytes) on C#:T#:L# cmd:0x28
YYYY-MM-DDTHH:MM:SS [vmhba#] Dropped frames (512 of 10 bytes) on C#:T#:L# cmd:0x28

 

Environment

  • VMware vSphere ESXi 7.x

  • VMware vSphere ESXi 8.x

Cause

  • Physical Fibre Channel (FC) fabric failure or physical link instability triggers virtual machine unresponsiveness during I/O intensive tasks. High Link Failure, error frames, invalid CRC, and Invalid Tx Word counts indicate underlying hardware issues with fiber cables, SFPs, or switch ports.

    • Invalid Tx Word Count is a physical layer error. It means the light signal traveling over the fiber cable between the HBA and the fabric switch is being interrupted, and the HBA cannot decode the signal correctly.

    • Loss of Signal Count indicates that the light actually went out. 

    • Link Failure Count is a hardware-level counter that tracks the number of times the Fibre Channel (FC) link has completely lost connectivity and had to re-initialize.

    • Invalid CRC indicates that the calculated checksum of the data in the frame does not match the stored value, thus the frame data been corrupted in transit.

Resolution

Engage the storage, fabric, and hardware vendor to investigate and mitigate physical layer issues within the environment.

Additional Information

How to verify the CRC errors on VMHBA / Fiber Channel adapters on ESXi hosts

Powered by Wolken Software