Snapshot taken with memory for a VM fails
search cancel

Snapshot taken with memory for a VM fails

book

Article ID: 414377

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Snapshot taken without memory for a Virtual machine completes successfully but when the snapshot is taken for the same VM with memory, it fails.

  • On the vCenter UI, the snapshot task fails with the below error :- 
    An error occurred while saving: 110 (Connection timed out). An error occurred while taking a snapshot: Unable to save snapshot file. An error occurred while saving snapshot file "/vmfs/volumes/[Datastore]/[VM-Name]/[VM-Name]-Snapshot.vmsn


  • On checking further from the VM's specific vmware logs from the host's /vmfs/volumes/[datastore]/[vm_name]/vmware.log file during the same time the snapshot with memory is taken, we see the below events:-
    YYYY-MM-DDTHH:MM:SSZ In(05) worker-###### - MainMem: Failed to write memory pages [306560,306687] at 0: Timeout.
YYYY-MM-DDTHH:MM:SSZ In(05) vcpu-0 - MainMem: Failed to write memory pages [439680,439807] at 0: Timeout.
YYYY-MM-DDTHH:MM:SSZ In(05) worker-###### - MainMem: End lazy IO (346752 done, sync = 0, error = 2).
YYYY-MM-DDTHH:MM:SSZ In(05) vmx - MainMem: Completed pending lazy checkpoint save (0).
YYYY-MM-DDTHH:MM:SSZ In(05) vmx - DUMPER: Partial save write failure.
YYYY-MM-DDTHH:MM:SSZ In(05) vmx - SnapshotVMXTakeSnapshotWork: Transition to mode 1.
YYYY-MM-DDTHH:MM:SSZ In(05) vmx - SnapshotVMXTakeSnapshotComplete: Done with snapshot 'VM Snapshot : 2312
YYYY-MM-DDTHH:MM:SSZ In(05) vmx - SnapshotVMXTakeSnapshotComplete: Snapshot 2312 failed: Unable to save snapshot file (13).
YYYY-MM-DDTHH:MM:SSZ In(05) vmx - SnapshotVMXTakeSnapshotComplete: Cleaning up incomplete snapshot 2312.
YYYY-MM-DDTHH:MM:SSZ In(05) vmx - SnapshotVMXTakeSnapshotComplete: Deleting incomplete snapshot 2312.

  • During the same time the snapshot gets failed, we see the "state in doubt; requested fast path state update" error from the host's vmkernel logs in the /var/run/log/vmkernel.log file path :-
    YYYY-MM-DDTHH:MM:SSZ Wa(180) vmkwarning: cpu23:2098278)WARNING: NMP: nmp_DeviceStartLoop:790: NMP Device "eui.#######################" is blocked. Not starting I/O from device.
YYYY-MM-DDTHH:MM:SSZ Wa(180) vmkwarning: cpu77:2098271)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "eui.####################" state in doubt; requested fast path state update

Environment

  • VMware vSphere ESXi 7.x
  • VMware vSphere ESXi 8.x

Cause

During the same time the snapshot gets failed, we see the "state in doubt; requested fast path state update" error from the vmkernel logs.

Host Bus Adapter (HBA) driver aborts a command because the command took longer than the timeout period of 5 seconds to complete. An operation can take longer than the timeout period because of several reasons including:

  • Array backup operations (LUN backup, replication, etc.)
  • General overload on the array
  • Read/Write Cache on the array (misconfiguration, lack of cache, etc.)
  • Incorrect tiered storage used (SATA over SCSI)
  • Fabric issues (Bad ISL, outdated firmware, bad fabric cable/GBIC)

Resolution

Engage your storage vendor to check if the storage is getting overloaded if it is observed that the device utilization is hitting 100%.
If storage utilization is under control and IO aborts are observed, engage your fabric vendor and ensure that the fabric is healthy

"state in doubt; requested fast path state update" error in vmkernel.log

Additional Information

Powered by Wolken Software