Virtual machine frequently hangs and restarts during the task of creating incremental snapshots.
search cancel

Virtual machine frequently hangs and restarts during the task of creating incremental snapshots.

book

Article ID: 446392

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

When executing incremental snapshot tasks (creation, deletion, or consolidation), the following symptoms are observed:

  • The virtual machine (VM) frequently hangs and unexpectedly powers off or restarts.

  • In vCenter Server, the snapshot creation, deletion, and consolidation tasks may show as completed, but the VM state becomes unstable immediately after.

  • vSphere HA is NOT triggered because the power-off is a controlled termination by the ESXi hypervisor, rather than a host failure or unhandled crash.

The following error appears in the vmware.log:
YYYY-MM-DDTHH:MM:SS In(05) ####-# #######-##-#### CPT: vm was stunned for 4742491 us
...
YYYY-MM-DDTHH:MM:SS In(05) ####-# #######-##-#### [msg.disklib.lockinfo.local] Locking conflict for file "/vmfs/volumes/########-########/<VM Name>/<VM Name>_2-flat.vmdk". Kernel open flags are 0x4008. Owner process on this host is world ID 43101544 with world name ###-####-#:<VM Name>.
....
YYYY-MM-DDTHH:MM:SS In(05) ####-# #######-##-#### [msg.disklib.lockinfo.local] Locking conflict for file "/vmfs/volumes/########-########/<VM Name>/<VM Name>_2-######-#####.vmdk". Kernel open flags are 0x4002. Owner process on this host is world ID 43101544 with world name ###-####-#:<VM Name>.
YYYY-MM-DDTHH:MM:SS In(05) ####-# #######-##-#### [msg.disklib.OPENPARENT] The parent of this virtual disk could not be opened
YYYY-MM-DDTHH:MM:SS In(05) ####-# #######-##-#### [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/########-########/<VM Name>/<VM Name>_2-000002.vmdk' or one of the snapshot disks it depends on.
YYYY-MM-DDTHH:MM:SS In(05) ####-# #######-##-#### [msg.checkpoint.continuesync.error] An operation required the virtual machine to quiesce and the virtual machine was unable to continue running.
YYYY-MM-DDTHH:MM:SS In(05) ####-# #######-##-#### ----------------------------------------
YYYY-MM-DDTHH:MM:SS In(05) ####-# - SnapshotVMX_ConsolidateCancel: Requesting snapshot consolidate cancel.
YYYY-MM-DDTHH:MM:SS In(05) ####-# - Msg_Post: Error
YYYY-MM-DDTHH:MM:SS In(05) ####-# - [msg.poweroff.commitOn] Performing disk cleanup. Cannot power off.
YYYY-MM-DDTHH:MM:SS In(05) ####-# - ----------------------------------------
YYYY-MM-DDTHH:MM:SS In(05) ####-# - ConsolidateItemCombine: Failed to unstun the virtual machine while consolidating disks.
YYYY-MM-DDTHH:MM:SS In(05) ####-# - ConsolidateEnd: Failed to stun/unstun virtual machine while consolidating disks.
YYYY-MM-DDTHH:MM:SS In(05) ####-# - VigorTransport_ServerSendResponse opID=#######-##-#### seq=3443: Completed Snapshot.Consolidate request with messages in 4772330 US.
YYYY-MM-DDTHH:MM:SS In(05) ####-# - Turning off snapshot info cache.
YYYY-MM-DDTHH:MM:SS In(05) ####-# - Turning off snapshot disk cache.
YYYY-MM-DDTHH:MM:SS In(05) ####-# - ConsolidateEnd: Snapshot consolidate complete: Unable to save snapshot file (13).
YYYY-MM-DDTHH:MM:SS In(05) ####-# - Vix: [mainDispatch.c:4211]: VMAutomation_ReportPowerOpFinished: statevar=3, newAppState=1881, success=1 additionalError=0
YYYY-MM-DDTHH:MM:SS In(05) ####-# - ConsolidateEnd: Snapshot consolidate requesting to power-off.
YYYY-MM-DDTHH:MM:SS In(05) vmx - Stopping VCPU threads...

Concurrently, the hostd.log indicated that a backup job was also accessing the identical snapshot disk <VM Name>_2-000001.
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2100738]: [Originator@6876 sub=Libs opID=#######-################] [NFC INFO]NfcAioCloseFile: Closing file '[<Datastore Name>] <VM Name>/<VM Name>_2-000001.vmdk'
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2100738]: [Originator@6876 sub=Libs opID=#######-################] [NFC INFO]NfcFile_Close: session=7CEBC00030 hdl=7CEA22EEB0  filename='/vmfs/volumes/########-########/<VM Name>/<VM Name>_2-000001.vmdk'
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2100738]: [Originator@6876 sub=Libs opID=#######-################] [NFC INFO]NfcFileConcludeFileStats: session=7CEBC00030 Local filename = '/vmfs/volumes/########-########/<VM Name>/<VM Name>_2-000001.vmdk'
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2100738]: [Originator@6876 sub=Libs opID=#######-################] [NFC INFO]Nfc_UnregisterFileHandle: sessionId=7CEBC00030 found=1 fh=7CEA22EEB0(/vmfs/volumes/########-########/<VM Name>/<VM Name>_2-000001.vmdk)
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2100738]: [Originator@6876 sub=Libs opID=#######-################] [NFC INFO]NfcFileStats: Path=/vmfs/volumes/########-########/<VM Name>/<VM Name>_2-000001.vmdk Type=NFC_DISK fileSessionDurationUS=8508608 timeToOpenFileUs=46826 bytesTransferred=0 fileTransferDurationUS=0

The following error appears in the hostd log on ESXi:
Log location on ESXi: /var/log/run/hostd.log 
"An operation required the virtual machine to quiesce and the virtual machine was unable to continue running."

Environment

VMware ESXi 8.0

VMware vCenter Server 8.0

Cause

This issue is caused by resource contention and file lock conflicts resulting from the simultaneous execution of a third-party (Nutanix) backup job and a separate incremental snapshot/consolidation task on the same virtual machine.

When both operations attempt to manipulate the VMDK file chain at the same time:

  1. The backup software opens a session via NFC to read the snapshot disk (00000#.vmdk).

  2. Simultaneously, vSphere attempts to complete a snapshot consolidation, which requires merging delta disks and unstunning the VM.

  3. Because the backup process is still interacting with the file chain, ESXi encounters a Locking conflict.

  4. To prevent severe disk corruption or data inconsistency, the ESXi hypervisor intentionally terminates (powers off) the VCPU threads.

Q1. Why did the virtual machine restart during this process?

A1.Refer to KB 323381.

Resolution

To resolve this issue, must eliminate concurrent disk management operations on the target VM:

  1. Stagger Schedules: Ensure that internal incremental snapshot testing schedules do not overlap with production backup windows.

  2. Pause Backups During Testing: Temporarily disable or pause automated backup policies for the specific VM while performing manual snapshot or disk consolidation maintenance.

  3. Review Backup Vendor Configuration: If both tasks are initiated by the same automation or backup suite, contact your software vendor (e.g., Nutanix Support) to optimize the snapshot integration and serialization settings.

Powered by Wolken Software