Guest OS reset request (reboot) when Storage vMotion task is running may cause the virtual machine to PANIC, and Storage vMotion fails

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information to resolve an issue when a Storage vMotion task failure occurs when the Guest OS in the powered on virtual machine requests reset.

Symptoms:

In vmware.log file of ESXi host, similar entries as mentioned below will be seen:

migration started :
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vmx - VigorTransportProcessClientPayload: opID=XXXXXXXX-XX-XX-XXXX seq=XXXXXX: Receiving Migrate.PrepareSource request.

Guest OS requests reset :
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - Chipset: The guest has requested that the virtual machine be hard reset.

*.vmdk file was closed :
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - Closing disk 'scsi0:5'
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - DISKLIB-VMFS  : "/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX-flat.vmdk" : closed.
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - Closing disk 'scsi0:4'
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - DISKLIB-VMFS  : "/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk" : closed.
yyyy-mm-ddTHH:mm:ss.nnnZ In(05) vcpu-0 - Closing disk 'scsi0:3'
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - DISKLIB-VMFS  : "/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk" : closed.
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - Closing disk 'scsi0:2'
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - DISKLIB-VMFS  : "/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk" : closed.
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - Closing disk 'scsi0:1'
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - DISKLIB-VMFS  : "/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk" : closed.
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - Closing disk 'scsi0:0'
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - DISKLIB-VMFS  : "/vmfs/volumes/XXXXXXXX-XXXXXXXX-XXXX-XXXXXXXXXXXX/XXXXXXXXX/XXXXXXX_2-flat.vmdk" : closed.

after that open *.vmdk file failed :
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - DISK: OPEN scsi0:2 '/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX.vmdk' persistent R[]
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - AIOGNRC: Failed to open '/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk' : Failed to lock the file (40003) (0x42003).
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk' locked (try 0)
yyyy-mm-ddThh:mm:ss.nnnZ In(05) svga - SVGA disabling SVGA
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - AIOGNRC: Failed to open '/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk' : Failed to lock the file (40003) (0x42003).
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk' locked (try 1)
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - AIOGNRC: Failed to open '/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk' : Failed to lock the file (40003) (0x42003).
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk' locked (try 2)
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - AIOGNRC: Failed to open '/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk' : Failed to lock the file (40003) (0x42003).
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk' locked (try 3)
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - AIOGNRC: Failed to open '/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk' : Failed to lock the file (40003) (0x42003).
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk' locked (try 4)
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - AIOGNRC: Failed to open '/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk' : Failed to lock the file (40003) (0x42003).
yyyy-mm-ddThh:mm:ss.nnnZ In(05) vcpu-0 - OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/XXXXXXXX-XXXXXXXX/XXXXXXXX/XXXXXXXX/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-flat.vmdk' : Failed to lock the file (262146).

after that virtual macine panicked :
yyyy-mm-ddThh:mm:ss.nnnZ Wa(03) worker-XXXXXXXXX - SVMotionMirroredModeThreadDiskCopy: Found internal error when woken up on diskCopySemaphore. Aborting storage vmotion.
:::
yyyy-mm-ddThh:mm:ss.nnnZ In(05) worker-XXXXXXXXX - SVMotionCleanupThread: Waiting for the cleanup semaphore to be signaled so that it is safe for the cleanup thread to proceed.
yyyy-mm-ddThh:mm:ssZ[+0.000] Wa(03) worker-XXXXXXXXX - Caught signal 11 -- tid XXXXXXXXX (addr 0)
:::
yyyy-mm-ddThh:mm:ssZ[+0.000] Cr(01) worker-XXXXXXXXX - PANIC: Unexpected signal: 11.
yyyy-mm-ddThh:mm:ssZ[+0.000] Wa(03) worker-XXXXXXXXX - A core file is available in "/var/core/vmx-zdump.000"

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 6.7

Cause

This issue occurs due to unexpected closure of virtual disks when a reset request is received for the Guest OS.

Resolution

The issue has been fixed in Vmware vSphere ESXi 6.7 P08 (ESXi670-202210001) and ESXi 7.0 Update 3f.

Workaround:

To workaround the issue, please follow one of the steps mentioned below:

• Do not issue the reset from the Guest OS (reboot) when Storage vMotion task is running.

• Power-off the virtual machine prior to starting Storage vMotion.