VM may crash during Storage VMotion when one or more of the disks are on latent storage with vVols as the back end

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

The follow sequence is observed in the vmware.log for the VM when migrating: 

2018-07-12T15:32:15.711Z| vmx| I125: DISKLIB-LIB   : DiskLib_ForceLoadFilters: Disk was opened with OPEN_NOFILTERS. Forcing a delayed load of all filters.
2018-07-12T15:32:15.711Z| vmx| I125: DISKLIB-LIB_BLOCKTRACK   : Resuming change tracking.
2018-07-12T15:32:19.833Z| vmx| I125: DISKLIB-CTK   : ChangeTrackerOpenOnDisk: Could not open tracking file /vmfs/volumes/vvol:###############-################/rfc##############/exampleVM-ctk.vmdk (4).
2018-07-12T15:32:19.833Z| vmx| I125: DISKLIB-CTK   : Could not open change tracking file "/vmfs/volumes/vvol:###############-################/rfc##############/exampleVM-ctk.vmdk": Could not open or create change tracking file.
2018-07-12T15:32:19.968Z| vmx| I125: DISKLIB-LIB_BLOCKTRACK   : Could not open change tracker /vmfs/volumes/vvol:###############-################/rfc##############/exampleVM-ctk.vmdk: Could not open or create change tracking file.
2018-07-12T15:32:19.968Z| vmx| I125: DISKLIB-LIB   : DiskLib_ForceLoadFilters: DiskLibBlockTrackResume failed : Could not open or create change tracking file (0x83c).
2018-07-12T15:32:19.968Z| vmx| W115: DiskDelayedFilterAttachAll Disk 'scsi3:0' force load filters error: Could not open or create change tracking file.
2018-07-12T15:32:19.968Z| vmx| I125: MigrateSetStateFinished: type=2 new state=12
2018-07-12T15:32:19.968Z| vmx| I125: MigrateSetState: Transitioning from state 11 to 12.
2018-07-12T15:32:19.968Z| vmx| I125: Migrate: Caching migration error message list:
2018-07-12T15:32:19.968Z| vmx| I125: [msg.migrate.multiwriter.delayed.filter.attach.failed] Delayed filter attach failed on destination.
2018-07-12T15:32:19.968Z| vmx| E105: PANIC: Delayed filter attach failed on destination virtual machine during migration.

2018-07-12T15:32:21.658Z| vmx| I125: DISKLIB-LIB   : DiskLib_ForceLoadFilters: Disk was opened with OPEN_NOFILTERS. Forcing a delayed load of all filters.
2018-07-12T15:32:21.658Z| vmx| I125: DISKLIB-LIB_BLOCKTRACK   : Resuming change tracking.
2018-07-12T15:32:25.766Z| vmx| I125: DISKLIB-CTK   : ChangeTrackerOpenOnDisk: Could not open tracking file /vmfs/volumes/vvol:###############-################/#############/exampleVM2-ctk.vmdk (4).
2018-07-12T15:32:25.766Z| vmx| I125: DISKLIB-CTK   : Could not open change tracking file "/vmfs/volumes/vvol:###############-################/#############/exampleVM2-ctk.vmdk": Could not open or create change tracking file.
2018-07-12T15:32:25.922Z| vmx| I125: DISKLIB-LIB_BLOCKTRACK   : Could not open change tracker /vmfs/volumes/vvol:###############-################/#############/exampleVM2-ctk.vmdk: Could not open or create change tracking file.
2018-07-12T15:32:25.922Z| vmx| I125: DISKLIB-LIB   : DiskLib_ForceLoadFilters: DiskLibBlockTrackResume failed : Could not open or create change tracking file (0x83c).
2018-07-12T15:32:25.922Z| vmx| W115: DiskDelayedFilterAttachAll Disk 'scsi3:0' force load filters error: Could not open or create change tracking file.
2018-07-12T15:32:25.922Z| vmx| I125: MigrateSetStateFinished: type=2 new state=12
2018-07-12T15:32:25.922Z| vmx| I125: MigrateSetState: Transitioning from state 11 to 12.
2018-07-12T15:32:25.922Z| vmx| I125: Migrate: Caching migration error message list:
2018-07-12T15:32:25.922Z| vmx| I125: [msg.migrate.multiwriter.delayed.filter.attach.failed] Delayed filter attach failed on destination.
2018-07-12T15:32:25.922Z| vmx| E105: PANIC: Delayed filter attach failed on destination virtual machine during migration.
2018-07-12T15:32:28.884Z| vmx| W115: A core file is available in "/vmfs/volumes/########-########-####-########/exampleVM2/vmx-zdump.000"

Environment

VMware vSphere ESXi 7.0.0

Cause

The vVol vMotion multiwriter disk handoff was designed to optimize disk handoff on VVols, as it may take more time than the vMotion downtime of 1 second to bind a VVol to an ESX host and open.

Resolution

There is no resolution at this time aside from ensuring that the destination datastore is not experiencing high disk latency.

Additional Information

Impact/Risks:
VM crash and downtime