Migrating a VM to a VSAN datastore fails

Products

• VMware vCenter Server
• VMware vCenter Server 7.0

Issue/Introduction

When CBRC is enabled on an ESXi 7.0 Update 1 and ESXi 7.0 Update 2 host, you experience these symptoms:

• VM migration to VSAN datastore fails.
• You see this error similar to:
  
  The operation is not supported on the object.
• You may also see issues with Horizon VDIs provisioning issues with an error

       “Error during provisioning cloning of VM failed: Fault type is VC_FAULT_FATAL The operation not supported on the object”

• In the /var/log/vmware/vpxd.log file, you see this entry similar to:
  
  2021-01-31T17:12:27.477Z error vpxd[18588] [Originator@6876 sub=vpxLro opID=65exxx53-01] [VpxLRO] Unexpected Exception: N5Vmomi5Fault12NotSupported9ExceptionE(Message is: The operation is not supported on the object.,
   --> Fault cause: vmodl.fault.NotSupported
   --> Fault Messages are:
   --> (null)
   --> )
   -->
   
• In the /var/run/log on the destination ESXi host, you see these entries similar to:
  
  2021-01-31T17:12:24.485Z info hostd[1000213017] [Originator@6876 sub=DiskLib opID=65ef3b53-01-7204 user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB_CREATE  : CREATE CHILD: “/vmfs/volumes/vsan:526475aead5xxxf2cfa1c8a1d6cc0/########-####-####-####-########52c1/vmotion.vsanVmfs6Vmotion-VM-1903-2-000001-digest.vmdk” -- vmfsSparse cowGran=1 allocType=1 policy=‘<?xml version=“1.0” encoding=“UTF-8"?>
  2021-01-31T17:12:24.513Z info hostd[1000213017] [Originator@6876 sub=Libs opID=6xxxx53-01-7204 user=vpxuser:VSPHERE.LOCAL\Administrator] DiskLib_IsVMFSSparseSupported: vmfssparse is not supported on /vmfs/volumes/vsan:526xxxxa75d-3f2cfa1c8a1d6cc0/########-####-####-####-########52c1: f532.
  2021-01-31T17:12:24.513Z info hostd[1000213017] [Originator@6876 sub=DiskLib opID=65exxxx3-01-7204 user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-VMFS_SPARSE : Failed to create extent /vmfs/volumes/vsan:52xxxxd5ca75d-3f2xxxxa1d6cc0/eae41660-xxxx-9a65-572f-0200775552c1//vmfs/volumes/vsan:526475ae########-######1c8a1d6cc0/########-####-####-####-########52c1/vmotion.vsanVmfs6Vmotion-VM-1903-2-000001-digest-delta.vmdk: The specified feature is not supported by this version (24).
  2021-01-31T17:12:24.513Z info hostd[1000213017] [Originator@6876 sub=DiskLib opID=65ef3b53-01-7204 user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB_CREATE  : Failed to create link: The specified feature is not supported by this version (24)
   
• In the /var/log/vmware/vpxd.log file of the VCSA appliance, you see entries similar to:
  
  2021-01-31T16:00:40.950Z error vpxd[18435] [Originator@6876 sub=VmProv opID=208c1018-01] Get exception while executing action vpx.vmprov.CopyVmFiles: N3Vim5Fault16NetworkCopyFault9ExceptionE(Message is: Could not complete network copy for file /vmfs/volumes/vsan:52d746e40xxxxxx-586f348625bb62f2/abd31660-4e7e-xxxx-xxxx-0200775889ff/vmotion.vsanMixVmfsVmotion-VM-3161-1-000002-digest.vmdk,
   --> Fault cause: vim.fault.NetworkCopyFault
   --> Fault Messages are:
   --> (null)
   --> )
   -->
   
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

This issue occurs because with storage vMotion, the digest vmdk creation is not handled well in vSphere 7.0 Update 2. Previously, vsanDatastore is using vmfs5 to create VSAN namespace object. vmfsSparse is selected as create type for digest vmdk by default. In vSphere 7.0 Update 2, vmfs6 is introduced for the VSAN namespace object and seSparse is expected for the digest vmdk. It is a new behavior in digest disk creation workflow. In failure cases, current logic still selects vmfsSparse as create type.

With vMotion, the source and destination host are different. Further, the storage is non-shared. The destination host does not have access to the source vmfs datastore. So, a remote NFC copy path is taken to copy the child disk and its digest. In this path, the objParams passed by NFC copy is NULL. As a result, the objType is choosing OBJTYPE_VSAN according to file path, which is under the vsanDatastore.

Resolution

This is a known issue affecting VMware ESXi 7.0 Update 2.

Currently, there is no resolution.

 

To work around this issue, consolidate all snapshots before migrating the virtual machine. Alternatively, delete all snapshots before migrating the virtual machine.