Error: "The operation is not supported on the object" when performing cold VM migrations after upgrading to vCenter 8.0

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• After upgrading to vSphere 8.0, cold (powered off) migrations of VMs fail
• vCenter UI may display the following error messages:
  
  vim.VirtualMachine.relocate: :vmodl.fault.NotSupported

Could not complete network copy for file /vmfs/volumes/###/<VM Name>/<VM Name>.vmdk
  
  
• Unable to copy powered off VMs 
• Both the source and destination ESXi hosts have provisioning VMKs configured
• Powered on migrations succeed as normal
• ESXi - /var/log/vmware/hostd.log may show the following log messages:
  
  [NFC ERROR]NfcTransportMigrateInit: Failed to initialize src migrate module for NFC streaming over vmkernel: Networking stack mismatch
[NFC ERROR]NfcSendFileInt: Failed to initialize migrate module: NFC_FAILED
[NFC ERROR]NfcSendFile: File send failed for file '/vmfs/volumes/##/<VM Name>/<VM NAME>.vmdk': NFC_FAILED (NFC operation failed)
File transfer [/vmfs/volumes/##/<VM Name>/<VM NAME>.vmdk -> /vmfs/volumes/##/<VM Name>/<VM NAME>.vmdk] failed:
NfcErrorCode communicated as part of fault
Error encountered while processing copy spec for file
Hostd[2099920]: --> N3Vim5Fault16NetworkCopyFault9ExceptionE(Fault cause: vim.fault.NetworkCopyFault
  
  
• Esxi - /var/log/vmkernel.log may show the following log messages
  
  vmkwarning: cpu22:2099852 opID=ce526129)WARNING: Nfc: 334: 52fdf5c5-8b1f-e343-d0dd-############S: failed to resolve net stack instance due to mismatch with vMotion server. Migration instance: vSphereProvisioning (0x############).
vmkwarning: cpu22:2099852 opID=ce526129)WARNING: Migrate: 790: 52fdf5c5-8b1f-e343-d0dd-############S: Destroying non-empty VOB migrant context migrateExtVobCtx
vmkwarning: cpu73:2099848 opID=cbf548da)WARNING: Nfc: 334: 529dc3c7-728c-d849-481c-############S: failed to resolve net stack instance due to mismatch with vMotion server. Migration instance: vSphereProvisioning (0x############).
vmkwarning: cpu73:2099848 opID=cbf548da)WARNING: Migrate: 790: 529dc3c7-728c-d849-481c-############S: Destroying non-empty VOB migrant context migrateExtVobCtx
vmkwarning: cpu86:27820922)WARNING: VMotion: 2569: 1797743340664855263 S: failed: VMM leader 23688121 exited 
  
  
  

   

Environment

• vSphere 8.0 upgrade
• ESXi host with provisioning stack enabled

Cause

Cold migrations attempt to use the provisioning network if enabled; this network is not configured by default. If not configured, the management network is used instead.

In vSphere 7.0, the provisioning VMK NIC has a null value in the Net.ProvisioningVmknics option.  In ESXi 8.0, if this option is not set, the ESXi host will use the management VMK, even if a provisioning VMK is configured, which will trigger the error.

Resolution

To resolve the issue, you need to modifying the Net.ProvisioningVmknics option from the vSphere UI as follows:

1. Select the ESXi host in inventory that you need to repair the option for.
   
   
2. Select the "Configure" tab for the host.
   
   
3. Scroll down to the System section and select the "Advanced System Settings" option.
   
   
4. Select the Edit button in the upper right hand corner.
   
5. Select the filter icon on the Key field and type in "Provision".
   
6. In the Net.ProvisioningVmknics field, add the VMK adapter that is configured for the Provisioning service.
    

 

Alternatively, you can SSH to the ESXi host and run the command:

vsish -e set /config/Net/strOpts/ProvisioningVmknics <vmkX>

Note: Replace <vmkX> with the vmk that you have configured for provisioning.