Error: "The operation is not supported on the object" when performing cold VM migrations after upgrading to vCenter 8.0

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• After upgrading to vSphere 8.0 migrations of VMs fails.
• vCenter UI may display the following error messages:
  
  vim.VirtualMachine.relocate: :vmodl.fault.NotSupported

Could not complete network copy for file /vmfs/volumes/###/<VM Name>/<VM Name>.vmdk
  
  
• Both the source and destination ESXi hosts have provisioning VMKs configured.
• Powered on migrations succeed as normal.
• ESXi logs may show the following log messages.
• /var/log/vmware/hostd.log

  [NFC ERROR]NfcTransportMigrateInit: Failed to initialize src migrate module for NFC streaming over vmkernel: Networking stack mismatch
  [NFC ERROR]NfcSendFileInt: Failed to initialize migrate module: NFC_FAILED
  [NFC ERROR]NfcSendFile: File send failed for file '/vmfs/volumes/##/<VM Name>/<VM NAME>.vmdk': NFC_FAILED (NFC operation failed)
  File transfer [/vmfs/volumes/##/<VM Name>/<VM NAME>.vmdk -> /vmfs/volumes/##/<VM Name>/<VM NAME>.vmdk] failed:
  NfcErrorCode communicated as part of fault
  Error encountered while processing copy spec for file
  Hostd[2099920]: --> N3Vim5Fault16NetworkCopyFault9ExceptionE(Fault cause: vim.fault.NetworkCopyFault

  
  
  
• /var/log/vmkernel.log

  vmkwarning: cpu22:2099852 opID=ce526129)WARNING: Nfc: 334: 52fdf5c5-8b1f-e343-d0dd-############S: failed to resolve net stack instance due to mismatch with vMotion server. Migration instance: vSphereProvisioning (0x############).
  vmkwarning: cpu22:2099852 opID=ce526129)WARNING: Migrate: 790: 52fdf5c5-8b1f-e343-d0dd-############S: Destroying non-empty VOB migrant context migrateExtVobCtx
  vmkwarning: cpu73:2099848 opID=cbf548da)WARNING: Nfc: 334: 529dc3c7-728c-d849-481c-############S: failed to resolve net stack instance due to mismatch with vMotion server. Migration instance: vSphereProvisioning (0x############).
  vmkwarning: cpu73:2099848 opID=cbf548da)WARNING: Migrate: 790: 529dc3c7-728c-d849-481c-############S: Destroying non-empty VOB migrant context migrateExtVobCtx
  vmkwarning: cpu86:27820922)WARNING: VMotion: 2569: 1797743340664855263 S: failed: VMM leader 23688121 exited 

  
  
  
  

   

Environment

• vSphere 8.0.x upgrade
• ESXi host with provisioning stack enabled.

Cause

• Cold migrations attempt to use the provisioning network if enabled. This network is not configured by default. If not configured, the management network is used instead.
• In vSphere 7.0, the provisioning VMK NIC has a null value in the Net.ProvisioningVmknics option.  In ESXi 8.0, if this option is not set, the ESXi host will use the management VMK, even if a provisioning VMK is configured, which will trigger the error.

Resolution

To resolve the issue, you need to modifying the Net.ProvisioningVmknics option from the vSphere UI as follows:

1. Select the ESXi host in inventory that you need to repair the option for.
   
   
2. Select the "Configure" tab for the host.
   
   
3. Scroll down to the System section and select the "Advanced System Settings" option.
   
   
4. Select the Edit button in the upper right hand corner.
   
5. Select the filter icon on the Key field and type in "Provision".
   
6. In the Net.ProvisioningVmknics field, add the VMK adapter that is configured for the Provisioning service.
    

 

Alternatively, you can SSH to the ESXi host and run the command:

vsish -e set /config/Net/strOpts/ProvisioningVmknics <vmkX>

Note: Replace <vmkX> with the vmk that you have configured for provisioning.