vMotion and cold migrations fail for VMs on a specific host with error: “An error occurred while communicating with the remote host”

Products

• VMware vSphere ESXi

Issue/Introduction

• When a specific ESXi host is used as the migration source, all virtual machine migrations, including vMotion and cold migration, fail with the following error in vCenter Server:

An error occurred while communicating with the remote host.
  
  
• When the same ESXi host is used as the migration destination, migrations complete successfully.
• During the vMotion process, the following entries appear in the logs:
  In the vCenter /var/log/vmware/vpxd/vpxd.log:
  
[YYYY-MM-DDTHH:MM:SS] error vpxd[36889] [Originator@6876 sub=VmProv opID=########-#######-####-#####-##:########-##-##] Failed to track task vim.Task:task-## on host vim.HostSystem:host-#######: Fault cause: vmodl.fault.HostCommunication

[YYYY-MM-DDTHH:MM:SS] error vpxd[36889] [Originator@6876 sub=VmProv opID=########-#######-####-#####-##:########-##-##] [VMOTION_RECOVER] Failed to prepare VMotion on source: . Aborting vMotion xaction

In the ESXi /var/run/log/hostd.log:
  
  [YYYY-MM-DDTHH:MM:SS] info hostd[2100876] [Originator@6876 sub=Vcsvc.VMotionSrc.<vMotion_migID>] ResolveCb: Failed with fault: (vim.fault.GenericVmConfigFault) {
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "msg.vigor.operationCancelled",
-->          message = "The operation was cancelled by the user.",
-->       }
-->    ],
-->    reason = "The operation was cancelled by the user.",
-->    msg = "The operation was cancelled by the user.
--> "
--> }

  
• Outside of the vMotion process, additional entries are observed in the ESXi /var/run/log/hostd.log at other times:
  
  [YYYY-MM-DDTHH:MM:SS] info hostd[2100933] [Originator@6876 sub=Solo.Vmomi opID=########-#### user=vpxuser] Result:
--> (vim.fault.PlatformConfigFault) {
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.esx.hostctl.default",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "reason",
-->                value = "Unable to delete DVPort "" that is in use, use list: <VM_name>.eth0"
-->             }
-->          ],
-->       }
-->    ],
-->    text = "",
-->    msg = ""
--> }
  
  

Environment

VMware vSphere ESXi 8.x

VMware vCenter Server 8.x

Cause

When the ESXi host was added to a vSphere Distributed Switch (vDS) and VM networking migration was performed, the VMs on this host did not register correctly with their assigned dvPorts.The affected VMs continued to use the old dvPort IDs instead of receiving new assignments. As a result, network metadata inconsistencies occurred, preventing the host from serving as a valid migration source.

Resolution

Either of the following methods can be used to resolve the issue.

Option 1: Re-register all the VMs

1. Log in to the vCenter web client.

2. Power off the virtual machine.

3. Right-click the VM and select Remove from Inventory.

4. In the datastore, browse to the VM directory and locate the .vmx file.

5. Select the .vmx file and click Register VM.

6. Power on the VM.

Option 2: Reassign port groups for all VMs on the affected host

1. Log in to the vCenter web client.

2. Navigate to the network inventory.

3. Right-click the vDS and select Add and Manage Hosts > Manage Host Networking.

4. Select the problematic ESXi host.

5. Proceed to Migrate VM Networking and check Migrate virtual machine networking to view all VMs on the host.

6. Select Configure per virtual machine.

7. Choose Assign Port Group. 

   • Select the same port group that the VM is already using.

   • This operation updates and corrects the VM’s dvPort ID assignment.

8. Click Next, then Finish.