Warning : "failed to connect to remote host from host : Host is down" in vmkernel logs during migration of VMs from ESXi hosts
search cancel

Warning : "failed to connect to remote host from host : Host is down" in vmkernel logs during migration of VMs from ESXi hosts

book

Article ID: 451036

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware Telco Cloud Platform

Issue/Introduction

  • vMotion fails to initiate or complete for VMs from a specific host. vmkernel logs of the source ESXi host shows below warning messages
YYYY-MM-DDTHH:MM:SS.SSSZ cpu61:<CPU ID>)MigrateNet: 2035: 7605599851734535238 S: Timed out attempting to connect to remote host.  Retrying...
YYYY-MM-DDTHH:MM:SS.SSSZ cpu61:<CPU ID>)MigrateNet: 1753: 7605599851734535238 S: Successfully bound connection to vmknic vmk1 - '<vmotion vmk IP'
YYYY-MM-DDTHH:MM:SS.SSSZ cpu61:<CPU ID>)WARNING: MigrateNet: 1903: 7605599851734535238 S: failed to connect to remote host <Dest host vmotion vmk IP> from host <Source host vmotion vmk IP>: Host is down.
YYYY-MM-DDTHH:MM:SS.SSSZ cpu61:<CPU ID>)WARNING: VMotionSend: 1206: 7605599851734535238 S: failed to create a connection with remote host <Dest host vmotion vmk IP>: The ESX hosts failed to connect over the VMotion network
YYYY-MM-DDTHH:MM:SS.SSSZ cpu61:<CPU ID>)Migrate: 102: 7605599851734535238 S: MigrateState: Failed
YYYY-MM-DDTHH:MM:SS.SSSZ cpu61:<CPU ID>)WARNING: Migrate: 256: 7605599851734535238 S: Failed: The ESX hosts failed to connect over the VMotion network (0xbad010b) @0x4200157b4d56
YYYY-MM-DDTHH:MM:SS.SSSZ cpu98:<CPU ID>)WARNING: Migrate: 6460: 7605599851734535238 S: Migration considered a failure by the VMX.  It is most likely a timeout, but check the VMX log for the true error.
YYYY-MM-DDTHH:MM:SS.SSSZ cpu98:<CPU ID>)Hbr: 3815: Migration end received (worldID=<worldID>) (migrateType=1) (event=1) (isSource=1) (sharedConfig=1)
  • The vmkping command on vmotion vmkernel interface to the destination host returns the following error
PING <vmotion vmk interface IP> (<vmotion vmk interface IP>): 56 data bytes
sendto() failed (Host is down)
  • Packet captures taken on "vmotion vmkernel interface" and on physical uplinks of the source ESXi host shows that ARP requests are transmitted but are never received on the destination ESXi host.
  • This issue is observed with HPE server Synergy 480 Gen10 hardware Model

Environment

VMware vSphere ESXi 7.x / 8.x

TCP 2.5 and above 

Cause

vMotion traffic originates from the ESXi host's physical uplink but fails to reach the destination host due to a physical network or hardware-layer fault, resulting in a failure of ARP resolution which could be due to packets being dropped or misrouted in the physical network

Resolution

To isolate and resolve the issue the following steps must be taken :

  1. Check for IP connectivity and packet loss over the vMotion network. Try having the source host ping (vmkping) the destination host's vMotion vmknic IP address using below command
    • vmkping ++netstack=vmotion -I vmk# <IP of destination host vMotion vmk>
  2. Perform Packet Captures: Use the pktcap-uw tool to confirm the traffic is leaving the ESXi host but not reaching the destination.
    • Run the following command on the source host to capture traffic in vmotion vmk interface 
      • pktcap-uw --vmk <vmotion vmk interface> --dir 2 -o - | tcpdump-uw -enr -
    • Run the following command on the source host to capture traffic in physical uplink interface, replace <vmnic_ID> with the uplink mapped to the vMotion network.
      • pktcap-uw --uplink <vmnic_ID> --dir 2 -o - | tcpdump-uw -enr -
  3. Verify Physical Link Integrity: Inspect physical cabling, SFPs, and switch port status for the uplinks associated with the vMotion port group.
  4. Audit Physical Switch Configuration: Engage the network infrastructure team to verify:
    • VLAN Tagging: Ensure the vMotion VLAN is correctly tagged on all intermediate trunk ports.
    • MTU Consistency: Confirm Jumbo Frames (if used) are configured consistently across the ESXi VMkernel, Virtual Switch, and Physical Switch ports.
    • MAC Learning/ARP Tables: Check for stale entries or security policies (such as MAC filtering) that might drop ARP broadcasts.
  5. Hardware Validation: Check the physical switch logs for interface errors or drops at the time of the vMotion failure.

If the issue persists after validating with hardware vendor and physical networks, contact Broadcom by referring KB Broadcom Support

Additional Information

For additional information refer KB 323392

Powered by Wolken Software