How to investigate network interruption to a VM after it is migrated from one ESXi host to another

Products

• VMware vSphere ESXi

Issue/Introduction

Virtual machines (VMs) deliver and receive their TCP/IP packets to and from other endpoints, along a specific data path, which involves a specific vmnic (network adapter) that is installed in the server running the ESXi hypervisor (the ESXi host).  

When a VM is migrated from a source ESXi host to a destination ESXi host, the packets being sent from, and delivered to, the VM, travel on a different data path, which involves a specific vmnic (network adapter) that is installed in the destination ESXi host.

It is normal and expected for a VM to perhaps miss one ping response when the data path is switched.

However, if when the VM is migrated to some hosts, it works fine, and other hosts, it does not, then please see the Resolution section.

Environment

ESXi

vCenter

Cause

There are a number of possible causes for this, depending on the results of each investigative step in the Resolution section.

However, the most common cause of this is a difference in the upstream physical switch configuration, between the physical switchport to which the vmnic in the original data path is connected, and the physical switchport to which the vmnic in the new data path is connected.

Resolution

1) The first step is to determine each of the source and destination data paths:

a) When the VM is on an ESXi host where the issue is NOT observed, SSH into the ESXi host and enter the command "esxtop"

b) Enter the small letter n (for networking info)

c) Locate the line(s) where the virtual NICs attached to the VM are, and identify the physical uplink being used to carry the packets (column "Pnic")

d) If the physical switch is configured to send CDP or LLDP info, locate that physical adapter in the "Configure" section of the vSphere client and reveal the switch and switchport to which that vmnic is connected.  Make a note of the physical switch and switchport info.

 

2) Migrate the VM to a destination host, then if the issue is observed, repeat steps a) through d) above on the destination host, and if revealed, make a note of the info.  

 

3) Contact your internal team that is responsible for managing the physical switch infrastructure external to the vmnic(s), provide them with the info for both the source and destination hosts, and ask them to log into the switches and compare the switchport configurations precisely to determine if they are identical.  They should be identical for proper operation.

• NOTE:  If there is no CDP or LLDP info available from the physical switches, provide the networking team with the MAC addresses of the physical adapters for each data path.  (i.e. the vmnic identified in the source, and the vmnic identified in the destination). 

 

4) If the networking team finds a discrepancy, ask them to correct it and repeat the migration.

 

5) If there is no discrepancy or if after correcting the discrepancy, the issue continues to present, please contact VMware by Broadcom Support and we will schedule a screen share session to help progress the investigation further.  At that session, a representative from the internal networking team with access to the physical switches in the environment must be present.