Troubleshooting virtual machine network connection issues

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Virtual machines (VMs) are experiencing network connectivity issues. The problem is isolated to one or more VMs and includes symptoms such as:

A VM can't ping/connect to certain IPs, or vice versa
VM is unreachable
VMs are unable to connect to the network
Multiple VMs not reachable
Communication between virtual machines broken when they are on different ESXi hosts
High latency between two virtual machines connectivity
No inbound or outbound network traffic is observed for specific virtual machines.
Internet access is unavailable from affected VMs
TCP/IP connections to or from these VMs fail

Common errors observed:

Destination Host Unreachable
Network error: Connection Refused
Network cable is unplugged
Ping request could not find host (IP address/hostname). Please check the name and try again.
Unable to resolve target system name (IP address/hostname)

Additionally, VMs located on different hosts may lose network connectivity with each other.

Cause

There are many different possible causes for a symptom of connectivity issues to / from virtual machines.

In order to find the most appropriate resolution, please investigate using the principles outlined below in the Resolution section.

Resolution

NOTE: If the virtual machine you are unable to reach is vCenter, then it is best to open a Networking Support Case with Broadcom Support using the instructions at Creating and managing Broadcom support cases to determine the best course of action.

NOTE: If the virtual machine you are unable to reach is vCenter, it is generally best to open a Networking Support Case with Broadcom Support using the instructions at Creating and managing Broadcom support cases.

This is especially true if the vCenter server networking is delivered by a Virtual Distributed Switch (vDS). In that specific scenario, see vCenter network connectivity lost - Recover vCenter network when connected to a Distributed Switch for the best course of action.

Another factor would be if an operation was being attempted to create a LAG on the host on which vCenter was running.

If these scenarios are not the case, please follow the process guidance below.

Process to troubleshooting virtual machine network connection issues:

Important Note: Validate that each troubleshooting step below is true for the environment. The steps provide instructions or a link to a document, for validating the step and taking corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution. Do not skip a step.

Ensure that the Port Group name(s) associated with the virtual machine's network adapter(s) exists in the vSwitch or Virtual Distributed Switch and is spelled correctly. If not, correct it using Edit Settings on the virtual machine and ensure that the Connected checkbox is selected.
- Note: On a virtual standard switch, virtual machines cannot be placed on a vmkernel port group. Standard switches require that each vmkernel has its own port group.
Ensure that the virtual machine has no underlying issues with storage or it is not in resource contention, as this might result in networking issues with the virtual machine. This can be done by logging into ESX/ESXi or Virtual Center/vCenter Server using the VI/vSphere Client and logging into the virtual machine console. For more information, see Troubleshooting a virtual machine that has stopped responding (VM hang/freeze).
Note: If there are network connectivity issues with the ESXi host or with multiple virtual machines, see ESXi hosts have intermittent or no network connectivity.
Verify that the virtual network adapter is present and connected. If it is connected, try changing to another network, and then changing it back to the original. For more information, see Verifying virtual network adapter is present and connected to the virtual machine
Verify that the networking within the virtual machine's guest operating system is correct. For more information, see Verifying the networking within a Windows based guest operating system for Windows, and Verifying the networking within a Linux based guest operating system for Linux.
Verify that the TCP/IP stack is functioning correctly. For more information, see Troubleshooting virtual machine TCP/IP connection issues.
- If the virtual machine's operating system becomes unresponsive or is stuck, ICMP echo requests (pings) to the VM will fail, resulting in the VM being unreachable on the network. In this scenario, kindly contact the OS admin/vendor for further assistance in resolving the issue.
If this virtual machine was converted from a physical system (P2V), verify that there are no hidden network adapters present. For more information, see Networking Error: IP address already assigned to another adapter.
Verify that the port group of the virtual machine is configured with two vmnics to eliminate a NIC or a physical configuration issue. To isolate a possible issue:
- If the load balancing policy is set to Route Based on Virtual Port ID at the vSwitch or vDS level:
  - Leave one vNIC connected with one uplink on the vSwitch or vDS, then try different vNIC and pNIC combinations until it is determined which virtual machine is losing connectivity.
- If using any port channel techniques (LACP or ether channel):
  - Ensure the physical switch ports are configured as port-channel. For more information on verifying the configuration on the physical switch, see Enable EtherChannel / Link Aggregation Control Protocol (LACP) in ESXi/vCenter and Example Configuration of LACP on VMware, Cisco, and HP switches.
  - Shut down all but one of the physical ports the NICs are connected to, and toggle this between all the ports by keeping only one port connected at a time. Take note of the port/NIC combination where the virtual machines lose network connectivity.
- Also check esxtop output using the n option (for networking) to see which pNIC the virtual machine uses. Try shutting down the ports on the physical switch one at at time to determine where the virtual machine is losing network connectivity. This also attempts to rule out misconfiguration on the physical switch port(s).
Verify that the virtual machine portgroup is configured with the correct VLAN ID, and check the switchports connected to the vmnic allowing that tagged VLAN traffic, for reference Sample configuration of virtual switch VLAN tagging (VST Mode) For additional VLAN configuration on virtual switches, physical switches and virtual machines, see VLAN configuration on virtual switches, physical switches, and virtual machines.
If the virtual machine requires communication with other virtual machines using a large packet size (MTU=9000) and both the VMs' network configuration is correctly set up for using jumbo frames, then check the physical switch port configuration and ensure that the ports on the physical switch are configured for jumbo frames (MTU=9000).
Take packet captures on the ESXi host managing the impacted VM(s) to aid in further investigation by the network team/admin that provides support for the physical network.
- See Capture and Trace Network Packets by Using the pktcap-uw Utility for details.

Note: ESXi does not log VM connectivity or communication (outside of port or physical link failures seen at the host level), so it is important to perform the troubleshooting steps in this article during the time the issue is taking place to be able to determine the cause.

Additional Information

Note: If connection to the default gateway is successful, but connections to IPs in other subnets are unsuccessful, then there may be an issue in routing/Layer 3. Please contact the local network team/admin that provides support for the physical network to determine why the Layer 3 connections are failing.

For more information, see: