Understanding Internet Control Message Protocol redirects

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

This article provides information about the behavior associated with Internet Control Message Protocol (ICMP) redirects and how to troubleshoot the behavior.

Symptoms:
Certain hosts in your environment are no longer able to communicate with VMware vCenter Server.

Cause

Network interrupts can cause a network device (usually by a router or firewall) to send an ICMP redirect. ICMP redirects are used to notify the hosts on the data link that a better route is available for a particular destination.

This can be problematic for VMware vSphere environments because it has the ability to interrupt connectivity between the hosts and vCenter Server. Under normal circumstances, you can expect a second ICMP redirect to be sent to the host when the issue has been resolved. However, it has been observed that this does not always happen.

Resolution

To confirm if an ICMP redirect is the cause of your connectivity problems, perform a network capture on the affected host and vCenter Server.

If both devices connect through the same router or firewall, their traffic should be destined for the same MAC address.

When you perform a capture, you see that one entity is sending the traffic to a MAC address that is unfamiliar. The unfamiliar MAC address is actually the MAC of the ICMP redirected gateway.

- To perform a capture on an ESXi host, run this command, where vmk# is the vmk associated with the management interface of the host and x.x.x.x is the IP of the vCenter Server.
```
# tcpdump-uw -e -i vmk# -nn host x.x.x.x
```
The option -i denotes the interface and -e allows tcpdump to display the MAC addresses (by default it does not).

To work around this issue, perform one of these options:
1. Work with the physical switch, router, or firewall vendor to determine if there is a way to force an ICMP redirect to be sent correcting the communication issue.
2. Reboot the host to clear the ICMP redirect from the hypervisor.
3. Delete the affected interface and recreate it.
This issue generally occurs on host Service Console or Management interfaces. The following steps walk you through removing and recreating a service console or management interface. Perform these steps from a console session. When the console session is removed, you lose connectivity to the host until the interface is recreated. The virtual machines remain unaffected during this process.
1. Find where the vmknic in question is located using the following command:
```
# esxcfg-vmknic -l 
```
2. Find the vSwitch or vDSwitch name on which the interface is running. Run the command:
```
# esxcfg-vswitch -l | less
```
3. Record the name exactly as it appears (it is case sensitive). If using a vDS, record the port number that the interface is associated with.
4. Remove the offending interface by running one of these commands:
  - To remove a vmk from a vSwitch, run:
```
# esxcfg-vmknic -d PortGroupName
```
  - To remove a vmk from a vDSwitch, run:
```
# esxcfg-vmknic -d -v port_number -s vDSname
```
5. Recreate the interface by running one of these commands:
  - To create a vmk interface on a vSwitch, run:
```
# esxcfg-vmknic -a -i x.x.x.x -n x.x.x.x -p PortGroupName vSwitchName
```
  - To create a vmk interface on a vDS, run:
```
# esxcfg-vmknic -a -i x.x.x.x -n x.x.x.x PortGroupName -s vDSwitchName
```

Additional Information

For more information on how to capture network traffic in ESXi and vCenter, please refer to the following articles:

Impact/Risks:
Deleting and recreating the Service Console or Management Interface of a host causes the host to drop from vCenter Server. The primary IP of the host is unreachable during this procedure, which may trigger other monitoring alerts in your environment. Use your discretion to determine who you should notify about the changes you are going to make. The entire delete and recreate process takes approximately 5 minutes.