After rebooting or updating ESXi, the host loses network connection

Products

• VMware vSphere ESXi

Issue/Introduction

• Updating, patching, or remediating completes successfully via Lifecycle Manager. Upon reboot, there is no ping connection to the remediated host.
• A restart of the management network (DCUI > Restart Management Network) restores network connectivity.
• Management network connectivity is still lost after subsequent reboots of the remediated host.
• The management vmk0 has multiple assigned vmnics.
  • For example:
vmk0's dvportgroup has 4 vmnics to choose from (uplink 1-4 / vmnic 0-3)

Cause

The ESXi host uses any of the configured vmnics for vmk0 in no particular order.

Since vmk0 uses multiple uplinks (e.g., vmnic0-vmnic4) for redundancy or load balancing, traffic distributes across them. If one of the upstream physical switch ports is missing the VLAN#, connectivity drops intermittently or entirely depending on which link the traffic hashes to at that moment.

Every host reboot or restart of the management network reselects which vmnic vmk0 uses.

Resolution

Ensure the vmnics are tagged for the appropriate VLAN.

To assist in narrowing down which vmnic is misconfigured, perform the following:

1. Put a single host into maintenance mode.
2. Identify the current vmnic in use.
   1. SSH to the host as root.
   2. Enter: esxtop
   3. Enter: n
   4. Under the Used-By column, look for vmk0. Look for the associated vmnic# in the column to the right of it.
      Example:
      
3. On another ESXi host, begin pinging the host that is in maintenance mode.
4. On the maintenance mode host, enter the command to down the vmnic in use:

   esxcli network nic down -n vmnic<#>

   where <#> represents the in-use vmnic. Using the above screenshot as a reference, the command would be:

   esxcli network nic down -n vmnic0

5. When the vmnic is down, the ESXi host automatically brings in one of the remaining assigned vmnics to replace it. We can reconfirm by going back to esxtop > n to see what replaces the previous one.
6. Confirm network connectivity by going onto the other ESXi host and reviewing if it is still able to ping the maintenance mode ESXi host.
   • Is it still pingable, or do we start losing pings?
7. Repeat the process by downing and checking pings for the remaining vmnics with:

   esxcli network nic down -n vmnic<#>

8. After all vmnics have been tested, bring up the vmnics by running:

   esxcli network nic up -n vmnic<#>