The "Permit Total Port Shutdown" setting in the BIOS for the physical network adapter affects traffic when the port is shut down from ESXi on a Dell server

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides detailed information on one option for configuring physical server network ports between an ESXi host and switches.

Symptoms:

The ESXi host is connected to stacked switches via two physical network ports (e.g. vmnic5 and vmnic7), with static mode EtherChannel configured on the switches. If one vmnic is shut down on the ESXi host using the command "esxcli network nic down -n vmnic5", the traffic from a VM to another VM's IP, which was routed through vmnic5, no longer functions.

The affected VM is connected to a port group configured with "Route Based on IP Hash" for uplinks vmnic5 and vmnic7. When vmnic5 is shut down on the ESXi host, traffic is expected to route through vmnic7, but the traffic does not function.

 

Environment

VMware ESXi 7.0.x

VMware ESXi 8.0.x

Cause

On a Dell server, there is a BIOS setting called "Permit Total Port Shutdown" for a network port. When this setting is disabled, the physical network port will not be fully shut down even if the command "esxcli network nic down -n vmnic5" is executed. As a result, the switches may still recognize the ESXi host's network port as being in an up state and may continue to forward traffic to vmnic5, causing the traffic to be lost at the ESXi host. However, for a different source/destination IP combination, the switches may forward traffic to vmnic7, in which case the traffic should function correctly.

Resolution

Enable the "Permit Total Port Shutdown" setting in the BIOS.

Note: Under "NIC in Slot # Port #: <Vendor> Ethernet ...

Contact the hardware vendor if further assistance is required.

 

Additional Information

Note: "Permit Total Port Shutdown" will halt all operations configured on the port including WakeOnLAN and shared LOM.

 

A detailed KB for "Route Based on IP Hash" can be found at KB article Understanding IP Hash load balancing: "For the link-state to work properly, both the physical switch and the ESXi host must recognize the link as down. If there is a discrepancy, traffic may not fail-over to other adapters in the team."

Impact/Risks:

Traffic drop occurs when one vmnic is down while another redundant vmnic remains active.