Fortigate Appliance VM in 'Transparent Mode' Experiences Loop Behavior

Products

• VMware vSphere ESXi

Issue/Introduction

Deployed Fortigate Appliance as a VM and it is running in 'Transparent Mode', or any other appliance that is bridging in the guest OS. 

Promiscuous mode is enabled on the virtual Switch or port group assigned to the vNICs of the appliance VM. 

Packets transmitted from this VM are received on the same vNIC they were transmitted from. 

Physical NICs are Intel cards using i40en drivers, Intel cards using icen drivers, or Broadcom cards using bnxtnet drivers. 

Environment

• VMware vSphere ESXi 

Cause

This looping behavior is due to the VMDQ Loopback feature on the physical NIC. Depending on the MAC address of a transmitted packet, the physical NIC might send the packet back into ESXi rather than upstream to the physical switch. Since promiscuous mode is usually required to be enabled for these bridging appliances, they will receive this packet and then transmit it again. 

Resolution

To turn OFF VMDQ loopback feature on NICs of Hosts where workloads are present as well as Hosts where bridge Edge VMs are present.

 

To turn off VMDQ loopback feature in Intel NIC (on an ESXi 7.0 host) follow the steps below:

1. Install the Intel esxcli plug-in tool by following the Intel esxcli plug-in for managing Intel(r) Ethernet Network Adapters (66772). 

2. Run the following command in SSH console.

# esxcli intnet misc vmdqlb -e 0 -n vmnicX

Note: The above configuration to disable VMDQ loopback feature is not consistent across reboot. To make this setting persistent, please add the command to rc.local by following Modifying the rc.local or local.sh file in ESX/ESXi to execute commands while booting (2043564)

 

Note: For the Intel E810 NIC with the icen driver, the VMDQ loopback feature is only available in version 1.14 and later. Therefore, for this NIC, the only solution is to upgrade the driver to version 1.14 or higher.

 

 

To turn off VMDQ loopback feature on the unified i40en VMware ESX Driver for Intel(R) Ethernet Controllers X710, XL710, XXV710, and X722 family (for a NIC that is on an ESXi 8.0 host):

 

 1. Update the Intel NIC firmware driver to 2.9.2.0

 2. Disable VMDQ on all vmnicX

# esxcli intnet misc vmdqlb set -l 0 -n vmnicX

---

Note:

VMDQ loopback feature is disabled by default with i40en 2.9.2 or later and icen 1.14.2 or later.
Refer to the release notes of the drivers for more details.

Note2:

Inbox driver does not have a feature to disable VMDQ loopback.

 

For Broadcom network cards:
This issue has been observed with driver version 229.0.146.0 and firmware 223.0.205.0 / pkg 22.31.13.70, but not with the latest driver version 232.0.254.0 and its corresponding firmware. For more information on how to download and install the driver, please refer to the KB article: Download and install async drivers in VMware ESXi.  

Additional Information

This feature is mainly beneficial for SR/IOV configurations, and there is no impact to disabling it if SR/IOV is not configured.