Modifying the vSwitch security policy is not persistent when SRIOV is enabled

Products

• VMware vSphere ESXi

Issue/Introduction

When the vSwitch is configured with the vSwitch Security feature “Allow MAC Address Change: true”, the SRIOV VMs (VF) refuses to accept the new MAC address. 

vSwitch Security policy (MAC change/Promiscuous Mode) for VF traffic on SRIOV enabled VMs are currently not fully supported

Environment

ESXi 6.x

ESXi 7.x

Cause

In-box i40en driver does not support Mac Change or Promiscuous Mode change.

Resolution

VMware is working as designed. The following is the latest Intel Driver Version: 1.9.5. Currently, with this Driver patch, the Trusted Virtual Function (VF) set on the VM can stay Trusted persistently between VM reboots.

NOTE: VLAN Tag Stripping Control feature is currently not available on Windows VF drivers. Some know issues, VF adapter cannot receive any packet after VM reboot. The probability of issue occurrence increases with the overall number of VFs and number of VMs reboots.

 
Workaround: power off and on VMs with VFs instead of rebooting them. More info: Release_Notes_i40en-1.9.5

Workaround:
At the moment, Intel team has provided the TrustMode configurations, which allows to override the policy on a per VF basis.

• Type ”esxcli intnet sriovnic vf set -v $VFID -n %VMNICID -s off” at VM startup, to disable spoof checking for that VF at the PF.
• Type ”esxcli intnet sriovnic vf set -v $VFID -n %VMNICID –t on” at VM startup, to set that VF as trusted mode at the PF.

Example: 

• Set VF 1 as trusted

  esxcli intnet sriovnic vf -n vmnic0 -v 1 -t on

• Set VF 1 as untrusted

  esxcli intnet sriovnic vf -n vmnic0 -v 1 -t off

• Enable VF spoof-check for VF 1

  esxcli intnet sriovnic vf -v 1 -n vmnic0 -s on

• Disable VF spoof-check for VF 1

  esxcli intnet sriovnic vf -v 1 -n vmnic0 -s off
  

For more information refer to release notes in  More info: Release_Notes_i40en-1.9.5 [ see attachment ]

Additional Information

SR-IOV uses physical functions (PFs) and virtual functions (VFs) to manage global functions for the SR-IOV devices. 

Physical function:
PFs are full PCIe functions that are capable of configuring and managing the SR-IOV functionality. It is possible to configure or control PCIe devices using PFs, and the PF has full ability to move data in and out of the device. 

Virtual function:
VFs are lightweight PCIe functions that support data flowing but have a restricted set of configuration resources.

Note: Theoretically a trusted VF can have the same set of privileged functionality as that of its parent PF. 

With SRIOV being enabled, the virtual machine and the physical adapter exchange data directly without using the VMkernel as an intermediary. Bypassing the VMkernel for networking reduces latency and improves CPU efficiency.

It basically relies on the interaction between the virtual functions (VF) and the physical function (PF) of the NIC port for better performance, and interaction between the driver of the PF and the host switch for traffic control. 

“MAC address changes” and “Forged transmits” are the port group settings that need to be made persistent, in order to allow MAC spoofed frames to be passed between the VF and the NIC.

The VF will be using trust mode configuration via the driver syntax as stated above in the workaround section and it is independent of vmkernel operations. 

Impact/Risks:
You will have to reset the interface in the guest operating system to get the latest MAC address from the virtual device and acquire an IP address.