Zscaler Branch Connector VIP Fails to Respond When Hosted on ESXi Due to Virtual MAC Handling

Products

• VMware NSX

Issue/Introduction

• A Zscaler Branch Connector HA VIP is unreachable when hosted on VMware ESXi, while the individual service IPs remains fully reachable.
• In a typical deployment, Zscaler Branch Connector VMs are configured with two IP addresses: one dedicated to management and another used as the service IP.
• The service IPs handle production traffic, while the Service IP VIP functions as the High Availability (HA) address, providing active/standby failover between the two service IPs.
• ZScaler HA VIP uses the CARP protocol.
• Packet captures on the ESXI uplink indicate that ARP and ICMP echo requests destined for the VIP are ingressing on both uplinks i.e those used for the management interface (ideally eth0) as well as the service IP interface (eth1).

Environment

VMware ESXi 7.x

VMware ESXi 8.x

Cause

In a NIC teamed environment where multiple uplinks are configured for a virtual switch and a port channel or LACP is not configured on the physical switch, the vSwitch will receive a multicast or broadcast packet from the physical network on each vSwitch uplink in the NIC team. All traffic received by the vSwitch will be forwarded to the virtual portgroup in promiscuous mode so the virtual machine guest OS will receive multiple multicast or broadcast packets.

For more information on promiscuous mode, see How promiscuous mode works at the virtual switch and portgroup levels.

Resolution

To prevent this Duplicate Multicast or Broadcast packets, follow the below steps : 

1. Enable Promiscuous mode on the Virtual Switch (vSwitch).
2. Enable MAC Address changes.
3. Enable Forged transmits.

If multiple physical ports/uplinks exist on the same vSwitch, then the Net.ReversePathFwdCheckPromisc option must be enabled in order to work around a vSwitch bug where the multicast traffic loops back to the host, which causes the CARP to not function with link states coalesced messages.

Complete these steps in order to modify the Net.ReversePathFwdCheckPromisc option:

1. Log into the VMware vSphere client.
2. Complete these steps for each VMware host where the vms will be executed, specially if it is an ESXi cluster:
   1. Click host, and navigate to the Configuration tab.
   2. Click System Advanced Settings from the left pane.
   3. Click edit and look for the variable Net.ReversePathFwdCheckPromisc option.
   4. Set the Net.ReversePathFwdCheckPromisc option to 1.
   5. Click OK.

In order for the setting to take effect, promiscuous mode must be toggled off and on (portgroup level). An operation such as a guest OS reboot or a vMotion to another ESXi host with the /Net/ReversePathFwdCheckPromisc setting enabled is sufficient. The setting does not require a reboot of the ESXi host to take affect.

This setting will discard packets coming from uplinks that are not associated with the particular client when promiscuous mode is enabled and will prevent duplicate packets from being received by the guest operating system.

Additional Information

Duplicate Multicast or Broadcast Packets are Received by a Virtual Machine When the Interface is Operating in Promiscuous Mode

Virtual machines unable to form HA cluster using CARP (Common Address Redundancy Protocol)