Zscaler Branch Connector VIP Fails to Respond When configured in NSX to Virtual MAC Handling

Products

• VMware NSX

Issue/Introduction

• A Zscaler Branch Connector HA VIP is unreachable when hosted on VMware ESXi, while the individual service IPs remains fully reachable.
• In a typical deployment, Zscaler Branch Connector VMs are configured with two IP addresses: one dedicated to management and another used as the service IP.
• The service IPs handle production traffic, while the Service IP VIP functions as the High Availability (HA) address, providing active/standby failover between the two service IPs.
• ZScaler HA VIP uses the CARP protocol.
• Packet captures on the ESXI uplink indicate that ARP and ICMP echo requests destined for the VIP are ingressing on both uplinks i.e those used for the management interface (ideally eth0) as well as the service IP interface (eth1).

Environment

VMware NSX

Cause

In a NIC teamed environment where multiple uplinks are configured for a virtual switch and a port channel or LACP is not configured on the physical switch, the vSwitch will receive a multicast or broadcast packet from the physical network on each vSwitch uplink in the NIC team. All traffic received by the vSwitch will be forwarded to the NSX segment in promiscuous mode so the virtual machine guest OS will receive multiple multicast or broadcast packets.

For more information on promiscuous mode, see How promiscuous mode works at the virtual switch and portgroup levels

Resolution

To configure the requested security and traffic settings in NSX (4.x/5.x/9.x), you must modify two distinct segment profiles: the Segment Security Profile and the MAC Discovery Profile.

Enable Promiscuous Mode, MAC Changes, and Forged Transmits 

These settings are part of the Segment Security Profile.

1. Log in to the NSX Manager UI.
2. Navigate to Networking > Segments > Segment Profiles.
3. Click the Segment Security tab.

You can edit the default-segment-security-profile (not recommended) or click Add Segment Security Profile to create a new one.

1. Set the following toggles to ON (Enabled)
2. Promiscuous Mode: Allows the guest adapter to receive all frames.
3. MAC Address Change: Allows the guest to change its MAC address.
4. Forged Transmits: Allows the guest to send frames with a source MAC different from the one assigned to the vNIC.
5. Click Save.

Disable Unknown Unicast Flooding

This setting is managed within the MAC Discovery Profile.

While still in Networking > Segments > Segment Profiles, click the MAC Discovery tab.

1. Click Add MAC Discovery Profile.
2. Locate the toggle for Unknown Unicast Flooding.
3. Set the toggle to OFF (Disabled).
4. Click Save.

Note: Disabling this prevents the segment from flooding frames with unknown destination MACs to all ports, which is often used to prevent traffic storms

Apply Profiles to the Segment

The settings will not take effect until the profiles are attached to your specific segment.

1. Navigate to Networking > Segments.
2. Locate your segment and click Edit.
3. Expand the Segment Profiles section.
4. Change the dropdowns for Segment Security and MAC Discovery to the custom profiles you created.
5. Click Save

If multiple physical ports/uplinks exist on the same vSwitch, then the Net.ReversePathFwdCheckPromisc option must be enabled in order to work around a vSwitch bug where the multicast traffic loops back to the host, which causes the CARP to not function with link states coalesced messages.

Complete these steps in order to modify the Net.ReversePathFwdCheckPromisc option:

Complete these steps for each VMware host where the vms will be executed, especially if it is an ESXi cluster:

1. Log into the VMware vSphere client.
2. Click host, and navigate to the Configuration tab.
3. Click System Advanced Settings from the left pane.
4. Click edit and look for the variable Net.ReversePathFwdCheckPromisc option.
5. Set the Net.ReversePathFwdCheckPromisc option to 1.
6. Click OK.

In order for the setting to take effect, promiscuous mode must be toggled off and on (portgroup level). An operation such as a guest OS reboot or a vMotion to another ESXi host with the /Net/ReversePathFwdCheckPromisc setting enabled is sufficient.

Note:  The setting does not require a reboot of the ESXi host to take effect.

This setting will discard packets coming from uplinks that are not associated with the particular client when promiscuous mode is enabled and will prevent duplicate packets from being received by the guest operating system.

Additional Information

Duplicate Multicast or Broadcast Packets are Received by a Virtual Machine When the Interface is Operating in Promiscuous Mode

Virtual machines unable to form HA cluster using CARP (Common Address Redundancy Protocol)

VM connectivity lost to fortigate gateway.