HCX - MON Compatibility with 3rd Party Gateway Routers using a First Hop Redundancy Protocol (FHRP)

Products

• VMware HCX

Issue/Introduction

This document describes an issue related to HCX MON functionality when Onprem Gateway has First Hop Redundancy Protocols (FHRP) enabled.

Symptoms:

When the VMware HCX Mobility Optimized Networking (MON) feature is enabled, a communication failure may be experienced from the cloud site Virtual Machines (VMs) towards the on premise default gateway router over the network extension. This occurs when a First Hop Redundancy Protocol (FHRP) is used by some 3rd party on premise default gateway routers.

First Hop Redundancy Protocols (FHRP) allow multiple redundant 3rd party default gateway routers that exist at the same site to share a virtual IP. Some FHRP protocols include:

• Virtual Router Redundancy Protocol (VRRP)
• Hot Standby Router Protocol (HSRP)
• Gateway Load Balancing Protocol (GLBP)

Note: HCX MON functionality is not compatible with Cisco Gateway Load Balancing Protocol (GLBP).

Cause

HCX Mobility Optimized Networking (MON) works by checking each data packets that goes from a MON-enabled cloud VM to a destination that is outside the subnet (i.e. on a different network). These packets can either be steered towards the Onprem Or Cloud router, depending on network adjacency and customer-defined policy routes.
Note: Refer knowledge Base 321599 for more info on HCX MON Operations.

The steering mechanism works by rewriting the destination MAC address of packet headers where the destination IP is Onprem default gateway/router .
In order for packets to be steered towards the Onprem router, the HCX Cloud/Target Network Extension (NE) appliance must learn the MAC address of the Onprem router.
Whenever any ethernet frame receives on the HCX Cloud/Target NE bridge port from cloud VM, it will check destination mac and then packet will be rewritten with the Onprem router MAC that was learned and subsequently forwarded to Onprem side over extended datapath.

Since the NE appliance is acting as a transparent bridge to the extended network, and thus has no in-band IP address on that network, it can not generate ARP packets to learn the Onprem gateway MAC, since an ARP packet should have a sender IP address.
To handle this requirement, the NE appliance has two primary methods of learning the Onprem gateway MAC:

1. Passively monitoring network traffic and noticing when a cloud VM (with router location set to Onprem) receives an ARP response from the Onprem gateway.
2. Sending ARP requests (probes) for the gateway where the sender protocol address is "0.0.0.0", also known as Duplicate Address Detection (D.A.D). Refer rfc5227 for more info on ARP Probe/Announcement.

Note: D.A.D packet may not be honored/responded by all families of routers (especially when VRRP or HSRP is enabled).
For this situation, below method can be used for learning Onprem gateway MAC:

1. Send a regular ARP request using a "shadow" IP, following this process:
   • Wait for a workload cloud VM to send an ARP request which will be detected by cloud NE appliance (i.e. an ethernet data packet that passes through it on the way to the Onprem)
   • Inspect the ARP REQ packet and find the sender IP and MAC address, referred to as "shadow" address.
2. The cloud NE appliance uses this shadow IP and MAC address to send real/regular ARP REQ for the Onprem router.

Ideally, When NE appliance sends the ARP REQ using shadow IP towards Onprem Gateway, the shadow MAC address should be appearing in the ARP packet under sender hardware address field, as well as in the Ethernet header source MAC address field, as shown below:



However, it has been found that the cloud NE appliance is putting its own bridge interface MAC address in the Ethernet header which is causing "ARP cache poisoning" effect to the Onprem router, where the router is updating its ARP cache for the shadow IP against NE bridge MAC.

This causes a period of packet loss until the router invalidates the current arp-cache and sends a new ARP request to re-learn the correct MAC. The cycle repeats every 5 seconds when the NE appliance sends the next instance of ARP probe packets.

Resolution

This is fixed in HCX 4.8.0 release.

Workaround:

To remediate the packet loss and recover extended datapath, user is recommended to disable the use of FHRP for the default gateway router at Onprem when the HCX MON feature is in use.

Also, The problematic ARP probe needs to be disabled by running below commands on the cloud/target NE appliance:

• Login to HCX Cloud/Target Manager using admin.
• Go to CCLI >> list >> Go {NE_Appliance} >> ssh

touch /opt/vmware/cgw/config/DisableVrrpSupport
systemctl restart cgw

Note: If user has enabled HCX NE HA (High Availability) feature then restarting the HCX "cgw" service will trigger an HA event.
Please follow below steps/recommendations to disable ARP probes when HA is enabled:

• The above commands must be executed on all members of an HA group.
• Run the command on the Standby NE Appliance first.
• Wait for HA status to go back into green state.
• Then run the command on the Active NE Appliance, which will trigger an auto failover to the standby and few seconds of packet loss may be experienced.
• There will be ~5 min delay from when an HA event occurs and status changes in the HCX UI. Wait for at least 5 mins for the HCX UI to show HA going back into green state after making the above changes.

IMPORTANT: Once HSRP/VRRP has been disabled and the router configuration has been changed to allow it to respond to the D.A.D. probes, then no further steps are necessary. The NE appliance will be able to learn the Onprem gateway MAC address from the response through D.A.D. probes.

However, If the router doesn’t respond to D.A.D probes even after disabling FHRP(HSRP/VRRP) protocol, then additional steps are needed to ensure that the Onprem router MAC address is learned always.

• As described in method 1 under cause section, the NE appliance learns the Onprem router MAC address from passively monitoring network traffic and watching for ARP RES from the Onprem gateway towards a cloud VM.
• To force this to happen constantly:
  • A workload VM can be placed/migrated in the cloud.
  • Attach to the MON-enabled extended segment.
  • Set the VM router location as Onprem (In HCX MON configuration using NE wizard).
  • The workload VM can start sending ARP packets (using "arping –b") for the Onprem gateway IP continuously, for example every 5 secs.

Note: When windows server attached to the extended segment, use ("arp -d") under CMD to start re-learning MAC address for the Onprem gateway IP.          

Restrictions & other Considerations

1. The above applied changes in HCX NE Appliances are not persistent across redeploy. If the NE appliance is redeployed, then the above steps must be executed post redeploy.
2. If the entire HA group is redeployed, then the steps must be repeated as per above recommendations on both nodes of the HA group (at cloud).
3. When Service Mesh "Resync" needs to be performed, please ensure the action shouldn't result an NE appliance redeploy which can be checked before confirming the "Resync" using UI.

Configuration Validation & Datapath Monitoring

1. Check if the file "/opt/vmware/cgw/config/DisableVrrpSupport" still exists in the Cloud NE Appliance filesystem. Follow the same method to login via CCLI.
2. Verify if Onprem gateway MAC has been learned. Use HCX NE ebtable rules as below:
   • Go to CCLI >> list >> Go {NE_Appliance} >> ssh
   • On Cloud NE appliance, run below CLI and check if "--to-dst" field is the Onprem router's MAC address.

ebtables -t nat -L | grep "dnat" | grep -v "ARP"

Example:

-d 2:50:56:##:##:58 -i vNic_2 -j dnat --to-dst 0:50:56:##:##:da --dnat-target ACCEPT

0:50:56:##:##:da is Onprem router's MAC address.
2:50:56:##:##:58 is HCX Stitching MAC address.

Additionally, Implement a central monitoring VM that has a continuous ping going to various cloud VMs, which can generate alerts if any intermittent packet loss occurred over extended datapath.

Additional Information

Impact/Risks:

• An intermittent communication failure may be experienced from the cloud site Virtual Machines (VMs) towards the on premise connector site default gateway over the extended network datapath.
• This issue will not impact the extended datapath over non-FHRP enabled gateways.
• This will not impact plain Layer-2 extension without MON functionality.