[VMC on Dell EMC] HCX traffic getting blocked at the SDDC Tier-0

Products

• VMware HCX
• VMware Cloud on Dell EMC

Issue/Introduction

To provide a known limitation when using MON in a VMC-D environment.

Symptoms:
A user is attempting to connect from a VM/Segment, that has been extended from OnPrem to VMC-D, to a management IP in the SDDC. The below items can be validated:
 

• The L2E segment has MON enabled
• Both the source VM and destination IP are within the same SDDC
• MON Policy Routes are default values or removed and the subnet 0.0.0.0/0 has been added
• MGW firewall rules are created to allow the compute workload access to a management resource
• Traceroute from the source VM to the destination IP can show the traffic routing to the OnPrem gateway and getting dropped at the VMC-D Tier-0

Cause

This is caused by the protocol, Reverse Path Forwarding (uRPF). This protocol checks incoming packets to see if (1) Do I have a matching entry for the source in the routing table? and (2) Do I use the same interface to reach this source where I received the packet?

 

When MON is enabled for an L2E segment in VMC, HCX will add /32 routes for all the VMs in the segment within the Tier-1 Gateway. In VMC-D, these /32 routes will be advertised to the Tier-0 Gateway over the downlink. The use case for MON is to allow VMs/Segments within the same Tier-1 Gateway to communicate with one another, without the need to route back OnPrem. As a compute VM/Segment will not be on the same Tier-1 Gateway as a management IP/VM, the MON policy routes are leveraged and will route the traffic to the OnPrem gateway.

 

Once the traffic comes back to VMC, the Tier-0 will receive it over the uplink and drop the packet due to uRPF. This is because the SDDC expects to see the traffic coming from the downlink, as the /32 routes are added to the Tier-0 route table, but the traffic is received by the Tier-0 over the uplink, in which uRPF drops it considering it an attack.

 

Resolution

There are only two resolutions to this issue:

1. Do not enable MON on the segment and route all traffic to OnPrem
2. Add a DENY subnet in the MON Policy Routes for the Management Subnet

Note: There is no way to disable uRPF in VMC-D uplink, as this is a security protocol to prevent any threats/attacks on the SDDC

Additional Information

Unicast Reverse Path Forwarding - https://kb.vmware.com/s/article/2127073
 
MON Use Cases - https://docs.vmware.com/en/VMware-HCX/4.7/hcx-user-guide/GUID-0E254D74-60A9-479C-825D-F373C41F40BC.html
 
MON Policy Routes - https://docs.vmware.com/en/VMware-HCX/4.7/hcx-user-guide/GUID-F45B1DB5-C640-4A75-AEC5-45C58B1C9D63.html#GUID-F45B1DB5-C640-4A75-AEC5-45C58B1C9D63

Impact/Risks:
A user will not be able to connect to a management IP/VM in VMC-D from a compute workload, unless the resolution is performed when using MON.