[VMC] Segments behind a Routed T1 Gateway cannot reach SDDC external endpoints

Products

• VMware Cloud on AWS
• VMware Cloud on Dell EMC

Issue/Introduction

Detail this common misconfiguration that can happen when setting up traffic behind a Routed Tier-1 Gateway.

Symptoms:
Customer has a Routed Custom T1 Gateway stood up. There are one/multiple network segments attached to this Custom T1 GW.
Customer is attempting to reach an endpoint which is external to the VMC SDDC and cannot get past the 100.64.x.x address of the CGW Firewall when attempting to run a traceroute between Source > Destination.
The correct DFW FW rules are in place which allows for the connectivity.
The correct Custom T1 Gateway Firewall rules are in place.
A traceflow from within NSX shows the traffic is stopped at the CGW FW. A traceroute will show that the traffic stops at the 100.64.x.x address of the CGW Firewall.
An Aria Operations for Logs query shows we are being dropped by the default deny all rule present on the CGW FW.

Cause

The traffic is being blocked by the Compute Gateway Firewall due to the lack of open FW rules.

Resolution

Ensure the appropriate firewall rules are applied at 3 distinct places:

1. The distributed firewall (DFW)
2. The custom Tier-1 Gateway FW (T1 GW FW)
3. The default Tier-1 Compute Gateway FW (T1 CGW FW)

The rules will need to exist at all 3 levels in order for the traffic to work end-to-end.

Workaround:
Run the workloads on the default T1 CGW with the correct CGW FW rules in place.

Additional Information

Per https://docs.vmware.com/en/VMware-Cloud-on-AWS/services/com.vmware.vmc-aws-networking-security/GUID-658253DB-F384-4040-94B2-DF2AC3C9D396.html, North/South network uplinks are managed by the default CGW. The CGW FW rules are enforced on the uplinks at the T0.

Impact/Risks:
The customer may not be able to setup workloads on the custom Tier-1 Gateway until resolved.