[VMC on AWS ] Routed network may stop working if 100.64.0.0/10 network address is used as on-prem endpoint device address

Products

• VMware Cloud on AWS

Issue/Introduction

Symptoms:
Routed networks are unable to communicate, when either endpoint uses RFC 6598 addressing space (100.64.0.0/10).
Performing a trace route from VMC VM to on-premises when the on-premises endpoint IP address falls under 100.64.0.0/10 subnet.

1 gateway (192.168.xx.xx) 0.522 ms 0.574 ms 0.521 ms
2 xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx) 1.233 ms 1.309 ms 1.380 ms
3 xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx) 0.859 ms 0.932 ms 0.906 ms
4 192.168.xx.xx (192.168.xx.xx) 49.779 ms 49.835 ms 49.804 ms
5 192.168.xx.xx (192.168.xx.xx) 80.187 ms 80.426 ms 80.411 ms
6 192.168.xx.xx (192.168.xx.xx) 87.924 ms 88.178 ms 87.495 ms
7 192.168.xx.xx (192.168.xx.xx) 86.341 ms 87.290 ms 87.515 ms
8 100.64.xx.xx (100.64.xx.xx) 87.890 ms 87.557 ms 87.958 ms

Cause

On-premises endpoint device (Direct Connect, VMware Transit Connect or VPN) is using an endpoint IP address from 100.64.0.0/10 subnet range. This is a reserved subnet for Carrier Grade Network Address Translation (NAT) per RFC 6598.

This block of addresses is specifically meant to be used by Internet Service Providers (or ISPs) that implement Carrier-Grade NAT, to connect their customer-premises equipment (CPE) to their core routers. For more information, see IPv4 Shared Address space  and tools.ietf.org/html/rfc6598.

100.64.0.0/10, 4194304 Private network Shared address space[3] for communications between a service provider and its subscribers when using a carrier-grade NAT.

Resolution

To resolve this issue, use the following guidelines/best practices:

• Do not use the IP range 100.64.0.0/10 addresses. They are always meant to be intermediary relay hops that are internal to the carrier networks.
• VMC uses the same network and specification to implement CGN style networking between the VMC Edge routers in internal routing components which can create a conflict.
• SDDCs that were created using version 1.10 or later have a smaller allocation of 100.64.0.0/16 instead of the entire 100.64.0.0/10 block. Address space other than the 100.64.0.0/16 block can be used for endpoints.
• SDDCs that were upgraded from releases prior to 1.10 do not reflect this change and will continue to have the 100.64.0.0/10 allocation present.
• VMC also uses this IP range between internal routing components it will create a conflict.
• If a packet with either a source or a destination address in the 100.64 address space that conflicts with overlapping address space internal to the SDDC, it will be delivered inside the edge and will never egress.