[VMC on AWS] Understanding Primary and Secondary VPC CIDRs

Products

• VMware Cloud on AWS

Issue/Introduction

To provide justification in why a Secondary VPC CIDR is not working as expected. 

Symptoms:
A subnet has been created in the VPC other than the connected subnet.
The subnet used to connect the VPC to the SDDC is the "Primary VPC CIDR". 
Any additional subnets in the VPC are "Secondary VPC CIDRs"
Communcation to and from the Secondary CIDR (xVPC communication) is not working as expected. 

Cause

In VMC, only the Primary VPC CIDR is supported to communicate over the VPC.
When the attempting to use the Secondary VPC CIDR, communication can be seen going over the Internet, Direct Connect, or TGW interfaces instead of the xVPC interface.
The NSX Edge installs the routes for the Connected (Primary) VPC CIDR. 
Routes for Secondary VPC CIDRs are not created on the NSX EDGEs and will cause communication to these CIDRs to fail. 
This is by design and is expected behavior. 

Resolution

There is no resolution as this is by design. 

Workaround:
There are two workarounds for this:
1. Update the remote instance to use an address that is within the Primary VPC CIDR. This is the recommended workaround. 
2. Create a new VPC subnet that includes the addresses of the required instances and relink this new VPC to the SDDC. To have the SDDC relinked to a new VPC, please open a ticket with the VMC support team: How do I get support

Additional Information

Documentation confirming only the Primary CIDR is supported in VMC:

• Designlet: VMware Cloud on AWS Connected VPC to Native AWS
• VPC, ENI, and EIP Requirements
• Connecting EC2 with SDDC Workloads
• Selecting IP Subnets and Connectivity for your SDDC (Section 2)

Impact/Risks:
Communication over the xVPC interface to the Secondary VPC CIDR will not work.