Cannot connect to harbor registry from VMS on VPC Segment in a new VKS deployment

Products

• VMware NSX

Issue/Introduction

• You are trying to deploy Supervisor services on a newly deployed environment.
• When attempting to connect to Harbor Registry from a VM on NSX Segment, it fails.
• Ping works from the NSX Segment-connected VM to the harbor, suggesting no L3 connectivity issue.
• curl to the harbor doesn't work and may fail with certificate-related errors.
  
  Reason: ReconcileFailed. Message: vendir: Error: Syncing directory '0': Syncing directory: with imgpkgBundle cont ents: Fetching image: Error while preparing a transport to talk with the registry: Unable to create round tripper: Get "https://harbor.test.com/v2/: tls: failed to verify certificate: x509: certificate signed by unknown authority (hint: The CA Certificate from URL is unknown/invalid. Add valid CA certificate to the kapp-controller configuration to reconcile successfully).

Environment

VMware NSX

Cause

Geneve encapsulated packets are dropped on the physical network owing to higher MTU than configured value of 1500 bytes.

Resolution

Make sure the underlay physical network supports at least 1600 bytes of MTU.

Additional Information

• NSX supports configuring the Edge TEP and Host TEP in the same VLAN or different VLAN.
• In most cases, the L2 interfaces on the switches in the datacenter fabric will be configured with 9000/9216 bytes of MTU.
• However, when the Edge TEP and Host TEPs are on different networks, the packets that need to traverse between the Edge and Host TEP will need to be routed via a L3 interface on the physical network. The L3 interfaces that are the gateways for the TEPs also need to be configured with at least 1600 bytes of MTU(9000/9216 preferred based on the hardware/vendor support) to support the additional header length that Geneve encapsulation adds.
  
  To verify the Edge and Host TEP connectivity:
  
  To validate the vmkping from the ESXi host to the Edge TEP.
  
  vmkping -S vxlan -s 1572 -I vmk10 -d <Edge TEP IP>
-S = Stack
-s = size
-I = Interface
-d = DontFragment

Note: Test it with both lower MTU and Jumbo MTU

vmkping -S vxlan -s 1472 -I vmk10 -d <Edge TEP IP>  --> This works confirms that there are no routing issues and L3 connectivity is good
vmkping -S vxlan -s 1572 -I vmk10 -d <Edge TEP IP> --> This doesn't work, while the previous one works, suggesting that the underlay is configured with only 1500 bytes and not meeting the prerequisites

       To validate the vmkping between Edge and ESXi host TEP.

Go into the TUNNEL VRF(vrf 0 In most cases, verify by running the command get gateways on the admin mode on NSX Edge) and run the command,

edge01(vrf)> ping <ESXi TEP IP> source <Edge TEP IP> size 1472 dfbit enable --> This works confirms that there are no routing issues and L3 connectivity is good
edge01(vrf)> ping <ESXi TEP IP> source <Edge TEP IP> size 1572 dfbit enable --> This doesn't work while the previous one works, suggesting that the underlay is configured with only 1500 bytes and does not meet the prerequisites.