kubectl commands are slow to complete in Tanzu Kubernetes Grid 1.2.1

Products

• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x

Issue/Introduction

Symptoms:

•  kubectl commands are slow to complete in Tanzu Kubernetes Grid (TKG) 1.2.1
• The issue is seen when Antrea is used as the CNI
• TKG node VM's are placed on NSX-T-backed virtual networks

Environment

VMware Tanzu Kubernetes Grid 1.x
VMware Tanzu Kubernetes Grid Plus 1.x

Cause

The issue happens if the TKG node VMs are placed on NSX-T-backed virtual networks and the CNI configured is Antrea, which creates a double-Geneve encapsulation. 

 

Resolution

This is a known bug affecting TKG clusters configured with Antrea CNI when the node VMs are placed in NSX-T backed virtual networks. 

It is resolved in NSX 3.1.3 or 3.2.1 if Enhanced Datapath is enabled

Workaround:

To workaround the issue, use Calico as the CNI, which uses VXLAN, instead of Antrea.

If Antrea is required, checksum offloading can be disabled using TKG Configuration File variable ANTREA_DISABLE_UDP_TUNNEL_OFFLOAD