TKG pod network unstable after enabling NSX IDS/IPS

Products

• Tanzu Kubernetes Grid
• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

When NSX IDS/IPS is enabled, the TKG pod network becomes unstable immediately

• Pod network inter name resolution has failed
  • CoreDNS itself is running without error logs
  • K8S SVC access failed between pods
  • Package reconcile is failed
• Node VM network traffic has no impact
  • Host network pods (etcd/kube-apiserver) also have no impact
  • SSH to Node VM works well

Environment

• All TKG version
• All NSX version

Cause

NSX IDS/IPS doesn't support VM layer overlay network (e.g. Encapsulated packet by Antrea), so the checksum in the packet is broken.
As a result, the k8s node VM drops the return packet, making the pod-to-pod network unstable.

Resolution

• Workaround 1: Disable NSX IDS/IPS function
• Workaround 2: Disable Antrea UDP checksum offloading in TKG