Not all Worker Nodes serving ingress traffic in cluster deployed by Cloud Director Container Service Extension

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Not all worker nodes serve ingress traffic in Tanzu Kubernetes Grid cluster deployed by Cloud Director Container Service Extension.
• The Kubernetes Service has externalTrafficPolicy=Local set.
• Testing the nodePort on each of the worker nodes will show that only some of them serve traffic, only the nodes on which the service pods actually run.
• Testing the Load Balancer External IP of the service will intermittently fail when the NSX Advanced Load Balancer forwards the request to a node where the pod of the service does not run.

Environment

VMware Cloud Director 10.x

Cause

Cloud Provider Interface (CPI) version 1.3.x and earlier does not support externalTrafficPolicy=Local, it only supports externalTrafficPolicy=Cluster.
These symptoms will occur if the Kubernetes Service in question has externalTrafficPolicy=Local set.

Resolution

To resolve this issue change the externalTrafficPolicy option of the Kubernetes Service from externalTrafficPolicy=Local to externalTrafficPolicy=Cluster.

For example the current service configuration could be viewed with a command like the following:

kubectl get svc -n <namespace of service> <name of service> -oyaml

The service configuration could then be changed using a command such like following:

kubectl edit svc -n <namespace of service> <name of service>