Large number of established connections from capvcd-controller-manager pods to Cloud Director API endpoint

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Kubernetes clusters have been deployed successfully in Cloud Director using Container Service Extension 4.0.x.
• Running netstat on the capvcd-controller-manager pods' manager containers show a large numbers of established connections to the Cloud Director API endpoint, for example:

kubectl exec -ti -n capvcd-system capvcd-controller-manager-<UUID> -c manager -- sh -c 'netstat -n | grep ESTABLISHED'

tcp        0      0 100.96.2.2:58742        <VCD_API_IP>:443        ESTABLISHED
...
tcp        0      0 100.96.2.2:58756        <VCD_API_IP>:443        ESTABLISHED

• A large number of active connections could be over 10000 from one capvcd-controller-manager pod when counting the results of the netstat command, for example:

kubectl exec -ti -n capvcd-system capvcd-controller-manager-<UUID> -c manager -- sh -c 'netstat -n | grep ESTABLISHED | wc -l'

• The large number of active connections to the Cloud Director API continues to grow while Kubernetes clusters are deployed.
• Performance of Cloud Director degrades dues to the large number of active connections and operations such as vApp deployments are impacted.
• Recreating the capvcd-controller-manager pods causes the number of active connections to drop and then rise again once the capvcd-controller-manager pods are recreated.

Environment

VMware Cloud Director 10.x

Cause

This issue can occur if there are issues with TCP sessions from the capvcd-controller-manager pods to the Cloud Director API endpoint when a Web Application Firewall is in front of the Cloud Director API endpoint.

Resolution

Ensure that the Web Application Firewall's settings are not interfering with TCP sessions from the capvcd-controller-manager pods to the Cloud Director API endpoint.

Additional Information

About VMware Cloud Director Container Service Extension.
Organization Virtual Data Center Prerequisites for Kubernetes Cluster Deployment.