Large number of established connections from capvcd-controller-manager pods to Cloud Director API endpoint
search cancel

Large number of established connections from capvcd-controller-manager pods to Cloud Director API endpoint

book

Article ID: 325559

calendar_today

Updated On:

Products

VMware Cloud Director

Issue/Introduction

Symptoms:

  • Kubernetes clusters have been deployed successfully in Cloud Director using Container Service Extension 4.0.x.
  • Running netstat on the capvcd-controller-manager pods' manager containers show a large numbers of established connections to the Cloud Director API endpoint, for example:
kubectl exec -ti -n capvcd-system capvcd-controller-manager-<UUID> -c manager -- sh -c 'netstat -n | grep ESTABLISHED'

tcp        0      0 10.96.2.2:58742        <VCD_API_IP>:443        ESTABLISHED
...
tcp        0      0 10.96.2.2:58756        <VCD_API_IP>:443        ESTABLISHED
  • A large number of active connections could be over 10000 from one capvcd-controller-manager pod when counting the results of the netstat command, for example:
kubectl exec -ti -n capvcd-system capvcd-controller-manager-<UUID> -c manager -- sh -c 'netstat -n | grep ESTABLISHED | wc -l'
  • The large number of active connections to the Cloud Director API continues to grow while Kubernetes clusters are deployed.
  • Performance of Cloud Director degrades dues to the large number of active connections and operations such as vApp deployments are impacted.
  • Recreating the capvcd-controller-manager pods causes the number of active connections to drop and then rise again once the capvcd-controller-manager pods are recreated.



Environment

VMware Cloud Director 10.x

Cause

This issue can occur if there are issues with TCP sessions from the capvcd-controller-manager pods to the Cloud Director API endpoint when a Web Application Firewall is in front of the Cloud Director API endpoint.

Resolution

Ensure that the Web Application Firewall's settings are not interfering with TCP sessions from the capvcd-controller-manager pods to the Cloud Director API endpoint.

Additional Information

Powered by Wolken Software