Large number of established connections from capvcd-controller-manager pods to Cloud Director API endpoint
book
Article ID: 325559
calendar_today
Updated On:
Products
VMware Cloud Director
Issue/Introduction
Symptoms:
Kubernetes clusters have been deployed successfully in Cloud Director using Container Service Extension 4.0.x.
Running netstat on the capvcd-controller-manager pods' manager containers show a large numbers of established connections to the Cloud Director API endpoint, for example:
tcp 0 0 10.96.2.2:58742 <VCD_API_IP>:443 ESTABLISHED ... tcp 0 0 10.96.2.2:58756 <VCD_API_IP>:443 ESTABLISHED
A large number of active connections could be over 10000 from one capvcd-controller-manager pod when counting the results of the netstat command, for example:
kubectl exec -ti -n capvcd-system capvcd-controller-manager-<UUID> -c manager -- sh -c 'netstat -n | grep ESTABLISHED | wc -l'
The large number of active connections to the Cloud Director API continues to grow while Kubernetes clusters are deployed.
Performance of Cloud Director degrades dues to the large number of active connections and operations such as vApp deployments are impacted.
Recreating the capvcd-controller-manager pods causes the number of active connections to drop and then rise again once the capvcd-controller-manager pods are recreated.
Environment
VMware Cloud Director 10.x
Cause
This issue can occur if there are issues with TCP sessions from the capvcd-controller-manager pods to the Cloud Director API endpoint when a Web Application Firewall is in front of the Cloud Director API endpoint.
Resolution
Ensure that the Web Application Firewall's settings are not interfering with TCP sessions from the capvcd-controller-manager pods to the Cloud Director API endpoint.