kubectl access is not working from TCA due to "Concurrent connection limit reached for IP"
search cancel

kubectl access is not working from TCA due to "Concurrent connection limit reached for IP"


Article ID: 345741


Updated On:


VMware VMware Telco Cloud Automation



This issue can happen when you are using kubectl to access a workload cluster. The kubeconfig is downloaded from TCA UI -> Virtual Infrastructure.
And I got random failure of TooManyRequests. 

While doing deployment of CNF, following error is seen in TCA Manger kubectl proxy log:

2023-02-10 20:28:40.005 UTC [Connection Acceptor '*:8500', , , TxId: ]WARN  c.p.m.c.t.http.HttpEndpointListener- Concurrent connection limit reached for IP: /
2023-02-15 16:35:45.771 UTC [Connection Acceptor '*:8500', , , TxId: ]
                ERROR c.p.m.c.t.http.HttpEndpointListener-
java.net.SocketException: Broken pipe (Write failed)
        at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source)
        at java.base/java.net.SocketOutputStream.write(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketOutputRecord.flush(Unknown Source)
        at java.base/sun.security.ssl.HandshakeOutStream.flush(Unknown Source)
        at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.onProduceCertificateVerify(Unknown Source)
        at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.produce(Unknown Source)
        at java.base/sun.security.ssl.SSLHandshake.produce(Unknown Source)
        at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(Unknown Source)
        at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(Unknown Source)
        at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(Unknown Source)
        at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(Unknown Source)
        at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
        at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
        at com.predic8.membrane.core.transport.http.HttpEndpointListener.writeRateLimitReachedToSource(HttpEndpointListener.java:236)
        at com.predic8.membrane.core.transport.http.HttpEndpointListener.run(HttpEndpointListener.java:134)
2023-02-15 16:35:45.772 UTC [Connection Acceptor '*:8500', , , TxId: ]
                WARN  c.p.m.c.t.http.HttpEndpointListener- Concurrent connection limit reached for IP: /
Received the following content 
2023-01-24 17:16:24.841 UTC [Connection Acceptor '*:8500', , , TxId: ]
                WARN  c.p.m.c.t.http.HttpEndpointListener- Concurrent connection limit reached for IP: /
Received the following content


VMware Telco Cloud Automation 2.1
VMware Telco Cloud Automation 2.1.1
VMware Telco Cloud Automation 2.2


Issue is happening because TCP connections that are established between TCA-M & TCA-CP are not quickly terminated when the connections from source to TCA-M are terminated. In TCA 2.1 TCP connections are periodically terminated every 8 seconds and the max number of concurrent connections allowed is 60. It is possible that there can be more than 60 connections initiated from same source with in 8 seconds and hence some connections will start getting rejected.


This issue is fixed in TCA 2.2 release.

With TCA 2.2, fix is to increase max concurrent connection limit to 2048 between TCA-M and TCA-CP.


As a work around customer can try any of the below 2 methods:

1) Re-start proxy service via TCA-CP appliance management UI. TCA-CP appliance management UI can be accessed over https://<tca-cp-ip-address>:9443 and logging in with username as admin and corresponding password. Go to Appliance Summary page here and click on STOP and then START for Proxy Service. Doing this will get rid of stale connections and restarting proxy service wont impact other TCA-CP functionality as such. 

2) get the kubeconfig for the workload cluster directly from management cluster. This can be done by either logging into management cluster control plane node or TCA-CP via SSH. If you are logging into TCA-CP via SSH, confirm that context is set to management cluster by doing 'kubectl config get-contexts' command. Usually, management cluster context will be available on TCA-CP for root user. So you can SSH as admin and switch user to root. After this, get the kubeconfig of corresponding workload cluster using this command:

kubectl get secret -n <cluster_name>  <cluster_name>-kubeconfig -o jsonpath='{.data.value}'|base64 -d  > cluster_name.kubeconfig