Remote kubectl access not working from Telco Cloud Automation (TCA) 2.3.x due to "Concurrent connection limit reached for IP" error

Products

• VMware
• VMware Telco Cloud Automation

Issue/Introduction

Replacing the proxy service will allow TCA manager (TCA-M) to release the connections to the pool in case of failure.

Symptoms:

1. The proxy service permanently denies connections from a given IP address with "Concurrent connection limit reached for IP" error.
2. HttpEndpointListener error seen in TCA Manger kubectl proxy log:

2023-11-24 11:30:07.376 UTC [Connection Acceptor ':8500', , , TxId: ]
ERROR c.p.m.c.t.http.HttpEndpointListener-
java.net.SocketException: Broken pipe (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.base/java.net.SocketOutputStream.write(Unknown Source)
at java.base/sun.security.ssl.SSLSocketOutputRecord.flush(Unknown Source)
at java.base/sun.security.ssl.HandshakeOutStream.flush(Unknown Source)
at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.onProduceCertificateVerify(Unknown Source)
at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.produce(Unknown Source)
at java.base/sun.security.ssl.SSLHandshake.produce(Unknown Source)
at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(Unknown Source)
at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(Unknown Source)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
at com.predic8.membrane.core.transport.http.HttpEndpointListener.writeRateLimitReachedToSource(HttpEndpointListener.java:239)
at com.predic8.membrane.core.transport.http.HttpEndpointListener.run(HttpEndpointListener.java:136)
2023-11-24 11:30:07.377 UTC [Connection Acceptor ':8500', , , TxId: ]
WARN c.p.m.c.t.http.HttpEndpointListener- Concurrent connection limit reached for IP: /X.X.X.X

 

Environment

VMware Telco Cloud Automation 2.3

Cause

Due to a bug on a 3rd party networking library, if a connection to Kubernetes is terminated due to a K8S overload / TCA-M max connection limit, the failed connections are not released back to the proxy's pool of available connection slots, causing the proxy to cease communications with the IP addresses associated with the failed connections.

Resolution

The problem can be addressed by manually replacing the proxy service (See Workaround) or by upgrading to TCA 3.1.

Workaround:

1. Download the proxy-0.1.0.war file using the link  or the copy attached to the KB.
2. Rename the file to proxy.war and copy to the /tmp directory of the TCA-M appliance.
3. SSH as admin into the TCA-M and and switch user to root.
4. Backup the existing proxy service by running the following command:
   cp /opt/vmware/proxy.war /opt/vmware/proxy.war.backup
5. Update the proxy service with the proxy.war file that was copied to the /tmp directory of the TCA-M by running the following command:
   cp /tmp/proxy.war /opt/vmware/proxy.war 
6. Restart the proxy server:
   systemctl restart proxy 

Additional Information

Similar issue seen in TCA 2.1.x and 2.2: kubectl access is not working from TCA due to "Concurrent connection limit reached for IP" (91493)

Impact/Risks:

Impacts TCA 2.3.x