Remote kubectl access not working due to "Concurrent connection limit reached for IP" error

Products

• VMware Telco Cloud Automation

Issue/Introduction

Replacing the proxy service will allow TCA manager (TCA-M) to release the connections to the pool in case of failure.

Symptoms:

1. The proxy service permanently denies connections from a given IP address with "Concurrent connection limit reached for IP" error.
2. HttpEndpointListener error seen in TCA Manger kubectl proxy log:

2023-11-24 11:30:07.376 UTC [Connection Acceptor ':8500', , , TxId: ]
ERROR c.p.m.c.t.http.HttpEndpointListener-
java.net.SocketException: Broken pipe (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.base/java.net.SocketOutputStream.write(Unknown Source)
at java.base/sun.security.ssl.SSLSocketOutputRecord.flush(Unknown Source)
at java.base/sun.security.ssl.HandshakeOutStream.flush(Unknown Source)
at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.onProduceCertificateVerify(Unknown Source)
at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.produce(Unknown Source)
at java.base/sun.security.ssl.SSLHandshake.produce(Unknown Source)
at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(Unknown Source)
at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(Unknown Source)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
at com.predic8.membrane.core.transport.http.HttpEndpointListener.writeRateLimitReachedToSource(HttpEndpointListener.java:239)
at com.predic8.membrane.core.transport.http.HttpEndpointListener.run(HttpEndpointListener.java:136)
2023-11-24 11:30:07.377 UTC [Connection Acceptor ':8500', , , TxId: ]
WARN c.p.m.c.t.http.HttpEndpointListener- Concurrent connection limit reached for IP: /#.#.#.#

 

 

Environment

2.3, 3.0

Cause

Due to a bug on a 3rd party networking library, if a connection to Kubernetes is terminated due to a K8S overload / TCA-M max connection limit, the failed connections are not released back to the proxy's pool of available connection slots, causing the proxy to cease communications with the IP addresses associated with the failed connections.

Resolution

Resolved in TCA 3.1.

Workaround:

1. Download the attached proxy-0.1.0.war file..
2. Rename the file to proxy.war and copy to the /tmp directory of the TCA-M appliance.
3. SSH as admin into the TCA-M and and switch user to root.
4. Backup the existing proxy service by running the following command:
   cp /opt/vmware/proxy.war /opt/vmware/proxy.war.backup
5. Update the proxy service with the proxy.war file that was copied to the /tmp directory of the TCA-M by running the following command:
   cp /tmp/proxy.war /opt/vmware/proxy.war 
6. Restart the proxy server:
   systemctl restart proxy 

Additional Information