Stack Trace Encountered in Policy Manager: "Could Not Access HTTP Invoker Remote Service" and "Remote Host Closed Connection During Handshake" Errors Observed
search cancel

Stack Trace Encountered in Policy Manager: "Could Not Access HTTP Invoker Remote Service" and "Remote Host Closed Connection During Handshake" Errors Observed

book

Article ID: 8807

calendar_today

Updated On:

Products

STARTER PACK-7 CA Rapid App Security CA API Gateway

Issue/Introduction

When connecting to the API Gateway via Policy Manager, some SSL-related stack traces may be observed. An example of such a stack trace snippet is below.

org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [https://securespangateway/ssg/manager/<Path>]; nested exception is javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake 
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:212) 
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:145) 
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) 
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) 
at com.sun.proxy.$Proxy9.getAssertionResourceData(Unknown Source) 
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at com.l7tech.gateway.common.spring.remoting.http.a.call(Unknown Source) 
at com.l7tech.gateway.common.spring.remoting.http.SecureHttpComponentsHttpInvokerRequestExecutor.doWithSession(Unknown Source) 
at com.l7tech.gateway.common.spring.remoting.http.RemotingContext.doRemoteInvocation(Unknown Source) 
at com.l7tech.console.util.AdminContextImpl.access$001(Unknown Source) 
at com.l7tech.console.util.c.call(Unknown Source) 
at com.l7tech.gui.util.at.construct(Unknown Source) 
at com.l7tech.gui.util.av.run(Unknown Source) 
at java.lang.Thread.run(Thread.java:745) 
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake 
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992) 
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) 
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747) 
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) 
at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:147) 
at org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:154) 
at org.apache.http.impl.AbstractHttpClientConnection.doFlush(AbstractHttpClientConnection.java:278) 
at org.apache.http.impl.AbstractHttpClientConnection.flush(AbstractHttpClientConnection.java:283) 
at org.apache.http.impl.conn.ManagedClientConnectionImpl.flush(ManagedClientConnectionImpl.java:175) 
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:232) 
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) 
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715) 
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520) 
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) 
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:827) 
at com.l7tech.gateway.common.spring.remoting.http.SecureHttpComponentsHttpInvokerRequestExecutor.executePostMethod(Unknown Source) 
at com.l7tech.gateway.common.spring.remoting.http.SecureHttpComponentsHttpInvokerRequestExecutor.doExecuteRequest(Unknown Source) 
at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:136) 
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:192) 
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:174) 
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:142) 
... 14 more 
Caused by: java.io.EOFException: SSL peer shut down incorrectly 
at sun.security.ssl.InputRecord.read(InputRecord.java:505) 
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)

Environment

All Policy Manager versions connecting to an API Gateway over HTTPS where the SSL session is not persisted.SaaS Portal & SaaS Gateway customers: Please review the bolded note in the Resolution section as it applies to the CA SaaS infrastructure.

Cause

The root cause of this behaviour is often the lack of what is called session persistence (aka "sticky sessions" or it's equivalent - language depends on the vendor of the load balancer).  In a typical environment, the Gateway cluster will be fronted by load balancer(s). When there are multiple Gateway nodes, sometimes - depending on the configuration of the load balancer - the load balancer may send traffic from the same Policy Manager request to multiple nodes during the handshake rather than just to one Gateway node.

Example scenario where this is often observed:

  1. Policy Manager attempts to connect to a VIP (Virtual IP) on a Load Balancer to reach the Gateway cluster.
  2. The Load Balancer hosting the VIP then relays the first few client packets to the first Gateway node, including the "Client Hello" packet near the beginning of the handshake process.
  3. As the first few packets are acknowledged (ACK), the Policy Manager then sends the next few packets to the VIP to continue the handshake process.
  4. This time, the next wave of packets to the VIP are "load balanced" by sending them to the second Gateway node.
  5. The second Gateway node, however, has no context of the original conversation from step 2. As such, it determines that the handshake process does not meet RFC requirements and terminates the connection.
  6. This termination generates a stack trace on the Policy Manager client as it claims the remote host closed the connection prematurely.

 

Resolution

In this kind of scenario, session persistence should be enabled on the load balancer to keep all the traffic of that single request / thread to one Gateway node rather than sending half the conversation to one node and the other half to another node.

The ideal scenario where this would not be observed:

  1. Policy Manager attempts to connect to a VIP (Virtual IP) on a Load Balancer to reach the Gateway cluster.
  2. The Load Balancer hosting the VIP then relays the first few client packets to the first Gateway node, including the "Client Hello" packet near the beginning of the handshake process.
  3. As the first few packets are acknowledged (ACK), the Policy Manager then sends the next few packets to the VIP to continue the handshake process.
  4. The next wave of packets to the VIP continue on to the same Gateway the conversation originally went to so it has complete context of the communication - most importantly the actual handshake.

The load balancer should be configured for session persistence. Please review vendor documentation on the load balancer to enable that functionality. A quick overview of this "best practice" for the load balancer configuration fronting the Gateway is in the Gateway product documentation.

Customers of the CA SaaS Portal & SaaS Gateway: In the CA SaaS infrastructure, the port to clients should use in Policy Manager after the Gateway hostname is 9443 rather than the usual 8443, as 9443 is the port configured for session persistence inside of the CA SaaS infrastructure.

Additional Information

Powered by Wolken Software