API Gateway: Unable to obtain HTTP response from server: Connection Reset

Products

• CA API Gateway

Issue/Introduction

This article will discuss common causes of "Connection reset" errors seen in the API Gateway logs. While not a complete list, this should help with some of the most common areas to focus on to troubleshoot the issue.

When using an HTTP routing assertion or any assertion that uses underlying routes, such as the 'Retrieve OAuth 2.0 Token' assertion, you may notice connection failures and a similar log entry to the one below in the SSG logs:

Error occurred. com.l7tech.common.http.GenericHttpException: Unable to obtain HTTP response from https://www.example.com: Connection reset at com.l7tech.common.http.prov.apache.components.f.getResponse(Unknown Source)

java.net.SocketException: Connection reset

Environment

This article applies to all API Gateway versions logging the "Connection reset" error.

Cause

Connection reset errors are typically caused by the remote server terminating the connection prematurely while not notifying the requestor. In other words, the backend closes the connection while the API Gateway is still sending information over the wire. The API Gateway becomes aware of this termination when it tries to send the next packet of data, as the backend will reject it due to it having already terminated the connection from its side. Since the API Gateway didn't receive any FIN or RST packets before the last data packet had been sent, it assumes it then classifies it as a "Connection reset" once it finds out the backend terminated the connection prematurely. 

There are a number of possible causes for the "Connection reset" error, the most common root causes are below, almost all of which are network-related:

• Backend server or load balancer (or any network devices in between the Gateway and backend server) isn't working as designed
  • Perhaps a bug in a recent software update, for example
• Backend server didn't receive a data packet in time for its own timeout settings and thus closed the connection
  • As long as we know the Gateway sent the data in time, then the backend not receiving it in time suggests a network issue, usually on a load balancer
• Load balancer may be overloaded in connection / traffic and is terminating connections as a result
• Load balancer or backend may have too short of a timeout window set for the request type
  • If it's a long-running batch transaction for example that may take a few minutes to complete, then all devices between the Gateway and backend should support a high-enough timeout period as to not end the connection prematurely

In summary, the issue is usually configuration related (i.e. too low timeout settings on load balancer or backend servers), or may be caused by a network issue which is causing the traffic to take too long to arrive and thus the read timeout gets hit unexpectedly. 

Resolution

Since the issue is most likely outside of the API Gateway, the customer's networking team and backend server admin team should be engaged to review and ensure timeout values are set appropriately and are high enough and that there is no delay in network traffic.

From the API Gateway perspective, the best way to confirm that it's not an API Gateway issue is to take a tcpdump on the API Gateway, then analyzing in a tool such as Wireshark. This should show what the data is doing "on the wire" of the API Gateway. From that network trace, it can be reviewed to see if there's a pattern to the connection reset timing, if it's seen only from a certain backend or multiple backends, and used to confirm if the Gateway received a RST or FIN packet before the Gateway attempted to send the next application data packets. Effectively, it's good to have a network admin review the tcpdump output and determine if it's a healthy connection termination attempt or if it's not following the TCP/IP specifications. In the vast majority of cases, we'll find that the connection termination was abrupt and not according to TCP/IP spec, which is why the "Connection reset" error is logged on the API Gateway.

Once (or if) the API Gateway is removed from the equation via the tcpdump review, then the customer will need to investigate the issue with their other teams on the network side and likely the backend server side, if determined not to be caused by the API Gateway.

Additional Information

• StackOverflow with a list of possible causes of "Connection reset" in Java applications
• Net-Informations article on Java "Connection reset" errors, possible causes, and possible resolutions