Troubleshooting the "Too many open files" Error When Keep-Alives Are Used

Products

• CA API Gateway

Issue/Introduction

• The "Too many open files" error can be solved by How to increase the maximum file open limit when receiving a "Too many open files". While that KB article is just a workaround for the current problem, there is a chance that this could happen in the future. This is relevant if Keep-Alive is used.

Environment

All supported versions of the API Gateway

Cause

• It is a good idea to monitor the number of open files over a period of time to see if they are increasing and to find the culprit of what is opening so many files. There are two commands below. The first outputs the lsof to '/Desktop/lsof.log' whereas the second command will display the top 20 processes that are opening the most files. This is explained more in this link.
• lsof > ~/Desktop/lsof.log 
• cat ~/Desktop/lsof.log | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20
• After monitoring the open files and discovering that the Gateway is opening many files, it is possible that it is related to "Keep-Alive". This is used in the "Route via HTTP(s)" assertion, within the "HTTP" Tab.
• When a proxy is configured, the Proxy-Connection: Keep-Alive header gets added by the Gateway. This is done by Apache HttpComponents and HttpClient library. It is not a standard header but is widely used by browsers and proxies. This is used because if a client is sending an HTTP/1.0 Connection header, some proxies may forward this header blindly to the backend server, which could confuse the server with properties of the client connection and those of the proxy connection. In order to avoid this situation, whenever the Client/Gateway knows that the request goes through a proxy, the Connection header is replaced with Proxy-Connection and hence even if the proxy is blindly forwarding the header to the server, it simply ignores it. When the Proxy-Connection: Keep-Alive header is present, most proxies send back Proxy-Connection:Close with the response so that the connection gets closed.

Resolution

In this situation, the proxy may be failing to send the Proxy-Connection:Close with the response which may trigger the "Too many open files" error logs. When the "Proxy-Connection:Close" header is added manually in the "Route Via HTTP(s)" assertion, the open connection should be closed and the exception disappeared.

Additional Information

• CentOS 7 / RHEL 7 - You may not find the lsof command. If not then install it manually
• Related: How to increase the maximum file open limit when receiving a "Too many open files"