API Gateway: 750 open files that belong to Java

Products

• CA API Gateway
• API SECURITY
• STARTER PACK-7

Issue/Introduction

When checking one of the servers, there are around 750 open files and they all belong to the java process for gateway user. Is this the expected behavior or not? Normally, the count of open files would be 163. Why there is sudden hike in the file count? The reason for the same. 

java      11113 gateway  mem    REG  253,1    83871 131413 /opt/mount1/SecureSpan/Gateway/runtime/lib/layer7-kerberos-8.2.01.jar

java      11113 gateway  mem    REG  253,1  1899491 131415 /opt/mount1/SecureSpan/Gateway/runtime/lib/layer7-policy-8.2.01.jar

java      11113 gateway  mem    REG  253,1   102995 131411 /opt/mount1/SecureSpan/Gateway/runtime/lib/layer7-identity-8.2.01.jar

java      11113 gateway  mem    REG  253,1    26477 131368 /opt/mount1/SecureSpan/Gateway/runtime/lib/httpmime-4.2.5.jar

java      11113 gateway  mem    REG  253,1   433368 131366 /opt/mount1/SecureSpan/Gateway/runtime/lib/httpclient-4.2.5.jar

java      11113 gateway  mem    REG  253,1   227275 131367 /opt/mount1/SecureSpan/Gateway/runtime/lib/httpcore-4.2.4.jar

java      11113 gateway  mem    REG  253,1    24163 131426 /opt/mount1/SecureSpan/Gateway/runtime/lib/mockwebserver-20130505.jar

java      11113 gateway  mem    REG  253,1   663730 131474 /opt/mount1/SecureSpan/Gateway/runtime/lib/xss4j-1.0.jar

java      11113 gateway  mem    REG  253,1    11959 131475 /opt/mount1/SecureSpan/Gateway/runtime/lib/xss4j.00.patch-1.1.jar

java      11113 gateway  mem    REG  253,1   220536 131468 /opt/mount1/SecureSpan/Gateway/runtime/lib/xml-apis-1.4.01.jar

java      11113 gateway  mem    REG  253,1  1394183 131467 /opt/mount1/SecureSpan/Gateway/runtime/lib/xercesImpl-2.11.0-l7p1.jar

java      11113 gateway  mem    REG  253,1   276420 131463 /opt/mount1/SecureSpan/Gateway/runtime/lib/xalan-serializer-2.7.2.jar

java      11113 gateway  mem    REG  253,1  1748611 131462 /opt/mount1/SecureSpan/Gateway/runtime/lib/xalan-2.7.2-l7p1.jar

java      11113 gateway  mem    REG  253,1   189284 131337 /opt/mount1/SecureSpan/Gateway/runtime/lib/concurrent-1.3.4.jar

java      11113 gateway  mem    REG  253,1    37506 131459 /opt/mount1/SecureSpan/Gateway/runtime/lib/whirlycache-1.0.1.jar

java      11113 gateway  mem    REG  253,1    98700 131436 /opt/mount1/SecureSpan/Gateway/runtime/lib/servlet-api-resources-2.5

Environment

API Gateway: 8.x, 9.x

Resolution

Looking at a lsof output, if you have a connection with a CLOSE_WAIT status, the connection will maintain an open file and will not release it until the socket is closed.

This behavior may be related to one or more of the following:

1. your Java version [needs to be updated]
2. the keep-alive time for the connection on the GW [needs to be increased]
3. the keep-alive connection [needs to be added to the header on your backend services]

NOTE: If you are using the 'Use Keep-Alive' option in your Route via HTTP(S) assertions, set the io.httpDisableKeepAlive Cluster Wide Property to false.

Additional Information

The number of open files for the gateway user can be around 750 (give or take). This is considered normal behavior.