Service stops responding and throws "Too many open files" exceptions in the log files

Products

• CA Application Test
• CA Continuous Application Insight (PathFinder)
• Service Virtualization

Issue/Introduction

DevTest service stops responding at some point. The exception "Too Many Open Files" is thrown in the log files.

Environment

All supported DevTest releases and platforms

Cause

The too many open files exception indicates that DevTest has reached the maximum open files limit set in the Linux system.

For most distributions, the default value for open files limit is only 1024. For DevTest servers this is too low.

To verify what is the open files limit set on the system you can execute:

# ulimit -n

Resolution

Contact your system administrator to increase the open files limit on the DevTest server facing the problem.

Consider increasing this limit to 10240 or higher.

While the recommendation is to increase the open files limit to 10240 or higher, it is difficult to specify a value you will need to apply to the environment.  It will depend on how you are using DevTest virtual services and test cases.

It is also possible that a customized extension or test step is consuming resources and not releasing it, causing your system to hit the open files limit.

One command line that can help you visualizing the list of open files by process is the lsof Linux command.

Using the –p option you can specify a process ID (PID). The results will be the listing of open files for the PID specified.

From this result you can have an idea of how much a process is consuming.

For more information regarding lsof command: https://www.unix.com/man-page/linux/8/lsof/

Additional Information

For more details regarding DevTest environmental restrictions, please take a look this KB: DevTest Environment Restrictions