java.io.IOException: Too many open files

Products

• CA Infrastructure Management
• CA Performance Management - Usage and Administration

Issue/Introduction

ISSUE:
Our team found that a lock file was accessed 65,000 times.

[root@<hostName> ~]# fgrep snmp_profile_rank /tmp/lsof.out |wc -l
65406

java 65503 root *497u REG 253,0 0 1835480 /tmp/snmp_profile_rank.lck
java 65503 root *498u REG 253,0 0 1835480 /tmp/snmp_profile_rank.lck
java 65503 root *499u REG 253,0 0 1835480 /tmp/snmp_profile_rank.lck
java 65503 root *500u REG 253,0 0 1835480 /tmp/snmp_profile_rank.lck
java 65503 root *501u REG 253,0 0 1835480 /tmp/snmp_profile_rank.lck

ENVIRONMENT:
CAPM 2.8 Redhat Linux

SYMPTOMS:
We are seeing errors in the logs for Device Manager (DMService.log) for too many open files:
java.nio.file.FileSystemException: /opt/CA/PerformanceCenter/jetty/lib/servlet-api-3.1.jar: Too many open files
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:1

CAPC web server is not responding

CAUSE:
The /temp/snmp_profile_rank.lck is used by CAPC REST in creating/updating SNMP profiles (used for lock purpose).
This can be caused by doing a lot of SNMP REST calls on CAPC, like automating SNMP Profile management.
The “file lock” mechanism we used in creating/updating snmp profiles via REST was not correct. It never closed the lock file.
That is why we got so many file opens on snmp_profile_rank.lck.
Basically, the open count increases by 1 for each SNMP REST calls on CAPC.

RESOLUTION:
Fixed in CAPM 3.0

 

 

 

 

Environment

Release: IMDAGG99000-2.8-Infrastructure Management-Data Aggregator
Component:

Resolution

...