Introscope Enterprise Manager crashes with the error: "Too many open files".

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

Note: This is discussed in the Community post: https://communities.ca.com/thread/10 1879332  

 

Introscope Enterprise Manager crashes after running for some time.

The Introscope Enterprise Manager log shows the below error:

[ERROR] [Acceptor] [Manager.Acceptor] IOException has been thrown.
Closing socket channel connect ion java.io.IOException: Too many open files

 

Note: Increasing the ulimit to 131072 did not resolve this issue.

ulimit -a

open files (-n) 131072

 

 

 

 

Environment

APM 10.xWindowsUNIX

Cause

 The Index of the Traces are corrupted.

When setting the Intrsocope Enterprise Manager in DEBUG mode. the log shows :

[ERROR] [MOM Collection Buffer consumer thread.] [Manager] Error querying for correlated traces
[DEBUG] [MOM Collection Buffer consumer thread.] [Manager] Correlation keys: [CorCrossProcessData:D9DE61710B8041F8655F11E7F37EF8AC]
com.wily.introscope.spec.server.QueryException: Could not query: directory '/usr/CA/wilyTEST/LOD01/traces/index' exists and is a directory, but cannot be listed: list() returned null
at com.wily.introscope.server.enterprise.entity.transactiontrace.EventManager.doExecuteEventQuery(EventManager.java:152)
at com.wily.introscope.server.enterprise.entity.transactiontrace.EventManager.executeEventQuery(EventManager.java:112)
at com.wily.introscope.server.enterprise.entity.transactiontrace.ClusteredEventManager.executeEventQuery(ClusteredEventManager.java:142)
at com.ca.apm.transactiontrace.appmap.mapper.LocalTransactionTraceCollector.executeClampedCorrelatedEventsQuery(LocalTransactionTraceCollector.java:134)
at com.ca.apm.transactiontrace.appmap.mapper.LocalTransactionTraceCollector.collectCorrelatedTraces(LocalTransactionTraceCollector.java:83)
at com.ca.apm.transactiontrace.appmap.mapper.arrivalbuffer.MomCollectionBuffer.process(MomCollectionBuffer.java:83)
at com.ca.apm.transactiontrace.appmap.mapper.arrivalbuffer.MomCollectionBuffer.process(MomCollectionBuffer.java:1)
at com.ca.apm.transactiontrace.appmap.mapper.arrivalbuffer.AbstractAsynchronousArrivalBuffer$QueueConsumer.run(AbstractAsynchronousArrivalBuffer.java:124)
at java.lang.Thread.run(Thread.java:745)
[DEBUG] [MOM Collection Buffer consumer thread.] [Manager] Collected 0 traces in 39 | []
[Acceptor] [Manager.Acceptor] IOException has been thrown. Closing socket channel connection
java.io.IOException: Too many open files

Resolution

 

Recreate the ...traces\index directory.

 

For the above scenario:

1. Stop the Inroscope Enterprise Manager

2. Remove the directory "'/usr/CA/wilyTEST/LOD01/traces/index"

3.Restart the Introsocope Enterprise Manager

4. The EM will regenerate the ...traces/index directory.

 

 

Additional Information

 

Enterprise Manager Process- Too many open files