EM unable to process Defects, seeing "http://#.#.#.#/wily/cem/tim/mod_python/timfiles/listDefects: Read timed out." error message
search cancel

EM unable to process Defects, seeing "http://#.#.#.#/wily/cem/tim/mod_python/timfiles/listDefects: Read timed out." error message

book

Article ID: 30192

calendar_today

Updated On:

Products

CA Application Performance Management Agent (APM / Wily / Introscope) INTROSCOPE

Issue/Introduction

Symptoms:

From CEM Console > System > Events , we can see the below error message:

Enterprise Manager cannot reach TIM: Error retrieving response
http://#.#.#.#/wily/cem/tim/mod_python/timfiles/listDefects: Read timed out."

 

 

Environment

APM 10.x

Cause

The TIM Collection Service (TCS) stops polling defects from the TIM\MTP, because of the huge amount of data needed to be processed (/opt/CA/APM/tim/data/out/defects)

 

Resolution

Manually delete all the files available under the <MTP\TIM box> data/out/defects folder:

1. Disable TIM(s) from CEM UI

2. Stop the Introscope EM collector running TCS (TIM Collector Service)

3. Manually delete all the files (defects-* and login-*) from the MTP\TIM box, depending on the TIM version it can be:

/etc/wily/cem/tim/data/out/defects
/opt/CA/APM/tim/data/out/defects

Go to each of the /worker<xx> folder and delete all files: rm -rf *

4. Restart the Introscope EM collector

5. Re-enable the TIM Monitor(s)

6. Check EM collector running TCS, you should be able to confirm that defects are being processed, for example:

[INFO] [TimPollThreadPool.Thread1] [Manager.com.timestock.tess.services.tim.TimIo] File 'apmr5b3-defect-14014641140000598108.xml' read, length=3810

 

NOTE: You can enable DEBUG logging on CEM service to troubleshoot further issues related to defect processing

Open {EMHOME}/config/IntroscopeEnterpriseManager.properties and set the following property value:

              log4j.logger.Manager.com.timestock=DEBUG,console,logfile

Here is an example of the output:

[DEBUG] [DefectLoginProcessingPool.Thread4] [Manager.com.timestock.tess.services.processors.DefectProcessor] Defect processing starting (1 defects, 1 details and 24 meta values) from monitor apmr5b3
[DEBUG] [DefectLoginProcessingPool.Thread4] [Manager.com.timestock.tess.services.processors.DefectProcessor] Defect processing took 0.379 seconds (defects: 1, biz events: 1, users: -1)

 

IMPORTANT:

If the defects data is important, then you can delete old defects based on time stamp from all the workers directories (at least match the value from “"Delete Defects after:" setting in “Incident Settings” page). This helps retain at least last X days defects data. 

Additional Information

 

See also: Unable to Enable TIM Monitor after upgrading TIM to 10.7 SP3 , seeing Communication Error 404
https://knowledge.broadcom.com/external/article?articleId=141458