Data Collectors not showing connected after upgrade to NetOps 20.2.8

book

Article ID: 210053

calendar_today

Updated On:

Products

CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

After upgrade to 20.2.8 the data collectors are showing as 'not connected' in the CAPC UI even thought they are running, and the version is wrong as well

 

Ran kill -3 on the dcmd process PID

This showed a deadlock with the APM agent associated with the data collector

/opt/IMDataCollector/apache-karaf-2.4.3/data/Karaf.out had the following java deadlock listed


"main" #1 prio=5 os_prio=0 tid=0x00007f3db800e000 nid=0x440f waiting for monitor entry [0x00007f3dbf64c000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at com.wily.util.properties.hot.ConfigurationManager.add(ConfigurationManager.java:101)
    - waiting to lock <0x00000001c04596f0> (a com.wily.util.properties.hot.ConfigurationManager)

Found one Java-level deadlock:
=============================
"Agent Execution":
  waiting to lock monitor 0x00007f3d400ad318 (object 0x00000001cba8f228, a java.lang.Class),
  which is held by "main"
"main":
  waiting to lock monitor 0x00007f3d6c004698 (object 0x00000001c04596f0, a com.wily.util.properties.hot.ConfigurationManager),
  which is held by "Agent Execution"

Cause

The APM agent for the dadaemon process created a deadlock, not allowing the Data Aggregator to start

Environment

Dx NetOps Performance Management

 

 

Resolution

ediit IMDataCollector/custom.d/_capm, and comment out the export line

# export WILY_DIR=

 

Then restart the data collector

Additional Information

Steps to resolve APM deadlock ont he DA: https://knowledge.broadcom.com/external/article?articleId=210046