Data aggregator not fully up after upgrade to NetOps 20.2.8

book

Article ID: 210046

calendar_today

Updated On:

Products

CA Performance Management - Usage and Administration

Issue/Introduction

After upgrade from 20.2.4 to 20.2.8 data aggregator is not coming up. Also unable to access the endpoint for the data aggregator using the web browser.

***

after restarting the dadaemon, ran kill -3 on the karaf process PID

This showed a deadlock with the APM agent associated with dadaemon

/opt/IMDataAggregator/apache-karaf-2.4.3/data/Karaf.out had the following java deadlock listed


"main" #1 prio=5 os_prio=0 tid=0x00007f3db800e000 nid=0x440f waiting for monitor entry [0x00007f3dbf64c000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at com.wily.util.properties.hot.ConfigurationManager.add(ConfigurationManager.java:101)
    - waiting to lock <0x00000001c04596f0> (a com.wily.util.properties.hot.ConfigurationManager)

Found one Java-level deadlock:
=============================
"Agent Execution":
  waiting to lock monitor 0x00007f3d400ad318 (object 0x00000001cba8f228, a java.lang.Class),
  which is held by "main"
"main":
  waiting to lock monitor 0x00007f3d6c004698 (object 0x00000001c04596f0, a com.wily.util.properties.hot.ConfigurationManager),
  which is held by "Agent Execution"

 

 

Cause

the APM agent for the dadaemon process created a deadlock, not allowing the Data Aggregator to start

Environment

Release : 20.2.8

Component : IM Data Aggregator

Resolution

comment out the following line in the following line in the /opt/IMDataAggregator/custom.d/_capm file

# export WILY_DIR=

Additional Information

Steps to resolve the deadlock Issue on the DC(s): https://knowledge.broadcom.com/external/article?articleId=210053