After upgrade from 20.2.4 to 20.2.8 data aggregator is not coming up. Also unable to access the endpoint for the data aggregator using the web browser.
after restarting the dadaemon, ran kill -3 on the karaf process PID
This showed a deadlock with the APM agent associated with dadaemon
/opt/IMDataAggregator/apache-karaf-2.4.3/data/Karaf.out had the following java deadlock listed
"main" #1 prio=5 os_prio=0 tid=0x00007f3db800e000 nid=0x440f waiting for monitor entry [0x00007f3dbf64c000]
java.lang.Thread.State: BLOCKED (on object monitor)
- waiting to lock <0x00000001c04596f0> (a com.wily.util.properties.hot.ConfigurationManager)
Found one Java-level deadlock:
waiting to lock monitor 0x00007f3d400ad318 (object 0x00000001cba8f228, a java.lang.Class),
which is held by "main"
waiting to lock monitor 0x00007f3d6c004698 (object 0x00000001c04596f0, a com.wily.util.properties.hot.ConfigurationManager),
which is held by "Agent Execution"
the APM agent for the dadaemon process created a deadlock, not allowing the Data Aggregator to start
Release : 20.2.8
Component : IM Data Aggregator
comment out the following line in the following line in the /opt/IMDataAggregator/custom.d/_capm file
# export WILY_DIR=
Steps to resolve the deadlock Issue on the DC(s): https://knowledge.broadcom.com/external/article?articleId=210053