Data aggregator dadaemon service going down on a regular basis

Products

• DX NetOps

Issue/Introduction

Data aggregator goes down occasionally

dadaemon and activemq don't appear to be using too much memory

dadaemon heap usage is consistently below 60%

 

 

 

Environment

Release : 21.2, 22.2

Cause

/var/log/messages shows the following, indicating the system is running out of memory

Sep 24 11:07:41 <DA> kernel: Out of memory: Kill process 7167 (java) score 789 or sacrifice child
Sep 24 11:07:41 <DA> kernel: Killed process 7167 (java), UID 0, total-vm:33536016kB, anon-rss:29981084kB, file-rss:0kB, shmem-rss:0kB
Sep 24 11:07:43 <DA> systemd: dadaemon.service: main process exited, code=exited, status=137/n/a
Sep 24 11:07:43 <DA> dadaemon: Stopping IM Data Aggregator.
Sep 24 11:07:43 <DA> dadaemon: CEF:0|Broadcom|DX NetOps Data Aggregator|21.2.9.361|101|STOPPING SERVICE|1|suser=root shost=HOSTNAME
Sep 24 11:07:44 <DA> dadaemon: Can't connect to the container. The container is not running.
Sep 24 11:07:44 <DA> dadaemon: Error stopping the Data Aggregator, error code=1
Sep 24 11:07:44 <DA> dadaemon: CEF:0|Broadcom|DX NetOps Data Aggregator|21.2.9.361|201|FAILED TO STOP SERVICE|8|suser=root shost=HOSTNAME
Sep 24 11:07:44 <DA> systemd: dadaemon.service: control process exited, code=exited status=1
Sep 24 11:07:44 <DA> systemd: Unit dadaemon.service entered failed state.
Sep 24 11:07:44 <DA> systemd: dadaemon.service failed.

Resolution

Add more memory to the system