DXC logstash process keeps terminating

book

Article ID: 140531

calendar_today

Updated On:

Products

CA Application Performance Management Agent (APM / Wily / Introscope) CA Application Performance Management (APM / Wily / Introscope) INTROSCOPE

Issue/Introduction

DXC logstash process dies, have to restart manually and lose critical site visibility while this occurs. Need to understand why the process terminates on its own.

Cause

The Logstash on the DXC is processing 80,000 metrics. It is generally rated for 50,000.

Environment

Release : 10.7.0

Component : APM Agents

Resolution

How to reduce metrics

1.       Reduce the number BA reporting

2.       Modify the BA profile to reduce urls monitored

3.       Modify the default DXC profile to reduce url monitored.

Here is how to Increase capacity

1.       Add more Logstashes

2.       Spin up another DXC
'

Here are some possible configuration settings and other changes
When it says "DXC logstash keeps terminating", it is the DxC APM Logstash Plugin process. You can set environment variable LS_HEAP_SIZE to specify max heap size for DxC logstash process. By default, it is 1G. You can set it in AxC/logstash/logstash-*/bin/logstash, or anywhere else appropriate for your environment.

- Please make sure there are enough memory on the box to support the increased heap size.

- And do verify the java command line for the logstash process afterwards to make sure the -Xmx option has been updated with increased value.

 

Additional Information

Log Analysis: In one instance that the clamp value had reached and there was an exception after that.

It is possible that the inflow of data to logstash was too much to handle which eventually led to its crash.

Increasing the logstash instances would be a wise thing to do at this point