AIOps - Unable to create New dashboard, error : [in flight requests] Data too large, data for [<http_request>] would be ...], which is larger than the limit of [8537636864/7.9gb]

Products

• DX Operational Intelligence

Issue/Introduction

Symptoms:

- When trying to create dashboards in Grafana using metadata datasource. It is returning the below error:

Elasticsearch error: [in flight requests] Data too large, data for [<http_request>] would be [12015005508/11.1gb], which is larger than the limit of [8537636864/7.9gb]

Environment

DX Operational Intelligence 2x
DX Application Performance Management 2x
DX AXA 2x

Cause

The message is due to the default 16GB RAM limits of your jarvis-elasticsearch pod and the required memory for the query. By design, the RAM_RATIO parameter for elasticsearch cannot be more than 0.5, so max allowed memory is 8GB as reported in the error message.

 

Resolution

Increase the memory of jarvis-elasticsearch deployments as described below.

 

IMPORTANT NOTE:  Before proceeding with the below steps, make sure that the physical host on which your jarvis-elasticsearch pod has sufficient additional physical memory available.

 

1) kubectl get pods -n<your-namespace> | grep elastic

2) kubectl describe pod jarvis-elasticsearch-<pod_id> -n<your-namespace>

3) kubectl edit deployment -n<your-namespace> jarvis-elasticsearch

Scroll down and change default memory values from 16G to 32G and save the file.

4) The pods from this deployment will be recreated automatically. You can check it by running again the command below:

kubectl get pods -n<your-namespace> | grep elastic

5) The jarvis-elasticsearch pods should be running as expected with the memory limit increased. Please run below to verify.

kubectl describe pod jarvis-elasticsearch-<pod_id> -n<your-namespace>

 

NOTE: You can enter the pod terminal using the following command if needed:

kubectl exec jarvis-elasticsearch-<pod_id> -it -n<your-namespace> -- bash

 

If the problem persists, collect Jarvis logs  (elastic, zookeeper, kafka, apis logs,  see below KB) and contact Broadcom Support for assistance:
 
DX AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting

 

You can also use the following queries to display more information about your current elasticsearch nodes in the cluster and their statistics:

Displays nodes in cluster (check memory, CPU, load): http(s)://<ELASTIC_URL>/_cat/nodes?v

 

Node stats: http(s)://<ELASTIC_URL>/_nodes/stats/indices?pretty

Additional Information

DX AIOPs - Troubleshooting, Common Issues and Best Practices

AIOPs - Troubleshooting, Common Issues and Best Practices