Elasticsearch error: [in flight requests] Data too large, data for [<http_request>] would be [12015005508/11.1gb], which is larger than the limit of [8537636864/7.9gb]
DX Operational Intelligence 2x
DX Application Performance Management 2x
DX AXA 2x
The message is due to the default 16GB RAM limits of your jarvis-elasticsearch pod and the required memory for the query. By design, the RAM_RATIO parameter for elasticsearch cannot be more than 0.5, so max allowed memory is 8GB as reported in the error message.
Increase the memory of jarvis-elasticsearch deployments as described below.
IMPORTANT NOTE: Before proceeding with the below steps, make sure that the physical host on which your jarvis-elasticsearch pod has sufficient additional physical memory available.
1) kubectl get pods -n<your-namespace> | grep elastic
2) kubectl describe pod jarvis-elasticsearch-<pod_id> -n<your-namespace>
3) kubectl edit deployment -n<your-namespace> jarvis-elasticsearch
Scroll down and change default memory values from 16G to 32G and save the file.
4) The pods from this deployment will be recreated automatically. You can check it by running again the command below:
kubectl get pods -n<your-namespace> | grep elastic
5) The jarvis-elasticsearch pods should be running as expected with the memory limit increased. Please run below to verify.
kubectl describe pod jarvis-elasticsearch-<pod_id> -n<your-namespace>
NOTE: You can enter the pod terminal using the following command if needed:
kubectl exec jarvis-elasticsearch-<pod_id> -it -n<your-namespace> -- bash
If the problem persists, collect Jarvis logs (elastic, zookeeper, kafka, apis logs, see below KB) and contact Broadcom Support for assistance:
DX AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
You can also use the following queries to display more information about your current elasticsearch nodes in the cluster and their statistics:
Displays nodes in cluster (check memory, CPU, load): http(s)://<ELASTIC_URL>/_cat/nodes?v
Node stats: http(s)://<ELASTIC_URL>/_nodes/stats/indices?pretty