CircuitBreakingException: [parent] Data too large, data for [<transport_request>]

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:

1). Notification sync between M&R and Smarts stops over a period of time.

2). Below exception is seen in Smarts tomcat alerts-ingestion.log:    

org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<transport_request>] would be [1028167600/980.5mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1028163656/980.5mb], new bytes reserved: [3944/3.8kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=3944/3.8kb, accounting=104630/102.1kb]

        at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:342) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128) ~[elasticsearch-7.3.1.jar:7.3.1]

 

Environment

Smarts - v10.1.x

Cause

1). Circuit breaker is responsible for avoiding out of memory exception and would lead to this error message.

2). The above error means that whenever there are multiple requests or heavy requests being sent to ElasticSearch (ES), the estimated size can shoot up significantly and is greater than limit_size (980.5mb here).

3). Below command can be used to verify ES limit_size:

    curl -XGET "http://localhost:9200/_nodes/stats?pretty=true"

    Example:  

parent" : {
          "limit_size_in_bytes" : 4080218931,
          "limit_size" : "3.7gb",
          "estimated_size_in_bytes" : 1508841520,
          "estimated_size" : "1.4gb",
          "overhead" : 1.0,
          "tripped" : 0
        }
      }

Resolution

Follow below steps to increase ElasticSearch (ES) limit_size:

1). Increase the heap size of ES in the file SM_HOME/SAM/smarts/toolbox/elasticsearch/config/jvm.options

2). Increase and set same value for -Xms and -Xmx parameters in the above file.

3). Restart ES service.