CircuitBreakingException: [parent] Data too large, data for [<transport_request>]

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:

• New notification not reflecting
• Notification sync between M&R and Smarts stops over a period of time.
• Below exception is seen in Smarts tomcat alerts-ingestion.log:   

org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<transport_request>] would be [1028167600/980.5mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1028163656/980.5mb], new bytes reserved: [3944/3.8kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=3944/3.8kb, accounting=104630/102.1kb]

        at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:342) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128) ~[elasticsearch-7.3.1.jar:7.3.1]

Environment

Smarts - 10.1.x

MnR - 7.x

Cause

• Circuit breaker is responsible for avoiding out of memory exception and would lead to this error message.
• The above error means that whenever there are multiple requests or heavy requests being sent to ElasticSearch (ES), the estimated size can shoot up significantly and is greater than limit_size (980.5mb here).
• Below command can be used to verify ES limit_size:

  curl -XGET "http://localhost:9200/_nodes/stats?pretty=true"

    Example:  

parent" : {
          "limit_size_in_bytes" : 4080218931,
          "limit_size" : "3.7gb",
          "estimated_size_in_bytes" : 1508841520,
          "estimated_size" : "1.4gb",
          "overhead" : 1.0,
          "tripped" : 0
        }
      }

Resolution

Follow below steps to increase ElasticSearch (ES) limit_size:

• Increase the heap size of ES in the file SM_HOME/SAM/smarts/toolbox/elasticsearch/config/jvm.options
• Increase and set same value for -Xms and -Xmx parameters in the above file.
• Restart ES service.