The Opensearch service keeps on crashing due to ' java.lang.OutOfMemoryError: Java heap space ' - VMware Identity Manager
search cancel

The Opensearch service keeps on crashing due to ' java.lang.OutOfMemoryError: Java heap space ' - VMware Identity Manager

book

Article ID: 449190

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

The opensearch log - ' horizon.log"  located at ' /opt/vmware/opensearch/logs/ ' has the following snippets  of ' java.lang.OutOfMemoryError: Java heap space ' .  

[xxxx-xx-xx:xx:xx,xxx][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [xxxxxxx.xx.xxxxxx.xxxxxxx.com] fatal error in thread [opensearch[xxxxxxx.xx.xxxxxx.xxxxxxx.com][clusterManagerService#updateTask][T#1]], exiting

java.lang.OutOfMemoryError: Java heap space

[xxxx-xx-xxT00:xx:xx,xxx][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [xxxxxxx.xx.xxxxxx.xxxxxxx.com] fatal error in thread [opensearch[xxxxxxx.xx.xxxxxx.xxxxxxx.com][refresh][T#4]], exiting

java.lang.OutOfMemoryError: Java heap space

[xxxx-xx-xxT00:xx:xx,xxx][WARN ][o.o.t.InboundHandler     ] [xxxxxxx.xx.xxxxxx.xxxxxxx.com] handling inbound transport message [InboundMessage{Header{332}{2.8.0}{580705902}{true}{false}{false}{false}{indices:admin/seq_no/retention_lease_background_sync[r]}}] took [12647ms] which is above the warn threshold of [5000ms]
[xxxx-xx-xxT00:xx:xx,xxx][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [xxxxxxx.xx.xxxxxx.xxxxxxx.com] attempting to trigger G1GC due to high heap usage [1067114624]
[xxxx-xx-xxT00:xx:xx,xxx][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [xxxxxxx.xx.xxxxxx.xxxxxxx.com] fatal error in thread [opensearch[xxxxxxx.xx.xxxxxx.xxxxxxx.com][refresh][T#1]], exiting

java.lang.OutOfMemoryError: Java heap space

[xxxx-xx-xxT00:xx:xx,xxx][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [xxxxxxx.xx.xxxxxx.xxxxxxx.com] GC did not bring memory usage down, before [1067114624], after [1067182608], allocations [1], duration [7]
[xxxx-xx-xxT00:xx:xx,xxx][WARN ][o.o.t.InboundHandler     ] [xxxxxxx.xx.xxxxxx.xxxxxxx.com] handling inbound transport message [InboundMessage{Header{333}{2.8.0}{580705767}{true}{false}{false}{false}{indices:admin/seq_no/retention_lease_background_sync[r]}}] took [18148ms] which is above the warn threshold of [5000ms]
[xxxx-xx-xxT00:xx:xx,xxx][WARN ][o.o.t.InboundHandler     ] [xxxxxxx.xx.xxxxxx.xxxxxxx.com] handling inbound transport message [InboundMessage{Header{325}{2.8.0}{809046567}{true}{false}{false}{false}{indices:admin/seq_no/retention_lease_background_sync[r]}}] took [26990ms] which is above the warn threshold of [5000ms]
[xxxx-xx-xxT00:xx:xx,xxx][WARN ][o.o.t.InboundHandler     ] [xxxxxxx.xx.xxxxxx.xxxxxxx.com] handling inbound transport message [InboundMessage{Header{336}{2.8.0}{580705531}{true}{false}{false}{false}{indices:admin/seq_no/retention_lease_background_sync[r]}}] took [7411ms] which is above the warn threshold of [5000ms]
[xxxx-xx-xxT00:xx:xx,xxx][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [xxxxxxx.xx.xxxxxx.xxxxxxx.com] attempting to trigger G1GC due to high heap usage [1070560280]
[xxxx-xx-xxT00:xx:xx,xxx][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [xxxxxxx.xx.xxxxxx.xxxxxxx.com] GC did not bring memory usage down, before [1070560280], after [1070684888], allocations [0], duration [771]
[xxxx-xx-xxT00:xx:xx,xxx][WARN ][o.o.t.InboundHandler     ] [xxxxxxx.xx.xxxxxx.xxxxxxx.com] handling inbound transport message [InboundMessage{Header{334}{2.8.0}{809046961}{true}{false}{false}{false}{indices:admin/seq_no/retention_lease_background_sync[r]}}] took [6376ms] which is above the warn threshold of [5000ms]

[xxxx-xx-xxT00:54:01,599][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [xxxxxxx.xx.xxxxxx.xxxxxxx.com] fatal error in thread [Thread-1554], exiting

Environment

VMware Identity Manager 3.3.7 

Cause

The provided log snippet indicates a fatal OutOfMemoryError: Java heap space exception within the OpenSearch daemon on node xxxxxxx.xx.xxxxxx.xxxxxxx.com. The G1 Garbage Collector is attempting to reclaim memory but failing (GC did not bring memory usage down), confirming that the daemon has hit its absolute configured JVM heap limit (approximately 1GB based on the byte counts in the log).

Because OpenSearch is fatally crashing, it cannot process incoming audit and analytics events.

Resolution

To stabilize the analytics layer, increase the OpenSearch JVM heap allocation and validate the upstream message queues. Execute the following steps on the affected node xxxxxxx.xx.xxxxxx.xxxxxxx.com.

1. Validate Upstream Queue Pressure

Determine how many messages are currently trapped in the RabbitMQ broker.

  • Execute: rabbitmqctl list_queues | grep analytics

  • Threshold: If the queue size is significantly greater than 100, messages are actively failing to deliver to OpenSearch.

2. Increase OpenSearch JVM Heap

You must allocate additional heap space to the OpenSearch daemon. The exact path may vary slightly depending on the specific 3.3.x patch level, but it is standardly located in the OpenSearch configuration directory.

  1. SSH into the affected node as root.

  2. Locate the JVM options file (typically /opt/vmware/opensearch/config/jvm.options or /usr/local/horizon/conf/).

  3. Modify the initial and maximum heap size parameters to a higher value (e.g., 2GB or 4GB, depending on your appliance's total RAM). Do not exceed 50% of the appliance's total physical memory.

    • Change -Xms1g to -Xms2g

    • Change -Xmx1g to -Xmx2g

  4. Save the file.

3. Reinitialize the Daemon : 

Force the daemon to restart and claim the newly allocated heap space.

  • Execute: /etc/init.d/opensearch restart

  • Validation: Run ' curl http://localhost:9200/_cluster/health?pretty ' to confirm the cluster status returns to green.

 

Powered by Wolken Software