"OutOfMemoryError: GC overhead limit exceeded" error on offline Replicators in vCloud Availability 3.0

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Creating new protections or migrations fail in vCloud Availability.
• Performing a test failover, failover, or migration for active replications fail.
• In the System Monitoring view of the vCloud Availability Portal, the Replicator show as offline.
• When navigating to the Replicator Management Portal, it is not available.
• In /opt/vmware/h4/replicator/log/replicator.log on the Replicator, you see entries similar to:

2019-09-26 08:19:57.826  WARN - [dd3aba8a-434a-4948-8b98-a5fd18d9a9f4_PU] [https-jsse-nio-8440-exec-10] c.v.h.c.e.ExceptionConversionService     : Unable to convert exception. Using fallback exception instead. org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCloud Availability 3.0.x

Cause

This issue occurs when the Replicator service spends mosts of its time performing garbage collection causing the service to become unresponsive.

Resolution

This is a know issue affecting vCloud Availability 3.0.
This issue is resolved in vCloud Availability 3.0.4, available at VMware Downloads.

Workaround:
To work around this issue, you have to restart the Replicator service.

1. SSH to the Replicator appliance and login as root.
2. Check the status of the Replicator service:

systemctl status replicator.service

3. If you see GC overhead limit messages in the output, restart the Replicator service:

systemctl restart replicator.service

4. Check the status of the Replicator service again to ensure it has been restarted successfully.
5. Navigate to the Replicator Management Portal to verify it is available again via the URL:

https://replicatorIp:8043/ui/admin

6. If the Management Portal is still not available, restart the Replicator appliance.