Deployments or Day 2 operations are stuck or hung in-progress against a vRealize Automation 8.x managed resource

Products

• VMware Aria Suite

Issue/Introduction

This issue occurs in vRealize Automation 8.1 P3 and 8.2 GA

Symptoms:

• Deployments or day 2 operations hang
• In the event broker service logs you see messages similar to:

2022-10-30T18:54:58.636Z [priority='ERROR' thread='parallel-xx' user='tango-blueprint-xxxxlULFRKcuxxxx' org='<Long_Org_ID>' context='<UUID_1>' parent='<UUID_2>' token='<UUID_3>'] c.v.a.e.b.s.s.impl.EventServiceImpl.lambda$publish$5:147 - Error handling Event[id=<UUID_4, eventTopicId=deployment.request.post, orgId=<Long_Org_ID>, projectId=<Project_ID>, userName=pa-tenantadmin, timeStamp=Fri Oct 30 18:53:27 GMT 2022, sourceType=Deployment, sourceIdentity=Deployment, targetType=null, targetId=null, correlationType=Deployment, correlationId=<UUID_5>, headers={uberctx-parent-id=<UUID_6>, uberctx-org-id=<Long_Org_ID>, uber-trace-id=<UUID_7>:<UUID_8>:<UUID_9>:1, uberctx-user-id=tango-blueprint-xxxxlULFRKcuxxxx, sampling-decision=true}]

com.vmware.automation.event.broker.server.service.exception.EventAlreadyPublishedException: null

• If the bundle was collected soon after the issue appears you will also see a message similar to:

2022-11-05T12:32:28.893Z [priority='ERROR' thread='reactor-http-epoll-16' user='' org='' context='' parent='' token=''] reactor.netty.http.server.HttpServer.error:319 - [id: 0x15fcb7d9, L:/<IP_Address_1>:4242 - R:/<IP_Address_2>:47804]

 java.lang.StackOverflowError: null
 

Environment

VMware vRealize Automation 8.x

Cause

Java stack size limitations generate null exceptions during request.

Resolution

This issue is resolved in vRealize Automation 8.2 P1 and 8.3 GA.

Workaround:
To workaround this issue, increase the Java stack size on each appliance virtual machine.

To increase the Java Stack size on each vRealize Automation appliance node, execute Steps #1-4 against each node in the cluster.

1. Open an SSH connection to each vRealize Automation appliance.
2. cd /opt/charts/ebs/templates
3. cp deployment.yaml /tmp/
4. vi deployment.yaml
   1. Within this file, locate the line with "- name: JAVA_OPTS"
   2. Just above this line insert the following 2 lines with the same indentation as JAVA_OPTS and save the file:

- name: EXT_JAVA_OPTS
value: -Xss1M -XX:MaxJavaStackTraceDepth=50000

5. Start the vRealize Automation 8.x application: /opt/scripts/deploy.sh

Note:  Run Step #5 against only one node in the cluster.