Aria Automation unable to start by Aria Suite Lifecycle with error code LCMVRAVACONFIG590070

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Aria Automation (Version 8.18.x) is experiencing intermittent startup failures, specifically generating error code LCMVRAVACONFIG590070.

Automation cluster will not start from vRLCM

From a powered off state, Automation cluster will not start after performing a power on from vRLCM, error below:

Error Code: LCMVRAVACONFIG590070
Failed to start services on VMware Aria Automation. Refer to the VMware Aria Suite Lifecycle log for additional details.
Failed to start services on VMware Aria Automation vraservername.example.com. For more information, login into the VMware Aria Automation and check /var/log/deploy.log file.

Environment

Aria Automation 8.18.x

Cause

The intermittent startup failure is caused by a timing conflict during the power-on process of Aria Automation. 
When this issue happened, following message can be seen in both deploy.log(Aria Automation) and  vmware_vrlcm.log(Aria Suite Lifecycle):

+ vracli cluster exec -- bash -c /opt/scripts/set_permissions_and_ownership.py
executing bash on command-executor-q7xxz failed: error: Internal error occurred: error sending request: Post "https:<IP Address>?command=run-on-execd&command=--&command=bash&command=-c&command=/opt/scripts/set_permissions_and_ownership.py&error=1&output=1": dial tcp <IP Address>:10250: connect: connection refused

Resolution

This has been resolved in 8.18.1 patch 4 and higher. 

1. Download the latest Cumulative Automation 8.18.1 patch from support.broadcom.com: Download Broadcom products, patches and software
2. Install the patch following the instructions in VMware Aria Automation 8.18.1 Cumulative Update #5

As a workaround, you can manually  manual restart Aria Automation services when this issue happens:

1. SSH into one node of the cluster and run /opt/scripts/deploy.sh