Failed to start services on VMware Aria Automation <Aria Automation FQDN>. For more information, please check the /var/log/deploy.log file on Aria Automation

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

The Power On task for Aria Automation on Aria Suite Lifecycle Manager is failing with the following error:

Error Code: LCMVRAVACONFIG590070
Error Message: "Failed to start services on VMware Aria Automation."
For more details, refer to the VMware Aria Suite Lifecycle log.
"Failed to start services on VMware Aria Automation <vra.fqdn> For more information, please check the /var/log/deploy.log file on Aria Automation."

However, all other Day 2 tasks for Aria Automation, such as Inventory Sync and Power Off, are functioning as expected.

Environment

Aria Suite Lifecycle 8.x

Aria Automation 8.x

Cause

Upon investigation, we found the following error in the deploy.log file on Aria Automation:

+ vracli cluster exec -- bash -c /opt/scripts/set_permissions_and_ownership.py executing bash on command-executor-d8j78 failed: Error from server: error dialing backend: dial tcp  ##.##.##.##:10250: connect: connection refused + on_exit + '[' 1 -ne 0 ']' + echo 'Deployment failed. Collecting log bundle ...' Deployment failed. Collecting log bundle ...

We further observed that enabling FIPS on Aria Suite Lifecycle allowed the task to complete successfully. This led us to investigate the security settings related to weak SHA1 algorithms and ciphers on both Aria Suite Lifecycle and Aria Automation.

Upon reviewing the /etc/ssh/sshd_config file on Aria Suite Lifecycle, we found the presence of the following entry: [email protected].

This weak cipher was causing the failure, as Aria Automation has these weak SHA1 algorithms and ciphers disabled. This was further confirmed by checking the /etc/ssh/sshd_config_effective file on Aria Automation.

Resolution

The issue was traced to the weak SHA1 algorithm and cipher configuration in the SSH settings.

To resolve this:

1. Validate SSH Configuration

   • Establish an SSH session to both the Aria Suite Lifecycle Manager and the Aria Automation nodes.

   • Check the /etc/ssh/sshd_config file to verify whether any weak SHA1 algorithms are present.

2. Remove Weak Algorithms

   • If found, remove the entry [email protected] (or any other SHA1-based algorithms) from the sshd_config file on both Aria Suite Lifecycle Manager and Aria Automation nodes.

   • Update the SSH configuration as per the guidance in the KB article below: https://knowledge.broadcom.com/external/article/326133/steps-for-removing-weak-sha1-algorithms.html

3. Restart SSH Service

   • Apply the updated configuration by restarting the SSH service using on Aria Suite Lifecycle Manager or Aria Automation nodes.

      

     systemctl restart sshd

4. Re-run the Power On Task

   • Retry the Power On task again from Aria Suite Lifecycle Manager.

After updating the file, the task completed successfully for Aria Automation, and the request also completed successfully within Aria Suite Lifecycle Manager.

Following the removal of weak SHA1 algorithms and updating the SSH configuration, the issue was fully resolved, and the power-on operation completed without errors.

Additional Information

Refer Steps for removing weak SHA1 algorithms and ciphers from VMware Aria Products
https://knowledge.broadcom.com/external/article/326133/steps-for-removing-weak-sha1-algorithms.html