Aria Automation upgrade fails with SSH channels configuration failed error

Products

• VMware Aria Suite

Issue/Introduction

• The upgrade fails with an SSH channel configuration error. The following message is seen in the /var/log/vrlcm/vmware_vrlcm.log logfile:
  
  2024-09-18T12:08:26.610Z INFO vrlcm[] [pool-3-thread-35] [c.v.v.l.u.SshUtils]  -- Command execution response: {
  "exitStatus" : 0,
  "outputData" : "Upgrade is in progress\n... Preparing for upgrade .....................................\n...............................................................\nRunning health check before upgrade for no
des and pods.\nHealth check before upgrade for nodes and pods passed YXYXYXYX SSH channels.\nSSH channels configuration failed.\n... Upgrade preparation failed. The system is unaffected. ...\n",
  "errorData" : null,
  "commandTimedOut" : false
}
2024-09-18T12:08:26.610Z INFO vrlcm[] [pool-3-thread-35] [c.v.v.l.d.v.h.VraPreludeInstallHelper]  -- Command Status code :: 0
2024-09-18T12:08:26.610Z INFO vrlcm[] [pool-3-thread-35] [c.v.v.l.d.v.h.VraPreludeInstallHelper]  -- ====================================================
2024-09-18T12:08:26.610Z INFO vrlcm[] [pool-3-thread-35] [c.v.v.l.d.v.h.VraPreludeInstallHelper]  -- Output Stream ::
2024-09-18T12:08:26.610Z INFO vrlcm[] [pool-3-thread-35] [c.v.v.l.d.v.h.VraPreludeInstallHelper]  -- ====================================================
2024-09-18T12:08:26.610Z INFO vrlcm[] [pool-3-thread-35] [c.v.v.l.d.v.h.VraPreludeInstallHelper]  -- Upgrade is in progress
... Preparing for upgrade .....................................
...............................................................
Running health check before upgrade for nodes and pods.
Health check before upgrade for nodes and pods passed YXYXYXYX
...............................................................
Configuring SSH channels.
SSH channels configuration failed.
... Upgrade preparation failed. The system is unaffected. ...
  
  
• The Aria Automation services were recently restarted before the upgrade attempt or the podnet-up health check is failing when running the health script:
  
  /opt/health/run.sh

Environment

Aria Automation 8.x

Cause

The upgrade is initiated before podnet-up check was succeeding. This can sometimes take a number of minutes after restarting services. Its progress can be monitored in the journal logs:

Check failing:

Sep 18 11:18:01 FQDN CROND[47280]: (root) CMDOUT (Running check podnet-up)
Sep 18 11:18:01 FQDN  CROND[47280]: (root) CMDOUT (make: *** [/opt/health/Makefile:85: podnet-up] Error 2)

Check succeeding:

Sep 18 11:19:01 FQDN  CROND[54453]: (root) CMDOUT ()
Sep 18 11:19:01 FQDN  CROND[54453]: (root) CMDOUT (Running check podnet-up)
Sep 18 11:19:01 FQDN  CROND[54453]: (root) CMDOUT ()

Resolution

1. To resolve the issue ensure that the podnet-up health check is succeeding prior to beginning the upgrade.

2. The health script can be manually run using command
/opt/health/run.sh

3. The expected output is :