Upgrade of Aria automation 8.x fails with connection refused error

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Upgrading vRA 8.x initially completes to show the new version, however, post reboot, unable to establish SSH session to continue syncing upgrade data.
• Upgrade logs report following events:

upgrade.log

Welcome to VMware Aria Automation Appliance 8.17.0
[INFO][2024-08-28 01:53:02][node1_fqdn] Syncing upgrade data with node: node3_fqdn
[INFO][2024-08-28 01:53:03][node1_fqdn] Synced upgrade data with node: node3_fqdn
[INFO][2024-08-28 01:54:01][node1_fqdn] VAMI checking upgrade progress
[INFO][2024-08-28 01:54:01][node1_fqdn] VAMI upgrade completed with error.
[INFO][2024-08-28 01:54:01][node1_fqdn] VAMI upgrade failed.
[INFO][2024-08-28 01:54:01][node1_fqdn] Preparing directory for upgrade data on remote node: node2_fqdn
[INFO][2024-08-28 01:54:01][node1_fqdn] Running remote command: mkdir -p /var/vmware/prelude/upgrade at host: node2_fqdn
[ERROR][2024-08-28 01:54:01][node1_fqdn][Exit Code: 255] Attempt failed to run command: mkdir -p /var/vmware/prelude/upgrade.
Pseudo-terminal will not be allocated because stdin is not a terminal.
Welcome to VMware Aria Automation Appliance 8.17.0
root@node2_fqdn: Permission denied (publickey,password).
[ERROR][2024-08-28 01:54:01][node1_fqdn] Remote command failed: mkdir -p /var/vmware/prelude/upgrade at host: node2_fqdn
[ERROR][2024-08-28 01:54:01][node1_fqdn] Remote command failed: mkdir -p /var/vmware/prelude/upgrade at one or more nodes
[WARNING][2024-08-28 01:54:01][node1_fqdn] Sync of upgrade data to node node2_fqdn cannot be completed at this time.

• Apart from the ssh communication issue in the description, the vami upgrade fails while setting up k8s in postupdate.log

postupdate.log

Waiting for infrastructure services..
+ /opt/scripts/wait-k8s-up -t 600 local-infra
Waiting for local-infra timed out: pod(s) not ready: kube-dns, network-health-monitor
Error setting up Kubernetes.Retrying.
2024-08-29 02:55:08Z Script /etc/bootstrap/postupdate-late.d/03-10-setup-k8s.sh failed, error status 1

Environment

VMware Aria Automation 8.x

Cause

• vRA 7.x appliance hardening rules are applied on a vRA 8.x environment. 
• By default, /etc/modprobe.d/modprobe.conf has the following content in prelude VA.

# install bridge /bin/false

• During the hardening process, STIG rule SV-PHTN-30-000032 is applied and the above command is uncommented. This disables modprobe from installing the bridge module leading to the error in journal:

root@vRA-FQDN [ ~ ]# /usr/lib/systemd/systemd-modules-load
Error running install command '/bin/false' for module bridge: retcode 1
Failed to insert module 'br_netfilter': Invalid argument

Resolution

Apply following action plan to resolve the upgrade issue:

1. Revert the environment to pre-upgrade snapshot.
2. Comment (#)  the line "install bridge /bin/false" in modprobe.conf file located in "/etc/modprobe.d/"
3. Reboot the Aria automation appliance.
4. Re-trigger the upgrade.