Aria Automation Upgrade fails with error LCMVRAVACONFIG90030 due to bridge module restriction

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Upgrading vRA 8.x initially completes to show the new version, however, post reboot, unable to establish SSH session to continue syncing upgrade data.

• When attempting to upgrade VMware Aria Automation, the process fails during Stage-2. In VMware Aria Suite Lifecycle Manager, you observe the following error:
  
  Error Code: LCMVRAVACONFIG90030
VMware Aria Automation VA Upgrade Status Check failed.
Upgrade on vRealize Automation VA <hostname> failed with state "fatal". To know more about the failure, run command 'vracli upgrade status -- details' on the vRealize Automation VA <hostname>. To revert to snapshot, click RETRY with 'revertSnapshot' property set to true

• Running vracli upgrade status --details on the VMware Aria Automation appliance shows that the cluster rebuild failed:
  

  root@<VRA_FQDN> [ ~ ]# vracli upgrade status --details

  Upgrade is in progress
Configuring upgrade.
Upgrade is in progress. This might take more than an hour to complete and the system might be rebooted several times.
Upgrade is complete. Continuing with post upgrade operations.
...............................................................
Analysing the result of the node that executes post upgrade operations.
The node running post upgrade operations is upgraded successfully.
...............................................................
Rebuilding cluster of appliance nodes.
Rebuilding cluster of appliance nodes failed.
... Upgrade terminated due to critical error. Follow the upgrade guide to recover the system. ...

• The /var/log/vmware/prelude/upgrade<date>.log report following events:

[INFO] ##:##:##.###### ##### warnings.go:###] "Warning: unknown field \"spec.topologyKeys\""
[ERROR][####-##-## ##:##:##][VRA_Node-FQDN] Remote command failed: mkdir -p /var/vmware/prelude/upgrade at host: Node2_FQDN
[ERROR][####-##-## ##:##:##][VRA_Node-FQDN] Remote command failed: mkdir -p /var/vmware/prelude/upgrade at one or more nodes
[ERROR][####-##-## ##:##:##][VRA_Node-FQDN][Exit Code: 255] Attempt failed to run command: mkdir -p /var/vmware/prelude/upgrade.
Pseudo-terminal will not be allocated because stdin is not a terminal.
ssh: connect to host Node3_FQDN port 22: Connection refused
[ERROR][####-##-## ##:##:##][VRA_Node-FQDN][Exit Code: 124] Attempt failed to run command: /opt/scripts/wait-k8s-up -t 600 local-infra.
Pseudo-terminal will not be allocated because stdin is not a terminal.
Welcome to VMware Aria Automation Appliance 8.18.1
[ERROR][####-##-## ##:##:##][VRA_Node-FQDN] Remote command failed: /opt/scripts/wait-k8s-up -t 600 local-infra at one or more nodes
[ERROR][####-##-## ##:##:##][VRA_Node-FQDN] System services failed to start.

• Apart from the SSH communication issue, the /var/log/bootstrap/postupdate.log also shows the following exception:

Waiting for infrastructure services..
+ /opt/scripts/wait-k8s-up -t 600 local-infra
Waiting for local-infra timed out: pod(s) not ready: kube-dns, network-health-monitor
Error setting up Kubernetes.Retrying.
2024-08-29 02:55:08Z Script /etc/bootstrap/postupdate-late.d/03-10-setup-k8s.sh failed, error status 1

Environment

VMware Aria Automation 8.x

Cause

This issue occurs when OS-level security hardening prevents the initialization of the bridge kernel module. Specifically, this happens if the install bridge /bin/false directive is active in /etc/modprobe.d/modprobe.conf

Resolution

Apply following action plan to resolve the upgrade issue:

1. Revert the environment to pre-upgrade snapshot.
2. Log in to all Aria Automation nodes via SSH and verify the configuration by running cat /etc/modprobe.d/modprobe.conf. Ensure that the following line is commented out:

    #install bridge /bin/false

    If the line is not commented, open the /etc/modprobe.d/modprobe.conf file using the vi editor and disable the bridge restriction by adding a # at the beginning of the above line.

3. Reboot the Aria automation appliance.
4. Re-trigger the upgrade.