Upgrade of Cluster VRA 8.x fails with Split brain scenario

Products

• VMware Aria Suite

Issue/Introduction

• To successfully upgrade a vRA cluster.

Symptoms:

• 3 node vRA cluster upgrade fails and each node tries to act as a separate cluster
• upgrade-noop-error log shows vRA fails to make the upgrade directory :

[INFO][2022-08-05 19:16:01][xxxx.xxxxx.xxxx.xxxx] Preparing directory for upgrade data on remote node: [INFO][2022-08-05 19:16:01][xxxx.xxxxx.xxxx.xxxx] Running remote command: mkdir -p /var/vmware/prelude/upgrade at host: *

[ERROR][2022-08-05 19:16:01][xxxx.xxxxx.xxxx.xxxx][Exit Code: 255] Attempt failed to run command: mkdir -p /var/vmware/prelude/upgrade.

 

Environment

VMware vRealize Automation 8.x

Cause

• This happens because the appliance fails to make the upgrade folder at "/var/vmware/prelude/upgrade" because the folder already exists from old upgrade attempts.

Resolution

• There is a workaround. 

Workaround:

• The workaround steps are below: 

1. Revert back to the snapshot before the upgrade.
2. Remove the stale upgrade folder from each node : rm -r /var/vmware/prelude/upgrade
3. Take new snapshots of the vRA nodes.
4. Trigger the upgrade again.

Additional Information

Impact/Risks:

• No major impact.