Troubleshooting VMware Aria Automation 8.x (formerly vRealize Automation 8.x) upgrades

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

This article provides steps to manually collect upgrade logs if the log bundle collection fails when a VMware Aria Automation upgrade fails.

When an upgrade fails, the following logs need to be analyzed:

Log File Path	Description
`/var/log/vmware/prelude/upgrade-*.log`	Upgrade reports. Review based on timestamp.
Single-node environment: `/opt/vmware/var/log/vami/.log` Cluster environments: `/opt/log/vmware/var/log/vami/.log`	Package installation details.
`/var/log/bootstrap/postupdate.log`	Initialization script details.
`/var/log/bootstrap/everyboot.log`	Initialization script details.
`/var/log/vmware/prelude/deploy-*.log`	Service startup details.

Note: Some log files have timestamps as part of the file name, while for others the information is appended. It is important to validate that the information is from the latest upgrade attempt.

Overview

Upgrade Prerequisites

Read the product release notes.
Check hardware requirements.
Check the Services status.
Ensure backups are available.
Ensure a pre-upgrade snapshot is taken.

Trigger Upgrade in Aria Suite Lifecycle

Validate prechecks are successful.

Upgrade Process Breakdown

The manifest is downloaded (upgrade-noop.log).
Searches for the bootstrap package and checks if the SSH port is open and able to connect (update-datetime.log).
Retrieves product version on all nodes.
Performs infrastructure health checks.
Shuts down infrastructure and application services.
Saves a local restore point for Kubernetes processes node data and activates the monitor on all nodes.
Once the upgrade monitor is activated and the cluster nodes are removed successfully, it proceeds with installation. This takes about 30 minutes.
VAMI upgrade starts.
Download of all packages occurs (vami.log).
Once downloaded, installation of packages starts (updatecli.log & postupdate.log).
Once completed, the appliance reboots and the VAMI upgrade is marked successful (update-datetime.log).
Cluster nodes are added back in and restore points are restored.
Infrastructure and application services start.
Upgrade cleanup is performed.

Environment

VMware vRealize Automation 8.x
VMware Aria Automation 8.x

Resolution

Manually Collecting Upgrade Failure Diagnostic Information

SSH to the VMware Aria Automation node indicated in the Aria Suite Lifecycle error.

Validate that there is available disk space in the root partition (/dev/sda4) by running the vracli disk-mgr command:

root@vranode1 [ /tmp ]# vracli disk-mgr
/dev/sda4(/):
        Total size: 47.80GiB
        Free: 33.58GiB(70.2%)
        Available(for non-superusers): 31.13GiB(65.1%)
        SCSI ID: (0:0)
/dev/sdb(/data):
        Total size: 140.68GiB
        Free: 109.54GiB(77.9%)
        Available(for non-superusers): 102.32GiB(72.7%)
        SCSI ID: (0:1)
/dev/sdc(/var/log):
        Total size: 21.48GiB
        Free: 9.09GiB(42.3%)
        Available(for non-superusers): 7.97GiB(37.1%)
        SCSI ID: (0:2)
/dev/sdd(/home):
        Total size: 29.36GiB
        Free: 27.41GiB(93.4%)
        Available(for non-superusers): 25.90GiB(88.2%)
        SCSI ID: (0:3)

Run the following command to collect the directories and logs related to the upgrade:

mkdir /tmp/upgradelogs && cp -pR /var/log/vmware/prelude /tmp/upgradelogs && cp -pR /opt/vmware/var/log/vami /tmp/upgradelogs && cp -pR /var/log/bootstrap /tmp/upgradelogs && tar -zcvf /tmp/upgradelogs.tar.gz /tmp/upgradelogs

Extract and continue the review with the collected file /tmp/upgradelogs.tar.gz for the failure code, or submit this data to Global Services for additional assistance in troubleshooting the upgrade.
After extracting the file from the appliance, remove the file and directory to save disk space:
```
cd /tmp
rm upgradelogs.tar.gz
rm -r upgradelogs
```