Upgrading to vRA 8.5.1 or later fails with the error "Upgraded terminated due to critical error"

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Upgrading to vRA 8.5.1 or later fails with the error Upgraded terminated due to critical error
• The command du -hs /root shows the content inside /root exceeding 98% of the available space of /metrics as displayed by df -h /metrics/
• A large number of log bundle tar files exist in the ~ home directory

Environment

VMware vRealize Automation 8.x

Cause

• During upgrades to vRA 8.5.1 and later, the root user's home directory is migrated into a different partition on Hard Disk 4 of the VM.
• If the content inside /root is larger that the capacity of Hard Disk 4 prior to upgrading, the new location might not have sufficient capacity to execute the migration successfully.

Resolution

VMware is aware of this issue.  See the workaround below for further details.

Workaround:

Procedure

1. Restore the latest pre-upgrade snapshot.
2. SSH / PuTTy into one node in the cluster.
3. Run the following command

   vracli cluster exec -- sh -lc "base64 -d <<< 'c2V0IC1lCnZlcj0kKHZyYWNsaSB2ZXJzaW9uIHwgaGVhZCAtbiAxIHwgYXdrICd7cHJpbnQgJDN9JyB8IGF3ayAtRiAuICd7cHJpbnQgJDEkMiQzfScpCmlmIFsgJHZlciAtZ2UgODUxIF07IHRoZW4KICBlY2hvICJUaGlzIEtCIGlzIG5vdCBhcHBsaWNhYmxlIGZvciB0aGlzIHZSQSB2ZXJzaW9ucyBiZWxvdyA4LjUuMSIKICBleGl0IDAKZmkKbXRyRnJTcGNLYj0kKGRmIC1QIC9tZXRyaWNzIHwgYXdrICdFTkR7cHJpbnQgJDR9JykKbXRyVG90U3BjS2I9JChkZiAtUCAvbWV0cmljcyB8IGF3ayAnRU5Ee3ByaW50ICQyfScpCnJ0U3pLYj0kKGR1IC1rcyAvcm9vdCB8IGF3ayAne3ByaW50ICQxfScpCm10clJlbUZyU3BjS2I9JCgoIG10ckZyU3BjS2ItcnRTektiICkpCm10ckV4cE1pbkZyU3BjS2I9JCgoIG10clRvdFNwY0tiKjEvMTAwICkpCmlmIFsgJG10clJlbUZyU3BjS2IgLWxlICRtdHJFeHBNaW5GclNwY0tiIF07IHRoZW4KICAgIGRpZmY9JCgoIChtdHJFeHBNaW5GclNwY0tiLW10clJlbUZyU3BjS2IpLzEwMjQvMTAyNCsxICkpCiAgICBlY2hvICJVcGdyYWRlIHJlcXVpcmVzIGFkZGl0aW9uYWwgJGRpZmYgR0Igb2YgZGlzayBzcGFjZSBvbiBWTSBIYXJkIGRpc2sgNCBvZiAkKCBjdXJyZW50X25vZGUgKS4gUmVzaXplIFZNIEhhcmQgZGlzayA0IGFjY29yZGluZ2x5IG9yIHJlbW92ZSBvYnNvbGV0ZSB1c2VyIGZpbGVzIGZyb20gL3Jvb3QgaW4gb3JkZXIgdG8gcHJvY2VlZCB3aXRoIHRoZSB1cGdyYWRlLiIKICAgIGV4aXQgMQplbHNlCiAgICBlY2hvICJObyBhZGRpdGlvbmFsIGRpc2sgc3BhY2UgaXMgcmVxdWlyZWQgb24gbm9kZTogJCggY3VycmVudF9ub2RlICkiCmZpCg==' | sh -"

4. If the check from Step#3 fails with the message Upgrade requires additional X GB of disk space on VM Hard disk 4 for any of the nodes in the cluster, follow the instructions from the script output for each respective node.
5. Run

   vracli disk-mgr resize

6. Repeat Step#3 to ensure enough space has been freed to perform the upgrade.

Additional Information

Increase vRealize Automation appliance disk space