Aria Operations cloud proxy upgrade fails due to lack of enough free space in root partition

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• During an Aria operations upgrade, one or more cloud proxy nodes may fail to upgrade.
• In the /storage/db/vmware-vrops-cprc/status/upgrade/vRealize-Operations-Cloud-Proxy-build_number/cprc.upgrade.process.log on the cloud proxy node, you may find entries similar to:

root - ERROR - Script command: [rpm -Uvh  --noverify --nodigest --nofiledigest --force /storage/db/vmware-vrops-cprc-upgrade-working-dir/vRealize-Operations-Cloud-Proxy-build_number/iso/product/vmware-cloudproxy-build_number.x86_64.rpm] failed with exit code: 2
root - ERROR - RPM upgrade failed with exit code: 2
root - ERROR - RPM upgrade failure stderr:    installing package vmware-cloudproxy-build_number.x86_64 needs ###MB more space on the / filesystem
 stdout: Preparing...                          ########################################
root - ERROR - Executing FAILURE_EXIT. Reason: Failed to install RPM: /storage/db/vmware-vrops-cprc-upgrade-working-dir/vRealize-Operations-Cloud-Proxy-build_number/iso/product/vmware-cloudproxy-build_number.x86_64.rpm
root - ERROR - Updating document for failure exit.
root - ERROR - Updating PAK document for failure exit.
root - ERROR - Updating upgrade status document for failure exit.
root - ERROR - !!! FAILURE EXIT: Failed to install RPM: /storage/db/vmware-vrops-cprc-upgrade-working-dir/
vRealize-Operations-Cloud-Proxy-build_number/iso/product/vmware-cloudproxy-build_number.x86_64.rpm

• Running df -h shows root partition utilization over 80%

Environment

Aria Operations 8.x

Cause

Root partition on the cloud proxy nodes do not have enough free space for the upgrade to complete

Resolution

• Check for large files under '/storage/db/vmware-vrops-cprc/upgrade/download' directory using the following command:
  

  du -sh *

• Remove any manually copied files that are consuming significant space (for example, upgrade .pak files) using 'rm'command.
• Delete the previously extracted Aria operations upgrade files if exists from '/storage/db/vmware-vrops-cprc/upgrade/extract/'
• Reduce the space utilization of the root partition to less than 80% by referring to Troubleshooting Storage Issues in Aria Operations
• Manually upgrade the cloud proxy nodes by referring to How to manually upgrade a VMware Aria Operations Cloud Proxy via CLI

Note: if there is a permission denied error when executing crpc-cli command during manual upgrade, use chmod +x /usr/lib/vmware-vrops-cprc/bin/shell/cprc-cli.sh to set execute permissions