Aria Operations upgrade fails with Error: upgrade.pak.file_copy_failed due to node running out of disk space

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

The upgrade of Aria Operations fails with Error: upgrade.pak.file_copy_failed as shown under the "Software Update" for the status of the upgrade.

Environment

Aria Operations 8.x

Cause

This issue is commonly caused due to one or more of the Aria Operations nodes running out of disk space during the upgrade.

Resolution

To verify if the upgrade failed due to a node running out of disk space, carry out the following steps:

1. SSH to each of the Aria Operations nodes
2. Login as root and run the following command to verify how much storage is available on each partition.  Note it is recommend for upgrade to have at least 30 GB free space available on each partition:

   df -h  

   • If one or more nodes does not have enough available disk space for /storage/db , please follow the procedure How to add disks to a node in Aria Operations cluster (318768) to add additional storage to the impacted nodes.
   • If one or more nodes does not have enough available disk space for /storage/log directory please follow the procedure in How to safely clean /storage/log in Aria Operations (315972) to safely clean up the directory.
3. Once all nodes have sufficient free disk space the upgrade can be restarted. 

Additional Information

We recommend to revert to the snapshot taken prior to the upgrade failure, then implement the procedures to clear or add additional disk space to each Aria Operations node and then starting the upgrade.