HCX Manager upgrade fails with an error as "Failed to backup certs, configs, secrets and database"

Products

• VMware HCX

Issue/Introduction

• HCX Manager upgrade fails.
• Error "Failed to backup certs, configs, secrets and database"
• HCX Manager upgrade logs shows below : /common/logs/upgrade/upgrade-detailed.log
  
  

  Validating the upgrade bundle
  Executing the pre—upgrade validation script
  WARN: It is advised to take a snapshot/backup of the HCX instance before you proceed with the upgrade. During the upgrade process all the services will be stopped and will be started automatically once the upgrade is complete.
  Validating appliance type of the upgrade bundle .
  Validating upgrade bundle integrity.
  Validation: upgrade bundle integrity (SHA256) check succeeded.
  Validating disk space availability for upgrade.
  Disk space availability check succeeded.
  Failed to backup certs, configs, secrets and database

  

• /common/logs/admin/app.log shows traces about running on low disk space.
  
  

   2024-11-02 05:39:06.610 UTC [VrealmUpgradeService_EventListener, , , TxId: ] INFO  c.v.v.h.m.a.EventListenerAdapter- Received AfterEventUpgrade event: {"eventType":"AfterUpgrade","id":"AfterUpgradeEvent","payload":{"upgradeStatus":{"endpointId":"20190609221616423-########-c1cf32b46112","version":"4.8.2.0.23247196","status":"FAILED","message":"Upgrade failed. Reason: Error : 
  
  2024-11-02T05:39:01 Backing-up certs, configs and database before the upgrade----[backup-start]-----Backup started on Sat Nov 2 05:39:01 UTC 2024Postgres Service is running.Running low on disk space. Backup cannot be performed.Running low on disk space. 
  2024-11-02T05:39:01 Failed to backup certs, configs, secretes and  database","upgradeEndTime":1730525941519}},"producer":"UpgradeTask"

  
  
  

Environment

VMware HCX

Cause

The HCX Manager upgrade failed due to /common directory running low on disk space.
NOTE: It is recommended to have around 40% free space under /common before HCX Manager upgrade.

Resolution

If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases

Please do run through the below steps to collect information and share via the SR raised.

• SSH to HCX Manager as admin.
  
  
• Change to root. Use root password.
  su -

• Run df -h to check space used by all partitions. 
  In this case /common usage was high. 
  
  
• Change to common directory and run the below commands and collect this output to a notepad. 
  cd /common
du -shc * 
du /common -ahx . | sort -rh | head -n 10


Additional Information

HCX - Connector or Cloud Manager unresponsive due to high utilization of "common" directory