HCX Manager upgrade fails with an error as "Failed to backup certs, configs, secrets and database"
search cancel

HCX Manager upgrade fails with an error as "Failed to backup certs, configs, secrets and database"

book

Article ID: 381186

calendar_today

Updated On:

Products

VMware HCX

Issue/Introduction

  • HCX Manager upgrade fails.
  • Error "Failed to backup certs, configs, secrets and database"
  • HCX Manager upgrade logs shows below : /common/logs/upgrade/upgrade-detailed.log

    Validating the upgrade bundle
    Executing the pre—upgrade validation script
    WARN: It is advised to take a snapshot/backup of the HCX instance before you proceed with the upgrade. During the upgrade process all the services will be stopped and will be started automatically once the upgrade is complete.
    Validating appliance type of the upgrade bundle .
    Validating upgrade bundle integrity.
    Validation: upgrade bundle integrity (SHA256) check succeeded.
    Validating disk space availability for upgrade.
    Disk space availability check succeeded.
    Failed to backup certs, configs, secrets and database

  • /common/logs/admin/app.log shows traces about running on low disk space.

     2024-11-02 05:39:06.610 UTC [VrealmUpgradeService_EventListener, , , TxId: ] INFO  c.v.v.h.m.a.EventListenerAdapter- Received AfterEventUpgrade event: {"eventType":"AfterUpgrade","id":"AfterUpgradeEvent","payload":{"upgradeStatus":{"endpointId":"20190609221616423-########-c1cf32b46112","version":"4.8.2.0.23247196","status":"FAILED","message":"Upgrade failed. Reason: Error : 
    
    2024-11-02T05:39:01 Backing-up certs, configs and database before the upgrade----[backup-start]-----Backup started on Sat Nov 2 05:39:01 UTC 2024Postgres Service is running.Running low on disk space. Backup cannot be performed.Running low on disk space. 
    2024-11-02T05:39:01 Failed to backup certs, configs, secretes and  database","upgradeEndTime":1730525941519}},"producer":"UpgradeTask"



Environment

VMware HCX

Cause

The HCX Manager upgrade failed due to /common directory running low on disk space.
NOTE: It is recommended to have around 40% free space under /common before HCX Manager upgrade.

Resolution

If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases

Please do run through the below steps to collect information and share via the SR raised.
  • SSH to HCX Manager as admin.

  • Change to root. Use root password.
    su -
  • Run df -h to check space used by all partitions. 
    In this case /common usage was high. 

  • Change to common directory and run the below commands and collect this output to a notepad. 
    cd /common
    du -shc * 
    du /common -ahx . | sort -rh | head -n 10


Additional Information