SDDC Manager Upgrade Fails with Error: Failure While Executing SDDC Manager Drift App
search cancel

SDDC Manager Upgrade Fails with Error: Failure While Executing SDDC Manager Drift App

book

Article ID: 448410

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

  • During an SDDC Manager upgrade in VMware Cloud Foundation (VCF) 9.1, the upgrade process fails at the SDDC Manager Deployment Drift pre-check step.



  • Reviewing the logs reveals 502 Bad Gateway errors resulting from the crash of the operationsmanager service:

    In /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log:

    YYYY-MM-DDThh:mm:ss.370+0000 ERROR [common,0000000000000000,0000] [c.v.e.s.t.services.TaskCollector,pool-2-thread-4] collect() failed
    org.springframework.web.client.HttpServerErrorException$BadGateway: 502 Bad Gateway on GET request for "http://localhost/operationsmanager/workflows/e8dba9d8-70b7-441f-ad44-49dded10a054": "<html>...<title>502 Bad Gateway</title>...</html>"

    In /var/log/vmware/vcf/lcm/thirdparty/upgrades/<xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx>/sddcmanager-migration-app/logs/ddcmanager_migration_app_upgrade.log:

    YYYY-MM-DDThh:mm:ss.078+0000 ERROR [vcf_lcm,0000000000000000,0000] [c.v.e.s.o.model.error.ErrorFactory,pool-5-thread-5] [Q0DAMQ] FAILED_TO_DISABLE_BASIC_AUTH_SDDC_MANAGER Failed to disable Basic Authentication API access on SDDC Manager com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to disable Basic Authentication API access on SDDC Manager ... Caused by: org.springframework.web.client.HttpServerErrorException$BadGateway: 502 Bad Gateway on PATCH request for "https://<REDACTED_HOSTNAME>/v1/sddc-manager"

  • Checking disk utilization via df -h shows that the partition /dev/mapper/data_vg-vmware is 100% full due to memory heap dump files (java_pidxxxxxx.hprof) accumulating in /var/log/vmware/vcf/operationsmanager/app-crash-logs:

Environment

VMware Cloud Foundation (VCF) 9.x

Cause

The operationsmanager service on the SDDC Manager exhausts its allocated Java heap memory during the pre-check task, causing the service to crash repeatedly and generate large heap dump (.hprof) files that fill up the /dev/mapper/data_vg-vmware storage partition.

Resolution

  1. Take a snapshot of the SDDC Manager virtual machine before making any changes.

  2. Establish an SSH session to the SDDC Manager and log in as the root user.

  3. Clean up old .hprof files in /var/log/vmware/vcf/operationsmanager/app-crash-logs to free up space on /dev/mapper/data_vg-vmware:
    rm -f /var/log/vmware/vcf/operationsmanager/app-crash-logs/*.hprof

  4. Open /etc/vmware/vcf/operationsmanager/vcf-operations-manager.conf using a text editor such as vi:
    vi /etc/vmware/vcf/operationsmanager/vcf-operations-manager.conf
  5. Increase the maximum Java heap size parameter from -Xmx2G to -Xmx4G.

  6. Restart the operationsmanager service and verify its operational status:
    systemctl restart operationsmanager
    systemctl status operationsmanager

 

Powered by Wolken Software