SDDC Manager Upgrade in VCF 5.x fails on "SDDC Manager Deployment Drift stage"
search cancel

SDDC Manager Upgrade in VCF 5.x fails on "SDDC Manager Deployment Drift stage"

book

Article ID: 327212

calendar_today

Updated On:

Products

VMware Cloud Foundation VMware SDDC Manager

Issue/Introduction

  • The follow error is observed in the SDDC UI.



  • Error in /var/log/vmware/vcf/lcm/thirdparty/upgrades/<upgrade id from SDDC UI>/sddcmanager-migration-app/logs 
    ERROR [vcf_lcm,0000000000000000,0000] [c.v.e.s.m.s.s.ExportDataService,pool-4-thread-16]  Error while fetching credentials data
org.springframework.web.client.HttpServerErrorException$GatewayTimeout: 504 Gateway Time-out: "<html><EOL><EOL><head><title>504 Gateway Time-out</title></head><EOL><EOL><body><EOL><EOL><center><h1>504 Gateway Time-out</h1></center><EOL><EOL><hr><center>nginx</center><EOL><EOL></body><EOL><EOL></html><EOL><EOL>"
        at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:114)


ERROR [vcf_lcm,0000000000000000,0000] [c.v.e.s.o.model.error.ErrorFactory,pool-4-thread-16]  [F####0] FAILED_TO_GET_CREDENTIAL_DATA_EXPORT Failed to get credentials data using export api.
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to get credentials data using export api.

     

  • credentialhistory table in SDDC Manager platform database has large number of entries 
    /usr/pgsql/13/bin/psql -h localhost -U postgres -d platform -c "select count(*) from credentialhistory;"
 count
-------
 10798
(1 row)

 

Environment

VMware Cloud foundation 5.x

Cause

The large number of credentialhistory records causes the credential export API to timeout. 

 

Resolution

Engineering is aware of this issue and a permanent fix is planned for inclusion in a future release.

 

Workaround Steps:

Option#1 Delete records older than 3 months.

  1. Take a snapshot of the SDDC Manager VM
  2. Delete credential history older than 3 months.  
    /usr/pgsql/13/bin/psql -h localhost -U postgres -d platform -c "DELETE FROM credentialhistory WHERE creationtime < EXTRACT(epoch FROM NOW() - INTERVAL '3 months') * 1000;"
  3. Retry the upgrade

Option#2 Temporarily move records to a backup table

  1. Take a snapshot of SDDC Manager VM
  2. SSH into SDDC Manager VM and elevate to root
  3. Connect to the SDDC DB 
    /usr/pgsql/13/bin/psql -h localhost -U postgres -d platform
  4. Create the credentialhistory backup. 
    create table credentialhistory_bak as table credentialhistory;
  5. Truncate the credentialhistory table. 
    truncate credentialhistory;
  6. Retry the SDDC Upgrade.
  7. After upgrade is complete, SSH into SDDC Manager VM and elevate to root
  8. Connect to the SDDC DB 
    /usr/pgsql/13/bin/psql -h localhost -U postgres -d platform
  9. Restore the credentialhistory table after the SDDC upgrade completes. 
    insert into credentialhistory select * from credentialhistory_bak;
  10. Drop the credential backup table. 
    drop table credentialhistory_bak;
Powered by Wolken Software