VCF upgrade fails on "SDDC Manager Deployment Drift stage" with an error "Failure while executing SDDC Manager drift app"

Products

• VMware SDDC Manager

Issue/Introduction

• SDDC update from 5.2.1.0, 5.2.1.1, 5.2.1.2 to 9.0 fails on "SDDC Manager Deployment Drift" stage with an error message as shown below.
  
  

 

 

• Under /var/log/vmware/vcf/lcm/thirdparty/upgrades/########-####-####-####-############/sddcmanager-migration-app/logs/sddcmanager_migration_app_upgrade.log file, below log snippets are available.
   
  YYYY-MM-DDTHH:MM:SS+0000 DEBUG [vcf_lcm,0000000000000000,0000] [c.v.v.s.t.DynamicTrustManager,pool-4-thread-3]  Error checking certificate chain CN=<SDDC FQDN>, OU=VMware Engineering, O=VMware, L=<City>, ST=<State>, C=<Country>,OU=VMware Engineering, O=<vCenter FQDN>, ST=<State>, C=<Country>, DC=local, DC=vsphere, CN=CA for validity.
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
        at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:242)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:113)
        at com.vmware.vcf.secure.truststore.DynamicTrustManager.checkServerTrusted(DynamicTrustManager.java:49)
        at java.base/sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:1436)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)

Caused by: java.security.SignatureException: Signature does not match.
        at java.base/sun.security.x509.X509CertImpl.verify(X509CertImpl.java:416)
        at java.base/sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166)
        at java.base/sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147)
        at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
        ... 70 common frames omitted

 

• Under /var/log/vmware/domainmanager/domainmanager.log, below log snippets are available:
  
  YYYY-MM-DDTHH:MM:SS+0000 DEBUG [vcf_om,###############,####] [c.v.e.s.c.c.CertificateRetrieverService,http-nio-127.0.0.1-XXXX-exec-5] Certificate chain validity check against current PKIXParameters failed
java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:157)
        at java.base/sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83)
        at java.base/java.security.cert.CertPathValidator.validate(CertPathValidator.java:309)
        at com.vmware.evo.sddc.common.certificateutil.CertificateRetrieverService.getTrustedCertificateValidatingChain(CertificateRetrieverService.java:83)
        at com.vmware.evo.sddc.common.util.SslUtil.getCertificateChain(SslUtil.java:85)
        at com.vmware.evo.sddc.common.util.SSOEntityService.getSecurityTokenService(SSOEntityService.java:429)
  
  

Cause

Certificates of the components under respective Domains (Workload/Management) are Inactive/Certificate not trusted. Which makes them inactive.

Resolution

1. Log into SDDC Manager.
2. Go to Inventory, then select your target Domain (WLD/MGMT).
3. Under the Certificates tab, click on the Inactive/Certificate not trusted option, where a small pop-up Window will appear.
4. Click on Trust the Certificate to make it as Active status.
   
   Example:
   
   

     4. Reboot the SDDC Manager appliance to enact these changes.
     5. Proceed with the upgrade attempt once the reboot is complete.