HCX Manager is not starting after upgrading from 4.9.2 to 4.11.3

Products

• VMware HCX

Issue/Introduction

• After upgrade of HCX server from 4.9.2 to 4.11.3 the HCX Manager is not starting.
• Unable to login to the UI
• After upgrading HCX Manager from 4.9.2 to 4.11.3 app-engine service fails to start.
• Found that mgmt. network adapter was "disconnected' - unsure when that occurred. 
• By Consoling into the VM via vCenter UI we see the following by running the command <systemctl status>:
  
• The </common/versions/version.properties> file shows successful upgrade:
  

  em.majorVersion=
  em.minorVersion=
  em.buildNumber=24972693
  em.preReleaseType=
  em.marketingMajorVersion=4
  em.marketingMinorVersion=11
  em.marketingMaintenanceVersion=3
  em.marketingPatchVersion=0
  hcx.releaseCodeNumber=280

  
  
  
• The <common/logs/upgrade/upgrade.log> confirms upgrade was successful:
  

  <timestamps> UTC Validating the upgrade bundle .............................................................................. [   OK ]
  <timestamps> UTC Backing-up certs, configs and database before the upgrade .................................................. [   OK ]
  <timestamps> UTC Stopping all the services .................................................................................. [   OK ]
  <timestamps> UTC Extracting distribution bundle ............................................................................. [   OK ]
  <timestamps> UTC Installing the upgrade image ............................................................................... [   OK ]
  <timestamps> UTC Updating each components version ........................................................................... [   OK ]
  <timestamps> UTC Upgrade successful, restarting the HCX Manager VM .......................................................... [   OK ]

• Reviewing </common/log/admin/app.log> in HCX Manager shows:
  

  <timestamps> UTC [main, , , TxId: ] INFO  c.v.v.h.orchestration.Application- Starting app and looking for config on zNode: /vchsApplication
  <timestamps> UTC [main-EventThread, , , TxId: ] INFO  c.v.v.h.orchestration.Application- com.vmware.vchs.hybridity.orchestration.Application, com.vmware.vchs.hybridity.orchestration.Application , inside process(WatchedEvent)
  
  
  <timestamps> UTC [main, , , TxId: ] ERROR c.v.v.h.c.utils.ApplianceConfigUtil- Failed to fetch section: proxy
  javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
          at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:981)
          at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1062)
          at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853)
          at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
          at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2202)
          at com.vmware.vchs.hybridity.util.KeyStoreUtil.decryptData(KeyStoreUtil.java:221)
          at com.vmware.vchs.hybridity.util.KeyStoreUtil.decryptData(KeyStoreUtil.java:198)
          at com.vmware.vchs.hybridity.config.utils.ApplianceConfigUtil.decrypt(ApplianceConfigUtil.java:239)
          at com.vmware.vchs.hybridity.config.utils.ApplianceConfigUtil.getSectionConfig(ApplianceConfigUtil.java:143)
          at com.vmware.vchs.hybridity.orchestration.Application.setProxy(Application.java:248)
          at com.vmware.vchs.hybridity.orchestration.Application.setProxySelector(Application.java:244)
          at com.vmware.vchs.hybridity.orchestration.Application.run(Application.java:220)
          at com.vmware.vchs.hybridity.orchestration.Application.main(Application.java:138)
  <timestamps> UTC [main, , , TxId: ] ERROR c.v.v.h.orchestration.Application- Exception during application initialization:
  java.lang.RuntimeException: Failed to fetch section: proxy
          at com.vmware.vchs.hybridity.config.utils.ApplianceConfigUtil.getSectionConfig(ApplianceConfigUtil.java:153)
          at com.vmware.vchs.hybridity.orchestration.Application.setProxy(Application.java:248)
          at com.vmware.vchs.hybridity.orchestration.Application.setProxySelector(Application.java:244)
          at com.vmware.vchs.hybridity.orchestration.Application.run(Application.java:220)
          at com.vmware.vchs.hybridity.orchestration.Application.main(Application.java:138)
  Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
          at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:981)
          at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1062)
          at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853)
          at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
          at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2202)
          at com.vmware.vchs.hybridity.util.KeyStoreUtil.decryptData(KeyStoreUtil.java:221)
          at com.vmware.vchs.hybridity.util.KeyStoreUtil.decryptData(KeyStoreUtil.java:198)
          at com.vmware.vchs.hybridity.config.utils.ApplianceConfigUtil.decrypt(ApplianceConfigUtil.java:239)
          at com.vmware.vchs.hybridity.config.utils.ApplianceConfigUtil.getSectionConfig(ApplianceConfigUtil.java:143)
          ... 4 common frames omitted

Environment

VMware HCX

Cause

• Key rotation failed to occur in proper sequence/time during reboot/upgrade cycle. 

Resolution

• Please open a case with Broadcom Support for assistance in resolving this issue.
  • Creating and managing Broadcom support cases