HCX Manager is not starting after upgrading from 4.9.2 to 4.11.3

Products

• VMware HCX

Issue/Introduction

• After upgrade of HCX server from 4.9.2 to 4.11.3 the HCX Manager is not starting.
• Unable to login to the UI
• After upgrading HCX Manager from 4.9.2 to 4.11.3 app-engine service fails to start.
• Found that mgmt. network adapter was "disconnected' - unsure when that occurred.
  
  
• By Consoling into the VM via vCenter UI we see the following by running the command <systemctl status>:
  
  
  
• The </common/versions/version.properties> file shows successful upgrade:
  

  em.majorVersion=
  em.minorVersion=
  em.buildNumber=24972693
  em.preReleaseType=
  em.marketingMajorVersion=4
  em.marketingMinorVersion=11
  em.marketingMaintenanceVersion=3
  em.marketingPatchVersion=0
  hcx.releaseCodeNumber=280

  
  
  
• The <common/logs/upgrade/upgrade.log> confirms upgrade was successful:
  

  <timestamps> UTC Validating the upgrade bundle .............................................................................. [   OK ]
  <timestamps> UTC Backing-up certs, configs and database before the upgrade .................................................. [   OK ]
  <timestamps> UTC Stopping all the services .................................................................................. [   OK ]
  <timestamps> UTC Extracting distribution bundle ............................................................................. [   OK ]
  <timestamps> UTC Installing the upgrade image ............................................................................... [   OK ]
  <timestamps> UTC Updating each components version ........................................................................... [   OK ]
  <timestamps> UTC Upgrade successful, restarting the HCX Manager VM .......................................................... [   OK ]

• Reviewing </common/log/admin/app.log> in HCX Manager shows:
  

  <timestamps> UTC [main, , , TxId: ] INFO  c.v.v.h.orchestration.Application- Starting app and looking for config on zNode: /vchsApplication
  <timestamps> UTC [main-EventThread, , , TxId: ] INFO  c.v.v.h.orchestration.Application- com.vmware.vchs.hybridity.orchestration.Application, com.vmware.vchs.hybridity.orchestration.Application , inside process(WatchedEvent)
  
  <timestamps> UTC [main, , , TxId: ] ERROR c.v.v.h.c.utils.ApplianceConfigUtil- Failed to fetch section: proxy
  javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
  
  <timestamps> UTC [main, , , TxId: ] ERROR c.v.v.h.orchestration.Application- Exception during application initialization:
  java.lang.RuntimeException: Failed to fetch section: proxy
  Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
  

Environment

VMware HCX

Cause

Key rotation failed to occur in proper sequence/time during reboot/upgrade cycle. 

Resolution

If you believe you have encountered this issue, open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases.