HCX - Unable to upgrade IX/NE Appliances : Error "Given final block not properly padded".

Products

• VMware HCX
• VMware Cloud on AWS

Issue/Introduction

This resource is to inform about IX/NE Redeployment/Upgrade failure and how to recover that.

HCX Fleet Appliances IX/NE may not be able to upgrade to the latest build 4.8 and below error/exceptions can be seen in app-engine logs:

/common/logs/admin/app.log

2023-12-19 12:34:36.529 UTC [InterconnectService_SvcThread-1129, IX:####-####9-####-####-########347, J:5ad397c3, , TxId: TxId: ####-####-####-####-########15b] ERROR c.v.v.h.s.i.DeployAppliance- DeployAppliance failed in state RECONFIGURE_VM. Error: Given final block not properly padded. Such issues can arise if a bad key is used during decryption..
javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

Environment

VMware HCX

Cause

HCX Manager maintains credential for fleet appliances IX/NE for login purposes. During HCX 4.8.0 upgrade, it has been observed that the credential for few appliances got malformed which impacted the upgrade/redeploy workflow only for those appliances.
Note: This symptom is very rare and should be considered as a corner case.

Resolution

This issue is fixed in HCX version 4.8.2.

Workaround:
As a potential workaround, below steps can be performed to fix the credential related issues in IX/NE Appliances and to unblock upgrade/redeploy workflow:

• Refer to HCX Connector/Cloud vCenter Plugin OR Standalone UI where IX/NE appliances cannot be upgraded and also "Given final block not properly padded" error can be seen.

Go to Interconnect >> Service Mesh >> Appliances >> Click on the IX or NE appliance >> Click "CHANGE PASSWORD"

•  Enter "root" and "admin" credentials in the below UI and then "UPDATE".
  

Note: User can choose the custom password for "root" & "admin" account.
Once "UPDATE" option is clicked, a "Task" will be initiated in the backend to update the root/admin credentials and its validity. Please refer SM >> Appliances >> Task section.

• After successful update of the root/admin credentials for IX or NE Appliances, the upgrade/redeploy operation can be re-attempted from Service Mesh Wizard.

 

Additional Information

Impact/Risks:

• The impact is applicable to IX & NE both appliances.
• Only Upgrade/Redeploy workflow related to IX/NE appliances won't be serviced.
• Existing Network Extension data path will remain unaffected.
• Migration workflow remains functional over existing IX Appliances running old version.