Standby cell deployment fails with "Password Decryption Error" in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

When attempting to deploy a new standby node into an existing VMware Cloud Director (VCD) database high availability cluster, the deployment sequence fails.

The installation logs reveal a decryption exception within the user key password retrieval routine:

private char[] getUserKeyPassword(PropertiesConfiguration responses, boolean passwordIsInvalid)
        throws IOException {
    char[] userSslPassword = null;
    ...
    } else if (!shouldPrompt && StringUtils.isNotEmpty(responses.getString(pks.getUserKeyPasswordProperty()))) {
        userSslPassword = encrypter.Decrypt(responses.getString(pks.getUserKeyPasswordProperty())).toCharArray(); <== Fails here
    }

Environment

VMware Cloud Director 10.6.1

Cause

This issue occurs when the deployment password supplied for the new standby cell node does not match the security credentials used during the initial setup of the database cluster, causing the decryption engine to fail when processing the cluster security keys.

Resolution

Take Proactive Virtual Machine Snapshots

• Capture a virtual machine snapshot of all active cells in the VMware Cloud Director server group and perform an embedded database backup before proceeding with the below steps.

Unregister the Failed Standby Node

• Remove and unregister the failed standby node entry from the active repmgr database cluster topology tracking using the unregister sequence detailed in How to remove a failed Standby Cell.

Clean Shared Transfer Folder

• Clean up residual appliance tracking records by purging the specific node-uuid subdirectory located within the cluster shared transfer folder path /opt/vmware/vcloud-director/data/transfer/appliance-nodes (Refer KB - How to remove a failed Standby Cell).

Retrieve Cluster Initialization Password

• Log into the primary cell command line interface and inspect the deployment values using the ovfenv utility or view the local runtime xml configuration metadata within the /opt/vmware/etc/vami/ovfEnv.xml file path.

Redeploy the Standby Appliance Node

• Redeploy the new standby cell appliance node into the high availability cluster context using the recovered initialization password parameters as documented in Configure Your VMware Cloud Director Standby and Application Cells

 

Additional Information

• How to remove a failed Standby Cell

• VAMI shows Cluster Health: Read Only Primary or Degraded state