Error: Unable to restore the VMware Cloud Director server configuration

Products

VMware Cloud Director

Issue/Introduction

When attempting to restore VMware Cloud Director it fails with the following error in the UI: Unable to restore the VMware Cloud Director sever configuration
In the /opt/vmware/var/log/vcd/restore.log you see entries similar to:

####-##-## ##:##:## | DEBUG | Invoking as root: /opt/vmware/appliance/bin/configure-vcd.sh restore false false
####-##-## ##:##:## | ERROR | Error running command: CompletedProcess(args='/opt/vmware/appliance/bin/configure-vcd.sh restore false false', returncode=20, stdout='Running in restore mode\nAppliance OS Phase has been Completed.\nStarting Appliance Cloud Director Configuration Phase.\nInvoking check-vcd-params script again.\nAll the required ovf parameters provided for the Cloud Director configuration phase\nInvoking nfs-setup script.\nMounting NFS file share ...\nInvoking setupvcd script.\nsetupvcd script failed to execute.\nFor more details, check /opt/vmware/var/log/vcd/configure-vcd.log.\n', stderr='')
####-##-## ##:##:## | ERROR | An error was encountered while restoring.
Traceback (most recent call last):
File "/opt/vmware/appliance/bin/api/restore.py", line 243, in restore
reconfigure_vcd(str(http_cert).lower(), str(pgmanagement_cert).lower())
File "/opt/vmware/appliance/bin/api/restore.py", line 168, in reconfigure_vcd
err = 'Unable to restore the VMware Cloud Director server configuration.'
File "/opt/vmware/appliance/bin/api/restore.py", line 50, in run
raise RestoreException(err)
RestoreException: Unable to restore the VMware Cloud Director server configuration.
####-##-## ##:##:## | INFO | An error occurred during configuration, running cleanup
In the /opt/vmware/vcloud-director/logs/configure-<date>.log you see entries similar to:

####-##-## ##:##:## | DEBUG | main | ConfigAgent | Configuring server certificates... |
####-##-## ##:##:## | INFO | main | ConfigAgent | HTTP SSL certificate entered: /opt/vmware/vcloud-director/data/transfer/certificates/http.pem |
####-##-## ##:##:## | INFO | main | ConfigAgent | HTTP SSL private key entered: /opt/vmware/vcloud-director/data/transfer/certificates/http.key |
####-##-## ##:##:## | ERROR | main | ConfigAgent | Cryptographic error: Password provided was incorrect for the supplied private key |
####-##-## ##:##:## | ERROR | main | ConfigAgent | Aborting configure due to OTHER (106) |
com.vmware.vcloud.configure.legacy.ConfigAgentException: Aborting configure due to OTHER (106)
Caused by: java.security.cert.CertificateException: Password provided was incorrect for the supplied private key
at com.vmware.vcloud.common.cmt.CertificateInputUtils.handleUserPasswordException(CertificateInputUtils.java:189)
at com.vmware.vcloud.common.cmt.CertificateInputUtils.promptAndLoadUserCertificates(CertificateInputUtils.java:158)
at com.vmware.vcloud.common.cmt.CertificateInputUtils.configureServerCertificates(CertificateInputUtils.java:99)
at com.vmware.vcloud.configure.legacy.ConfigAgent.configureCertificates(ConfigAgent.java:894)
at com.vmware.vcloud.configure.legacy.ConfigAgent.configureCell(ConfigAgent.java:632)
at com.vmware.vcloud.configure.legacy.ConfigAgent.start(ConfigAgent.java:424)
at com.vmware.vcloud.configure.legacy.ConfigAgentExecutor.start(ConfigAgentExecutor.java:83)
at com.vmware.vcloud.configure.legacy.ConfigAgentExecutor.main(ConfigAgentExecutor.java:78)
Caused by: java.security.spec.InvalidKeySpecException: org.bouncycastle.pkcs.PKCSException: unable to read encrypted data: Error finalising cipher
at com.vmware.vcloud.common.crypto.Crypt.parsePKCS8PrivateKey(Crypt.java:262)
at com.vmware.vcloud.common.cmt.CertificateInputUtils.promptAndLoadUserCertificates(CertificateInputUtils.java:154)
... 6 more
Caused by: org.bouncycastle.pkcs.PKCSException: unable to read encrypted data: Error finalising cipher
at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.decryptPrivateKeyInfo(Unknown Source)
at com.vmware.vcloud.common.crypto.Crypt.parseEncryptedPKCS8PrivateKey(Crypt.java:302)
at com.vmware.vcloud.common.crypto.Crypt.parsePKCS8PrivateKey(Crypt.java:252)
... 7 more
Caused by: org.bouncycastle.crypto.InvalidCipherTextException: Error finalising cipher
at org.bouncycastle.jcajce.io.CipherInputStream.finaliseCipher(Unknown Source)
at org.bouncycastle.jcajce.io.CipherInputStream.nextChunk(Unknown Source)
at org.bouncycastle.jcajce.io.CipherInputStream.read(Unknown Source)
at org.bouncycastle.util.io.Streams.pipeAll(Unknown Source)
at org.bouncycastle.util.io.Streams.readAll(Unknown Source)
... 10 more
Caused by: javax.crypto.BadPaddingException: Error finalising cipher data: pad block corrupted
at org.bouncycastle.jcajce.provider.BaseCipher.engineDoFinal(Unknown Source)
at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2085)
... 15 more

Environment

VMware Cloud Director 10.6.X
VMware Cloud Director 10.5.X

Cause

This issue is encountered when the user.certificate.path/user.key.path does not point to /opt/vmware/vcloud-director/etc/user.http.pem and user.http.key when restoring from backup in global.properties.

Resolution

This issue is resolved in 10.6.1.2. For the previous versions, please refer to the workaround mentioned below.

Workaround

To workaround the issue please perform the following steps:

Extract the files from the backup zip folder that was created when you take a backup of the VMware Cloud Director.
Take a backup of the following file in the backup folder: global.properties
Edit the original global.properties file in the backup folder by editing the following lines:

From:

user.certificate.path = /opt/vmware/vcloud-director/data/transfer/certificates/http.pem
user.key.path = /opt/vmware/vcloud-director/data/transfer/certificates/http.key

To:

user.certificate.path = /opt/vmware/vcloud-director/etc/user.http.pem
user.key.path = /opt/vmware/vcloud-director/etc/user.http.key
Save the changes and zip the folder again and attempt the restore again using the edited backup.

Additional Information

Backup and Restore of Your VMware Cloud Director Appliance