Cross vCenter migraton fails with Error "Unable to decrypt the ciphertext" when Using Native Key Provider (NKP)

Products

• VMware vCenter Server

Issue/Introduction

• Cross-vCenter vMotion or clone operations for encrypted VMs (including those with vTPM) may fail at 100% with an error that it was unable to decrypt the key:

Error message: A general system error occurred: Unable to decrypt the ciphertext. Failed to decrypt the key '[KEY_ID]'.

• When viewing the VPXD logs on the vCenter Server (/var/log/vmware/vpxd/vpxd.log), the following errors are seen:

  YYYY-MM-DDTHh:MM:SS info vpxd[PID] [Originator@6876 sub=CryptoManager opID=<>] Key info for VM <VM Name> after xvc-migrate operation:
  -->   Added Keys :
  -->   Old VM Keys: <KEY_ID>
  -->   New VM Keys:
  -->   Fault      : (vmodl.fault.SystemError) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    faultMessage = (vmodl.LocalizableMessage) [
  -->       (vmodl.LocalizableMessage) {
  -->          key = "com.vmware.esx.kms.key_operation.decrypt.error",
  -->          arg = (vmodl.KeyAnyValue) [
  -->             (vmodl.KeyAnyValue) {
  -->                key = "arg",
  -->                value =
  
  message = "Failed to decrypt the key
  
  -->       (vmodl.LocalizableMessage) {
  -->          key = "com.vmware.esx.kms.keyoperationsvc.decrypt.error",
  -->          arg = <unset>,
  -->          message = "Unable to decrypt the ciphertext."
  -->       }
  -->    ],
  -->    reason = ""
  -->    msg = "A general system error occurred: "

 

For a clone operation, the above error message will begin with the following:

YYYY-MM-DDTHh:MM:SS info vpxd[PID] [Originator@6876 sub=CryptoManager opID=<>] Key info for VM <VM Name> after xvc-clone operation:

Environment

vCenter Server 8.x

vCenter Server 9.x

Cause

This error occurs when the destination vCenter does not have a matching Native Key Provider (NKP) or when the Key ID on the destination vCenter does not match the source vCenter.

Decryption is not possible unless the exact NKP (including the unique Key ID and key material) is backed up from the source vCenter then restored to the destination vCenter.

Resolution

In order to resolve the issue, proceed to add the Native Key Provider (NKP) of source vCenter Server to the destination vCenter Server

1. Export the NKP backup from the source vCenter. Refer to Back up a vSphere Native Key Provider

Note: Ensure the destination vCenter has no existing NKP configured. If an incorrect NKP was previously restored or created with the same name, it must be removed.

2. Restore the NKP to the destination vCenter Server. Refer to Recovering a vSphere Native Key Provider
   
   
3. Re-attempt the Cross-vCenter vMotion.