Error: "A general runtime error occurred. Key <key_id> not found" during Cross-vCenter vMotion of Encrypted VMs using Standard Key provider

Products

• VMware vCenter Server

Issue/Introduction

When attempting to perform a cross-vCenter vMotion of encrypted virtual machines (VMs), the operation fails with a general runtime error. The failure prevents migration of VMs from a source vCenter Server to a destination vCenter Server.

Observed Error Message: A general runtime error occurred. Key <key_id> not found.

Additionally, the vpxd.log on the destination vCenter Server shows entries similar to:

key = "com.vmware.vim.vpxd.encryption.kmsClusterNotFound",
message = "Key provider <KMS Name> not found."

Cause

The issue occurs due to a mismatch in the Key Management Server (KMS) name configured on the source and destination vCenter Servers. Even a minor discrepancy, such as an extra space, special character, or case difference can prevent the destination vCenter from recognizing the encryption key provider used by the source.

In some scenarios, the KMS name on one vCenter included an extra character, causing the destination vCenter to be unable to locate the corresponding Key Provider during the vMotion compatibility check.

Resolution

To resolve this issue:

1. Take a snapshot or backup of the vCenter Server configuration to ensure recovery in case of issues. Use of offline snapshots in ELM deployments is very strongly recommended for a safe rollback point as mentioned in "VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice"

2. Remove the incorrectly named Key Provider from the vCenter Server, referring to Delete a Standard Key Provider

3. Re-add the Key Provider with the correct and consistent name, ensuring it matches the source vCenter configuration exactly, referring to Add a Standard Key Provider Using the vSphere Client

   • Ensure the KMS connection is established and trusted successfully.

4. Retry the cross-vCenter vMotion operation for the encrypted VM.

5. Re-Key the ESXi hosts by invoke API hostSystem.configureCryptoKey from the vCenter MOB, referring to the workaround section in article 311922

Additional Information

To validate which KMS server is being used by which host, run the following command from the vCenter server command line:

psql -d VCDB -U postgres -c "select dns_name,crypto_key_provider_id from vpx_host;"