vMotion Fails for Encrypted Virtual Machines with KMS Item Not Found Error

Products

• VMware vCenter Server

Issue/Introduction

vMotion migration operations fail for encrypted virtual machines. The vCenter Server is unable to retrieve the designated cryptographic key necessary to facilitate the transfer of the encrypted virtual machine to the target host.

The var/log/vmware/vpxd/vpxd.log on the vCenter Server contains errors similar to the following indicating a Key Management Server (KMS) retrieval failure:

info vpxd[06039] [Originator@6876 sub=CryptoManager opID=CdrsLoadBalancer-2f996dfc-72d84932-01-01] Key info for VM <REDACTED_HOSTNAMES> before migrate operation: 
-->   New Keys: {########-####-####-####-##########}/<REDACTED_SECRETS> 
-->   VM Keys : {########-####-####-####-##########}/<REDACTED_SECRETS> 
info vpxd[06039] [Originator@6876 sub=CryptoManager opID=CdrsLoadBalancer-2f996dfc-72d84932-01-01] Sending VM standard keys {012184F9-A26B-00A3-00C4-BD751ED2E2A1}/<REDACTED_SECRETS> to [vim.HostSystem:host-1014,<REDACTED_HOSTNAMES>] for VM <REDACTED_HOSTNAMES> before migrate operation. 
error vpxd[06039] [Originator@6876 sub=CryptoManagerKmipWrapper opID=CdrsLoadBalancer-2f996dfc-72d84932-01-01] Get, Object, Id=%s failed on key {012184F9-A26B-00A3-00C4-BD751ED2E2A1} on KMS <REDACTED_HOSTNAMES>:1028 - Server Error:Item Not Found, Explanation:Item {012184F9-A26B-00A3-00C4-BD751ED2E2A1} not found. 
--> 
warning vpxd[06039] [Originator@6876 sub=Default opID=CdrsLoadBalancer-2f996dfc-72d84932-01-01] Failed to get key {012184F9-A26B-00A3-00C4-BD751ED2E2A1} on key provider <REDACTED_SECRETS>, error 7: 
--> Reason: 
--> Failed to get key {########-####-####-####-##########} on KMS <REDACTED_HOSTNAMES>: QLC_ERR_GENERAL_ERROR

Environment

vCenter Server 8.0.3
vSphere ESXi 8.0.3

Cause

An external misconfiguration on the backend Key Management Server (KMS) appliance cluster, coupled with site-to-site infrastructure connectivity degradation, disrupts cryptographic key synchronization and replication across the HSM nodes (A physical or virtual cryptographic appliance). This prevents the managing vCenter instance from retrieving or validating the specific key ID linked to the virtual machine's encryption state.

Resolution

 

• Resolve any underlying cross-site network routing or interconnect constraints to restore connectivity between the cross-site HSM peers.

• Verify the external KMS server backend connectivity to ensure successful cryptographic key synchronization and replication.

• Validate that all cluster hosts are uniformly aligned with the correct key provider ID mapping by executing the following command on the vCenter Server via SSH:

  To connect to the embedded vPostgres database (VCDB) on a vCenter Server Appliance, you must first open an SSH session as root and type shell to switch to the BASH environment. Once in the BASH shell, execute the following command: 

  /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

  To validate which KMS server is being used by which host, run the following command :

  select dns_name,crypto_key_provider_id from vpx_host; 

• Validate the  Network Connectivity Check (via vCenter SSH) : To verify that vCenter has an open network path to the KMS server (bypassing any UI caching), you can use the exact same netcat tool you were using before, but execute it from the vCenter appliance.

  1. SSH into your vCenter Server Appliance.

  2. Run the following command:

     nc -zv <KMS_IP> 1028
     

  If this returns succeeded or Connected, your vCenter has the required physical network access to the KMS.

• Retry the vMotion operation for the encrypted virtual machines.

 

Additional Information

Virtual Machine Encryption Interoperability