vMotion failing for an encrypted VM with Error: "A general runtime error occurred. Key ##### not found"

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

In the environment where:

• External KMS in use (Standard KMS server)
• vTPM / TPM 2.0 in use
• Key provider is in healthy state

• vCenter Server is reading the correct ESXi host keys:

To validate ESXi host key, you can execute below commands on vCenter server:

     Connect to the VCDB via the postgres shell. Refer Interacting with the vCenter Server Appliance 6.5/6.7/7.0/8.0 embedded vPostgres Database

    psql -U postgres -d VCDB -h localhost

    Identify the information for the host in question

    select id, dns_name, crypto_state, crypto_key_id, crypto_key_provider_id, crypto_enable from vpx_host where dns_name like '%<esxi-host-fqdn>%';

• Though, vMotion fails for an encrypted VM with Error: "A general runtime error occurred. Key ##### not found"

• Encryption Keys might recently reset on (source or Destination) ESXi host(s):

• The correct key is not available on the ESXi host.
• vCenter Server cannot retrieve keys from the key server.
• Error persist even after rebooting ESXi host

 

Environment

VMware ESXi 8.x

VMware ESXi 7.x

VMware vCenter Server 8.x

VMware vCenter Server 7.x

Cause

The KMS user for vCenter is not the owner of some of the ESXi keys. Hence vCenter Server cannot retrieve keys from the key server.

Resolution

If the key is no longer available on the key server, vCenter Server generates a virtual machine alarm. Also, an error message appears in the event log.

• Ask the key server (KMS) administrator to restore the key.
• Retrieve the key ID by using the Managed Object Browser (MOB) or the vSphere API.
• Retrieve the keyId from VirtualMachine.config.keyId.keyId.

 

 

• Ask the key server (KMS) administrator to reactivate the key that is associated with that key ID.
• After restoring the key, see Unlock Locked Virtual Machines.
• If the key can be restored on the key server, vCenter Server retrieves it and pushes it to the ESXi host the next time it is needed.

Additional Information

Resolve Missing Encryption Key Issues