Error: kmsClusterNotFound - ESXi host encryption mode disabled after reboot in vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

After an ESXi host reboots or undergoes maintenance, the host encryption mode may show as Incapable or Disabled in vCenter Server. This occurs even if a new, healthy Key Provider is active and set as the default.

• vCenter reports an alarm: Host requires encryption mode enabled.
• Virtual Machine migrations (vMotion) fail for encrypted VMs.
• In /var/log/vmware/vpxd/vpxd.log, the following errors appear:
  com.vmware.vim.vpxd.encryption.kmsClusterNotFound
Failed to enable encryption: Fault cause: vmodl.fault.NotSupported
Sent notification to host for host key error, event type: com.vmware.vc.host.Crypto.ReqEnable.KMSClusterError 

 

• You may also encounter the following traces in /var/log/vmware/vpxd/vpxd.log:

  key = "com.vmware.esx.kms.key_operation.decrypt.error", message = "Failed to decrypt the key ... Unable to decrypt the ciphertext." method: configureCryptoKey; code: 500(Internal Server Error); fault: (vmodl.fault.SystemError) com.vmware.vc.host.Crypto.ReqEnable.KeyMissingOnKMS

Environment

VMware vCenter Server 8.x
VMware vSphere ESXi 8.x
External Key Management Server (KMS)

Cause

This issue occurs when a previously used Key Provider was deleted from vCenter, but one or more ESXi hosts still have host keys bound to that deleted provider's ID. During a reboot or reconnection, vCenter attempts to automatically sync the host key using the stale provider metadata. When the lookup fails, the host is placed in an "incapable" state.

Resolution

To permanently resolve this, you must rebind the affected ESXi hosts to a currently valid Key Provider.

1. Identify your active Key Provider in the vSphere Client (Configure > Security > Key Providers).
2. Set the new provider as the Default 
3. Access the vCenter Managed Object Browser (MOB): https://####-vcenter-fqdn/mob/?moid=host-####&method=configureCryptoKey
   • Note: Replace ####-vcenter-fqdn with your vCenter address and host-#### with the specific host ID.
4. In the keyProvider parameter, enter the exact name of your active Key Provider.
5. Click Invoke Method.
6. Return to the vSphere Client and verify the Host Encryption Mode returns to Safe.

Additional Information

For defects and enhancements, subscribe to this article to be updated on fix status: How to subscribe to Broadcom KB articles.

For instructions on how to retrieve logs for your environment, refer to the Broadcom Log Collection KB.