vSAN Encrypted Disk Groups Locked Due to Incorrect Default Key Provider Configuration

Products

VMware vSAN

Issue/Introduction

After reinstalling an ESXi host in a vSAN cluster, disk groups remain unmounted in a locked state. Manual attempts to mount disk groups fail, and host encryption configuration returns runtime errors indicating key provider communication failures.

Symptoms:

Disk group on Host remained in a Locked state.

Host State: Host Encryption Mode set to Disabled

Disk groups status displays:

"Disk encrypted and locked due to the encryption key is not available."

Mounting the disk group fails with error:

 "A general system error occurred: Failed to load dek for encrypted disk on encryption off host."

Interacting with key providers returns:

 "A general runtime error occurred. Key provider Built-In is not compatible with the host <host_fqdn>. Reason: TPM2 device is required."

Enabling host encryption fails with:

"A general runtime error occurred. Cannot generate key. CreateKey failed on key provider Entrust KeyControl, error code:QLC_ERR_COMMUNICATE; Failed."

Environment

VMware vSAN 8.x

VMware vCenter 8.x

Cause

The root cause is a vCenter Server misconfiguration where an inactive or unreachable external Key Management Interoperability Protocol (KMIP) provider is set as the "Default Key Provider."

Following the reinstallation of an ESXi host, vCenter Server attempts to query the "Default" key provider to generate and retrieve the Host Key. If the "Default" provider is an unreachable external KMIP service rather than the active Native Key Provider, key retrieval fails. This leaves the ESXi host incapable of unlocking the encrypted vSAN disks, resulting in communication errors (such as QLC_ERR_COMMUNICATE) in the /var/log/vmware/vpxd/vpxd.log.

YYYY-MM-DDTHH:MM:SS.MS+HH:MM error vpxd[09314] [Originator@6876 sub=CryptoManager opID=SWI-7#######] Failed to enable encryption on [[FQDN]:host-[MASKED MOID],[FQDN]]: N5Vmomi12RuntimeFault9ExceptionE(Fault cause: [FQDN]
--> )
--> [context][MASKED CONTEXT][/context]
YYYY-MM-DDTHH:MM:SS.MS+HH:MM info vpxd[09170] [Originator@6876 sub=vpxLro opID=q-261##################################################] [VpxLRO] -- BEGIN lro-648194123 -- ResourceModel -- [FQDN] -- ########-####-####-####-############(########-####-####-####-############)
YYYY-MM-DDTHH:MM:SS.MS+HH:MM info vpxd[08687] [Originator@6876 sub=vpxLro opID=2ef64###] [VpxLRO] -- BEGIN lro-648194124 -- session[########-####-####-####-############]########-####-####-####-############ -- [FQDN] -- ########-####-####-####-############(########-####-####-####-############)
YYYY-MM-DDTHH:MM:SS.MS+HH:MM info vpxd[08687] [Originator@6876 sub=vpxLro opID=2ef64###] [VpxLRO] -- FINISH lro-648194124
YYYY-MM-DDTHH:MM:SS.MS+HH:MM error vpxd[09314] [Originator@6876 sub=CryptoManager opID=SWI-7#######] Failed to enable encryption on [[FQDN]:host-[MASKED MOID],[FQDN]]: N5Vmomi5Fault12NotSupported9ExceptionE(Fault cause: [FQDN]
--> )
--> ___PLCHLDR___11___.
YYYY-MM-DDTHH:MM:SS.MS+HH:MM error vpxd[09314] [Originator@6876 sub=CryptoManager opID=SWI-7#######] Failed to recover [[FQDN]:host-[MASKED MOID],[FQDN]] crypto state from "incapable" to "safe" in [[FQDN]:domain-[MOID],Missile].

YYYY-MM-DDTHH:MM:SS.MS+HH:MM error vpxd[09314] [Originator@6876 sub=CryptoManagerKmipWrapper opID=SWI-7#######] Failed to connect to key server ##.##.##.#:5696 - Err:QLC_ERR_COMMUNICATE Failed to establish the connection: Q_ERROR_FAILED
-->

Resolution

Prerequisites:

Take a snapshot or backup of the vCenter Server instance before proceeding.

Steps:

Log in to the vSphere Client.
Select the vCenter Server object in the inventory tree.
Click Configure > Key Providers.
Select the active Native Key Provider from the list.
Click Set as Default.
Verify that the Native Key Provider is marked as Default.
Monitor the host status until it transitions into Crypto Safe mode.
Verify that vSAN disk groups mount automatically and return to a healthy state.

If the issue persists, contact Broadcom Support by following the steps outlined in Contact Broadcom Support.

Additional Information

Ensure that any decommissioned or inactive KMS providers are removed from the vCenter configuration to prevent future selection as the default.