ESXi hosts fail to enter Crypto Safe Mode due to Key Retrieval failures

Products

• VMware vSAN

Issue/Introduction

ESXi hosts are unable to enter Crypto Safe Mode, which prevents the unlocking of encrypted disks or the creation of new vSAN disk groups. Even after manual intervention, such as removing existing disk groups, the system fails with the following error: "Manually recover the missing key to the key provider." This typically occurs when the host can reach the Key Management Server (KMS) but is blocked from pulling the key by the underlying Hardware Security Module (HSM).

Environment

• Product: VMware ESXi / vSAN

• Version: All Versions. 

• KMS Vendor: HyTrust with HSM integration

Cause

A failure occurred within the HSM for the HyTrust KMS that prevents ESXi hosts from successfully retrieving cryptographic keys despite having network connectivity to the provider.

Resolution

1. Validate KMS Connectivity: Verify that the connection status between vCenter/ESXi and the Key Provider is "Normal" in the vSphere Client.
   
   Troubleshooting vSAN Encryption 

2. Verify HSM Health: Contact the HyTrust or Security administration team to verify the health and status of the HSM.

3. Correct External Blockers: Ensure the ESXi hosts are correctly authorized within the HyTrust environment to request and pull the specific keys associated with the cluster.

4. Retry Crypto Mode: Once the external environment issues are resolved, the impacted hosts  will enter Crypto Safe Mode and any vSAN Disk-groups should mount.

5. Verify Disk Access: Confirm that disk groups can be created and that previously locked encrypted disks are now accessible.

If you have any additional questions, or concerns regarding this issue please open a case with Broadcom Support for further investigation.