Recovering Encrypted vSAN Datastores from KMS Circular Dependencies
search cancel

Recovering Encrypted vSAN Datastores from KMS Circular Dependencies

book

Article ID: 453198

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

VMware vSAN Data-at-Rest Encryption relies on an external Key Management Server (KMS) to supply the Key Encryption Keys (KEK) required to lock and unlock physical disk groups. This architecture ensures strict security by separating the cryptographic key material from the storage media.

A critical architectural flaw, known as a circular dependency, occurs when the KMS virtual machines are deployed directly onto the same encrypted vSAN datastore they are configured to manage. While the cluster is online and keys are cached in the ESXi host memory, this configuration appears functional. However, if the vSAN cluster experiences a full power outage or the hosts are rebooted, the hosts' memory cache is cleared.

Upon startup, the ESXi hosts cannot mount the encrypted vSAN disk groups without establishing a network connection to the KMS. Conversely, the KMS virtual machines cannot be powered on because the underlying vSAN datastore is locked. This creates a catastrophic "chicken-and-egg" deadlock, rendering the entire storage cluster and all resident virtual machines completely inaccessible.

This technical document outlines the emergency procedures required to safely break this deadlock. It details how to verify the cluster's cryptographic state, reintroduce the original encryption keys from outside the locked vSAN environment, and restore datastore access while strictly avoiding administrative actions that could result in permanent data loss.

Environment

VMware vSAN 8.x

VMware vSAN 9.x

Resolution

Critical Warnings & Broadcom Guidance

A "circular dependency" occurs when Key Management Server (KMS) virtual machines are stored on the same encrypted vSAN datastore they are responsible for unlocking. If the cluster goes offline, the hosts cannot reach the KMS to unlock the datastore, and the datastore cannot mount to boot the KMS.

To prevent permanent data loss and further complications, strictly adhere to the following Broadcom guidelines:

  1. Never locate the KMS on the vSAN datastore it unlocks.

  2. Do not reboot ESXi hosts if encryption keys are unavailable. Keys can remain cached in memory after a KMS outage; rebooting destroys this cache and removes the remaining ability to access encrypted data.

  3. Do not attempt to recreate disk groups or disable encryption to bypass the lock. Disabling encryption involves format conversion and requires the original keys.

  4. Permanent Data Loss: If you do not have external backups of the KMS appliances or their exported cryptographic keys, the data on the vSAN cluster is permanently inaccessible.


Phase 1: Verify Encryption State and Connectivity

1. Check the Disk Group Encryption State Run the following commands on a single ESXi host to determine if the disk groups are mounted or locked:

esxcli vsan encryption info get
esxcli vsan encryption hostkey get
esxcli vsan encryption kms list
esxcli vsan cluster get
esxcli vsan storage list
esxcli vsan debug disk list
esxcli storage filesystem list

2. Verify KMS Availability From an ESXi host, test if the KMS IPs are reachable via ICMP and the KMIP port (TCP 5696):

vmkping <KMS_IP_1>
vmkping <KMS_IP_2>

nc -zv <KMS_IP_1> 5696
nc -zv <KMS_IP_2> 5696


Phase 2: Recovery Options to Break the Deadlock

To recover the cluster, you do not necessarily need the original virtual machines, but you must have the original cryptographic key database. A brand-new KMS installation containing no original keys will not decrypt existing vSAN data.

You must provide the ESXi hosts access to the original keys from outside the locked vSAN environment using one of the following methods.

Option A: Restore the KMS to Non-Encrypted Storage (Preferred)

If you have a VM-level backup of your KMS appliances (e.g., CloudLink), restore them to an independent location.

  1. Locate an External Backup: Find your VM backups or exported key databases.

  2. Deploy to Independent Storage: Restore the KMS VMs to a non-encrypted standalone ESXi host, a management cluster, or a local VMFS datastore (e.g., a temporary NFS share / iSCSI volume).

  3. Restore Network Access: Ensure the restored KMS VMs are assigned their exact original IP addresses and that port 5696 is reachable from your locked vSAN hosts.

Option B: Deploy a Fresh KMS and Import the Key Database

If a VM-level restore is unavailable but you have a backup of the key database:

  1. Deploy a Fresh Appliance: Deploy a new KMS appliance OVA directly onto an ESXi host's local or external non-vSAN datastore using the ESXi Host Client.

  2. Match the Networking: Assign the new KMS VM the original IP address.

Import the Keys: Log into the new KMS interface and import your original encryption keys/database backup.

Option C: The last resort 

If recovering the KMS cluster/keys is not possible then the only option is to recrate the KMS cluster & Disk Groups and restore from backup


Phase 3: The Unlock Process

As soon as the externally hosted KMS is online, reachable on port 5696, and holding the correct cryptographic keys:

  1. The ESXi hosts will automatically connect and request the Key Encryption Key (KEK) and Host Key.

  2. The drives will automatically unlock and the vSAN datastore will mount.

  3. You can safely power on vCenter and your production workloads.

NOTE: If you need any assistance with KMS Recovery, you will have to submit a case with your KMS vendor.

Additional Information

DO NOT DO LIST

Do not recreate vSAN disk groups

That can destroy the remaining possibility of recovering the encrypted data.

Do not initialize new disks

Same reason.

Do not remove the KMS configuration

The ESXi hosts already know which KMS they need.

Do not configure a new KMS and assume it will decrypt the datastore

A new KMS doesn't automatically possess the old encryption keys.

Do not disable vSAN encryption

KMS keys are required to disable vSAN ecryption but this isn't a recovery mechanism for missing keys.

Do not repeatedly reboot ESXi hosts

Especially while the encryption state is unresolved. Broadcom explicitly warns against rebooting when KMS/key access is unavailable.

Do not perform destructive vSAN recovery operations

Until the encryption/key situation is understood.

Powered by Wolken Software