vSAN Encryption Health Alarm: Key State Missing in KMS

Products

• VMware vSAN

Issue/Introduction

In a vSAN environment utilizing Data-at-Rest Encryption, the Skyline Health check reports a red status for vCenter and all hosts are connected to Key Management Servers.

• vSAN Health service logs (var/log/vmware/vsan-health/vmware-vsan-health-service.log). show warnings similar to: WARNING vsan-mgmt [VsanHealthEncUtil::_AggregateEncryptionConfigHealth] Key state {'<UUID>': 'KeyStateMissingInKMS'} is red
• vSAN Health service logs (var/log/vmware/vsan-health/vmware-vsan-health-service.log) Queries to the KMS provider return errors similar to: ERROR [VsanVcEncryption::GetKmipKeyAttributes] Error reason: Server Error:General Failure, Explanation:[NCERRResourceNotFound: Resource not found]

Environment

• VMware vSAN (all versions)

Cause

This issue occurs when the Key Encryption Key (KEK) IDs registered in the vCenter/vSAN configuration no longer exist in the active Key Management Server (KMS) vault.

This is typically seen after:

• A departmental separation or migration where legacy KMS servers were decommissioned.
• Accidental deletion of keys from the KMS database.
• Re-establishing trust with a new KMS that does not contain the historical keys.

Resolution

If all vSAN disk groups remain mounted and operational (meaning the Disk Encryption Keys are still in the host memory), you can resolve the health alarm by generating a new KEK.

1. Confirm that the trust relationship between vCenter Server, ESXi hosts, and the active KMS provider is healthy.
2. Perform a Shallow Rekey operation on the vSAN cluster:
   • Navigate to the vSAN Cluster > Configure > vSAN > Services.
   • Under Data-At-Rest Encryption, click EDIT.
   • Select RE-KEY and ensure Shallow Rekey is selected.
3. Re-test vSAN Encryption Health to verify all health checks return to green.

Note: A Shallow Rekey generates a new KEK on the active KMS server and re-wraps the existing Disk Encryption Keys (DEKs) without re-encrypting the data on disk, making it a non-disruptive operation.

Additional Information

For further troubleshooting of KMS connectivity, see Troubleshooting vSAN Encryption - KMS

To speak with a customer representative or a Support Engineer, Scroll to the bottom of the page and click on your respective region.