All VMs in all management domain hosts show invalid status with vSAN Data at Rest Encryption in use

Products

• VMware vSAN
• VMware vCenter Server

Issue/Introduction

vSAN Data at Rest Encryption is in use with the vSphere Native Key Provider  or External KMS in use.

When connecting to the ESXi hosts web interface, you see all virtual machines showing a path to the VMs namespace path and the status is "invalid".

• All virtual machines on a vSAN encrypted datastore show an "invalid" status.
• The vSAN datastore capacity reports as 0 bytes used/available.
• Errors persist after reboots when using either vSphere Native Key Provider (NKP) or an External KMIP-compliant KMS.

Environment

ESXi 7.X
ESXi 8.X
vCenter 8.X

Cause

• The root cause is the permanent loss of the Key Encryption Key (KEK) or Host Keys in the KMS infrastructure.
• This occurs if the keys are deleted from the KMS database, the KMS configuration is lost, or no valid KMS backup is available to restore the keys used by the cluster.

Resolution

1. Verify Key Presence: Confirm with the KMS administrator that the specific Key IDs (found in ESXi esxcli vsan encryption info get) still exist on the server.
2. Validate Trust: Ensure client certificates and CA certificates are valid and the trust relationship between vCenter and the External KMS is active.
3. Data Recovery: If the keys are unrecoverable, the only path to restoration is to recreate the vSAN disk groups (effectively wiping the encrypted data) and restoring all virtual machines from external backups.

Additional Information

For other vSAN encryption issues see KB 326769 - Troubleshooting vSAN Encryption