All VMs in all management domain hosts show invalid status with vSAN Data at Rest Encryption in use

Products

• VMware vSAN
• VMware vCenter Server

Issue/Introduction

vSAN Data at Rest Encryption is in use with the vSphere Native Key Provider as the KMS.

When connecting to the ESXi hosts web interface, you see all virtual machines showing a path to the VMs namespace path and the status is "invalid".

When connected to one of the hosts in the cluster, you also see zero available space for the vSAN datastore when you run the command, df.  

cat df.txt
Filesystem        Bytes        Used    Available Use% Mounted on
vsan                  0           0            0   0% /vmfs/volumes/vsanDatastore

 

Environment

ESXi 7.X
ESXi 8.X
vCenter 8.X

Cause

• The vSphere Native Key Provider (NKP) being used as KMS for vSAN Data at Rest (DAR) and the NKP is offline or its configuration is lost.
  • While the key provider is offline the vSAN disks cannot be decrypted or mounted in vSAN resulting in a data unavailable situation until the key provider in accessible again.
• All hosts in the cluster have been rebooted or had their vSAN disk group(s) unmounted.

Resolution

Restore access to the NKP. If it is no longer available, restore it from backup.

If the key provider cannot be recovered the disk groups must be recreated with either a new key provider or with encryption disabled, and all VM data must be restored from backup.

Additional Information

For other vSAN encryption issues see KB 326769 - Troubleshooting vSAN Encryption