All VMs "Invalid" on vSAN Encrypted datastore

Products

• VMware vSAN

Issue/Introduction

• After planned power-outage, all VMs are showing as "Invalid"
• vSAN Datastore is 0 in size
• vSAN Encrytpion is enabled 
• vCenter VM resides on the vSAN datastore
• KMS cluster is in use
• KMS cluster can be successfully pinged

Environment

VMware vSAN (All Versions)

Cause

This is caused due to the vSAN disks not being able to decrypt themselves due to communication issues to the KMS cluster resulting in not being able to validate the KEKs with the KMS cluster.

From vmkernel you see:

2025-04-06T20:56:02.376Z In(182) vmkernel: cpu0:2131939 opID=195ee45e)KeyCache: 924: Trying to resolve key for keyId 5######5-####-####-####-d##########e

2025-04-06T20:56:02.376Z In(182) vmkernel: cpu0:2131939 opID=195ee45e)KeyCache: 860: Request to populate 5######5-####-####-####-d##########e in keycache

2025-04-06T20:56:02.376Z In(182) vmkernel: cpu0:2131939 opID=195ee45e)KeyCache: 880: RPC_SendAndGetReply() reply result: Not found

2025-04-06T20:56:02.376Z In(182) vmkernel: cpu0:2131939 opID=195ee45e)KeyCache: 930: Did not find keyID 5######5-####-####-####-d##########e in key cache

Resolution

Engage your KMS administrator/vendor for further investigation.

Additional Information

Troubleshooting vSAN Encryption