Verifying vSAN Data-at-rest encryption functionality

Products

• VMware vSAN

Issue/Introduction

Users may see alerts in vSAN Skyline Health for "vCenter and all hosts are connected to Key Management Servers". In some cases, users may not have easy access to logs, or may not be able to export logs to support due to the environment being "air-gapped" (no outside access, and inability to share details due to security reasons). In such cases, users need a way to verify whether or not vSAN Data-at-rest encryption is functioning.

Environment

VMware vSAN 8
VMware vSAN 9

Resolution

Provided vSAN Data-at-rest encryption shows as enabled, the below steps can be followed to verify that ESXi is receiving encryption/decryption keys from KMS, and that it is able to use them to encrypt/decrypt VMs.

1. Pick one host and evacuate all VMs off of it.
2. Create a test VM on that host, using the vSAN datastore. If this succeeds, then we know the host is able to create and encrypt using its current key.
3. Power on the VM. If this succeeds, we know the host can decrypt and open the VM using its current key.
4. Reboot the host. When the host comes back up, it will reach out for the current key from the KMS (in this case NKP).
5. Power on the test VM. If this succeeds, then we know that even after rebooting and getting a fresh key, it can still decrypt and open the VM.

The above steps would need to be tested on each ESXi host in the vSAN cluster. If any of these steps fails, please open a ticket with Broadcom support. Collect and provide as much information as possible about which step failed, and the error message received.

Additional Information

The below commands can be run from ESXi to collect additional information about the host's encryption config and connectivity to KMS.

Retrieve vSAN encryption information:
esxcli vsan encryption info get

Retrieve KMS configurations for vSAN encryption:
esxcli vsan encryption kms list

Retrieve host key from the keycache:
esxcli vsan encryption hostkey get

Retrieve encryption certificate file paths on the ESXi hosts:
esxcli vsan encryption cert path list

Retrieve KMS server certificate contents from the ESXi host (similar to 'cat /etc/vmware/ssl/vsan_kms_castore.pem'):
esxcli vsan encryption cert get