All VM's are offline and ESXii servers are unable to connect to the KMS Server in Azure to unlock vSAN

Products

• VMware vSAN
• VMware vCenter Server

Issue/Introduction

A vSAN encrypted host will not enter Encryption Mode. The operation fails with a QLC_ERR_VALUE_MISSING error.

vSAN diskgroups become locked after the ESXi host was rebooted

 

Logging into the vCenter appliance via SSH using the root password, and upon checking vpxd.log, the following error can be seen:

[Timestamp] info  vpxd[...] The Vecs string entry 'password-<UUID>/<kms1.example.com>' does not exist in VECS
[Timestamp] error vpxd[...] Failed to create key on KMS <IP>:5696 - Err:QLC_ERR_VALUE_MISSING Password

[Timestamp] info  vpxd[...] The Vecs string entry 'password-<UUID>/<kms2.example.com>' does not exist in VECS
[Timestamp] error vpxd[...] Failed to create key on KMS <IP>:5696 - Err:QLC_ERR_VALUE_MISSING Password

[Timestamp] warning vpxd[...] Failed to generate key on key provider <UUID>, error 7:
--> Reason:
--> Failed to generate key on KMS <IP1>: QLC_ERR_VALUE_MISSING
--> Failed to generate key on KMS <IP2>: QLC_ERR_VALUE_MISSING
--> Custom attributes: (null)

Environment

vSAN 8.x

Cause

The password for the Key Provider is either missing or incorrect in the VECS Store, which prevents the vCenter from authenticating with the KMS.

Resolution

1. Log in to the vCenter Server.
2. Navigate to: vCenter > Configure > Security > Key Providers
3. Edit the affected Key Provider's settings.
4. Re-enter and save the correct password.

If the optional password is unknown, engaged the KMS provider to assist with resetting the password.