vCenter KMS status Key State Red Other Status Green In vSAN Skyline Health

Products

• VMware vSAN

Issue/Introduction

Customer KMS server logs have errors related to incorrect or wrong username authenticating when attempting to establish trust from vCenter or hosts.

Existing vSAN cluster using Data-At-Rest Encryption and the vSAN Skyline Health check begin indicating vCenter and all hosts are connected to Key Management Servers alarm.
The only Red indicator under vCenter KMS status is Key State, the others are green.

Shallow rekey fails with "General vSAN error. There was an issue generating keys with KMS cluster #########"

Attempting to enable Encryption Mode (or crypto safe mode) on host fails with "A general runtime error occurred. Cannot generate key. CreateKey failed on key provider #########, error code: QLC_ERR_NO_BATCH_COUNT; Failed. Check log for details."

Environment

vSAN 7.x
vSAN 8.x

Cause

At some point the [email protected] username was entered into the username field for optional password authentication while editing the the KMS in vCenter. The entry is then in the vCenter database and cannot be cleared from the UI.

Resolution

Open a case with VMware by Broadcom Support to resolve this issue.