KMS health alarm under vSAN health in vCenter

Products

• VMware vSAN
• VMware SDDC Manager
• VMware vCenter Server

Issue/Introduction

The message "Key Management Server Health Status Alarm" appears under vSAN health in the vCenter.

This can occur following a re-key of the vSAN cluster. It has been show to happen with a shallow re-key but is possible with any re-key.

The following error appears in the /var/run/log/vsansystem.log on one of the cluster hosts:

YYYY-MM-DDTHH:MM:SS.SSSZ Er(###) vsansystem[#######]: [vSAN@6876 sub=Libs opId=xxx-#######-hhh-X#######-hhhh] VsanUtil: Activate, Id=%s failed on key xxxxxc5d792c4e6e8377b63da3f3hhhhhhhhhhh31d89455780ce89abchhhhhhh, Server Error:Permission Denied, Explanation:[NCERRInvalidParamValue]: Can not change activation date unless key is in Pre-Active state.

The message will be in the first host to attempt to validate the key in question (NOTE: "key" above is nonsensical for privacy purposes). So the vsansystem.log on *each* host in the cluster must be checked to validate or invalidate the issue.

Cause

The above message is the Crypto Manager on refusing to change the activation date on the key as it is holding an incorrect state for the key. The work around of manually activating the key on the KMS will resolve the VC error message and activate the key but the issue can recur as the CM's in the KMS cluster will be out-of-sync on the key state unless the activation happens manually.

Resolution

This can occur if one or more CM's in the KMS cluster are out of time sync with the rest of the KMS cluster. Confirm NTP settings on all the CM's in the KMS.