"Database temporarily unavailable or has Network problems" while establishing trust with KMS Certificate

Products

• VMware vCenter Server
• VMware vSAN

Issue/Introduction

 

• KMS nodes are showing as "Not Connected (Trust not established)" and when attempt to "Make vCenter Trust KMS", the process fails with error "Database temporarily unavailable or has Network problems"
• Logs similar to below entries will be logged in /var/log/vmware/vpxd/vpxd.log  

YYYY-MM-DDTTT:HH:MM.SSZ error vpxd[42757] [Originator@6876 sub=CryptoManagerKmipWrapper opID=1e425428-SWI-7f0bf07c] Failed to connect to key server, QLC_ERR_NEED_AUTH
YYYY-MM-DDTTT:HH:MM.SSZ warning vpxd[42757] [Originator@6876 sub=CryptoManager opID=1e425428-SWI-7f0bf07c] DiscoverVersions failed: QLC_ERR_NEED_AUTH

• Communication between vCenter Server and KMS Server works fine using curl command

curl -v telnet://<KMS-Server-IP>:5696   >>>>>>5696 is default port

• KMS Provider logs indicate an unknown authority. 

Environment

VMware vSAN 7.x
VMware vSAN 8.x
VMware vCenter Server Appliance 7.x
VMware vCenter Server Appliance 8.x

Cause

• Stale certificates (different serial number) for KMS nodes in VECS store KMS_ENCRYPTION.
• Certificate Authority (CA) trust issues on the KMS nodes themselves. 

Resolution

If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases. 

1. Reboot the vCenter VM to verify if the reboot fixed the issue or

2. Remove the KMS key provider and re-add and verify  or

3. If the issue is identified as a KMS provider CA trust-related issue, your KMS provider's support team will need to be engaged to ensure that the CA is trusted by their nodes.

 

Additional Information

Refer the Vendor Site for Troubleshooting key generation error during encryption Troubleshooting Key Generation Error During Encryption