KMS Trust Fails with "Database temporarily unavailable or has Network problems" / 16-Certificate Limit Reached

Products

• VMware vCenter Server

Issue/Introduction

• KMS nodes show as "Not Connected (Trust not established)".
• Clicking "Make vCenter Trust KMS" fails with error: "Database temporarily unavailable or has Network problems".
• In /var/log/vmware/vpxd/vpxd.log, the following errors appear:
  YYYY-MM-DDTHH:MM:SSZ error vpxd[06614] [Originator@6876 sub=CryptoManager opID=#######-auto-2qz-h5:#####-3e] Failed to add more trusted certificate to cluster trustedCert. A cluster can configure at most 16 trusted certificates.
YYYY-MM-DDTHH:MM:SSZ error vpxd[06614] [Originator@6876 sub=Default opID=#######-auto-2qz-h5:#####-3e] [VpxLRO] -- ERROR lro-30590 -- ##############(##########-721b) -- CryptoManager -- vim.encryption.CryptoManagerKmip.uploadKmipServerCert:
:vim.fault.DatabaseError
--> Result:

  --> (vim.fault.DatabaseError) {

  -->    faultCause = (vmodl.MethodFault) null,

  -->    faultMessage = <unset>

  -->    msg = ""

  YYYY-MM-DDTHH:MM:SSZ error vpxd[06898] [Originator@6876 sub=CryptoManagerKmipWrapper] Failed to connect to KMS <KMS node>:5697 - Err:QLC_ERR_NEED_AUTH Failed to establish the connection, authorisation needed

• KMS store shows 16 entries
  /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store KMS_ENCRYPTION | grep "trustedCert" | wc -l
16

Environment

VMware vCenter Server 8.x

Cause

The vCenter VMware Endpoint Certificate Store (VECS) has an architectural limit of 16 certificates in KMS_ENCRYPTION store. During certificate rotations or repeated trust attempts, stale certificate aliases are retained in the KMS_ENCRYPTION store. Once the 16-slot limit is reached, vCenter cannot add new certificates, resulting in a database error in the UI.

Resolution

To resolve this issue, you must manually clear the database lockout by deleting the stale certificate entries:

• Take an snapshot of the vCenter Server. Refer Snapshot Best practices for vCenter Server Virtual Machines
• Log in to the vCenter Server Appliance via SSH.
• Export the current certificates to identify stale entries
  
  /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store KMS_ENCRYPTION --text > /tmp/KMS_ENCRYPTION.txt


• Locate the specific KMS node and note its alias.
• Back up the stale certificate:
  
  /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store KMS_ENCRYPTION --alias <alias-name> --output /storage/core/<alias-name>.crt


• Delete the stale certificate entry:
  
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store KMS_ENCRYPTION --alias <alias-name>


• Re-attempt to "Make vCenter Trust KMS" from the vSphere Client.