Fixing the KMS connectivity issues in "vCenter - Configure - Key Providers"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• When cloning a virtual machine or creating a new virtual machine, the "Select Storage" pane in the wizard gets stuck at "Validating" and fails with timeout.
• When right click the virtual machine and click "Edit Settings", the page will stuck for approximately 2 minutes before being able to display properly.

 

Environment

VMware vSphere 6.7.x

Vmware vSphere 7.0.x

Cause

In the "Select Storage" page, the UI is trying to get the default key provider and its health state. When the key provider has KMS with connectivity issues, the health check will timeout.

Similarly, when trying to edit virtual machine settings, UI also needs to contact key provider. If there's issue with connection to key provider, the UI will not load until the connection to KMS finally times out after approximately 2 minutes.

 

Below vpxd logs can be referred to to identify the issue:

YYYY-MM-DDTHH:MM:SS.903+08:00 error vpxd[06271] [Originator@6876 sub=CryptoManagerKmipWrapper opID=q-231998:h5ui-getProperties:urn:vmomi:Folder:group-d1:BBBA834E-7D0B-4714-8175-DC52CDA3004E:1099917763:VCenterKmipPropertyProvider:1109211-a0q9-h5:70161015-86-WorkQueue-38654da0] Failed to connect to KMS <KMS ip address>:5696 - Err:QLC_ERR_COMMUNICATE Failed to establish TCP connection to server

-->

YYYY-MM-DDTHH:MM:SS.903+08:00 error vpxd[06271] [Originator@6876 sub=CryptoManager opID=q-231998:h5ui-getProperties:urn:vmomi:Folder:group-d1:BBBA834E-7D0B-4714-8175-DC52CDA3004E:1099917763:VCenterKmipPropertyProvider:1109211-a0q9-h5:70161015-86-WorkQueue-38654da0] Cannot accept empty cert

YYYY-MM-DDTHH:MM:SS.903+08:00 warning vpxd[06271] [Originator@6876 sub=VpxProfiler opID=q-231998:h5ui-getProperties:urn:vmomi:Folder:group-d1:BBBA834E-7D0B-4714-8175-DC52CDA3004E:1099917763:VCenterKmipPropertyProvider:1109211-a0q9-h5:70161015-86-WorkQueue-38654da0] WorkQueue [TotalTime] took 120147 ms

Resolution

To resolve the issue, we need to either fix the KMS connectivity issues with the following steps or delete KMS settings if key provider is not in use.

1. Navigate to vCenter->Configure->Key Providers.
2. Click the problematic key provider. The list will be in "Retrieving" status for 2 minutes. If the user fails to establish a TCP connection with the KMS, then connection status will populate with "error". In this case, check the KMS first.