"Unable to fetch key provider details on host.get provider info error, please check the health logs." in vSAN skyline health reports warning under Cluster configuration consistency

Products

• VMware vSphere ESXi
• VMware vSAN 7.x

Issue/Introduction

• When the user session count reached maximum on the vCenter Server, the warnings would appear "Unable to fetch key provider details on host.get provider info error, please check the health logs."
• vCenter would report events "User session count is limited to 550. Existing session count is 550 for user"
• The vmware-vsan-health-summary-result.log vSAN-Health logs would report the below events. Log location: /var/log/vmware/vsan-health/vmware-vsan-health-summary-result.log

 

$ vmware-vsan-health-summary-result.log:

[YYYY-MM-DDTHH:MM:SS] INFO vsan-mgmt [17356] [VsanHealthSummary LogUtil::PrintHealthResult opID=noOpId] Cluster <**Cluster name**> Overall Health yellow

Test consistentconfig health : yellow

   Issues: Host Disk Issue Recommendation

(Host-4****, '', UnableToFetchKeyProviderDetailsOnHost.GetProvider InfoError, PleaseCheckTheHealthLogs., ''), 

(Host-4****, '', UnableToFetchKeyProviderDetailsOnHost.GetProvider InfoError, PleaseCheckTheHealthLogs., ''), 

(Host-4****, '', UnableToFetchKeyProviderDetailsOnHost.GetProvider InfoError, PleaseCheckTheHealthLogs., ''), 

(Host-4****, ''' UnableToFetchKeyProviderDetailsOnHost.GetProvider InfoError, PleaseCheckTheHealthLogs., ''),

 

Group encryption health : unknown

  Test kmsconnection health skipped

HostsKmsStatus: Hosts KmsCluster KmsAlias ConnectionStatus KeyState Issue Recommendation

(Host-4****, vCenter, '', Yellow, Yellow, UnableToFetchKeyProviderDetailsOnHost.GetProvider InfoError, PleaseCheckTheHealthLogs., ''), 

(Host-4****, vCenter, '', Yellow, Yellow, UnableToFetchKeyProviderDetailsOnHost.GetProvider InfoError, PleaseCheckTheHealthLogs., ''),

(Host-4****, vCenter, '', Yellow, Yellow, UnableToFetchKeyProviderDetailsOnHost.GetProvider InfoError, PleaseCheckTheHealthLogs., ''), 

(Host-4****, vCenter, '', Yellow, Yellow, UnableToFetchKeyProviderDetailsOnHost.GetProvider InfoError, PleaseCheckTheHealthLogs., ''),

 

 

[YYYY-MM-DDTHH:MM:SS] ERROR vsan-mgmt [24256] [VsanVapiUtil::GetVapiConfigStubBySolUser opID=noOpId] Fail to connect VAPI by solution user machine

com.vmware.vapi.std.errors_client.ServiceUnavailable: {messages: [LocalizableMessage(id='com.vmware.vapi.endpoint.failedToLoginMaxUserSessionCount Reached', default_message='User session count is limited to 550. Existing session count is 550 for user machine-****bbda-****_****_****_ *******@vsphere.local.', args=['550', '550', 'machine-****bbda-****_****_****_*******@domain.local'], params=None, localized=None)], data: None, error_type: SERVICE UNAVAILABLE}

 

Environment

VMware vSAN 7.x
VMware vCenter 7.x

Cause

This behavior is seen as vapi-endpoint incoming connections are limited to 550.

Resolution

The issue is fixed in the VMware vCenter version 7.0.U3q Build 23788036 and  higher version. To download VC 7.0U3q please refer this link. 

Workaround:

As a workaround, the incoming connections limit can be increased by appending the following property on /etc/vmware-vapi/endpoint.properties:
http.endpoint.default.maxConnections=550
 
The value can be increased maximum to 1000. It is recommended to keep lesser then 900.

This change requires vCenter services restart.

 

Steps to apply workaround as below.

• Take a snapshot of the vCenter server.

• Connect to vCenter server using SSH:

• Take a backup of : /etc/vmware-vapi/endpoint.properties:

cp /etc/vmware-vapi/endpoint.properties /etc/vmware-vapi/endpoint.properties.backup

• Edit the endpoint.properties file using the command below.

vi /etc/vmware-vapi/endpoint.properties

• Modify the session.maxSessionsPerUser to 750 as below.

http.endpoint.default.maxConnections=750

• Restart vmware-vapi-endpoint service

service-control --stop vmware-vapi-endpoint && service-control --start vmware-vapi-endpoint