vSAN cluster configuration consistency check warning Unable to fetch key provider details on host.get provider info error

Products

• VMware vSphere ESXi
• VMware vSAN 7.x

Issue/Introduction

vSAN skyline health reports warning under Cluster configuration consistency as "Unable to fetch key provider details on host.get provider info error, please check the health logs.".

 

Environment

VMware vSAN 7.x
VMware vCenter 7.x

Cause

This behavior is seen as vapi-endpoint incoming connections are limited to 550. When the user session count reached maximum on the vCenter Server, the warnings would appear.

 

The vmware-vsan-health-summary-result.log file on vCenter would report the below events.

$ vmware-vsan-health-summary-result.log:

2024-06-19T20:11:06.835+10:00 INFO vsan-mgmt[17356] [VsanHealthSummaryLogUtil::PrintHealthResult opID=noOpId] Cluster <**Cluster name**> Overall Health : yellow
      Test consistentconfig health : yellow
         Issues: Host  Disk  Issue  Recommendation
                 (Host-4****, '', UnableToFetchKeyProviderDetailsOnHost.GetProviderInfoError,PleaseCheckTheHealthLogs., ''), (Host-4****, '', UnableToFetchKeyProviderDetailsOnHost.GetProviderInfoError,PleaseCheckTheHealthLogs., ''),
                 (Host-4****, '', UnableToFetchKeyProviderDetailsOnHost.GetProviderInfoError,PleaseCheckTheHealthLogs., ''), (Host-4****, '', UnableToFetchKeyProviderDetailsOnHost.GetProviderInfoError,PleaseCheckTheHealthLogs., ''),

   Group encryption health : unknown
      Test kmsconnection health : skipped
         HostsKmsStatus: Hosts  KmsCluster  KmsAlias  ConnectionStatus  KeyState  Issue  Recommendation
                         (Host-4****, vCenter, '', Yellow, Yellow, UnableToFetchKeyProviderDetailsOnHost.GetProviderInfoError,PleaseCheckTheHealthLogs., ''), (Host-4****, vCenter, '', Yellow, Yellow, UnableToFetchKeyProviderDetailsOnHost.GetProviderInfoError,PleaseCheckTheHealthLogs., ''),
                         (Host-4****, vCenter, '', Yellow, Yellow, UnableToFetchKeyProviderDetailsOnHost.GetProviderInfoError,PleaseCheckTheHealthLogs., ''), (Host-4****, vCenter, '', Yellow, Yellow, UnableToFetchKeyProviderDetailsOnHost.GetProviderInfoError,PleaseCheckTheHealthLogs., ''),

 

The vmware-vsan-health-service.log on vCenter would report events "User session count is limited to 550. Existing session count is 550 for user"

$ less vmware-vsan-health-service.log
2024-06-19T20:11:08.435+10:00 ERROR vsan-mgmt[24256] [VsanVapiUtil::GetVapiConfigStubBySolUser opID=noOpId] Fail to connect vAPI by solution user machine
com.vmware.vapi.std.errors_client.ServiceUnavailable: {messages : [LocalizableMessage(id='com.vmware.vapi.endpoint.failedToLoginMaxUserSessionCountReached', default_message='User session count is limited to 550. Existing session count is 550 for user machine-****bbda-****-****-****-*******@vsphere.local.', args=['550', '550', 'machine-****bbda-****-****-****-*******@vsphere.local'], params=None, localized=None)], data : None, error_type : SERVICE_UNAVAILABLE}

 

Resolution

The issue is fixed in the VMware vCenter version 7.0.U3q Build 23788036 and above.

Workaround:

As a workaround, the incoming connections limit can be increased by appending the following property on /etc/vmware-vapi/endpoint.properties:
http.endpoint.default.maxConnections=550
 
The value can be increased maximum to 1000. It is recommended to keep lesser then 900.

This change requires vCenter services restart.

 

Steps to apply workaround as below.

• Take a snapshot of the vCenter server.

• Connect to vCenter server using SSH:

• Take a backup of : /etc/vmware-vapi/endpoint.properties:

cp /etc/vmware-vapi/endpoint.properties /etc/vmware-vapi/endpoint.properties.backup

• Edit the endpoint.properties file using the command below.

vi /etc/vmware-vapi/endpoint.properties

• Modify the session.maxSessionsPerUser to 750 as below.

http.endpoint.default.maxConnections=750

• Restart vmware-vapi-endpoint service

service-control --stop vmware-vapi-endpoint && service-control --start vmware-vapi-endpoint