vSphere Web client unresponsive in vCenter Server 6.0

Products

• VMware vCenter Server

Issue/Introduction

vAPI Endpoint being out of memory is a known issue affecting vSphere 6.0.

Symptoms:

• The vAPI Endpoint is in out of memory state in federated environment for vSphere 6.0.
• In the endpoint.log file, you see entries similar to:
  
  
  • Unable to login to inventory service
  • Unable to login/logout to/from inventory service

• The wrapper.log file of vAPI Endpoint contains out of memory exceptions with many vlsi-client threads in the thread dump.
  
  Note: Logs are located at:
  
  
  • vCenter Server Appliance: /var/log/vmware/vapi/endpoint/
  • Windows vCenter Server: C:\ProgramData\VMware\vCenterServer\logs\vapi\endpoint\

You might also see these symptoms, though they may not be related to this issue:

• Heap dumps are present in the endpoint's log folders.
• User symptoms:
  • Login to the vSphere Web Client takes a long time (3-8 minutes).
  • You see an alarm for vmware-vapi-endpoint crashing.
  • Creating or modifying storage tags and profiles fails in the Sphere Web Client.
  • Navigating in the vSphere Web Client is slow or unresponsive.

Environment

VMware vCenter Server Appliance 6.0.x
VMware vCenter Server 6.0.x

Cause

This issue is caused by an out of memory condition in the vAPI Endpoint service. This is a known issue that manifests when a vCenter Server node cannot reach the Inventory Service on one or more nodes in the federated environment.

For more information about the out of memory condition, see the Additional Information section.

Resolution

This issue is resolved in vCenter Server 6.0 Update 3, available at Broadcom Support portal.

 

To work around this issue, ensure that all vCenter Server nodes are accessible in the federation:

1. Check if the Inventory Service is running.
2. Check the connectivity between all vCenter Server nodes
3. Restart the vAPI Endpoint service:

 

vCenter Server Appliance:

1. Connect to vCenter Server Appliance with SSH.
2. Switched to the BASH Shell. For more information, see Toggling the vCenter Server Appliance 6.x default shell (2100508)
   
   
3. Run these commands:
   • service-control --stop vmware-vapi-endpoint
   • service-control --start vmware-vapi-endpoint
     
     
4. Ensure the service has started successfully by navigating to:
   
   https://hostname_or_IP/vapiendpoint/health

Windows vCenter Server:

1. Login to vCenter Server.
2. Open a command prompt.
3. Run these commands:
   
   cd C:\Program Files\VMware\vCenter Server\bin
   
   
4. Run these commands:
   
   • service-control.bat --stop vapiEndpoint
   • service-control.bat --start vapiEndpoint
     
     
5. Ensure the service has started successfully by navigating to:
   
   https://hostname_or_IP/vapiendpoint/health

Additional Information

Out of memory condition

 

This issue is reported as an exception by the endpoint in the logs:

Unable to login to inventory service or Unable to login/logout to/from inventory service.

 

Every four minutes, the vAPI Endpoint service triggers a reconfiguration and tries to connect to Inventory Service on all the other vCenter Server nodes. If this fails, the memory for some objects is not cleaned up until vAPI Endpoint service is restarted. This causes a leak in memory for every run and eventually an out of memory condition.

When the vAPI Endpoint is out of memory, the wrapper will restart the service automatically clearing any leaked objects. If connectivity to all vCenter Server nodes has not been restored, the endpoint will soon be out of memory again.

Toggling the vCenter Server Appliance 6.x default shell