vAPI endpoint status changes to yellow in vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

To work around this issue, restart the vAPI Endpoint service to clear the faulty configuration state.

Symptoms:

• vCenter Server alarm reports this warning when using vCenter Server Appliance 6.0:
   
  • vmware-vapi-endpoint service reports yellow status continually
     
• In the endpoint.log file, you see entries similar to:
  
  Unexpected error while initializing endpoint runtime state org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'some-bean-name' is defined
  
  Note: The endpoint log files are located at:
  • vCenter Server Appliance - /var/log/vmware/vapi/endpoint
  • Windows vCenter Server - C:\ProgramData\VMware\vCenterServer\logs\vapi\endpoint
     
• When logged into the Web Client, all users (including [email protected]) will see this message for all objects:
  
  You do not have permissions to view this object or this object does not exist.

Environment

VMware vCenter Server Appliance 6.0.x
VMware vCenter Server 6.0.x

Cause

For vAPI Endpoint service to enter a perpetually misconfigured state and report yellow status, a rare issue in the environment must occur. As of now the only known issue to cause this is a no route to host to the remote server https://vcenter-url/sts/STSService/vsphere.local .

The root cause is due to a known issue in the vAPI Endpoint - when vAPI Endpoint service is re configuring itself and a badly handled exception occurs, it enters a state where a broken spring bean is saved and every subsequent reconfiguration fails and the service reports yellow status from there on. The badly handled exception might be:

Error communicating to the remote server server-url
com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: java.net.NoRouteToHostException: No route to host: connect.

Note: vAPI Endpoint reconfigures itself every four minutes in order to adapt to any changes in the vCenter Server environment (for example, a new node has joined a node has changed IP address). Though vAPI Endpoint service cannot reconfigure itself any more and reports yellow status, it uses a cached configuration (its last successful one) and thus can still service requests without error as long as there is no actual need for reconfiguration.

Resolution

This is a known issue affecting vCenter Server 6.0 and vCenter Server Appliance 6.0.

vCenter 6.0 U3 makes the system more stable and reduces the opportunity for false health alarms.

To work around this issue, restart the vAPI Endpoint service to clear the faulty configuration state.

To restart the vAPI Endpoint service:

• In vCenter Server Appliance:
   
  1. Connect to vCenter Server Appliance with SSH.
  2. Switched to the BASH Shell. For more information, see Toggling the vCenter Server Appliance 6.x default shell (2100508).
  3. Run this command:
     
     service-control --stop vmware-vapi-endpoint
      
  4. Run this command:
     
     service-control --start vmware-vapi-endpoint
      
  5. Open https://hostname_or_IP/vapiendpoint/health from your client PC in a web browser and confirm that the service started successfully.
      
• In Windows vCenter Server:
   
  1. Log in to vCenter Server.
  2. Open a command prompt.
  3. Run this command:
     
     cd C:\Program Files\VMware\vCenter Server\bin
      
  4. Run this command:
     
     service-control.bat --stop vapiEndpoint
      
  5. Run this command:
     
     service-control.bat --start vapiEndpoint
      
  6. Ensure the service has started successfully by navigating to:
     
     https://hostname_or_IP/vapiendpoint/health.

Additional Information

Toggling the vCenter Server Appliance 6.x default shell
Stopping, starting, or restarting VMware vCenter Server Appliance 6.x services
vCenter Server で vAPI Endpoint ステータスが黄色に変わる