VMware vAPI Endpoint Service Health Alarm status changed from green to yellow
search cancel

VMware vAPI Endpoint Service Health Alarm status changed from green to yellow

book

Article ID: 322831

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

- vAPI endpoint status changes to yellow alarm in vCenter intermittently.
- In the /var/log/vmware/vapi/endpoint/endpoint.log file, you see entries similar to:
 
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild cis-api-connections-builder
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:9090/ds/vapi
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:16666/cls/
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:16666/cls/
YYYY-MM-DDTXX:XX:XX.XXXZ | WARN  | state-manager1            | ApiConnectionsLsUtil           | Cannot find metadata source definitions in VAPI endpoint Service Endpoint of type com.vmware.cis.data.provider with protocol vapi.json.http at http://localhost:16666/cls/
YYYY-MM-DDTXX:XX:XX.XXXZ | WARN  | state-manager1            | ApiConnectionsLsUtil           | Unable to find metadata endpoint in service Service with localization key cis.content-library.ServiceDescription and id dd <ID>.
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | ApiConnectionsStateBuilder     | Cannot resolve protocol priorities between the following services. Will use the first one.
        First: dd<ID>\com.vmware.cis.cls.vapi at http://localhost:16666/cls/
        Second: dd<ID>\com.vmware.cis.cls.vapi at http://<VC FQDN>:80/cls/
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:16666/cls/
YYYY-MM-DDTXX:XX:XX.XXXZ | WARN  | state-manager1            | ApiConnectionsLsUtil           | Cannot find metadata source definitions in VAPI endpoint Service Endpoint of type com.vmware.cdc.provider with protocol va
pi.json.http at http://localhost:16666/cls/
YYYY-MM-DDTXX:XX:XX.XXXZ | WARN  | state-manager1            | ApiConnectionsLsUtil           | Unable to find metadata endpoint in service Service with localization key cis.content-library.ServiceDescription and id dd <ID>.
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:10080/invsvc/vapi
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:12346/site/api
YYYY-MM-DDTXX:XX:XX.XXXZ | WARN  | state-manager1            | ApiConnectionsLsUtil           | Cannot find metadata source definitions in VAPI endpoint Service Endpoint of type com.vmware.vapi.endpoint with protocol v
api.json.http at http://localhost:12346/site/api
YYYY-MM-DDTXX:XX:XX.XXXZ | WARN  | state-manager1            | ApiConnectionsLsUtil           | Unable to find metadata endpoint in service Service with localization key cis.vapi.endpoint.serviceDescriptionResourceKey
and id 363e5980-5e66-4a86-b43d-08ba5f953193.
YYYY-MM-DDTXX:XX:XX.XXXZ | WARN  | state-manager1            | ApiConnectionsLsUtil           | Cannot find metadata source files/URLs in VAPI endpoint Service Endpoint of type com.vmware.vapi.endpoint with protocol va
pi.json.http at http://<VC FQDN>:80/site/api
YYYY-MM-DDTXX:XX:XX.XXXZ | WARN  | state-manager1            | ApiConnectionsLsUtil           | Unable to find metadata endpoint in service Service with localization key cis.vapi.endpoint.serviceDescriptionResourceKey
and id 363e5980-5e66-4a86-b43d-08ba5f953193.
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:13080/statsreport/to
pn
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:8900/vmonapi
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:8093/<vCenter>
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild vim-adapter-settings-builder
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild vapi-vcenter-servlet-builder
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild api-interfaces-builder
YYYY-MM-DDTXX:XX:XX.XXXZ | WARN  | state-manager1            | ApiInterfacesFactory           | Retrieving interfaces for service 9dd<ID>\com.vmware.cis.ds has failed.
YYYY-MM-DDTXX:XX:XX.XXXZ | WARN  | state-manager1            | ApiInterfacesFactory           | Retrieving interfaces for service 9dd<ID>\com.vmware.cis.ds has failed.
com.vmware.vapi.internal.core.abort.RequestAbortedException: Http request aborted.
        at com.vmware.vapi.internal.protocol.common.Util$1.onAbort(Util.java:68)
        at com.vmware.vapi.internal.core.abort.AbortHandleImpl.abort(AbortHandleImpl.java:39)
        at com.vmware.vapi.endpoint.api.TimedApiProvider$1.run(TimedApiProvider.java:58)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
YYYY-MM-DDTXX:XX:XX.XXXZ | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH YELLOW Failed to refresh 9dd<ID>\com.vmware.cis.ds vAPI provider.

Environment

VMware vCenter Server Appliance 6.x and 7.x

Cause

vapi-endpoint fails to respond to health requests and gets restarted
vapi-endpoint crashes with out of memory when very large inventory tags is accessed

Resolution

A stop and start of the vAPI-endpoint service can resolve this issue. 

1. Connect to the VCSA appliance via SSH

2.  Enable shell

3. Run this command to stop the service:
service-control --stop vmware-vapi-endpoint

4. Run this command to start the service:
service-control --start vmware-vapi-endpoint

Enabling and disabling of the alarm can also resolve this issue.  See Create or Edit Alarms

If the issue persists after trying all of the steps above, please open a support ticket and reference this KB.

Powered by Wolken Software