vSphere Tag collection fails in vRealize Operations Manager

Products

VMware Aria Suite

Issue/Introduction

The purpose of this article is to help users troubleshoot and resolve Tag collection issues from vSphere in vRealize Operations Manager.

It is expected that user permissions have already been correctly set.
Review Minimum Collection User Permissions in vRealize Operations Manager 6.x for correct user permissions required to collect tags from vCenter.
The vAPI service is only available in vSphere 6.x.

Symptoms:

vRealize Operations Manager fails to collect vSphere tags from vCenter
vSphere tags do not appear in vRealize Operations Manager even though permissions are correctly set.
Logging into the remote collector where the vCenter adapter is running and checking /data/vcops/log/collector.log shows the below error:

ERROR [Collector worker thread 3] (40) com.integrien.adapter.vmware.TagServiceTagHandler.collectTagData - collect tag data failed, get exception java.util.concurrent.ExecutionException: com.vmware.vapi.client.exception.TransportProtocolException: HTTP response with status code 503 (enable debug logging for details): <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 503 Service Unavailable</title>
</head>
<body><h2>HTTP ERROR 503</h2>
<p>Problem accessing /api. Reason:
<pre> Service Unavailable</pre></p>
</body>
</html>

Environment

VMware vRealize Operations Manager 6.6.x
VMware vRealize Operations Manager 7.0.x
VMware vRealize Operations 8.x
VMware vRealize Operations Manager 7.5.x
VMware vRealize Operations Manager 6.7.x
VMware vRealize Operations 8.0.x

Cause

The cause of this issue when the above messages are received is due to two possible causes:

The vCenter vAPI service is unresponsive or is experiencing issues.
The vCenter vAPI service cannot accommodate the API request as it is too large. This usually occurs if you have a large number of tags in the environment

Resolution

In order to resolve this issue, the user can do the following in order:

Restart the vAPI Service

Follow Stopping, Starting or Restarting VMware vCenter Server Appliance 6.x services to restart the vmware-vapi-endpoint service.

Check if vSphere tags show up within the next collection cycle in vRealize Operations Manager.
If the above does not resolve the issue, proceed to Set the Global Request Rate.

Set the Global Request Rate

Log into the vSphere web client as an Administrator.
Navigate to Administration > System Configuration > Objects > Services.
Select the vAPI Enpoint service.
Click Settings.
Click Edit.
Filter for Global request rate.
Change the default value of Global request rate from 360 to 0 to disable the limitation and click OK.

Collection should now work for vSphere tags.

Additional Information

You can also restart the vAPI service via SSH by performing the below commands:
service-control --stop vmware-vapi-endpoint
service-control --start vmware-vapi-endpoint

Impact/Risks:
Restarting the vAPI service will not allow API calls to be processed for a short time by vCenter