vCenter collections alerts are generated daily and Timeout while waiting for vSphere" in Aria Operations for Logs

Products

• VMware Aria Suite

Issue/Introduction

This issue can have multiple causes and resolutions.  These will be marked in the vCenter Storage and Collection Routine for WCP events (Kubernetes) sections.
 

vCenter Storage

• vRealize Log Insight triggers vCenter task/event collection alerts at same time every day.
• Running df -h on the vCenter shows the /storage/seat partition several hundred GB.

Collection Routine for WCP events (Kubernetes)

• Aria Operations for Logs triggers vCenter task/event collection alerts at same time every day.
• Within the /storage/core/loginsight/var/plugins/vsphere/li-vsphere.log file you may see entries similar to:

2020-05-11 19:25:16.167+0000] ["pool-11-thread-1"/[IP of vCenter] ERROR] [com.vmware.loginsight.scheduled.ScheduledPluginService] [Error running vSphere WCP collection]
java.lang.Exception: vCenter API vcenter.example.com is not available, response code: 403, message: {"type":"com.vmware.vapi.std.errors.unauthorized","value":{"messages":[{"args":[],"default_message":"Unable to authorize user","id":"vapi.security.authorization.invalid"}]}}

 

[2020-04-24 06:32:51.740+0000] ["ScheduledPluginServiceScheduledTask-thread-1"/##.##.##.## ERROR] [com.vmware.loginsight.scheduled.ScheduledPluginService] [Timeout on waiting '##.##.##.##', timeout=600000 millis. vCenter can't complete requested range in time. Increasing vsphere's config event-collection-timeout-millis may mitigate the problem but that may introduce delays to other vCenter collectors.]

java.util.concurrent.CancellationException

 

 

vSphere integration error: "Timeout while waiting for vSphere"

Environment

• Aria Operations for Logs 8.x
• VMware vRealize Log Insight 8.x

Cause

vCenter Storage

Larger vCenter environments with broad task, event, and stat retention settings can cause vCenter database bloat.

Collection Routine for WCP events (Kubernetes)

These alerts are caused by a collection routine for WCP events (Kubernetes).

Resolution

vCenter Storage

To resolve this issue, follow Delete old tasks, events and statistics data in vCenter Server 5.x, 6.x, 7.x and 8.x and set all of the parameters to 0.
Do not interrupt the script until complete.

Collection Routine for WCP events (Kubernetes)

You can disable the collection of these events and prevent the error by following the steps below:

1. Open SSH session as root to Aria Operations for Logs Primary node.
2. Using a text editor, edit /usr/lib/loginsight/application/etc/loginsight-config-base.xml
3. Set wcp-collection-enabled to false.

Example: <wcp-collection-enabled value="false" />

4. Save the file and exit the text editor
5. Run the following command to restart the loginsight service:

service loginsight restart

     6. Repeat steps 2-5 to each Aria Operations for Logs node as needed.

Additional Information

vCenter Storage

To prevent this issue from occurring in the future, reduce the event, alarm, and statistics retention settings on the vCenter object in the vCenter web client.

1. Log into the vCenter web client as an administrative user.
2. Select the vCenter object from any environment view.
3. Navigate to Configure > Settings > General and click Edit.
4. Under Statistics, reduce or disable any desired values.
5. Under Database, reduce the Task retention (days), and Event retention (days) settings to any desired values.
6. Click Save.