How To Clear Data Inserter Queue in vRealize Hyperic

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• If your Hyperic server was down for any period of time, you may notice sluggish performance, delays in alerts or delays in metrics appearing.
• When looking at the ServerHome/logs/server.log, you may notice something similar to the following repeated several times:

17-08-2016 08:26:00,682 EDT WARN [measurementSendReport-3200213] [org.hyperic.lather.server.LatherServlet@317] org.hyperic.lather.LatherRemoteException: Unable to insert data org.hyperic.hq.measurement.server.session.DataInserterException: No space left in the DataInserter queue. Try again later, or increase your queue size

17-08-2016 08:26:00,810 EDT WARN [measurementSendReport-3200214] [com.hyperic.hq.measurement.server.session.BatchAggregateDataInserter@351] No space left in the DataInserter queue. Agents are sending data in faster than we can process it, queuesize=500000

• The DataInserter queue under Administration > HQ health > Diagnostic > Batch Aggregate DataInserter contains 1,000 or more metrics.

Environment

VMware vCenter Hyperic Server 5.8.1

Cause

Hyperic agents are sending in data faster than can be processed.

Resolution

Over time, the data inserter and Top N queues will clear on their own but to return Hyperic back to a normal state, there is a process where the queues can be paused, allowing Hyperic to catch up.
To resolve the issue, follow the steps below.

1. Log into the vRealize Hyperic UI as an administrator.
2. To clear the DataInserter queue, navigate to Administration > Groovy Console.
3. Run the following commands in the Groovy Console:

import org.hyperic.hq.context.Bootstrap
import org.hyperic.hq.measurement.server.session.*
def inserter = Bootstrap.getBean(MeasurementInserterHolder.class)
inserter.dataInserter.doNotAcceptData.set(true)

Note: The result will say null, which is expected.

4. Navigate to Administration > HQ health > Diagnostic > Batch Aggregate DataInserter to check the queue size.

Note: the queue information will look similar to:
# entries:  0

5. If the entries are higher than 0, refresh the page until you see 0.
6. Navigate to Administration > Groovy Console.
7. Run the following commands in the Groovy Console:

import org.hyperic.hq.context.Bootstrap
import org.hyperic.hq.measurement.server.session.*
def inserter = Bootstrap.getBean(MeasurementInserterHolder.class)
inserter.dataInserter.doNotAcceptData.set(false)

8. Repeat step 4. 

Note: If the DataInserter queue gets to be too high again, repeat steps 2-4; repeat as necessary until your queue has normalized.
 

Once your queue is clear, you should notice significant improvement in performance along with seeing all metrics coming in from agents.