VMware vRealize Operations Manager fails to show data for the Health charts in the dashboard section but reports the number
search cancel

VMware vRealize Operations Manager fails to show data for the Health charts in the dashboard section but reports the number

book

Article ID: 342872

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Symptoms:
  • VMware vRealize Operations Manager (formerly known as vCenter Operations Manager) fails to show data for the Health charts in the dashboard section but reports the number
  • Incorrect number of virtual machine is reported by vCenter Server in the vRealize Operations Manager page
  • In the activemq.log file, located at $ALIVE_BASE/user/log on the vRealize Operations Manager, you see entries similar to:
22:43:47,771 | INFO | Transport failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///127.0.0.1:33036
23:10:48,570 | INFO | Transport failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///127.0.0.1:33228
23:25:50,128 | INFO | Transport failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///127.0.0.1:33650
23:25:50,132 | INFO | Slow KahaDB access: Journal append took: 7710 ms, Index update took 0 ms | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
23:25:50,169 | INFO | Slow KahaDB access: cleanup took 7747 | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
23:40:56,228 | INFO | Transport failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///127.0.0.1:33890
  • In the analytics.log file, located at $ALIVE_BASE/user/log on the vRealize Operations Manager, you see entries similar to:
22:52:12,675 ERROR [Threshold checker worker thread 7] com.integrien.analytics.vm.faults.impl.state.vm.VMDiskFullState.setAlertCriticality - Failed to find valid fault score. Will reject fault event FaultEvent [eventName=Guest
file system '/applic' is out of space, eventObjectName=ncdlmorast0002, eventSource=NCDWTARASP0003, eventStatus=Guest file system '/applic' is out of space, faultClass=VmDiskFull?diskName=/applic, faultState=, rollup=4, faultScore=-1, cr
iticality=4, eventClass=12, eventSubclass=5, notificationMessage=Guest file system '/applic' is out of space]
22:52:12,686 ERROR [Threshold checker worker thread 7] com.integrien.analytics.vm.faults.impl.state.vm.VMDiskFullState.setAlertCriticality - Failed to find valid fault score. Will reject fault event FaultEvent [eventName=Guest
file system '/var' is out of space, eventObjectName=ncdlmorast0002, eventSource=NCDWTARASP0003, eventStatus=Guest file system '/var' is out of space, faultClass=VmDiskFull?diskName=/var, faultState=, rollup=4, faultScore=-1, criticality
=4, eventClass=12, eventSubclass=5, notificationMessage=Guest file system '/var' is out of space]


Environment

VMware vCenter Operations Manager (vApp) 5.8.x

Cause

This issue occurs if the database for the messages queue is corrupt. This can be caused by a lack of disk space or an unexpected shutdown, such as powering off the virtual appliance without shutting down the guest first. In this case, the Kaha database becomes full.

Resolution

To resolve this issue, reset the Kaha database.

To reset the Kaha database:
  1. Run this command on the UI virtual machine as Admin to stop vRealize Operations service:

    vcops-admin stop

  2. Run these commands on the Analytics virtual machines as Admin:

    cd /data/activemq/data

    mv kahadb kahadb.old

  3. Run this command on the UI virtual machine as Admin to start vRealize Operations:

    vcops-admin start

  4. Run this command on the UI virtual machine as Admin to check that all services are running:

    vcops-admin status
Note: Data should be populated in the vRealize Operations user interface after the status appears as running. Wait for a minimum of 15 minutes (3 collection cycles) and ensure that the datapoints collection continues.


Additional Information

About Kaha DB and Active MQ:

Active MQ is an open source message broker that writes to a database called kaha database. Here it stores and forwards to other services. If the service cannot accept the messages, it writes it to this database.
Kaha database is the default database in vRealize Operations Manager.