The ActiveMQ service fails to start after the installation drive becomes full in vRealize Operations Enterprise
search cancel

The ActiveMQ service fails to start after the installation drive becomes full in vRealize Operations Enterprise

book

Article ID: 343585

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Symptoms:
    • ActiveMQ fails to start, or ActiveMQ starts then stops unexpectedly
    • The %Alive_Base%\user\log\activemq.log contains entries similar to:

      Failed to start ActiveMQ JMS Message Broker. Reason: java.io.EOFException | org.apache.activemq.brocker.brokerService | wrapperSimpleAppmain java.io.EOFException
      at java.io.randomAccessFile.readInt(RandomAccessFile.java:725)
      at org.apache.kahadb.journal.dataFileAccessor.readLocationDetails(DataFileAccessor.java:115)
      at org.apache.kahadb.journal.journal.getNextLocation(Journal.java:498)
      at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:382)
      at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:382)
      at org.apache.activemq.store.kahadb.MessageDatabase.start(MessageDatabase.java:382)
      at org.apache.activemq.store.kahadb.kahadbPersistenceAdapter.start(kahadbPersistenceAdapter.java:181)


    Environment

    VMware vCenter Operations Enterprise 1.0.x

    Cause

    ActiveMQ writes to a database called kahadb. It stores messages until they can be dispatched to other services. If one of the other services stops accepting messages, ActiveMQ writes the messages into kahadb until the other services can accept the messages again. As a result, a lot of database files are created on the disk.

    When the disk gets full, the database writer is not able to write a complete file and therefore cannot write the end of file (EOF) marker. When ActiveMQ then tries to load the database, it fails, because the database is corrupt.

    Resolution

    To resolve this issue, delete or rename the directory that contains kahadb. This removes the corrupt file and frees up space. ActiveMQ recreates the file if it does not find an existing one.
    1. Stop vRealize Operations Enterprise (formerly known as vCenter Operations Enterprise) services. For more information, see Starting and stopping VMware Alive Enterprise 7.2 and VMware vRealize Operations Enterprise 1.0 services (1034903).
    2. Navigate to %ALIVE_BASE%\activemq\data.
    3. Delete or rename the folder kahadb.
    4. Start vRealize Operations Enterprise (formerly known as vCenter Operations Enterprise) services. For more information, see Starting and stopping VMware Alive Enterprise 7.2 and VMware vRealize Operations Enterprise 1.0 services (1034903).
    5. Verify that a new kahadb directory is created in the same path.

      Note: The user interface may take several minutes to begin to show the health of resources.


    Additional Information

    Starting and stopping VMware Alive Enterprise 7.2 and VMware vRealize Operations Enterprise 1.0 services