Vertica nodes are down and cannot be started

Products

• Network Observability
• CA Performance Management

Issue/Introduction

The Data Repository (DR) nodes fail to start via admintools, reporting transaction commit failures. This typically occurs when the catalog partition has no remaining disk space.

When attempting to start the database, the startup.log or console displays errors similar to the following:

Error found in the startup.log file:

Failure during commit of transaction 0xfff0000000000cc1, cannot proceed
HINT: Vertica semantics require commit to succeed or the node to go down; restart this node
LOCATION: commitTransaction, ...

 

Checking disk space with the command "df -h" shows the catalog directory is at 100% (or near 100%) capacity.

Environment

• DX NetOps Performance Management (all versions)
• Data Repository (Vertica)

Cause

The vertica.log file in the catalog directory has grown significantly, exhausting the available disk space, preventing Vertica from completing the transaction commit required for startup.

Resolution

To resolve this issue and prevent it from recurring, follow these steps:

1. Log in to the impacted Data Repository node(s) as the dradmin user.
2. Navigate to the catalog directory and clear the current vertica.log file to free up space immediately:
   bash

    

   echo "" > vertica.log

3. Start the Vertica database using the admintools utility. If further assistance is needed for startup, see Unable to start the Vertica database.
4. Once the database is running, configure Log Rotation to ensure the vertica.log is managed automatically and does not fill the disk again. Refer to vertica.log keeps growing in size on the CAPM Data Repository for the configuration command.

Additional Information

For more details on how Vertica manages log files, see How does the Data Repository (Vertica) rotate logs?

Additional Information

See the Configure Log Rotation for the Data Repository documentation topic for additional Vertica log rotation details and configuration.