Vertica database fails to start due to spread service failure

Products

• Network Observability
• CA Performance Management

Issue/Introduction

Running a single node Vertica database for DX NetOps Performance Management.

Vertica runs version 10.1.1-20.

The node ran out of disk space. More space was added and the server was rebooted. After this it fails to start.

Trying to start it using the adminTools UI we see this message.

Database startup failed. Too many nodes having invalid/missing last good epoch values to continue.

Tried a force start using this command:

/opt/vertica/bin/admintools -t start_db --force -d <database_name>

It returns failure with the message:

Database <dbName> did not start successfully: Nodes Failed to come up

In the vertica.log file we see the following message about it trying to start the spread daemon which is required to run before the DB can be started.

About to launch spread with '/opt/vertica/spread/sbin/spread -c <Catalog_Path>/v_<dbName_node0001_catalog/spread.conf

We see it try to start the spread daemon but then returns this INFO message with no specific error.

Spread daemon does not appear to be running on <host_IP> -- exiting!

Environment

All supported DX NetOps Performance Management releases

Cause

The spread.conf and vertica.conf files in the <Catalog_Path>/v_<dbName_node0001_catalog/ directory are empty. The files are zero bytes.

Resolution

The files can be rebuilt using copies from another working Vertica DB cluster running the same Vertica version.

The spread.conf will need to have it's IPs updated when rebuilding it on the down node from the working node.

The vertica.conf config should remain unchanged. Just rebuild the file on the down node from the content of the working node.

After rebuilding the files the DB should restart normally using the adminTools UI.

Additional Information

If a second environment is not available please open a support case for further assistance.