VMware vRealize Log Insight 2.5 startup fails with the error: Unable to gossip with any seeds

Products

VMware

Issue/Introduction

This article describes a specific problem affecting VMware vRealize Log Insight 2.5.

Symptoms:

Starting VMware vRealize Log Insight 2.5 fails.
In the /storage/var/loginsight/loginsight_daemon_stdout.log file or /storage/var/loginsight/runtime.log file, you see an entry similar to:

Daemon startup failed: Failed to start Cassandra Server: StartupException(description:Unable to connect to Cassandra node at localhost:nnnn: com.vmware.loginsight.cassandra.CassandraException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1 (com.datastax.driver.core.TransportException: [localhost/127.0.0.1] Cannot connect))).
In the /storage/var/loginsight/cassandra.log file, you see an entry similar to:

ERROR [main] yyyy-mm-dd hh:mm:ss,nnn CassandraDaemon.java (line ccc) Exception encountered during startup java.lang.RuntimeException: Unable to gossip with any seeds

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

This issue occurs when the Apache Cassandra service within the Log Insight 2.5 virtual appliance cannot establish communication between cluster nodes. This may be noticed immediately after upgrading a cluster from Log Insight 2.0 to 2.5.

This issue is triggered by an incorrect DNS resolution for the Log Insight Master node's own IP address.

Resolution

This is a known issue affecting VMware vRealize Log Insight 2.5.

To work around this issue, modify the /etc/hosts configuration file on the Master node and set the Fully Qualified Domain Name (FQDN) of the host and the hostname associated with the IP address of the node.

To correct the /etc/hosts configuration file on the Master node in the Log Insight cluster:

Open a console or SSH session to the Log Insight virtual appliance and log in as root.
Validate that the Log insight service is not running by running this command:

service loginsight status

You should see daemonNotRunning. If the service is running, these steps are not necessary.
Determine the Log Insight node's IP address by running this command:

/opt/vmware/share/vami/vami_ip_addr eth0
Open the /etc/hosts configuration file in a text editor. For more information, see the Editing using vi section of Editing files on an ESX host using vi or nano (1020302).
Inspect the contents of the /etc/hosts configuration file. A problematic configuration has the Log Insight node's fully-qualified domain name (FQDN) or hostname listed on the same line as the loopback address 127.0.0.1. For example:

127.0.0.1 loginsight-master.example.com loginsight-master localhost
Modify the /etc/hosts configuration file to specify the node's fully-qualified domain name and hostname associated with the IP address of the node noted in Step 3. For example:

10.11.12.13 loginsight-master.example.com loginsight-master 127.0.0.1 localhost
If the /etc/hosts configuration file contains a line for the ::1 IPv6 loopback address, remove it. For example, remove this line entirely:

::1 loginsight-master.example.com loginsight-master localhost ip6-localhost ip6-loopback
Save and close the /etc/hosts configuration file.
Start the Log Insight service using the command:

service loginsight start

Note: If Worker nodes in the Log Insight cluster restarted and failed to connect to the Master node during this procedure, restart the Log Insight service on each Worker node.

Open a console or SSH session to each Log Insight cluster Worker node and log in as root.
Restart the Log Insight service by running this command:

service loginsight restart

Additional Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box.
Editing files on an ESX host using vi or nano