"ERROR: Unable to validate cassandra setting for node:XXX" when configuring a Cassandra cluster in VMware Cloud Director.

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Running the command to configure Cassandra cluster nodes in VMware Cloud Director:

./cell-management-tool cassandra --configure --create-schema --cluster-nodes [Node IP 1], [Node IP 2], [Node IP 3], [Node IP 4] --username XXX --password 'XXX' --ttl 15 --port 9042

• The command fails with an error of the form:

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /XXX:9042 (com.datastax.driver.core.exceptions.TransportException: [/XXX:9042] Cannot connect))
    at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:232)

Environment

VMware Cloud Director 10.x

Cause

This error can occur due to a misconfiguration in the networking setup from the Cloud Director Cells to the Cassandra cluster nodes.
For example, a misconfigured firewall rule preventing connectivity from the Cloud Director Cells to the Cassandra cluster nodes.

Resolution

In order to configure a Cassandra Metrics Database for Cloud Director, the Cloud Director Cells must have network access to the Cassandra cluster nodes.

To test if the Cassandra nodes can be reached from Cloud Director you can perform a series of curl commands to check connectivity of the components:

1. Log in to a Cloud Director Cell as root.
2. Using the IP address and Cassandra port of one of the Cassandra cluster nodes attempt a curl command to test connectivity:

curl -k -v telnet://1.2.3.4:9042

3. If the Cassandra cluster node is reachable at this IP and port from the Cloud Director Cell we would expect output similar to the following:

*   Trying 1.2.3.4:9042...
* Connected to 1.2.3.4 (1.2.3.4) port 9042 (#0)

4. Repeat the test against all Cassandra nodes in the Cassandra cluster.

If the curl command fails this suggests that there is an issue with connectivity from the Cloud Director Cells to the Cassandra cluster nodes.
Troubleshoot the networking configuration further by using the same curl commands on the Cassandra nodes to themselves and each other.
Continue to curl from other devices further away from the Cassandra nodes at each network hop before the Cloud Director Cells to determine where the network issue may lie.
Once the network issues are resolved and the curl succeeds on the Cloud Director Cells the cell-management-tool cassandra --configure command can be retried.

NOTE: If the curl is not successful from any location then this suggests there is an issue with the Cassandra cluster itself and not with VMware Cloud Director.

Additional Information

For more information see Install and Configure a Cassandra Database for Storing Historic Metric Data