Error :"All host(s) tried for query failed" when starting VMware Cloud Director services on the cell having Cassandra configured

Products

• VMware Cloud Director

Issue/Introduction

• VMware Cloud Director (VCD) services on the cell fails to start.  
• You have Cassandra configured.
• In /opt/vmware/vcloud-director/logs/cell-runtime.log you see entry like the one reported below:
  
  DATE TIME | FATAL    | Module: com.vmware.vcloud.statsfeeder-application | ModuleRegistry                 | Error in module: "com.vmware.vcloud.statsfeeder
-application"; Bean: cassandraClient, Message: Error creating bean with name 'cassandraClient': Invocation of init method failed; nested exception is com.datastax.driver
.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: ##.##.##.## / ##.##.##.##:9042 (com.datastax.driver.core.exceptions.TransportExce
ption: [##.##.##.## / ##.##.##.##:9042] Channel has been closed), ##.##.##.## / ##.##.##.##:9042 (com.datastax.driver.core.exceptions.TransportException: [##.##.##.## / ##.##.##.##:9042] Channel has been closed), ##.##.##.## / ##.##.##.##:9042 (com.datastax.driver.core.exceptions.TransportException: [##.##.##.## / ##.##.##.##:
9042] Channel has been closed), ##.##.##.## / ##.##.##.##:9042 [only showing errors of first 3 hosts, use getErrors() for more details]) |
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cassandraClient': Invocation of init method failed; nested exception is com.datas
tax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: ##.##.##.## / ##.##.##.##:9042 (com.datastax.driver.core.exceptions.Tra
nsportException: [##.##.##.## / ##.##.##.##:9042] Channel has been closed), ##.##.##.## / ##.##.##.##:9042 (com.datastax.driver.core.exceptions.TransportException: [
##.##.##.## / ##.##.##.##:9042] Channel has been closed), ##.##.##.## / ##.##.##.##:9042 (com.datastax.driver.core.exceptions.TransportException: [##.##.##.## / ##.##.##.##:9042] Channel has been closed), ##.##.##.## / ##.##.##.##:9042 [only showing errors of first 3 hosts, use getErrors() for more details])

• Cassandra node status shows that the nodes are up and running
• The VCD cell can communicate with the Cassandra nodes. To check the connectivity with the Cassandra nodes run the following command replacing the IP of each Cassandra node:
  
  curl -kv telnet://ip

 

 

Environment

VMware Cloud Director 10.x

Cause

SSL connection for Cassandra node is enabled.

Resolution

Disable SSL connection for Cassandra node on all the affected VCD cell by updating the global.properties file.

1. SSH the affected cell 
2. Login using root credential
3. Take a backup of the global properties running the command below:
   
   cp /opt/vmware/vcloud-director/etc/global.properties /opt/vmware/vcloud-director/etc/global.properties.back
4. Edit the global properties file running the command below: 
   
   vi /opt/vmware/vcloud-director/etc/global.properties
5. Add the configuration parameter cassandra.use.ssl to 0 in the global.properties file. Below is reported on example of the global.properties that includes the Cassandra value:
   
   

   # Database connection settings
database.jdbcUrl=jdbc:postgresql://##.##.##.##:5432/vcloud?socketTimeout=90&ssl=true
database.username=vcloud
database.password=#############################################

   # Product display name
product.display_name = VMware Cloud Director

   # Maximum number blocking tasks (per cell) to be resumed when expired.
# This property is taken into account only when blocking task's default timeout action is "Resume".
# Uncomment this line and set the desired value. Otherwise, the default value (1000) will be used.
# blockingTasks.timeoutResumeRate = 1000
product.version=10.5.1.23400185
product.build_date=2024-02-29T16:46:47-0800
vcloud.cell.ip.primary = ##.##.##.##
vcloud.ssl.key.password = #############################################
user.certificate.path = /opt/vmware/vcloud-director/etc/user.http.pem
user.key.path = /opt/vmware/vcloud-director/etc/user.http.key
user.key.password = #############################################
system.info = #################
system.version = 2
cassandra.use.ssl = 0
vcloud.cell.uuid = ####-#######-####-########caf9
vcloud.cell.ips = [##.##.##.##,##.##.##.##,##.##.##.##]
   
   

6. Restart the VCD services on the affected cell