Rally - On-premises: Troubleshooting Database Connection Issues

Products

• CA Agile Central On Premise (Rally)

Issue/Introduction

Issues with the services VM are often tied to communication issues between the services virtual machine (VM) and the database.  These communication issues can arise due to the following reasons:

• Database VM is offline or inaccessible
• DNS server is not resolving the database hostname
• Database configuration incorrect in settings
• Docker services have mismatched IPs
• Incorrect subnet masks
• Unsupported network topology
• Network configuration conflicts with Docker routes

Environment

Release: RLOPPL99000-2.0-Rally-On Premise-Perpetual
Component:

Resolution

Database VM offline or inaccessible

1. Ensure that the database VM is powered on
2. Try to ping the database IP address from the services VM
3. Use curl to open a port to the database IP on port 1521

Example:

$ curl 10.17.71.10:1521

A successful response will look like this:

curl: (52) Empty reply from server

A failed response will look like this:

curl: (7) Failed connect to 10.17.71.10:1521; No route to host

DNS server not resolving database hostname

1. Access your Rally dashboard
2. Click Settings in the navigation bar
3. If the "JDBC service URL" is a hostname, ensure you can resolve that hostname from the services VM

Example:
$ ping rally-oracle.company.com
A successful response will look like this:

PING rally-oracle.company.com (10.17.71.10) 56(84) bytes of data.

A failed response will look like this:

ping: rally-oracle.company.com: Name or service not known

 

Database address incorrect in settings

1. Access your Rally dashboard
2. Click Settings in the navigation bar
3. Verify the "JDBC service URL" is correct and correct format.

Examples:

10.17.71.10:1521/f4wp1
rally-oracle.company.com:1521/f4wp1

 

Docker services have mismatched IPs

When the services VM starts, it runs scripts that attempts to align the docker container IPs with the IP assigned to the VM.  On occasion, this process will not complete successfully and typically a reboot resolves this issue.  To verify the IPs, perform the following steps:

1. Locate the IP assigned to the services appliance
2. Access your Rally dashboard
3. Click Cluster in the navigation bar
4. Look at the Image row in the Services section and ensure that the IP address listed there matches the IP assigned to the services VM.  If it does not, restart the VM

Incorrect Subnet Mask in Network Configurations

The nmtui tool does not have a separate field for subnet mask but instead uses CIDR notation to specify the subnet mask.  nmtui does not warn if the CIDR suffix is not present however it will default to a value of /32 as seen below. 

<Please see attached file for image>

If this happens, the network stack will believe it is the only device on the subnet and will send all traffic to its default route and will not be able to find the database server.  To resolve this situation, it will be necessary to verify the subnet mask in CIDR notation with the network administrator and update the subnet mask in nmtui.  This will need to be verified on both the services VM and the database VM.
 

Unsupported Network Topology

Rally does not support splitting the services VM and the database VM onto two different subnets or networks.  In order to maintain maximum performance and ensure reliable communications, both VMs should be in the same broadcast domain as each other.
 

Network configuration conflicts with Docker routes

Due to the way Docker handles routing traffic between containers, they will have their own IP routing tables.  If the VMs are configured with IP addresses that overlap these routing tables, then communication issues may occur.  In general it is best to not use the following networks:

• 10.0.0.0/24
• 10.0.1.0/24
• 10.0.2.0/24
• 10.0.4.0/24
• 10.255.0.0/16
• 172.17.0.0/16
• 172.18.0.0/16