config-server unable to reach the mirror service

Products

• VMware Tanzu Platform - Cloud Foundry
• VMware Tanzu Platform Spring
• VMware Tanzu Platform Spring Essentials
• VMware Tanzu Spring Runtime
• VMware Tanzu Spring Essentials

Issue/Introduction

You would observe the following error message upon checking the config-server dashboard from the Apps Manager 

"error": "ssh://mirror@<IP>:22/var/vcap/store/mirror/############/#######: DefaultConnectFuture [mirror@/<IP>:22]: Failed to get operation result within specified timeout: 5000

Backing application logs show Health status: UNKNOWN

 

Environment

Spring Cloud Services

Cause

Check if the IP address shown in the config-server dashboard is the same as the SCS broker VM IP, if they are different, then you would see the config-server unable to reach to the mirror-service as expected and hence that error described in the Introduction section.

IP address of SCS broker VM can change if the AZ is migrated for the SCS tile.

However, the existing Config Server instances did not receive the updated metadata. This results in an "Orphaned Service" where the Config Server app is trying to reach a Mirror Service that no longer exists at the old address.

Resolution

Resolution involves two steps, to fix the IP address in the DB with the new IP and then for existing service instances we need to update the new IP in the credhub.

Updating the mirror-service IP in the SCS DB:

1) ssh to the SCS broker VM using the below command 

bosh -d p_spring-cloud-services-############# ssh spring-cloud-services/##############

2) Connect to the mysql instance

mysql --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf

3) Use the mirrordb

use mirrordb;

4) Check the list of occurances of mirror service IP address 

select * from mirror;
+----+----------------------------------------------+--------------------------------------------------------------------------------------------------+-----------------------------------------------------+
| id | mirror_name                                  | mirror_uri                                                                                       | source_uri                                          |
+----+----------------------------------------------+--------------------------------------------------------------------------------------------------+-----------------------------------------------------+
|  1 | 54e015c6059267d477f08c22f0c472ec/config-repo | ssh://mirror@<IP-Address>:22/var/vcap/store/mirror/54e015c6059267d477f08c22f0c472ec/config-repo | https://github.com/spring-cloud-samples/config-repo |
+----+----------------------------------------------+--------------------------------------------------------------------------------------------------+-----------------------------------------------------+
1 row in set (0.00 sec)

5) Update the IP addresses as below:

UPDATE mirror  SET mirror_uri = REPLACE(mirror_uri, 'old-ip', 'new-ip');

Note: Replace the old-ip with the old IP (the one you see in the config-server dashboard) and the new-ip with the one that the SCS broker VM currently gets assigned.

Updating the credhub DB for the existing Service Instances:

For the existing service instances where the credentials in the credhub still points to the old IP we can update using the steps below:

• Identify the CredHub Path: Run cf env <config-server-app-name> and locate the CredHub path in the VCAP_SERVICES or credentials block. Example: /c/p.spring-cloud-services-scs-service-broker/<GUID>/mirror-service

• Export the Configuration: Extract the current JSON to a local file to preserve RSA keys and formatting:

  credhub get -n <PATH> -j | jq '.value' > mirror_config.json

• Update the IP Address:
  Open mirror_config.json and replace the stale IP (e.g., old-ip) with the new Broker VM IP (e.g., new-ip).
• Upload the fix 
  

  credhub set -n <PATH> -t json -v "$(cat mirror_config.json)"

• Restart the config-server app 
  

  cf restart <config-server-app-name>