No route to host – connect (2) for port - error while creating Cloud Cache (CC) service instance

Products

• VMware Tanzu Gemfire

Issue/Introduction

Summary

When you upgrade the Cloud Cache (CC) from one version to another, your upgrade/install may be successful. However, you may get the below error when trying to create a service instance with new version from Apps Manager UI.
 

 

Symptoms

This happens when BOSH doesn’t recognize the changed IP addresses for newly created VMs during the upgrade. Further, the HAProxy IP addresses under TAS tile > Settings > Networking has been changed during the event of an upgrade or network infrastructure settings can cause this error. 

This is how the log entry looks like when you try to create a service instance on upgraded CC tile:

[32m[2020-03-13 14:55:29.85 (UTC)]> cf create-service p-cloudcache extra-small test-service [0m  
          Creating service instance test-service in org system / space SMOKE_TESTS-1-SPACE-79202f2df6aeadf9 as SMOKE_TESTS-1-USER-338b51f1ace4281d...  
          FAILED  
          Server error, status code: 500, error code: 10001, message: No route to host - No route to host - connect(2) for "xxx.xxx.xxx.xxx" port xxxx(xxx.xxx.xxx.xxx:xxx)

Environment

OS: N/A

Resolution

The best solution to fix this problem by avoiding manual intervention is to make sure to run Register Broker Errands on CC Tile > Settings > Errands and then Apply Changes.

These Errands by BOSH Director makes sure it refreshes the IP addresses or network settings for CC and registers in its database.