"get_task" timeout error repeats when attempting to create VMware Tanzu GemFire for VMs cluster

Products

• VMware Tanzu Gemfire

Issue/Introduction

When attempting to deploy/create a VMware Tanzu GemFire for VMs cluster service instances you may see the following error repeat: 

L Error: Timed out sending 'get_task' to instance: 'server/266e7ca1-c19c-4a99-882b-3a7365d503c2', agent-id: '17430413-debe-4066-a778-ad73f7d57739' after 45 seconds

When trying to delete the instance after the failure, you see the following error:

L Error: Timed out sending 'run_script' to instance: 'locator/8022b827-bbd7-4e88-bc89-889a5ee7b79d', agent-id: 'f65f2e04-a95d-4bff-be11-145b7c55d80a' after 45 seconds

Environment

Product Version: 1.11

Resolution

A common cause for this type of timeout error is duplicate IP addresses caused when the BOSH Director does not shutdown and removed bosh-health-check VMs successfully. In this situation, a bosh-health-check VM may occupy the IP address that the BOSH Director believes is free. As a result, it removes any references to the IP address but not the duplicate.

To work around this issue, use the "cf purge-service-instance" and "bosh -d delete-deployment --force" commands to get rid of the service instance.

This may need to be done repeatedly or several times until the cluster can be successfully created.