Task times out after 45 seconds during Apply Changes

Products

• VMware Tanzu Platform
• VMware Tanzu Application Service
• VMware Tanzu Application Service for VMs

Issue/Introduction

During Apply Changes, bosh may try to create a VM during a deployment or while running an errand.

 

Task 69875 | 19:11:23 | L starting jobs: deploy-service-broker/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (0) (canary) (00:03:01)
                      L Error: Timed out sending 'get_state' to instance: 'deploy-service-broker/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx', agent-id: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' after 45 seconds
Running errand 'deploy-service-broker':
  Expected task '69875' to succeed but state is 'error'

 

After the VM is created and the bosh agent responds to the PING request mentioned above the Bosh director will ask the Agent to perform a task like clear arp entries or run pre-start scripts. Director will submit this task to the agent and wait for a finished status. During the interim the Director expects the agent to continuously report running state to get_task request. If the Agent fails to report then the Director will put the VM in an unresponsive state. In these cases the Director is not concerned with how long the agent takes to complete the task, instead the Director is concerned how long it takes for the Agent to respond to the get_task request.

 

Resolution

There could be a duplicate IP on the network. In order to prevent the timeout, you must remove, reconfigure, or power off the device so that the IP address conflict no longer hinders bosh from assigning the address to a new device.

Check the bosh task logs (bosh task <task_number> --cpi) and search for the IP that was chosen for the instance in question and ensure no other VM exists with that IP.

Look for the IP address in the output of bosh vms, bosh instances --details, and in the IaaS (vSphere, Azure, GCP, AWS, etc.) console. Log into a VM on the same network as the Tanzu foundation and attempt to connect to the IP address in questions using ping, traceroute, nc, and similar commands.

Check if the VM instance exists in the IaaS and check if there are any IAAS events that correlate with vm creation or start up failures.