BOSH Target Error, "cannot access director"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

BOSH Director is not responding and subsequently, the user cannot run the BOSH operations through Operations (Ops) Manager. Below is the log of the BOSH operation:

ubuntu@pivotal-ops-manager:~$ bosh target 
Current target is https://XX.XXX.XXX.XX:XXXXX (p-bosh)
ubuntu@pivotal-ops-manager:~$ bosh login 
[WARNING] cannot access director, trying 4 more times...
[WARNING] cannot access director, trying 3 more times...
[WARNING] cannot access director, trying 2 more times...
[WARNING] cannot access director, trying 1 more times...
cannot access director (Connection refused - connect(2) for "10.111.111.11" port 25555 (10.111.111.11:25555))

Environment

Resolution

In such a case, follow the steps listed below for troubleshooting:

1. ssh vcap@<bosh director IP> to see if it is possible to login to BOSH director VM. The password can be found on the Ops Manager dashboard.

2a. If it doesn't work, check underlying IaaS. If the VM is running well and the IP address is correctly allocated (As example, for vSphere, you can login to the vCenter to check the VM status and open the console window).

2b. If it works, please sudo -i to switch to root user. Issue monit status to check if any jobs are failing.

3. Check if there is any CPU/disk/memory resource issue with the VM, with commands df -H, free, top, etc.

4. Try to restart those failing jobs with monit restart <job name>.

5a. If it doesn't work, check logs under /var/vcap/sys/log/ and contact Pivotal support with the archived logs.

5b. If it works, please execute the BOSH Target again to double check.