Apply Changes failed in uploading runtime config releases to the director

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Apply Changes failed in uploading runtime config releases to the director with the following error:

Fetching info:
  Performing request GET 'https://10.x.x.x:25555/info':
    Performing GET request:
      Retry: Get "https://10.x.x.x:25555/info": dial tcp 10.x.x.x:25555: connect: connection refused
Exit code 1

Running Bosh commands also fail with similar error as well.

Environment

Tanzu Kubernetes Grid Integrated Edition

Cause

It is possible that there was an old Director VM that is still running and have the same IP address.  In this particular scenario where there are two Director VM's with the same IP address, this could cause an issue with Ops Manager connecting to the old VM and would fail because the old Director VM doesn't have the persistent disk attached to it and therefore doesn't have any jobs running.  In the /var/tempest/workspaces/default/deployments/bosh-state.json, the VM CID of the Director indicates the new VM. 

However, in the Ops Manager UI->Director Tile->Status tab, the VM CID is that of the old VM.  It is possible that the Ops Manager VM, at some point, re-issued an ARP request for the Director IP address and caught the old VM's ARP response, hence it's connecting to the old VM.  The old VM should have no jobs running and the persistent disk (/dev/sdc1) is not attached to it.  You could also deduce which VM is the old one versus new by reviewing the Apply Changes logs and finding the recent successful Bosh Director VM creation and noting down the VM CID.  There could be various causes of the duplicated VM/IP, but the duplicate IP address is an issue and needs to be sorted out.

Resolution

The following steps can be followed to resolve the duplicate IP address issue.

1. Note down the new Director VM CID from the file /var/tempest/workspaces/default/deployments/bosh-state.json in Ops Manager.
2. Note down the old Director VM CID from the Ops Manager UI->Director Tile->Status tab.
3. In vCenter, search for the Director's IP address and review the results.  The old and new VM CID's should be the results.
4. Review the old VM's settings in vCenter and confirm that the persistent disk (usually the third hard disk) is not attached to it. 
5. Review the new VM's settings in vCenter and confirm that the persistent disk (third hard disk) is attached to it.
6. Shut down the old VM in vCenter.  Reboot the new VM in vCenter.  After the new VM comes up and running, the Director should now be available.
7. The powered off old VM can be deleted from vCenter afterwards.  It would be good to do this after successfully performing checks on the Director.