BOSH SSH fails on Ops Manager due to duplicate IP address on network

Products

• Ops Manager
• VMware Tanzu Application Service for VMs

Issue/Introduction

After upgrading Ops Manager from version 2.11 to 3.0.31, customer observed 2 unusual behaviors:

1. When executing 'bosh ssh' to any VM (for example, a diego_cell in Tanzu Application Service for VMs - TAS), the connection would succeed, but immediately after the session was established, it would be forcibly terminated. Not only did the SSH session to the TAS VM end, but the "outer" SSH session from a jumpbox to the Ops Manager VM was terminated also.
2. HealthWatch showed the Ops Manager intermittently unhealthy.

Resolution

The behavior of SSH in this case would be consistent with the presence of a duplicate IP address. It's possible that the old Ops Man VM is still present and powered on in the customer's network.
2 methods of testing for this:
1. Search in vSphere for the IP address of the Ops Man VM. If there is more than one device using that address, that explains the SSH behavior. Power off or delete the devices with the duplicate IP's.
2. Power off the Ops Man VM, then attempt to connect to the address with "nc <IP-address> 22" (for port 22, the SSH port). If you are able to connect to that address with the Ops Man VM powered off, you need to find the device with the duplicate IP.