Deployments fail with IP Reservation conflicts when Healthwatch or On Demand Service tiles are installed

Products

• Operations Manager

Issue/Introduction

Symptoms:
Deployments fail with CPI error 

[create_missing_vm(mysql/GUID (0)/1)] DEBUG -- DirectorJobRunner: [external-cpi] [cpi-373588] response: {"result":null,"error":{"type":"Unknown","message":"Detected IP conflicts with other VMs on the same networks: vm-GUID on network service-network1 with ip 192.168.0.10","ok_to_retry":false},"log":"Rescued Unknown: Detected IP conflicts with other VMs on the same networks: vm-GUID on network service-network1 with ip 192.168.0.10

Environment

Cause

This problem occurs when Operations Manager redeploys the BOSH Director at the same time Healthwatch or an On Demand Tile is creating the a new BOSH deployment. Operations Manager uses a version of the BOSH CLI that does not wait for running deployment to complete before taking the BOSH Director offline. This may result in orphaned VMs resting in a powered on state in the IaaS which is consuming an IP address that the BOSH Director will not remember.

Resolution

Identify and locate the Orphaned VM by search for the Duplicate IP address or by the VM GUID returned in the error from the failed BOSH deployment. Once located you can remove the orphaned VM from the IaaS and then retry the deployment.

This is a known issue that is fixed in BOSH CLI version v5.3.1 which includes a new feature that prevents the director from being recreated when there are existing deployments in flight. BOSH CLI will wait for the running deployment to complete before recreating the BOSH Director.

This is fixed in Operations Manager Versions 2.4.1.