OpsManager Upgrade fails with CPI error: "Failed to create network interface... The specified address is already in use"

Products

• VMware Tanzu Platform Core

Issue/Introduction

When upgrading Operations Manager (e.g., from version 3.1.x to 3.3.x) or applying changes, the BOSH Director fails to provision the new VM. The task fails with a Cloud Provider Interface (CPI) error similar to the following:

CPI 'create_vm' method responded with error: CmdError{"type":"Bosh::Clouds::CloudError","message":"Failed to create network interface for nic_group 'MYGROUP': The specified address is already in use.","ok_to_retry":false}

Environment

• VMware Tanzu Operations Manager
• BOSH Director
• Infrastructure: vSphere, AWS, or Azure

Cause

This issue occurs when the private IP address assigned to the OpsManager or BOSH VM is still being held by a stale network interface (NIC) or another resource in the underlying infrastructure. Even if the previous VM has been deleted, the network interface may persist, preventing the CPI from creating a new NIC with that same IP address.

Resolution

To resolve this issue, manually clear the IP conflict in the infrastructure:

1. Identify the IP Address: Review the BOSH task logs to identify the specific IP address associated with the nic_group mentioned in the error (e.g., MYGROUP).
2. Locate Stale Resources:
   • In your IaaS console (vCenter, AWS Console, etc.), search for any existing Network Interfaces or "Zombies" assigned that IP.
   • Ensure there are no other VMs manually configured with that IP on the same subnet.
3. Delete the Interface: Manually delete the network interface associated with the BOSH private IP.
4. BOSH Database Cleanup (Optional): If the IP is still showing as "in use" within the BOSH database but not in the IaaS, you may need to manually clean the BOSH IP pool or reserve the IP in the BOSH Director tile settings to force the use of a different address.
5. Retry: Re-initiate the OpsManager upgrade or Apply Changes.

---

Additional Information

Related Article:

• BOSH Director is failing on creating a VM with the error "'Address 10.x.y.z is in use.'"