BOSH Director is failing on creating a VM with the error "'Address 10.x.y.z is in use.' in 'create_vm' CPI method" in a TKGi environment on AWS

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

BOSH Director is consistently failing on creating a VM with the error "'Address 10.x.y.z is in use.' in 'create_vm' CPI method" in a TKGi environment on AWS.

Task 110999 | 20:30:00 | Creating missing vms: worker/99999999-131c-4cec-9520-99999999999 (19) (00:09:59)
                       L Error: Unknown CPI error 'Unknown' with message 'Address 10.x.y.z is in use.' in 'create_vm' CPI method (CPI request ID: 'cpi-903999')
Task 110999 | 20:30:00 | Error: Unknown CPI error 'Unknown' with message 'Address 10.x.y.z is in use.' in 'create_vm' CPI method (CPI request ID: 'cpi-903999')

The issue persists even after performing the steps in the following KB.

https://knowledge.broadcom.com/external/article?articleNumber=293831

 

 

Environment

VMware Tanzu Kubernetes Grid Integrated Edition

Cause

The particular IP address, referenced in the error message, is being used somewhere in the AWS environment, and was not assigned by/through BOSH.  To help with identifying what uses the particular IP address, the following AWS documentation can be followed as a guide.

https://repost.aws/knowledge-center/vpc-find-owner-unknown-ip-addresses

 

Resolution

Once the device/resource (using the particular IP address) has been identified, then a decision has to be made to either un-assign the IP address or exclude it in BOSH so that it won't be assigned to any VM.

The following steps can be followed to exclude it in BOSH and resolve the issue. 

1. Delete the particular IP address 10.x.y.z from the BOSH database by following the steps from this KB:
   https://knowledge.broadcom.com/external/article?articleNumber=293831
   
   
2. Reserve the particular IP address 10.x.y.z in the BOSH Director tile settings, specifically in the Create Networks tab, and then Apply Changes.  You can select just the Bosh Tile, when applying the changes.
   
   
3. Retry the BOSH/TKGI operation (deploy, upgrade, update, etc.) that failed previously.