BOSH Director Apply Changes fails with "Address <IP-ADDRESS> is in use."

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

It may happen that during the upgrade of the BOSH director Tile, Apply Changes fails with following error:

Deploying:
  Creating instance 'bosh/0':
    Creating VM:
      Creating vm with stemcell cid 'ami-######':
        CPI 'create_vm' method responded with error: CmdError{"type":"Unknown","message":"Address ###.###.###.### is in use.","ok_to_retry":false}
Exit code 1
===== 2022-04-02 11:24:01 UTC Finished "/usr/local/bin/bosh --no-color --non-interactive --tty create-env /var/tempest/workspaces/default/deployments/bosh.yml"; Duration: 629s; Exit Status: 1
{"type": "step_finished", "id": "bosh_product.deploying"}
Exited with 1.

 

 

Environment

Product Version: 2.10

Resolution

This usually happens because there is another VM using same IP. The solution for that would be to either delete or move the other VM to a different network.

But we have also seen this issue and there is actually no duplicate IP address. In this case, what is causing the issue is "current_vm_cid" field being empty in /var/tempest/workspaces/default/deployments/bosh-state.json. To solve this issue just do the following - 

• Take a backup of bosh-state.json:
  

  cd /var/tempest/workspaces/default/deployments
  cp bosh-state.json bosh-state.json.bkp

• Modify the original bosh-state.json by adding the bosh director VM id (Instance ID) from AWS console to "current_vm_cid". After modification it should look like:
  

   "current_vm_cid": "i-######",

• Attempting to "Apply Changes" should now succeed without error.