BOSH Director Apply Changes fails with "Address <IP-ADDRESS> is in use."

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

It may happen that during the upgrade of the BOSH director Tile, Apply Changes fails with following error.

Deploying:
  Creating instance 'bosh/0':
    Creating VM:
      Creating vm with stemcell cid 'ami-abcdefghijklmn':
        CPI 'create_vm' method responded with error: CmdError{"type":"Unknown","message":"Address www.xxx.yyy.zzz. is in use.","ok_to_retry":false}
Exit code 1
===== 2022-04-02 11:24:01 UTC Finished "/usr/local/bin/bosh --no-color --non-interactive --tty create-env /var/tempest/workspaces/default/deployments/bosh.yml"; Duration: 629s; Exit Status: 1
{"type": "step_finished", "id": "bosh_product.deploying"}
Exited with 1.

This usually happens because there is another VM using same IP. The solution for that would be to either delete or move the other VM to a different network.

But we have also seen this issue and there is actually no duplicate IP address. In this case, what is causing the issue is "current_vm_cid" field being empty in /var/tempest/workspaces/default/deployments/bosh-state.json. To solve this issue just do the following,

Take a backup of bosh-state.json:

cd /var/tempest/workspaces/default/deployments
cp bosh-state.json bosh-state.json.bkp

Modify the original bosh-state.json by adding the bosh director VM id from AWS console to "current_vm_cid". After modification it should look like:

 "current_vm_cid": "ami-opqrstuvwxyz",

Attempting to "Apply Changes" should now succeed without error.

 

Environment

Product Version: 2.10