How to Recover the out of Sync VMs on vSphere from a Failed BOSH Deployment

Products

• Operations Manager

Issue/Introduction

This article will assist in bringing back virtual machines (VMs) that are out of sync after a failed deployment.

'bosh cck' will show the following where a disk is not mounted on the problem VM:

 

Environment

Cause

A BOSH deployment can fail to cause the VMs' to go out of sync:

Started preparing deployment > Binding existing deployment. Failed: VM `vm-ef777cc9-9780-4a9e-a4f1-dbfe9bbd2e5a' is out of sync: expected to be a part of deployment `cf-9c65e8a93ceb573a2c18' but is actually a part of deployment `' (00:00:01)
Error 400003: VM `vm-ef777cc9-9780-4a9e-a4f1-dbfe9bbd2e5a’ is out of sync: expected to be a part of deployment `cf-9c65e8a93ceb573a2c18' but is actually a part of deployment `'

 

Resolution

Below are the steps to resolve this issue:

1. Shutdown the vm in your IaaS.

2. Run the following the following command to list the VM details:

    'bosh instances --details':

3. Ensure that the Disk CID in the 'unknown' instance (shown in the screenshot above) matches the 'Disk File' in the VM in vSphere:

4. If the disk CIDs' matches, run 'bosh cck' and select option 4:'Delete VM reference'. This will delete the reference for this VM from the microbosh database (Note: This will not remove the VM from your infrastructure): 

5. Once completed, run 'bosh deploy' to continue with the deployment.