TAS VM missing and cannot be updated due to detatched state.

Products

• Operations Manager

Issue/Introduction

When running bosh vms, it may be observed that there is an instance of a VM missing. Usually an Apply Changes or bosh cck would recover this missing VM, but in this case these operations don't pick up on this.

Running bosh is --details instead shows that the VM is in a detatched state.
 

diego_cell/a26be528-b6f5-40d6-b4bf-3f184be93ac1 - detatched

Environment

Product Version: 2.10

Resolution

In some instances it may be possible to force the instance out of the detached state by running  bosh start on the instance:

bosh -d <DEPLOYMENT NAME> start <INSTANCE>/<GUID> --no-converge

In the event that the above does not work, the state may be updated manually in the database. To do this, we will use the BOSH director console:

• SSH into the BOSH director
• Elevate to root privileges
• Start the BOSH console:

/var/vcap/jobs/director/bin/console

# Will result in a terminal similar to:
irb(main):001:0>

• Once in the terminal, retrieve the list of instances in a detached state:

Bosh::Director::Models::Instance.where(state: "detached").all.map{|n| "#{n.uuid} #{n.job} #{n.index}"}

# Example output:
> ["b87d02d7-54d6-4c9c-826f-f8b08937dd61 diego_cell 1", "fb4ff31a-a059-4c50-b330-0d2549258eba diego_cell 0"]

• The output will retrieve a comma separated list of instance GUID, associate job name, and the instance index
• Verify that the list of GUIDs returned matches the expected instances that are to be updated
• If all of the GUIDs match, proceed with the following to update all detached instances:

Bosh::Director::Models::Instance.where(state: "detached").update(state: "started")

• If there are GUIDs that do not match the expected instances, each can be updated individually by replacing <GUID> in the following with the desired GUID:

Bosh::Director::Models::Instance.where(uuid: "<GUID>").first.update(state: "started")

Afterward, proceed with an Apply Changes or other BOSH operation as normal.