Persistent Disk is not Attached to the Director VM

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

The Director-Virtual Machine (VM) in your deployment no longer has its persistent disk (mounted to /var/vcap/store) attached.

An error message similar to the following appears:

/home/tempest-web/tempest/web/vendor/bundle/ruby/2.2.0/gems/bosh_vsphere_cpi-2.2.0/lib/
cloud/vsphere/cloud.rb:348:in `block in detach_disk': Disk 
'disk-a2971b65-8c2b-41f4-86e3-c0cbe6a98948' is not attached to VM 
'vm-351eb059-ccaa-4ad0-88de-8dac566bc6d5' (Bosh::Clouds::DiskNotAttached) 

 

Environment

Cause

This error can happen for a few different reasons. The most common causes would be if a user, script or automation at the Infrastructure as a Service (IaaS) level has specifically detached the disk from the virtual machine (VM) or if BOSH stopped or failed unexpectedly during a previous deployment.  Depending on the timing of that failure, this can leave BOSH in a state where it thinks the disk is still attached and will thus fail on subsequent attempts to detach the disk. 

This is not IaaS specific and it can happen to any BOSH-deployed VM with a persistent disk, not just the Director.

 

Resolution

Check if you can find the missing disk in your IaaS environment. If you browse each of your datastores and search for that name, you should be able to locate the disk. It is a detached disk, so it is separate from the Director VM.

Note- It is very important that this disk still exists.

If it does exist, please perform the following steps:

1. In your IaaS, power off old Director VM.
2. SSH to your Ops Manager VM. Run cd /var/tempest/workspaces/default/deployments.
3. Make a backup copy of bosh_deployments.yml and then edit it (edit the actual file, not the backup).
4. Look for the line vm_cid. This should have a Globally Unique Identifier (GUID), which is the name of the current Director VM in your IaaS. Delete that whole line. Save and close the file.
5. Go to the Ops Manager Web UI. Click "Apply Changes". This should create a new VM for the Director.
6. Update /var/tempest/workspaces/default/deployments/bosh-deployments.yml file with new vm_cid reference.
7. Once the install completes, firstly remove old Director VM from the disk and then remove the backup copy of bosh_deployments.yml.

If the steps do not resolve the problem, submit a ticket to Tanzu support at https://support.broadcom.com/