Persistent disk has been renamed after storage vMotion on vSphere

Products

• Operations Manager

Issue/Introduction

You recently performed a Storage vMotion vSphere on a Bosh VM managed that has got a persistent disk, then hit an issue when Bosh is updating/recreating this Bosh VM managed:

Inconsistent mount information:
Record shows that disk 'disk-62275e14-####-####-####-f56380c58c20' should be mounted on vm-c110804d-####-####-####-fc6115d50d30.
However it is currently :
Not mounted in any VM (mount_info_mismatch 37): Reattach disk and reboot instance (00:00:58)
L Error: Unknown CPI error 'Unknown' with message 'Failed to add disk 'scsi0:3'.' in 'attach_disk' CPI method (CPI request ID: 'cpi-350669')
Task 80644 | 10:43:39 | Error: Error resolving problem '11': Unknown CPI error 'Unknown' with message 'Failed to add disk 'scsi0:3'.' in 'attach_disk' CPI method (CPI request ID: 'cpi-350669')

Environment

Product Version: 2.10
OS: vSphere

Cause

In vSphere environment, the persistent disk created by Bosh is an independent disk that is not stored in the VM folder. Any persistent disk is stored in a vSphere folder  (OpsManager GUI -> Bosh director tile -> vCenter config -> Disk path Folder)

From Storage vMotion documentation 

"vCenter Server moves the virtual machine to the new storage location. Names of migrated virtual machine files on the destination datastore match the inventory name of the virtual machine."

From OpsManager documentation 

"If a datastore is part of a vSphere Storage Cluster using DRS storage (sDRS), you must deactivate the s-vMotion feature on any datastores used by Ops Manager. Otherwise, s-vMotion activity can rename independent disks and cause BOSH to malfunctionn"


How to prevent this behavior when a datastore migration is required:
We recommend following this procedure instead of doing manual Storage vMotion on the Bosh VMs with any persistent disk 

 

 

Resolution

How to recover:

The command "bosh instances --details" gives you the persistent disk name from the Bosh director database.
You need to rename the persistent disk to the original name and move it back to the original vSphere folder (OpsManager GUI -> Bosh director tile -> vCenter config -> Disk path Folder)

1. disable Bosh resurrector  bosh update-resurrection off

2. Using vSphere UI, shutdown the guest OS for the Bosh VM and detach the persistent disk

3. Rename the persistent disk via vSphere UI. In our example, the original persistent disk name was disk-62275e14-####-####-####-f56380c58c20 as per the output below:

Record shows that disk 'disk-62275e14-####-####-####-f56380c58c20' should be mounted

4. Move the persistent disk to the original vSphere folder (from OpsManager GUI -> Bosh director -> vCenter Config -> Disk path Folder)

5. Recreate this specific Bosh VM using bosh -d <DEPLOYMENT> cck

6. Re-enable Bosh resurrector bosh update-resurrection on