Upgrade Apply Changes errors out with: "Error: Unknown CPI error 'Unknown' with message 'undefined method `datastore'" on vSphere

Products

• Operations Manager

Issue/Introduction

When running "Apply Changes" in Operations Manager (Ops Manager), BOSH fails to attach persistent disk during upgrades. It outputs the following error message: 

Unknown CPI error 'Unknown' with message 'undefined method `datastore' for #<VimSdk::Vim::Vm::Device::VirtualCdrom::RemotePassthroughBackingInfo:0x0000560d4a903f28>'

OSH fails to attach persistent disk during upgrades. "Apply Changes" fails and errors out with:

Task 2395646 | 05:37:01 | Updating instance worker: worker/1aa8a864-6b0c-4e73-b712-6a7662e71184 (14) (00:03:32)
           L Error: Unknown CPI error 'Unknown' with message 'undefined method `datastore' for #VimSdk::Vim::Vm::Device::VirtualCdrom::RemotePassthroughBackingInfo:0x0000560d4a903f28>' in 'detach_disk' CPI method (CPI request ID: 'cpi-277333')
Task 2395646 | 05:40:33 | Error: Unknown CPI error 'Unknown' with message 'undefined method `datastore' for #VimSdk::Vim::Vm::Device::VirtualCdrom::RemotePassthroughBackingInfo:0x0000560d4a903f28>' in 'detach_disk' CPI method (CPI request ID: 'cpi-277333')

 

Cause

During upgrades, the Virtual Machines (VMs) gets restarted. While the VMs restart, BOSH checks for persistent disk on certain VMs. When the disk is not attached to the CD-ROM in vCenter, it fails.

In some cases, attempting to move a server to maintenance mode fails if the CD-ROM has an env.iso file attached. To get around this you need to manually detach the env.iso file. However, this workaround is temporary and this issue will reoccur the next time BOSH tries to rebuild the VMs during an upgrade.

Below is an image to show the task when the VM is reconfigured in vCenter.

Environment

Product Version: 2.7

Resolution

From a high level, to fix this issue, you need to:

• Connect to vCenter and identify the affected VM.
• Reconfigure the CD-ROM or DVD drive back to "<VM-CID>/env.iso" for affected VMs.
• Then "Apply Changes".

Below are detailed instructions on how to fix this issue:

1. Connect to vCenter and identify the affected VM. For example: "bosh instances --details | grep worker/1aa8a864-6b0c-4e73-b712-6a7662e71184"
2. Change the CD-ROM or DVD drive to a Datastore ISO file.

• CD-ROM or DVD: Click on browse and choose  "<VM-CID>/env.iso" file from the list that corresponds to the VM. If you cannot find the env.iso file inside the VM folder, shutdown the VM from the vSphere UI and let BOSH resurrector recreate the unresponsive VM. If BOSH resurrector is not enabled in BOSH director tile, run the command bosh -d <deployment> cck against the deployment where is located the VM and choose to recreate the unresponsive VM.
• The VM-CID can be identified using the above bosh instances command. 

3. Then click "Apply Changes" in Ops Manager to continue with your upgrade.