No valid placement found for VM compute and storage requirement when upgrading Opsman/BOSH
search cancel

No valid placement found for VM compute and storage requirement when upgrading Opsman/BOSH

book

Article ID: 448428

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid Integrated Edition VMware Tanzu Application Platform

Issue/Introduction

When upgrading Opsman/BOSH or update its stemcell  you may get the below error when deploying BOSH:

CPI 'create_vm' method responded with error: CmdError{"type":"Bosh::Clouds::CloudError","message":"No valid placement found for VM compute and storage requirement","ok_to_retry":false}

This can happen for multiple reasons please check KB 293771 first, and in case none of the symptoms and resolutions in the KB does not match, there is corner case where the following or similar scenario could have taken place

For example:

  1. In Bosh vcenter config with original configuration from persistent disk using datastores for example lun names [lun001, lun002, lun003] bosh director was originally created 
  2. At later stage the datastores were replaced with datastore cluster. Where same array of luns  are part of the datastore cluster [lun001, lun002, lun003] 
  3. Another change updates the luns and rotate with new luns - transtion process [lun001, lun002, lun003]   ==> [newlun001, newlun002, newlun003] 
  4. The old luns are placed in maintenance mode and are no longer in use
  5. Update bosh director stemcell you likely will see the error above and the bosh director vm will fail to be created.

Environment

Bosh 3.x 

Cause

Even the deployment process uses the correct datastore cluster filter the disk placement algorithm is also dependent on the CID details. 

In the context of BOSH and Tanzu Operations Manager, CID stands for Cloud Identifier (or Cloud ID).

Because the core BOSH Director engine is completely cloud-agnostic, it does not understand the native language of vSphere, AWS, or GCP. To bridge this gap, BOSH uses an abstraction layer called the CPI (Cloud Provider Interface).

When BOSH needs to create an infrastructure resource (like a VM, a persistent disk, or a network), it commands the CPI to talk to the IaaS. The cloud fabric (vCenter) provisions the asset and hands back a unique reference string. That string is the CID.

This CID is stored as-is in the director/deployment state. When attach_disk/placement runs, the CPI decodes that embedded, frozen pattern — not the current manifest's persistent_datastore_pattern — to decide two things: where to look for the disk, and whether its current location is "acceptable".

Because of the lun mismatch the pattern recorded in the CID (at the time of disk creation) - opsman deployment process fails to find suitable datastore and results in error.

Example from apply changes to bosh disk detach:

Started deploying
  Waiting for the agent on VM 'vm-eb105bbe-XXXX-XXXX-XXXX-bb3726e82d98'... Finished (00:00:00)
  Running the pre-stop scripts 'unknown/0'... Finished (00:00:00)
  Stopping jobs on instance 'unknown/0'... Finished (00:00:01)
  Running the post-stop scripts 'unknown/0'... Finished (00:00:00)
  Unmounting disk 'disk-868a87da-a0ad-4b68-97b1-8092043b5542.XihsdW4wMDF8bHVuMDAyfGx1bjAwMykkIn0lCg==... Finished (00:00:10)

 

Where decrypt of the CID looks like below:

echo "XihsdW4wMDF8bHVuMDAyfGx1bjAwMykkIn0lCg==" | base64 -d
^(lun001|lun002|lun003)$"}%

If all datastores from the old type are removed during deployment the system will attempt to move the persistent disk to one of the above luns and because it cannot find them as available the operation fails

Resolution

To restore bosh system there are two options: 

1. Bring one of the old luns back to the datastore cluster so the name matches and opsman deployment process be able to find placement 

2. Use option to enable persistent disk recreation (NOTE you might still have to complete option 1 in order to bring bosh VM up and running)

  • BOSH Director for vSphere -->  Director Config --> Recreate All Persistent Disks
  • Apply change only to bosh Director tile 

Additional Information

Due to the logs from opsman are relatively short with only above error, for troubleshooting purposes enable debug logging from BOSH Director for vSphere -->  Director Config --> Debug (from default)

This procedure will greatly increase the logs generated including the CPI logs which are critical to determine preciecely the error hidden behind the more generic error from the above

Once the logs are generated and collected revert the logging back to default.

Powered by Wolken Software