"Unknown CPI error" during Bosh deploy caused by "The operation is not allowed in the current state of the host"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

• Bosh operations that clone new VMs from Stemcells fail on the 'create_vm' step
• This might impact Bosh cck, recreate, or create operations.
• The Bosh task will show failures with messaging like:
  
  Error: Unknown CPI error 'Unknown' with message 'The operation is not allowed in the current state of the host.' in 'create_vm'
  
  
• When viewing the bosh task with --cpi flag, the following errors will be presented:
  
  Error running task 'VirtualMachine.clone'. Failed with message 'The operation is not allowed in the current state of the host.' and fault message ''.
Error in creating vm: The operation is not allowed in the current state of the host
  
  
• These errors will also be presented in the vCenter GUI

Cause

This problem occurs due to connectivity failures from vCenter to the hostd daemon on the ESXi host on which the backing Bosh Stemcell is registered. While there are numerous causes for host to vCenter connectivity, KB 318013 provides one example of an issue that can impact hostd in a way that leaves the host in a state that doesn't appear degraded in vCenter.

Resolution

1. Use the following command with the bosh task ID that is failing to identify the stemcell VM ID:
   
   bosh task <TASK_ID> --cpi | grep "stemcell"
   
   
2. Search for the stemcell from step 1 in the vCenter GUI to identify which ESXi host the stemcell VM resides on.
3. Attempt a storage vMotion to move the stemcell to another ESXi host that isn't degraded.
4. Correct the ESXi degradation if the VM can't be vMotioned.