Bosh create_vm errors out with "The virtual machine version is not compatible with the version of the host"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

During the creation of a VM, when BOSH tries to clone the stemcell, the BOSH task fails with an error:

Task 279 | 17:33:45 | Creating missing vms: worker/cacf218c-904e-421f-96dd-692d0eeceabf (1) (00:00:58)
                    L Error: Unknown CPI error 'Unknown' with message 'The virtual machine version is not compatible with the version of the host '172.18.79.11'.' in 'create_vm' CPI method (CPI request ID: 'cpi-812194')
Task 279 | 17:34:43 | Error: Unknown CPI error 'Unknown' with message 'The virtual machine version is not compatible with the version of the host '172.18.79.11'.' in 'create_vm' CPI method (CPI request ID: 'cpi-812194')

The underlying root cause of this problem is currently under investigation but this can likely occur due to stemcell corruption or underlying datastore or ESXi issues.

From the vSphere client UI, the following failures can be observed in the recently failed tasks:

Environment

Product Version: 1.9
OS: Ubuntu

Resolution

To resolve this problem, the following steps can be followed:

• Use the bosh stemcells command to identify the stemcell version associated with the stemcell id in the error
• Use bosh upload-stemcell <stemcell> --fix to reupload the stemcell 

Once the upload is complete the VM creation should succeed now.