Unknown CPI error related to missing properties "summary.storage.committed" when creating VM

Products

VMware Tanzu Application Service

Issue/Introduction

This error message can be seen during VM deployments in Apply Changes. The error message can be found with other error messages like missing snapshot errors as seen below:

Task ######## | hh:mm:ss | Compiling packages: envoy_windows/################################################################ (00:04:34)
                         L Error: Unknown CPI error 'Unknown' with message 'The object[s] '&lt;[Vim.VirtualMachine] vm-#######&gt;' should have the following properties: ["summary.storage.committed"]
, but they were missing these: #&lt;Set: {"summary.storage.committed"}&gt;
.' in 'create_vm' CPI method (CPI request ID: 'cpi-######')
Task ######## | hh:mm:ss | Compiling packages: ccg-plugin/################################################################ (00:04:34)
                         L Error: Unknown CPI error 'Unknown' with message 'The object[s] '&lt;[Vim.VirtualMachine] vm-#######&gt;' should have the following properties: ["snapshot"]
, but they were missing these: #&lt;Set: {"snapshot"}&gt;
.' in 'create_vm' CPI method (CPI request ID: 'cpi-######')

Environment

TPCF on vSphere

Cause

When a stemcell is deployed, it is uploaded to a datastore and gets replicated to other datastores. When VMs are deployed using a specific stemcell, a clone of this stemcell VM is generated. If there are metadata inconsistencies, this cloning process can fail.

You can see evidence of the cloning issues by obtaining cpi logs from the failed task:

bosh task <task_id> --cpi

The messages can look similar to the example below:

W, [YYYY-MM-DDThh:mm:ss.269743 ########]  WARN -- [req_id cpi-######]: Error running task 'VirtualMachine.clone'. Failed with message 'The name 'sc-########-####-####-####-############ / <datastore_name>' already exists.' and fault message ''.
I, [YYYY-MM-DDThh:mm:ss.269743 ########]  INFO -- [req_id cpi-######]: Stemcell is already being replicated, waiting for sc-########-####-####-####-############ %2f <datastore_name> to be ready

Resolution

Checked the following metadata using govc commands.

To install govc you can run the following commands. You may have to modify this procedure if you do not have internet access directly from the OpsMan/jumbox host. Change the user, password, and vCenter hostname in the GOVC_URL to match yours.

wget https://github.com/vmware/govmomi/releases/download/v0.52.0/govc_Linux_x86_64.tar.gz
export GOVC_INSECURE=true
export GOVC_URL='https://[email protected]:[email protected]'

Once installed check the following information.

VM info that bosh cpi is looking for:

govc vm.info -json=true "/NAME_OF_YOUR_DATACENTER/vm/PATH/TO/YOUR/STEMCELL/VM_NAME/sc-########-####-####-####-############ %2f <datastore_name>"

Value of property "summary.storage.committed":

govc vm.info -json=true "/NAME_OF_YOUR_DATACENTER/vm/PATH/TO/YOUR/STEMCELL/VM_NAME/sc-########-####-####-####-############ %2f <datastore_name>"  | jq '.virtualMachines[] | .summary.storage.committed'

Snapshot status:

govc vm.info -json=true "/NAME_OF_YOUR_DATACENTER/vm/PATH/TO/YOUR/STEMCELL/VM_NAME/sc-########-####-####-####-############ %2f <datastore_name>"  | jq '.virtualMachines[] | .snapshot'

If you are able to retrieve all this information and the values for summary.storage.committed and the snapshot status are valid this means vSphere metadata got corrupted.

Proceed to re-upload the stemcell using command:

bosh upload-stemcell <path-to-stemcell-file.tgz> --fix

Try executing the original deployment again, this time the new stemcell upload should have corrected the metadata inconsistencies and the VMs should be deployed correctly.