"Unknown CPI error 'Unknown' with message 'VM vm-xxx was expected in NSX-T but was not found'" error when creating new VM in BOSH

Products

• Operations Manager

Issue/Introduction

BOSH fails to create a new VM and shows the following error:

Task 1001 | 18:01:01 | Applying problem resolutions: VM for 'diego_cell/b111c9db-1234-41e5-93e3-31151378b174 (7)' missing. (missing_vm 130): Recreate VM and wait for processes to start (00:04:51)
  L Error: Unknown CPI error 'Unknown' with message 'VM vm-b111d01a-9f0d-49a6-9394-37854fc42886 was expected in NSX-T but was not found' in 'create_vm' CPI method (CPI request ID: 'cpi-209250')

This error means that BOSH vSphere CPI successfully created the VM on the ESX host but the NSX-T inventory did not get updated.

Environment

Product Version: 2.10

Resolution

This article outlines how to investigate this issue. Identify which ESX host was hosting the new VM from the BOSH task CPI output with the following command:

bosh task <ID> --cpi | grep 'Acquiring generic service ticket for URL'

For example:

$ bosh task 1001 --cpi | grep 'Acquiring generic service ticket for URL' I, [2021-06-03T18:01:01.848071 #4860] INFO -- [req_id cpi-831921]: Acquiring generic service ticket for URL: https://esx-host-11.lab.vmware.com/folder/vm_fd59ffb358da/env.json?dsName=vsanDatastore and Method: httpPut

Your infrastructure team should investigate the specific ESX host where the VM was created to identify any issues with the NSX-T inventory service and synchronization.