VM creation fails with "Unknown CPI error 'Unknown' with message 'VM vm-<id> was expected in NSX-T but was not found' in 'set_vm_metadata' CPI method (CPI request ID: 'cpi-id')"

Products

• Operations Manager

Issue/Introduction

During a VM creation, Apply Changes or BOSH Healthwatch etc. for Tanzu Application Service (TAS) with NSX-T fails with the following error message:

Task <id> | 04:12:04 | Compiling packages: genx/<id> (00:11:03)
L Error: Unknown CPI error 'Unknown' with message 'VM vm-<id> was expected in NSX-T but was not found' in 'set_vm_metadata' CPI method (CPI request ID: 'cpi-id')

Environment

Product Version: 2.7

Resolution

To get more information about the failure, perform the following command from the BOSH CLI:

bosh -e <environment> -d <deployment> task  <task number> --cpi

Check the output for login return values to determine the time it takes the user to login to VCenter:

[2020-02-12T17:47:08.822283 #id] DEBUG -- [req_id cpi-id]: Running method 'Login'...
I, [2020-02-12T17:48:38.963541 #23141]  INFO -- [req_id cpi-id]: Finished info in 93.1 seconds
 
 [2020-04-12T17:50:58.517503 #id] DEBUG -- [req_id cpi-id]: Running method 'Login'...
I, [2020-04-12T17:54:28.694825 #id]  INFO -- [req_id cpi-id]: Finished info in 240.22 seconds

Note: The expected login times are around or under 1 second. If this is the case, this slow login time will cause vSphere and NSX CPI to conflict and lead to creating objects in an unexpected order. 

Workaround

Slow login times must be addressed on IaaS level and the issue could range from the following:

• Possible network latency
• VCenter resource congestion
• VC - External PSC issue
• Active Directory misbehaving for AD users
• Other unknown causes