Unknown CPI error 'Unknown' with message '859: unexpected token at ''' in 'set_vm_metadata'

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

This issue appears intermittently during cluster creation in particular VM creation process  

Error message visible in the bosh task operation:

Unknown CPI error 'Unknown' with message '859: unexpected token at ''' in 'set_vm_metadata'

Environment

TKGi 1.19

TKGi 1.20 

NSX Compatible with above but lower than 4.2.x

 

Cause

This message is misleading and a better error handling is now taking place in the latest Opsman/Bosh 3.0.38.

This error originally comes from the NSX manager where a null response is registered which is then handled by bosh who is unable to decode the received message:

We have identified a significant reliance on Basic Authentication for the external IDP. It's not a direct NSX issue but it can overload the vIDM server.  NSX needs to check every time with vIDM which can lead to performance degradation duringActive directory verification process.

Failing API: 
./nsx_manager_/var/log/proxy/localhost_access_log.txt.2.gz 127.0.0.1 - "GET /api/v1/fabric/vifs?owner_vm_id=503d010f-a2f9-7dcd-bd5c-584de4e14770 HTTP/1.1" 500 629 65292 65292

 

Resolution

To mitigate this behaviour, it could help using local authentication mechanisms such as local users or Principal Identity, in any case, it is always much more efficient to use sessions to avoid repeated authentication.

To address this issue, in the fix 4.2 branch, an improvement have been  introduced: a cache mechanism for successful vIDM logins. This enhancement helps mitigate the situation by reducing the number of authentication requests sent to the IDP, improving efficiency, and optimizing performance.

Additionally Ops Manager/Bosh handles above type errors better and provide more details about why the CPI operation could not complete.  

Opsman/Bosh 3.0.38

NSX 4.2 

Alternatively using local users or Principal Identity instead of AD account for NSX authentication inside bosh director configuration can help prevent this error message from happening.

 

Additional Information

https://github.com/cloudfoundry/bosh-vsphere-cpi-release/releases/tag/v97.0.18