Deploying bosh director tile fails with Error connecting to NSX IP: got HTTP response with status code 500', type: IaasConfigurationVerifier

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Due to cluster restructuring Tanzu foundation had to be redeployed.

During initial configuration an error occurred: "Error connecting to NSX IP: got HTTP response with status code 500', type: IaasConfigurationVerifier"

All required elements have been configured

Environment

TKGi 1.23

Opsman 3.0.41 +

Cause

There could be multiple reasons on why the configuration would fail like typo or a unexpected character however the error is specifically related to section vCenter config under Bosh Director tile.

From the support bundle the requests to update the vCenter config:

Processing by Infrastructure::IaasConfigurationsController#update as HTML

Leads to non error message:

Completed 422 Unprocessable Content

It is not clear why the error would occur, and also is not visible any 500 error messages in the logs.

The process IaasConfigurationVerifier is verifying the configuration to vcenter as well as if the vcenter is configured with NSX it will verify the configuration of the NSX.

As part of this verification the process tries to connect to NSX and run basic query which potentially can result in error (500 internal server error)  

In this specific situation the generated superuser certificate was not correctly registered to NSX following nsxt-generate-pi-cert procedure

To verify the NSX is responding as expected below two curl requests can be used:

curl -v --cacert nsx_t_ca.crt --cert nsx_t_superuser.crt --key nsx_t_superuser.key https://nsxmgrxFQDN/policy/api/v1/infra

curl -v --cacert nsx_t_ca.crt --cert nsx_t_superuser.crt --key nsx_t_superuser.key https://nsxmgrxFQDN/api/v1/node

Where:

nsx_t_ca.crt               - NSX CA cert configured on the tile and NSX manager 

nsx_t_superuser.crt   - Superuser Certificate

nsx_t_superuser.key - Superuser key

nsxmgrxFQDN           - NSX manager FQDN

Resolution

Complete the registration of the superuser certificate to NSX manager.