TKGI cluster deployment fails for 1 or more node on NSX-T logical switch

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• You have TKGI deployed and use NSX-T for networking.
• When deploying a new TKGI cluster, this fails for one or more nodes.
  • For example, in a TKGI three node cluster, one node will succeed and one or both of the other two will fail to deploy.
• The NSX-T manager cluster consists of 3 appliances, each appliance can service API calls independently. 
• In one of the NSX-T manager's log /var/log/proxy/localhost_access_log.txt you see the following 500 HTTP response code:

2023-03-02T10:47:04.431Z 192.168.1.10 - "POST /api/v1/ns-groups/11ddb73f-544d-444e-83a3-400e928e411f?action=ADD_MEMBERS HTTP/1.1" 500 427 470 470

• In the same manager's log /var/log/proton/localhost_access_log.txt you see the following 500 http response code for the same group:

2023-03-02T10:47:04.431Z - "POST /nsxapi/api/v1/ns-groups/11ddb73f-544d-444e-83a3-400e928e411f?action=ADD_MEMBERS HTTP/1.1" 500 427 461 461

• On the other NSX-T manager's log /var/log/proxy/localhost_access_log.txt we see a 200 HTTP response code for the same group:

2023-03-02T10:47:04.264Z 192.168.1.10 - "POST /api/v1/ns-groups/11ddb73f-544d-444e-83a3-400e928e411f?action=ADD_MEMBERS HTTP/1.1" 200 34132 353 351

• And also in this log /var/log/proton/nsxapi.log

2023-03-02T10:47:04.262Z - "POST /nsxapi/api/v1/ns-groups/11ddb73f-544d-444e-83a3-400e928e411f?action=ADD_MEMBERS HTTP/1.1" 200 34132 342 342

Environment

VMware NSX-T

Cause

This issue occurs due to concurrent API calls to edit the same group. NSX-T manager incorrectly replies to one of the API calls with a 500 HTTP response code, which results in the TKGI cluster deployment failure, the response code should be 409, to allow the caller to retry.

Resolution

This is a known issue impacting NSX-T data center.

Workaround:
If you believe you have encountered this issue, please open a support request and reference this KB.