Unable to update HW VTEP properties via the PUT /hardwaregateways/{id} API due to the error "Credential is configured with another transport node"

Products

• VMware NSX

Issue/Introduction

Symptoms:
Cannot update ToR Gateway properties via the PUT API, for e.g., to change its replication cluster. The API returns a 400 response "Credential is configured with another transport node."  However, other API's such as adding and removing bindings are successful.

In vsm.log, you see entries similar to:

2020-05-19 15:19:58.225 EDT ERROR http-nio-127.0.0.1-7441-exec-22 NvpRestClientManagerImpl:994 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] NVP API returns error: [400] <h1>400 Bad Request</h1>Creden
tial is configured with another transport node.
2020-05-19 15:19:58.227 EDT ERROR http-nio-127.0.0.1-7441-exec-22 TorServiceImpl:274 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Failed to update a TOR Gateway
org.springframework.web.client.HttpClientErrorException: 400 Bad Request
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:86) ~[spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:708) ~[spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:661) ~[spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:621) ~[spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:539) ~[spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at com.vmware.vshield.vsm.vdn.nvpcontroller.service.impl.NvpRestClient.sendRequestOnce(NvpRestClient.java:313) ~[vsphere-1.0.jar:?]
        at com.vmware.vshield.vsm.vdn.nvpcontroller.service.impl.NvpRestClient.sendRequestEx_aroundBody4(NvpRestClient.java:281) ~[vsphere-1.0.jar:?]
        at com.vmware.vshield.vsm.vdn.nvpcontroller.service.impl.NvpRestClient$AjcClosure5.run(NvpRestClient.java:1) ~[vsphere-1.0.jar:?]

Cause

This is caused in environments that are upgraded from 6.4.3 or prior versions to 6.4.4 or later.

While servicing many requests related to hardware gateways, NSX makes a call to the controller to complete the request. The controller validates the call by performing a ToR UUID validation check against its own internal table. Prior to NSX-v 6.4.4, this validation check was much looser and the ToR UUID was essentially a random UUID. But in 6.4.4 and later, the validation requires the ToR UUID to be a function of its certificate. Therefore, when upgrading from 6.4.3 to 6.4.4, the ToR UUIDs (which were random in 6.4.3) fail the validation on some call paths in the controller. This results in failure at the controller and a corresponding failure of the call at NSX.

Resolution

Currently, there is no resolution.

Workaround:
The only solution currently is to delete the ToR node and re-add it. Note that a ToR can be deleted only if it has no bindings. So all its bindings must be removed first. When the same ToR is re-added, a new UUID will be generated for it on the controller, which will meet the validation that the controller expects on subsequent calls. The new UUID is propagated back to NSX where it's stored in its database. At that point further operations on the ToR should work.