Post NSX-T upgrade, Transport Nodes are stuck in Partial Success

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• Upgrade was recently completed to NSX-T version 3.2.x.
• The Transport node profile (TNP) was changed.
• The Transport nodes (TN) goes into an NSX install failed state after changing the TNP.
• The TN status shows as Partial Success.
• On the NSX-T manager log /var/log/proton/nsxapi.log we see the following entries:

INFO L2HostConfigTaskExecutor4 TransportNodeStateServiceImpl 4665 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Incoming Revision: [11058] Obj: [TnStateInternal [id=<TN UUID>, retryCount=0, vmkMigrationFailures=0, revision=11058, stageToStatusMap={HostConfig=TnStageStatus [stageName=HostConfig, status=PARTIAL_SUCCESS, errorCode=8700, errorParams=[9a5faf0b-e0cd-4549-a5bc-d8e6359ff2f1, NSXA Vim not ready; 

• In the manager log /var/log/proxy/reverse-proxy.log we see the following entries:

INFO https-jsse-nio-10.218.34.91-443-exec-4 RegistrationTokenAuthenticationProvider 9298 AAA [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Successful login for reg-token <token UUID>, delegated user , roles (none)

• On the TN log /var/run/log/nsxcli.log we see the following entries:

31412205 cli.utils.apiclient ERROR POST /api/v1/fabric/nodes/<TN UUID>?action=register_node returned status: 403
...
2137855 cli.audit INFO CMD: join management-plane 10.34.45.14 thumbprint <thumb print> token <token-obfuscated> node-uuid <TN UUID> (duration: 8.769s), Operation status: CMD_EXECUTED_WITH_ERROR_RESULT
5987505 cli.descriptors.cli_command_service INFO {0} CMD: join management-plane <manager IP address> thumbprint <thumb print> token <token-obfuscated> node-uuid <TN UUID>
...
3041798 cli.utils.apiclient ERROR POST /api/v1/fabric/nodes/<TN UUID>?action=register_node returned status: 403
b'{\n "module_name" : "common-services",\n "error_message" : "The credentials were incorrect or the account specified has been locked.",\n "error_code" : 403\n}\n'
3041798 cli.commands.host_shared.register INFO Node registration failed, rc=403: None
3041798 cli.commands.host_shared.register INFO Error from MP API: The credentials were incorrect or the account specified has been locked.

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

During the upgrade, the API used to register the TN to the management plane, selected an invalid token which lead to this issue.

Resolution

This issue is resolved in NSX-T 3.2.2 available at VMware downloads.

Workaround:
If you believe you have encountered this issue, please open a support request with VMware NSX-T GSS and refer to this KB article.