NSX Federation Onboard fails for APH

Products

• VMware NSX

Issue/Introduction

After upgrade to 4.1.1 version, some customers may have to replace expired/expiring certificates. If the APH cert replacement was not done properly, communication issues between GM and LM sites may arise, leading to Federation onboarding failures.

You may see similar log entries:

2024-01-29T19:17:34.922Z manager-node-ID NSX 1240 - [nsx@6876 comp="global-manager" subcomp="ip_utils" username="nsx-sha" level="WARNING" s2comp="rpc-client" invalid="true"] Event RpcStubCreateEvent(stub=ShaOdsService_Stub, retry=True): Exception in initializing RPC client for ShaOdsService_Stub: Onboard fails for APH APH_UUID
2024-01-29T19:17:35.488Z manager-node-ID  NSX 2391 - [nsx@6876 comp="nsx-manager" errorCode="MP403" level="ERROR" subcomp="http"] No credentials passed. Disallowing the call.
2024-01-29T19:17:35.488Z manager-node-ID  NSX 2391 - [nsx@6876 comp="nsx-manager" errorCode="MP403" level="ERROR" subcomp="http"] The credentials were incorrect or the account specified has been locked.

appl-proxy-rpc.log:6:2024-01-29T19:13:01.950Z manager-node-ID NSX 1490 - [nsx@6876 comp="global-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="1533" level="INFO"] StreamSocket[611706 Open f:64 i:526113441 ? -> ssl://NSX-LM-IP:1236] on_connect 335544539-short read
appl-proxy-rpc.log:7:2024-01-29T19:13:01.950Z gmnsxtmanager3 NSX 1490 - [nsx@6876 comp="global-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="1533" level="WARNING"] StreamConnection[611706 Connecting to ssl://NSX-LM-IP:1236 sid:611706] Couldn't connect to 'ssl://NSX-LM-IP:1236' (error: 335544539-short read

Environment

NSX 4.1.1

Cause

The APH information is incorrect and communication is broken across the sites.

Resolution

The resolution is to either remove the affected LM site(s) from GM then onboard it again, or you can use the API call below to rejoin the LM to GM:

 

First, get the GM node thumbprint:

admin>: get certificate api thumbprint

Second, send an POST API call 

POST https://<Active GM node IP>/api/v1/sites?action=onboard_site

with the following inside the body of the API call. 

{

      “address”: “LM node IP”,

      “username”: “admin”,

      “password”: “password”,

      “thumbprint”: “GM node thumbprint”,

      “site_name”: “site name”

}