Disconnection between NSX Global Managers and NSX Local Managers post NSX upgrade from 3.x to 4.x in NSX federation setup

Products

• VMware NSX

Issue/Introduction

• When the LM is upgraded from 2.5 or 3.0.x or 3.1.x to 4.x LM-GM disconnection issue may be seen 

• Validate the below logs from the Global Manager: /var/log/vmware/appl-proxy-rpc.log

2024-05-23T09:02:31.851Z manager-node.example.com NSX 82660 - [nsx@6876 comp="global-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="82691" level="WARNING"] StreamConnection[162723 Connecting to ssl://#.#.#.#:1236 sid:162723] Couldn't connect to 'ssl://x.x.x.x:1236' (error: 336130315-wrong version number)
2024-05-23T09:02:31.851Z manager-node.example.com NSX 82660 - [nsx@6876 comp="global-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="82691" level="WARNING"] StreamConnection[162723 Error to ssl://#.#.#.#:1236 sid:-1] Error 336130315-wrong version number

• Validate the below logs from the Local Manager: /var/log/vmware/appl-proxy-rpc.log

2024-05-23T09:07:57.727Z <manager-node> NSX 1846 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-rpc" tid="1876" level="INFO"] Frame format is not recognized
2024-05-23T09:07:57.727Z <manager-node> NSX 1846 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-rpc" tid="1876" level="ERROR" errorCode="RPC400"] RpcConnection[166221 Negotiating on tcp://0.0.0.0:1236 0] Frame format is not recognized

2024-05-23T09:08:05.749Z <manager-node> NSX 1846 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="1876" level="INFO"] StreamSocket[166229 Closing f:69 i:257659720 tcp://0.0.0.0:1236 <- #.#.#.#:45080] DoClose
2024-05-23T09:08:06.751Z <manager-node> NSX 1846 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="1876" level="INFO"] StreamConnection[166230 Connected on tcp://0.0.0.0:1236 sid:166230] Accepted connection from tcp://#.#.#.#:45094

 

• Validate if the below highlighted lines (in bold) are present in the file "/etc/vmware/nsx-appl-proxy/appl-proxy.xml" 

     <applProxyPublicCfgFile>/etc/vmware/nsx-appl-proxy/appl-proxy-public-cfg.xml</applProxyPublicCfgFile>
     <applProxyPrivateKeyFile>/etc/vmware/nsx-appl-proxy/appl-proxy-privkey.pem</applProxyPrivateKeyFile>
     <applProxyCertificateFile>/etc/vmware/nsx-appl-proxy/appl-proxy-cert.pem</applProxyCertificateFile>
     <applProxyArPrivateKeyFile>/etc/vmware/nsx-appl-proxy/appl-proxy-ar-privkey.pem</applProxyArPrivateKeyFile> 
     <applProxyArCertificateFile>/etc/vmware/nsx-appl-proxy/appl-proxy-ar-cert.pem</applProxyArCertificateFile>

<external_ar>
<ip>0.0.0.0</ip>
<ipv6>::</ipv6>
<!-- <fqdn>localhost</fqdn> →
<!-- <fqdnv6>localhost</fqdnv6> →
<port>1236</port>
<!-- <path>unix:///tmp/aphexternal.sock</path> →
<sslEnabled>true</sslEnabled>
</external_ar>

• For Greenfield deployment of 3.x OR 3.x >> 4.x NSX upgrade, this issue is not likely to hit.

Environment

VMware NSX-T Data Center 3.x
VMware NSX 4.x

Cause

• "/etc/vmware/nsx-appl-proxy/appl-proxy.xml" file is not getting upgraded to the intended install version. 
• From NSX 4.x onwards, we read the sslEnabled value for external_ar from appl-proxy.xml file, and if not present default value is sslEnabled=false, leading to Global Manager-Local Manager sync disconnects.

Resolution

The fix is in upcoming NSX releases
For workaround to overcome this situation, please contact Broadcom Support

Below are the logs needed:

• All Global Manager support bundles (including Active and Standby)
• All Local Managers support bundles (including all Locations)
• "/etc/vmware/nsx-appl-proxy/appl-proxy.xml" file from all the above nodes