Disconnection between NSX Global Managers and NSX Local Managers post NSX upgrade from 3.x to 4.x in NSX federation setup
search cancel

Disconnection between NSX Global Managers and NSX Local Managers post NSX upgrade from 3.x to 4.x in NSX federation setup

book

Article ID: 369820

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • NSX-T Federation sites are being upgraded to NSX 4.1.2.3
  • The connections between GM and LM are down and no global object is sync to the LM site.
  • After re-entering the LM site credential under location manager, the connection status remained disconnected. 
  • From the UI, there maybe error message as "unable to fetch full sync status" along with other error message showing cannot fetch status for objects such as TransportZoneListResultDto and error message as "500139". 
  • Validate the below logs from the Global Manager: /var/log/vmware/appl-proxy-rpc.log

2024-05-23T09:02:31.851Z manager-node.example.com NSX 82660 - [nsx@6876 comp="global-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="82691" level="WARNING"] StreamConnection[162723 Connecting to ssl://#.#.#.#:1236 sid:162723] Couldn't connect to 'ssl://#.#.#.#:1236' (error: 336130315-wrong version number)
2024-05-23T09:02:31.851Z manager-node.example.com NSX 82660 - [nsx@6876 comp="global-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="82691" level="WARNING"] StreamConnection[162723 Error to ssl://#.#.#.#:1236 sid:-1] Error 336130315-wrong version number

  • Validate the below logs from the Local Manager: /var/log/vmware/appl-proxy-rpc.log

    2024-05-23T09:07:57.727Z <manager-node> NSX 1846 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-rpc" tid="1876" level="INFO"] Frame format is not recognized
    2024-05-23T09:07:57.727Z <manager-node> NSX 1846 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-rpc" tid="1876" level="ERROR" errorCode="RPC400"] RpcConnection[166221 Negotiating on tcp://#.#.#.#:1236 0] Frame format is not recognized

         2024-05-23T09:08:05.749Z <manager-node> NSX 1846 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="1876" level="INFO"] StreamSocket[166229 Closing f:69 i:257659720 tcp://0.0.0.0:1236 <- #.#.#.#:45080] DoClose
    2024-05-23T09:08:06.751Z <manager-node> NSX 1846 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="1876" level="INFO"] StreamConnection[166230 Connected on tcp://0.0.0.0:1236 sid:166230] Accepted connection from tcp://#.#.#.#:45094

  • Validate if the below highlighted lines are present in "/etc/vmware/nsx-appl-proxy/appl-proxy.xml" for both GM and LM:

     <applProxyPublicCfgFile>/etc/vmware/nsx-appl-proxy/appl-proxy-public-cfg.xml</applProxyPublicCfgFile>
     <applProxyPrivateKeyFile>/etc/vmware/nsx-appl-proxy/appl-proxy-privkey.pem</applProxyPrivateKeyFile>
     <applProxyCertificateFile>/etc/vmware/nsx-appl-proxy/appl-proxy-cert.pem</applProxyCertificateFile>
     <applProxyArPrivateKeyFile>/etc/vmware/nsx-appl-proxy/appl-proxy-ar-privkey.pem</applProxyArPrivateKeyFile> 
     <applProxyArCertificateFile>/etc/vmware/nsx-appl-proxy/appl-proxy-ar-cert.pem</applProxyArCertificateFile>

<external_ar>
<ip>0.0.0.0</ip>
<ipv6>::</ipv6>
<!-- <fqdn>localhost</fqdn> →
<!-- <fqdnv6>localhost</fqdnv6> →
<port>1236</port>
<!-- <path>unix:///tmp/aphexternal.sock</path> →
<sslEnabled>true</sslEnabled>
</external_ar>

Environment

VMware NSX-T Data Center 3.x
VMware NSX 4.1.2.3

Cause

  • "/etc/vmware/nsx-appl-proxy/appl-proxy.xml" file is not getting upgraded to the intended install version after the upgrade. 
  • From NSX 4.x onwards, the sslEnabled value for external_ar is read from appl-proxy.xml file, and if not present, default value is sslEnabled=false, leading to Global Manager-Local Manager sync disconnects.

Resolution

This issue is fixed in VMware NSX 4.2.1 or above. Customer using Federation should upgrade their environment to NSX 4.2.1 or above to avoid this issue or use the below workaround. 

 

Workaround: 

Note: Replace /etc/vmware/nsx-appl-proxy/appl-proxy.xml with new file "appl-proxy.xml_4.1.2.3" from this KB attachment, and new file needs to be same name as appl-proxy.xml

 

If workaround did not resolve the issue or if the version affected isn't 4.1.2.3, please contact Broadcom Support and upload the following log files to the case. 

  • All Global Manager support bundles (including Active and Standby)
  • All Local Managers support bundles (including all Locations)
  • "/etc/vmware/nsx-appl-proxy/appl-proxy.xml" file from all the above nodes

 

Attachments

appl-proxy.xml_4.1.2.3 get_app
Powered by Wolken Software