Disconnection between NSX Global Managers and NSX Local Managers post NSX upgrade from 3.x to 4.x despite following KB 369820

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX-T Federation sites are being upgraded to NSX 4.1.2.3
• The connections between GM and LM are down and no global object is sync to the LM site.
• Customer performed action as per KB 369820, however, the connections remains down.
• Local Managers GUI displayed that CONTROLLER and MESSAGING-MANAGER services are down.
• The /var/log/messaging-manager/messaging-manager.log file on local managers contains the following fault:

YYYY-MM-DDTHH:MM:SS INFO unix:MESSAGING_TNCONNCFGSVC_CLIENT:worker-0 AbstractTransportProtocol - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="messaging"] unix:MESSAGING_TNCONNCFGSVC_CLIENT: Failed to connect to /var/run/vmware/appl-proxy/aphservice.sock because of FileNotFoundException:

• The /var/log/messaging-manager/appl-proxy.log file on local managers contains the following error:

YYYY-MM-DDTHH:MM:SS scc-nsxt2 NSX 5613 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" tid="5613" level="ERROR" invalid="true"] Caught exception in reading appl-proxy config file /etc/vmware/nsx-appl-proxy/appl-proxy.xml: /etc/vmware/nsx-appl-proxy/appl-proxy.xml: cannot open file

Environment

VMware NSX-T Data Center 3.x
VMware NSX 4.1.2.3

Cause

According to KB 369820, it is necessary to download the attachment appl-proxy.xml_4.1.2.3, then upload it to the NSX Manager as instructed in the KB article. It is crucial to replace the existing file at /etc/vmware/nsx-appl-proxy/appl-proxy.xml with this downloaded file.

After uploading, ensure you rename the attachment to match the original file path: /etc/vmware/nsx-appl-proxy/appl-proxy.xml, otherwise NSX service messaging-manager can not start correctly.

 

Resolution

• Rename the file /etc/vmware/nsx-appl-proxy/appl-proxy.xml_4.1.2.3 to /etc/vmware/nsx-appl-proxy/appl-proxy.xml in three NSX managers.
• Perform a rolling restart of the three local managers by restarting them one at a time.

Additional Information

KB 369820