Local Manager UI points to error "Unable to fetch full sync status" for local and "Lost communication to GM sites"

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX Local Manager UI shows below error in location manager:

Unable to fetch full sync status" for local and "Lost communication to GM sites"

• Validate APH connectivity on LM site showing the issue. It would point all GMs in the disconnected state in aph_rest_output.json
  
  API to run on Local Manager

"/api/v1/messaging/cluster-connection/status": {
    "results": [
...
      {
        "address": "ssl://172.16.1.X:1236",
        "conn_status": "Disconnected",
        "node_id": "########-####-####-####-##########06",
        "node_type": "ApplianceProxyHub"
      },
      {
        "address": "ssl://172.16.1.Y:1236",
        "conn_status": "Disconnected",
        "node_id": "########-####-####-####-##########a4",
        "node_type": "ApplianceProxyHub"
      },
      {
        "address": "ssl://172.31.1.Z:1236",
        "conn_status": "Disconnected",
        "node_id": "########-####-####-####-##########76",
        "node_type": "ApplianceProxyHub"
      },

However, on GMs the connectivity status will be CONNECTED as below :

API to run on Global Manager

  "/api/v1/messaging/cluster-connection/status": {
    "results": [
.......
      {
        "address": "ssl://172.16.2.X:1236",
        "conn_status": "Connected",
        "node_id": "########-####-####-####-##########ba",
        "node_type": "ApplianceProxyHub"
      },
      {
        "address": "ssl-tcp://172.16.2.Y:34828",
        "conn_status": "Connected",
        "node_id": "########-####-####-####-##########fc",
        "node_type": "ApplianceProxyHub"
      },
      {
        "address": "ssl://172.16.2.Z:1236",
        "conn_status": "Connected",
        "node_id": "########-####-####-####-##########7c",
        "node_type": "ApplianceProxyHub"
      },

Environment

VMware NSX-T Data Center

Cause

APH status shows Connected/Disconnected for self (Local Manager) and Global Managers could happen in two scenarios -

1. APH connectivity itself is down - which is probably less likely to occur as Global Manager is in CONNECTED state.

2. Connectivity is UP but Connection status change didn't reflect in the Local Manager side.

Resolution

Pending Resolution

Workaround:

Restarting appl-proxy as below on local Managers with the following commands:

root@manager1:~# /etc/init.d/nsx-appl-proxy stop

root@manager1:~# /etc/init.d/nsx-appl-proxy start

Additional Information

This behavior is observed in NSX-T version 3.2.2.

Impact/Risks:

No functional impact as object replication and other tasks work fine.