Upgrade process stopped on one of the node in the cluster

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• The Upgrade failed on the Manager cluster part.
• Run the command: "get service install-upgrade" on all the Managers. and identified the one in stopped state.
• Run the following API on the Manager where the install-upgrade is in "stopped" state and confirm the Application Proxy Hub (APH) is disconnected.

curl -H "X-NSX-Username:admin" -H 'Content-Type:application/json' -X GET "http://localhost:7441/api/v1/messaging/cluster-connection/status"
{
  "results": [
    {
      "address": "ssl://10.11.11.93:1234",
      "conn_status": "Disconnected",
      "node_id": "db6e3c82-4fa5-445c-9526-eeaa0c8fb7c3",
      "node_type": "ApplianceProxyHub"
    },
    {
      "address": "ssl://10.11.11.92:1234",
      "conn_status": "Connecting",
      "node_id": "3f25b742-7b12-4259-8dc7-673b4278d70f",
      "node_type": "ApplianceProxyHub"
    },
    {
      "address": "ssl://10.11.11.91:1234",
      "conn_status": "Disconnected",
      "node_id": "ccdbf0ec-6ffd-4d28-86ac-97d0e21d7be4",
      "node_type": "ApplianceProxyHub"
    }
  ]

• The following log sample can be seen in the Manager Support Log bundle <bundle-name>/desired_state_manager.json

  "/nsxapi/api/v1/aaa/role-bindings": "Connection refused: 127.0.0.1:7440",
  "/nsxapi/api/v1/aaa/roles": "Connection refused: 127.0.0.1:7440",
  "/nsxapi/api/v1/aaa/roles-with-feature-permissions": "Connection refused: 127.0.0.1:7440",
  "/nsxapi/api/v1/aaa/user-info": "Connection refused: 127.0.0.1:7440",

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Resolution

Currently there is no resolution.

Workaround:

Follow these steps for the workaround:

• Stop the APH service on the faulty Manager: "/etc/init.d/nsx-appl-proxy stop"
• Copy the file /etc/vmware/nsx/appliance-info.xml from one of the healthy Manager to the identified faulty one.
• Start the APH service on the faulty Manager: "/etc/init.d/nsx-appl-proxy start"
• Resume the upgrade.

Additional Information

Impact/Risks:
The environment cannot be upgraded. The manager upgrade is failing.