NSX Controller showing as disconnected in the vSphere web client

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX Controller is showing as disconnected in the vSphere Web Client.
• Running the show control-cluster connections command on the controllers, you see many api_provider open connections.

nsx-controller # show control-cluster connections
role                port            listening open conns
--------------------------------------------------------
api_provider        api/443         Y         1234
--------------------------------------------------------
persistence_server  server/2878     Y         1
                    client/2888     Y         0
                    election/3888   Y         0
--------------------------------------------------------
switch_manager      ovsmgmt/6632    Y         0
                    openflow/6633   Y         0
--------------------------------------------------------
system              cluster/7777    Y         0 

• The disconnected controller may also show as connected to the cluster from CLI even though it shows as disconnected from the UI.

nsx-controller # show control-cluster status
Type                Status                                       Since
--------------------------------------------------------------------------------
Join status:        Join complete                               06/11 12:38:52
Majority status:    Connected to cluster majority                06/14 16:28:53
Restart status:     This controller can be safely restarted      06/14 16:26:10
Cluster ID:         35f43765-xxxx-xxxx-xxxx-35e1806070bc
Node UUID:          4280c211-xxxx-xxxx-xxxx-119d1b07cee0
 Role                Configured status   Active status
--------------------------------------------------------------------------------
api_provider        enabled             activated
persistence_server  enabled             activated
switch_manager      enabled             activated
logical_manager     enabled             activated
directory_server    enabled             activated

Environment

VMware NSX for vSphere 6.3.x
VMware NSX for vSphere 6.2.x

Resolution

TCP sockets exhaustion may be caused by a firewall considering the connections idle and blocking them. It is recommended to configure the TCP idle timeout on the Firewall to at least 3 hours for TCP port 443 connections.

To resolve this issue:

1. If the disconnected controller is wrongly shown as connected via CLI or show a high number of api_provider, reboot it.
2. Identify which of the remaining two controllers is the primary. This is the one showing the same UUID as Cluster ID and Node UUID.

   nsx-controller # show control-cluster status
   Type                Status                                       Since
   --------------------------------------------------------------------------------
   Join status:        Join complete                                06/14 10:03:10
   Majority status:    Connected to cluster majority                06/14 16:27:45
   Restart status:     This controller can be safely restarted      06/14 16:27:25
   Cluster ID:         35f43765-xxxx-xxxx-xxxx-35e1806070bc
   Node UUID:          35f43765-xxxx-xxxx-xxxx-35e1806070bc
   Role                Configured status   Active status
   --------------------------------------------------------------------------------
   api_provider        enabled             activated
   persistence_server  enabled             activated
   switch_manager      enabled             activated
   logical_manager     enabled             activated
   directory_server    enabled             activated

3. Reboot the secondary controller from vSphere Web Client.
4. Once complete, reboot the primary controller from vSphere Web Client.
5. Once complete, check on all three controllers the status and roles.

   nsx-controller # show control-cluster connections
   role                port            listening open conns
   --------------------------------------------------------
   api_provider        api/443         Y         10
   --------------------------------------------------------
   persistence_server  server/2878     -         0
                       client/2888     Y         2
                       election/3888   Y         0
   --------------------------------------------------------
   switch_manager      ovsmgmt/6632    Y         0
                       openflow/6633   Y         0
   --------------------------------------------------------
   system              cluster/7777    Y         1​
   nsx-controller # show control-cluster status
   Type                Status                                       Since
   --------------------------------------------------------------------------------
   Join status:        Join complete                                06/14 17:13:22
   Majority status:    Connected to cluster majority                06/14 17:13:04
   Restart status:     This controller can be safely restarted      06/14 17:13:18
   Cluster ID:         35f43765-xxxx-xxxx-xxxx-35e1806070bc
   Node UUID:          35f43765-xxxx-xxxx-xxxx-35e1806070bc
   Role                Configured status   Active status
   --------------------------------------------------------------------------------
   api_provider        enabled             activated
   persistence_server  enabled             activated
   switch_manager      enabled             activated
   logical_manager     enabled             activated
   directory_server    enabled             activated
   nsx-controller # show control-cluster roles
                            Listen-IP  Master?    Last-Changed  Count
   api_provider         Not configured       No  06/14 16:27:25     60
   persistence_server              N/A       No  06/14 16:27:25     48
   switch_manager            127.0.0.1       No  06/14 16:27:25     60
   logical_manager                 N/A       No  06/14 16:27:25     60
   directory_server                N/A       No  06/14 16:27:25     64