‘get managers’ output on NSX manager nodes report incorrect status.

Products

• VMware NSX

Issue/Introduction

• Though all nsx manager nodes are Up and Connected, running ‘get managers’ command would show incorrect output as few nodes are on Standby.

Example : Run below command as admin user of NSX manager CLI.

• nsx-mngr-01> get managers
  Tue Jul 16 2024 UTC 07:38:08.081
  - <Ip address of NSX manager3>    Connected (NSX-RPC) *
  - <Ip address of NSX manager2>     Connected (NSX-RPC)
  - <Ip address of NSX manager1>    Standby (NSX-RPC)
  
  nsx-mngr-02> get managers
  Tue Jul 16 2024 UTC 07:38:18.008
  - <Ip address of NSX manager3>    Connected (NSX-RPC) *
  - <Ip address of NSX manager2>     Standby (NSX-RPC) *
  - <Ip address of NSX manager1>    Connected (NSX-RPC) *
  
  nsx-mngr-03> get managers
  Tue Jul 16 2024 UTC 07:48:51.915
  - <Ip address of NSX manager3>    Standby  (NSX-RPC) *
  - <Ip address of NSX manager2>    Connected (NSX-RPC)
  - <Ip address of NSX manager1>    Connected (NSX-RPC)

 

Port 1234 connectivity from the NSX manager : Run below command as root user of NSX manager CLI.

• nsx-mngr-01:~# netstat -anlp | grep 1234
  tcp        0      0 0.0.0.0:1234            0.0.0.0:*               LISTEN      3424/appl-proxy
  tcp        0      0<Ip Address>:33236    <Ip Address>:1234      ESTABLISHED 3424/appl-proxy
  tcp        0      0 <Ip Address>:1234      <Ip Address>:51862    ESTABLISHED 3424/appl-proxy
  tcp        0      0 <Ip Address>:1234      <Ip Address>:51752     ESTABLISHED 3424/appl-proxy
  tcp        0      0 <Ip Address>:1234      <Ip Address>:56854    ESTABLISHED 3424/appl-proxy
  tcp        0      0  <NSX manager 01 IP address>:1234       <Ip Address>:42133     TIME_WAIT   -  
  
  nsx-mngr2-02:~# netstat -anlp | grep 1234
  tcp        0      0 0.0.0.0:1234            0.0.0.0:*               LISTEN      18306/appl-proxy
  tcp        0      0 <Ip Address>:1234      <Ip Address>:53888     ESTABLISHED 18306/appl-proxy
  tcp        0      0 <Ip Address>:1234      <Ip Address>:36556    ESTABLISHED 18306/appl-proxy
  tcp        0      0 <Ip Address>:1234      <Ip Address>:33722    ESTABLISHED 18306/appl-proxy
  tcp        0      0 <Ip Address>:1234      <Ip Address>:33236     ESTABLISHED 18306/appl-proxy
  tcp        0      0  <NSX manager 02 IP address>:1234       <Ip Address>:53356     TIME_WAIT   -
  
  nsx-mngr-03:~# netstat -anlp | grep 1234
  tcp        0      0 0.0.0.0:1234            0.0.0.0:*               LISTEN      19321/appl-proxy
  tcp        0      0 <Ip Address>:51752    <Ip Address>:1234      ESTABLISHED 19321/appl-proxy
  tcp        0      0 <Ip Address>:53888     <Ip Address>:1234      ESTABLISHED 19321/appl-proxy
  tcp        0      0  <NSX manager 03 IP address>:1234       <Ip Address>:53356     TIME_WAIT   -

 

This is just a command output, the inconsistency does not have any impact.

Environment

Issue is reported in NSX versions below 3.2.X , 4.0.x, and 4.1.x 

Cause

On impacted versions, only outgoing connections to other manager nodes over TCP 1234 are considered as “Connected” . The connections over incoming TCP 1234 and to self are marked as  “Standby”

Resolution

Issue is fixed in NSX manager versions 3.2.4 and 4.2.

Additional Information

After the upgrade to fixed version, we are expecting to see all the NSX managers in Connected state.

Example:

nsx-mngr-01> get managers
Wed Jul 17 2024 UTC 09:04:50.106
- <Ip address of NSX manager1>    Connected (NSX-RPC)
- <Ip address of NSX manager2>    Connected (NSX-RPC) *
- <Ip address of NSX manager3>    Connected (NSX-RPC)

The * symbol denotes the current master for management plane.