All Virtual servers of one Load Balancer are shown down on NSX Manager UI, however Virtual servers are working normally

Products

• VMware NSX

Issue/Introduction

• All Virtual servers of one Load Balancer are shown down on NSX Manager UI, however Virtual servers are working normally.
• The virtual server number is around 400, pool number around 300, pool member number is around 500.
• Virtual servers down was due to the attached server pools were down with error "LbPoolStatus is DOWN: Members:’x.x.x.x:xx,x.x.x.x:xx’ with status ‘UNUSED’ ".
• syslog.log of active edge node:

<edge_node_name> NSX xxx - [nsx@xxx comp="nsx-edge" s2comp="nsx-monitoring" entId="xxxx" tid="xxx" level="WARNING" eventState="On" eventFeatureName="load_balancer" eventSev="warning" eventType="pool_status_down"] The load balancer pool xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx status is down.

• In nsx manager support bundle, the monitor reports UNKNOWN and pool status is DOWN:
                     "display_name": "xxxxx",
                    "member_num": "1",
                    "members": [
                        {
                           "display_name": "xxxxx",
                           "ip": "x.x.x.x",
                            "monitors": [
                                {
                                    "display_name": "tcp",
                                   "id": "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx",
                                   "status": "unknown", <<<<<<<
                                    "type": "TCP"
                                }
                            ],
                           "port": "xx",
                            "status": "unused",
                            "type": "primary"
                        }
                    ],
                    "primary_disabled": "0",
                    "primary_down": "0",
                    "primary_graceful_disabled": "0",
                    "primary_unknown": "0",
                    "primary_unused": "1",
                    "primary_up": "0",
                    "status": "down",
                    "type": "l7",
                   "uuid": "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx",
                    "vss": {
                       "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx": "<vs_name>"
• The issue may recover itself after hours.

 

Cause

If there are several monitors configured in a pool, the size of the returned status of the member would be large. Then causes edge fail to get pool status and mark pool status as down.

Resolution

This issue is fixed in 4.2.3.2, 4.2.4 and 9.0.2.0.

Workaround:

Reduce the monitor number configured in the pool. For example, keep 2 monitors for each pool.