Pool members are showing 'down' in NSX native load balancer and NSX LB is generating 'Bad gateway' error

Products

• VMware NSX

Issue/Introduction

• NSX native load balancer is configured with SNAT Translation Mode set to Automap.
• Monitor profile is configured.
• Pool members are showing down.

• Pool members are accessible using their individual IP addresses, however when accessed through LB VIP 'Bad Gateway' error is seen in web browser.
• Pool members have distributed firewall (DFW) rule applied to them. Default rule is set to deny.

Environment

VMware NSX
VMware NSX-T Data Center

Cause

• SNAT IP is not allowed in DFW rule so health monitor and traffic to pool members is getting dropped by distributed firewall.
• Distributed firewall logs on host where pool member is present shows packets dropped.
  [root@esx:~] cat var/run/log/dfwpktlogs.log

Resolution

Check the SNAT IP for the LB and allow traffic from SNAT IP to pool members.

SNAT IP can be checked using edge CLI or NSX manager UI.

NSX manager UI :

When SNAT Translation Mode is set to Automap a group a created in NSX manager under Inventory > Groups with name is following format NLB.PoolLB.[<pool-name>][<LB name>].


Edge CLI :

Run the following command -
edge> get load-balancer <lb-uuid> snat-pools

edge> get load-balancer ccf7####-####-####-####-####6fca8c0b snat-pools

SNAT                    : nat_####_1
Min Port                : 4096
Max Port                : 65535
Port Overload Factor    : 32
Random Port             : False
Snat IP                 : 100.##.##.1 Allocated Port: 0