Traffic is routed to different pool member by LB with persistence

Products

• VMware NSX

Issue/Introduction

• Traffic through LB is forwarded to different pool member even though the virtual server has persistent profile configured.
• syslog throws 99: Cannot assign requested address
  /var/log/syslog
<TIMESTAMP> <HOSTNAME> NSX <PID> LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="lb" level="FATAL"] [<LB_UUID>] connect() to <POOL_MEMBER_IP_ADDRESS>:80 failed (99: Cannot assign requested address) while connecting to upstream, client: <CLIENT_IP>, server: , request: "POST /Jfrl.Cosmos/MethodInvokeService.asmx HTTP/1.1", upstream: "<URL>", host: "<VS_IP>"
• SNAT mode is set to Transparent on Server Pool
  To check if SNAT mode is set to Transparent which is equivalent to SNAT Translation Mode is set to Deactivated on UI or Auto Map is set to False:
  Examples:
  edge/lb-pool in Edge Support Bundle
  "snat_translation": {
    "auto_map": false,
    "ip_address": [
        "192.168.10.11/null"
    ],
    "port_overload": 32
},

get load-balancer <LB_ID> pool <POOL_ID> snat-pools in Edge CLI
  edge> get load-balancer <LB_ID> pool <POOL_ID> snat-pools
  <TIMESTAMP>
  edge>
  
  NOTE: In above example, the command returns no configuration so that there is no SNAT configured.
  In case there is SNAT configured, the above command shows some configuration.
  edge01> get load-balancer <LB_ID> pool <POOL_ID> snat-pools
  <TIMESTAMP>
  SNAT                    : <SNAT_NAME>
  Min Port                : 4096
  Max Port                : 65535
  Port Overload Factor    : 32
  Random Port             : False
  Snat IP                 : 100.64.0.1 Allocated Port: 0

Environment

NSX 4.x

Cause

Without SNAT, LB tries to use the same client port. If there are multiple client using the same port, LB can only pick one and assign and fail to assign to others.
In that case, LB chooses another pool member.

Resolution

Enable SNAT.

Additional Information

"502 bad gateway" response with L7 transparent mode