SNAT Port Exhausted

Products

• VMware NSX
• VMware NSX

Issue/Introduction

Symptoms:

A large number of requests to backend servers via NSX Load Balancer (LB) fail with error code 500

Edge Syslog files include the error:

/var/log/syslog

[nsx@6876 comp="nsx-edge" subcomp="nsx-edge-lb.lb_log" level="FATAL"] [ID] [crit] 2674#0: *45311383 get src ip address failed  while connecting to upstream

LB Error log with Debug log level:

[error] 13790#0: SNAT: alloc SNAT from Pool: nat_1681928195_1 failed                                                                                                                                                                                                                                                                                                                                        
[error] 13790#0: *18447598 get source ip failed from ip pool: "nat_1681928195_1" while connecting to upstream, client:IP_Client, server: , request: "POST /v1/callback/tiny/logs/services/ HTTP/1.1", upstream: "http://upstream_IP:32001/v1/callback/tiny/logs/services/", host: "Host:32001"                                                                                                     
[crit] 13790#0: *18447598 get src ip address failed  while connecting to upstream, client:IP_Client, server: , request: "POST /v1/callback/tiny/logs/services/ HTTP/1.1", upstream: "http://upstream_IP:32001/v1/callback/tiny/logs/services/", host: "Host:32001"                                                                                                                                 
[debug] 13790#0: *18447598 free: 00000000029AE5E0, unused: 32                                                                                                                                                                                                                                                                                                                                                  
[debug] 13790#0: *18447598 reusable connection: 0      

                                                                                                                                                                                                                                                                                                                              

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 2.5.x

Script Settings

Minutes to Remind:
Hours to Remind:
Auto Refresh Interval (seconds):
Auto Refresh:
Auto Check-In:
Check-In Time:
Check-Out Time:
Template File:
Save

Open Trumbowyg

▶

Cause

The SNAT Port exhausted as the Auto Map is used when configuring the SNAT Translation in Server Pools.

The port could be used between 4096 and 65535

get load-balancer LB_UUID snat-pools
SNAT                    : nat_1681928195_1
Min Port                : 4096
Max Port                : 65535
Port Overload Factor    : 32
Random Port             : False
Snat IP                 : SNAT_IP Allocated Port         

Script Settings

Minutes to Remind:
Hours to Remind:
Auto Refresh Interval (seconds):
Auto Refresh:
Auto Check-In:
Check-In Time:
Check-Out Time:
Template File:
Save

Open Trumbowyg

▶

 

Script Settings

Minutes to Remind:
Hours to Remind:
Auto Refresh Interval (seconds):
Auto Refresh:
Auto Check-In:
Check-In Time:
Check-Out Time:
Template File:
Save

Open Trumbowyg

◀

Resolution

Change SNAT Translation Mode to "IP List"  in Server Pools to extend IP/Ports

Script Settings

Minutes to Remind:
Hours to Remind:
Auto Refresh Interval (seconds):
Auto Refresh:
Auto Check-In:
Check-In Time:
Check-Out Time:
Template File:
Save

Open Trumbowyg

▶

Additional Information

Impact/Risks:
Request to Backend servers via LB would be failed as All SNAT Ports are in use