DUAS: "Cannot assign requested address" during peak of connections cause commands / uprocs to Abort

Issue/Introduction:

In Dollar Universe version 5:

The following error messages are present in universe.log:
======================================================================================
<< 2014-01-20 05:03:10 0002038/uxshw /CALL_UXIOSRV /000000000 - u_io_callsrv(on NODE,COMPANY,X) returns error -1
<< 2014-01-20 05:03:10 0002038/uxshw /GAAPIUSR /134459202 - %UNI_-E-U_EGAT05B3207, Update to an unknown User
<< 2014-01-20 05:03:10 0002043/uxshw /u_io_callsrv /000000000 - u_connect error : Errno syserror 99: Cannot assign requested address (host [HOSTNAME])
======================================================================================

In this example, the issue occurs between 5:00 and 5:30 AM everyday.

The connection to the IO server can increase dramatically. In our example, from 1K to 27K in less a few seconds (less than 10 seconds) using the netstat command while greping for the IO server's port number (ex: 10600).

The high volume of connections to the IO server causes the system to run out of available TCP sockets.

In Dollar Universe 6, the following kind of errors appear in universe.log during the launch of the script that creates the Production Plan Batch ( creation and distribution of uprocs / sessions / resources / tasks...):

|ERROR|X|cmd|pid=p.t| owls_connect_auth | k_connect_auth_timeout(COMPANY/SIO) returns error [200]
|ERROR|X|cmd|pid=p.t| o_callsrv_connect_r| Connection error 99 [Errno syserror 99: Cannot assign requested address]
|ERROR|X|cmd|pid=p.t| owls_cmd_return | Can not connect to server. Error!

In the output of the command line the following error appear:

Commande : uxadd upr exp upr=UPROC label="label" fperiod=D appl=MM memo=O geninf nbper=0 upt=cl_int vupr=000 nopartage nointeractive
Can not connect to server.
Error! commande en erreur!!

Cause:

Exhaustion of ephemeral ports:
Dynamic TCP Port Range is set by default on Linux from 32768 to 61000, so more than 29K TCP connections during a short period of time ( less than 2 minutes) can reach this system limitation.

Environment:

Component: Automic Dollar Universe
Release: 5.x

Resolution:

Check if there's a Dollar Universe job or cron job that uses Dollar Universe commands that is launched multiple times(thousands times).
If there is, then check to see if it is necessary or if the commands can be delayed between each other.

If it is not, you may need to stop the job.

If it is necessary, check with your system administrator on how to make ensure that the machine can handle/provide more TCP connections.
On Linux, for example, you can increase the dynamic port range to avoid the issue to a higher value (21000 to 64000 for example).

[[email protected] ~]# sysctl -w net.ipv4.ip_local_port_range="21000 64000"
net.ipv4.ip_local_port_range = 21000 64000

Note that this modification will be lost upon a system reboot so you need to check with your Unix / Linux administrator to make the change permanent.

If that still does not resolve the problem, please verify that the latest fix for Dollar Universe 5.x has been applied:

Dollar Universe 5.6 - Fix FX25010
Dollar Universe 5.3 - Fix FX25002

Additional Information:

More information here:
https://en.wikipedia.org/wiki/Ephemeral_port

Create a Support Ticket in case you need to download the 5.x Fixes.