Launcher crashed when running a lot of executions.
Article ID: 86488
Updated On:
Products
CA Automic Dollar Universe
Issue/Introduction
Affects Release version(s): 5
Error Message :
In Universe logs :
################
uxioserv /u_io_create_thread /000000000 - u_io_create_thread - System error 12 : thread launch type 0 failed (unable to create new thread), thread_number xxx
################
In IO serv logs :
################
[Log File DOLLAR UNIVERSE] - not enough resources, check UXMAXFD environment variables and/or system limits for maximum number of file descriptors/threads/processes per process/user/system wide - system function [fopen] error [EMFILE] - : Too many open file
[u_io_create_thread] - not enough memory - system function [pthread_create] error [ENOMEM] - : Cannot allocate memory
################
Patch level detected:Dollar Universe 5.6
Product Version: Dollar.Universe 5.6.0 FX25010
Description :Launcher crashed when running a lot of executions.
Error Message :
In Universe logs :
################
uxioserv /u_io_create_thread /000000000 - u_io_create_thread - System error 12 : thread launch type 0 failed (unable to create new thread), thread_number xxx
################
In IO serv logs :
################
[Log File DOLLAR UNIVERSE] - not enough resources, check UXMAXFD environment variables and/or system limits for maximum number of file descriptors/threads/processes per process/user/system wide - system function [fopen] error [EMFILE] - : Too many open file
[u_io_create_thread] - not enough memory - system function [pthread_create] error [ENOMEM] - : Cannot allocate memory
################
Patch level detected:Dollar Universe 5.6
Product Version: Dollar.Universe 5.6.0 FX25010
Description :Launcher crashed when running a lot of executions.
Environment
OS: IBM AIX
OS Version: All
OS Version: All
Cause
Cause type:
Configuration
Root Cause: The UXMAXFD Dollar Universe variable sets the maximum amount of files descriptors that a process (server, engine , ...) can open.File descriptors limitation for the current shell session especially can be check and modified with ulimit -n.When UXMAXFD is not defined, Dollar Universe can use maximum of 256 file descriptors, which can be quickly too low if you are running a lot of jobs simultaneously.
The value is linked to files or network sockets requested by the use of Dollar Universe.- If UXMAXFD is lower or equal to ulimit -n, everything goes fine until Dollar Universe reaches UXMAXFD value, then new actions will wait until enough file descriptors are available.
Configuration
Root Cause: The UXMAXFD Dollar Universe variable sets the maximum amount of files descriptors that a process (server, engine , ...) can open.File descriptors limitation for the current shell session especially can be check and modified with ulimit -n.When UXMAXFD is not defined, Dollar Universe can use maximum of 256 file descriptors, which can be quickly too low if you are running a lot of jobs simultaneously.
The value is linked to files or network sockets requested by the use of Dollar Universe.- If UXMAXFD is lower or equal to ulimit -n, everything goes fine until Dollar Universe reaches UXMAXFD value, then new actions will wait until enough file descriptors are available.
Resolution
Always respect this rule :
"UXMAXFD" < "ulimit -n" < "cat /proc/sys/fs/file-max"
Ask System administrator if there is no special limitation on your node's host before modifying ulimit -n.
When starting, ioserv announce how many FD it is allowed to take in the logs, but this value will only refer to UXMAXFD, not to ulimit -n, even if ulimit -n is lesser (Which would be BAD).
Check OS limits with :
ulimit -n
cat /proc/sys/fs/file-max
In Dollar Universe 5.x :
- Load environment
- Stop company
- Verify that all Dollar Universe process are correctly stopped.
- Edit uxsetenv files in $UXMGR (uxsetenv, uxsetenv_ksh, uxsetenv_csh)
- Add UXMAXFD (with a correct value).
Example for uxsetenv and uxsetenv_ksh:
UXMAXFD=4096
export UXMAXFD
Example for uxsetenv_csh:
setenv UXMAXFD 4096
- Reload environment
- Start company
Fix Status: No Fix
"UXMAXFD" < "ulimit -n" < "cat /proc/sys/fs/file-max"
Ask System administrator if there is no special limitation on your node's host before modifying ulimit -n.
When starting, ioserv announce how many FD it is allowed to take in the logs, but this value will only refer to UXMAXFD, not to ulimit -n, even if ulimit -n is lesser (Which would be BAD).
Check OS limits with :
ulimit -n
cat /proc/sys/fs/file-max
In Dollar Universe 5.x :
- Load environment
- Stop company
- Verify that all Dollar Universe process are correctly stopped.
- Edit uxsetenv files in $UXMGR (uxsetenv, uxsetenv_ksh, uxsetenv_csh)
- Add UXMAXFD (with a correct value).
Example for uxsetenv and uxsetenv_ksh:
UXMAXFD=4096
export UXMAXFD
Example for uxsetenv_csh:
setenv UXMAXFD 4096
- Reload environment
- Start company
Fix Status: No Fix
Additional Information
Workaround :
N/A
N/A
Feedback
Yes
No
Powered by
