Dollar Universe fails to start after reboot with error ServerException

Products

• CA Automic Dollar Universe

Issue/Introduction

There was a system reboot and post which the Dollar Universe fail to start with below error

NIO CDJ Connection problem - ServerException - connect: error creating socket [DUHKEX/NODENAME/X]

Environment

Component: Dollar Unvierse

Release: 6.x, 7.x

Cause

The log review shows that with a server reboot there was an OS update performed resulting in consecutive boot sequence starting multiple restarts, which interrupted the Dollar Universe shutdown process. As a result the Dollar Universe services shutdown disgracefully and the ports in used are not released. Please refer log snippet

/var/log/messages

Aug 11 05:46:28 <3.6> HOSTNAME dollaruniverse: Shutting down DUAS services: DUAS shutdown procedure

Aug 11 05:47:44 <3.6> HOSTNAME systemd: Stopping Update UTMP about System Boot/Shutdown...

Aug 11 05:51:04 <3.6> HOSTNAME systemd: Starting Update UTMP about System Boot/Shutdown...

Aug 11 05:51:41 <3.6> HOSTNAME dollaruniverse: The IO server seems to have been stopped in an unsafe mode.

universe.log

| 2024-08-11 05:46:12 |WARN |X|CDJ|pid=p.t| UXOS_HdlTermProcess | execution handler: SIGNAL = (15) PID = (6311) PPID = (1) GPID = (6311)

| 2024-08-11 05:46:12 |WARN |X|BVS|pid=p.t| UXOS_HdlTermProcess | execution handler: SIGNAL = (15) PID = (6344) PPID = (1) GPID = (6344)

| 2024-08-11 05:46:12 |FATAL|S|IO |pid=p.t| u_io_srv_main | u_select_accept returns -1 - end of the IO server

| 2024-08-11 05:46:12 |FATAL|X|IO |pid=p.t| u_io_srv_main | u_select_accept returns -1 - end of the IO server

 

| 2024-08-11 05:46:12 |ERROR|S|IO |pid=p.t| u_io_srv_main | COM Error u_select -1: Errno syserror 4: Interrupted system call,select error

| 2024-08-11 05:51:59 |ERROR|X|CDJ|pid=p.t| u_ouv_serv | failed in u_listen: Errno syserror 98: Address already in use

| 2024-08-11 05:51:59 |FATAL|X|CDJ|pid=p.t| u_cdj_main | Unable to start CDJ service: port number in use (already started?)

Resolution

1: If performing system(OS) updates/upgrades make sure to stop the services manually perform the system upgrade and start the services 

2: If the system(OS) is booted externally and performing auto updates, make sure to stop and start the $U services manually. In case of $U service startup issue post system reboot, make sure to check if there is already processes running by command ps -efHwl | grep ux and stop/kill the same and start the $U services.

Additional Information

DUAS: Communication issues between UVMS/UVC and Nodes through the network