Launcher freezes on xmlRMutexLock with mutliple uxjobinit stuck

Products

• Automic Dollar Universe

Issue/Introduction

On heavily used nodes, the Dollar Universe (DUAS) Launcher process may remain frozen. Jobs that should have been launched remain in "Launch Wait" status.

This issue is typically characterized by one or more uxjobinit processes appearing to loop at the system level.

This happens randomly, not necessarily at the same time and not necessarily on the same DUAS each time.
The phenomenon is uncommon and occurs approximately once a week on a random node out of a total of 150 nodes.

Symptoms

• Launcher process is frozen with a "Wake up date" in the past.
• Jobs are stuck in "Launch Wait" status.
• ps or tasklist output shows one or more uxjobinitprocesses that were launched at the same timestamp as the Launcher's last wake-up time.
• universe.log may show WATCHDOG warnings related to locking errors in u_fmsb60.

• the pstack of the uxioserv shows the same taken 3 times after the issue occurs, with 7 threads hung that seem to be waiting on xmlRMutexLock :

  Thread 47 (Thread 0x7f6b3b5fe700 (LWP 3241068)):
  #0  0x00007f6c07a0744c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  #1  0x00007f6c08ce687d in xmlRMutexLock () from /duas_folder/bin/lib/libuniutil.so
  #2  0x00007f6c08d0fcca in xmlFetchXMLCatalogFile () from /duas_folder/bin/lib/libuniutil.so
  #3  0x00007f6c08d1120b in xmlCatalogListXMLResolve () from /duas_folder/bin/lib/libuniutil.so

Environment

Dollar Universe 6.10.101+ or 7.x

Cause

This issue is caused by a mutex locking conflict (Defect DE189831 attempted previously to be fixed via DE123073) where uxjobend or uxjobinit and the Launcher process deadlocks when accessing the u_fmsb60 file.

Resolution

Workaround:

Kill the hung uxjobinit / uxjobend processes and restart the node

Solution:

Update to a fix version listed below or a newer version if available.

Fix version(s): 
Component: Dollar Universe Application Server (node)
Dollar Universe 7.01.31 - planned release November 2026