Agent Linux fails to start or launch jobs with UserServiceDriverProcessMonitor failed by receiving messages

Products

• CA Automic Workload Automation - Automation Engine
• Automic SaaS

Issue/Introduction

The Agent Linux Java 21.x or 24.x fails to submit jobs on Amazon Linux 2023 servers with the following error in the Agent log:

20250124/150922.977 - UserServiceDriverProcessMonitor [pid=xxx], failed by receiving messages, reason : null, thread will end.
20250124/150922.988 - U02000186 Report file '/path/temp/OAACFDFN.TXT' for Job 'JOBNAME (1004003)' cannot be opened. Error: 'java.io.IOException Failed to initialize child driver for user 'username''.

When investigating, it was found that Amazon Linux 2023 does not provide the library libcrypt.so.1, so a ldd on the binaries user_service_pkg (agent Java Linux) or ucjxlx6 (agent Linux 21.x) we could find the reason of the issue:

ldd user_service_pkg
        linux-vdso.so.1 (0x00007fff1a18a000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f338f637000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f338f632000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f338f62d000)
        libcrypt.so.1 => not found
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f338f200000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f338f516000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f338f4fc000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f338ee00000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f338f642000)

ldd ucxjlx6
        linux-vdso.so.1 (0x00007ffe2e347000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fab16079000)
        libcrypt.so.1 => not found
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fab16074000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fab1606f000)
        libstdc++.so.6 => ./libstdc++.so.6 (0x00007fab15f2e000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fab15e51000)
        libgcc_s.so.1 => ./libgcc_s.so.1 (0x00007fab15e3c000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fab15c00000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fab16084000)

On Amazon Linux 2 server it all works as expected without having to install any additional library.

Environment

OS: Amazon Linux 2023, RHEL 9.x and other recent OS distributions 

Agent Linux Java 21.x and 24.x and also Agent Linux 21.x (C-based)

Cause

Some Linux Distributions (like Amazon Linux 2023 or RHEL 9) don't deliver by default the library libcrypt.so.1 needed for the Agent Linux.

Resolution

Install the necessary libcrypt library before attempting to start the Agent.

To do so, for example, on Amazon Linux:

dnf -y install libxcrypt-compat

After installing it, both agents Linux Java(v21 and v24) and Agent Linux v21 will work fine.

Additional Information

The compatibility with Amazon Linux has not yet been validated by Broadcom Engineering teams, this should be done via the following RFE:

F42818: Compatibility: Amazon Linux 2023 for AE and Unix agent

It is tentatively scheduled to be done during the second half 2025 and the Compatibility Matrix will be updated accordingly when done.