New Install of DX NetOps Portal on RHEL 8.x returns errors

Products

CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

Installing a new instance of the DX NetOps Performance Management Portal web server returns errors. The Performance_Center_Install_*.log shows these errors.

Custom Action: com.ca.im.installanywhere.action.GenerateInstallFailure
Status: FATAL ERROR
Additional Notes: FATAL ERROR - class com.ca.im.installanywhere.action.GenerateInstallFailure FatalInstallException: Error loading UDF plugin. v6_* functions will be unavailable for GlobalSync, etc!!!

Custom Action: com.ca.im.installanywhere.action.GenerateInstallFailure
Status: FATAL ERROR
Additional Notes: FATAL ERROR - class com.ca.im.installanywhere.action.GenerateInstallFailure FatalInstallException: MySql is not running.

The msyql install log shows these errors.

Execute Script/Batch file: Initializing...
Status: SUCCESSFUL
Custom Action: com.ca.im.installanywhere.action.GenerateInstallFailure
Status: FATAL ERROR
Additional Notes: FATAL ERROR - class com.ca.im.installanywhere.action.GenerateInstallFailure FatalInstallException: MySql DB initialization failed!
Checking /etc/profile to see if any PATH= line contains :/capm/CA/MySql/bin"
Looking up version property in file: /etc/my.cnf
Unable to locate MySQL configuration file: /etc/my.cnf
No existing MySQL configuration version found - install needed.
Backing up file /etc/my.cnfFile does not exist.Backing up file /my.cnfFile does not exist.
Did not find PATH= line that contains search string.

Later in the mysql install log we then see:

Custom Action: com.ca.im.installanywhere.action.GenerateInstallFailure
Status: FATAL ERROR
Additional Notes: FATAL ERROR - class com.ca.im.installanywhere.action.GenerateInstallFailure FatalInstallException: Error loading UDF plugin. v6_* functions will be unavailable for GlobalSync, etc!!!
XMLScriptWriter: No Installation Objects were skipped
ERROR (1): There was an error running MySQL --initialize-insecure. Please check /capm/CA/MySql/mysql_initialize_results.txt for the reason.
Execute Custom Code
class com.ca.im.installanywhere.action.GenerateInstallFailure FatalInstallException MySql DB initialization failed!
Starting MySql... Exit Code: 1 Err: [Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
]
Updating root pw... Exit Code: 1 Err: [mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/tmp/mysql.sock' exists!

In the /capm/CA/MySql/mysql_initialize_results.txt file we then see:

2023-11-13T20:30:37.032142Z 0 [Note] [MY-010458] [Server] --initialize specified on an existing data directory.
2023-11-13T20:30:37.038179Z 0 [ERROR] [MY-013276] [Server] Failed to set datadir to '/capm/CA/MySql/data/' (OS errno: 13 - Permission denied)
2023-11-13T20:30:37.038183Z 0 [ERROR] [MY-013236] [Server] The designated data directory /capm/CA/MySql/data/ is unusable. You can remove all files that the server added to it.
2023-11-13T20:30:37.038258Z 0 [ERROR] [MY-010119] [Server] Aborting

Environment

All supported DX NetOps Performance Management Portal web server releases

Cause

Multiple causes have been observed. One or more may be found on your system.

Missing the ncurses-compat-libs package, a requirement for successful Portal install and operation.
The selinux tool is enabled in ENFORCING mode instead of being disabled or in PERMISSIVE mode.
Custom home directory path, the /opt replacement, is configured with incorrect permissions. In this example /capm was created as the home directory.

Resolution

To check for the missing required package run "rpm -qa | grep ncurses". If it's not found, install it using "yum install ncurses-compat-libs". Once installed clean up the failed install and run it again.

To check for the state of selinux run "sestatus". Anything other than disabled or permissive will cause problems. If found in an enabled and ENFORCING mode, disable it. After disabling selinux clean up the failed install and run it again.

Using a custom home directory instead of the default system created /opt directory? Ensure it's created with 755 (rwxr-wr-w) permissions or we'll see problems with access rights for the mysql user created for MySql DB administration by the tool. It'll fail to start after install causing the install steps after mysql is installed to fail. To check the permissions use "ls -la /opt" (default path shown for example purposes). If incorrect use "chmod 755 <path>" (Ex: chmod 755 /opt) to change it. After fixing the permissions clean up the failed install and run it again.

To clean up the failed install delete the (default path shown) /opt/CA directory and everything underneath as well as the /var/.com.zerog.registry.xml file.

Additional Information

The ncurses-compat-lib is one of a number of required packages for a successful Portal install. Ensure all required packages are installed. See the Package Requirements documentation topic for additional information.

Another potential clue for the missing packages is in some Linux OS's we can see them missing using the ldd command. For example if it's missing we'd see something similar to this. Note the two not found libraries from the ncurses-compat-libs package.

ldd /opt/CA/MySql/bin/mysql
linux-vdso.so.1 (0x00007ffe281f7000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa78d0c5000)
librt.so.1 => /lib64/librt.so.1 (0x00007fa78cebd000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fa78ccb9000)
libncurses.so.5 => not found
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fa78c924000)
libm.so.6 => /lib64/libm.so.6 (0x00007fa78c5a2000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fa78c38a000)
libc.so.6 => /lib64/libc.so.6 (0x00007fa78bfc5000)
/lib64/ld-linux-x86-64.so.2 (0x00007fa78d2e5000)
libtinfo.so.5 => not found

Another clue for selinux being a problem is the systems /var/log/messages file. Depends on the correct logging levels being enabled on the OS. See the Verify the Prerequisites documentation topic for a note about the requirement to have selinux disabled or in permissive mode only. If system config provides /var/log/messages file logging we might see this showing the mysql start failures due to selinux being enabled.

Jan 17 15:30:06 default systemd[1]: Starting MySQL daemon...
Jan 17 15:30:06 default systemd[2610]: mysql.service: Failed to execute command: Permission denied
Jan 17 15:30:06 default systemd[2610]: mysql.service: Failed at step EXEC spawning /opt/CA/MySql/support-files/mysql.server: Permission denied
...
Jan 17 15:30:06 default systemd[1]: mysql.service: Control process exited, code=exited status=203
Jan 17 15:30:06 default systemd[1]: mysql.service: Failed with result 'exit-code'.
Jan 17 15:30:06 default systemd[1]: Failed to start MySQL daemon.
...
Jan 17 15:30:07 default setroubleshoot[2613]: failed to retrieve rpm info for /opt/CA/MySql/support-files/mysql.server
...
Jan 17 15:30:08 default setroubleshoot[2613]: SELinux is preventing /usr/lib/systemd/systemd from execute access on the file /opt/CA/MySql/support-files/mysql.server. For complete SELinux messages run: sealert -l 1e5d83d5-a783-4b9a-9004-6187683a5dee
Jan 17 15:30:08 default setroubleshoot[2613]: SELinux is preventing /usr/lib/systemd/systemd from execute access on the file /opt/CA/MySql/support-files/mysql.server.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that systemd should be allowed execute access on the mysql.server file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c '(l.server)' --raw | audit2allow -M my-lserver#012# semodule -X 300 -i my-lserver.pp#012