Error Cannot load modules/mod_mime.so undefined symbol: ap_set_content_type_ex when upgrading CA Access Gateway

Products

• SITEMINDER

Issue/Introduction

Trying to do an in-place upgrade of CA Access Gateway, the following errors appear:

Services could not start!!
--------------------------

Services could not start!!

Please restart the services manually.

Error output:

httpd: Syntax error on line 110 of <CA_ACCESS_GATEWAY_HOME>/httpd/conf/httpd.conf:
Cannot load modules/mod_mime.so into server: <CA_ACCESS_GATEWAY_HOME>/httpd/modules/mod_mime.so:
undefined symbol: ap_set_content_type_ex

and upgrade is not performed successfully, whereby CA Access Gateway is unable to start

Environment

CA SiteMinder 12.8.6 (possibly also other versions) running in Linux

Cause

Doing

ldd <CA_ACCESS_GATEWAY_HOME>/httpd/modules/mod_mime.so
        linux-vdso.so.1 (0x00007fff50de7000)
        /lib64/liboneagentproc.so (0x00007f3b0e32d000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3b0e10d000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f3b0dd37000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3b0e75d000)

 /lib64/liboneagentproc.so is a Dynatrace module. So the machine to be upgraded is running this product which has injected itself in SPS operation. During the upgrade this prevents successful completion

CA Access Gateway has no certified integration with Dynatrace, which should in any case be uninstalled during the upgrade process and if so wished be installed back afterwards at your own risk.

Resolution

The following procedure may solve the issue:

1. Comment the reference to /lib64/liboneagentproc.so in "/etc/ld.so.preload"
2. Disable Dynatrace
3. Make sure no tomcat and httpd process were running.
4. Delete the sps.pid file from <CA_ACCESS_GATEWAY_HOME>/tmp if it exists
5. Re-run upgrade 

Additional Information

There have been other use cases reported where despite having disabled Dynatrace or it not being present in the system the same error is obtained post-upgrade.

In general the error reported means that the mod_mime.so module is unable to successfully load its dependencies and so it fails with this generic error.

In the case of Dynatrace it may be due to Dynatrace interaction, but in other use cases, where Dynatrace is not present, this may also arise as a result of an incorrect or incomplete installation (for instance the dependencies for mod_mime.so may not have been properly upgraded). In such situations it is recommended to try reinstalling Access Gateway and pay attention to possible errors in installation logs to make sure all modules are compatible

After the upgrade it is possible that Tomcat is not able to start with error

java.lang.IllegalArgumentException: Invalid FIPS mode [ ] Provided at com.ca.sso.FipsMode.<clinit>(FipsMode.java:39)

in this case it is advisable to apply 

Getting fips error starting Access Gateway after upgrade