Only XCOM 11.6 32-bit is supported under Red Hat Enterprise Linux (RHEL) 5.
XCOM 11.6 64-bit uses updated third party software versions which addressed known vulnerabilities and those latest third party software versions are compiled with higher levels of compiler which are not compatible with RHEL 5. Some basic validation tests on RHEL 5.6 problems showed problems which are due to the RHEL 5.6 GNU C Library version being older than what XCOM 11.6 64-bit was built with.
So in summary to use XCOM 11.6 64-bit requires upgrade to the supported Red Hat Enterprise Linux 6 or higher per RI59036.
In addition it should be noted that XCOM 11.6 32-bit reached end of service on February 28, 2018:
https://support.ca.com/us/product-content/status/support-life-cycle/indexes/ca-xcom-data-transport-release-and-support-lifecycle-dates.html So it is strongly encouraged to upgrade to a supported XCOM 11.6 64-bit release with corresponding supported version of Red Hat Enterprise Linux.