When attempting to start the Enterprise Manager, it quickly fails and goes into an orderly shutdown. In the IntroscopeEnterpriseManager.log file, which is in INFO mode, there is one ERROR line to the effect of:
4/28/17 03:54:04.925 PM EDT [ERROR] [main] [Manager] The EM failed to start. /tmp/librocksdbjni2906454849760766685.so: /tmp/librocksdbjni2906454849760766685.so: failed to map segment from shared object: Operation not permitted
In DEBUG mode there is a much larger stack trace beginning with:
4/28/17 04:00:30.663 PM EDT [DEBUG] [main] [Manager] Error starting up EM
com.wily.introscope.server.enterprise.EnterpriseManagerCannotStartException: An error occurred while trying to start Isengard.
Contrary to the stack trace here, Isengard — which is responsible for low-level networking in the EM — is actually not at fault. What is at fault is the usage of a library known as RocksDB, specifically in the context of the Environment statement above. RocksDB is a third-party library ( http://rocksdb.org ) that was implemented in APM 10.5 for the metadata portion of the SmartStor database where metrics are stored. During EM initialization, the RocksDB library extracts a portion of native code in the form of a shared library (.so file) that it puts into the temporary directory defined by Java system property java.io.tmpdir. Generally this is the /tmp directory on the operating system. RocksDB is then expecting to use and execute the code within that shared library in the same way that any program uses a shared library: the library needs to be opened and mapped such that the bytes from it can be executed.
The issue arises when the operating system prohibits execution of the code within the shared object file (in this example, /tmp/librocksdbjni2906454849760766685.so but yours will likely have a different number which is random). This is the case when underlying filesystem prohibits code execution because it is mounted with the "noexec" flag. This is to say that despite the permissions on the .so file itself, the underlying filesystem (and thus the kernel) prohibits execution of the files contained therein. This is a common security practice in organizations that run hardened systems, especially on shared, world-writable filesystems like /tmp.
There are two possible resolutions:
Specify a new location for the temporary directory that the EM uses via the java.io.tmpdir Java system property. To achieve this, edit the Enterprise Manager's Introscope_Enterprise_Manager.lax file and update the property lax.nl.java.option.additional to add a define option to specify your desired temp directory path:
Of course please make sure that the path you choose is on an executable filesystem.