Opensearch service fails to start

Products

• Network Observability
• VMware Smart Assurance

Issue/Introduction

Opensearch service fails to start, error seen in opensearch log :

"Caused by: java.lang.UnsatisfiedLinkError: /tmp/[directory]/libbc-probe.so: failed to map segment from shared object"

Environment

All Supported Smarts versions

Cause

System cannot map executable segments from the /tmp directory when it is mounted with the noexec option. 

Resolution

1. Run the following command to check if /tmp is mounted with noexec: 

   mount | grep '/tmp'

2. If /tmp has noexec in it then do:
   1. Navigate to the <SAM_BASE>/bin directory.
   2. Open the sm_opensearch file for editing using sm_edit utility located at <SAM_BASE>/bin/system.
   3. In opened file, locate the JAVA_CMD section and add the following line: 

      -Djava.io.tmpdir='$OPENSEARCH_TMPDIR' \

      The JAVA_CMD string now explicitly points to the opensearch temporary directory instead of the system /tmp.
3. Now softlink of opensearch to be created, to do so follow the below:
   1. Navigate to the <SAM_BASE>/bin directory.
   2. Remove the existing softlink for sm_opensearch as 

      rm -rf sm_opensearch

   3. Run ./runcmd to recreate softlink.
4. After this, kill any stale opensearch processes if exist.
5. Start the opensearch service and confirm that it is running as

   sm_service show <opensearch service name>

NOTE: This fix will be available as part of default installation from Smarts 25.4.7 version onwards.