Unable to load notification in event dashboard due to OpenSearch Error

Products

• VMware Smart Assurance
• Network Observability

Issue/Introduction

• Failed to load notification in M&R due to error: Msa Internal Server Error. Reason: OpenSearch Error. Caused by: Could not create a validated object, cause: ValidateObject failed
• Upon enabling debug and running setup_opensearch.pl script noticed error in setup_opensearch.log: 'Proxy tunneling failed: ForbiddenUnable to establish SSL connection'.
• Failed to load notification in M&R due to error: Msa Internal Server Error. Reason: OpenSearch Error. Caused by: Request failed: [search_phase_execution_exception] all shards failed
• Post disabling proxy, Opensearch connection is established but still notification are not loading. 

Environment

Smarts 24.3.13 and later releases

M&R 7.x

Cause

Proxy was enabled in Linux VM for wget

Resolution

• Disable the proxy setting in linux VM for wget 
• Even after above if notification is still not loading in M&R then follow below steps to clear the stale notification or notification cache: 
  • Stop the SAM service as $SAM_Basedir/bin# sm_service stop ic-sam-pres
  • Clear notification cache by running below commands:
    • $SAM_Basedir/bin# sm_service stop smarts-opensearch
    • $SAM_Basedir/bin# sm_service stop smarts-rabbitmq
    • $SAM_Basedir/bin# sm_service stop smarts-tomcat
    • $SAM_Basedir# rm -rf local/repos/rmq
    • $SAM_Basedir# rm -rf local/repos/opensearch
    • $SAM_Basedir/bin# sm_service start smarts-rabbitmq
    • Note: Check the setup_rabbitmq.log to see if rabbitmq service has started correctly, if not stop and start the service again.
    • $SAM_Basedir/bin# sm_service start smarts-opensearch
    • $SAM_Basedir/bin# sm_service start smarts-tomcat
  • Start the SAM service as $SAM_Basedir/bin# sm_service start ic-sam-pres

Additional Information

/root/.wgetrc file in linux VM containing default settings and options that wget reads each time is invoked by the root user can be modified to disable the proxy setting.

/etc/wgetrc contains Global configuration settings for all users.