VMware Smart Assurance Smarts | RabbitMQ fails to exchange notifications to ElasticSearch towards M&R
search cancel

VMware Smart Assurance Smarts | RabbitMQ fails to exchange notifications to ElasticSearch towards M&R

book

Article ID: 323801

calendar_today

Updated On:

Products

VMware

Issue/Introduction

This article provides a workaround/ best practice to ensure there that the rabbitmq configuration is loaded successfully.

Symptoms:
RabbitMQ appears to not exchange the notifications from the SAM to MnR (post rabbitmq repository cleanup) with an error like:
 ERROR 'ALERTS-IN' EventConsumerService alerts-ingestion-main AMQP connection failure 
 ERROR 'ALERTS-IN' EventConsumerService alerts-ingestion-main AMQP connection failure 
 ERROR 'ALERTS-IN' EventConsumerService alerts-ingestion-main start 
   java.lang.RuntimeException: No available ingestion Queue


Environment

VMware Smart Assurance - SMARTS

Cause

The repository cleanup of the rabbitmq means that there is no existing configuration, and hence the sm_rabbitmq.pl script reports a failure in merge operation as follows:

Waiting for RabbitMQ to be available: . OK
Merging default configuration: Failed!
204 No Content 

Further, we can see that the script is written to close operation on encountering a failure and hence the script closes on line: 205

Died at /apg/InCharge10/SAM/smarts/perl/lib/setup_rabbitmq.pl line 205.


Although, we can see that the RabbitMQ Queues and Users are created on the RabbitMQ console, a crucial piece of code remains un-executed as the script exits at line #205. 
Hence, the visual queues for the same are that the Consumers appear to be un-initialized. This can be validated via:

http://<RabbitMQ_hosted_IP>:15672/api/consumers

Resolution

If you are to clean-up the rabbitmq repository ensure that the setup_rabbitmq.log does not report the mentioned error:

Merging default configuration: Failed!
204 No Content

If the above error is seen, we could simply restart the rabbitmmq service as during the first restart operation re-creates the repository and the second restart would be able to achieve the merge and no error is thrown for the sm_rabbitmq.pl script. 

Note:
Smarts-tomcat and Presentation-SAM also need to be restarted as well, post restarting smarts-rabbitmq


Additional Information

Cleanup of RabbitMQ and ElasticSearch repos should be followed in the below sequence:

1) Stop smarts-rabbitmq, smarts-tomcat, smarts-elasticsearch and Presentation SAM
2) Clean-up the rmq repos by moving the contents under SAM/smarts/local/repos/rmq directory to a temporary location.
     Ensure the same for Sitemod_SAM/smarts/local/repos/rmq directory
3) Clean-up elasticsearch repos by moving the contents under SAM/smarts/local/repos/elasticsearch directory to a temporary location.
    Ensure the same for Sitemod_SAM/smarts/local/repos/elasticsearch
4) Start smarts-elasticsearch and ensure it's running.
5) Start smarts-rabbitmq alone and check setup_rabbitmq.log.
    If it has failure messages (as described in this article) then restart the same once again.
    Ensure no such failures seen in the setup_rabbitmq.log.
6) Start smarts-tomcat, wait for a while and ensure consumer is running
    http://<RabbitMQ_hosted_IP>:15672/api/consumers


Impact/Risks:
The notification exchange is affected means that the notification flow to the M&R is affected and no data will be exchanged with this current status.