RabbitMQ fails to exchange notifications to ElasticSearch towards M&R

Products

VMware Smart Assurance

Issue/Introduction

RabbitMQ appears to not exchange the notifications from the SAM to MnR (post rabbitmq repository cleanup) with an error like:

 ERROR 'ALERTS-IN' EventConsumerService alerts-ingestion-main AMQP connection failure 
 ERROR 'ALERTS-IN' EventConsumerService alerts-ingestion-main AMQP connection failure 
 ERROR 'ALERTS-IN' EventConsumerService alerts-ingestion-main start 
   java.lang.RuntimeException: No available ingestion Queue

Environment

VMware SmartAssurance Service Assurance Manager
Watch4net / M&R 7.x

Cause

The repository cleanup of the rabbitmq means that there is no existing configuration, and hence the sm_rabbitmq.pl script reports a failure in merge operation as follows:

Waiting for RabbitMQ to be available: . OK
Merging default configuration: Failed!
204 No Content

Further, we can see that the script is written to close operation on encountering a failure and hence the script closes on line: 205

Died at /apg/InCharge10/SAM/smarts/perl/lib/setup_rabbitmq.pl line 205.

Although, we can see that the RabbitMQ Queues and Users are created on the RabbitMQ console, a crucial piece of code remains un-executed as the script exits at line #205.
Hence, the visual queues for the same are that the Consumers appear to be un-initialized. This can be validated via:

http://<RabbitMQ_hosted_IP>:15672/api/consumers

Impact/Risks:

The notification exchange is affected means that the notification flow to the M&R is affected and no data will be exchanged with this current status.

Resolution

If you are to clean-up the rabbitmq repository ensure that the setup_rabbitmq.log does not report the mentioned error:

Merging default configuration: Failed!
204 No Content

If the above error is seen, we could simply restart the rabbitmmq service as during the first restart operation re-creates the repository and the second restart would be able to achieve the merge and no error is thrown for the sm_rabbitmq.pl script.

Note:
Smarts-tomcat and Presentation-SAM also need to be restarted as well, post restarting smarts-rabbitmq

Additional Information

Cleanup of RabbitMQ and ElasticSearch repos should be followed in the below sequence:

1) Stop smarts-rabbitmq, smarts-tomcat, smarts-elasticsearch and Presentation SAM

2) Clean-up the rmq repos by moving the contents under SAM/smarts/local/repos/rmq directory to a temporary location.
Ensure the same for Sitemod_SAM/smarts/local/repos/rmq directory

3) Clean-up elasticsearch repos by moving the contents under SAM/smarts/local/repos/elasticsearch directory to a temporary location.
Ensure the same for Sitemod_SAM/smarts/local/repos/elasticsearch

4) Start smarts-elasticsearch and ensure it's running.

5) Start smarts-rabbitmq alone and check setup_rabbitmq.log.
If it has failure messages (as described in this article) then restart the same once again.
Ensure no such failures seen in the setup_rabbitmq.log.

6) Start smarts-tomcat, wait for a while and ensure consumer is running
http://<RabbitMQ_hosted_IP>:15672/api/consumers