Memory growth after changing RabbitMQ password and SM_AMQPNotificationPublisher::SM-AMQPNotificationPublisher queue constantly growing

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:

• SM_AMQPNotificationPublisher::SM-AMQPNotificationPublisher queue is constantly growing    
• Smarts SAM presentation domain shows memory growth when using EDAA after changing RabbitMQ password.
  
  

Environment

Smarts 10.1.X

Cause

There are three default accounts in the RabbitMQ installation:

• an admin account
• an account used by the SAM publisher to deliver messages into RabbitMQ
• an account used by the ingestion process to pull messages from RabbitMQ.

You can change the password for each one of these accounts. 

Smarts Service Assurance Manager (Smarts SAM) presentation domain maintains a queue of objects to push to the RabbitMQ process. If the publisher password is mismatched between the entry in SAM/smarts/local/conf/clientConnect.conf and the publisher password in RabbitMQ, this queue will not be processed and therefore grow over time.

To check if this queue is the cause of your SAM domains memory growth run the following command and check the output for the SM-AMQPNotificationPublisher queue

<Base-SAM-Dir>/smarts/bin/dmctl -s <SAM DOMAIN> exec dmdebug --queues --output=queues.txt

 

An example from SAM/smarts/local/logs/queues.txt is shown below:

SM_AMQPNotificationPublisher::SM-AMQPNotificationPublisher                                                              [1 worker(s)]
Current size 27966, max 27966; processed entries 1

As the Current size equals the max and processed entries is 1 this is a good indication of this scenario. 

Resolution

Please ensure publisher password is consistent in both clientConnect.conf and the RabbitMQ application.

The publisher is a component in the presentation SAM domain that sends notifications into RabbitMQ.
The RabbitMQ login for this process is publisher with the default password of changeme.

To change this password, you must update both RabbitMQ and the SAM configuration with a new matching password.

Procedure:

1. Log in to the RabbitMQ web-based management interface as "admin" user. 
2. Click the Admin tab and select publisher from the list of RabbitMQ login users.
3. Click Update this user.
4. Enter the new password in both password fields.
5. Click Update User.
   
6. To change the password in the SAM configuration:
          a. Log in to the host where the presentation SAM is installed.
          b. Type
   

   <Base-SAM-Dir>/SAM/bin/sm_edit conf/clientConnect.conf

   
          c. In the clientConnect.conf file, look for a line beginning with AMQP/*
              It is similar to this:
   

   *:AMQP/*:publisher:<E-1.0>948D22513AC23588FE1F3E1F85BF6B037208CF4BE61FA0A8EE0BA7307A4B9F63

   
          d. In the AMQP/*" line, replace the password field with the same password that you have entered for the RabbitMQ publisher account:
   

   *:AMQP/*:publisher:<Enter the Password Here>

   
          e. Save clientConnect.conf and exit sm_edit.
7. Restart the publisher by connecting to your presentation SAM using dmctl.
   

   <Base-SAM-Dir>/smarts/bin/dmctl -s <SAM DOMAIN>  invoke SM_AMQPNotificationPublisher::SMAMQPNotificationPublisher stop