Smarts SAM: smarts-rabbitmq service will not start or stay running; Upgraded to Smarts SAM 9.3 and smarts-rabbitmq service will not start; Error in /var/log/messages: Failed to create cookie file '<home dir of a user>.

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:


Cannot start the smarts-rabbitmq service

Smarts solution pack is not able to properly fetch data from Smarts SAM 9.3 deployment.

There is a problem when upgrading the Smarts SAM software that causes the services not to run correctly.  The service that is currently impacted is the smarts-rabbitmq service.

Environment

VMware Smart Assurance - SMARTS

Cause

No error messages regarding this problem will appear in the Smarts SAM 9.3 log files.  This problem can be noted for the following error messages in the /var/log/messages file:

bash-3.2# tail -f /var/log/messages |grep rabbitmq

Jun 24 18:52:40 <hostname> sm_serviced[23713]: SMS-N-LOG_TRANS-Service smarts-rabbitmq has transitioned from state NOT RUNNING to state STARTING via event Start Action
Jun 24 18:52:40 <hostname> sm_serviced[23713]: SMS-N-LOG_START-STARTING service smarts-rabbitmq
Jun 24 18:52:41 <hostname> sm_serviced[23713]: SMS-N-LOG_TRANS-Service smarts-rabbitmq has transitioned from state STARTING to state RUNNING via event Start Time 2
Jun 24 18:52:42 <hostname> sm_serviced[23713]: SMS-N-LOG_OUTPUT-OUTPUT service smarts-rabbitmq: {error_logger,{{2014,6,24},{18,52,41}},"Failed to create cookie file '<home dir of a user>
Jun 24 18:52:43 <hostname> sm_serviced[23713]: SMS-N-LOG_TRANS-Service smarts-rabbitmq has transitioned from state RUNNING to state NOT RUNNING via event Termination
Jun 24 18:52:43 <hostname> sm_serviced[23713]: SMS-N-LOG_UNEXPECTEDTERMINATION-Service smarts-rabbitmq has terminated unexpectedly
Jun 24 18:53:05 <hostname> sm_serviced[23713]: SMS-N-LOG_TRANS-Service smarts-rabbitmq has transitioned from state NOT RUNNING to state NOT RUNNING via event Stop Action

Where <home dir of a user> is the home directory of a user on the system and <hostname> is the host name of the system the Smarts SAM 9.3 software is installed on.
 

Resolution

Please use the following steps to correct this issue:

1. Log into the Smarts SAM 9.3 host as root and create a back up of all Smarts services installed on the host with the following command:

<Base Dir>/SAM/smarts/bin/sm_service show --cmdline > <Backup Directroryc>/ServiceBackUp.txt

2. Stop all Smarts services running on teh SAM 9.3 host with the following command:

<Base Dir>/SAM/smarts/bin/sm_service stop --all

3. Uninstall the Smarts SAM 9.3 software.
   • Be sure not to accidentally delete <Backup Directroryc>/ServiceBackUp.txt
4. Navigate to /etc/init.d and remove the file ic-serviced
5. Navigate to /var/smarts and remove the file serviced.db and the folder pipes
6. Reinstall the Smarts SAM 9.3 software and be sure to select the installation of the follow three services at minimum
   1. smarts-tomcat
   2. smarts-elasticsearch
   3. smarts-rabbitmq
7. After the installation is complete review the currently installed services with the command:

<Base Dir>/SAM/smarts/bin/sm_service show

8. Review the <Backup Directroryc>/ServiceBackUp.txt that backed up all the services and recreate all the services that are missing.
9. Start the new Smarts SAM 9.3 service in the order below with the commands given:
   1. <Base Dir>/SAM/smarts/bin/sm_service start smarts-tomcat
   2. <Base Dir>/SAM/smarts/bin/sm_service start smarts-elasticsearch
   3. <Base Dir>/SAM/smarts/bin/sm_service start smarts-rabbitmq

The smarts-rabbitmq service should now start up properly and continue running.