Smarts NCM: Jobs not running after applying patch; commmgrd process stops then fails to start; Error from commmgr.log CommMgr ThreadThrottler::shutdown...done CommMgr Shutdown complete

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:


No jobs running in NCM. Commmgrd process fails to start on the Device Server(s).

 

After applying an official NCM Patch, the commmgrd process fails to start on the Device Server(s). As a result, no jobs will run in NCM.

**The command service vcmaster status will show that commmgrd is not running.**

The following errors will repeat over and over in $VOYENCE_HOME/logs/commmgr.log and $VOYENCE_HOME/logs/voyence_ds.log on the Device Server(s)

commmgr.log
Oct 14 06:23:55 -1995712768/60#1: ------ CommMgr Sub-thread Queue Manager - 2 #-1995712768 terminated
Oct 14 06:23:55 -1970513952#1: CommMgr ThreadThrottler::shutdown waiting 26 seconds for 3 threads until shutdown...
Oct 14 06:23:55 -1970513952#1: CommMgr ThreadThrottler::shutdown waiting 26 seconds for 2 threads until shutdown...
Oct 14 06:23:55 -1970513952#1: CommMgr ThreadThrottler::shutdown waiting 26 seconds for 1 threads until shutdown...
Oct 14 06:23:55 -2004109568/30#1: Manager 0-30 shutting down
Oct 14 06:23:55 -1991514368/RTStatus#1: ------ Manager 0 30 Sub-thread Notifcation Thread - RealTimeUpdate #-1991514368 terminated
Oct 14 06:23:55 -2004109568/30#2: The real time update thread has terminated
Oct 14 06:23:55 -2004109568/30#1: Manager 0 30 ThreadThrottler::reclaimThread Notifcation Thread - RealTimeUpdate #-1991514368 No retText returned...using "Abnormal Termination"
Oct 14 06:23:55 -2004109568/30#1: MasterAccess 0-100 shutting down
Oct 14 06:23:55 -2004109568/30#1: MasterAccess 0-30 shutting down
Oct 14 06:23:55 -2004109568/30#1: ------ CommMgr Sub-thread Queue Manager - 1 #-2004109568 terminated
Oct 14 06:23:56 -1970513952#1: CommMgr ThreadThrottler::shutdown waiting 25 seconds for 1 threads until shutdown...
Oct 14 06:23:56 -1970513952#1: CommMgr ThreadThrottler::shutdown...done
Oct 14 06:23:56 -1970513952#1: Script Manager shutting down
Oct 14 06:23:57 -1970513952#2: InfrastructureCfgMgr shutting down
Oct 14 06:23:57 -1970513952#1: CommMgr Shutdown complete...
Oct 14 06:23:57  commmgr#1| 23632: -------- commmgr: stopped 9.2.2a.0.17 --------

voyence_ds.log
Oct 15 09:24:37 1396344800#1: commmgrd(2289): Process terminated gracefully...normal restart needed (status=0x86)
Oct 15 09:24:37 1396344800#1: commmgrd(2289): retry count reached...delaying 10 seconds
Oct 15 09:24:47 1396344800#1: commmgrd(2289): process backoff completd
Oct 15 09:24:52 1396344800#1: Starting commmgrd
Oct 15 09:24:52 1396344800#1: commmgrd(3537): starting daemon process...
Oct 15 09:24:52 1396344800#1: commmgrd(3537): process startup executed

Environment

VMware Smart Assurance - NCM

Cause

This issue is caused by files getting locked during the installation of the Patch. This can happen if the vcmaster service is still running during the installation of the patch and perhaps some processes are actively in use.

To confirm the issue view the install log for the Patch. This file is named hotfix_install.log and will be located in $VOYENCE_HOME/logs:



Within the file hotfix_install.log you should see some NonFatalErrors as follows:

Summary
-------

Installation: Successfulwith errors.

64 Successes
0 Warnings
4 NonFatalErrors
0 FatalErrors

Search the file hotfix_install.log for the following string 'ERROR'. You should find the following 3 errors in sequence:


Install File:             /opt/smarts-ncm/bin/evdispatchd
                          Status: ERROR
                          Additional Notes: ERROR - ZeroGpl: /opt/smarts-ncm/bin/evdispatchd (Text file busy)

Install File:             /opt/smarts-ncm/bin/commmgrd
                          Status: ERROR
                          Additional Notes: ERROR - ZeroGpl: /opt/smarts-ncm/bin/commmgrd (Text file busy)

Install File:             /opt/smarts-ncm/bin/autodiscd
                          Status: ERROR
                          Additional Notes: ERROR - ZeroGpl: /opt/smarts-ncm/bin/autodiscd (Text file busy)

Resolution

The files autodiscd, commmgrd and evdispatchd outlined above were locked during the Patch installation.

To resolve this issue, follow these steps:

• Confirm that there are no running jobs related to the Device Server where you are installing the patch. You can check this in the Schedule Manager in the NCM GUI.
• You will need to stop the vcmaster service on the Device Server where you are installing the patch before installation begins: service vcmaster stop
• Once you have confirmed the status of vcmaster as not running, begin the installation of the patch: service vcmaster status
• Once installation has completed, again check the hotfix_install.log file to confirm there are no longer any NonFatalErrors.
• Once this is confirmed, restart the vcmaster service: service vcmaster start
• At this point, the commmgrd process should start as part of the vcmaster service

If you still receive a warning that the commmgrd service is not starting, please open an SR with EMC Technical Support.