Smarts NCM :Excessive logging of WARN messages for AuthorizationSucceededEvent event thrown by integration adapters

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:


Under some NCM instance production specific usage profiles, use of the integration modules may cause the server.log or powerup.log files to become filled with very rapidly occurring, high volume writes of the following or similar WARN messages tied to the AuthorizationSucceededEvent (the integration module named may vary by NCM instance):

2018-05-22 10:49:48,341 WARN [com.voyence.configmgr.integration.modules.flatfile.jmx.FlatFileIntegrationModule] (ActiveMQ Session Task-2779) Ignoring received event AuthorizationSucceededEvent. This module not setup to handle this event.
2018-05-22 10:49:48,341 WARN [com.voyence.configmgr.integration.modules.mail.jmx.MailIntegrationModule] (ActiveMQ Session Task-2731) Ignoring received event AuthorizationSucceededEvent. This module not setup to handle this event.
2018-05-22 10:49:48,342 WARN [com.voyence.configmgr.integration.modules.snmp.jmx.SNMPIntegrationModule] (ActiveMQ Session Task-2727) Ignoring received event AuthorizationSucceededEvent. This module not setup to handle this event.

This has been tied to large log file size, and rapid rolling of the affect log files at a pace that makes them largely useless for other troubleshooting. In some cases, we have seen the entire rolled log set cycle completely out of scope in under thirty (30) minutes.

Environment

VMware Smart Assurance - NCM

Cause

Many of the integration modules were not written with message handlers to gracefully receive and discard the AuthorizationSucceededEvent message because it is considered to be normal function. Under most usage profiles, these messages can be safely ignored, as they do not impact NCM functionality in any way except in the narrow use case where the sheer volume of these messages in the logs becomes problematic.

Resolution

1.  During an appropriate maintenance window, log into a Linux shell on the NCM Application Server (AS) host as 'root'.
2.  Set NCM related shell session variables by running the following command:
 

    source /etc/voyence.conf

3.  Make a backup copy of the current logging configuration responsible for the excessive error logging for NCM integration modules by running the following command:
 

    cp -p $VOYENCE_HOME/ncmcore/webapps/ncm-webapp/WEB-INF/classes/log4j.xml $VOYENCE_HOME/ncmcore/webapps/ncm-webapp/WEB-INF/classes/_backup_log4j.xml

4.  Open the original file cited in Step 3. above for editing in a text editor of your choice (ex.: POSIX Linux 'vi' text editor):
5.  Locate the comment section immediately prior to the '<root>' tag near the end of the file (on our about line 297).
6.  Insert the following new configuration stanza prior to the comment section cited in Step 4 above:

    <category name="com.voyence.configmgr.integration">

      <priority value="OFF" />

      <appender-ref ref="FILE"/>

    </category>

7.  Save and exit the file
8.  Restart NCM services on the AS host by running the following command:

    service vcmaster restart

This should suppress the excessive integration module logging of these normal function messages so they no longer adversely impact NCM logging.