Action Shell Command exception The parameter or environment lists are too long

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

We use a simple script to notify ServiceNow about status updates (from the ConnectionStatus metric) the script is defined with The Action use the Shell Command with "Pass Text to Shell". In general the script works well, but sometimes we get the following problem reported in the IntroscopeEnterpriseManager.log.

9/17/18 04:20:34.873 PM CEST [ERROR] [Alarm Pooled Worker] [Manager.Action] Action "ServiceNow Notification" failed to execute shell command "./zis/servnow.sh"

enabling DEBUG logging shows the following extra detail

 9/17/18 04:20:34.654 PM CEST [DEBUG] [Alarm Pooled Worker] [Manager.Action] Before executing shell command : ./zis/servnow.sh from Action : ServiceNow Notification
9/17/18 04:20:34.873 PM CEST [ERROR] [Alarm Pooled Worker] [Manager.Action] Action "ServiceNow Notification" failed to execute shell command "./zis/servnow.sh"
9/17/18 04:20:34.873 PM CEST [VERBOSE] [Alarm Pooled Worker] [Manager.Action] Action failed with Exception: java.io.IOException: Cannot run program "./zis/servnow.sh": java.io.IOException: error=7, The parameter or environment lists are too long.


 

Environment

APM any release Linux / Unix platform

Cause

When using a large metric group if a significant number of alerts are triggered the size of the message passed to the shell command can exceed the maximum size that the shell environment can handle. The restriction is OS dependent see https://www.in-ulm.de/~mascheck/various/argmax/

Resolution

The requirement to prevent the problem is to reduce the maximum size of the text that is passed to the shell command. The size will depend on the number of metrics triggering the alert. Possible workarounds are

 - define the alert triggering the action to "Notify on individual metric" - this means that if 100 metrics trigger the alert then instead of all the details of those being passed in one stream to the action , 100 discrete alerts will be triggered.
 - define multiple alerts based on smaller metric groupings so that the limit could not be reached even if all metrics in the grouping trigger

An Idea has been raised for this situation to be better handled in the tool i.e. provide more options