The problem was found with custom alarm messages for the ntperf probe and may apply when creating custom alarm messages for other probes. 

The probe generates the alarm as expected, however it fails to clear the alarm when it should. 

The alarm subsystem value is used for alarm to alarm clear correlation. With the subsystem value missing from the custom alarm the correlation could not take place and the alarm was left open. 

Release:

Component:

Add the subsystem to the alarm message. 

To add open raw configure > messages > select the message > new key name = subsystem value = 1.1.1 

Note: if the problem is for a different probe the subsystem number will be different, the important thing is it matches the original alarm subsystem number. 

Here is an example of the custom alarm from ntperf.cfg

   <AlarmOnValue_Major>
      text = $hostname $robotname $watcher: $description $object/$instance/$counter = $value | $value $operation (limit= $limit) failed. 
      level = 4
      token = alarm_on_value
      i18n_token = as#system.ntperf.performance_winnt_host_robot_watcher_limit_failed
   </AlarmOnValue_Major>

Here is the original alarm message:
​   <AlarmOnValue>
      text = $hostname $robotname $watcher: $description $object/$instance/$counter = $value | $value $operation (limit= $limit) failed
      level = 5
      subsystem = 1.1.1
      token = alarm_on_value
      i18n_token = as#system.ntperf.performance_winnt_host_robot_watcher_limit_failed
   </AlarmOnValue>

Here is the clear alarm message:
   <ClearOnValue>
      text = $hostname $robotname $watcher: $description $object/$instance/$counter = $value - clear
      level = 0
      subsystem = 1.1.1
      token = clear_on_value
      i18n_token = as#system.ntperf.performance_winnt_host_robot_watcher_limit_clear
   </ClearOnValue>