Set up and then collect baselines for some cdm probe metrics on multiple robots.
After waiting for 1 month, set up Standard Deviation dynamic thresholds for these QoS metrics.
No matter how high you set the threshold number (tried as high as 1000) the alarm continues to fire as long as the current QoS value is greater than the baseline value of the QoS metric.
The clocks on some of the robots where metrics are being baselined were an hour ahead of the correct time. This caused the baseline_engine to "dump" the QoS metrics collected each hour resulting in invalid data in the baseline_engine cache_dir.
Correct the system time on the problematic robots. Once the times are corrected, dynamic thresholds should begin to work as expected.
The current design of the baseline_engine probe requires that the times on all robots collecting baseline metrics be at or close to the right time.