UIM - Why does qos_processor take a long time to perform enrichment
Article ID: 34664
Updated On:
Products
Issue/Introduction
For an explanation of what the qos_processor does, see this article:
UIM - What does the qos_processor probe do?
For an explanation of how the qos_processor does what it does, and why this may result in a delay, continue reading.
Environment
Component: UIMQOS
Resolution
The enrichment portion of qos_processor works by running a Ruby script called enrichment.rb This can be set up to make alterations (or "enrichments") to various parts of a qos message. ?One common usage is to set a custom origin on QoS whose robot/hub origin is different than what is desired by the customer.
A simple example/snippet of a script to change the origin on all cdm QOS would be something like:
if ($monitor.probe == 'cdm')
$monitor.origin = 'BINGO'
end
This script would look for any QOS from any cdm probe and update the origin on that QoS to "BINGO".
Important Note: qos_processor listens/subscribes to all QOS_MESSAGE messages on the bus as they come in, and marks them for enrichment, but it does not actually perform the enrichment/modifications in the script in real time. ?It performs the modifications on a scheduled interval.
If you take a look at the table S_QOS_DATA you will see a field called 'modifier'.
When you first create a QoS definition, but before any samples have been sent, this field will be set to NULL.
As soon as the first QoS sample is sent by a probe and processed by qos_processor, the modifier will be set to "nimsoft" by default, if there is no enrichment.rb script in the qos_processor\scripts\ folder. ?
Now, when you create an enrichment.rb script to perform some sort of enrichment on the QoS and set enrichment-enabled to "true", the qos_processor probe creates a hash or checksum of the contents of the enrichment.rb script itself which looks like this:
enrichment#5BFAF6316A3F2FC42CFDD66F7CB253F6E2BE074A?
As each QoS sample is received and processed by qos_processsor, the 'modifier' column of any QoS which has been processed will be updated to the same hash instead of "nimsoft". ?If the hash on a particular QoS object remains set to "nimsoft" this indicates that the qos_processor has not yet seen a sample come through.
When you make any change to the enrichment.rb script (including adding a couple of lines of whitespace), the hash changes, and next time the script runs, it will again update the modifier column to match this hash the next time a QoS sample comes in. ?The probe monitors the change time of the file every 30 seconds and automatically detects any changes to the script and creates a new hash without requiring a probe restart.
The enrichment itself is scheduled by a config key called "monitor-script-execution-interval" and the default value is 60 minutes.
Every "monitor-script-execution-interval" minutes, the enrichment.rb script is run against every row in S_QOS_DATA where the modifier column matches the current hash value. ?This avoids unnecessary enrichment of old or stale QoS. ?The probe only performs the modification/enrichment that the script calls for on this interval, so with the default options it is expected that there will be up a 60-minute delay using the default settings before the enrichment is actually performed.
This can be set to a lower interval, but it may take some time for the script to run in a very large environment (or with a very complicated script), and unexpected results may be observed if the interval is set lower than the amount of time it takes the script to execute (some trial and error may be needed to find the lowest setting for your particular environment.)
Regardless of environment size or script simplicity,Support does not recommend setting this value to an interval lower than 5 minutes.
Feedback
