The oi_connector probe is very unstable and not processing messages to AIOps


Article ID: 237365


Updated On:


Support Offerings


This is for the BROADCOM_IT tenant. We have observed that the OI_Connector is very unstable. We have observed that at time the oi_connector does not pass messages to AIOps and we have to restart the OI_Connector and the APM bridge to resolve the issue.


- occurs when 1 or more purestorage probes are enabled


Release : 20.3

Component : UIM - OI_CONNECTOR


Summary of configuration changes:
Increased java min/max memory to 12GB/14GB
task_count = 2000
payload_bulk_size = 1000
qos_bulk_size set to 3000
java min/max set to 13G and 15G respectivelybulk_size = 500
hub_bulk_size set from 2000 to 1750
thread_count_insert set from 12 to 24 (best practice).
After setting the value to 24, at peak message count we saw throughput avg of 4-5k per sec increase to ~20000 per second when it was needed.


Error in log:
[Qos Monitor Enricher #5, qos_processor] Failed to enqueue pending qos object, queue capacity exceeded. The update will be made when memory is released.
No data will be lost. S_QOS_DATA { qos_name: QOS_PROCESS_CPU, source: xxxxxxx, target: yyyy, nim_origin: YOUR_ORIGIN, origin: YOUR_ORIGIN, modifier: nimsoft,
host:, robot: uim_robot_name, probe: processes, table_id: null, qos_def_id: null } 

So we added the key:

database-update-queue-capacity and set it to 50000
java heap memory min/max set to 8GB/10GB respectively

oi_connector loglevel

After setting the oi_connector loglevel back down from 5 (debug) to 1 (fatal error messages only)  it allowed the queue to function more efficiently again and Sent all of the messages and queued messages remained within a range of 0 and 2000 messages queued at any given time.

The QOS Message queue attached to OI Connector is now stable, and able to process the messages quickly and efficiently.

Additional Information

oi_connector probe axagateway.uimQos queue continues processing for several days or up to 2 weeks but when the purestorage probe is enabled, it starts to backup and cannot keep up.