ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

The oi_connector probe is very unstable and not processing messages to AIOps

book

Article ID: 237365

calendar_today

Updated On:

Products

Support Offerings

Issue/Introduction

This is for the BROADCOM_IT tenant. We have observed that the OI_Connector is very unstable. We have observed that at time the oi_connector does not pass messages to AIOps and we have to restart the OI_Connector and the APM bridge to resolve the issue.

Cause

- occurs when 1 or more purestorage probes are enabled

Environment

Release : 20.3

Component : UIM - OI_CONNECTOR

Resolution

Summary of configuration changes:
 
oi_connector
Increased java min/max memory to 12GB/14GB
task_count = 2000
payload_bulk_size = 1000
qos_bulk_size set to 3000
java min/max set to 13G and 15G respectivelybulk_size = 500
 
data_engine
hub_bulk_size set from 2000 to 1750
thread_count_insert set from 12 to 24 (best practice).
 
After setting the value to 24, at peak message count we saw throughput avg of 4-5k per sec increase to ~20000 per second when it was needed.

qos_processor

Error in log:
[Qos Monitor Enricher #5, qos_processor] Failed to enqueue pending qos object, queue capacity exceeded. The update will be made when memory is released.
No data will be lost. S_QOS_DATA { qos_name: QOS_PROCESS_CPU, source: xxxxxxx, target: yyyy, nim_origin: YOUR_ORIGIN, origin: YOUR_ORIGIN, modifier: nimsoft,
host: 1.2.3.4, robot: uim_robot_name, probe: processes, table_id: null, qos_def_id: null } 

So we added the key:

database-update-queue-capacity and set it to 50000
java heap memory min/max set to 8GB/10GB respectively

oi_connector loglevel

After setting the oi_connector loglevel back down from 5 (debug) to 1 (fatal error messages only)  it allowed the queue to function more efficiently again and Sent all of the messages and queued messages remained within a range of 0 and 2000 messages queued at any given time.

The QOS Message queue attached to OI Connector is now stable, and able to process the messages quickly and efficiently.

Additional Information

oi_connector probe axagateway.uimQos queue continues processing for several days or up to 2 weeks but when the purestorage probe is enabled, it starts to backup and cannot keep up.