We have observed that the OI_Connector is very unstable. We have observed that at times the oi_connector does not pass messages to AIOps and we have to restart the oi_connector and apm_bridge to resolve the issue.
qos_processor
Error in log:So we added the key:
database-update-queue-capacity and set it to 50000oi_connector loglevel
After setting the oi_connector loglevel back down from 5 (debug) to 1 (fatal error messages only) it allowed the queue to function more efficiently again and sent all of the messages and queued messages remained within a range of 0 and 2000 messages queued at any given time which is reasonable.
The QOS Message queue attached to OI Connector was then stable, and able to process the messages quickly and efficiently.
oi_connector probe axagateway.uimQos queue continues processing for several days or up to 2 weeks but when the purestorage probe is enabled, it starts to backup and cannot keep up.
apm_bridge:
Every time topology is created or updated it creates a file. A file is saved in the apm_bridge cache folder. If these files build up, while CPU/Memory usage slowly increases, the cache files can be deleted because they are only used temporarily
1. Deactivate the apm_bridge probe
2. Rename/delete the cache & store folder within apm_bridge. Cache folder will have a large number of 1 KB files in most deployments so deleting will take a very long time, so renaming the folder would work.
3. Activate the probe