MCS profiles which are deployed at the Group level are not getting applied

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

MCS profiles deployed at the Group level are not getting applied to the server even after waiting for more than 48 hours. Restarted ade probe and mon_config_service probes 24 hours back but still no change or improvement.

Environment

Release : 9.1.0

Component : UIM - MON_CONFIG_SERVICE 9.21

Cause

- mon_config_service needed more memory

- max count reached for profiles

Resolution

Via raw configure for the mon_config_service probe, we increased the java heap memory:

Changed: <startup> options = -Xms512m -Xmx1024m </startup> to <startup> options = -Xms2048m -Xmx4096m </startup>

Then cold started the probe (Deactivate-Activate)

We also tuned a few settings:

discoverythreads = 10 to 20

device_processing_threads = 100 to 200

config_deployment_threads = 20 to 40

select (select top(1) objectvalue from SSRV2AuditTrail where objecttype = 'profile' and objectid = dp.profileId order by timestamp desc) as reason, d.name as device_name, d.status as device_status, r.address as robot_address, r.robot_active, r.status as robot_status, dp.status, dp.* from SSRV2Profile dp join ssrv2device d on d.cs_id = dp.cs_id join cm_device cd on cd.cs_id = dp.cs_id join cm_nimbus_robot r on r.dev_id = cd.dev_id where dp.status <> 'ok' and dp.cs_id is not null

data_engine needed to be restarted.

Restarted distsrv and ADE probes. After 30 minutes we checked the profile status again. After increasing the memory and threads, only 3 profiles out of of a total of 20 profiles' status were displayed as new. One of the reasons the profile could not have been deployed is that profile reached max count for a specific device.The error we saw for device was for device Id instead of csId. We can get the corresponding csId from ssrv2device based on device id.The resultant csId can be queried in ssrv2profile to check if it already contains profile and max count has reached.

Also, we can query ssrv2audittrail table for the missing profiles if they are not applicable using below query .Other reasons for a profile not being deployed could be that a robot is down or not reachable but it is is unlikely in this case. select * from ssrv2profile where cs_id = ( select cs_id from ssrv2device where dev_id = xxxxx) select * from ssrv2audittrail where objectvalue like '%maximum%' From the logs at trace level we also saw the metric gatherer is skipping qos collection. This is due to the interval configured in the statistics section in mon_config_service configuration. This should not have any functional impact and is most likely not any reason for profiles not getting deployed on devices. After increasing the memory, the profiles were applied and all profiles were then updated to an 'ok' state instead of 'new'. It appears that the profiles got applied to devices and only a few of them were not applied due to max count reached for profiles.

In order to highlight missing profiles, we need to somehow make the user aware of that fact in a dashboard or UI but this would be an enhancement request.