Missing Robots and/or Robot metrics in Operator Console

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)
• Unified Infrastructure Management for Mainframe
• CA Unified Infrastructure Management On-Premise (Nimsoft / UIM)
• CA Unified Infrastructure Management SaaS (Nimsoft / UIM)

Issue/Introduction

Possible symptoms:

Note that the cm_nimbus_robot table most likely will not have entries for Robots that are missing.

You can check the robot table by running this query:

select * from cm_nimbus_robot

Or if it is related to a specific origin or robot with specific user tags you can use an argument like

where user_tag_1 = 'xyz'

or

where origin ='ABC'

If the robot does exist in the CM_NIMBUS_ROBOT table, then the other possibility is that the device was removed from the OC and the option to prevent rediscovery was selected when the device was removed.

Customers like to have VM templates available with robots already on them, and if the niscache is already populated with entries, the robot controller will not notice this and/or automatically clean up or re-create those files.

Therefore, we recommend when cloning a VM, to delete the niscache contents first.

Environment

• Any UIM/OC version

Cause

• The most likely cause for missing robots in OC is duplicate device id's.
• Cloning VM's has been identified as one cause of duplicate dev ids for robots.
• The other possible reason is that the device was removed using the device deletion option from the Inventory and the option to prevent rediscovering the device was selected.

Resolution

You can identify duplicate dev ids for Robots in the discovery_server.log at log level 3.

Here are two example entries from discovery_server log:

You may see a message like this:

Jan 30 11:23:13:898 WARN [hubWorker4, com.nimsoft.nimbus.probe.service.discoveryserver.nimbusscan.HubInfoFetcherDelegate] robot <hostname> / ##.##.##.## (status=0) has duplicate dev_id DB00C38BB3EA481CF292EDD9D9670DBBF

         or this:

##.##.##.## (status=0) has duplicate dev_id

Approach:

• With 23.4, validate that the ROBOT_NIS_CACHE message type has been added to each of the UIM hub(s) queues: 

  Configure the ROBOT_NIS_CACHE Subject queue in DX UIM 23.4

• Develop a list of robots showing dev_id duplication errors in the discovery_server log and make sure you have all of them documented.
• You can run the discovery_server at loglevel 3, long enough to get a list of Robots that need the change to eliminate the duplicate dev_id's .
• You may need to increase the logsize for the discovery_server so that you can see sufficient information.
• You have to make your changes, restart the Robot and then check the log again to be sure you've identified them all.
• You can also check against the Robots that should have been listed in the cm_nimbus_robot table in the first place.
• Once you have a list of robots with duplicate device_ids to resolve the issue, RDP/login to each robot machine and delete the contents of the niscache directory, then restart the robot.

Steps to resolve duplicate ids and allow robots to be discovered:

1. Login to each robot machine.
2. Stop the Nimsoft Robot Watcher service
3. Delete the contents of the niscache folder, for example in: ...\Program Files (x86)\Nimsoft\niscache
   On Unix and Linux systems, be mindful of hidden files (file names that start with a "." and remove those as well.
4. Start the Nimsoft Robot Watcher service.
5. Restart the discovery_server probe
6. Restart the nis_server probe
7. Recheck the discovery_server log for duplicate dev_id errors and repeat if necessary.

If a device was deleted from OC and inadvertently added to the blacklist and you want to add it back or if you wish to remove all blacklisted devices from these tables, then follow the instructions to use the discovery_server callbacks as specified in the following section of the documentation:

Allow Rediscovery of Deleted Devices

When OC is not displaying metrics/monitoring values for the given Robots

Additionally, if after resolving the issue of missing Robots, OC metrics view is not displaying the metrics for those robots, then check the ci metric ID mappings between S_QOS_DATA and cm_configuration_item_metric tables.

If the metric IDs for a given metric in S_QOS_DATA do not have a corresponding row in cm_configuration_item_metric, they simply won't display in OC.

So... for those hosts, you can run a query like:

select s.ci_metric_id, cm.ci_metric_id from s_qos_data s left join cm_configuration_item_metric cm on s.ci_metric_id = cm.ci_metric_id

One will be the ci_metric_id from S_QOS_DATA, the other will either be the ci_metric_id from cm_configuration_item_metric (which is good), or a NULL (which means the ci_metric_id in S_QOS_DATA is bad and needs to be NULL'd out itself, then the data_engine restarted).

So check the rows/two resultant ci_metric_id columns. Any blanks for ci_metric_id in the second column listed in the query output are bad entries. You can then use those 'bad' ci_metric_id values and run:

update s_qos_data set ci_metric_id = NULL

Use one robot as a test, then restart the data_engine.

This will take some time but the mappings should be corrected after this - but please try it on a single robot first to be sure.