Smarts NCM: Unable to discover devices & pull their configurations. Which SSH errors in the logs need to verified ?

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:




Unable to discover Devices and pull their configurations accordingly. The device is certified in the respective DSR Version.

Verified SSH and SNMP are working fine. 

Can login to the device through ssh console from device server, but unable to pull the configuration from NCM application. 

Which errors need to be verified in the logs ?

Environment

VMware Smart Assurance - NCM

Cause

Initially enable Debug and perform a device rediscovery. This issue usually occurs due to two reasons :

1. Invalid device SSH credentials, please find an example of the error snippets :

Dec 04 05:01:53 :1161767232/10.170.49.100@7004/SSH#2: DeviceObj::testAccountCredentials  !!! SSH Credentials are not Valid
Dec 04 05:01:54 :1161767232/10.170.49.100@7001/SSH#2: Testing Session Credentials using SSH...
Dec 04 05:01:58 :1161767232/10.170.49.100@7001/SSH#1: ssh_connect(): 10.170.49.100 - Remote device connection failed; check credentials
Dec 04 05:01:58 :1161767232/10.170.49.100@7001/SSH#1: ssh_connect(): 10.170.49.100 - Remote device connection failed; check credentials

2. SSH Keys are old. The below error snippets are found in the commmgr logs :

[root@STSOT003 logs]# grep 10.170.49.100 *.log
commmgr.2.log:Dec 08 23:44:07 -176137920/pull(2708908)#1: ssh_connect(): 10.170.49.100 - SSH keys did not match
commmgr.2.log:Dec 08 23:44:12 -176137920/pull(2708908)#1: ssh_connect(): 10.170.49.100 - SSH keys did not match
commmgr.2.log:Dec 08 23:44:17 -176137920/pull(2708908)#1: ssh_connect(): 10.170.49.100 - SSH keys did not match
commmgr.6.log:Dec 08 04:12:28 746658112/pull(2706028)#1: ssh_connect(): 10.170.49.100 - SSH keys did not match
commmgr.6.log:Dec 08 04:12:33 746658112/pull(2706028)#1: ssh_connect(): 10.170.49.100 - SSH keys did not match
commmgr.6.log:Dec 08 04:12:38 746658112/pull(2706028)#1: ssh_connect(): 10.170.49.100 - SSH keys did not match

Resolution

Resolution option 1 : Validate the credentials and verify by logging to the device with the same credentials

Resolution option 2 : Validate the SSH Keys. In case the SSH Keys are old or mismatching, delete the SSH Keys for the devices in concern. Update the new credentials in the devices & verify by rediscovery / pulling configurations accordingly.