VMware Smart Assurance NCM 10.1.4: Device Server restarts when discovery/pull config operations are executed on devices with mechanism SSH

Products

• VMware

Issue/Introduction

Symptoms:

Initiated device discovery with account credentials and the job/task on the GUI gives below result:
Action Failed... 
Operation canceled, device server restart
 

Enabled debug with instructions of KB https://kb.vmware.com/s/article/323701?lang=en_US

$VOYENCE_HOME/autodisc.log has following messages:
Jun 24 05:27:32 :-2099173632/10.x.x.x@23/snmp#6: Term: Connecting to 10.x.x.x port 22
Jun 24 05:27:32 :-2099173632/10.x.x.x@23/snmp#6: Term: We claim version: SSH-2.0-PuTTY_Release_0.68
Jun 24 05:27:32 :-2099173632/10.x.x.x@23/snmp#6: Term: Server version: SSH-2.0-OpenSSH_6.9
Jun 24 05:27:32 :-2099173632/10.x.x.x@23/snmp#6: Term: Using SSH protocol version 2
Jun 24 05:27:32 :-2099173632/10.x.x.x@23/snmp#6: Term: Doing ECDH key exchange with curve Curve25519 and hash SHA-256
Jun 24 05:27:32 :-2099173632/10.x.x.x@23/snmp#6: Term: Server also has ecdsa-sha2-nistp256/ssh-dss/ssh-rsa host keys, but we don't know any of them
Jun 24 05:27:32 :-2099173632/10.x.x.x@23/snmp#6: Term: Host key fingerprint is:
Jun 24 05:27:32 :-2099173632/10.x.x.x@23/snmp#6: Term: ssh-ed25519 256 e1:64:68:7e:e2:cb:4f:6e:9a:c8:a0:9e:39:67:f4:3c
Jun 24 05:27:32 :2099173632/10.x.x.x@23/snmp#6: Term: Initialised AES-256 SDCTR client>server encryption
Jun 24 05:27:32 :2099173632/10.x.x.x@23/snmp#6: Term: Initialised HMAC-SHA-256 client>server MAC algorithm
Jun 24 05:27:32 :2099173632/10.x.x.x@23/snmp#6: Term: Initialised AES-256 SDCTR server>client encryption
Jun 24 05:27:32 :2099173632/10.x.x.x@23/snmp#6: Term: Initialised HMAC-SHA-256 server>client MAC algorithm
Jun 24 05:27:32 :-2099173632/10.x.x.x@23/snmp#6: Term: Attempting keyboard-interactive authentication
Jun 24 05:27:41 2130442048#1: ======== autodisc pid 16182: started 10.1.4.0.48========

 

$VOYENCE_HOME/voyence_ds.log has following messages, indicating autodiscd service restart:
Jun 24 05:27:36 1211868992#4: Wait returned 0x86, pid=32645
Jun 24 05:27:36 1211868992#1: autodiscd(32645): Process terminated gracefully...normal restart needed (status=0x86)
Jun 24 05:27:41 1211868992#1: Starting autodiscd
Jun 24 05:27:41 1211868992#4: Arg String: -r
Jun 24 05:27:41 1211868992#4: Arg 0: autodiscd
Jun 24 05:27:41 1211868992#4: Arg 1: -r
Jun 24 05:27:41 1211868992#1: autodiscd(16182): starting daemon process...
Jun 24 05:27:41 1211868992#1: autodiscd(16182): process startup execute

Environment

VMware Smart Assurance - NCM

Cause

NCM versions from 9.6.x till 10.1.3, have support of below KEX Algorithms for device connectivity via CLI & OpenSSH 6.9 has been validated as supported.

diffie-hellman-group1-sha1
diffie-hellman-group14-sha1
diffie-hellman-group-exchange-sha256
diffie-hellman-group-exchange-sha1

 

NCM 10.1.4 has new enhancement in code to support new KEX algorithms of SSH.
Refer https://docs.vmware.com/en/VMware-Smart-Assurance/10.1.4/rn/Smart-Assurance-Network-Configuration-Manager-1014-Release-Note.html#whatsnews
Support for Key exchange algorithms enabled in OpenSSH 8.3
Added support for the following Key exchange algorithms which are enabled in OpenSSH 8.3:
curve25519-sha256
[email protected]
ecdh-sha2-nistp256
ecdh-sha2-nistp384
ecdh-sha2-nistp521

 

Due to these changes in the code, for few devices, NCM keyboard interactive function was failing to add the prompts which results in prompts being null and service crash.

Resolution

As of August 2021, this issue is identified as a bug and will be addressed in upcoming NCM 10.1.6 release.

Workaround:
For a workaround on NCM 10.1.4, file a Service Request with VMware Technical Support referencing this KB article.

Additional Information

Refer https://kb.vmware.com/s/article/2006985 for instructions to file a Service Request with VMware.