Network Configuration Manager captures failing after upgrading to DX NetOps Spectrum 23.3.13.

Products

• Network Observability
• Spectrum

Issue/Introduction

After upgrading to DX NetOps Spectrum 23.3.13, we have noticed that the majority of Network Configuration Manager (NCM) captures are failing during the Global Sync task.

After enabling NCM debugging (Reference How to debug NCM problems in Spectrum), we see the following in the $SPECROOT/NCM/NCMSERV.OUT  and $SPECROOT/NCM/logs/debug/*.log files:

NCMSERV.OUT:
Jan 14, 2025 15:10:07.680 - 
io.grpc.StatusException: UNKNOWN: 8523794
Auth fail for methods 'publickey,keyboard-interactive,password'
at io.grpc.Status.asException(Status.java:551)
at com.aprisma.spectrum.scmd.CustomSSH.jschCustomCapture(CustomSSH.java:197)
at com.aprisma.spectrum.scmd.CustomSSH.capture(CustomSSH.java:56)
at com.aprisma.spectrum.scmd.ScmServiceImpl.captureRunningConfigImpl(ScmServiceImpl.java:704)
at com.aprisma.spectrum.scmd.ScmServiceImpl.captureRunningConfigRun(ScmServiceImpl.java:474)
at com.aprisma.spectrum.scmd.ScmServiceImpl$1.run(ScmServiceImpl.java:4151)
at java.base/java.lang.Thread.run(Unknown Source)

debug log:
Jan 14, 2025 15:09:57.862 (captureRunningConfig for host: xxx.xxx.xxx.xxx) - Diff Masks : 
str: "[!#]"
Jan 14, 2025 15:09:57.862 (captureRunningConfig for host: xxx.xxx.xxx.xxx) - calling custom family jsch 
Jan 14, 2025 15:09:57.863 (captureRunningConfig for host: xxx.xxx.xxx.xxx) -  CustomSSH jschCaptureConfig starting: 
Jan 14, 2025 15:09:57.863 (captureRunningConfig for host: xxx.xxx.xxx.xxx) -     host: xxx.xxx.xxx.xxx
Jan 14, 2025 15:09:57.863 (captureRunningConfig for host: xxx.xxx.xxx.xxx) -     filename: /data/opt2/SPECTRUM/NCM/cache/SCM_0x267025
Jan 14, 2025 15:09:57.863 (captureRunningConfig for host: xxx.xxx.xxx.xxx) -     username: xxxxxxxx
Jan 14, 2025 15:09:57.863 (captureRunningConfig for host: xxx.xxx.xxx.xxx) -     password: xxxxxxxx
Jan 14, 2025 15:10:07.680 (captureRunningConfig for host: xxx.xxx.xxx.xxx) -  Exception occured :Auth fail for methods 'publickey,keyboard-interactive,password'
Jan 14, 2025 15:10:07.680 (captureRunningConfig for host: xxx.xxx.xxx.xxx) - Session already closed 
Jan 14, 2025 15:10:07.680 (captureRunningConfig for host: xxx.xxx.xxx.xxx) -  error status not equal to SUCESS : 8523794
Jan 14, 2025 15:10:07.680 (captureRunningConfig for host: xxx.xxx.xxx.xxx) - captureRunningConfigRun attempts down to: 0
Jan 14, 2025 15:10:07.680 (grpc-default-executor-4) - Exception caught in GRPC captureRunningConfig8523794
Auth fail for methods 'publickey,keyboard-interactive,password'

Environment

Version: 23.3.13
Component: Applications / Network Configuration Manager

Cause

• "Verify Startup Equals Running Configuration" set to Yes in Network Configuration Manager 
• ncm_tftp_wait=1 not in the $SPECROOT/SS/.vnmrc file

Resolution

If the "Verify Startup Equals Running Configuration" is set to Yes, set it to No

• Log into OneClick as an ADMIN user
• Find and select the Configuration Manager folder in the Explorer tab of the Navigation panel
• Click on the Information tab in the Contents panel
• Expand the Global Synchronization subview and note the value of "Verify Startup Equals Running Configuration". If set to Yes, set to No.

Add ncm_tftp_wait=1 to the $SPECROOT/SS/.vnmrc file

• Log into the SpectroSERVER system as the user that owns the Spectrum installation
• cd to the $SPECROOT/SS directory
• Make a backup of the current .vnmrc file
• Edit the .vnmrc file and add ncm_tftp_wait=1
• Save the file
• Restart the SpectroSERVER process for the change to take affect