Cisco Devices Randomly Dropping While Using SSH Connection

Products

• Spectrum

Issue/Introduction

Customer reports that since upgrade of Spectrum to 10.4.1 SSH Sessions to Cisco devices have been dropping out randomly while actively being used.
Cisco device logs show error for example - 000294: Jul 28 14:41:19: %SSH-3-BAD_PACK_LEN: Bad packet length -1769979419.

I checked the NCMSERV.OUT logs but only seeing the following presently - 

Jul 29, 2020 1:34:46 PM io.grpc.netty.NettyServerTransport notifyTerminated
SEVERE: Transport failed
java.io.IOException: An existing connection was forcibly closed by the remote host
 at sun.nio.ch.SocketDispatcher.read0(Native Method)
 at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
 at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
 at sun.nio.ch.IOUtil.read(IOUtil.java:192)
 at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
 at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
 at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1100)
 at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372)
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioBythannel.java:123)
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 at java.lang.Thread.run(Thread.java:748)

Environment

Release : 10.4.x/20.2.x/21.2.x/22.2.x

Component : Spectrum Core / SpectroSERVER / Spectrum OneClick

Cause

NCMService was still running while the SpectroSERVER was shutdown, this was the reason for the error message in NCMSERV.OUT

Resolution

SSH session was ended because the SpectroSERVER was shutdown, the time in the VNM.OUT of the SpectroSERVER shutdown matches the time in the NCMService where the SSH session was dropped