Customer reports that since upgrade of Spectrum to 10.4.1 SSH Sessions to Cisco devices have been dropping out randomly while actively being used.
Cisco device logs show error for example - 000294: Jul 28 14:41:19: %SSH-3-BAD_PACK_LEN: Bad packet length -1769979419.
I checked the NCMSERV.OUT logs but only seeing the following presently -
Jul 29, 2020 1:34:46 PM io.grpc.netty.NettyServerTransport notifyTerminated
SEVERE: Transport failed
java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1100)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioBythannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
Release : 10.4.x/20.2.x/21.2.x/22.2.x
Component : Spectrum Core / SpectroSERVER / Spectrum OneClick
NCMService was still running while the SpectroSERVER was shutdown, this was the reason for the error message in NCMSERV.OUT
SSH session was ended because the SpectroSERVER was shutdown, the time in the VNM.OUT of the SpectroSERVER shutdown matches the time in the NCMService where the SSH session was dropped