Cisco Devices Randomly Dropping While Using SSH Connection

book

Article ID: 196270

calendar_today

Updated On:

Products

CA Spectrum

Issue/Introduction


Customer reports that since upgrade of Spectrum to 10.4.1 SSH Sessions to Cisco devices have been dropping out randomly while actively being used.
Cisco device logs show error for example - 000294: Jul 28 14:41:19: %SSH-3-BAD_PACK_LEN: Bad packet length -1769979419.

I checked the NCMSERV.OUT logs but only seeing the following presently - 

Jul 29, 2020 1:34:46 PM io.grpc.netty.NettyServerTransport notifyTerminated
SEVERE: Transport failed
java.io.IOException: An existing connection was forcibly closed by the remote host
 at sun.nio.ch.SocketDispatcher.read0(Native Method)
 at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
 at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
 at sun.nio.ch.IOUtil.read(IOUtil.java:192)
 at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
 at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
 at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1100)
 at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372)
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioBythannel.java:123)
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 at java.lang.Thread.run(Thread.java:748)


Cause

NCMService was still running while the SpectroSERVER was shutdown, this was the reason for the error message in NCMSERV.OUT

Environment

Release : 10.4.x

Component : Spectrum Core / SpectroSERVER / Spectrum OneClick

Resolution

SSH session was ended because the SpectroSERVER was shutdown, the time in the VNM.OUT of the SpectroSERVER shutdown matches the time in the NCMService where the SSH session was dropped