Stale Entries in the Transport Node and Message Client Table

Products

• VMware NSX

Issue/Introduction

Host and Edge Nodes are not able to connect to NSX-T Managers via port 1234 (NSX Proxy).

In the Edge Nodes' syslog (/var/log/syslog), you can observe log lines such as:

 

2021-06-29T03:53:16.258Z <hostname> NSX 1874 - [nsx@6876 comp="nsx-edge" subcomp="nsx-proxy" s2comp="nsx-net" tid="1875" level="WARNING"] StreamConnection[21154 Connecting to ssl:// #.#.#.#:1234 sid:21154] Couldn't connect to 'ssl://#.#.#.#:1234' (error: 335544539-short read)
2021-06-29T03:53:16.258Z <hostname> NSX 1874 - [nsx@6876 comp="nsx-edge" subcomp="nsx-proxy" s2comp="nsx-net" tid="1875" level="WARNING"] StreamConnection[21154 Error to ssl:// #.#.#.#:1234 sid:-1] Error 335544539-short read

** "335544539-short read" means that the TCP connections fail after SSL handshake. This implies that there is a problem in the certificates, which is a generic problem (not inherently specific to this problem).

Additionally, in the Proton log (/var/log/proton/nsxapi.log) , you also see the following log lines:

2021-06-29T01:06:32.472Z INFO nsx-rpc:PROTON_TNCONNCFGSVC_CLIENT:user-executor-0 NodeAphCallObserver - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] exception in next() com.vmware.nsx.management.common.exceptions.ObjectNotFoundException: The requested object : Node/<NODE-UUID> could not be found. Object identifiers are case sensitive.
 at com.vmware.nsx.management.container.dao.AbstractDao.findByIdentifier(AbstractDao.java:150)
 at com.vmware.nsx.management.container.dao.IdentifiableObjectDaoDelegate.findByIdentifier(IdentifiableObjectDaoDelegate.java:205)
 at com.vmware.nsx.management.fabricnode.service.NodeServiceImpl.findById(NodeServiceImpl.java:51)
 at com.vmware.nsx.management.fabricnode.service.NodeServiceImpl.getNode(NodeServiceImpl.java:46)
 at com.vmware.nsx.management.fabricnode.aph.NodeAphRealizer.getTnConnCfgFromDb(NodeAphRealizer.java:243)
 at com.vmware.nsx.management.fabricnode.aph.NodeAphCallObserver.next(NodeAphCallObserver.java:58)
 at com.vmware.nsx.management.fabricnode.aph.NodeAphCallObserver.next(NodeAphCallObserver.java:30)
 at com.vmware.nsx.rpc.call.NsxRpcCall$ActiveCallStateBase.invokeNext(NsxRpcCall.java:266)
 at com.vmware.nsx.rpc.call.NsxRpcCall$LocalDoneCallState.doReceive(NsxRpcCall.java:524)
 at com.vmware.nsx.rpc.call.NsxRpcCall.doReceive(NsxRpcCall.java:999)
 at com.vmware.nsx.rpc.channel.NsxRpcChannel.doReceiveEstablishedCall(NsxRpcChannel.java:687)
 at com.vmware.nsx.rpc.channel.NsxRpcChannel.doReceive(NsxRpcChannel.java:624)
 at com.vmware.nsx.rpc.channel.task.ChannelReceiveTask.doRun(ChannelReceiveTask.java:21)
 at com.vmware.nsx.rpc.channel.task.ChannelTask.run(ChannelTask.java:45)
 at com.vmware.nsx.rpc.channel.NsxRpcChannel.processOperations(NsxRpcChannel.java:835)
 at com.vmware.nsx.rpc.core.Scheduler.process(Scheduler.java:112)
 at com.vmware.nsx.rpc.core.Scheduler.run(Scheduler.java:90)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

** ObjectNotFoundException means that there is a stale object. In this case it is an Edge Node, whose uuid is <Node-UUID>.

To be encountering this issue, both unique log lines must be present, as the first log line itself is not independently indicative of this specific issue. 

Users would not able to make any configuration changes on Host Transport Nodes and Edge Transport Nodes if this issue is present.

 

Environment

VMware NSX

Cause

When an Edge Transport Node is deleted, its corresponding entry in all Management Plane database tables should be deleted. In a rare situation, there can be stale entries database tables for either Transport Nodes and/or Messaging Clients (A table Transport Nodes depend on). These stale entries generate messages, which cannot be serviced by the Management Plane. These messages block the message queue and prevent the Management Plane servicing other messages in the message queue. As a result, the Management Plane cannot send messages to any Transport Nodes, including the certificates used for authentication between Transport Nodes and the Management Plane. When hosts do not have the updated certificates from the Management Plane, they cannot form a connection to the Management Plane.
 
The cause of the stale entries cannot be determined because the log is overwritten.

Resolution

This issue is resolved in VMware NSX 3.1.1, available at Broadcom downloads.

If you believe you have encountered this issue and are unable to upgrade, please open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases.