NSX Edge lose VMCI connection to NSX Manager

Products

• VMware NSX for vSphere

Issue/Introduction

When the NSX Manager is unavailable for an extended period of time, you experience these symptoms:

• The NSX Edge may lose VMCI connection to the NSX Manager.
• In the NSX Edge logs, you see entries similar to:
  
  2018-04-14T13:11:29+00:00 ### syslog-ng[837]: [default]: [syslog.notice] Configuration reload request received, reloading configuration;
  2018-04-14T13:11:29+00:00 ### MsgMgr[1075]: [daemon.info] VSM msg handler created - amqp broker host:127.0.0.1 ip:0x100007f port:10000
  2018-04-14T13:11:29+00:00 ### MsgMgr[1075]: [daemon.err] SSL_connect:: Underlying syscall error (Bad file descriptor)
  2018-04-14T13:11:34+00:00 ### MsgMgr[1075]: [daemon.info] VSM msg handler created - amqp broker host:127.0.0.1 ip:0x100007f port:10000
  
  ...
  2018-04-16T11:54:17+00:00 ### MsgMgr[1075]: [daemon.info] VSM msg handler created - amqp broker host:127.0.0.1 ip:0x100007f port:10000
  2018-04-16T11:54:17+00:00 ### MsgMgr[1075]: [daemon.err] SSL_connect:: Underlying syscall error (Bad file descriptor)
  2018-04-16T11:54:22+00:00 ### MsgMgr[1075]: [daemon.info] VSM msg handler created - amqp broker host:127.0.0.1 ip:0x100007f port:10000
  2018-04-16T11:54:22+00:00 ### MsgMgr[1075]: [daemon.err] SSL_connect:: Underlying syscall error (Bad file descriptor)
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX for vSphere 6.3.x

Cause

This issue occurs because when the NSX Manager is not available, the msgmgr process on the NSX Edge waits 30 seconds for the NSX Manager to respond. If there is no response, msgmgr will retry without properly closing the previously open files. This results in a very large number of open file descriptors if the NSX Manager is down for a long time.

Resolution

This issue is resolved in:

• VMware NSX for vSphere 6.3.7
• VMware NSX for vSphere 6.4.2

Workaround:

Restart the NSX Edge appliance which should clean up the open socket connections. Once this completes, the NSX Edge should be able to re-connect back to the NSX Manager.