NSX Manager reports "EAM Status Down" Alarm comes up and gets resolved itself after few minutes

Products

• VMware NSX

Issue/Introduction

• The following error received intermittently and resolves itself after few minutes:

• The following error message found in /var/log/proton.nsxapi.log:
  
   2024-09-24T18:26:32.086Z  WARN EamPollingThread VcUtils 70368 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] ConnectException occurred
 java.net.ConnectException: Connection timed out (Connection timed out)
   at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_372]
   at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_372]
   at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_372]
   at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_372]
   at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_372]

Environment

VMware NSX-T Data Center 3.x
VMware NSX 4.x

Cause

NSX Manager appliances on impacted versions have the ip_blackhole feature enabled by default. This may result in certain traffic flows being rejected resulting in TCP timeouts. This causes the EAM check to fail.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.2.4 and newer.

This issue is resolved in VMware NSX 4.1.2.2 and newer.

Customers impacted by this should upgrade to above version or a later version.

 

Workaround

Open SSH session to NSX Manager as root:

• Use the following command to disable ip_blackhole live (no reboot required)
  
  echo 0 > /proc/sys/kernel/grsecurity/ip_blackhole
  
  
• Use the following command to keep the change persistent after reboot (add the same command as above to the startup script):
  
  echo 'echo 0 > /proc/sys/kernel/grsecurity/ip_blackhole' >> /opt/vmware/nsx-node-api/bin/set_params.sh
  
  
• Then confirm that ip_blackhole was disabled:
  
  cat /proc/sys/kernel/grsecurity/ip_blackhole
  
  
• Repeat the above steps for all NSX manager nodes

Additional Information

EAM service down Alarm is generated on NSX with multiple VCs registered
EAM service down Alert is generated on NSX