NSX Manager reports "EAM Status Down" Alarm comes up and gets resolved itself after few minutes
search cancel

NSX Manager reports "EAM Status Down" Alarm comes up and gets resolved itself after few minutes

book

Article ID: 381091

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • The following error received intermittently and resolves itself after few minutes:

  • The following error message found in nsxapi.log (/var/log/proton) of the NSX Manager Node reporting the "EAM Status Down" alarm.

     2024-09-24T18:26:32.086Z  WARN EamPollingThread VcUtils 70368 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] ConnectException occurred
     java.net.ConnectException: Connection timed out (Connection timed out)
       at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_372]
       at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_372]
       at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_372]
       at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_372]
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_372]

Environment

VMware NSX-T Data Center 3.x
VMware NSX 4.x

Cause

Any version running kernel version 5.15.92 with the ip_blackhole feature still enable.

The kernel version can be checked from below command in NSX manager Bash shell (root user).

root@NSX-Manager:~# uname -a

Run cat /proc/sys/kernel/grsecurity/ip_blackhole to see if the feature is enabled or not.
The above command returns 1 which means ip_blackhole feature is enabled.

NSX Manager appliances on impacted versions have the ip_blackhole feature enabled by default. This may result in certain traffic flows being rejected resulting in TCP timeouts. This causes the EAM check to fail.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.2.4 and newer.

This issue is resolved in VMware NSX 4.1.2.2 and newer.

Customers impacted by this should upgrade to above version or a later version.

Workaround

Open SSH session to NSX Manager as root:

  • Use the following command to disable ip_blackhole live (no reboot required)

    echo 0 > /proc/sys/kernel/grsecurity/ip_blackhole

  • Use the following command to keep the change persistent after reboot (add the same command as above to the startup script):

    echo 'echo 0 > /proc/sys/kernel/grsecurity/ip_blackhole' >> /opt/vmware/nsx-node-api/bin/set_params.sh

  • Then confirm that ip_blackhole was disabled:

    cat /proc/sys/kernel/grsecurity/ip_blackhole

  • Repeat the above steps for all NSX manager nodes

Additional Information

Powered by Wolken Software