Malware prevention SVMs are not powered on after being deployed due to EamPollingThread not running on one or more of the NSX Managers

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

NSX version is older than 4.2.0, and 
SVM is deployed on all the hosts in the cluster, but not powered on for quite some time, and
EamPollingThread is not running on 1 or more of the NSX managers

Logs: 
We keep seeing following types of messages in the EAM logs again and again for the same Agent ID. (/var/log/vmware/eam/eam.log)

HostAgent(ID: <some-agent-id>) is waiting for a hook, provisioned: true, poweredOn: false, prePowerOn: false, keeping it yellow until hooks are processed.

eg.

2024-08-21T16:36:15.148Z |  INFO | host-agent-0 | AgentWorkflowListener.java | 135 | HostAgent(ID: 'Agent:6a44cbf0-ceb9-4c34-b789-2595baa56329:null') is waiting for a hook, provisioned: true, poweredOn: false, prePowerOn: false, keeping it yellow until hooks are processed.

 

These logs tell us vCenter is waiting for NSX to tell vCenter to power on the machine.

Verify and see if EamPollingThread is running on the NSX managers by sshing into each NSX Manager and running:

grep EamPollingThread /var/log/proton/nsxapi.log

We should see log lines for EamPollingThread every 3-4 minutes. If you don’t see any recent logs, then that means EamPollingThread is not running on this NSX Manager.

 

Environment

NSX versions prior to 4.2.0 

Cause

EamPollingThread can lock up during a socket read operation (specifically, a function called SocketInputStream.socketRead0()). This can happen intermittently due to a known bug in the implementation of this function within Java Development Kit (JDK) version 1.8.'

Resolution

JDK 1.8 is used in the NSX versions prior to 4.2.0 
NSX 4.2.0 uses JDK version 1.11 which is not susceptible to this issue, thus NSX 4.2.0 and newer have this issue resolved. 

Workaround: 
On each NSX Manager that EamPollingThread is not running on, run the following:
'service proton restart'        

Note : Proton should be restarted on the Primary Manager Node which has the VIP assigned.

Restarting proton on all the managers can clear out the cache and force the NSX Manager to make a new API call for vAPI Token. It forces a full sync.