EAM service down Alarm is generated on NSX with multiple VCs registered

Products

• VMware NSX

Issue/Introduction

• EAM status down alarm is reported by one of the vCenter in the NSX-T
• Issue is noticed when there are multiple vCenter servers registered as compute manager in NSX-T
• The eam status of the vCenter reported the eam down alarm is up and healthy in the vCenter-1
• Running the command service-control --status vmware-eam via vCenter shell confirms the eam is up
• The eam mob is accessible at : https://<vc>/eam/mob/
• The /var/log/vmware/eam/eam.log file reports no error in vCenter-1
• The /var/log/vmware/eam/eam_api.log logging entries for NSX logging in and out
  
  2024-05-31T03:42:30.683Z | INFO | vlsi | ClientAuthenticator.java | 254 | User logged in: com.vmware.nsx.management.nsxt (version eam/6.7) 2024-05-31T03:45:22.997Z | INFO | vim-inv-update | ClientAuthenticator.java | 445 | User logged out of vCenter: com.vmware.nsx.management.nsxt


• EAM issue occurred for vCenter with cmID 7aae51a7-xxxx-xxxx-xxxx-e4d774f52c8e. However it's being recorded against different vcenter with different cmID 3a####5c-####-####-####-cca4####2a26 as seen in nsxapi.log file in NSX-T manager
  
  
  2024-05-21T04:06:20.945Z ERROR EamPollingThread EamServiceImpl 1171 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP40451" level="ERROR" subcomp="manager"] Error occured while fetching eam status for cmId 7aae51a7-####-####-####-e4d774f52c8e, (eam.fault.EamServiceNotInitialized) {
   faultCause = null,
    faultMessage = null
 }
2024-05-21T04:06:20.945Z INFO EamPollingThread HostPinnedExecuteOperationImpl 1171 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [HostPinned]: CmPlugin response obtained for action [GetEamStatus], compute-manager: 7aae51a7-####-####-####-e4d774f52c8e. Status KeyValuePair{key='eamStatus', value='down'}.
 2024-05-21T04:06:20.948Z INFO EamPollingThread EamServiceStatusServiceImpl 1171 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Current EAM status UP, updated EAM status DOWN for cmID 3a3ee45c-####-####-####-cca419fd2a26

Cause

There is an issue with eam service health on one of the vcenter compute manager registered to the NSX-T. The NSX-T uses incorrect CM-ID when generating EAM down alarm. 

Resolution

This is a known issue and engineering team is aware of it. This issue is resolved in version 4.1.1.0.0.21575634. 

Workaround:

To workaround the issue, resolve the eam issue on all the compute managers registered to the NSX-T so that eam down alarm is not generated.