NSX-v Manager reports EAM Status as Down, even when EAM is up and running on vCenter Server

Products

• VMware NSX for vSphere

Issue/Introduction

• Windows vCenter Server is running an application or service that listens on TCP port 8089.
• NSX-v Manager reports EAM Status as Down. The click here to resolve option to restart the 'VMware ESX Agent Manager" service does not resolve the issue.
• NSX-v Manager vsm.log contains message like the following:

<Date>T<Time> |  WARN | eam-0 | EamAddress.java | 206 | Check the vCenter Server Settings. One of FQDN, ManagedIP, AutoManagedIPV4, AutoManagedIPV6 should be correctly configured. The EAM file server becomes unreachable.
<Date>T<Time> |  WARN | eam-0 | EamAddress.java | 206 | Check the vCenter Server Settings. One of FQDN, ManagedIP, AutoManagedIPV4, AutoManagedIPV6 should be correctly configured. The EAM file server becomes unreachable.
<Date>T<Time> | DEBUG | eam-0 | ActiveObservers.java | 168 | Registered observer com.vmware.eam.vc.impl.VcOptions@68b7cf67 for ManagedObjectReference: type = OptionManager, value = VpxSettings, serverGuid = 7167####-####-####-####-####5a0bb795
<Date>T<Time> | DEBUG | eam-0 | ActiveObservers.java | 168 | Registered observer ClientAuthenticator listening for changes to ManagedObjectReference: type = SessionManager, value = SessionManager, serverGuid = 7167####-####-####-####-####5a0bb795 for ManagedObjectReference: type = SessionManager, value = SessionManager, serverGuid = 7167####-####-####-####-####5a0bb795
<Date>T<Time> | DEBUG | eam-0 | ActiveObservers.java | 209 | Initial state
7167####-####-####-####-####5a0bb795::OptionManager:VpxSettings: com.vmware.eam.vc.impl.VcOptions@68b7cf67,
7167####-####-####-####-####5a0bb795::SessionManager:SessionManager: ClientAuthenticator listening for changes to ManagedObjectReference: type = SessionManager, value = SessionManager, serverGuid = 7167####-####-####-####-####5a0bb795,
7167####-####-####-####-####5a0bb795::ExtensionManager:ExtensionManager: EsxAgentManager listening for changes to ManagedObjectReference: type = ExtensionManager, value = ExtensionManager, serverGuid = 7167####-####-####-####-####5a0bb795,
<Date>T<Time> |  INFO | eam-0 | VcListener.java | 197 | Starting to poll changes...
<Date>T<Time> |  INFO | localhost-startStop-1 | Log4JConfigurator.java | 36 | just initialized
<Date>T<Time> |  INFO | localhost-startStop-1 | Log4JConfigurator.java | 36 | just initialized
<Date>T<Time> | DEBUG | eam-0 | ExecutorImpl.java | 57 | JOB SUBMITTED: [#639501465] com.vmware.eam.vc.vim.impl.ChangeStream$3@261e0499
<Date>T<Time> | DEBUG | eam-0 | ExecutorImpl.java | 57 | JOB SUBMITTED: [#1563554245] com.vmware.eam.vc.vim.impl.ChangeStream$3@5d31f1c5
<Date>T<Time> | DEBUG | vc-callback | AuditedJob.java | 70 | JOB STARTED: [#639501465] com.vmware.eam.vc.vim.impl.ChangeStream$3@261e0499
<Date>T<Time> |  WARN | vc-callback | EamAddress.java | 234 | Ignores an empty VC address option with key VirtualCenter.ManagedIP
<Date>T<Time> |  WARN | vc-callback | EamAddress.java | 234 | Ignores an empty VC address option with key VirtualCenter.AutoManagedIPV6
<Date>T<Time> | DEBUG | vc-callback | EamAddress.java | 197 | EAM address set to VCDVC2-VA-1.wh.intermedia.net based on VC option VirtualCenter.FQDN
<Date>T<Time> |  WARN | vc-callback | EamAddress.java | 234 | Ignores an empty VC address option with key VirtualCenter.ManagedIP

 

 

 

Environment

VMware NSX for vSphere 6.4.x

Cause

The NSX-v manager by default connects to EAM on port 8089. However, in some environments, reverse proxy on vCenter can be configured to listen on ports other than 8089.
Prior to NSX 6.4, this port number could not be changed.
When vCenter is installed, VPXD selects a TCP port (8089, by default) for the ESX Agent Manager service. If this port is already in use by some other application, then VPXD selects the next free TCP port, 8090, for the ESX Agent Manager service.
By default the Splunk Add-on for VMware connects to the Data Collection Node (DCN) on the default Splunk management port, TCP 8089.

Resolution

As of NSX 6.4, it is now possible to change the EAM TCP port number that NSX connects to, from the default of TCP 8089:

1. Verify that NSX version 6.4.0 or later
2. Verify that vCenter is listening on TCP port 8089, to some other service other than EAML

- https://vCenter-Server-IP-Address/8089
- if Splunk or some other app is installed on vCenter Server, this will serve up the Splunk Atom Feed: splunkd slash page

3. Verify that vCenter EAM is listening on TCP 8090

- examine the TCP port defined in eam.properties file on vCenter Server:
vc.tunnelSdkUri=https://sdkTunnel:##{VC_SDK_TUNNEL_PORT}##/sdk/vimService

4. Verify that NSX is using the default EAM TCP port 8089, using the following REST API call:

GET https://NSX-Manager-IP-Adddress/api/2.0/services/vcconfig/tunnelport
- this should return a 200 OK, and integer 8089 in the response body

5. Change the NSX EAM TCP port to 8090, using the following REST API call:

PUT https://NSX-Manager-IP-Adddress/api/2.0/services/vcconfig/tunnelport/8090

6. Verify that NSX is now using TCP port 8090, using the following REST API call:

GET https://NSX-Manager-IP-Adddress/api/2.0/services/vcconfig/tunnelport
- this should return a 200 OK, and integer 8090 in the response body

7. From the NSX Manager Web UI, stop and then start the NSX Management service
8. From the vSphere Web Client, Networking and Security, Installation, Host Preparation, verify that EAM Status is Up