NSX Host Preparation fails with an EAM error: "Host is no longer in vCenter inventory"
search cancel

NSX Host Preparation fails with an EAM error: "Host is no longer in vCenter inventory"

book

Article ID: 319519

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
  • NSX Host preparation failure.
    • VXLAN / FDW
    • Guest Introspection
    • Third Party Integration
  • EAM is not preparing newly added host to NSX Cluster.
  • EAM MOB may be loading a blank page.
  • EAM logs contains entries similar to:

    | ERROR | vc-callback | AgencyImpl.java | 1295 | Failed to add agent to agency (AgencyImpl(ID:'Agency:XXXXX')): java.lang.RuntimeException: Host is no longer in vCenter inventory.


Cause

This scenario occurs by the method used to migrate ESXi host(s) between clusters. For example, migrating an ESXi host from standalone (No cluster affiliation) into an NSX prepared cluster.  

The host parent relationship is present in two properties:

  • (Cluster) ComputeResource's host (a list of hosts' IDs)
  • HostSystem's parent (id of a (Cluster)ComputeResource)

These two properties are not updated simultaneously.

Seldom (due to several conditions - network latency, concurrent locks, etc.) when an ESXi host is moved from standalone (or directly added to the cluster), the property change for cluster.hosts is received but host.parent still points to the previous object (that no longer exists).

When this condition occurs and EAM tries to create an internal representation of the host by traversing its parent, the error "Host is no longer in vCenter inventory" occurs and no agent (no preparation) is created for this host.

There are two different versions of this issue depending on the vCenter version:

Cause 1 - "Host is no longer in vCenter inventory" - vCenter 6.0 prior to U3c

  1. ESXi Host is moved from Standalone to NSX prepared cluster.
  2. ESXi Host is removed from vCenter without first removing EAM / NSX Configuration.
  3. Later, ESXi Host is added back to vCenter and to a NSX prepared cluster.
  4. EAM tries to prepare it by creating an Agent which eventually fails since the HostSystemMo in VPXD is pointing to the old or wrong parent ID that is no longer existing in the VPXD inventory.

    Note: This is happening for a short time window when moving standalone host into a cluster because standalone host's parents are stopped after the host joins a cluster.
     
  5. EAM fails with 'RuntimeException: Host is no longer in vCenter inventory' and host moId is still persisted in EAM's runtime.

Cause 2 - "Host is no longer in vCenter inventory" - vCenter 6.0 U3c and U3d

  1. Similar to cause 1; ESXi Host is moved from Standalone to NSX prepared cluster.
  2. vCenter 6.0 U3c and 6.0 U3d introduce a different problem with a mechanism which cleans up every record for the created Agent regarding adding host except the mapping record for that Agent in the Agency.
  3. The software defect is that EAM will unload all of the existing agents from the agency in EAM Runtime.
  4. EAM logs contains entries similar to:

    | ERROR | vc-callback | AgencyImpl.java | 1295 | Failed to add agent to agency (AgencyImpl(ID:'Agency:XXXXX')): java.lang.RuntimeException: Host is no longer in vCenter inventory
    | WARN | eam-0 | AgencyImpl.java | 1954 | Disposing agency: AgencyImpl(ID:'Agency:XXXXXXX') due to failed load up.
    | ERROR | eam-0 | AgencyImpl.java | 1895 | Failed to load agent: ManagedObjectReference: type = Agent, value = ########-####-####-####-########46fa, serverGuid = ########-####-####-####-########a719. Error: null java.lang.NullPointerException


EAM MOB Fails to Load “Host not covered by scope anymore”

  • Potential side effect consequence of either Cause 1 or Cause 2.
  • Caused by restarting EAM service while it is in bad state due to issues described above.
  • EAM logs contains entries similar to:

    | ERROR | eam-0 | VcListener.java | 116 | An unexpected error in the changes polling loop
    | ERROR | eam-0 | AgencyImpl.java | 1895 | Failed to load agent: ManagedObjectReference: type = Agent, value = ########-####-####-####-########2e9a, serverGuid = ########-####-####-####-########a719. Error: Host not covered by scope anymore

Resolution

This issue is resolved in VMware vCenter Server 6.5, available at Broadcom Downloads.

Note: Otherwise, database maintenance is required 

Workaround:
To work around this issue:

  1. Depending on the severity of the situation, restart of EAM might resolve agents missing from EAM runtime.
  2. A preventative workaround would be to add host to non-NSX prepared cluster prior to moving to NSX Cluster. (Avoid moving standalone host to NSX Prepared cluster)
    • There is a VPXD parent ID assigned to host while it is standalone, this is not the case while host is part of a cluster. The software defect is related to the VPXD parent ID and mechanism of removal while transitioning to NSX prepared cluster.



Additional Information

Powered by Wolken Software