"The backing EAM agency for this deployment could not be found" error after restore from backup

Products

• VMware NSX

Issue/Introduction

Symptoms:

• In the NSX Manager, you may see high priority alerts similar to:
  
  Nov 8 04:35:00 NSXMGR 2016-11-08 04:35:00.422 GMT+00:00 INFO TaskFrameworkExecutor-18 SystemEventDaoImpl:133 - [SystemEvent] Time:'Tue Nov 08 04:35:00.410 GMT+00:00 2016', Severity:'High', Event Source:'Security Fabric', Code:'250024', Event Message:'The backing EAM agency for this deployment could not be found. It is possible that the VC services may still be initializing. Please try to resolve the alarm to check existence of the agency. In case you have deleted the agency manually, please delete the deployment entry from NSX.', Module:'Security Fabric', Universal Object:'false
• In the /var/log/vmware/eam/eam.log file or the C:\ProgramData\VMware\vCenterServer\logs\eam\eam.log file of the ESX Agent Manager (EAM), you see entries similar to:
  
  2017-05-11T20:30:04.202-04:00 | WARN | vlsi | DispatcherImpl.java | 306 | Managed object doesn't exist (vmodl.fault.ManagedObjectNotFound) { faultCause = null, faultMessage = null, obj = ManagedObjectReference: type = Agency, value = ########-4dad-4a2c-b92e-###########, serverGuid = eam-server-guid } at com.vmware.vim.vmomi.server.impl.BodyHandlerImpl.handleBody(BodyHandlerImpl.java:108) at com.vmware.vim.vmomi.server.impl.DispatcherImpl.doDispatch(DispatcherImpl.java:283) at com.vmware.vim.vmomi.server.impl.DispatcherImpl.dispatch(DispatcherImpl.java:303) at com.vmware.vim.vmomi.server.http.impl.CorrelationDispatcherTask.run(CorrelationDispatcherTask.java:58) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX for vSphere 6.4.x

Cause

This issue occurs due to inconsistency between NSX Manager DB and vCenter Server EAM DB. After restoring NSX Manager from a backup file, NSX DB is reverted to the past data whereas vCenter Server EAM DB is unchanged.

Resolution

To resolve this issue, delete old deployment unit data from NSX DB by using NSX Manager REST API DELETE Method:

1. Execute the following method via REST client to get the current inconsistent state and the solution.
   
   
   Method: GET
   URL: https://{NSXM-IP}/api/2.0/si/fabric/sync/conflicts
    
2. Delete deployment unit data.
   
   REST API:
   
   Method: PUT
   URL: https://{NSXM-IP}/api/2.0/si/fabric/sync/conflicts
   Body:
   <conflictResolverInfo><deploymentUnitAction>DELETE</deploymentUnitAction></conflictResolverInfo>
   
   CURL:
   curl -X PUT -d "<conflictResolverInfo><deploymentUnitAction>DELETE</deploymentUnitAction></conflictResolverInfo>" -H "Content-Type: Application/xml" -k -u username:password https://NSXM-IP/api/2.0/si/fabric/sync/conflicts
   

For more information on how to make API calls to the NSX Manager, see the Using the NSX REST API section in the VMware NSX for vSphere API Guide.

Note: username:password have to be entered in the API call and need to be authorized to preform API calls.