EAM fails with OutOfMemoryErrors

Products

• VMware vCenter Server
• VMware NSX for vSphere

Issue/Introduction

During host preparation in VMware NSX for vSphere 6.x, you experience these symptoms:

• The vSphere ESX Agent Manager (EAM) threads get stuck or EAM service may not start.
• In the C:\ProgramData\VMware\VMware VirtualCenter\Logs\eam.log file on the vCenter Server, you see entries similar to:
  
  2015-04-23T06:31:38.998Z | DEBUG | agent-4 | AgentImpl.java | 2251 | Agent AgentImpl(ID:'Agent:e83ee5ce-####-####-####-###########20:3c36c535-####-####-####-##########11') (VcHostSystem(ID: host-359)) scheduled a VIB task and is waiting for it to complete.
  2015-04-23T06:31:40.379Z | DEBUG | compute-res-0 | ExecutorImpl.java | 57 | JOB SUBMITTED: [#1420322380] ClusterVibConditionalJob(ManagedObjectReference: type = ClusterComputeResource, value = domain-c343, serverGuid = 885ab205-####-####-####-##########79), trace: .:scheduleNewVibTask:.
  2015-04-23T06:31:40.379Z | DEBUG | compute-res-0 | AuditedJob.java | 70 | JOB STARTED: [#1420322380] ClusterVibConditionalJob(ManagedObjectReference: type = ClusterComputeResource, value = domain-c343, serverGuid = 885ab205-####-####-####-##########79), trace: .:scheduleNewVibTask:.
  
  INFO | jvm 1 | 2015/04/23 06:32:20 | java.lang.OutOfMemoryError: Java heap space
  INFO | jvm 1 | 2015/04/23 06:32:20 | Dumping heap to /var/log/vmware/eam/java_pid24722.hprof ...
  INFO | jvm 1 | 2015/04/23 06:32:22 | Heap dump file created [138746180 bytes in 2.438 secs]
  INFO | jvm 1 | 2015/04/23 06:32:22 | Exception in thread "host-359-0" java.lang.OutOfMemoryError: Java heap space
  INFO | jvm 1 | 2015/04/23 06:32:22 | at sun.nio.cs.StreamEncoder.write(Unknown Source)
  INFO | jvm 1 | 2015/04/23 06:32:22 | at java.io.OutputStreamWriter.write(Unknown Source)
  INFO | jvm 1 | 2015/04/23 06:32:22 | at java.io.Writer.write(Unknown Source)
  
  INFO | jvm 1 | 2015/04/23 06:32:23 | at com.vmware.eam.vc.VcListener$4.call(VcListener.java:338)
  INFO | jvm 1 | 2015/04/23 06:32:23 | at com.vmware.eam.vc.VcListener$4.call(VcListener.java:335)
  INFO | jvm 1 | 2015/04/23 06:32:23 | at com.vmware.eam.lock.RaceProtection.exec(RaceProtection.java:68)
  INFO | jvm 1 | 2015/04/23 06:32:23 | at com.vmware.eam.lock.RaceProtection.exec(RaceProtection.java:51)
  INFO | jvm 1 | 2015/04/23 06:32:23 | at com.vmware.eam.vc.VcListener.unregisterObserver(VcListener.java:335)
  
  2015-04-23T06:35:43.033Z | DEBUG | agent-3 | AgentImpl.java | 2251 | Agent AgentImpl(ID:'Agent:86cab9ca-####-####-####-##########a5:3c36c535-####-####-####-##########11') (VcHostSystem(ID: host-363)) scheduled a VIB task and is waiting for it to complete.
  
  For more information on vCenter Server log location, see Location of vCenter Server log files (1021804).
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 6.0.x
VMware NSX for vSphere 6.0.x
VMware vCenter Server Appliance 6.0.x
VMware NSX for vSphere 6.1.x
VMware NSX for vSphere 6.2.x

Cause

This issue occurs due to the bug in EAM where it tries to poll the same updates again from VM exhausting the memory of the EAM.

Resolution

This issue is resolved in:

• vCenter Server 6.0 Update 3
• vCenter Server 6.5 Update 1

To work around the issue, increase the EAM heap size to the desired level using cloudvm-ram-size utility.
 

To increase the heap size:

1. Stop the VMware VirtualCenter Server service that also stops the vCenter Inventory service. For more information, see Stopping, starting, or restarting VMware vCenter Server services (1003895).
   
   Note: To stop the EAM service on the vCenter Server, run this command:
   
   service vmware-eam stop
    
2. Open command line utility and navigate to the folder C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin.
    
3. Run the command:
   
   cloudvm-ram-size -C NewMemorysizeinMB EsxAgentManager
   
   In case of vCenter Appliance:
   
   cloudvm-ram-size -C NewMemorysizeinMB vmware-eam
   
   In case of Windows vCenter Server:
   
   cloudvm-ram-size -C NewMemorysizeinMB vspherewebclientsv
    
4. Restart the VMware VirtualCenter Server service.
   
   To restart the eam service on vCenter Server, run this command:
   
   service vmware-eam restart

To confirm that heap size is increased, run the commands:

1. cloudvm-ram-size -J vmware-eam -O /etc/vmware/java/vmware-eam_jvm.conf
   
   This output shows the allocated heap size for EAM.
    
2. Echo $?
   
   0 indicates success, any non-zero value indicates failure.
    
3. If there is a failure, check the file permissions of vmware-eam_jvm.conf and ensure read and write privileges on the file.
   
   ls -ltr /etc/vmware/java/vmware-eam_jvm.conf
   -rw------- 1 eam users 77 Mar 10 06:56 /etc/vmware/java/vmware-eam_jvm.conf

Additional Information

How to stop, start, or restart vCenter Server services
Location of vCenter Server log files