"WARNING: VMware ESX Agent Manager may have failed to start", EAM Service fails to start after vCenter Server reboot
search cancel

"WARNING: VMware ESX Agent Manager may have failed to start", EAM Service fails to start after vCenter Server reboot

book

Article ID: 315479

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

The purpose of this kb is to show how to recover the EAM service on vCenter Server 6.0 by manually replacing the eam.properties file.

Symptoms:
  • EAM service fails to start after a reboot.
  • EAM service fails to start after recovering the vCenter from 100% storage utilization issue.
  • Unable to access the EAM MOB web interface.
  • Unable to start the service vmware-eam
  • You see the warning message:
WARNING: VMware ESX Agent Manager may have failed to start
  • In the /var/log/vmware/eam/wrapper.log (on VCSA)/ %Programdata%/VMware/vCenterServer/eam/wrapper.log (on Windows) you see entries similar to:

    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: Encountered an error running main:
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'esxAgentMgr' defined in class path resource [eam-server-beans.xml]: Could not resolve placeholder 'eam.hostd_restart_timeout'
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: at org.springframework.beans.factory.config.PropertyPlaceholderConfigurer.processProperties (PropertyPlaceholderConfigurer.java:268)
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: at org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcessBeanFactory (PropertyResourceConfigurer.java:75)
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors (AbstractApplicationContext.java:554)
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors (AbstractApplicationContext.java:528)
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.refresh (AbstractApplicationContext.java:363)
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: at org.springframework.context.support.ClassPathXmlApplicationContext. (ClassPathXmlApplicationContext.java:139)
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: at org.springframework.context.support.ClassPathXmlApplicationContext. (ClassPathXmlApplicationContext.java:83)
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: at com.vmware.eam.util.Main.main(Main.java:50)
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: at java.lang.reflect.Method.invoke(Unknown Source)
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)
    INFO | jvm 1| [YYYY-MM-DDTHH:MM:SS] | WrapperSimpleApp Error: at java.lang.Thread.run(Unknown Source)
    STATUS | wrapper | [YYYY-MM-DDTHH:MM:SS] |


Environment

VMware vCenter Server 6.x
VMware vCenter Server 6.0.x

Cause

The eam.properties is being removed during start of the vmware-eam service (EAM) in an environment without any disk space. Before EAM starts, eam.properties is getting updated using cis.utils.replace_properties_in_file. This rewrites the whole file which causes a loss of the data inside the file.

Resolution

Process to Resolve the issue:

Note: Strongly recommended to backup the vCenter Server before following below procedure.


vCenter Server Appliance
  1. Connect to your vCenter appliance using SSH or console, enter bash shell using shell command.
     
  2. Browse to /etc/vmware-eam/.
     
  3. Clone the eam.properties file to have a backup from the original file

    cp eam.properties eam.properties.old
     
  4. Determine the host ID,
cat /etc/vmware/install-defaults/sca.hostid:

The output should be a string like : 0a12dfe2-b3b7-4eb6-9737-1ec5a8e770ec

Check the vCenter FQDN: hostname -f

  1. Edit the eam.properties file "vi eam.properties"
  2. Enter insert mode by pressing i, paste the below inside the file considering modifying the highlighted strings to fit the specifications of the environment:
Note: You can also run the attached script "Recreate_eam.propertiers.sh" for quick and accurate process.
vc.proxy.host=localhost
vc.proxy.port=80
# Hostname or IP of the EAM server
# Fill only if EAM is not running on the same host as VC
eam.host=
# EAM service port used to configure the HTTP connector of the application server.
eam.int.http.port=15005
# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib
# file server.
eam.ext.port=443
eam.ext.scheme=https
# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib
# file server.
eam.ext.port.deprecated=80
eam.ext.scheme.deprecated=http
eam.support_linked_clone=true
eam.clear_db_on_startup=false
eam.debug_ref_count=false
eam.recent_event_size=20
# Enable/disable VUM integration
vum.integration=true
# Value is specified in minutes (set to 24h = 1440m)
eam.scan_for_unknown_agent_vms=1440
# The timeout to wait for hostd to restart on a host (set to 5m=300s)
eam.hostd_restart_timeout=300
# The following entries will be added verbatim to the advanced options
# of hosts on which EAM is enabled. All are optional.
Net.DVFilterBindIpAddress=169.254.0.1
Net.TrafficFilterIpAddress=
# The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).
Net.DVFilterVswifIpAddress=169.254.0.2
# Resource bundle configuration
eam.resourcebundle.filename=eam-resourcebundle.jar
# VLSI embedded tcServer configuration
#
tcserver.tmp.dir=/var/tmp/vmware/eam/tomcat
eam.web.root=/usr/lib/vmware-eam/web
# EAM SSL configuration
#
eam.keystore.type=VKS
eam.key.alias=vpxd-extension
eam.keystore.storename=vpxd-extension
# CM configuration
#
cm.url=http://localhost:18090/cm/sdk/?hostid=0a12dfe2-b3b7-4eb6-9737-1ec5a8e770ec #<====== same string from step 4
cm.wait.attempts=360
cm.wait.intervalSeconds=5
# SSO configuration
#
sso.wait.attempts=360
sso.wait.intervalSeconds=5
# VC SSL configuration
#
vc.truststore.type=VKS
vc.truststore.storename=TRUSTED_ROOTS
vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService #<====== Keep this unchanged
vc.tunnelSdkUri=https://<vCenter_FQDN>:8089/sdk/vimService #<====== Change this to your vCenter FQDN/IP
drs.demandCapacityRatio=100
 
  1. Save and exit the file by pressing <ESc> and enter :wq 
  2. Run this command to change the permission of the file:

    chmod 0644 eam.properties
     
  3. Start the EAM service:
service-control --start vmware-eam

Note: Delete the eam.properties.old or backup file if EAM is working fine.
 

vCenter Server on Windows
  1. Login to the vCenter Server using Console or RDP.
     
  2. Browse to %ProgramData%\VMware\vCenterServer\cfg\vmware-eam.
  3. Create a Backup of eam.properties using Windows Explorer Copy function
  4. Determine the host ID and FQDN,
Open %ProgramData%/VMware/vCenterServer/cfg/install-defaults/sca.hostid:
It will contain a string like : 0a12dfe2-b3b7-4eb6-9737-1ec5a8e770ec
 
Check the vCenter FQDN: Control Panel -> System -> Full computer name
  1. Open eam.properties file using Wordpad and add the values mentioned below considering modifying the highlighted strings to fit the specifications of the environment:
vc.proxy.host=localhost
vc.proxy.port=80
# Hostname or IP of the EAM server
# Fill only if EAM is not running on the same host as VC
eam.host=
# EAM service port used to configure the HTTP connector of the application server.
eam.int.http.port=15005
# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib
# file server.
eam.ext.port=443
eam.ext.scheme=https
# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib
# file server.
eam.ext.port.deprecated=80
eam.ext.scheme.deprecated=http
eam.support_linked_clone=true
eam.clear_db_on_startup=false
eam.debug_ref_count=false
eam.recent_event_size=20
# Enable/disable VUM integration
vum.integration=true
# Value is specified in minutes (set to 24h = 1440m)
eam.scan_for_unknown_agent_vms=1440
# The timeout to wait for hostd to restart on a host (set to 5m=300s)
eam.hostd_restart_timeout=300
# The following entries will be added verbatim to the advanced options
# of hosts on which EAM is enabled. All are optional.
Net.DVFilterBindIpAddress=169.254.0.1
Net.TrafficFilterIpAddress=
# The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).
Net.DVFilterVswifIpAddress=169.254.0.2
# Resource bundle configuration
eam.resourcebundle.filename=eam-resourcebundle.jar
# VLSI embedded tcServer configuration
#
tcserver.tmp.dir=C:/ProgramData/VMware/vCenterServer/runtime/tmp/vmware/eam/tomcat
eam.web.root=C:/Program Files/VMware/vCenter Server/eam/web
# EAM SSL configuration
#
eam.keystore.type=VKS
eam.key.alias=vpxd-extension
eam.keystore.storename=vpxd-extension
# CM configuration
#
cm.url=http://localhost:18090/cm/sdk/?hostid=0a12dfe2-b3b7-4eb6-9737-1ec5a8e770ec #<====== same string from step 4
cm.wait.attempts=360
cm.wait.intervalSeconds=5
# SSO configuration
#
sso.wait.attempts=360
sso.wait.intervalSeconds=5
# VC SSL configuration
#
vc.truststore.type=VKS
vc.truststore.storename=TRUSTED_ROOTS
vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService #<====== Keep this unchanged
vc.tunnelSdkUri=https://<vCenter_FQDN>:8089/sdk/vimService #<====== Change this to your vCenter FQDN/IP
drs.demandCapacityRatio=100
  1. Save the file 
  2. Start the EAM service:
service-control --start vmware-eam

Note: Delete the eam.properties.old or backup file if EAM is working fine.



Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ
Path for Install Defaults on Windows vCenter Server : %ProgramData%\VMware\vCenterServer\cfg\install-defaults\

For more information:
Resolution: Internal error occurs during vSphere ESX Agent Manager pre-upgrade checks

Attachments

Recreate_eam.properties.sh get_app
Powered by Wolken Software