"Internal error occurs during vSphere ESX Agent Manager pre-upgrade checks" upgrading the vCenter Server Appliance

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

Upgrading the vCenter Server Appliance fails on Stage 2 Pre-upgrade checks with the error:

Error: Internal error occurs during vSphere ESX Agent Manager pre-upgrade checks.

In the /var/log/vmware/upgrade/CollectRequirements_com.vmware.eam_xx.log file, you see similar to:

Waiting for VMware ESX Agent Manager.......

WARNING: VMware ESX Agent Manager may have failed to start.
..
            "id": "install.ciscommon.command.errinvoke",
            "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: '",
            "translatable": "An error occurred while invoking external command : '%(0)s'"
...
ERROR:root:Unable to start service vmware-eam, Exception: {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vmware-eam"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vmware-eam'",
            "translatable": "An error occurred while starting service '%(0)s'"
..
Traceback (most recent call last):
  File "/tmp/vmware-root/tmpvmware252/payload/componentPhaseLauncher.py", line 438, in main
    executionResult = systemExtension(exeContext)
  File "/tmp/vmware-root/tmpvmware252/libs/sdk/extensions.py", line 94, in __call__
    result = self.extension(*args)
  File "/tmp/vmware-root/tmpvmware252/libs/sdk/extensions.py", line 110, in _func
    return func(*args)
  File "/tmp/vmware-root/tmpvmware252/payload/component-scripts/eam/__init__.py", line 104, in preCheck
    sourceVcPlatform
  File "/tmp/vmware-root/tmpvmware252/payload/component-scripts/eam/__init__.py", line 141, in _findUpgradeMistmatches
    mismatch = check(version, platform)
  File "/tmp/vmware-root/tmpvmware252/payload/component-scripts/eam/__init__.py", line 179, in _eamServiceStartedCheck
    if _startEamService(version, platform):
  File "/tmp/vmware-root/tmpvmware252/payload/component-scripts/eam/__init__.py", line 313, in _startEamService
    serviceManager.start(serviceName)
  File "/tmp/vmware-root/tmpvmware252/libs/sdk/service_manager.py", line 966, in wrapper
    return getattr(controller, attr)(*args, **kwargs)
  File "/tmp/vmware-root/tmpvmware252/libs/sdk/service_manager.py", line 874, in start
    super(VMwareServiceController, self).start(serviceName)
  File "/tmp/vmware-root/tmpvmware252/libs/sdk/service_manager.py", line 732, in start
    raise IllegalServiceOperation(errorText)
IllegalServiceOperation: Service cannot be started. Error: INFO:root:Service: vmware-eam, Action: start
[YYYY-MM-DDTHH:MM:SS] Invoked command: ['/sbin/service', u'vmware-eam', 'start']
[YYYY-MM-DDTHH:MM:SS] RC = 1
Stdout = Starting VMware ESX Agent Manager...
Waiting for VMware ESX Agent Manager.......
WARNING: VMware ESX Agent Manager may have failed to start.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server Appliance 6.0.x
VMware vCenter Server 7.0.x
VMware vCenter Server 6.0.x
VMware vCenter Server Appliance 6.5.x

Cause

This issue occurs when the /etc/vmware-eam/eam.properties file is empty on the source vCenter Server Appliance.

Resolution

To resolve this recreate the eam.properties file.

IMPORTANT: It is strongly recommended to backup the vCenter Server Appliance before following this procedure.

Process to recreate the eam.properties file:

Connect to your vCenter appliance using SSH and root credentials.
Navigate to the /etc/vmware-eam/ directory.
Copy the eam.properties file to create a backup:

cp eam.properties eam.properties.old
Run this command:

cat /etc/vmware/install-defaults/sca.hostid
Make note of the output.

For example: ########-####-####-####-########70ec

Edit the eam.properties file with this command:

vi eam.properties
Paste the below inside the file modifying the highlighted strings to match the environment:

#########################################################################
# Copyright 2013-2014 VMware, Inc. All rights reserved. VMware Confidential
#########################################################################
vc.proxy.host=localhost
vc.proxy.port=80
# Hostname or IP of the EAM server
# Fill only if EAM is not running on the same host as VC
eam.host=
# EAM service port used to configure the HTTP connector of the application server.
eam.int.http.port=15005
# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib
# file server.
eam.ext.port=443
eam.ext.scheme=https
# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib
# file server.
eam.ext.port.deprecated=80
eam.ext.scheme.deprecated=http
eam.support_linked_clone=true
eam.clear_db_on_startup=false
eam.debug_ref_count=false
eam.recent_event_size=20
# Enable/disable VUM integration
vum.integration=true
# Value is specified in minutes (set to 24h = 1440m)
eam.scan_for_unknown_agent_vms=1440
# The timeout to wait for hostd to restart on a host (set to 5m=300s)
eam.hostd_restart_timeout=300
# The following entries will be added verbatim to the advanced options
# of hosts on which EAM is enabled. All are optional.
Net.DVFilterBindIpAddress=169.254.0.1
Net.TrafficFilterIpAddress=
#The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).
Net.DVFilterVswifIpAddress=169.254.0.2
# Resource bundle configuration
eam.resourcebundle.filename=eam-resourcebundle.jar
# VLSI embedded tcServer configuration
#
tcserver.tmp.dir=/var/tmp/vmware/eam/tomcat
# EAM SSL configuration
#
eam.keystore.type=VKS
eam.key.alias=vpxd-extension
eam.keystore.storename=vpxd-extension
# CM configuration
#
cm.url=http://localhost:18090/cm/sdk/?hostid=########-####-####-####-########70ec <====== Use the output recorded on step 5
cm.wait.attempts=360
cm.wait.intervalSeconds=5
# SSO configuration
#
sso.wait.attempts=360
sso.wait.intervalSeconds=5
# VC SSL configuration
#
vc.truststore.type=VKS
vc.truststore.storename=TRUSTED_ROOTS
vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService
vc.tunnelSdkUri=<vCenter_FQDN>:8089/sdk/vimService <====== Change this to your vCenter FQDN/IP

drs.demandCapacityRatio=100

Save and exit the file.
Make sure the below line is added to the eam.properties. If its missing append the below line to the end of the eam.properties:
- eam.web.root=/usr/lib/vmware-eam/web
Change permissions on the file with this command:

chmod 0644 eam.properties

Start the EAM service:

service-control --start vmware-eam

Note: You may delete the eam.properties.old file after the upgrade is complete. If you see an error similar to:

Could not resolve placeholder 'eam.web.root' in string value "${eam.web.root}"; nested exception is java.lang.IllegalArgumentException

Additional Information

"Internal error occurs during Export of vSphere ESX Agent Manager" error when migrating vCenter Server 6.0 to 6.5 (2147898)
"WARNING: VMware ESX Agent Manager may have failed to start", EAM Service fails to start after vCenter Server reboot (2148582)