vCenter Server EAM service goes down due to bad properties file
search cancel

vCenter Server EAM service goes down due to bad properties file

book

Article ID: 313871

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

The purpose of this KB is to show how to recover the EAM service from 6.7 U3 onwards.

Symptoms:

  • Mostly such issues can happen after disk space issues.
  • Below logging will be recorded in /var/log/vmware/eam/eam.log

    YYYY-MM-DDThh:mm:ss.msZ | WARN | localhost-startStop-1 | AbstractApplicationContext.java | 550 | Exception encountered during context initialization - cancelling refresh attempt:
    org.springframework.beans.factory.BeanCreationException:
    Error creating bean with name 'vimMOIdTransformer' defined in class path resource [eam-server-vim-beans.xml]: Cannot resolve reference to bean 'vpxdServerGuidProdiver' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vpxdConnection' defined in class path resource
    [eam-server-vim-beans.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate
    [com.vmware.eam.vc.VcConnection]: Constructor threw exception; nested exception is java.lang.RuntimeException: Wrong Vc sdk tunnel URL: <fqdn>:8089/sdk/vimService
    Caused by: java.net.URISyntaxException: Illegal character in opaque part at index 43: <fqdn>:8089/sdk/vimService
 



Environment

VMware vCenter Server Appliance 6.7 U3 and later
VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

Before EAM starts, eam.properties and eam-vim.properties is getting updated using an internal script. This tries to rewrite the whole file, but because of disk space issues it is unable to and this impacts the eam services.

Prior to vCenter Server 6.7 U3, vCenter kept all EAM configurations in a single file called /etc/vmware-eam/eam.properties.
Beginning in vCenter 6.7 U3, EAM configurations are broken into two files called /etc/vmware-eam/eam.properties and /etc/vmware-eam/eam-vim.properties.  Most of the configurations are now located in the /etc/vmware-eam/eam-vim.properties file, however with the older versions /etc/vmware-eam/eam.properties files contains all the information.

Resolution

Warning: Please make sure you have a healthy backup of vCenter Server or offline snapshot of vCenter virtual machine (VM powered off)

  1. Connect to your VCSA using SSH or console, enter bash shell using shell command.
  2. Browse to /etc/vmware-eam/.
  3. Backup the eam.properties and eam-vim.properties file to have a backup from the original file.
Here we are using "mv" to rename the file itself.

# mv eam.properties eam.properties.old
# mv eam-vim.properties eam-vim.properties.old
  1. Modify/update the eam.properties and insert below lines and save the file. File content is same for 6.7 U3/7/x/8.x

    vi eam.properties
    Copy the below content.
    Save the file
    'Esc' key then ':wq!'
    enter

 


#########################################################################
# Copyright 2013-2017,2021 VMware, Inc.  All rights reserved. VMware Confidential
#########################################################################

eam.support_linked_clone=true
eam.debug_ref_count=false
eam.mm.policy.one.by.one=false

5. Modify/update the eam-vim.properties and insert below lines and save the file

 vi eam-vim.properties
Copy the below content as per the vCenter version. 
Save the file
'Esc' key then ':wq!'
enter  



For vCenter 6.7 U3 and later : Update the red marked parameters as per environment.

 

#########################################################################
# Copyright 2013-2016 VMware, Inc.  All rights reserved. VMware Confidential
#########################################################################
# VC location
vc.proxy.host=localhost
vc.proxy.port=80
vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService
vc.tunnelSdkUri=https://<FQDN>:8089/sdk/vimService
# VC SSL trust
vc.truststore.type=VKS
vc.truststore.storename=TRUSTED_ROOTS
# EAM location
# Hostname or IP of the EAM server
# Fill only if EAM is not running on the same host as VC
eam.host=FQDN
# EAM AuthN
#
eam.keystore.type=VKS
eam.keystore.storename=vpxd-extension
eam.key.alias=vpxd-extension
############################################################################
#
# The data below is used by not-eam-vim beans hence is going to be moved out
#
############################################################################
# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib
# file server.
eam.ext.port=443
eam.ext.scheme=https
# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib
# file server.
eam.ext.port.deprecated=80
eam.ext.scheme.deprecated=http
drs.demandCapacityRatio=100



For vCenter 7.0 & 8.0 :

eam-vim.properties 

Change the fqdn in all places and the ls.node.id .

To find ls.node.id, run :  cat /etc/vmware/install-defaults/vmdir.ldu-guid

#########################################################################
# Copyright 2013-2019 VMware, Inc.  All rights reserved. VMware Confidential
#########################################################################
# VC location
vc.proxy.host=localhost
vc.proxy.port=80
vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService
vc.tunnelSdkUri=https://<fqdn>:8089/sdk/vimService
# VC SSL trust
vc.truststore.type=VKS
vc.truststore.storename=TRUSTED_ROOTS
# EAM location
# Hostname or IP of the EAM server
# Fill only if EAM is not running on the same host as VC
eam.host=<fqdn>
# EAM AuthN
#
eam.keystore.type=VKS
eam.keystore.storename=vpxd-extension
eam.key.alias=vpxd-extension
# LookupService
ls.url=https://<fqdn>:443/lookupservice/sdk
ls.node.id= <copy-id-from-above-ldu-guid>
############################################################################
#
# The data below is used by not-eam-vim beans hence is going to be moved out
#
############################################################################
# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib
# file server.
eam.ext.port=443
eam.ext.scheme=https
# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib
# file server.
eam.ext.port.deprecated=80
eam.ext.scheme.deprecated=http
drs.demandCapacityRatio=100

 

  1. Validate the file permissions. Should be as shown below.

    For 6.7 U3 and 7.0 : 
    -rw-r--r-- 1 eam cis 1.3K MM DD hh:mm eam-vim.properties
    -rw-r--r-- 1 eam cis 317  MM DD hh:mm eam.properties

    For 8.0 : 
    -rw-r--r-- 1 root root 1.3K MM DD hh:mm eam-vim.properties
    -rw-r--r-- 1 root root 317  MM DD hh:mm eam.properties


    Below commands can be used to fix the file permissions and ownership.

chmod 644 eam-vim.properties
chown eam:cis eam-vim.properties (For 6.7 & 7.0)
chown root:root eam-vim.properties (For 8.0)

  1. Restart EAM services

service-control --restart vmware-eam


Additional Information

Impact/Risks:
This affects EAM which communicates between vCenter and NSX to install vibs for NSX-V and Guest Introspection.

Powered by Wolken Software