Patching the vCenter Server from 7.0U1 - U2b to 7.0U3o failed because eam service failed to start.

Products

• VMware vCenter Server

Issue/Introduction

Symptoms -

• The attempt to patch the vCenter Server from version 7.0U1-U2b to 7.0U3o failed due to the EAM (ESX Agent Manager) service failing to start.

• The source vCenter was previously restored from a file-based backup.

• The EAM service encountered the following error in the /var/log/vmware/eam/jvm.log.stderr - 

Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: org.apache.juli.logging.LogConfigurationException: java.lang.reflect.InvocationTargetException
        at org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:136)
        at org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:153)
        at org.apache.juli.logging.LogFactory.getLog(LogFactory.java:208)
        at org.apache.catalina.startup.Bootstrap.<clinit>(Bootstrap.java:50)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:134)
        ... 3 more
Caused by: org.apache.logging.log4j.core.config.ConfigurationException: No type attribute provided for component strategy
        at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationBuilder.createComponent(PropertiesConfigurationBuilder.java:330)
        at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationBuilder.processRemainingProperties(PropertiesConfigurationBuilder.java:344)
        at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationBuilder.createAppender(PropertiesConfigurationBuilder.java:225)
        at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationBuilder.build(PropertiesConfigurationBuilder.java:158)
        at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:56)
        at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:35)
        at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:302)
        at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:465)
        at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:240)
        at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
        at org.apache.logging.log4j.LogManager.getContext(LogManager.java:356)
        at org.apache.logging.log4j.appserver.tomcat.TomcatLogger$PrivateManager.getContext(TomcatLogger.java:182)
        at org.apache.logging.log4j.appserver.tomcat.TomcatLogger$PrivateManager.getLogger(TomcatLogger.java:186)
        at org.apache.logging.log4j.appserver.tomcat.TomcatLogger.<init>(TomcatLogger.java:67)

• The file /etc/vmware-eam/log4j2-tomcat.properties contains an invalid line due to a missing line break - 

rootLogger.level = info
rootLogger.appenderRef.catalina.ref = catalinaAppenderappender.catalina.strategy.type = DefaultRolloverStrategy
appender.catalina.strategy.max = 20
appender.localhost.strategy.type = DefaultRolloverStrategy
appender.localhost.strategy.max = 20

Environment

VMware vCenter Server 7.0.3
VMware vCenter Server 7.0.1
VMware vCenter Server 7.0.2

Cause

• After restoring from file-based backup, /etc/vmware-eam/log4j2-tomcat.properties does not have a line break at the end of file (fixed in VC 7.0U2c).

root@vc [ ~ ]# cat -A /etc/vmware-eam/log4j2-tomcat.properties | tail -n 1

rootLogger.appenderRef.catalina.ref = catalinaAppenderr

• When patching to version 7.0U3o, the following four lines are appended to the end of log4j2-tomcat.properties - 

appender.catalina.strategy.type = DefaultRolloverStrategy
appender.catalina.strategy.max = 20
appender.localhost.strategy.type = DefaultRolloverStrategy
appender.localhost.strategy.max = 20

• As a result, an invalid line inserted into log4j2-tomcat.properties during patching -

rootLogger.appenderRef.catalina.ref = catalinaAppenderappender.catalina.strategy.type = DefaultRolloverStrategy

Resolution

• This issue was removed in 7.0U2c and later.
• Workaround -
  
  • Revert to the snapshot/backup taken before patching.
  • Run following command to add a line break at the end.
    
    • echo >> /etc/vmware-eam/log4j2-tomcat.properties
  • Retry update.