The HMS service does not start after upgrade to VMware vSphere Replication when the upgrade process fails to upgrade the embedded database

Products

VMware Live Recovery VMware vSphere ESXi

Issue/Introduction

Symptoms:

After upgrading the vSphere Replication appliance from 5.5 to 5.8 or 6.0, the HMS service fails to start.
vSphere Replication is configured to use the embedded vPostgreSQL database
The /opt/vmware/hms/logs/hms-configtool.log file contains entries similar to:

YYYY-MM-DD HH:MM:SS.SSS DEBUG com.vmware.hms.configtool.App [main] (..hms.configtool.UpgradeProcessor) | Previous HMS version: 5.5.0.0
YYYY-MM-DD HH:MM:SS.SSS DEBUG com.vmware.hms.configtool.App [main] (..hms.configtool.UpgradeProcessor) | Upgrading from 5.5.0.0 to 6.0.0.0.
YYYY-MM-DD HH:MM:SS.SSS ERROR com.vmware.hms.configtool.App [main] (..hms.configtool.DatabaseProcessor$JdbcRunnable) | Cannot execute sql statement.
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
The /opt/vmware/hms/logs/hms.log file contains entries similar to:

YYYY-MM-DD HH:MM:SS.SSS FATAL hms [main] (..vmware.hms.App) |

HMS SERVER ERROR

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [META-INF/hmsconfig.xml]: Invocation of init method failed; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: hibernate] Unable to build EntityManagerFactory
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1512)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1117)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:922)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
at com.vmware.jvsl.util.SpringService.init(SpringService.java:152)
at com.vmware.jvsl.util.SpringService.init(SpringService.java:125)
at com.vmware.jvsl.util.SpringService.<init>(SpringService.java:61)
at com.vmware.hms.HmsSingletonSpringService.<init>(HmsSingletonSpringService.java:109)
at com.vmware.hms.HmsSingletonSpringService.init(HmsSingletonSpringService.java:88)
at com.vmware.hms.App.main(App.java:123)
Caused by: javax.persistence.PersistenceException: [PersistenceUnit: hibernate] Unable to build EntityManagerFactory
at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:915)
at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:890)
at org.hibernate.ejb.HibernatePersistence.createContainerEntityManagerFactory(HibernatePersistence.java:74)
at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:290
)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:310)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509)
... 15 more
Caused by: org.hibernate.HibernateException: Missing column: collectionMoRefServerGuid in public.changerecordentity
at org.hibernate.mapping.Table.validateColumns(Table.java:289)
at org.hibernate.cfg.Configuration.validateSchema(Configuration.java:1275)
at org.hibernate.tool.hbm2ddl.SchemaValidator.validate(SchemaValidator.java:155)
at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:500)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1744)
at org.hibernate.ejb.EntityManagerFactoryImpl.<init>(EntityManagerFactoryImpl.java:94)
at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:905)
... 21 more

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere Replication 5.8.x
VMware vSphere Replication 5.5.x
VMware vSphere Replication 5.6.x
VMware vSphere Replication 6.0.x

Cause

This issue occurs when upgrading to VMware vSphere Replication 5.5, 5.8 or 6.0, the upgrade of the embedded database schema takes place before the vPostgreSQL service is fully started.

Resolution

To resolve the issue, rerun the database schema upgrade.

To rerun the database schema upgrade for upgrades to 5.8 and 6.0 and 8.x:

Take a snapshot of the virtual machine using the vSphere Client.
Log in to the vSphere Replication appliance as root using the virtual machine console or using SSH.
Type the command service hms-vpostgres restart|
Rerun the database schema upgrade with this command:

/opt/vmware/hms/bin/hms-configtool -cmd upgrade -configfile /opt/vmware/hms/conf/hms-configuration.xml
After the upgrade process completes, log in to the vSphere Replication Virtual Appliance Management Interface (VAMI) and complete the registration of the appliance.

Note: This procedure works for upgrading from replication version 5.1.3 to 5.5.1.5. In the procedure to rerun the database upgrade, run these commands instead of the one given for step 4:

cd /usr/java
ln -s jre1.7.0_76 default
/opt/vmware/hms/hms-configtool -cmd upgrade -configfile /opt/vmware/hms/conf/hms-configuration.xml