vSphere Replication appliance update from 9.0.0 to 9.0.2 takes long time to complete

Products

• VMware vCenter Server

Issue/Introduction

Symptoms

VR update from 9.0.0 to 9.0.2 takes long time to finish.
At times it takes 2-3 hours to complete.

We see following in the hms.log for one of the replicating VM:

2024-12-17 21:18:40.037 ERROR hms.replication.monitor [hms-pcm-updatehandler-thread-1508] (..hms.replication.ReplicationMonitor$PCUpdatesHandler) [operationID=4e2efad1-9508-45f0-b5b0-13333199d287-HMSINT-22178, operationID=995487fb-030d-47af-80c8-fce90d89ffa3-HMSINT-22169] | Update handling failed for group GID-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, (vmodl.query.Change) {
   dynamicType = null,
   dynamicProperty = null,
   name = currentRpoViolation,
   op = assign,
   val = 187
}


Due to the ongoing tasks by VR for this replicating VM (GID-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx), the JDBC connection is not available and DB schema is temporarily not accessible for update.

hms.log further reports following:

org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.GenericJDBCException:
 Unable to acquire JDBC Connection
        at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:467) ~[spring-orm-5.3.31.jar:5.3.31]
        at com.vmware.jvsl.persistence.transaction.HmsJpaTransactionManager.doBegin(HmsJpaTransactionManager.java:88) ~[jvsl-persistence-9.0.0.jar:?]
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400) ~[spring-tx-5.3.31.jar:5.3.31]
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) ~[spring-tx-5.3.31.jar:5.3.31]

Caused by: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection

Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!

Environment

vSphere Replication 9.0.x

Cause

This is caused due to VRMS database being busy running tasks related to VM replication and the handle of DB not available for the update/upgrade.

Resolution

a) Pause all VM replications (Outgoing as well as Incoming)
b) Allow VR to complete the ongoing tasks running in the backend and let it free up the JDBC connection.
c) Once done, VR upgrade would automatically complete.
d) If issue persist, revert the snapshot (if taken before the update/upgrade), perform the power-cycle of the VR appliance. Attempt to update again.