vSphere Replication Management Service is inaccessible after upgrading to vCenter Server 5.5

Products

• VMware Live Recovery
• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• In the virtual appliance management interface (VAMI) of the vSphere Replication Management Service (VRMS) appliance on both sides, the vSphere Replication (VR) service is stopped.
• Manually starting the VR service fails on the appliance.
• Restarting the VRMS appliance fails to start the VR service.
• The process identifier (PID) for the vSphere Replication service (/var/run/hms.pid) still exists on the VRMS appliance, even though the service is already stopped.
• The VRMS appliance is configured with an external database.
• The vCenter Server's SSL certificates changed during the upgrade to vSphere 5.5.
• In the hms.log file (located in /opt/vmware/hms/logs/ ), you see entries similar to:

Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified
at com.vmware.jvsl.run.CheckedRunnable.withoutChecked(CheckedRunnable.java:27)
at com.vmware.jvsl.run.VlsiRunnable$1.run(VlsiRunnable.java:102)
at com.vmware.jvsl.run.ExecutorRunnable.withExecutor(ExecutorRunnable.java:17)
at com.vmware.jvsl.run.VlsiRunnable.withClient(VlsiRunnable.java:96)
at com.vmware.jvsl.run.VcRunnable.withVc(VcRunnable.java:139)
at com.vmware.hms.cfg.VcExtensionChecker.checkThumbprint(VcExtensionChecker.java:746)
at com.vmware.hms.cfg.VcExtensionChecker.afterPropertiesSet(VcExtensionChecker.java:800)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFcctory.java:1514)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1452)
... 97 more
Caused by: java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain
not verified
at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:70)
at com.vmware.jvsl.run.VcRunnable.connect(VcRunnable.java:108)
at com.vmware.hms.cfg.VcExtensionChecker$2.connect(VcExtensionChecker.java:756)
at com.vmware.jvsl.run.VlsiRunnable$1$1.run(VlsiRunnable.java:106)
at com.vmware.jvsl.run.CheckedRunnable.withoutChecked(CheckedRunnable.java:19)
... 105 more
Caused by: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:224)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:131)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified
at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$HostnameVerifier.verify(ThumbprintTrustManager.java:217)
at org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:399)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.updateSecureConnection(DefaultClientConnectionOperator.java:167)
at org.apache.http.impl.conn.AbstractPoolEntry.layerProtocol(AbstractPoolEntry.java:275)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.layerProtocol(AbstractPooledConnAdapter.java:138)
at org.apache.http.impl.client.DefaultRequestDirector.establishRoute(DefaultRequestDirector.java:704)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:421)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:111)
... 3 more
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(Unknown Source)
at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$HostnameVerifier.verify(ThumbprintTrustManager.java:206)

Environment

VMware vCenter Server 5.5.x

Cause

This issue occurs because the vSphere Replication database contains the old vCenter Server certificate.

Resolution

To resolve this issue, re-configure the vSphere Replication Management Service appliance via the VAMI to obtain a new certificate from vCenter Server when it is powered on the next time.

To reconfigure VRMS:

1. Connect to the VAMI of the VRMS in a Web browser. The URL for the VAMI of the vSphere Replication server is https://VR_Server_Address:5480.
2. Log in to the virtual appliance
3. Click Configuration.
4. Click Save and Restart.
   
   Note: Do not change any configuration information.

The vSphere Replication appliance restarts and obtains a new vCenter Server certificate.