Existing replications fail to synchronize after upgrading the Cloud Director Availability 4.x On-Premises Appliance

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Replications configured before the upgrade of the Cloud Director Availability On-Premises Appliance fail to synchronize.
• Configuring new replications from the upgraded Cloud Director Availability On-Premises Appliance synchronize successfully.
• In the /opt/vmware/h4/replicator/log/replicator.log file on the Cloud Replicator Appliance, you see entries similar to:

2020-10-29 08:11:20.841 DEBUG - [83adceb3-####-####-####-########654-IO] [job-58] c.v.h.r.replication.SyncSourceJob : Requesting manual sync for H4-1ec6e895-####-####-####-########796 2020-10-29 08:11:20.862 DEBUG - [83adceb3-####-####-####-########654-IO] [job-58] c.v.h.r.replication.SyncSourceJob : Requesting online instance for vm vm-97

• In the /opt/vmware/h4/lwdproxy/log/lwdproxy.log file on the Cloud Director Availability On-Premises Appliance, you see entries similar to:

2020-10-29 08:11:20,220 INFO [Worker-3-3] c.v.h.p.h.InitSessionHandler [InitSessionHandler.java:74] PeerId: null
2020-10-29 08:11:20,356 WARN [Worker-3-3] c.v.h.p.h.InitSessionHandler [InitSessionHandler.java:224] Handshake relay to server /127.0.0.1:8049 failed for group H4-########-####-####-####-########f796
javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:848)
        at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:811)

2020-10-29 08:11:20,356 WARN [Worker-3-3] c.v.h.p.u.TrafficCounter [TrafficCounter.java:65] Unknown counter: 005056ff########-######b1-####-####-####-########f37c-2fb784f0
2020-10-29 08:11:20,356 WARN [Worker-3-3] i.n.c.DefaultChannelPipeline [DefaultChannelPipeline.java:1152] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Director Availability 4.x

Cause

This issue occurs when the lightweight delta service certificate on the Cloud Director Availability On-Premises Appliance is regenerated due to the upgrade of the appliance, but this change doesn't get updated on the cloud site for existing replications.

Resolution

This is a known issue affecting Cloud Director Availability 4.x.

Currently there is no resolution.

Workaround:
To work around this issue, you need to reconfigure affected replications. To do this in the least disruptive way, you can perform the following steps.

1. Log into the Cloud Director Availability Portal.
2. Select an affected replication.
3. Click All actions.
4. Under Settings, click Replication settings.
5. Toggle the Compress replication traffic option.
6. Click Apply.
7. Click All actions again.
8. Under Settings, click Replication settings.
9. Toggle the Compress replication traffic option back to its original setting.
10. Click Apply.