Unable to synchronize existing On-Premises to Cloud replications after renewing the Cloud Director Availability On-Premises Appliance certificate

Products

• VMware Cloud Director

Issue/Introduction

In the Cloud Director Availability Portal, On-Premises to Cloud replications experience RPO violations. The replications display an overall health status of yellow, but maintain a green replication state. Creating new replications from the same On-Premises site completes successfully with an overall health of green.

In the /opt/vmware/h4/lwdproxy/log/lwdproxy.log file on the Source Cloud Replicator Appliance, you see entries similar to the following:

[DATE] WARN [Worker-3-8] c.v.h.p.u.TrafficCounter [TrafficCounter.java:65] Unknown counter: ################-########-########-################-########
[DATE] WARN [Worker-3-8] i.n.c.DefaultChannelPipeline [DefaultChannelPipeline.java:1152] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

In the /opt/vmware/h4/lwdproxy/log/lwdproxy.log file on the Destination Cloud Replicator Appliance, you see entries similar to the following:

[DATE] WARN [Worker-3-1] c.v.h.p.a.LwdAuthorizer [LwdAuthorizer.java:49] Access denied for client ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:## over GID:H4-########-####-####-####-############
[DATE] WARN [Worker-3-1] c.v.h.p.h.InitSessionHandler [InitSessionHandler.java:80] Access denied for LWDS peer ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:## over group H4-########-####-####-####-############ 
[DATE] WARN [Worker-3-1] c.v.h.p.u.TrafficCounter [TrafficCounter.java:65] Unknown counter: ################-########-########-################-########

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Director Availability 4.x

Cause

This issue occurs when the lightweight delta service certificate on the Cloud Director Availability On-Premises Appliance is regenerated, but this change does not update on the cloud site for existing replications.

Additionally, lwdproxy state synchronization only happens when all registered managers are online. If some managers are offline (e.g., during a temporary network outage or emergency stop), the lwdproxy state reset does not happen. As a result, some lwdproxy forwarding rules may linger, preventing consistency across components.

Resolution

Consistency will automatically be restored once you resolve all manager connectivity issues. This occurs when the temporary network outage goes away or by executing the repair manager or remove manager operation for each offline manager.

Workaround: To work around this issue immediately, you can reconfigure the affected replications by simply re-saving the replication settings.

1. Log in to the Cloud Director Availability Portal.

2. Select an affected replication.

3. Click All actions.

4. Under Settings, click Replication settings.

5. Click Apply to resave the settings.