Active replications enter an error replication state due to stale host information in VMware Cloud Director Availability

Products

• VMware Cloud Director

Issue/Introduction

• In the Cloud Director Availability Provider Portal, you see replications with RPO Violations with the error:

Invalid or inaccessible datastore.

• In /var/log/vmware/hbrsrv.log on the destination Replicator, you see entries similar to:

2020-06-17T12:51:02.515+01:00 error hbrsrv[08281] [Originator@6876 sub=IO] HandshakeCb; <SSL(<io_obj p:0x00007f17f000fea8, h:110, <TCP 'xx.x.x.xx : 50144'>, <TCP 'xx.xx.xx.xxx : 80'>>)>; error: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:

--> PeerThumbprint: A1:xx:xx:xx:A8:20:xx:A7:xx:76:xx:56:xx:AC:xx:22:xx:xx:xx:F3:xx:8F:xx:7A:xx:xx:xx:xx:0D:xx:xx:xx

--> ExpectedThumbprint: xx:xx:F6:xx:xx:D9:xx:xx:xx:xx:xx:xx:xx:75:xx:0C:xx:xx:xx:xx:32:xx:xx:DF:70:xx:xx:xx:xx:xx:xx:xx

--> ExpectedPeerName: xx.xx.x.xx

--> The remote host certificate has these problems:

-->

--> * Host name does not match the subject name(s) in certificate.

-->

--> * unable to get local issuer certificate)

• In /var/log/hostd.log on the destination ESXi host, you see entries referencing the C4 disk of the replication job similar to:

2020-06-16T11:20:43.975Z info hostd[2124740] [Originator@6876 sub=DiskLib opID=hsl-8379887c-dc19 user=vpxuser] DISKLIB-LINK  : "/vmfs/volumes/5xxxxxx8-9xxxxxx6-0xxb-0xxxxxxxxxxb/C4-5xxxxxx4-bxx5-4xx2-axx6-cxxxxxxxxxxb/6xxxxxxa-fxx9-2xx9-1xxc-fxxxxxxxxxx0_CB-sxxe.vmdk" : failed to open (Failed to lock the file).

2020-06-16T11:20:43.975Z info hostd[2124740] [Originator@6876 sub=DiskLib opID=hsl-8379887c-dc19 user=vpxuser] DISKLIB-CHAIN : "/vmfs/volumes/5xxxxxx8-9xxxxxx6-0xxb-0xxxxxxxxxxb/C4-5xxxxxx4-bxx5-4xx2-axx6-cxxxxxxxxxxb/6xxxxxxa-fxx9-2xx9-1xxc-fxxxxxxxxxx0_CB-sxxe.vmdk" : failed to open (Failed to lock the file).

2020-06-16T11:20:43.976Z info hostd[2124740] [Originator@6876 sub=DiskLib opID=hsl-8379887c-dc19 user=vpxuser] DISKLIB-LIB   : Failed to open '/vmfs/volumes/5xxxxxx8-9xxxxxx6-0xxb-0xxxxxxxxxxb/C4-5xxxxxx4-bxx5-4xx2-axx6-cxxxxxxxxxxb/6xxxxxxa-fxx9-2xx9-1xxc-fxxxxxxxxxx0_CB-sxxe.vmdk' with flags 0x8 Failed to lock the file (16392).

 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Director Availability 4.x

Cause

This issue occurs when the trusted host information on the Cloud Replicator Appliance or Cloud Director Availability On-Premises Appliance is outdated or contains duplicated entries.

Resolution

This is a known issue affecting VMware Cloud Director Availability 4.x.

To resolve this issue, contact Broadcom Support and note this Article ID (315013) in the problem description. For more information, see Creating and managing Broadcom support cases

Workaround:
To work around this issue, pause and then resume any impacted replications to change their health status to green and resume their synchronizations.